From steve at pearwood.info Wed Jul 1 00:47:01 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 1 Jul 2009 08:47:01 +1000 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610906301219h3300d610jaf8e376b14de3d42@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <94bdd2610906301219h3300d610jaf8e376b14de3d42@mail.gmail.com> Message-ID: <200907010847.02606.steve@pearwood.info> On Wed, 1 Jul 2009 05:19:07 am Tarek Ziad? wrote: > 2009/6/30 Guido van Rossum : ... > > So what *is* the distutils-sig consensus? > > The consensus is to have one and only one way to install > distributions in Python, "One and only one way"? Shouldn't that be "one obvious way"? There is a big difference -- the first implies that Python should actively prohibit (somehow!) or at least inhibit any other install mechanisms, while the second implies that there should be a preferred, obvious, but not compulsory, mechanism. If you say "one and only one", and I take you at your word, then I can only assume that you want to actively prevent me from manually dropping .py files into my PYTHONPATH and having them work. I would guess that's probably not what you mean, but that's what it sounds like you're proposing. I don't like guessing -- would you please clarify what you mean? Some questions and comments regarding the PEP: Rationale: http://www.python.org/dev/peps/pep-0376/#id13 "There are too many ways to do it." Why is having multiple ways of adding distributions a problem to be solved? On my Linux system, I can add packages with rpm/yum, or I can compile them from source and manage them myself. I possibly even could (but never have!) install apt-get and use it to manage packages. Another rationale which should be added: There is no standard way of uninstalling distributions. How distributions are installed: http://www.python.org/dev/peps/pep-0376/#id14 "The problem is that many people use easy_install (from the setuptools project [4]) or pip [5] to install their packages, and these third-party tools do not install packages in the same way that Distutils does" Why is that a problem to be solved? Uninstall information: http://www.python.org/dev/peps/pep-0376/#id15 "Under some circumstances, you might not be able to know for sure that you have removed everything, or that you didn't break another distribution by removing a file that is shared among several distributions." I don't see how this proposal will help in the second case. If you install distribution Spam, containing file spam.py, and then install distribution Ham, which requires spam.py, what is to prevent you from removing Spam and breaking Ham? If you don't propose a solution for the dependency problem, you should say so. The RECORD format: http://www.python.org/dev/peps/pep-0376/#id19 "Each record is composed of three elements. ... the MD5 hash of the file, encoded in hex. Notice that pyc and pyo generated files will not have a hash because they are automatically produced from py files." What if your distribution is not a source distribution and only provides pyc and pyo files? -- Steven D'Aprano From ziade.tarek at gmail.com Wed Jul 1 01:34:53 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Jul 2009 01:34:53 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <200907010847.02606.steve@pearwood.info> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <94bdd2610906301219h3300d610jaf8e376b14de3d42@mail.gmail.com> <200907010847.02606.steve@pearwood.info> Message-ID: <94bdd2610906301634s439dacb3m76770b3c585407ba@mail.gmail.com> On Wed, Jul 1, 2009 at 12:47 AM, Steven D'Aprano wrote: > On Wed, 1 Jul 2009 05:19:07 am Tarek Ziad? wrote: >> 2009/6/30 Guido van Rossum : > ... >> > So what *is* the distutils-sig consensus? >> >> The consensus is to have one and only one way to install >> distributions in Python, > > "One and only one way"? Shouldn't that be "one obvious way"? > > There is a big difference -- the first implies that Python should > actively prohibit (somehow!) or at least inhibit any other install > mechanisms, while the second implies that there should be a preferred, > obvious, but not compulsory, mechanism. > > If you say "one and only one", and I take you at your word, then I can > only assume that you want to actively prevent me from manually > dropping .py files into my PYTHONPATH and having them work. I would > guess that's probably not what you mean, but that's what it sounds like > you're proposing. I don't like guessing -- would you please clarify > what you mean? one and only one way to install a distribution using its setup.py script, among the ways provided by easy_install, distutils and pip. e.g. avoiding having different format/location for the generated .egg.info; > > > Some questions and comments regarding the PEP: > > Rationale: > http://www.python.org/dev/peps/pep-0376/#id13 > > "There are too many ways to do it." > > Why is having multiple ways of adding distributions a problem to be > solved? On my Linux system, I can add packages with rpm/yum, or I can > compile them from source and manage them myself. I possibly even could > (but never have!) install apt-get and use it to manage packages. If you choose not to manually manage them, the packaging system you use is installing files in standard places where you expect to find them, and provides tools to know what is installed. Right now we don't have this standard in Python stdlib to interact with the distributions in a Python installation. And since packaging systems that are used out there could use the same standard, since we would like to make distutils a reference package for this kind of need, it seems like a good idea to define this standard in here. then each packaging system is free to implement its features on the top of it, ala wsgiref. > > Another rationale which should be added: > > There is no standard way of uninstalling distributions. > > > How distributions are installed: > http://www.python.org/dev/peps/pep-0376/#id14 > > "The problem is that many people use easy_install (from the setuptools > project [4]) or pip [5] to install their packages, and these > third-party tools do not install packages in the same way that > Distutils does" > > Why is that a problem to be solved? Being able to retrieve the metadata of an installed distribution, or the list of its installed files, no matter which tool installed it. > > > Uninstall information: > http://www.python.org/dev/peps/pep-0376/#id15 > > "Under some circumstances, you might not be able to know for sure that > you have removed everything, or that you didn't break another > distribution by removing a file that is shared among several > distributions." > > I don't see how this proposal will help in the second case. If you > install distribution Spam, containing file spam.py, and then install > distribution Ham, which requires spam.py, what is to prevent you from > removing Spam and breaking Ham? > > If you don't propose a solution for the dependency problem, you should > say so. This problem is solved as described later in the PEP, with the API that allows you to get the list of the distributions that use a given file. (thanks to the RECORD files) If Spam and Ham use smap.py, and if you uninstall Spam, this file will not be removed because the API will tell you its used in both distributions. > > > The RECORD format: > http://www.python.org/dev/peps/pep-0376/#id19 > > "Each record is composed of three elements. > ... > the MD5 hash of the file, encoded in hex. Notice that pyc and pyo > generated files will not have a hash because they are automatically > produced from py files." > > What if your distribution is not a source distribution and only provides > pyc and pyo files? > Good question, I have never created such distribution. Aren't they read-only files ? What happens if someone change them, do we really want to keep them since it breaks the distribution ? How they can be changed in the first place, without the source ? Tarek From ziade.tarek at gmail.com Wed Jul 1 01:46:00 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Jul 2009 01:46:00 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> Message-ID: <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> On Tue, Jun 30, 2009 at 10:06 PM, Scott David Daniels wrote: > Tarek Ziad? wrote: >> >> On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore wrote: >>> >>> [1] I'd actually like it if the PEP defined an uninstall command - >>> something like "python -m distutils.uninstall packagename". It can be >>> as minimalist as you like, but I'd like to see it present. >> >> it's already there: >> >> http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function > > That (at least as I read it) is a function, not a command. > If it is a command, give an example of its use from the command line > for us poor "don't want to research" people. ?If the following works: > > ? ?$ python setup.py uninstall some_package > > Then explicitly say so for us poor schlubs. > Right, I'll add that. Although it will be a reference implementation only. From ziade.tarek at gmail.com Wed Jul 1 01:49:58 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Jul 2009 01:49:58 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <4A4A7F88.7080609@gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <20090630174159.731CE3A4099@sparrow.telecommunity.com> <94bdd2610906301235g60bfea7dsb67a72a04213d8d0@mail.gmail.com> <4A4A7F88.7080609@gmail.com> Message-ID: <94bdd2610906301649l34525ab3h45c733e8bd60eb46@mail.gmail.com> On Tue, Jun 30, 2009 at 11:11 PM, Nick Coghlan wrote: >> >> >> previous formats will not be supported but that won't break anything >> of course since the new APIs will work only over the distribution >> installed with the new version of distutils. > > To address PJE's question in the PEP, it may be worth expanding on this > in the backwards compatibility section explaining how the new distutils > metadata system avoids getting confused by the old pre-standardisation > installation formats (e.g. it may be that the directory names and/or > filenames all deliberately differ from current approaches precisely so > they can coexist without interfering with each other) > I'll work on making it clearer, > Also, I find the following paragraph (near the start of the section > linked above) confusing: > > [PEP 376] >> Notice that this change is based on the standard proposed by >> EggFormats, although this standard proposes two ways to install >> files: >> >> * A self-contained directory that can be zipped or left unzipped and >> contains the distribution files and the .egg-info directory. >> * A distinct .egg-info directory located in the site-packages directory. > > This is unclear as to what "this standard" is referring to (since the > PEP itself is proposing a standard, but the sentence is also referring > to the existing EggFormats de facto standard). If the PEP only supports > the latter of the two options (which appears to be the case) it should > say so explicitly. ok > > Another minor nit from the same section: > > [PEP 376] >> Any '-' characters are currently replaced with '_'. > > This should say something like "Any '-' characters other than the one in > 'egg-info' and the one separating the name from the version number are > included in the replacement of non-alphanumeric characters with '_'" ok > > Other questions/comments: > > - What is a "local absolute path"? Absolute path I understand, relative > path I understand, but "local absolute" is a novel term to me. local means that the "/" separator that is used in the RECORD file for example, no matter what platform you are on, is translated using the local separator (/ or \) I'll make it clearer, Regards Tarek From ziade.tarek at gmail.com Wed Jul 1 02:07:11 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Jul 2009 02:07:11 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> Message-ID: <94bdd2610906301707t12d0e879hd7ccfb47abb8a710@mail.gmail.com> On Tue, Jun 30, 2009 at 10:11 PM, Paul Moore wrote: >> We said during the summit at Pycon that we wanted this feature in >> Distutils, (Guido said so) > > "We" in this context denotes the people at the summit. Please remember > that people who weren't there still have an opinion - and it may well > differ. I'm not saying that it *does* differ, just that the view of > the summit should not be viewed as conclusive (unless the way Python > is developed is very different from how I understand it). Sorry I didn't mean it this way. The summit was just the kickoff for this PEP, but with the strong desire to have a standard + some API for it. Also, notice that I put in "We" also the people that answered the survey I did before the summit. (http://tarekziade.wordpress.com/2009/03/26/packaging-survey-first-results/) I'll make other fixes in the PEP with the feedback you gave, thanks Tarek From pje at telecommunity.com Wed Jul 1 02:13:19 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 30 Jun 2009 20:13:19 -0400 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610906301634s439dacb3m76770b3c585407ba@mail.gmail.co m> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <94bdd2610906301219h3300d610jaf8e376b14de3d42@mail.gmail.com> <200907010847.02606.steve@pearwood.info> <94bdd2610906301634s439dacb3m76770b3c585407ba@mail.gmail.com> Message-ID: <20090701001021.F1C553A4099@sparrow.telecommunity.com> At 01:34 AM 7/1/2009 +0200, Tarek Ziad? wrote: >On Wed, Jul 1, 2009 at 12:47 AM, Steven D'Aprano wrote: > > I don't see how this proposal will help in the second case. If you > > install distribution Spam, containing file spam.py, and then install > > distribution Ham, which requires spam.py, what is to prevent you from > > removing Spam and breaking Ham? > > > > If you don't propose a solution for the dependency problem, you should > > say so. > >This problem is solved as described later in the PEP, with the API >that allows you to get the >list of the distributions that use a given file. (thanks to the RECORD files) > >If Spam and Ham use smap.py, and if you uninstall Spam, this file will >not be removed >because the API will tell you its used in both distributions. That's not the scenario he's talking about. He's talking about the case where Ham has an 'install_requires' of Spam. That is, a runtime dependency, not a shared file. >Good question, I have never created such distribution. >Aren't they read-only files ? setuptools' bdist_egg command has an option to exclude source from an .egg, but it doesn't do anything special with permissions. I don't think any other current install tools support source-free installation. From pje at telecommunity.com Wed Jul 1 02:21:18 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 30 Jun 2009 20:21:18 -0400 Subject: [Python-Dev] PEP 376 In-Reply-To: <4A4A7F88.7080609@gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <20090630174159.731CE3A4099@sparrow.telecommunity.com> <94bdd2610906301235g60bfea7dsb67a72a04213d8d0@mail.gmail.com> <4A4A7F88.7080609@gmail.com> Message-ID: <20090701001819.8ABDF3A4099@sparrow.telecommunity.com> At 07:11 AM 7/1/2009 +1000, Nick Coghlan wrote: >To address PJE's question in the PEP, it may be worth expanding on this >in the backwards compatibility section explaining how the new distutils >metadata system avoids getting confused by the old pre-standardisation >installation formats (e.g. it may be that the directory names and/or >filenames all deliberately differ from current approaches precisely so >they can coexist without interfering with each other) In particular, it should be explicitly noted that the API does not support .egg files (or directories), and thus cannot substitute for the pkg_resources API, as far as detecting installed packages. That is, users who need to unequivocally detect the presence/version of a project on sys.path, etc. will still have to use pkg_resources. (Because pkg_resources supports the format proposed by the PEP, but the PEP does not support .egg files and directories.) From ziade.tarek at gmail.com Wed Jul 1 02:19:53 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Jul 2009 02:19:53 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <20090701001021.F1C553A4099@sparrow.telecommunity.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <94bdd2610906301219h3300d610jaf8e376b14de3d42@mail.gmail.com> <200907010847.02606.steve@pearwood.info> <94bdd2610906301634s439dacb3m76770b3c585407ba@mail.gmail.com> <20090701001021.F1C553A4099@sparrow.telecommunity.com> Message-ID: <94bdd2610906301719u78734951reeb46bef4e1e302a@mail.gmail.com> 2009/7/1 P.J. Eby : > At 01:34 AM 7/1/2009 +0200, Tarek Ziad? wrote: >> >> On Wed, Jul 1, 2009 at 12:47 AM, Steven D'Aprano >> wrote: >> > I don't see how this proposal will help in the second case. If you >> > install distribution Spam, containing file spam.py, and then install >> > distribution Ham, which requires spam.py, what is to prevent you from >> > removing Spam and breaking Ham? >> > >> > If you don't propose a solution for the dependency problem, you should >> > say so. >> >> This problem is solved as described later in the PEP, with the API >> that allows you to get the >> list of the distributions that use a given file. (thanks to the RECORD >> files) >> >> If Spam and Ham use smap.py, and if you uninstall Spam, this file will >> not be removed >> because the API will tell you its used in both distributions. > > That's not the scenario he's talking about. ?He's talking about the case > where Ham has an 'install_requires' of Spam. ?That is, a runtime dependency, > not a shared file. Ah, right sorry I misunderstood... They are no plans to handle dependency installation / uninstallation / managment at distutils level. so if you remove Ham, it will not check what distributions use it. So yes, I'll add a note on this, That said, the APIs will be powerfull enough for a third-party package managers to handle this case by throwing for example a warning or an exception. From kevin at bud.ca Wed Jul 1 04:12:19 2009 From: kevin at bud.ca (Kevin Teague) Date: Tue, 30 Jun 2009 19:12:19 -0700 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> Message-ID: <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> On Jun 30, 2009, at 4:46 PM, Tarek Ziad? wrote: > On Tue, Jun 30, 2009 at 10:06 PM, Scott David > Daniels wrote: >> Tarek Ziad? wrote: >>> >>> On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore >>> wrote: >>>> >>>> [1] I'd actually like it if the PEP defined an uninstall command - >>>> something like "python -m distutils.uninstall packagename". It >>>> can be >>>> as minimalist as you like, but I'd like to see it present. >>> >>> it's already there: >>> >>> http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall- >>> function >> >> That (at least as I read it) is a function, not a command. >> If it is a command, give an example of its use from the command line >> for us poor "don't want to research" people. If the following works: >> >> $ python setup.py uninstall some_package >> >> Then explicitly say so for us poor schlubs. >> > > Right, I'll add that. Although it will be a reference implementation > only. > Uninstall as a command feels a little weird. Since "python setup.py [some-command]" implies that the setup.py contains information about the distribution that the command is being applied to. So instead of: $ python setup.py uninstall some_package It could just be: $ python setup.py uninstall Except then you'd need to have a complete distribution kicking around with which to run the "python setup.py uninstall" command just to tell the uninstall command the distribution name you want to uninstall. But then with the other uninstall format you could uninstall any distribution from within any other distribution, which is convenient, but weird ... e.g.: $ cd Spam-1.0/ $ python setup.py uninstall Foo Although even the other version of the command could do weird stuff: $ cd Spam-1.0/ $ python setup.py install $ cd ../Spam-2.0/ $ python setup.py uninstall Which would presumably remove the Spam 1.0 distribution when run from the 2.0 version of it! Or perhaps this command should only allow uninstall to be run from a distribution whose name and version match the one that it was installed from ... I dunno what the right solution is. My two-cents is either to punt and only include an uninstall function as currently proposed, or for only supporting some form of the "python setup.py uninstall" style since I would guess that the most common use-case for uninstall is: user downloads a distribution, runs "python setup.py install", tries out the package, decides they don't like package, then runs "python setup.py uninstall" to restore their python back to it's original state. For doing anything more complex than that, people should be encouraged to use another one of the existing tools for managing their distributions, IMHO. From Scott.Daniels at Acm.Org Wed Jul 1 05:06:34 2009 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Tue, 30 Jun 2009 20:06:34 -0700 Subject: [Python-Dev] PEP 376 In-Reply-To: <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> Message-ID: Kevin Teague wrote: > On Jun 30, 2009, at 4:46 PM, Tarek Ziad? wrote: >> On Tue, Jun 30, 2009 at 10:06 PM, Scott David >> Daniels wrote: >>> Tarek Ziad? wrote: >>>> On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore wrote: >>>>> [1] I'd actually like it if the PEP defined an uninstall command - >>>>> something like "python -m distutils.uninstall packagename". It can be >>>>> as minimalist as you like, but I'd like to see it present. >>>> it's already there: >>>> http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function >>> That (at least as I read it) is a function, not a command. >>> If it is a command, give an example of its use from the command line >>> for us poor "don't want to research" people. If the following works: >>> $ python setup.py uninstall some_package >>> Then explicitly say so for us poor schlubs. >> Right, I'll add that. Although it will be a reference implementation >> only. > Uninstall as a command feels a little weird.... > I dunno what the right solution is. My two-cents is either to punt and > only include an uninstall function as currently proposed, or for only > supporting some form of the "python setup.py uninstall" style.... But for us poor schlubs, we want you brilliant packagers to actually come to a hard decision. If you want approval either admit you have no solution in the PEP (and detail the issues that prevent a decision), provide a minimally acceptable command, or expect that nobody sees the value of what you propose.... --Scott David Daniels Scott.Daniels at Acm.Org From guido at python.org Wed Jul 1 06:09:35 2009 From: guido at python.org (Guido van Rossum) Date: Tue, 30 Jun 2009 21:09:35 -0700 Subject: [Python-Dev] PEP 376 In-Reply-To: References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> Message-ID: 2009/6/30 Guido van Rossum : > On Tue, Jun 30, 2009 at 5:32 AM, Nick Coghlan wrote: >> Martin v. L?wis wrote: >>>> If no one objects, I'd like to push PEP 376 in the "accepted" status >>>> and go ahead with its implementation, >>>> with continuous feedback at Distutils-SIG as we did to build it. >>> >>> I think this isn't quite the process. In the past, every PEP required >>> BDFL pronouncement, which you should now seek. >> >> Agreed. While Guido is highly likely to just accept the distutils-sig >> consensus on something like this, that doesn't eliminate the need for >> him to actually *say* that he is approving the PEP on that basis. > > Sure. :-) > > So what *is* the distutils-sig consensus? > > And is there consensus outside of it? (Remember the ipaddr debacle. > It's easy for people to miss an important PEP.) Looks like the discussion isn't quite over yet... I did read about a third of the PEP but still feel pretty lost about how it all fits together; I won't have time to read more until next week, probably. I made some minor edits for typos and grammar (being PEP editor and all :-). I noted an inconsistency: first you say that the RECORD file uses the excel dialect, but at the end of the same section you say it uses the default csv settings. Sounds like you need to delete one or the other. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fuzzyman at voidspace.org.uk Wed Jul 1 10:20:33 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 01 Jul 2009 09:20:33 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> Message-ID: <4A4B1C51.3060601@voidspace.org.uk> Kevin Teague wrote: > > On Jun 30, 2009, at 4:46 PM, Tarek Ziad? wrote: > >> On Tue, Jun 30, 2009 at 10:06 PM, Scott David >> Daniels wrote: >>> Tarek Ziad? wrote: >>>> >>>> On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore >>>> wrote: >>>>> >>>>> [1] I'd actually like it if the PEP defined an uninstall command - >>>>> something like "python -m distutils.uninstall packagename". It can be >>>>> as minimalist as you like, but I'd like to see it present. >>>> >>>> it's already there: >>>> >>>> http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function >>> >>> That (at least as I read it) is a function, not a command. >>> If it is a command, give an example of its use from the command line >>> for us poor "don't want to research" people. If the following works: >>> >>> $ python setup.py uninstall some_package >>> >>> Then explicitly say so for us poor schlubs. >>> >> >> Right, I'll add that. Although it will be a reference implementation >> only. >> > > Uninstall as a command feels a little weird. Since "python setup.py > [some-command]" implies that the setup.py contains information about > the distribution that the command is being applied to. So instead of: > > $ python setup.py uninstall some_package It could be: $ python -m distutils uninstall some_package Asymmetrical with the install of course. Michael > > It could just be: > > $ python setup.py uninstall > > Except then you'd need to have a complete distribution kicking around > with which to run the "python setup.py uninstall" command just to tell > the uninstall command the distribution name you want to uninstall. But > then with the other uninstall format you could uninstall any > distribution from within any other distribution, which is convenient, > but weird ... e.g.: > > $ cd Spam-1.0/ > $ python setup.py uninstall Foo > > Although even the other version of the command could do weird stuff: > > $ cd Spam-1.0/ > $ python setup.py install > $ cd ../Spam-2.0/ > $ python setup.py uninstall > > Which would presumably remove the Spam 1.0 distribution when run from > the 2.0 version of it! Or perhaps this command should only allow > uninstall to be run from a distribution whose name and version match > the one that it was installed from ... > > I dunno what the right solution is. My two-cents is either to punt and > only include an uninstall function as currently proposed, or for only > supporting some form of the "python setup.py uninstall" style since I > would guess that the most common use-case for uninstall is: user > downloads a distribution, runs "python setup.py install", tries out > the package, decides they don't like package, then runs "python > setup.py uninstall" to restore their python back to it's original > state. For doing anything more complex than that, people should be > encouraged to use another one of the existing tools for managing their > distributions, IMHO. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From ziade.tarek at gmail.com Wed Jul 1 16:32:23 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Jul 2009 16:32:23 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <4A4B5632.5020209@gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> <4A4B5632.5020209@gmail.com> Message-ID: <94bdd2610907010732p76fc39d7ia516c930fb5da987@mail.gmail.com> On Wed, Jul 1, 2009 at 2:27 PM, Nick Coghlan wrote: > However, having uninstall as a command supported by setup.py also makes > a certain amount of sense. > Yes, that could be an alias that just calls the uninstall global function using the distribution name. From ncoghlan at gmail.com Wed Jul 1 14:27:30 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 01 Jul 2009 22:27:30 +1000 Subject: [Python-Dev] PEP 376 In-Reply-To: <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> Message-ID: <4A4B5632.5020209@gmail.com> Kevin Teague wrote: > Uninstall as a command feels a little weird. Since "python setup.py > [some-command]" implies that the setup.py contains information about the > distribution that the command is being applied to. So instead of: > > $ python setup.py uninstall some_package > > It could just be: > > $ python setup.py uninstall I was actually thinking of something more along the lines of: python -m distutils uninstall (Older Pythons won't reliably let you execute a package like that, but 3.1+ and 2.7+ let you do it by including a __main__ module in the package) However, having uninstall as a command supported by setup.py also makes a certain amount of sense. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ziade.tarek at gmail.com Wed Jul 1 10:41:14 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Jul 2009 10:41:14 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0907010135n1a9d96abt8c3372e695a6f3ff@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <79990c6b0907010135n1a9d96abt8c3372e695a6f3ff@mail.gmail.com> Message-ID: <94bdd2610907010141w7340b1a9v8bd001938f15fb58@mail.gmail.com> On Wed, Jul 1, 2009 at 10:35 AM, Paul Moore wrote: > 2009/7/1 Tarek Ziad? : >>> That (at least as I read it) is a function, not a command. >>> If it is a command, give an example of its use from the command line >>> for us poor "don't want to research" people. ?If the following works: >>> >>> ? ?$ python setup.py uninstall some_package >>> >>> Then explicitly say so for us poor schlubs. >>> >> >> Right, I'll add that. Although it will be a reference implementation only. > > -1. Where does the setup.py file come from? If I have docutils > installed, and want to remove it, must I download the source again so > that I can get the setup.py, so I can run the uninstall? This is daft > - particularly given that the point of PEP 376 is to ensure that all > of the required information is available from the installed package! > > As I suggested before: > > ? ?python -m distutils.uninstall packagename yes sorry if it was unclear, I was not thinking about adding something based on setup.py, but just saying that I was going to add this feature in the PEP. and it will be of the form: python -m distutils.uninstall packagename > > Calling it a "reference implementation" should not imply that it's not > built to be usable and complete. If it's there,m people should be able > to use it. It will be usable and complete, but very limited. As someone mentioned, it will not take care of dependencies and prevent you from removing a distribution that is mentioned in another distrubution in a setuptools' install_requires metadata. That said, when PEP 345 evolves like we have planned to (adding install_requires in the metadata) Then we should be able to provide this kind of warning with no pain. From ziade.tarek at gmail.com Wed Jul 1 15:34:01 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Jul 2009 15:34:01 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0907010537r29d46f40y3b682273346abee8@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <79990c6b0907010521q392b73d8ne3834592b57002af@mail.gmail.com> <79990c6b0907010537r29d46f40y3b682273346abee8@mail.gmail.com> Message-ID: <94bdd2610907010634v4581fadfr97b20067658262b7@mail.gmail.com> Right, the API part is almost empty, http://docs.python.org/distutils/apiref.html I'll complete it for the relevant part. On Wed, Jul 1, 2009 at 2:37 PM, Paul Moore wrote: > 2009/7/1 Paul Moore : >> One other thought. You haven't documented the DistributionMetaData class. The > > Just noticed that it's defined in distutils. But it's not documented there :-( > > Maybe just add a bit to the PEP saying that the class exists in > distutils, give a brief summary, and say that as part of implimenting > the PEP it will be formally documented. (Even though distutils is a > mess of undocumented functionality, I don't think that you can base > new work on undocumented internals - you should document them first, > or things will never get any better :-() > > Paul. > -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Wed Jul 1 16:29:56 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Jul 2009 16:29:56 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> Message-ID: <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com> On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore wrote: > "The problem is that many people use easy_install or pip to install > their packages..." > > As already noted by others, it's not clear why this is a problem. But > worse is the fact that the paragraph reads to me as saying that > easy_install/pip/etc don't follow the *current* standard structure. > Given this fact (which *is* a problem!) I fail to see why I should > expect them to follow any *new* standard - so how does this PEP > actually address the issue? Rather, it seems to say to me that the > existing tools have a history of ignoring the standard approach, so > this PEP is going to be useless :-( > > [I expect that isn't what you're trying to say, so you may just need > to clarify your meaning. But I do think it's important to address the > question of how this PEP is going to ensure that existing tools adapt. > In particular, setuptools seems to have completely stagnated, so I see > very little likelihood that setuptools is going to change to conform > to the PEP - how does that affect things?] Right it's unclear, I'll work on this part. To resume : - Phase 1 : introduction of the egg-info file in distutils Philipp introduced the creation of a file named xxx.egg-info file in 2006 (see http://bugs.python.org/issue1459476) alongside distutils-installed package, that contains the metadata of the distribution. - Phase 2: two new formats in the setuptools project Then he created two new formats in the setuptools project: 1. an .egg-info directory containing a PKG-INFO file, which is similar to the previous .egg-info file. This happened because other files were added in that directory. 2. an .egg directory, possibly zipped, that is a self-contained version of the distribution, containing the metadata and all other files. Setuptools uses 2. by default. There's an option when you use setuptools to force 1. (?single-version-externally-managed *or* --root). - Phase 3: adopting a unified standard. It occurs that having a .egg-info directory (1.) is way better than a single file because it's a place-holder for other files. It is also adopted by the community when it comes to install setuptools-based package.: - This is what "pip" uses to install packages in a more flat manner. - It's also the policy under debian (http://wiki.debian.org/DebianPython/NewPolicy) - and under Fedora (http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Setuptools) The .egg directory (2.) is more controversial because it a self-contained directory that doesn't install packages so it makes two different ways to work with distributions for packagers. So I have proposed in the PEP to adopt the standalone .egg-info directory as the new distutils unified standard. This means that all the third-party tools out there already conform to that standard, and that packages installed in other formats will not benefit from the new APIs. which means that people that want to work with distributions installed as .egg directories will have to use setuptools APIs. Which makes sense. > > ".egg-info becomes a directory" > > Don't refer to the setuptools documentation (EggFormats)! It is > obscure and user-hostile, as well as not actually being the same as > the PEP's proposal. Rather, just document the structure of .egg-info. > If you want, you can then add a cross-reference note, saying something > like "The setuptools structure, as proposed in the EggFormats > documentation for that package [ref], is a subset of this standard. In > order to conform to this PEP, setuptools will have to be amended to > only install .egg-info directories in the format defined by this PEP". I'll work that way. > > "However, it will impact the setuptools and pip projects, but given > the fact that..." > > Confusing. Will these tools need to change (I believe so) or not? If > they will need to change, that hardly counts as "no deep consequences" > - there's the whole backward compatibility issue for them to handle. I'll add this in a backward compatibility section, as suggested earlier by someone. > > "Adding a RECORD file" > > "The RECORD format" > > "Adding an INSTALLER file in the .egg-info directory" These section will have more details about the way they interact with bdist_* for the rest of the mail, I'll clarify the rest of the PEP using your feedback; > "New APIs in pkgutil" > > You say "the best place to put these APIS seems to be pkgutil". You > should be more definite - "these APIs will be added to the pkgutil > module". ok > General rule - don't document (and commit yourself to) any internal > details that the user can't access from the public API. It just makes > backward compatibility harder to maintain. The purpose is to provide this documentation to third-party projects that want to implement a packaging system on the top of these classes. Maybe this should be removed from the PEP and but into another document targeted to developers ? > > "Adding an uninstall function" > > One point - what happens if the uninstall method can't remove a file > or directory? Will it leave things in a broken/partially installed > state? This needs to be documented (although huge robustness and > recovery measures don't need to be taken in a simple reference > implementation). This is likely to be a somewhat common issue on > Windows in particular, where you can't delete open files. Good question. I didn't go that far yet because I didn't implement that part in the prototype. Maybe the files could be moved to a temporary place before they are all deleted, allowing to cancel the uninstallation in case a file couldn't be moved. > "Installer marker" > > There are times when I would imagine it would be useful to force > uninstallation of the basic package, even if that leaves stuff behind > related to a particular installer. A force=True argument would be the > easiest, but if you'd rather not add that, it would be useful to > document the workaround: > > ? ?inst = get_distribution('docutils').get_egginfo_file('INSTALLER').read() > ? ?uninstall('docutils', installer=inst) Ok, > > If nothing else, it shows a simple example of how to use the APIs. A > few more such examples in the PEP might be nice. > > But as I said, overall, I'm much happier with the PEP now that I've > read it through and taken the time to digest it. Great thanks for the feedback. Tarek From p.f.moore at gmail.com Wed Jul 1 13:44:32 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 1 Jul 2009 12:44:32 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> Message-ID: <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> 2009/6/30 Paul Moore : > Thank you. I'll try to make the time to go through the PEP and comment > more fully. OK, I've now read the PEP through in full. My comments follow. I have deliberately avoided mentioning points that others have already raised, to keep things shorter. First of all, after I got over the use of setuptools terminology, the document is very good. It's clear what it's trying to achieve, and covers most of the points I'd want it to. Occasionally the presentation is confusing, see below, but I think everything was there. I do still feel that the setuptools focus will put some readers off. The introductory remarks, in particular, assume a reasonable level of familiarity from readers about setuptools, easy_install, pip and the like. Not having that familiarity isn't a problem in fact, but the assumption is there in the tone. (For example, personally, I hate the unnecessarily cute term "egg", but it's established by now, so that's a lost cause :-() Onto the PEP. "The problem is that many people use easy_install or pip to install their packages..." As already noted by others, it's not clear why this is a problem. But worse is the fact that the paragraph reads to me as saying that easy_install/pip/etc don't follow the *current* standard structure. Given this fact (which *is* a problem!) I fail to see why I should expect them to follow any *new* standard - so how does this PEP actually address the issue? Rather, it seems to say to me that the existing tools have a history of ignoring the standard approach, so this PEP is going to be useless :-( [I expect that isn't what you're trying to say, so you may just need to clarify your meaning. But I do think it's important to address the question of how this PEP is going to ensure that existing tools adapt. In particular, setuptools seems to have completely stagnated, so I see very little likelihood that setuptools is going to change to conform to the PEP - how does that affect things?] ".egg-info becomes a directory" Don't refer to the setuptools documentation (EggFormats)! It is obscure and user-hostile, as well as not actually being the same as the PEP's proposal. Rather, just document the structure of .egg-info. If you want, you can then add a cross-reference note, saying something like "The setuptools structure, as proposed in the EggFormats documentation for that package [ref], is a subset of this standard. In order to conform to this PEP, setuptools will have to be amended to only install .egg-info directories in the format defined by this PEP". "However, it will impact the setuptools and pip projects, but given the fact that..." Confusing. Will these tools need to change (I believe so) or not? If they will need to change, that hardly counts as "no deep consequences" - there's the whole backward compatibility issue for them to handle. "Adding a RECORD file" You say "at installation time" - please clarify. Do you only mean setup.py install? What about bdist_wininst and bdist_msi? What about third party bdist style tools? How will they ensure they get a RECORD file? "The RECORD format" The line separator shouldn't be os-dependent. What value is used for a pure-python (ie, platform independent) package? Unless the file is generated when the install is done, in which case see the previous point... Absolute file paths - this implies that RECORD has to be generated by the installer (which is the only place that knows absolute paths) which means that every bdist_xxx installer has to write its own RECORD file. Does the PEP offer no support for this? In any case, the bdist_wininst and bdist_msi code (which is core distutils) will need to be amended to manage RECORD files appropriately. "Adding an INSTALLER file in the .egg-info directory" Same question again - does the PEP offer supporting APIs to help bdist_xxx installers do this? What about the core bdist_xxx commands? "New APIs in pkgutil" You say "the best place to put these APIS seems to be pkgutil". You should be more definite - "these APIs will be added to the pkgutil module". The Distribution/ZippedDistribution and DistributionDir/ZippedDistributionDir pairs seem to imply that users need to explicitly instantiate these classes (and hence know whether a distribution is zipped). This is cleared up later, but you should add a note here - "Users will not need to create instances of these classes manually, they are returned by the public functions in pkgutil, such as get_distributions()". "DistributionDirMap class" This seems to be an internal implementation detail, and as such should not be documented. It's never returned from any of the public APIs, and the only created instance is a hidden internal global instance. The whole part about purging the cache is also unnecessary - the user has no interface to allow them to do this, so you don't need to document that it's possible. Actually, (Zipped)DistributionDir instances are never returned to the user via the public API, either - so these don't need to be documented. General rule - don't document (and commit yourself to) any internal details that the user can't access from the public API. It just makes backward compatibility harder to maintain. "Adding an uninstall function" With regard to my earlier comments, my apologies - I misread the text and assumed the documented function simply returned the files to remove, but didn't do the removal itself. (Which makes it far more useless than it actually is). One point - what happens if the uninstall method can't remove a file or directory? Will it leave things in a broken/partially installed state? This needs to be documented (although huge robustness and recovery measures don't need to be taken in a simple reference implementation). This is likely to be a somewhat common issue on Windows in particular, where you can't delete open files. "Installer marker" There are times when I would imagine it would be useful to force uninstallation of the basic package, even if that leaves stuff behind related to a particular installer. A force=True argument would be the easiest, but if you'd rather not add that, it would be useful to document the workaround: inst = get_distribution('docutils').get_egginfo_file('INSTALLER').read() uninstall('docutils', installer=inst) If nothing else, it shows a simple example of how to use the APIs. A few more such examples in the PEP might be nice. But as I said, overall, I'm much happier with the PEP now that I've read it through and taken the time to digest it. Paul. From p.f.moore at gmail.com Wed Jul 1 16:56:04 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 1 Jul 2009 15:56:04 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com> Message-ID: <79990c6b0907010756x1794cc3at16fa05a8b993ebfa@mail.gmail.com> 2009/7/1 Tarek Ziad? : > Right it's unclear, I'll work on this part. > > To resume : > > - Phase 1 : introduction of the egg-info file in distutils > > Philipp introduced the creation of a file named xxx.egg-info file in 2006 > (see http://bugs.python.org/issue1459476) ?alongside > distutils-installed package, that contains > the metadata of the distribution. > > - Phase 2: ?two new formats in the setuptools project > > Then he created two new formats in the setuptools project: > > 1. an .egg-info directory containing a PKG-INFO file, which is similar > to the previous .egg-info file. This happened > ? because other files were added in that directory. > > 2. an .egg directory, possibly zipped, that is a self-contained > version of the distribution, containing the metadata and all other > files. > > Setuptools uses 2. by default. There's an option when you use > setuptools to force 1. (?single-version-externally-managed *or* > --root). > > - Phase 3: adopting a unified standard. > > It occurs that having a .egg-info directory (1.) is way better than a > single file because it's a place-holder for other files. > It is also adopted by the community when it comes to install > setuptools-based package.: > > - This is what "pip" uses to install packages in a more flat manner. > - It's also the policy under debian > (http://wiki.debian.org/DebianPython/NewPolicy) > - and under Fedora > (http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Setuptools) > > The .egg directory (2.) is more controversial because it a > self-contained directory that doesn't install packages > so it makes two different ways to work with distributions for packagers. > > So I have proposed in the PEP to adopt the standalone .egg-info > directory as the new distutils unified standard. > > This means that all the third-party tools out there already conform to > that standard, and that packages installed > in other formats will not benefit from the new APIs. which means that > people that want to work with distributions installed as .egg > directories will have to use setuptools APIs. > Which makes sense. Ah, that makes a *lot* more sense. So, in a "compatibility" section, you could point out that the proposed standard is compatible with setuptools "single version externally managed" format, and that the setuptools multi-version format (the egg file/directory) is a non-standard format designed to allow multiple versions to be installed at once - something out of scope for this PEP. With that explanation, people like me who glaze over at the complexities of setuptools scenarios should be able to see what's going on. Thanks for clarifying. >> General rule - don't document (and commit yourself to) any internal >> details that the user can't access from the public API. It just makes >> backward compatibility harder to maintain. > > The purpose is to provide this documentation to third-party projects that want > to implement a packaging system on the top of these classes. Hmm, but you don't explain how they should do that. I certainly wouldn't know how to. As the public API is pkgutil.get_distribution (and the like) which isn't a class so cannot be subclassed, it's not clear how a 3rd party could hook in a subclass of (Zipped)Distribution. And as there's nothing in the public API that consumes these classes, creating them by hand is of no use, either. > Maybe this should be removed from the PEP and but into another > document targeted to developers ? No. If it's needed by developers, it should be defined in the PEP, not elsewhere. But it should be defined in a way that developers can use - not just for the sake of defining it. Unless there is a concrete use case (which should be stated as an example in the PEP) then *not* documenting the classes is of more help to developers, as they don't have to follow a definition that isn't used. Paul. From p.f.moore at gmail.com Wed Jul 1 14:21:41 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 1 Jul 2009 13:21:41 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> Message-ID: <79990c6b0907010521q392b73d8ne3834592b57002af@mail.gmail.com> 2009/7/1 Paul Moore : > 2009/6/30 Paul Moore : >> Thank you. I'll try to make the time to go through the PEP and comment >> more fully. > > OK, I've now read the PEP through in full. My comments follow. One other thought. You haven't documented the DistributionMetaData class. The And a minor nit: """ Distribution class A new class called Distribution is created with a the path of the .egg-info directory provided to the contructor. It reads the metadata contained in PKG-INFO when it is instanciated. """ instantiated Paul. From rdmurray at bitdance.com Wed Jul 1 15:39:09 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 1 Jul 2009 09:39:09 -0400 (EDT) Subject: [Python-Dev] PEP 376 In-Reply-To: References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> Message-ID: On Tue, 30 Jun 2009 at 20:06, Scott David Daniels wrote: > Kevin Teague wrote: >> On Jun 30, 2009, at 4:46 PM, Tarek Ziad? wrote: >> > On Tue, Jun 30, 2009 at 10:06 PM, Scott David >> > Daniels wrote: >> > > Tarek Ziad? wrote: >> > > > On Tue, Jun 30, 2009 at 8:37 PM, Paul Moore >> > > > wrote: >> > > > > [1] I'd actually like it if the PEP defined an uninstall command - >> > > > > something like "python -m distutils.uninstall packagename". It can >> > > > > be >> > > > > as minimalist as you like, but I'd like to see it present. >> > > > it's already there: >> > > > http://www.python.org/dev/peps/pep-0376/#adding-an-uninstall-function >> > > That (at least as I read it) is a function, not a command. >> > > If it is a command, give an example of its use from the command line >> > > for us poor "don't want to research" people. If the following works: >> > > $ python setup.py uninstall some_package >> > > Then explicitly say so for us poor schlubs. >> > Right, I'll add that. Although it will be a reference implementation >> > only. >> Uninstall as a command feels a little weird.... >> I dunno what the right solution is. My two-cents is either to punt and >> only include an uninstall function as currently proposed, or for only >> supporting some form of the "python setup.py uninstall" style.... > > But for us poor schlubs, we want you brilliant packagers to actually > come to a hard decision. If you want approval either admit you have no > solution in the PEP (and detail the issues that prevent a decision), > provide a minimally acceptable command, or expect that nobody sees the > value of what you propose.... I haven't read the PEP in detail since it's outside my area of interest and knowledge(*), but my understanding of the goal is that the PEP is providing an _infrastructure_ for system-level package management tools. The uninstall function is part of that infrastructure, but since distutils isn't a package manager itself (it's an install program), distutils as currently organized can't really handle uninstall except as outlined in a section you may have clipped from the above context (ie: when setup.py from the original package is available). A possible implementation at the python/distutils level might be to have a 'pyuninstall' command installed (like pydoc et al are installed) which handles uninstallation. The question is what do the people who do real package management (linux distribution level package management and the equivalent) think? I'm guessing they are happy with just having the function for their package management tools to call when needed, since (I'm hoping) they are part of the distutils sig.... So, if my understanding of the overall goal is correct, it looks to me like the PEP is missing a "motivation" section that talks about system package managers. --David (*) I'll make time to read it anyway soon. From ziade.tarek at gmail.com Wed Jul 1 10:32:15 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Jul 2009 10:32:15 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <4A4B1C51.3060601@voidspace.org.uk> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> <4A4B1C51.3060601@voidspace.org.uk> Message-ID: <94bdd2610907010132ib0c6dcat6a7cfce1d40b6ed7@mail.gmail.com> On Wed, Jul 1, 2009 at 10:20 AM, Michael Foord wrote: >> >> Uninstall as a command feels a little weird. Since "python setup.py >> [some-command]" implies that the setup.py contains information about the >> distribution that the command is being applied to. So instead of: >> >> $ python setup.py uninstall some_package > > It could be: > > ? $ python -m distutils uninstall some_package > > Asymmetrical with the install of course. > Yes exactly, I was going to add: $ python -m distutils.uninstall some_package The whole point of the RECORD file is to be able to uninstall without depending on the original archive used to install From jcea at jcea.es Wed Jul 1 19:26:18 2009 From: jcea at jcea.es (Jesus Cea) Date: Wed, 01 Jul 2009 19:26:18 +0200 Subject: [Python-Dev] [Fwd: [issue6397] Implementing Solaris "poll" in the "select" module] Message-ID: <4A4B9C3A.4090606@jcea.es> -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -------------- next part -------------- An embedded message was scrubbed... From: =?utf-8?b?SmVzw7pzIENlYSBBdmnDs24=?= Subject: [issue6397] Implementing Solaris "poll" in the "select" module Date: Wed, 01 Jul 2009 16:49:15 +0000 Size: 2187 URL: From brett at python.org Thu Jul 2 01:21:56 2009 From: brett at python.org (Brett Cannon) Date: Wed, 1 Jul 2009 16:21:56 -0700 Subject: [Python-Dev] I am back Message-ID: Assuming Mailman flipped the right bits to start delivering mail to me again, my Python sabbatical is now over. Anything happen while I was gone that I should be aware of that is not covered in a PEP? -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Jul 1 14:37:36 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 1 Jul 2009 13:37:36 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0907010521q392b73d8ne3834592b57002af@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <79990c6b0907010521q392b73d8ne3834592b57002af@mail.gmail.com> Message-ID: <79990c6b0907010537r29d46f40y3b682273346abee8@mail.gmail.com> 2009/7/1 Paul Moore : > One other thought. You haven't documented the DistributionMetaData class. The Just noticed that it's defined in distutils. But it's not documented there :-( Maybe just add a bit to the PEP saying that the class exists in distutils, give a brief summary, and say that as part of implimenting the PEP it will be formally documented. (Even though distutils is a mess of undocumented functionality, I don't think that you can base new work on undocumented internals - you should document them first, or things will never get any better :-() Paul. From ncoghlan at gmail.com Wed Jul 1 14:37:07 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 01 Jul 2009 22:37:07 +1000 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610906301649l34525ab3h45c733e8bd60eb46@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <20090630174159.731CE3A4099@sparrow.telecommunity.com> <94bdd2610906301235g60bfea7dsb67a72a04213d8d0@mail.gmail.com> <4A4A7F88.7080609@gmail.com> <94bdd2610906301649l34525ab3h45c733e8bd60eb46@mail.gmail.com> Message-ID: <4A4B5873.1050500@gmail.com> Tarek Ziad? wrote: >> - What is a "local absolute path"? Absolute path I understand, relative >> path I understand, but "local absolute" is a novel term to me. > > local means that the "/" separator that is used in the RECORD file for example, > no matter what platform you are on, is translated using the local separator > (/ or \) > > I'll make it clearer, OK, I understand now. That makes Paul's questions about how to correctly generate the RECORD file for bdist_xxx installations all the more relevant though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From eric.pruitt at gmail.com Thu Jul 2 02:47:15 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Wed, 1 Jul 2009 19:47:15 -0500 Subject: [Python-Dev] Bytes, Strings, Encoding In-Reply-To: <171e8a410907010817j715f1d5ye638ea4b2469afd9@mail.gmail.com> References: <171e8a410907010817j715f1d5ye638ea4b2469afd9@mail.gmail.com> Message-ID: <171e8a410907011747i461a409ew9253e3fc82a10acd@mail.gmail.com> Hello, I am working on the subprocess.Popen module for Python 2.7 and am now moving my changes over to Python 3.1 however I am having trouble with the whole byte situation and I can't quite seem to understand how to go back and forth between bytes and strings. I am also looking for the Python 3k equivalent for the Python 2.X built-in buffer class. One version of the file with my modifications can be found here < http://code.google.com/p/subprocdev/source/browse/subprocess.py?spec=svn5b570f8cbfcaae859091eb01b21b183aa5221af9&r=5b570f8cbfcaae859091eb01b21b183aa5221af9>. Lines 1 - 15 are me attempting to get around certain changes between Python 3.0 and Python 2.7. Further down on line 916, we have the function "send" and "recv" in which I am having the most trouble with bytes and strings. Any help is appreciated. Feel free to comment on my blog http://subdev.blogspot.com/ or reply to the last. Thanks in advance, Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Thu Jul 2 03:48:52 2009 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 1 Jul 2009 20:48:52 -0500 Subject: [Python-Dev] Bytes, Strings, Encoding In-Reply-To: <171e8a410907011747i461a409ew9253e3fc82a10acd@mail.gmail.com> References: <171e8a410907010817j715f1d5ye638ea4b2469afd9@mail.gmail.com> <171e8a410907011747i461a409ew9253e3fc82a10acd@mail.gmail.com> Message-ID: <1afaf6160907011848h122cc2ebp964d225d453052c1@mail.gmail.com> 2009/7/1 Eric Pruitt : > Hello, > > I am working on the subprocess.Popen module for Python 2.7 and am now moving > my changes over to Python 3.1 however I am having trouble with the whole > byte situation and I can't quite seem to understand how to go back and forth > between bytes and strings. I am also looking for the Python 3k equivalent > for the Python 2.X built-in buffer class. Look at the "memoryview" type as a replacement for buffer. io.Buffered* is not the same. > > One version of the file with my modifications can be found here? < > http://code.google.com/p/subprocdev/source/browse/subprocess.py?spec=svn5b570f8cbfcaae859091eb01b21b183aa5221af9&r=5b570f8cbfcaae859091eb01b21b183aa5221af9 >>. Lines 1 - 15 are me attempting to get around certain changes between > Python 3.0 and Python 2.7. Further down on line 916, we have the function > "send" and "recv" in which I am having the most trouble with bytes and > strings. You translate between bytes and strings through encoding and decoding. Have you read https://secure.wikimedia.org/wikipedia/en/wiki/Unicode? > > Any help is appreciated. Feel free to comment on my blog > http://subdev.blogspot.com/ or reply to the last. Have you talked with your mentor about these things? -- Regards, Benjamin From guido at python.org Thu Jul 2 04:05:21 2009 From: guido at python.org (Guido van Rossum) Date: Wed, 1 Jul 2009 19:05:21 -0700 Subject: [Python-Dev] [Fwd: [issue6397] Implementing Solaris "poll" in the "select" module] In-Reply-To: <4A4B9C3A.4090606@jcea.es> References: <4A4B9C3A.4090606@jcea.es> Message-ID: The select module already supports the poll() system call. Or is there a special variant that only Solaris has? 2009/7/1 Jesus Cea : > > -- > Jesus Cea Avion ? ? ? ? ? ? ? ? ? ? ? ? _/_/ ? ? ?_/_/_/ ? ? ? ?_/_/_/ > jcea at jcea.es - http://www.jcea.es/ ? ? _/_/ ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/ > jabber / xmpp:jcea at jabber.org ? ? ? ? _/_/ ? ?_/_/ ? ? ? ? ?_/_/_/_/_/ > . ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?_/_/ ?_/_/ ? ?_/_/ ? ? ? ? ?_/_/ ?_/_/ > "Things are not so easy" ? ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/ > "My name is Dump, Core Dump" ? _/_/_/ ? ? ? ?_/_/_/ ? ? ?_/_/ ?_/_/ > "El amor es poner tu felicidad en la felicidad de otro" - Leibniz > > > ---------- Forwarded message ---------- > From:?"Jes?s Cea Avi?n" > To:?jcea at jcea.es > Date:?Wed, 01 Jul 2009 16:49:15 +0000 > Subject:?[issue6397] Implementing Solaris "poll" in the "select" module > > New submission from Jes?s Cea Avi?n : > > In Python 2.6 we added support for Linux "epoll" and *BSD "kqueue" in > the select module. I think we should add support for Solaris "poll" > interface too. > > What do you think?. > > I volunteer to do the work, if you agree this is a feature we want to > have. I think so. > > ---------- > assignee: jcea > components: Library (Lib) > messages: 89989 > nosy: jcea > severity: normal > stage: needs patch > status: open > title: Implementing Solaris "poll" in the "select" module > type: feature request > versions: Python 2.7, Python 3.2 > > _______________________________________ > Python tracker > > _______________________________________ > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Wed Jul 1 07:30:44 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 01 Jul 2009 07:30:44 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> Message-ID: <4A4AF484.7070600@v.loewis.de> > I dunno what the right solution is. I feel that it is straight forward. Either provide a /usr/bin/python-uninstall utility, or arrange to make python -mdistutils.uninstall work, so one would do python -mdistutils.uninstall some_package Regards, Martin From aahz at pythoncraft.com Thu Jul 2 04:32:51 2009 From: aahz at pythoncraft.com (Aahz) Date: Wed, 1 Jul 2009 19:32:51 -0700 Subject: [Python-Dev] I am back In-Reply-To: References: Message-ID: <20090702023251.GA28507@panix.com> On Wed, Jul 01, 2009, Brett Cannon wrote: > > Anything happen while I was gone that I should be aware of that is not > covered in a PEP? Yes. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "as long as we like the same operating system, things are cool." --piranha From pje at telecommunity.com Thu Jul 2 04:41:52 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 01 Jul 2009 22:41:52 -0400 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com > References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com> Message-ID: <20090702023855.F00293A4109@sparrow.telecommunity.com> At 04:29 PM 7/1/2009 +0200, Tarek Ziad? wrote: >- Phase 1 : introduction of the egg-info file in distutils > >Philipp introduced the creation of a file named xxx.egg-info file in 2006 >(see http://bugs.python.org/issue1459476) alongside >distutils-installed package, that contains >the metadata of the distribution. > >- Phase 2: two new formats in the setuptools project > >Then he created two new formats in the setuptools project: This is backwards. The .egg-info file was added to distutils and setuptools *after* support for the other formats. In other words, it was a third format added to allow other software (such as setuptools and friends) to detect the presence of packages installed via the distutils, and thereby prevent other tools from installing duplicates. (.egg-info directories and .egg files/directories already existed.) >This means that all the third-party tools out there already conform to >that standard, and that packages installed >in other formats will not benefit from the new APIs. which means that >people that want to work with distributions installed as .egg >directories will have to use setuptools APIs. >Which makes sense. Yes and no. Not providing uninstall support is reasonable, but the PEP also has features to query packages in general. (There's also no technical reason why comparable manifest and uninstall support can't be provided for .egg files and directoriees, since they already have an implicit manifest: their contents. However, since I'm not currently possessed of the time to provide a patch myself, I'm not going to lobby for this as a feature.) > > If you want, you can then add a cross-reference note, saying something > > like "The setuptools structure, as proposed in the EggFormats > > documentation for that package [ref], is a subset of this standard. In > > order to conform to this PEP, setuptools will have to be amended to > > only install .egg-info directories in the format defined by this PEP". > >I'll work that way. (Technically, it's an extension of a subset of EggFormats, since it adds RECORD and INSTALLER.) > > > > "However, it will impact the setuptools and pip projects, but given > > the fact that..." > > > > Confusing. Will these tools need to change (I believe so) or not? If > > they will need to change, that hardly counts as "no deep consequences" > > - there's the whole backward compatibility issue for them to handle. > >I'll add this in a backward compatibility section, as suggested >earlier by someone. They don't need to change, unless they're going to support being uninstalled by the tool (in which case they'll need to add RECORD and INSTALLER). From p.f.moore at gmail.com Wed Jul 1 10:47:07 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 1 Jul 2009 09:47:07 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610907010141w7340b1a9v8bd001938f15fb58@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <79990c6b0907010135n1a9d96abt8c3372e695a6f3ff@mail.gmail.com> <94bdd2610907010141w7340b1a9v8bd001938f15fb58@mail.gmail.com> Message-ID: <79990c6b0907010147g4d3b3848i36eafbcd0ba56ef2@mail.gmail.com> 2009/7/1 Tarek Ziad? : >> As I suggested before: >> >> ? ?python -m distutils.uninstall packagename > > yes sorry if it was unclear, I was not thinking about adding something > based on setup.py, > but just saying that I was going to add this feature in the PEP. and > it will be of the form: > > ?python -m distutils.uninstall packagename > >> >> Calling it a "reference implementation" should not imply that it's not >> built to be usable and complete. If it's there,m people should be able >> to use it. > > It will be usable and complete, but very limited. As someone mentioned, > it will not take care of dependencies and prevent you from removing a > distribution that is mentioned in another distrubution in a > setuptools' install_requires > metadata. Thank you. That is precisely the right level for the PEP, IMHO. > That said, when PEP 345 evolves like we have planned to (adding > install_requires in the metadata) > Then we should be able to provide this kind of warning with no pain. That's good, but not something that bothers me too much. (I'm still of the view that dependencies should be handled by documenting them properly - metadata isn't sufficient on its own). Paul. From alexandre at peadrop.com Thu Jul 2 04:55:49 2009 From: alexandre at peadrop.com (Alexandre Vassalotti) Date: Wed, 1 Jul 2009 22:55:49 -0400 Subject: [Python-Dev] [Fwd: [issue6397] Implementing Solaris "poll" in the "select" module] In-Reply-To: References: <4A4B9C3A.4090606@jcea.es> Message-ID: On Wed, Jul 1, 2009 at 10:05 PM, Guido van Rossum wrote: > The select module already supports the poll() system call. Or is there > a special variant that only Solaris has? > I think Jesus refers to /dev/poll?i.e., the interface for edge-triggered polling on Solaris. This is the Solaris equivalent of FreeBSD's kqueue and Linux's epoll. -- Alexandre From stephen at xemacs.org Thu Jul 2 05:44:57 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 02 Jul 2009 12:44:57 +0900 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com> Message-ID: <877hyrwxxi.fsf@uwakimon.sk.tsukuba.ac.jp> Tarek Ziad? writes: > On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore wrote: > > General rule - don't document (and commit yourself to) any > > internal details that the user can't access from the public > > API. It just makes backward compatibility harder to maintain. > > The purpose is to provide this documentation to third-party > projects that want to implement a packaging system on the top of > these classes. That's a judgment you must make. However, Paul's opinion seems to be that it is internal, and not needed by third-parties who are working "on the top" of these classes. If upon consideration you agree, you should take those "details" out of the PEP proper. If you disagree, you should promote them to the "official"/public API. The point of a PEP is not to construct a duck; it is to explain what "quack" means. > Maybe this should be removed from the PEP and but into another > document targeted to developers ? Yes. In the reference implementation (aka prototype), which should have its own documentation as usual. The reference implementation for the PEP shows *how* these things can be done. It must do that, or the PEP has no force. However, the reference implementation need not be "industrial strength"; the actual implementation that eventually goes into the stdlib may very well be an optimized and enhanced version with many data structures and algorithms that differ from the reference implementation. Another general principle: even in the draft PEP, say "is", not "will be". If you're wrong, and that won't work; if it's insufficiently precise; if you find a more elegant way to express the solution; if you simply haven't thought carefully about something yet; if, for whatever reason, you may need to change the PEP, then you change it in the next draft. That's why we have a review cycle, so you can change it. If you yourself have a question, or the draft is incomplete at some point, then you can explain in a note (either a footnote or parentheses). You can even mark a whole section as "speculative" (eg, "nothing in this section has been implemented yet, so everything is subject to change"). But a rule should be stated as a rule, and precisely, so that reviewers can criticize it accurately. If you're uncomfortable saying "is", then maybe that part of the draft isn't ready for public review yet. On the other hand, you can use the reviewers' knowledge here: write "A is B", then as a note "(In some cases A is actually C, and this should be treated specially. But how?)" From p.f.moore at gmail.com Wed Jul 1 10:35:35 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 1 Jul 2009 09:35:35 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> Message-ID: <79990c6b0907010135n1a9d96abt8c3372e695a6f3ff@mail.gmail.com> 2009/7/1 Tarek Ziad? : >> That (at least as I read it) is a function, not a command. >> If it is a command, give an example of its use from the command line >> for us poor "don't want to research" people. ?If the following works: >> >> ? ?$ python setup.py uninstall some_package >> >> Then explicitly say so for us poor schlubs. >> > > Right, I'll add that. Although it will be a reference implementation only. -1. Where does the setup.py file come from? If I have docutils installed, and want to remove it, must I download the source again so that I can get the setup.py, so I can run the uninstall? This is daft - particularly given that the point of PEP 376 is to ensure that all of the required information is available from the installed package! As I suggested before: python -m distutils.uninstall packagename Calling it a "reference implementation" should not imply that it's not built to be usable and complete. If it's there,m people should be able to use it. Paul. From p.f.moore at gmail.com Thu Jul 2 09:59:20 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 2 Jul 2009 08:59:20 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> Message-ID: <79990c6b0907020059o6c66181eradf8717a69ce9edc@mail.gmail.com> 2009/7/1 R. David Murray : > I haven't read the PEP in detail since it's outside my area of interest > and knowledge(*), but my understanding of the goal is that the PEP is > providing an _infrastructure_ for system-level package management tools. > The uninstall function is part of that infrastructure, but since > distutils isn't a package manager itself (it's an install program), > distutils as currently organized can't really handle uninstall except > as outlined in a section you may have clipped from the above context > (ie: when setup.py from the original package is available). >From a Windows user's POV, "install program" = "package manager". And an install program needs an uninstall feature. I know Linux users with their advanced dependency-managing package managers feel that this is a stone-age view, and they may be right, but the PEP needs to take the Windows situation into account. > The question is what do the people who do real package management > (linux distribution level package management and the equivalent) think? > I'm guessing they are happy with just having the function for their > package management tools to call when needed, since (I'm hoping) they > are part of the distutils sig.... Don't forget that the maintainers of the bdist_wininst, bdist_msi and bdist_rpm code *are* the distutils maintainers - so to that extent, the PEP has to say how *those* aspects of package managers are covered. Unless another PEP is accepted saying that support for bdist_xxx is being dropped [1], this part of distutils cannot be ignored. > > So, if my understanding of the overall goal is correct, it looks to me > like the PEP is missing a "motivation" section that talks about system > package managers. Possibly. If so, though, it must discuss the above 3 cases which are part of core distutils. Paul. [1] A PEP I plan on strongly opposing! From ziade.tarek at gmail.com Thu Jul 2 10:55:51 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Jul 2009 10:55:51 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <20090702023855.F00293A4109@sparrow.telecommunity.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com> <20090702023855.F00293A4109@sparrow.telecommunity.com> Message-ID: <94bdd2610907020155w43aee580wac3604f828cb6b54@mail.gmail.com> On Thu, Jul 2, 2009 at 4:41 AM, P.J. Eby wrote: > Yes and no. ?Not providing uninstall support is reasonable, but the PEP also > has features to query packages in general. > > (There's also no technical reason why comparable manifest and uninstall > support can't be provided for .egg files and directoriees, since they > already have an implicit manifest: their contents. ?However, since I'm not > currently possessed of the time to provide a patch myself, I'm not going to > lobby for this as a feature.) > Yes that's not technical : I am in favor of keeping a single format. I don't see but confusion in keeping several formats, From ziade.tarek at gmail.com Thu Jul 2 11:01:34 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Jul 2009 11:01:34 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <877hyrwxxi.fsf@uwakimon.sk.tsukuba.ac.jp> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com> <877hyrwxxi.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <94bdd2610907020201w5f93b6a1j70fa599d6a1011a8@mail.gmail.com> On Thu, Jul 2, 2009 at 5:44 AM, Stephen J. Turnbull wrote: > That's a judgment you must make. ?However, Paul's opinion seems to be > that it is internal, and not needed by third-parties who are working > "on the top" of these classes. ?If upon consideration you agree, you > should take those "details" out of the PEP proper. ?If you disagree, > you should promote them to the "official"/public API. > > The point of a PEP is not to construct a duck; it is to explain what > "quack" means. Yes, while the APIs I have written in the prototype+PEP helped us claryfing what we wanted, I agree they would be better in a second document. They are two target audience, the users of distutils and the builders of package managers, so removing this details from the PEP will also make it easier to read for the first crowd. > > Another general principle: even in the draft PEP, say "is", not "will > be". Ok I'll fix that. That's a French stuff : in french, "will be" isn't speculative at all. Thx From jcea at jcea.es Thu Jul 2 11:52:32 2009 From: jcea at jcea.es (Jesus Cea) Date: Thu, 02 Jul 2009 11:52:32 +0200 Subject: [Python-Dev] [Fwd: [issue6397] Implementing Solaris "poll" in the "select" module] In-Reply-To: References: <4A4B9C3A.4090606@jcea.es> Message-ID: <4A4C8360.7000701@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Alexandre Vassalotti wrote: > On Wed, Jul 1, 2009 at 10:05 PM, Guido van Rossum wrote: >> The select module already supports the poll() system call. Or is there >> a special variant that only Solaris has? >> > > I think Jesus refers to /dev/poll?i.e., the interface for > edge-triggered polling on Solaris. This is the Solaris equivalent of > FreeBSD's kqueue and Linux's epoll. Yes. In python 2.6, Linux and *BSD benefice of access to high performing "poll"-like interfaces, non portable. Solaris has "/dev/poll" too, so we can provide another non-portable interface :). In fact I think it could be interesting to investigate the direct use of "/dev/poll" in Solaris under "select.poll" implementation, if available, in a transparent way. With this could be possible to improve efficiency without adding another non-portable interface. But before investigating options, I want to know if there is interest on it. I think so, because Solaris/OpenSolaris is an important platform, I will take care of the code, and we already have non-portable interfaces: "select.epoll" for Linux and "select.kqueue" for *BSD. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSkyDXZlgi5GaxT1NAQJSawP/SUf2AVfzYbqv8F3zZjNBspFu5c3bYVEK 6A2YSW0WIrNiQtcI9Ka99dCCzHI6cwohlDhxQddJEG4w30hewsFBFJqftSpPLhkv ToSUNDYe37QmRgVrcns//repWoaMwsrt8MMQU6MLb9pyaoT4Pffe1vcbqg+/XgUN EJENI7BjSOw= =BTMr -----END PGP SIGNATURE----- From ben+python at benfinney.id.au Thu Jul 2 12:16:41 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 02 Jul 2009 20:16:41 +1000 Subject: [Python-Dev] PEP 376 References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> <79990c6b0907020059o6c66181eradf8717a69ce9edc@mail.gmail.com> Message-ID: <873a9ftmnq.fsf@benfinney.id.au> Paul Moore writes: > 2009/7/1 R. David Murray : > > The uninstall function is part of that infrastructure, but since > > distutils isn't a package manager itself (it's an install program), > > distutils as currently organized can't really handle uninstall > > except as outlined in a section you may have clipped from the above > > context (ie: when setup.py from the original package is available). > > >From a Windows user's POV, "install program" = "package manager". And > an install program needs an uninstall feature. I know Linux users with > their advanced dependency-managing package managers feel that this is > a stone-age view, and they may be right, but the PEP needs to take the > Windows situation into account. Actually, your view seems quite reasonable to me: as a GNU user with my advanced dependency-managing package manager, I feel just as strongly that a program that installs is implicitly promising that it can also uninstall cleanly. I think that calling distutils an ?install program? is confusing. When I discuss distutils, I don't call it a program at all; it's a library (or perhaps ?framework?) that provides part of the job of package installation. -- \ ?I think it would be a good idea.? ?Mahatma Gandhi (when asked | `\ what he thought of Western civilization) | _o__) | Ben Finney From p.f.moore at gmail.com Thu Jul 2 13:26:00 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 2 Jul 2009 12:26:00 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <873a9ftmnq.fsf@benfinney.id.au> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> <79990c6b0907020059o6c66181eradf8717a69ce9edc@mail.gmail.com> <873a9ftmnq.fsf@benfinney.id.au> Message-ID: <79990c6b0907020426x5165e0bfl886e876dbae47d57@mail.gmail.com> 2009/7/2 Ben Finney : > I think that calling distutils an ?install program? is confusing. When > I discuss distutils, I don't call it a program at all; it's a library > (or perhaps ?framework?) that provides part of the job of package > installation. I agree, but I consider the fact that distutils "provides" python setup.py install means that it "is an install program" in some people's minds. These are the same people who complain about the lack of a distutils uninstall feature. Personally, I want distutils to provide all the individual bits that I get from bdist_wininst, so that I can put together an alternative for specialised situations (notably, non-site installs of some packages). So uninstall and being able to list installed packages are important to me - hence my interest in this PEP. Paul. From ncoghlan at gmail.com Thu Jul 2 13:32:57 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 02 Jul 2009 21:32:57 +1000 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610907010132ib0c6dcat6a7cfce1d40b6ed7@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> <4A4B1C51.3060601@voidspace.org.uk> <94bdd2610907010132ib0c6dcat6a7cfce1d40b6ed7@mail.gmail.com> Message-ID: <4A4C9AE9.5000806@gmail.com> Tarek Ziad? wrote: > On Wed, Jul 1, 2009 at 10:20 AM, Michael Foord wrote: >>> Uninstall as a command feels a little weird. Since "python setup.py >>> [some-command]" implies that the setup.py contains information about the >>> distribution that the command is being applied to. So instead of: >>> >>> $ python setup.py uninstall some_package >> It could be: >> >> $ python -m distutils uninstall some_package >> >> Asymmetrical with the install of course. >> > > Yes exactly, I was going to add: > > $ python -m distutils.uninstall some_package A directly executable submodule is an even better idea than making distutils itself executable. Definitely worth mentioning in the PEP in the section on uninstallation support. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ziade.tarek at gmail.com Thu Jul 2 13:40:10 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Jul 2009 13:40:10 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <4A4C9AE9.5000806@gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> <4A4B1C51.3060601@voidspace.org.uk> <94bdd2610907010132ib0c6dcat6a7cfce1d40b6ed7@mail.gmail.com> <4A4C9AE9.5000806@gmail.com> Message-ID: <94bdd2610907020440t747e94fema65b940be596db29@mail.gmail.com> On Thu, Jul 2, 2009 at 1:32 PM, Nick Coghlan wrote: >> >> Yes exactly, I was going to add: >> >> $ python -m distutils.uninstall some_package > > A directly executable submodule is an even better idea than making > distutils itself executable. Definitely worth mentioning in the PEP in > the section on uninstallation support. > The PEP is updated for this, and takes into account several feedback from this thread, (like removing implementation details) I am working now on describing the behavior with the bdist_* commands From ncoghlan at gmail.com Thu Jul 2 13:39:48 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 02 Jul 2009 21:39:48 +1000 Subject: [Python-Dev] I am back In-Reply-To: <20090702023251.GA28507@panix.com> References: <20090702023251.GA28507@panix.com> Message-ID: <4A4C9C84.1030702@gmail.com> Aahz wrote: > On Wed, Jul 01, 2009, Brett Cannon wrote: >> Anything happen while I was gone that I should be aware of that is not >> covered in a PEP? > > Yes. I agree with Aahz ;) Cheers, Nick. P.S. Welcome back :) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ben+python at benfinney.id.au Thu Jul 2 14:24:14 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 02 Jul 2009 22:24:14 +1000 Subject: [Python-Dev] PEP 376 References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <94bdd2610906301646x16cb991hdfa55676fd4700b9@mail.gmail.com> <8D2D3C9B-DB50-48A5-8216-CEE70A62278A@bud.ca> <79990c6b0907020059o6c66181eradf8717a69ce9edc@mail.gmail.com> <873a9ftmnq.fsf@benfinney.id.au> <79990c6b0907020426x5165e0bfl886e876dbae47d57@mail.gmail.com> Message-ID: <87y6r7s26p.fsf@benfinney.id.au> Paul Moore writes: > 2009/7/2 Ben Finney : > > I think that calling distutils an ?install program? is confusing. > > When I discuss distutils, I don't call it a program at all; it's a > > library (or perhaps ?framework?) that provides part of the job of > > package installation. > > I agree, but I consider the fact that distutils "provides" python > setup.py install means that it "is an install program" in some > people's minds. These are the same people who complain about the lack > of a distutils uninstall feature. Well, I don't suffer from that confusion between the library and a specific program, yet I still want distutils to support uninstall :-) and hence am interested in seeing the fate of this PEP. -- \ ?Men never do evil so completely and cheerfully as when they do | `\ it from religious conviction.? ?Blaise Pascal (1623?1662), | _o__) Pens?es, #894. | Ben Finney From ziade.tarek at gmail.com Thu Jul 2 14:34:59 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Jul 2009 14:34:59 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> Message-ID: <94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com> On Wed, Jul 1, 2009 at 1:44 PM, Paul Moore wrote: > "Adding a RECORD file" > > You say "at installation time" - please clarify. Do you only mean > setup.py install? What about bdist_wininst and bdist_msi? What about > third party bdist style tools? How will they ensure they get a RECORD > file? > > "The RECORD format" > > The line separator shouldn't be os-dependent. What value is used for a > pure-python (ie, platform independent) package? Unless the file is > generated when the install is done, in which case see the previous > point... > > Absolute file paths - this implies that RECORD has to be generated by > the installer (which is the only place that knows absolute paths) > which means that every bdist_xxx installer has to write its own RECORD > file. Does the PEP offer no support for this? In any case, the > bdist_wininst and bdist_msi code (which is core distutils) will need > to be amended to manage RECORD files appropriately. Since all bdist_* commands are using the install command in a temporary build directory to create a binary distribution, the RECORD file (and the INSTALLER) will be pre-generated then installed alongside the packages/modules. For absolute paths now that gets installed, what would be the difference between the pre-generated RECORD file and the RECORD file installed on the win32 target system, if any ? From p.f.moore at gmail.com Thu Jul 2 14:39:29 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 2 Jul 2009 13:39:29 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com> Message-ID: <79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com> 2009/7/2 Tarek Ziad? : > For absolute paths now that gets installed, what would be the > difference between the pre-generated > RECORD file and the RECORD file installed on the win32 target system, if any ? When is an absolute path generated? If you can give me a small sample of a distribution that installs a file in an absolute path, I'll do some testing on Windows. But the immediate answer is that there are *no* reliable "absolute" paths on Windows, so you're not looking at the likes of /usr/doc, but rather paths that are relative to sys.prefix, but not to the package directory. In that case, the key point is that if the installer is built on a system where Python is installed in a different directory than the system where the installer is run, paths need to be relocated. (E.g., C:\Python27 vs D:\Apps\Python27). Paul. From anderson.lizardo at openbossa.org Thu Jul 2 15:03:48 2009 From: anderson.lizardo at openbossa.org (Anderson Lizardo) Date: Thu, 2 Jul 2009 09:03:48 -0400 Subject: [Python-Dev] "re" module and Python 1.6 (GPL incompatible) license? Message-ID: <55b616790907020603x52b10d07t272d550ce9805dd0@mail.gmail.com> Hi, I noticed that some files of the "re" module still have the (GPL incompatible) 1.6 license notice. Is that on purpose or unintentionally forgotten? If that is on purpose, does that mean that some GPL Python code cannot use the "re" module? For reference, here are the links to these files (from trunk): http://svn.python.org/view/python/trunk/Lib/re.py?view=markup http://svn.python.org/view/python/trunk/Modules/_sre.c?view=markup Additionally, the files below do not have a complete license header, or they just point to one of the files above: http://svn.python.org/view/python/trunk/Lib/sre_compile.py?view=markup http://svn.python.org/view/python/trunk/Lib/sre_constants.py?view=markup http://svn.python.org/view/python/trunk/Lib/sre_parse.py?view=markup http://svn.python.org/view/python/trunk/Modules/sre.h?view=markup http://svn.python.org/view/python/trunk/Modules/sre_constants.h?view=markup PS: if this gets fixed somehow, can the fix be backported to the Python 2.5.x releases? Let me know what needs to be done to get this fixed on next release. Thanks, -- Anderson Lizardo OpenBossa Labs - INdT Manaus - Brazil From ziade.tarek at gmail.com Thu Jul 2 15:45:23 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Jul 2009 15:45:23 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com> <79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com> Message-ID: <94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com> On Thu, Jul 2, 2009 at 2:39 PM, Paul Moore wrote: > 2009/7/2 Tarek Ziad? : >> For absolute paths now that gets installed, what would be the >> difference between the pre-generated >> RECORD file and the RECORD file installed on the win32 target system, if any ? > > When is an absolute path generated? If you can give me a small sample > of a distribution that installs a file in an absolute path, I'll do > some testing on Windows. Try this setup.py file: """ from distutils.core import setup setup(name='foo', version='1.0', data_files=[('/tmp', ['data'])]) """ with this MANIFEST.in file: "" include data """ and add a 'data' file alongside If you install it, data is copied in /tmp. If you create a bdist distribution it will be created in the root of the dumb directory which is used to generate the binary distro. Of course you'll have to change '/tmp' to 'c:\tmp' (which makes me realizes that there's no way to force the installation data_files in another drive under windows: the binary distribution will be the same, no matter what drive you use in the absolute path used in data_files I'll add an issue on this) > > But the immediate answer is that there are *no* reliable "absolute" > paths on Windows, so you're not looking at the likes of /usr/doc, but > rather paths that are relative to sys.prefix, but not to the package > directory. In that case, the key point is that if the installer is > built on a system where Python is installed in a different directory > than the system where the installer is run, paths need to be > relocated. (E.g., C:\Python27 vs D:\Apps\Python27). I get the point: they are three levels we should handle in the RECORD file 1. absolute paths 2. paths relative to sys.prefix or sys.exec_prefix 3. paths relative to the directory where the .egg-info directory is located the RECORD file doesn't handle 2. indeed. For instance you can add a script: setup( .. scripts=['myscript.py'] ..) that will get installed in : sys.prefix + 'Scripts/myscript.py' under win32 for instance So a possible solution is to add 2. in the RECORD files by using a notation such as "$PREFIX" "$EXEC_PREFIX" when we detect that the file is under on of these paths. The query functions will then be able to use sys.prefix and sys.exec_prefix for recompose the absolute pat on the target system > > Paul. > -- Tarek Ziad? | http://ziade.org From solipsis at pitrou.net Thu Jul 2 16:18:42 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 2 Jul 2009 14:18:42 +0000 (UTC) Subject: [Python-Dev] PEP 376 References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> Message-ID: Guido van Rossum python.org> writes: > > I noted an inconsistency: first you say that the RECORD file uses the > excel dialect, but at the end of the same section you say it uses the > default csv settings. Sounds like you need to delete one or the other. If I remember correctly, the default csv dialect is just the Excel dialect, so this would be two different ways of saying the same things. Regards Antoine. From p.f.moore at gmail.com Thu Jul 2 16:35:18 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 2 Jul 2009 15:35:18 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com> <79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com> <94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com> Message-ID: <79990c6b0907020735q5d76deb1p31d904c981286ca8@mail.gmail.com> 2009/7/2 Tarek Ziad? : > Try this setup.py file: > > """ > from distutils.core import setup > > setup(name='foo', version='1.0', data_files=[('/tmp', ['data'])]) > """ > > with this MANIFEST.in file: > "" > include data > """ > > and add a 'data' file alongside Ta. I'll give it a go tonight. But haven't you made the point yourself by saying I'll need to change the directory? This will fail for me as I don't have a "/tmp" directory. So I'd expect a bdist installer (*any* bdist installer) to fail, as it's got no way to tell where to put that file. I can only imagine this type of thing being done by someone packaging for Unix, with no interest in cross-platform portability (I suspect MAC OS directory names differ from Unix as well). So maybe absolute filenames should be banned in distutils? (OK, less radically, maybe someone should ask where that functionality is used in practice, and based on the answers come up with a more portable approach). Paul. From ziade.tarek at gmail.com Thu Jul 2 16:50:51 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Jul 2009 16:50:51 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0907020735q5d76deb1p31d904c981286ca8@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com> <79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com> <94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com> <79990c6b0907020735q5d76deb1p31d904c981286ca8@mail.gmail.com> Message-ID: <94bdd2610907020750lc49ddd3kf21f7dceda13bc2b@mail.gmail.com> On Thu, Jul 2, 2009 at 4:35 PM, Paul Moore wrote: > Ta. I'll give it a go tonight. But haven't you made the point yourself > by saying I'll need to change the directory? This will fail for me as > I don't have a "/tmp" directory. So I'd expect a bdist installer > (*any* bdist installer) to fail, as it's got no way to tell where to > put that file. I can only imagine this type of thing being done by > someone packaging for Unix, with no interest in cross-platform > portability (I suspect MAC OS directory names differ from Unix as > well). It will fail if your path starts with "/" and you are under win32 because there's a converter that will raise an error in that case at installation time. But you can use some code in your setup.py to provide the right directory name when install is called. I don't see it as a problem. I'm just afraid it's impossible to use efficiently under win32 because of the drive letter. But that's rather a bug. > > So maybe absolute filenames should be banned in distutils? (OK, less > radically, maybe someone should ask where that functionality is used > in practice, and based on the answers come up with a more portable > approach). > I'll try to see if I can collect that info out of PyPI with a script, to list the real-world usages. Maybe Martin you have a simple way to do this on PyPI ? Tarek From skip at pobox.com Thu Jul 2 18:07:13 2009 From: skip at pobox.com (skip at pobox.com) Date: Thu, 2 Jul 2009 11:07:13 -0500 Subject: [Python-Dev] I am back In-Reply-To: <20090702023251.GA28507@panix.com> References: <20090702023251.GA28507@panix.com> Message-ID: <19020.56113.916994.809864@montanaro.dyndns.org> Aahz> On Wed, Jul 01, 2009, Brett Cannon wrote: >> Anything happen while I was gone that I should be aware of that is >> not covered in a PEP? Aahz> Yes. In particular, Brett, you probably didn't hear that the King of Pop died last week. It was hardly mentioned in the major news outlets... Skip From josiah.carlson at gmail.com Thu Jul 2 18:30:04 2009 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Thu, 2 Jul 2009 09:30:04 -0700 Subject: [Python-Dev] Bytes, Strings, Encoding In-Reply-To: <1afaf6160907011848h122cc2ebp964d225d453052c1@mail.gmail.com> References: <171e8a410907010817j715f1d5ye638ea4b2469afd9@mail.gmail.com> <171e8a410907011747i461a409ew9253e3fc82a10acd@mail.gmail.com> <1afaf6160907011848h122cc2ebp964d225d453052c1@mail.gmail.com> Message-ID: On Wed, Jul 1, 2009 at 6:48 PM, Benjamin Peterson wrote: > 2009/7/1 Eric Pruitt : >> Hello, >> >> I am working on the subprocess.Popen module for Python 2.7 and am now moving >> my changes over to Python 3.1 however I am having trouble with the whole >> byte situation and I can't quite seem to understand how to go back and forth >> between bytes and strings. I am also looking for the Python 3k equivalent >> for the Python 2.X built-in buffer class. > > Look at the "memoryview" type as a replacement for buffer. > io.Buffered* is not the same. FYI, it is used slightly differently. Rather than x = buffer(y, offset), you use x = memoryview(y)[offset:]. >> One version of the file with my modifications can be found here? < >> http://code.google.com/p/subprocdev/source/browse/subprocess.py?spec=svn5b570f8cbfcaae859091eb01b21b183aa5221af9&r=5b570f8cbfcaae859091eb01b21b183aa5221af9 >>>. Lines 1 - 15 are me attempting to get around certain changes between >> Python 3.0 and Python 2.7. Further down on line 916, we have the function >> "send" and "recv" in which I am having the most trouble with bytes and >> strings. > > You translate between bytes and strings through encoding and decoding. > Have you read https://secure.wikimedia.org/wikipedia/en/wiki/Unicode? > >> >> Any help is appreciated. Feel free to comment on my blog >> http://subdev.blogspot.com/ or reply to the last. > > Have you talked with your mentor about these things? > > > -- > Regards, > Benjamin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com > From stefan_ml at behnel.de Thu Jul 2 19:06:53 2009 From: stefan_ml at behnel.de (Stefan Behnel) Date: Thu, 02 Jul 2009 19:06:53 +0200 Subject: [Python-Dev] I am back In-Reply-To: <19020.56113.916994.809864@montanaro.dyndns.org> References: <20090702023251.GA28507@panix.com> <19020.56113.916994.809864@montanaro.dyndns.org> Message-ID: skip at pobox.com wrote: > Aahz> On Wed, Jul 01, 2009, Brett Cannon wrote: > > >> Anything happen while I was gone that I should be aware of that is > >> not covered in a PEP? > > Aahz> Yes. > > In particular, Brett, you probably didn't hear that the King of Pop died > last week. It was hardly mentioned in the major news outlets... *shrug* who cares about foreign politics anyway. I don't even know where Pop is, and Google Maps doesn't seem to either. Stefan From python at mrabarnett.plus.com Thu Jul 2 19:27:25 2009 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 02 Jul 2009 18:27:25 +0100 Subject: [Python-Dev] I am back In-Reply-To: References: <20090702023251.GA28507@panix.com> <19020.56113.916994.809864@montanaro.dyndns.org> Message-ID: <4A4CEDFD.8070704@mrabarnett.plus.com> Stefan Behnel wrote: > skip at pobox.com wrote: >> Aahz> On Wed, Jul 01, 2009, Brett Cannon wrote: >> >> >> Anything happen while I was gone that I should be aware of that is >> >> not covered in a PEP? >> >> Aahz> Yes. >> >> In particular, Brett, you probably didn't hear that the King of Pop died >> last week. It was hardly mentioned in the major news outlets... > > *shrug* who cares about foreign politics anyway. I don't even know where > Pop is, and Google Maps doesn't seem to either. > I believe that "Pop" is an abbreviation for "Populous". You can read about it here: http://en.wikipedia.org/wiki/Populous :-) From Scott.Daniels at Acm.Org Thu Jul 2 20:01:15 2009 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Thu, 02 Jul 2009 11:01:15 -0700 Subject: [Python-Dev] PEP 376 In-Reply-To: References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> Message-ID: Antoine Pitrou wrote: > Guido van Rossum python.org> writes: >> I noted an inconsistency: first you say that the RECORD file uses the >> excel dialect, but at the end of the same section you say it uses the >> default csv settings. Sounds like you need to delete one or the other. > > If I remember correctly, the default csv dialect is just the Excel dialect, so > this would be two different ways of saying the same things. Right, but Guido's point is, decide which one is the is the definition and stick to talking about it in that form. --Scott David Daniels Scott.Daniels at Acm.Org From brett at python.org Thu Jul 2 19:54:53 2009 From: brett at python.org (Brett Cannon) Date: Thu, 2 Jul 2009 10:54:53 -0700 Subject: [Python-Dev] I am back In-Reply-To: <4A4C9C84.1030702@gmail.com> References: <20090702023251.GA28507@panix.com> <4A4C9C84.1030702@gmail.com> Message-ID: On Thu, Jul 2, 2009 at 04:39, Nick Coghlan wrote: > Aahz wrote: > > On Wed, Jul 01, 2009, Brett Cannon wrote: > >> Anything happen while I was gone that I should be aware of that is not > >> covered in a PEP? > > > > Yes. > > I agree with Aahz ;) > Ah, the smart ass remarks never cease. =) > > Cheers, > Nick. > > P.S. Welcome back :) Thanks! -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Jul 2 20:00:09 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 02 Jul 2009 14:00:09 -0400 Subject: [Python-Dev] PEP 376 In-Reply-To: References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> Message-ID: Scott David Daniels wrote: > Antoine Pitrou wrote: >> Guido van Rossum python.org> writes: >>> I noted an inconsistency: first you say that the RECORD file uses the >>> excel dialect, but at the end of the same section you say it uses the >>> default csv settings. Sounds like you need to delete one or the other. >> >> If I remember correctly, the default csv dialect is just the Excel >> dialect, so >> this would be two different ways of saying the same things. > > Right, but Guido's point is, decide which one is the is the definition > and stick to talking about it in that form. Since I think it more likely that Excel will change than cvs default, I think the latter should be used. From tjreedy at udel.edu Thu Jul 2 20:44:36 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 02 Jul 2009 14:44:36 -0400 Subject: [Python-Dev] I am back In-Reply-To: References: <20090702023251.GA28507@panix.com> <4A4C9C84.1030702@gmail.com> Message-ID: Brett Cannon wrote: > > > On Thu, Jul 2, 2009 at 04:39, Nick Coghlan > wrote: > > Aahz wrote: > > On Wed, Jul 01, 2009, Brett Cannon wrote: > >> Anything happen while I was gone that I should be aware of that > is not > >> covered in a PEP? > > > > Yes. > > I agree with Aahz ;) > > > Ah, the smart ass remarks never cease. =) The biggest news is the release of 3.1, but that might have been recorded in the relevant PEP, and it certainly hard to miss. In any case, welcome back. Terry From brett at python.org Thu Jul 2 21:04:45 2009 From: brett at python.org (Brett Cannon) Date: Thu, 2 Jul 2009 12:04:45 -0700 Subject: [Python-Dev] semi-regular check that all committers are subscribed to python-committers Message-ID: If you have commit privileges on Python but are not subscribed to python-committers, please let me know and I will subscribe you. Membership on python-committers is mandatory as that's where we announce branch freezes, etc. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Thu Jul 2 21:33:50 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 02 Jul 2009 21:33:50 +0200 Subject: [Python-Dev] "re" module and Python 1.6 (GPL incompatible) license? In-Reply-To: <55b616790907020603x52b10d07t272d550ce9805dd0@mail.gmail.com> References: <55b616790907020603x52b10d07t272d550ce9805dd0@mail.gmail.com> Message-ID: <4A4D0B9E.3070301@v.loewis.de> > PS: if this gets fixed somehow, can the fix be backported to the > Python 2.5.x releases? Definitely not - it's not security critical. Regards, Martin From dirkjan at ochtman.nl Thu Jul 2 21:56:34 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 2 Jul 2009 21:56:34 +0200 Subject: [Python-Dev] semi-regular check that all committers are subscribed to python-committers In-Reply-To: References: Message-ID: On Thu, Jul 2, 2009 at 21:04, Brett Cannon wrote: > If you have commit privileges on Python but are not subscribed to > python-committers, please let me know and I will subscribe you. Membership > on python-committers is mandatory as that's where we announce branch > freezes, etc. Would it be helpful if I was subscribed? I'm not strictly a committer, but what with the Mercurial migration I imagine it won't be long before the first cset bearing my name appears in the Python repository (plus I just saw there was some discussion about the hg migration there which I could weigh in on). Cheers, Dirkjan From p.f.moore at gmail.com Thu Jul 2 22:06:53 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 2 Jul 2009 21:06:53 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com> <79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com> <94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com> Message-ID: <79990c6b0907021306r4fc85c51lb8e6063e5bbaf663@mail.gmail.com> 2009/7/2 Tarek Ziad? : > On Thu, Jul 2, 2009 at 2:39 PM, Paul Moore wrote: >> 2009/7/2 Tarek Ziad? : >>> For absolute paths now that gets installed, what would be the >>> difference between the pre-generated >>> RECORD file and the RECORD file installed on the win32 target system, if any ? >> >> When is an absolute path generated? If you can give me a small sample >> of a distribution that installs a file in an absolute path, I'll do >> some testing on Windows. > > Try this setup.py file: > > """ > from distutils.core import setup > > setup(name='foo', version='1.0', data_files=[('/tmp', ['data'])]) > """ > > with this MANIFEST.in file: > "" > include data > """ > > and add a 'data' file alongside > > If you install it, data is copied in /tmp. If you create a bdist > distribution it will be created in the root of the dumb directory > which is used to generate the binary distro. > > Of course you'll have to change '/tmp' to 'c:\tmp' >python setup.py bdist_wininst running bdist_wininst running build installing to build\bdist.win32\wininst running install_data creating build creating build\bdist.win32 creating build\bdist.win32\wininst creating build\bdist.win32\wininst\DATA Traceback (most recent call last): File "setup.py", line 3, in setup(name='foo', version='1.0', data_files=[('/tmp', ['data'])]) File "C:\Apps\Python26\lib\distutils\core.py", line 152, in setup dist.run_commands() File "C:\Apps\Python26\lib\distutils\dist.py", line 975, in run_commands self.run_command(cmd) File "C:\Apps\Python26\lib\distutils\dist.py", line 995, in run_command cmd_obj.run() File "C:\Apps\Python26\lib\distutils\command\bdist_wininst.py", line 171, in run install.run() File "C:\Apps\Python26\lib\distutils\command\install.py", line 589, in run self.run_command(cmd_name) File "C:\Apps\Python26\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "C:\Apps\Python26\lib\distutils\dist.py", line 995, in run_command cmd_obj.run() File "C:\Apps\Python26\lib\distutils\command\install_data.py", line 62, in run dir = convert_path(f[0]) File "C:\Apps\Python26\lib\distutils\util.py", line 164, in convert_path raise ValueError, "path '%s' cannot be absolute" % pathname ValueError: path '/tmp' cannot be absolute Looks like bdist_wininst doesn't support absolute paths. Which isn't surprising, really. Same with bdist_msi. Actually, with setup.py install: >python setup.py install running install running build running install_data Traceback (most recent call last): File "setup.py", line 3, in setup(name='foo', version='1.0', data_files=[('/tmp', ['data'])]) File "C:\Apps\Python26\lib\distutils\core.py", line 152, in setup dist.run_commands() File "C:\Apps\Python26\lib\distutils\dist.py", line 975, in run_commands self.run_command(cmd) File "C:\Apps\Python26\lib\distutils\dist.py", line 995, in run_command cmd_obj.run() File "C:\Apps\Python26\lib\distutils\command\install.py", line 589, in run self.run_command(cmd_name) File "C:\Apps\Python26\lib\distutils\cmd.py", line 333, in run_command self.distribution.run_command(command) File "C:\Apps\Python26\lib\distutils\dist.py", line 995, in run_command cmd_obj.run() File "C:\Apps\Python26\lib\distutils\command\install_data.py", line 62, in run dir = convert_path(f[0]) File "C:\Apps\Python26\lib\distutils\util.py", line 164, in convert_path raise ValueError, "path '%s' cannot be absolute" % pathname ValueError: path '/tmp' cannot be absolute So how do absolute paths work??? It sure ain't like this! Changing '/tmp' to c:/tmp' gives this: >python setup.py bdist_wininst running bdist_wininst running build installing to build\bdist.win32\wininst running install_data creating build\bdist.win32\wininst\DATA\c:tmp error: could not create 'build\bdist.win32\wininst\DATA\c:tmp': The directory name is invalid Changing it to 'c:\\tmp' gives (sort of) success: python setup.py bdist_wininst running bdist_wininst running build installing to build\bdist.win32\wininst running install_data creating build\bdist.win32\wininst\tmp copying data -> build\bdist.win32\wininst\tmp running install_egg_info Creating build\bdist.win32\wininst\PURELIB\ Writing build\bdist.win32\wininst\PURELIB\foo-1.0-py2.6.egg-info creating 'c:\users\gustav\appdata\local\temp\tmpcs9rmi.zip' and adding '.' to it adding 'PURELIB\foo-1.0-py2.6.egg-info' adding 'tmp\data' creating dist removing 'build\bdist.win32\wininst' (and everything under it) The installer packs a *relative* path tmp/data - which isn't right. bdist_msi copies the data file to c:\tmp - when building the msi!. This is very wrong. I can't introspect a msi file to see what ends up in it, but if I install it, no c:\tmp\data is installed, so it does look like it's screwed up. Frankly, handling of absolute paths looks so broken, that I'd explicitly state that it's not supported. I'd raise bugs for some of these things, but to be honest, I'm not at all sure what I'd expect to be correct behaviour (on Windows). In my view absolute paths shouldn't be supported - I can see that might be different on Linux, but I don't see how it's ever going to be cross-platform. Not sure where that leaves things. Absolute path handling seems broken enough (on Windows) that whatever the PEP says isn't much of an issue, as it's not usable anyway. One definite point, though - given the behaviour of bdist_msi, it thought it had installed something that it simply didn't - so there's a serious risk that the distutils uninstall function could uninstall something that the installer never installed - that's far worse than simply deleting the files and messing up the installer metadata. Maybe the answer is that distutils reject *all* uses of absolute paths on Windows. The PEP can then say whatever you want as far as I'm concerned, as it doesn't affect me. If the existing bugs don't get fixed, though, I'd say that the RECORD file should be explicitly disallowed from containing absolute paths on Windows (for safety reasons if nothing else). Paul. From brett at python.org Thu Jul 2 22:08:13 2009 From: brett at python.org (Brett Cannon) Date: Thu, 2 Jul 2009 13:08:13 -0700 Subject: [Python-Dev] semi-regular check that all committers are subscribed to python-committers In-Reply-To: References: Message-ID: On Thu, Jul 2, 2009 at 12:56, Dirkjan Ochtman wrote: > On Thu, Jul 2, 2009 at 21:04, Brett Cannon wrote: > > If you have commit privileges on Python but are not subscribed to > > python-committers, please let me know and I will subscribe you. > Membership > > on python-committers is mandatory as that's where we announce branch > > freezes, etc. > > Would it be helpful if I was subscribed? I'm not strictly a committer, > but what with the Mercurial migration I imagine it won't be long > before the first cset bearing my name appears in the Python repository > (plus I just saw there was some discussion about the hg migration > there which I could weigh in on). Maybe you should just get the commit privileges now. Any objections? -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From dirkjan at ochtman.nl Thu Jul 2 22:10:49 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 2 Jul 2009 22:10:49 +0200 Subject: [Python-Dev] semi-regular check that all committers are subscribed to python-committers In-Reply-To: References: Message-ID: On Thu, Jul 2, 2009 at 22:08, Brett Cannon wrote: > ?Maybe you should just get the commit privileges now. Any objections? Not from me, obviously. Cheers, Dirkjan From pje at telecommunity.com Thu Jul 2 22:59:11 2009 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 02 Jul 2009 16:59:11 -0400 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0907021306r4fc85c51lb8e6063e5bbaf663@mail.gmail.co m> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <94bdd2610907020534v14d20ec9t235f8a0f34673617@mail.gmail.com> <79990c6b0907020539u5cb57bcdx1efefc5fd1496e5e@mail.gmail.com> <94bdd2610907020645r4e742b44r5afa5a28aa54e43d@mail.gmail.com> <79990c6b0907021306r4fc85c51lb8e6063e5bbaf663@mail.gmail.com> Message-ID: <20090702205612.57D8A3A4109@sparrow.telecommunity.com> At 09:06 PM 7/2/2009 +0100, Paul Moore wrote: >Maybe the answer is that distutils reject *all* uses of absolute paths >on Windows. The PEP can then say whatever you want as far as I'm >concerned, as it doesn't affect me. If the existing bugs don't get >fixed, though, I'd say that the RECORD file should be explicitly >disallowed from containing absolute paths on Windows (for safety >reasons if nothing else). In non bdist cases (e.g. setup.py with manual options, but also easy_install, pip, etc.) you can specify where data files and scripts go on Windows, and they don't have to be relative locations. So, absolute paths need to be supported, despite the existence of broken bdist targets on Windows. From dirkjan at ochtman.nl Thu Jul 2 22:42:32 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 2 Jul 2009 22:42:32 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) Message-ID: In response to some rumblings on python-committers and just to request more feedback, a progress report. I know it's long, I've tried to put to keep it concise and chunked, though. - First of all, I've got the basic conversion down, I've done it a few times now, with progressively better results. You can view some results at http://hg.python.org/, which has a preliminary cpython repository. *** The changeset hashes for that repo will change, so you won't be able to commit or pull from it in the future.*** - Second of all, some planning. I've thought about it a bit, and I think we should aim for going live with hg on August 1. Given that I'm on vacation from 8-18 July (and I'm not sure whether I'll be able to actually work on it during that time, though I imagine I'll be able to spend some time on it at least), that's quite ambitious, so I'm going to say it's okay if it slips by a few days. Putting a deadline out there is a good thing, anyway. - Third of all, to make this possible, it would be helpful if I got more feedback on the PEP. Last time I raised it, there was virtually nothing. This time, I'll include it inline so there's hopefully less of a barrier to reviewing it. - Fourth, Mercurial 1.3 was just released! Bet you didn't see that coming. It's looking like a pretty good release, with an experimental version of the much-coveted subrepository support (like svn:externals). This also means that the latest version of hgsubversion, the tool I used for the conversion, will be more accessible for converting other projects. You know you want to! - Fifth, here's a list of things, off the top of my head, that still need doing: * Get agreement on branch strategy and branch processing (list of branches + proposed handling at http://hg.python.org/pymigr/file/tip/all-branches.txt) <--- PLEASE REVIEW * Get agreement on tag processing (first come up with a list) * Set up hg-ssh infra (should be easy) * Set up hooks (should be mostly straightforward) * Set up roundup integration (should be made easier by quick revision map hgweb extension) * Write docs - Sixth (this is the good part), less obvious things that have been done or don't need doing: * .hgignore generation (I've been convinced it's too hard, the current version will do) * revlog reordering (it's painless and a big win) I'll get through all of these myself, but obviously any help would be welcome. For any hg users, writing docs should be an easy start. For others, please review the PEP (below), the branch map in http://hg.python.org/pymigr/file/tip/all-branches.txt and the author map at http://hg.python.org/pymigr/file/tip/author-map (not that much has changed since the start, so if you've looked at it already, feel free to skip this part). Right now I'm a little stuck on branch processing, because it's a long running script that needs a bunch of debugging, but I'll get going on that again. I think that's all I can think of for now, I'll update the PEP with new bits soon. Here it is, ready for your review: ============================================================== Motivation After having decided to switch to the Mercurial DVCS, the actual migration still has to be performed. In the case of an important piece of infrastructure like the version control system for a large, distributed project like Python, this is a significant effort. This PEP is an attempt to describe the steps that must be taken for further discussion. It's somewhat similar to PEP 347 [1], which discussed the migration to SVN. To make the most of hg, I (Dirkjan) would like to make a high-fidelity conversion, such that (a) as much of the svn metadata as possible is retained, and (b) all metadata is converted to formats that are common in Mercurial. This way, tools written for Mercurial can be optimally used. In order to do this, I want to use the hgsubversion [2] software to do an initial conversion. This hg extension is focused on providing high-quality conversion from Subversion to Mercurial for use in two-way correspondence, meaning it doesn't throw away as much available metadata as other solutions. Such a conversion also seems like a good time to reconsider the contents of the repository and determine if some things are still valuable. In this spirit, the following sections also propose discarding some of the older metadata. Timeline TBD; needs fully working hgsubversion and consensus on this document. Transition plan Branch strategy Mercurial has two basic ways of using branches: cloned branches, where each branch is kept in a separate repository, and named branches, where each revision keeps metadata to note on which branch it belongs. The former makes it easier to distinguish branches, at the expense of requiring more disk space on the client. The latter makes it a little easier to switch between branches, but often has somewhat unintuitive results for people (though this has been getting better in recent versions of Mercurial). I'm still a bit on the fence about whether Python should adopt cloned branches and named branches. Since it usually makes more sense to tag releases on the maintenance branch, for example, mainline history would not contain release tags if we used cloned branches. Also, Mercurial 1.2 and 1.3 have the necessary tools to make named branches less painful (because they can be properly closed and closed heads are no longer considered in relevant cases). A disadvantage might be that the used clones will be a good bit larger (since they essentially contain all other branches as well). This can me mitigated by keeping non-release (feature) branches in separate clones. Also note that it's still possible to clone a single named branch from a combined clone, by specifying the branch as in hg clone http://hg.python.org/main/#2.6-maint. Keeping the py3k history in a separate clone problably also makes sense. XXX To do: size comparison for selected separation scenarios. Converting branches There are quite a lot of branches in SVN's branches directory. I propose to clean this up a bit, by employing the following the strategy: * Keep all release (maintenance) branches * Discard branches that haven't been touched in 18 months, unless somone indicates there's still interest in such a branch * Keep branches that have been touched in the last 18 months, unless someone indicates the branch can be deprecated Converting tags The SVN tags directory contains a lot of old stuff. Some of these are not, in fact, full tags, but contain only a smaller subset of the repository. I think we should keep all release tags, and consider other tags for inclusion based on requests from the developer community. I'd like to consider unifying the release tag naming scheme to make some things more consistent, if people feel that won't create too many problems. For example, Mercurial itself just uses '1.2.1' as a tag, where CPython would currently use r121. Author map In order to provide user names the way they are common in hg (in the 'First Last ' format), we need an author map to map cvs and svn user names to real names and their email addresses. I have a complete version of such a map in my migration tools repository [3]. The email addresses in it might be out of date; that's bound to happen, although it would be nice to try and have as many people as possible review it for addresses that are out of date. The current version also still seems to contain some encoding problems. Generating .hgignore The .hgignore file can be used in Mercurial repositories to help ignore files that are not eligible for version control. It does this by employing several possible forms of pattern matching. The current Python repository already includes a rudimentary .hgignore file to help with using the hg mirrors. It might be useful to have the .hgignore be generated automatically from svn:ignore properties. This would make sure all historic revisions also have useful ignore information (though one could argue ignoring isn't really relevant to just checking out an old revision). Revlog reordering As an optional optimization technique, we should consider trying a reordering pass on the revlogs (internal Mercurial files) resulting from the conversion. In some cases this results in dramatic decreases in on-disk repository size. Other repositories Richard Tew has indicated that he'd like the Stackless repository to also be converted. What other projects in the svn.python.org repository should be converted? Do we want to convert the peps repository? distutils? others? Infrastructure hg-ssh Developers should access the repositories through ssh, similar to the current setup. Public keys can be used to grant people access to a shared hg@ account. A hgwebdir instance should also be set up for easy browsing and read-only access. If we're using ssh, developers should trivially be able to start new clones (for longer-term features that profit from a separate branch). Hooks A number of hooks is currently in use. The hg equivalents for these should be developed and deployed. The following hooks are being used: * check whitespace: a hook to reject commits in case the whitespace doesn't match the rules for the Python codebase. Should be straightforward to re-implement from the current version. We can also offer a whitespace hook for use with client-side repositories that people can use; it could either warn about whitespace issues and/or truncate trailing whitespace from changed lines. Open issue: do we check only the tip after each push, or do we check every commit in a changegroup? * commit mails: we can leverage the notify extension for this * buildbots: both the regular and the community build masters must be notified. Fortunately buildbot includes support for hg. I've also implemented this for Mercurial itself, so I don't expect problems here. * check contributors: in the current setup, all changesets bear the username of committers, who must have signed the contributor agreement. In a DVCS, the committers are not necessarily the same people who push, and so we can't check if the committer is a contributor. We could use a hook to check if the committer is a contributor if we keep a list of registered contributors. hgwebdir A more or less stock hgwebdir installation should be set up. We might want to come up with a style to match the Python website. It may also be useful to build a quick extension to augment the URL rev parser so that it can also take r[0-9]+ args and come up with the matching hg revision. After migration Where to get code It needs to be decided where the hg repositories will live. I'd like to propose to keep the hgwebdir instance at hg.python.org. This is an accepted standard for many organizations, and an easy parallel to svn.python.org. The 2.7 (trunk) repo might live at http://hg.python.org/main/, for example, with py3k at http://hg.python.org/py3k/. For write access, developers will have to use ssh, which could be ssh://hg at hg.python.org/main/. A demo installation will be set up with a preliminary conversion so people can experiment and review; it can live at http://hg.python.org/example/. code.python.org was also proposed as the hostname. Personally, I think that using the VCS name in the hostname is good because it prevents confusion: it should be clear that you can't use svn or bzr for hg.python.org. hgwebdir can already provide tarballs for every changeset. I think this obviates the need for daily snapshots; we can just point users to tip.tar.gz instead, meaning they will get the latest. If desired, we could even use buildbot results to point to the last good changeset. Python-specific documentation hg comes with good built-in documentation (available through hg help) and a wiki [4] that's full of useful information and recipes. In addition to that, the parts of the developer FAQ [5] concerning version control will gain a section on using hg for Python development. Some of the text will be dependent on the outcome of debate about this PEP (for example, the branching strategy). Think first, commit later? In recent history, old versions of Python have been maintained by a select group of people backporting patches from trunk to release branches. While this may not scale so well as the development pace grows, it also runs into some problems with the current crop of distributed versioning tools. These tools (I believe similar problems would exist for either git, bzr, or hg, though some may cope better than others) are based on the idea of a Directed Acyclic Graph (or DAG), meaning they keep track of relations of changesets. Mercurial itself has a stable branch which is a ''strict'' subset of the unstable branch. This means that generally all fixes for the stable branch get committed against the tip of the stable branch, then they get merged into the unstable branch (which already contains the parent of the new cset). This provides a largely frictionless environment for moving changes from stable to unstable branches. Mistakes, where a change that should go on stable goes on unstable first, do happen, but they're usually easy to fix. That can be done by copying the change over to the stable branch, then trivial-merging with unstable -- meaning the merge in fact ignores the parent from the stable branch). This strategy means a little more work for regular committers, because they have to think about whether their change should go on stable or unstable; they may even have to ask someone else (the RM) before committing. But it also relieves a dedicated group of committers of regular backporting duty, in addition to making it easier to work with the tool. Now would be a good time to consider changing strategies in this regard, although it would be relatively easy to switch to such a model later on. The future of Subversion What happens to the Subversion repositories after the migration? Since the svn server contains a bunch of repositories, not just the CPython one, it will probably live on for a bit as not every project may want to migrate or it takes longer for other projects to migrate. To prevent people from staying behind, we may want to remove migrated projects from the repository. Build identification Python currently provides the sys.subversion tuple to allow Python code to find out exactly what version of Python it's running against. The current version looks something like this: * ('CPython', 'tags/r262', '71600') * ('CPython', 'trunk', '73128M') Another value is returned from Py_GetBuildInfo() in the C API, and available to Python code as part of sys.version: * 'r262:71600, Jun 2 2009, 09:58:33' * 'trunk:73128M, Jun 2 2009, 01:24:14' I propose that the revision identifier will be the short version of hg's revision hash, for example 'dd3ebf81af43', augmented with '+' (instead of 'M') if the working directory from which it was built was modified. This mirrors the output of the hg id command, which is intended for this kind of usage. For the tag/branch identifier, I propose that hg will check for tags on the currently checked out revision, use the tag if there is one ('tip' doesn't count), and uses the branch name otherwise. sys.subversion becomes * ('CPython', '2.6.2', 'dd3ebf81af43') * ('CPython', 'default', 'af694c6a888c+') and the build info string becomes * '2.6.2:dd3ebf81af43, Jun 2 2009, 09:58:33' * 'default:af694c6a888c+, Jun 2 2009, 01:24:14' This reflects that the default branch in hg is called 'default' instead of Subversion's 'trunk', and reflects the proposed new tag format. References [1] http://www.python.org/dev/peps/pep-0347/ [2] http://bitbucket.org/durin42/hgsubversion/ [3] http://hg.xavamedia.nl/cpython/pymigr/ [4] http://www.selenic.com/mercurial/wiki/ [5] http://www.python.org/dev/faq/#version-control ===================================================== Cheers, Dirkjan From greg at krypto.org Fri Jul 3 00:41:48 2009 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 2 Jul 2009 15:41:48 -0700 Subject: [Python-Dev] [Fwd: [issue6397] Implementing Solaris "poll" in the "select" module] In-Reply-To: <4A4C8360.7000701@jcea.es> References: <4A4B9C3A.4090606@jcea.es> <4A4C8360.7000701@jcea.es> Message-ID: <52dc1c820907021541l3ac2301ej6efa032f50e19429@mail.gmail.com> On Thu, Jul 2, 2009 at 2:52 AM, Jesus Cea wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Alexandre Vassalotti wrote: >> On Wed, Jul 1, 2009 at 10:05 PM, Guido van Rossum wrote: >>> The select module already supports the poll() system call. Or is there >>> a special variant that only Solaris has? >>> >> >> I think Jesus refers to /dev/poll?i.e., the interface for >> edge-triggered polling on Solaris. This is the Solaris equivalent of >> FreeBSD's kqueue and Linux's epoll. > > Yes. In python 2.6, Linux and *BSD benefice of access to high performing > "poll"-like interfaces, non portable. Solaris has "/dev/poll" too, so we > can provide another non-portable interface :). In fact I think it could > be interesting to investigate the direct use of "/dev/poll" in Solaris > under "select.poll" implementation, if available, in a transparent way. > With this could be possible to improve efficiency without adding another > non-portable interface. > > But before investigating options, I want to know if there is interest on > it. I think so, because Solaris/OpenSolaris is an important platform, I > will take care of the code, and we already have non-portable interfaces: > "select.epoll" for Linux and "select.kqueue" for *BSD. +1 I'm sure *Solaris users would appreciate this. Go for it. > > - -- > Jesus Cea Avion ? ? ? ? ? ? ? ? ? ? ? ? _/_/ ? ? ?_/_/_/ ? ? ? ?_/_/_/ > jcea at jcea.es - http://www.jcea.es/ ? ? _/_/ ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/ > jabber / xmpp:jcea at jabber.org ? ? ? ? _/_/ ? ?_/_/ ? ? ? ? ?_/_/_/_/_/ > . ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?_/_/ ?_/_/ ? ?_/_/ ? ? ? ? ?_/_/ ?_/_/ > "Things are not so easy" ? ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/ ? ?_/_/ ?_/_/ > "My name is Dump, Core Dump" ? _/_/_/ ? ? ? ?_/_/_/ ? ? ?_/_/ ?_/_/ > "El amor es poner tu felicidad en la felicidad de otro" - Leibniz > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.8 (GNU/Linux) > Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org > > iQCVAwUBSkyDXZlgi5GaxT1NAQJSawP/SUf2AVfzYbqv8F3zZjNBspFu5c3bYVEK > 6A2YSW0WIrNiQtcI9Ka99dCCzHI6cwohlDhxQddJEG4w30hewsFBFJqftSpPLhkv > ToSUNDYe37QmRgVrcns//repWoaMwsrt8MMQU6MLb9pyaoT4Pffe1vcbqg+/XgUN > EJENI7BjSOw= > =BTMr > -----END PGP SIGNATURE----- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > From greg at krypto.org Fri Jul 3 00:47:48 2009 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 2 Jul 2009 15:47:48 -0700 Subject: [Python-Dev] pthread sem PyThread_acquire_lock In-Reply-To: <4A4931E7.1020108@v.loewis.de> References: <4A466846.1050904@aim.com> <4A468507.1060503@v.loewis.de> <4A46A40B.5080202@aim.com> <536685ea0906291233r1b3e8dc4ndf25e7f404a87e22@mail.gmail.com> <4A4931E7.1020108@v.loewis.de> Message-ID: <52dc1c820907021547j6549b1a3od44d3c3b02e4cbb7@mail.gmail.com> On Mon, Jun 29, 2009 at 2:28 PM, "Martin v. L?wis" wrote: >> AFAIK, ignoring EINTR doesn't preclude the calling of signal handlers. > > This is my understanding as well - so I don't think Python actually > "swallows" the signal. > >> A great example is reading from a socket. Whether or not it can be >> interrupted depends on the platform, so catching Ctrl+C often requires >> a timeout loop. >> >> Also, remember that signals are asynchronous in the sense that they >> are handled outside the normal execution flow of a program. Checking >> for EINTR probably isn't the best way to determine if a signal has >> been sent to the program. > > I think it would be reasonable to support "asynchronous" exceptions, > and Python supports SIGINT fairly well most of the time. > > It might be possible to support keyboard interrupts throughout > the system, but changing Python to do so could also cause > incompatibilities. So any change must be done with greatest care, > but simultaneously, should also try to arrange to cover all cases. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org If you want signals to actually be handled in a timely manner, its best to leave the main thread of the program alone as a signal handling thread that just spends its time in a loop of time.sleep(123) calls rather than blocking on any sort of lock. Spawn other threads to do the actual work in your program. Signals are delivered indirectly in the existing CPython implementation by setting an internal flag that the main interpreter thread polls on occasion so blocking calls that do not interrupt and return early being called from the main thread will effectively block signals. From exarkun at divmod.com Fri Jul 3 01:04:51 2009 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Thu, 2 Jul 2009 19:04:51 -0400 Subject: [Python-Dev] pthread sem PyThread_acquire_lock In-Reply-To: <52dc1c820907021547j6549b1a3od44d3c3b02e4cbb7@mail.gmail.com> Message-ID: <20090702230451.22176.208772802.divmod.quotient.12204@henry.divmod.com> On Thu, 2 Jul 2009 15:47:48 -0700, "Gregory P. Smith" wrote: >On Mon, Jun 29, 2009 at 2:28 PM, "Martin v. L?wis" wrote: >>> AFAIK, ignoring EINTR doesn't preclude the calling of signal handlers. >> >> This is my understanding as well - so I don't think Python actually >> "swallows" the signal. >> >>> A great example is reading from a socket. Whether or not it can be >>> interrupted depends on the platform, so catching Ctrl+C often requires >>> a timeout loop. >>> >>> Also, remember that signals are asynchronous in the sense that they >>> are handled outside the normal execution flow of a program. Checking >>> for EINTR probably isn't the best way to determine if a signal has >>> been sent to the program. >> >> I think it would be reasonable to support "asynchronous" exceptions, >> and Python supports SIGINT fairly well most of the time. >> >> It might be possible to support keyboard interrupts throughout >> the system, but changing Python to do so could also cause >> incompatibilities. So any change must be done with greatest care, >> but simultaneously, should also try to arrange to cover all cases. >> >> Regards, >> Martin >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > >If you want signals to actually be handled in a timely manner, its >best to leave the main thread of the program alone as a signal >handling thread that just spends its time in a loop of time.sleep(123) >calls rather than blocking on any sort of lock. Spawn other threads >to do the actual work in your program. Signals are delivered >indirectly in the existing CPython implementation by setting an >internal flag that the main interpreter thread polls on occasion so >blocking calls that do not interrupt and return early being called >from the main thread will effectively block signals. Yes, this is all true now. The question is why the implementation works that way, and whether it is desirable to keep it working that way. Considering *some* of the lock implementations make themselves not interruptable by threads while others don't bother, it seems like *some* change to the status quo is desirable. Jean-Paul From benjamin at python.org Fri Jul 3 01:26:58 2009 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 2 Jul 2009 18:26:58 -0500 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: <1afaf6160907021626w297c1727kcbbf1a0c586f07ac@mail.gmail.com> 2009/7/2 Dirkjan Ochtman : > In response to some rumblings on python-committers and just to request > more feedback, a progress report. I know it's long, I've tried to put > to keep it concise and chunked, though. Thanks very much for working on this, Dirkjan. It may seem rather thankless now, but I'm sure once we wish to Mercurial, the praise will flow. :) > > - Second of all, some planning. I've thought about it a bit, and I > think we should aim for going live with hg on August 1. Given that I'm > on vacation from 8-18 July (and I'm not sure whether I'll be able to > actually work on it during that time, though I imagine I'll be able to > spend some time on it at least), that's quite ambitious, so I'm going > to say it's okay if it slips by a few days. Putting a deadline out > there is a good thing, anyway. Sounds good. > - Fifth, here's a list of things, off the top of my head, that still need doing: > > * Get agreement on branch strategy and branch processing (list of > branches + proposed handling at > http://hg.python.org/pymigr/file/tip/all-branches.txt) <--- PLEASE > REVIEW The io-c branch doesn't need to stay. > * Get agreement on tag processing (first come up with a list) > * Set up hg-ssh infra (should be easy) > * Set up hooks (should be mostly straightforward) > * Set up roundup integration (should be made easier by quick revision > map hgweb extension) > * Write docs > > - Sixth (this is the good part), less obvious things that have been > done or don't need doing: I suppose this includes modifying sys.subversion as described in the PEP? > The SVN tags directory contains a lot of old stuff. Some of these are > not, in fact, full tags, but contain only a smaller subset of the > repository. I think we should keep all release tags, and consider > other tags for inclusion based on requests from the developer > community. I'd like to consider unifying the release tag naming scheme > to make some things more consistent, if people feel that won't create > too many problems. For example, Mercurial itself just uses '1.2.1' as > a tag, where CPython would currently use r121. +1 to unifying tag name style to the current cpython procedure. > Author map > > In order to provide user names the way they are common in hg (in the > 'First Last ' format), we need an author map to map > cvs and svn user names to real names and their email addresses. I have > a complete version of such a map in my migration tools repository [3]. > The email addresses in it might be out of date; that's bound to > happen, although it would be nice to try and have as many people as > possible review it for addresses that are out of date. The current > version also still seems to contain some encoding problems. > Generating .hgignore What effect will the encoding problems have? Does hg require ASCII characters? > Richard Tew has indicated that he'd like the Stackless repository to > also be converted. What other projects in the svn.python.org > repository should be converted? Do we want to convert the peps > repository? distutils? others? I think everything should be converted unless there's a reason not to. (such as the maintainer indicates she doesn't what to migrate) > A number of hooks is currently in use. The hg equivalents for these > should be developed and deployed. The following hooks are being used: > > ? ?* check whitespace: a hook to reject commits in case the > whitespace doesn't match the rules for the Python codebase. Should be > straightforward to re-implement from the current version. We can also > offer a whitespace hook for use with client-side repositories that > people can use; it could either warn about whitespace issues and/or > truncate trailing whitespace from changed lines. Open issue: do we > check only the tip after each push, or do we check every commit in a > changegroup? It might as well be on every commit because it will have to normalized on push anyway. > code.python.org was also proposed as the hostname. Personally, I think > that using the VCS name in the hostname is good because it prevents > confusion: it should be clear that you can't use svn or bzr for > hg.python.org. +1 for hg.python.org > > Think first, commit later? > > In recent history, old versions of Python have been maintained by a > select group of people backporting patches from trunk to release > branches. While this may not scale so well as the development pace > grows, it also runs into some problems with the current crop of > distributed versioning tools. These tools (I believe similar problems > would exist for either git, bzr, or hg, though some may cope better > than others) are based on the idea of a Directed Acyclic Graph (or > DAG), meaning they keep track of relations of changesets. The problem is that Python is much more complicated than the average project. We have many commits that are only applicable one maintenance branch, or just 2.x, or just 3.x; the trunk and py3k will never be subsets of each other. Regardless of where we make commits initially, we need to ability to manage special cases like this easily. -- Regards, Benjamin From skippy.hammond at gmail.com Fri Jul 3 01:01:24 2009 From: skippy.hammond at gmail.com (Mark Hammond) Date: Fri, 03 Jul 2009 09:01:24 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: <4A4D3C44.60504@gmail.com> On 3/07/2009 6:42 AM, Dirkjan Ochtman wrote: > In response to some rumblings on python-committers and just to request > more feedback, a progress report. I know it's long, I've tried to put > to keep it concise and chunked, though. Although this has come up in the past, I don't recall a resolution. What is your plan to handle svn:eol-style? We have some files in the tree which need that support and it isn't clear to me how that would work with the existing win32text extension provided with current mercurial releases. (I've an outstanding patch to hg which should address some of these issues, but without the 'rules' being versioned I fear that would still fall short.) Even more generally, how will you suggest Windows users work? Will local files, in general, have windows line endings or unix? If the latter, will there be hooks in-place to prevent editors on Windows 'accidently' mixing eol styles? If so, this cycles back to the first question - how would we know which files get treated that way? Thanks, Mark From dinov at microsoft.com Fri Jul 3 01:45:39 2009 From: dinov at microsoft.com (Dino Viehland) Date: Thu, 2 Jul 2009 23:45:39 +0000 Subject: [Python-Dev] [IronPython] Exception for setting attributes of built-in type In-Reply-To: <5b0248170906141209y4398defanacfd0bb62504e550@mail.gmail.com> References: <5b0248170906141209y4398defanacfd0bb62504e550@mail.gmail.com> Message-ID: <1A472770E042064698CB5ADC83A12ACD034067D5@TK5EX14MBXC116.redmond.corp.microsoft.com> Seo wrote: > Exception for setting attributes of built-in type differs between > CPython and IronPython. This is not purely theoretical, as > zope.interface tries to set Implements declaration as __implemented__ > attribute of built-in type object, and excepts TypeError. > > Python 2.6.1 > >>> object.flag = True > TypeError: can't set attributes of built-in/extension type 'object' > > IronPython 2.6 > >>> object.flag = True > AttributeError: 'object' object has no attribute 'flag' > > I was surprised that CPython raises TypeError. Library Reference seems > to mention it here: > > exception AttributeError > Raised when an attribute reference or assignment fails. (When an > object does not support attribute references or attribute assignments > at all, TypeError is raised.) > http://docs.python.org/library/exceptions.html > > What does it mean that "an object does not support attribute > references or attribute assignments at all"? Even though Guido said we didn't have to match it I've updated IronPython so that we'll throw the same exception now. The change will be in tomorrow's source push to CodePlex. We've matched specific exceptions in the past and this one is at least documented :) From brett at python.org Fri Jul 3 03:20:13 2009 From: brett at python.org (Brett Cannon) Date: Thu, 2 Jul 2009 18:20:13 -0700 Subject: [Python-Dev] I am back In-Reply-To: References: <20090702023251.GA28507@panix.com> <4A4C9C84.1030702@gmail.com> Message-ID: On Thu, Jul 2, 2009 at 11:44, Terry Reedy wrote: > Brett Cannon wrote: > >> >> >> On Thu, Jul 2, 2009 at 04:39, Nick Coghlan > ncoghlan at gmail.com>> wrote: >> >> Aahz wrote: >> > On Wed, Jul 01, 2009, Brett Cannon wrote: >> >> Anything happen while I was gone that I should be aware of that >> is not >> >> covered in a PEP? >> > >> > Yes. >> >> I agree with Aahz ;) >> >> >> Ah, the smart ass remarks never cease. =) >> > > The biggest news is the release of 3.1, but that might have been recorded > in the relevant PEP, and it certainly hard to miss. > Yeah, it hit Twitter immediately thanks to Benjamin. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Jul 3 05:49:05 2009 From: guido at python.org (Guido van Rossum) Date: Thu, 2 Jul 2009 20:49:05 -0700 Subject: [Python-Dev] pthread sem PyThread_acquire_lock In-Reply-To: <20090702230451.22176.208772802.divmod.quotient.12204@henry.divmod.com> References: <52dc1c820907021547j6549b1a3od44d3c3b02e4cbb7@mail.gmail.com> <20090702230451.22176.208772802.divmod.quotient.12204@henry.divmod.com> Message-ID: On Thu, Jul 2, 2009 at 4:04 PM, Jean-Paul Calderone wrote: > On Thu, 2 Jul 2009 15:47:48 -0700, "Gregory P. Smith" >> If you want signals to actually be handled in a timely manner, its >> best to leave the main thread of the program alone as a signal >> handling thread that just spends its time in a loop of time.sleep(123) >> calls rather than blocking on any sort of lock. ?Spawn other threads >> to do the actual work in your program. ?Signals are delivered >> indirectly in the existing CPython implementation by setting an >> internal flag that the main interpreter thread polls on occasion so >> blocking calls that do not interrupt and return early being called >> from the main thread will effectively block signals. > > Yes, this is all true now. ?The question is why the implementation > works that way, and whether it is desirable to keep it working that > way. > > Considering *some* of the lock implementations make themselves not > interruptable by threads while others don't bother, it seems like > *some* change to the status quo is desirable. I'm probably to blame for this, but IIRC when we did it, lock acquisitions were not interruptible by signals. However, that was before posix threads were even standardized. Nowadays I believe we use pthreads everywhere except on Windows. But I know nearly nothing about pthreads. If pthreads mutex acquisition is (or can easily be made) interruptible by signals I'd be all for adding signal handling to them, assuming it doesn't slow things down much. Also, if there are ways to do the "try to acquire a lock or wait until a given timeout" natively and while handling signals, that would be a great improvement. I'll probably regret saying this within an hour though, -- --Guido van Rossum (home page: http://www.python.org/~guido/) From dirkjan at ochtman.nl Fri Jul 3 13:28:02 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Fri, 3 Jul 2009 13:28:02 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4D3C44.60504@gmail.com> References: <4A4D3C44.60504@gmail.com> Message-ID: On Fri, Jul 3, 2009 at 01:01, Mark Hammond wrote: > Although this has come up in the past, I don't recall a resolution. > > What is your plan to handle svn:eol-style? ?We have some files in the tree > which need that support and it isn't clear to me how that would work with > the existing win32text extension provided with current mercurial releases. > ?(I've an outstanding patch to hg which should address some of these issues, > but without the 'rules' being versioned I fear that would still fall short.) What files would need what? Are there any files that really need to be \r\n on Windows and \n on Unix (and possibly \r on Mac)? I remember one file was discussed separately, but I think the outcome there was that it could just always be \r\n (since it wasn't used at all on non-Windows platforms). Anyway, knowing specific requirements (or where to find them) would help here. > Even more generally, how will you suggest Windows users work? ?Will local > files, in general, have windows line endings or unix? ?If the latter, will > there be hooks in-place to prevent editors on Windows 'accidently' mixing > eol styles? ?If so, this cycles back to the first question - how would we > know which files get treated that way? There will be a server-side hook to check whitespace. People will also be able to install it for commit-time. I think just using \n by default everywhere is a good default (though I almost always use Windows client machine, I do all nearly all of my development through a terminal on several Linux boxen), except where it isn't. People who want to use can set up the win32text stuff to get \r\n on Windows if they feel they need that -- we can provide information about that in the dev FAQ (although it would be nice if someone else who was more familiar with it -- like yourself! -- would write it). Cheers, Dirkjan From lukasz.langa at stxnext.pl Fri Jul 3 14:57:37 2009 From: lukasz.langa at stxnext.pl (=?UTF-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 3 Jul 2009 13:57:37 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com> References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com> Message-ID: On 2009-06-22 at 14:23, Tarek Ziad? wrote: > Hello, > > We have polished out PEP 376 and its code prototype at Distutils-SIG. > It seems to fullfill now all the requirements, > so I am mailing it here again, for a new round of feedback, if needed. Hello. I have read the current version of the PEP from trunk and would like to make some comments about it. The .egg-info as a directory ============================ At our company, after much fiddling about the current distutils and setuptools, we have developed a solution that enables us to serve pseudo-eggs. What I mean by that is that by plugging into the PEP 302 infrastructure, we were able to make non-filesystem-based repositories behave like `.egg` files. There are a couple of use cases for that implementation, the most important ones being: * being able to deploy encrypted and sealed blobs with a binary loader, to discourage fiddling around the source code by its users * being able to serve modules from a database back-end instead of the filesystem This solution depends on the `EGG-INFO` to be fetched from the `.egg` itself. We wouldn't want to use a static filesystem-based folder because it breaks the encryption use-case by revealing the `SOURCES.txt` information. It also breaks the database back-end usecase by presenting static versioning information whereas the whole point of using a database backed "egg" is to have the newest version installed at all times. Moreover, the proposed ``egginfo_dirname()`` routine is a step-back from the ``pkg_resources`` approach where we don't enforce resources to reside on a traditional filesystem. The uninstall feature ===================== This is a great feature but I don't seem to understand why there seems to be a consensus here that it's good for it not to even warn about dependencies, let alone manage them. ``easy_install`` and co. does manage dependencies while installing, why shouldn't it be symmetrical in that regard? Moreover, wouldn't dependency management be useful while using ``easy_install`` to *upgrade* an installation of a package? The second issue is that while having ``python -m distutils.uninstall`` is the preferred way to go, I would argue that enabling an ``uninstall`` entry-point in ``setup.py`` is desirable as well, if only for retaining an intuitive symmetrical system where I can do ``python setup.py install`` and ``python setup.py uninstall`` as well. I'm not forced to download the sources just to uninstall the package (``distutils.uninstall`` covers this use case) but if I do have the source code available, it would feel very natural to use the ``setup.py`` provided. There already is the precedent of ``setup.py develop --uninstall``. Having a ``setup.py uninstall`` could handle this use case as well. More high-level remarks ======================= This isn't probably the best place to cover this but may serve as a rationale for the above .egg-info problems we have with the proposed changes. It's great that you push the metadata functionality to core distutils. At the same time, it would be feasible to think over the whole `egg` concept. We would argue that having the `egg` format as a container more-less agnostic to the underlying data storage structure would be **very** useful. Imagine installable `egg` support packages which would enable you to: * use other compression formats beside ZIP * use other means of module storage beside the filesystem * use sealed and encrypted archives for deployment * use programmatic `egg`s for licensing, etc. * ``easy_install`` from different protocols, archive formats, etc. Decoupling the `egg` format from the concrete "zip-based" or "directory-based" implementations is a step forward in that direction. In that regard, the solution PEP 376 is proposing isn't ultimately solving the issues we're having. Thanks for your time. -- Best regards, ?ukasz Langa Senior Developer STX-Next Sp. z o.o. tel: +48 791 080 144 skype: lukaszlanga From stephen at xemacs.org Fri Jul 3 15:29:41 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 03 Jul 2009 22:29:41 +0900 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp> Dirkjan Ochtman writes: > Mercurial has two basic ways of using branches: cloned branches, where > each branch is kept in a separate repository, and named branches, > where each revision keeps metadata to note on which branch it belongs. > The former makes it easier to distinguish branches, at the expense of > requiring more disk space on the client. The latter makes it a little > easier to switch between branches, but often has somewhat unintuitive > results for people (though this has been getting better in recent > versions of Mercurial). I'll have to try them again now that 1.3 is out, but I found Mercurial named branches fundamentally unsuited to release management. > I'm still a bit on the fence about whether Python should adopt cloned > branches and named branches. Since it usually makes more sense to tag > releases on the maintenance branch, for example, mainline history > would not contain release tags if we used cloned branches. Ditto named branches. The problem is that (unless the internal implementation has changed very recently) a Mercurial revision can be on exactly one named branch (or on the trunk). > A disadvantage might be that the used clones will be a good bit larger > (since they essentially contain all other branches as well). This can > me mitigated by keeping non-release (feature) branches in separate > clones. Which defeats the purpose of having named branches, really. (I mean the version control purpose; obviously it still can save disk space.) > Also note that it's still possible to clone a single named > branch from a combined clone, by specifying the branch as in hg clone > http://hg.python.org/main/#2.6-maint. Unless you're really short on space, though, that's not a big deal. What would be more important to me (not that I matter for the purpose of Python, but in XEmacs -- also a Mercurial shop -- I do :-) would be the other way around: pulling an external branch into a named branch. I have a feeling that working with such a repository with others would be a little difficult. > too many problems. For example, Mercurial itself just uses '1.2.1' as > a tag, where CPython would currently use r121. Stick with the CPython notation. At XEmacs, continuity of tags has made our beta testers happy. (Well, the two of them who bothered to mention it, anyway. :-) > code.python.org was also proposed as the hostname. Personally, I think > that using the VCS name in the hostname is good because it prevents > confusion: it should be clear that you can't use svn or bzr for > hg.python.org. Agreed, although "can't" is a little too strong. It might work (there are a lot of places where http://ftp.example.com works just fine, for example), but we don't want people to expect it to, and "http://REPOHOST.python.org/" should take your browser or your client to the official repo (which will be the hg repo), not to some index of repos that happen to live on the same host. > Mercurial itself has a stable branch which is a ''strict'' subset of > the unstable branch. As others (MvL, I think) have commented, this isn't really relevant to Python which generally has four mainlines going at once. I don't see why the requirements are going to change with the shift to hg, and I see no reason why hg won't handle the existing workflow just fine. Note that PEP 374 was written on the assumption that the existing workflow will *not* change until the committers have gotten used to Mercurial, and then it will change in the natural way. Ie, some one will say, "you know, this bit of red tape isn't needed any more if we do X, so let's do X", and after a cascade of 100*"+1 " we do X. :-) Other than that, looks to be shaping up well. (Note: I don't have any comments on subversion-specific aspects, as XEmacs went directly from CVS to Mercurial). From mhammond at skippinet.com.au Fri Jul 3 15:31:17 2009 From: mhammond at skippinet.com.au (Mark Hammond) Date: Fri, 03 Jul 2009 23:31:17 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A4D3C44.60504@gmail.com> Message-ID: <4A4E0825.5020407@skippinet.com.au> On 3/07/2009 9:28 PM, Dirkjan Ochtman wrote: > On Fri, Jul 3, 2009 at 01:01, Mark Hammond wrote: > >> Although this has come up in the past, I don't recall a resolution. >> >> What is your plan to handle svn:eol-style? We have some files in the tree >> which need that support and it isn't clear to me how that would work with >> the existing win32text extension provided with current mercurial releases. >> (I've an outstanding patch to hg which should address some of these issues, >> but without the 'rules' being versioned I fear that would still fall short.) >> > > What files would need what? Are there any files that really need to be > \r\n on Windows and \n on Unix (and possibly \r on Mac)? I remember > one file was discussed separately, but I think the outcome there was > that it could just always be \r\n (since it wasn't used at all on > non-Windows platforms). Anyway, knowing specific requirements (or > where to find them) would help here. > > >> Even more generally, how will you suggest Windows users work? Will local >> files, in general, have windows line endings or unix? If the latter, will >> there be hooks in-place to prevent editors on Windows 'accidently' mixing >> eol styles? If so, this cycles back to the first question - how would we >> know which files get treated that way? >> > > There will be a server-side hook to check whitespace. People will also > be able to install it for commit-time. > > I think just using \n by default everywhere is a good default (though > I almost always use Windows client machine, I do all nearly all of my > development through a terminal on several Linux boxen), except where > it isn't. So we must work without effective EOL support? I fear we will end up like the mozilla hg repo with some files in windows line endings and some with linux. While my editing tools are good enough to preserve existing EOL styles, I've found myself accidentally checking in new \r\n terminated files in a repo which otherwise uses \n line endings. IMO, SVN's EOL support was better than no EOL support. > People who want to use can set up the win32text stuff to get > \r\n on Windows if they feel they need that -- we can provide > information about that in the dev FAQ (although it would be nice if > someone else who was more familiar with it -- like yourself! -- would > write it). > This is exactly why I was asking for your advice - I can't work out how to work effectively with win32text as it stands myself, so remain stuck accidently checking in files with inappropriate line endings and stuck working out how to move pywin32's CVS repo with abandoning the very primitive EOL safety it offers... Cheers, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From dirkjan at ochtman.nl Fri Jul 3 15:43:54 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Fri, 3 Jul 2009 15:43:54 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E0825.5020407@skippinet.com.au> References: <4A4D3C44.60504@gmail.com> <4A4E0825.5020407@skippinet.com.au> Message-ID: On Fri, Jul 3, 2009 at 15:31, Mark Hammond wrote: > So we must work without effective EOL support?? I fear we will end up like > the mozilla hg repo with some files in windows line endings and some with > linux.? While my editing tools are good enough to preserve existing EOL > styles, I've found myself accidentally checking in new \r\n terminated files > in a repo which otherwise uses \n line endings.? IMO, SVN's EOL support was > better than no EOL support. This is why we'll have hooks -- to prevent you from pushing changesets with inappropriate, to say the least, and, if you're willing to do a little bit of extra work, to prevent you from committing them. > This is exactly why I was asking for your advice - I can't work out how to > work effectively with win32text as it stands myself, so remain stuck > accidently checking in files with inappropriate line endings and stuck > working out how to move pywin32's CVS repo with abandoning the very > primitive EOL safety it offers... As long as the difference between \r\n- and \n-based files is clear and can be reasoned about, I don't see why having some of both (I'm assuming an overwhelming majority will have one, and only a few the other) is a big problem. But feel free to enlighten me! Cheers, Dirkjan From dirkjan at ochtman.nl Fri Jul 3 15:49:31 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Fri, 3 Jul 2009 15:49:31 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Fri, Jul 3, 2009 at 15:29, Stephen J. Turnbull wrote: > I'll have to try them again now that 1.3 is out, but I found Mercurial > named branches fundamentally unsuited to release management. Can you explain why, please? It's not clear from what you say below. > Ditto named branches. ?The problem is that (unless the internal > implementation has changed very recently) a Mercurial revision can be > on exactly one named branch (or on the trunk). That's still true. > Which defeats the purpose of having named branches, really. ?(I mean > the version control purpose; obviously it still can save disk space.) Why does it defeat the purpose? What, in your opinion, is the purpose? > Unless you're really short on space, though, that's not a big deal. > What would be more important to me (not that I matter for the purpose > of Python, but in XEmacs -- also a Mercurial shop -- I do :-) would be > the other way around: pulling an external branch into a named branch. > I have a feeling that working with such a repository with others would > be a little difficult. Can you give an example? > Stick with the CPython notation. ?At XEmacs, continuity of tags has > made our beta testers happy. ?(Well, the two of them who bothered to > mention it, anyway. :-) Right; Benjamin also mentioned that processing the tags just to be consistent with the recent tagging scheme would probably be the best solution. > As others (MvL, I think) have commented, this isn't really relevant to > Python which generally has four mainlines going at once. ?I don't see > why the requirements are going to change with the shift to hg, and I > see no reason why hg won't handle the existing workflow just fine. It will handle it, for sure, but I think it would all go easier if we could work with stricter subset branches (and leave the effective cherrypicking for the occasional problem). Cheers, Dirkjan From p.f.moore at gmail.com Fri Jul 3 15:54:13 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 3 Jul 2009 14:54:13 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com> Message-ID: <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com> 2009/7/3 ?ukasz Langa : > > On 2009-06-22 at 14:23, Tarek Ziad? wrote: > >> Hello, >> >> We have polished out PEP 376 and its code prototype at Distutils-SIG. >> It seems to fullfill now all the requirements, >> so I am mailing it here again, for a new round of feedback, if needed. > > > Hello. > I have read the current version of the PEP from trunk and would like to make > some comments about it. > > The .egg-info as a directory > ============================ > At our company, after much fiddling about the current distutils and > setuptools, we have developed a solution that enables us to serve > pseudo-eggs. What I mean by that is that by plugging into the PEP 302 > infrastructure, we were able to make non-filesystem-based repositories > behave like `.egg` files. There are a couple of use cases for that > implementation, the most important ones being: > * being able to deploy encrypted and sealed blobs with a binary loader, to > discourage fiddling around the source code by its users > * being able to serve modules from a database back-end instead of the > filesystem > > This solution depends on the `EGG-INFO` to be fetched from the `.egg` > itself. We wouldn't want to use a static filesystem-based folder because it > breaks the encryption use-case by revealing the `SOURCES.txt` information. > It also breaks the database back-end usecase by presenting static versioning > information whereas the whole point of using a database backed "egg" is to > have the newest version installed at all times. This is a good point. Distutils only installs files in the filesystem - it has no facilities for installing packages based on any other sort of PEP 302 based importers. Hence, PEP 376 in principle should only relate to filesystem-based distributions. But it also mentions zipfile-based distributions: "Notice that the API is organized in five classes that work with directories and Zip files (so it works with files included in Zip files, see PEP 273 for more details [8])." This is wrong. The PEP should either (1) restrict itself to filesystem based implementations (leaving the problem of other PEP 302 loaders to systems that manage these) or (2) be defined in a sufficiently general way that it can be implemented for *any* PEP 302 based loader - which probably means extending the PEP 302 protocols - and supplying zipfile functions as an example of how this is used. I believe that (1) is unlikely to be sufficient for real world use. Zip files (eggs, py2exe embedded modules, etc) are far too important a real world use case to ignore. The problem with (2) is that it requires significant extra work. But special-casing zip files (as the present implementation appears to do) will break as soon as any other PEP 302 compliant format becomes popular. > Moreover, the proposed ``egginfo_dirname()`` routine is a step-back from the > ``pkg_resources`` approach where we don't enforce resources to reside on a > traditional filesystem. On the other hand, pkgutil.get_data is the standard library means of reading resources from a package. This is PEP 302 compliant now. This new PEP doesn't affect that. > The uninstall feature > ===================== > This is a great feature but I don't seem to understand why there seems to be > a consensus here that it's good for it not to even warn about dependencies, > let alone manage them. ``easy_install`` and co. does manage dependencies > while installing, why shouldn't it be symmetrical in that regard? Moreover, > wouldn't dependency management be useful while using ``easy_install`` to > *upgrade* an installation of a package? easy_install is not a standard library feature - the fact that it has no uninstall facility is not relevant. python setup.py install is the standard library feature. The new uninstall is intended to correspond to that. > More high-level remarks > ======================= > This isn't probably the best place to cover this but may serve as a > rationale for the above .egg-info problems we have with the proposed > changes. > > It's great that you push the metadata functionality to core distutils. At > the same time, it would be feasible to think over the whole `egg` concept. > We would argue that having the `egg` format as a container more-less > agnostic to the underlying data storage structure would be **very** useful. > Imagine installable `egg` support packages which would enable you to: > * use other compression formats beside ZIP > * use other means of module storage beside the filesystem > * use sealed and encrypted archives for deployment > * use programmatic `egg`s for licensing, etc. PEP 302 gives you (in theory) all of this now. I'd recommend you look at it - and at Brett's importlib, which will make experimenting with such things far easier. What PEP 302 doesn't provide is package management. But Python itself doesn't provide package management, except in the form of distutils setup.py install - which is solely filesystem based. Maybe there's a case for extending PEP 302 and distutils to allow integrated management of other forms of importer format, but that's a huge other project, which no-one to my knowledge is even looking at. > * ``easy_install`` from different protocols, archive formats, etc. easy_install is not a standard library feature, so is not relevant here. > Decoupling the `egg` format from the concrete "zip-based" or > "directory-based" implementations is a step forward in that direction. In > that regard, the solution PEP 376 is proposing isn't ultimately solving the > issues we're having. Eggs are fundamentally a PEP 302 zip file format. There are some extra bits of metadata for setuptools/easy_install in there (as I understand things) but essentially they are zip files. When you say "decoupling the egg format", I assume you mean "decoupling the egg metadata" - which is fine, but to properly decouple, you need API level access to the metadata. PEP 376 offers read-only access, but as you rightly point out, it is only for filesystem data (and some form of zip file, which appears to be limited in some way, as it isn't PEP 302 based, and the actual format isn't defined anywhere). The basic point here is that PEP 376 needs to define precisely how pkgutil.get_distributions() scans sys.path looking for ".egg-info directories". What does it do for sys.path entries that don't correspond to filesystem directories? (Note - these may or may not be zip files. Even if they are zip files, an earlier entry on sys.path_hooks could have taken precedence. At the very least, you should only process path entries as zip files if their importer - in sys.path_importer_cache or via an explicit path hook scan - is a zipimporter object.). To be honest, this is a major can of worms. But if PEP 376 is not going to support PEP 302, then it must state that fact explicitly, to avoid giving people false expectations - particularly with Brett's importlib in Python 3.1, which will make it far easier for people to experiment with new packaging formats such as the ones Lukasz mentions above. And it MUST fail gracefully in the face of unsupported importer types. Paul. From ncoghlan at gmail.com Fri Jul 3 16:04:57 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 04 Jul 2009 00:04:57 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E0825.5020407@skippinet.com.au> References: <4A4D3C44.60504@gmail.com> <4A4E0825.5020407@skippinet.com.au> Message-ID: <4A4E1009.4050901@gmail.com> Mark Hammond wrote: > So we must work without effective EOL support? I fear we will end up > like the mozilla hg repo with some files in windows line endings and > some with linux. While my editing tools are good enough to preserve > existing EOL styles, I've found myself accidentally checking in new \r\n > terminated files in a repo which otherwise uses \n line endings. IMO, > SVN's EOL support was better than no EOL support. If Mercury doesn't have automatic whitespace conversion along the lines of svn:eol-style, then the existing white-space checking script probably needs to be updated to enforce the appropriate line endings for all files. If we default to Unix line endings for most files, then the checking script can be made aware of which files should always have Windows line endings (I believe the various Visual Studio files need them, batch files are probably best left with Windows line endings in the repository, and I expect there are other files in PC and PCbuild that need them as well). However, I expect that would still be painful to work with for Windows developers, even if it prevented any line ending problems from making their way into the main repository. I believe that is where the win32text extensions can help. Looking at the Wiki page for win32text [1], I believe it would be a matter of configuring the extension to encode and decode all files with the extensions: *.py *.pyw *.h *.c *.in *.rst *.asdl That said, I don't see a way to tell win32text to also translate files which don't have an extension at all (e.g. NEWS or ACKS), and there doesn't seem to be a way to tell it to *skip* files matching certain patterns (if there was, we could just tell it to ignore extensions like .bat, .sln, .vcproj, .vsprops, .dps, .dsw, .wse, .ico, .bmp and convert everything else) Cheers, Nick. [1] http://mercurial.selenic.com/wiki/Win32TextExtension -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Fri Jul 3 16:28:02 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 04 Jul 2009 00:28:02 +1000 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com> References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com> <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com> Message-ID: <4A4E1572.3000801@gmail.com> Paul Moore wrote: > To be honest, this is a major can of worms. But if PEP 376 is not > going to support PEP 302, then it must state that fact explicitly, to > avoid giving people false expectations - particularly with Brett's > importlib in Python 3.1, which will make it far easier for people to > experiment with new packaging formats such as the ones Lukasz mentions > above. And it MUST fail gracefully in the face of unsupported importer > types. importlib is in 2.7 as well. I agree that even if the reference implementation of PEP 376 only handles normal directories and zipfiles, the PEP itself needs to explain how the developer of a custom PEP 302 importer or loader can hook into the mechanism in order to provide metadata that distutils can understand. Or, as you suggest, state explicitly that the proposal at this stage is specifically limited to filesystem and zipfile packages and defer extension to arbitrary PEP 302 importers and loaders to a later PEP. To be honest, I'd personally be OK with the latter strategy - while other PEP 302 loaders and importers do exist (as Lukasz'z post shows), it would be unfortunate to unduly hold up improved installation metadata for the vast majority of typical use cases while we try to figure out ways to support the more esoteric use cases. Supporting both would obviously be better, but given the choice between the status quo and partial support, I believe this is a case where the partial support would still be worthwhile. I will note (and I believe this is also the main point that Lukasz was making) that having the distribution metadata outside the distribution as currently proposed in PEP 376 is going to make any eventual PEP 302 integration much harder - 302 importers only provide a mechanism for accessing files inside the distribution, not "adjacent" to them, so the mechanism in the PEP doesn't generalise properly. I suspect this limitation of the PEP 302 APIs is the origin of the setuptools format that embeds the metadata inside the distribution - it lets you get at the metadata without having to assume that it exists directly on the filesystem anywhere. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ziade.tarek at gmail.com Fri Jul 3 16:31:31 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 3 Jul 2009 16:31:31 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com> References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com> <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com> Message-ID: <94bdd2610907030731s395a962byced058b97c2ef6d@mail.gmail.com> 2009/7/3 Paul Moore : > This is a good point. Distutils only installs files in the filesystem > - it has no facilities for installing packages based on any other sort > of PEP 302 based importers. Hence, PEP 376 in principle should only > relate to filesystem-based distributions. But it also mentions > zipfile-based distributions: "Notice that the API is organized in five > classes that work with directories and Zip files (so it works with > files included in Zip files, see PEP 273 for more details [8])." > > This is wrong. The PEP should either (1) restrict itself to filesystem > based implementations (leaving the problem of other PEP 302 loaders to > systems that manage these) or (2) be defined in a sufficiently general > way that it can be implemented for *any* PEP 302 based loader - which > probably means extending the PEP 302 protocols - and supplying zipfile > functions as an example of how this is used. > > I believe that (1) is unlikely to be sufficient for real world use. > Zip files (eggs, py2exe embedded modules, etc) are far too important a > real world use case to ignore. The problem with (2) is that it > requires significant extra work. But special-casing zip files (as the > present implementation appears to do) will break as soon as any other > PEP 302 compliant format becomes popular. >> Moreover, the proposed ``egginfo_dirname()`` routine is a step-back from the >> ``pkg_resources`` approach where we don't enforce resources to reside on a >> traditional filesystem. > > On the other hand, pkgutil.get_data is the standard library means of > reading resources from a package. This is PEP 302 compliant now. This > new PEP doesn't affect that. > Right. While it would be feasible to make pgutil works with PEP 302 loaders, we would still need to define in a generic way the content of the RECORD files. Right now it works for directory and zipped files since it's expressed with '/' separated paths. And if I understand PEP 302 right, any backend would be able to handle these paths no matter how they are stored, as long as the implement APIs like get_data() > What PEP 302 doesn't provide is package management. But Python itself > doesn't provide package management, except in the form of distutils > setup.py install - which is solely filesystem based. > > Maybe there's a case for extending PEP 302 and distutils to allow > integrated management of other forms of importer format, but that's a > huge other project, which no-one to my knowledge is even looking at. Sounds like a fully-featured packaging managment system, which is imho, out of scope. And I don't see PEP 376 making it impossible for someone to build such a packaging system on the top of distutils. I've started one myself for the sake of experimentation, with built-in multiversion support, for a full replacement of site-packages. > Eggs are fundamentally a PEP 302 zip file format. There are some extra > bits of metadata for setuptools/easy_install in there (as I understand > things) but essentially they are zip files. When you say "decoupling > the egg format", I assume you mean "decoupling the egg metadata" - > which is fine, but to properly decouple, you need API level access to > the metadata. PEP 376 offers read-only access, but as you rightly > point out, it is only for filesystem data (and some form of zip file, > which appears to be limited in some way, as it isn't PEP 302 based, > and the actual format isn't defined anywhere). And also PEP 376 goal is to define a single standard location of egg-info files for filesystem data. The zip form was built so it could work with zipped site-packages directories, like what the py2app project uses. > > The basic point here is that PEP 376 needs to define precisely how > pkgutil.get_distributions() scans sys.path looking for ".egg-info > directories". What does it do for sys.path entries that don't > correspond to filesystem directories? (Note - these may or may not be > zip files. Even if they are zip files, an earlier entry on > sys.path_hooks could have taken precedence. At the very least, you > should only process path entries as zip files if their importer - in > sys.path_importer_cache or via an explicit path hook scan - is a > zipimporter object.). I'll add more details on that part. right now it visits directories and zip files. Tarek From ziade.tarek at gmail.com Fri Jul 3 16:44:34 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 3 Jul 2009 16:44:34 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <4A4E1572.3000801@gmail.com> References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com> <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com> <4A4E1572.3000801@gmail.com> Message-ID: <94bdd2610907030744p6cc1fbbavd5fcc723c5a8917b@mail.gmail.com> On Fri, Jul 3, 2009 at 4:28 PM, Nick Coghlan wrote: > I will note (and I believe this is also the main point that Lukasz was > making) that having the distribution metadata outside the distribution > as currently proposed in PEP 376 is going to make any eventual PEP 302 > integration much harder - 302 importers only provide a mechanism for > accessing files inside the distribution, not "adjacent" to them, so the > mechanism in the PEP doesn't generalise properly. > > I suspect this limitation of the PEP 302 APIs is the origin of the > setuptools format that embeds the metadata inside the distribution - it > lets you get at the metadata without having to assume that it exists > directly on the filesystem anywhere. Maybe we could rework the pkgutil classes for PEP 376 so they look like an implementation of the PEP 302 protocol for directories and zip files, with an extra "get_metadata()" API and state that it could be an extension for PEP 302 later. From g.brandl at gmx.net Fri Jul 3 17:17:02 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 03 Jul 2009 17:17:02 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: Dirkjan Ochtman schrieb: > - Fifth, here's a list of things, off the top of my head, that still need doing: > > * Get agreement on branch strategy and branch processing (list of > branches + proposed handling at > http://hg.python.org/pymigr/file/tip/all-branches.txt) <--- PLEASE > REVIEW Do you have a key to the second column in that file? E.g. the difference between "strip" and "discard" is not clear to me. "strip partial"? Why are there branch names starting with "../"? [PEP 385] > ============================================================== > > Motivation > > After having decided to switch to the Mercurial DVCS, the actual > migration still has to be performed. In the case of an important piece > of infrastructure like the version control system for a large, > distributed project like Python, this is a significant effort. This > PEP is an attempt to describe the steps that must be taken for further > discussion. It's somewhat similar to PEP 347 [1], which discussed the > migration to SVN. > > To make the most of hg, I (Dirkjan) would like to make a high-fidelity > conversion, such that (a) as much of the svn metadata as possible is > retained, and (b) all metadata is converted to formats that are common > in Mercurial. This way, tools written for Mercurial can be optimally > used. In order to do this, I want to use the hgsubversion [2] software > to do an initial conversion. This hg extension is focused on providing > high-quality conversion from Subversion to Mercurial for use in > two-way correspondence, meaning it doesn't throw away as much > available metadata as other solutions. > > Such a conversion also seems like a good time to reconsider the > contents of the repository and determine if some things are still > valuable. In this spirit, the following sections also propose > discarding some of the older metadata. > Timeline > > TBD; needs fully working hgsubversion and consensus on this document. > Transition plan > Branch strategy > > Mercurial has two basic ways of using branches: cloned branches, where > each branch is kept in a separate repository, and named branches, > where each revision keeps metadata to note on which branch it belongs. > The former makes it easier to distinguish branches, at the expense of > requiring more disk space on the client. The latter makes it a little > easier to switch between branches, but often has somewhat unintuitive > results for people (though this has been getting better in recent > versions of Mercurial). > > I'm still a bit on the fence about whether Python should adopt cloned > branches and named branches. Since it usually makes more sense to tag > releases on the maintenance branch, for example, mainline history > would not contain release tags if we used cloned branches. Also, > Mercurial 1.2 and 1.3 have the necessary tools to make named branches > less painful (because they can be properly closed and closed heads are > no longer considered in relevant cases). > > A disadvantage might be that the used clones will be a good bit larger > (since they essentially contain all other branches as well). This can > me mitigated by keeping non-release (feature) branches in separate > clones. Also note that it's still possible to clone a single named > branch from a combined clone, by specifying the branch as in hg clone > http://hg.python.org/main/#2.6-maint. Keeping the py3k history in a > separate clone problably also makes sense. * Does it work with "hg pull" etc. too, afterwards? * Does it support more than one branch? > XXX To do: size comparison for selected separation scenarios. > Converting branches > > There are quite a lot of branches in SVN's branches directory. I > propose to clean this up a bit, by employing the following the > strategy: > > * Keep all release (maintenance) branches > * Discard branches that haven't been touched in 18 months, unless > somone indicates there's still interest in such a branch > * Keep branches that have been touched in the last 18 months, > unless someone indicates the branch can be deprecated I would just kill all feature branches unless someone indicates it is still used. There are very few active feature branches. (I guess in the case a branch gets killed erroneously it could still be re-created after the conversion?) > Converting tags > > The SVN tags directory contains a lot of old stuff. Some of these are > not, in fact, full tags, but contain only a smaller subset of the > repository. I think we should keep all release tags, and consider > other tags for inclusion based on requests from the developer > community. I'd like to consider unifying the release tag naming scheme > to make some things more consistent, if people feel that won't create > too many problems. For example, Mercurial itself just uses '1.2.1' as > a tag, where CPython would currently use r121. +1 for readable tag names. +1 for throwing out old questionable tag names. > Generating .hgignore > > The .hgignore file can be used in Mercurial repositories to help > ignore files that are not eligible for version control. It does this > by employing several possible forms of pattern matching. The current > Python repository already includes a rudimentary .hgignore file to > help with using the hg mirrors. > > It might be useful to have the .hgignore be generated automatically > from svn:ignore properties. This would make sure all historic > revisions also have useful ignore information (though one could argue > ignoring isn't really relevant to just checking out an old revision). I guess that's not necessary. People can just add stuff to .hgignore when they see something that should be there. > hg-ssh > > Developers should access the repositories through ssh, similar to the > current setup. Public keys can be used to grant people access to a > shared hg@ account. A hgwebdir instance should also be set up for easy > browsing and read-only access. If we're using ssh, developers should > trivially be able to start new clones (for longer-term features that > profit from a separate branch). +1. > Hooks > > A number of hooks is currently in use. The hg equivalents for these > should be developed and deployed. The following hooks are being used: > > * check whitespace: a hook to reject commits in case the > whitespace doesn't match the rules for the Python codebase. Should be > straightforward to re-implement from the current version. We can also > offer a whitespace hook for use with client-side repositories that > people can use; it could either warn about whitespace issues and/or > truncate trailing whitespace from changed lines. Open issue: do we > check only the tip after each push, or do we check every commit in a > changegroup? Only checking the tip would make it possible for people to revert their whitespace commits, but then -- if they have the local hook -- they shouldn't do that anyway. > * commit mails: we can leverage the notify extension for this As long as it can send diffs... > * check contributors: in the current setup, all changesets bear > the username of committers, who must have signed the contributor > agreement. In a DVCS, the committers are not necessarily the same > people who push, and so we can't check if the committer is a > contributor. We could use a hook to check if the committer is a > contributor if we keep a list of registered contributors. That gets very ugly as soon as you start pulling from repos that just fix a small typo or so. > code.python.org was also proposed as the hostname. Personally, I think > that using the VCS name in the hostname is good because it prevents > confusion: it should be clear that you can't use svn or bzr for > hg.python.org. Yes, and it mirrors svn.python.org. > Mercurial itself has a stable branch which is a ''strict'' subset of > the unstable branch. This means that generally all fixes for the > stable branch get committed against the tip of the stable branch, then > they get merged into the unstable branch (which already contains the > parent of the new cset). This provides a largely frictionless > environment for moving changes from stable to unstable branches. > Mistakes, where a change that should go on stable goes on unstable > first, do happen, but they're usually easy to fix. That can be done by > copying the change over to the stable branch, then trivial-merging > with unstable -- meaning the merge in fact ignores the parent from the > stable branch). > > This strategy means a little more work for regular committers, because > they have to think about whether their change should go on stable or > unstable; they may even have to ask someone else (the RM) before > committing. But it also relieves a dedicated group of committers of > regular backporting duty, in addition to making it easier to work with > the tool. Strong +1 for that. > I propose that the revision identifier will be the short version of > hg's revision hash, for example 'dd3ebf81af43', augmented with '+' > (instead of 'M') if the working directory from which it was built was > modified. This mirrors the output of the hg id command, which is > intended for this kind of usage. > > For the tag/branch identifier, I propose that hg will check for tags > on the currently checked out revision, use the tag if there is one > ('tip' doesn't count), and uses the branch name otherwise. > sys.subversion becomes > > * ('CPython', '2.6.2', 'dd3ebf81af43') > * ('CPython', 'default', 'af694c6a888c+') > > and the build info string becomes > > * '2.6.2:dd3ebf81af43, Jun 2 2009, 09:58:33' > * 'default:af694c6a888c+, Jun 2 2009, 01:24:14' > > This reflects that the default branch in hg is called 'default' > instead of Subversion's 'trunk', and reflects the proposed new tag > format. Looks good to me. cheers, Georg From status at bugs.python.org Fri Jul 3 18:07:18 2009 From: status at bugs.python.org (Python tracker) Date: Fri, 3 Jul 2009 18:07:18 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20090703160718.2935E78611@psf.upfronthosting.co.za> ACTIVITY SUMMARY (06/26/09 - 07/03/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2231 open (+26) / 16000 closed (+38) / 18231 total (+64) Open issues with patches: 890 Average duration of open issues: 661 days. Median duration of open issues: 413 days. Open Issues Breakdown open 2203 (+26) pending 28 ( +0) Issues Created Or Reopened (65) _______________________________ Remove checks for win NT 06/29/09 CLOSED http://bugs.python.org/issue4856 reopened amaury.forgeotdarc patch Rstrip Incorrectly Strips Some Strings 06/26/09 CLOSED http://bugs.python.org/issue6346 created mkubilus hpux11.00-parisc: dtoa.c: "Failed to find an exact-width 32-bit 06/26/09 CLOSED http://bugs.python.org/issue6347 created srid patch solaris/aix: Py_Initialize: can't initialize sys standard stream 06/26/09 http://bugs.python.org/issue6348 created srid email.Maildir cannot roundtrip messages. 06/27/09 CLOSED http://bugs.python.org/issue6349 created bremner Example at end of HTMLParser documentation uses old-style print 06/27/09 CLOSED http://bugs.python.org/issue6350 created MLModel Compiler warning in _cursesmodule.c 06/28/09 http://bugs.python.org/issue6351 created hagen Compiler warning in unicodeobject.c 06/28/09 http://bugs.python.org/issue6352 created hagen patch "L" integer suffix in Python 3.1 tutorial 06/28/09 CLOSED http://bugs.python.org/issue6353 created delroth patch Old floating point representation in 3.1 tutorial 06/28/09 CLOSED http://bugs.python.org/issue6354 created delroth bogus NULL check in PyCapsule 06/28/09 CLOSED http://bugs.python.org/issue6355 created bkramer patch Segfault of sys.setrecursionlimit if limit exceeds the value 180 06/28/09 CLOSED http://bugs.python.org/issue6356 created str1442 tempfile.NamedTemporaryFile does not accept the delete= paramete 06/28/09 CLOSED http://bugs.python.org/issue6357 created stutzbach os.popen exit code inconsistent 06/28/09 http://bugs.python.org/issue6358 created abbeyj pyexpat.c calls trace function incorrectly for exceptions 06/28/09 http://bugs.python.org/issue6359 created nedbat Simplify string decoding in xmlrpc.client. 06/29/09 http://bugs.python.org/issue6360 created alexandre.vassalotti patch I/O object wrappers shouldn't close their underlying file when d 06/29/09 http://bugs.python.org/issue6361 created alexandre.vassalotti multiprocessing: handling of errno after signals in sem_acquire( 06/29/09 http://bugs.python.org/issue6362 created ryles __future__ statements break doctest 06/29/09 CLOSED http://bugs.python.org/issue6363 created pooryorick freeze tool broken in Python 3.x 06/29/09 http://bugs.python.org/issue6364 created hagen distutils duplicates package directory for C extensions in 3.1 f 06/29/09 CLOSED http://bugs.python.org/issue6365 created scoder rare assertion failure in test_multiprocessing 06/29/09 http://bugs.python.org/issue6366 created pitrou Change the instruction pointer in python 06/29/09 CLOSED http://bugs.python.org/issue6367 created vinoth Fix unused variable warning introduced by commit r73603. 06/29/09 CLOSED http://bugs.python.org/issue6368 created vshenoy patch binhex buggy in py3k 06/29/09 CLOSED http://bugs.python.org/issue6369 created pitrou patch Bad performance of colllections.Counter at initialisation from a 06/29/09 CLOSED http://bugs.python.org/issue6370 created SilentGhost Minor internal link errors in Optparse doc 06/29/09 CLOSED http://bugs.python.org/issue6371 created pakal Make all switches keyword-only 06/29/09 CLOSED http://bugs.python.org/issue6372 created ptn SystemError in encoder 06/29/09 CLOSED http://bugs.python.org/issue6373 created glynnc patch Confused by subprocess API documentation 06/29/09 CLOSED http://bugs.python.org/issue6374 created pakal patch compile failure in Modules/python.c 06/29/09 http://bugs.python.org/issue6375 created adamg patch description of Decinal.logical_invert is incorrect 06/29/09 CLOSED http://bugs.python.org/issue6376 created mft distutils compiler switch ignored 06/29/09 http://bugs.python.org/issue6377 created atuining patch Patch to make 'idle.bat' run idle.pyw using appropriate Python i 06/29/09 http://bugs.python.org/issue6378 created srid patch Add first() function that would take first matching case from an 06/29/09 CLOSED http://bugs.python.org/issue6379 created gandalf Deadlock during the "import" in the fork()'ed child process if f 06/29/09 http://bugs.python.org/issue6380 created hdn test_urllib2_localnet sporadic failures closing socket 06/30/09 http://bugs.python.org/issue6381 created r.david.murray test_socketserver fails on trunk in test_ForkingTCPServer 06/30/09 http://bugs.python.org/issue6382 created r.david.murray error in unicodedata.numeric(u"\u2187") and 2188 06/30/09 CLOSED http://bugs.python.org/issue6383 created vernondcole patch, needs review Permalink to built-in exception class hierarchy documentation 06/30/09 CLOSED http://bugs.python.org/issue6384 created MLModel two lines of typographical inconsistency in doc of exception cla 06/30/09 CLOSED http://bugs.python.org/issue6385 created MLModel importing yields unexpected results when initial script is a sym 06/30/09 http://bugs.python.org/issue6386 created jbeulich floor division gives different result for int versus float. 06/30/09 CLOSED http://bugs.python.org/issue6387 created drj platform.python_implementation does not work on IronPython 06/30/09 CLOSED http://bugs.python.org/issue6388 created tbleier os.chmod() documentation refers to non-existent documentation in 06/30/09 CLOSED http://bugs.python.org/issue6389 created mhearne808 easy File reads past EOF in "w+b" mode 06/30/09 CLOSED http://bugs.python.org/issue6390 created bmearns Incorrect description of unittest.TestCase.run 06/30/09 CLOSED http://bugs.python.org/issue6391 created zhouer IDLE 3.1 will not open 07/01/09 CLOSED http://bugs.python.org/issue6392 created matthall OS X: python3 from python-3.1.dmg crashes at startup 07/01/09 http://bugs.python.org/issue6393 created marketdickinson getppid support in os module on Windows 07/01/09 http://bugs.python.org/issue6394 created janglin patch Add Pickle Support to the codecs Module 07/01/09 http://bugs.python.org/issue6395 created ThomasH No conversion specifier in the string, no __getitem__ method in 07/02/09 http://bugs.python.org/issue6396 reopened amaury.forgeotdarc Implementing Solaris "poll" in the "select" module 07/01/09 http://bugs.python.org/issue6397 created jcea README typo 07/01/09 http://bugs.python.org/issue6398 created nmichaels patch Error reporting by logging.config.fileConfig() 07/02/09 http://bugs.python.org/issue6399 created mike incorrect commands import 07/02/09 CLOSED http://bugs.python.org/issue6400 created alejolp Unexpected module garbage collection 07/02/09 CLOSED http://bugs.python.org/issue6401 created j_pok Crash after attempt to set the error indicator via PyErr_SetStri 07/02/09 CLOSED http://bugs.python.org/issue6402 created iankko distutils builds extension modules to root package directory 07/02/09 CLOSED http://bugs.python.org/issue6403 created ivank 3.1 NEWS.txt points to alpha 2 file 07/02/09 CLOSED http://bugs.python.org/issue6404 created srid Redundant redeclarations in descrobject.h 07/03/09 CLOSED http://bugs.python.org/issue6405 created flub patch NameError on 2to3 tool 07/03/09 CLOSED http://bugs.python.org/issue6406 created alejolp patch multiprocessing Pool should allow custom task queue 07/03/09 http://bugs.python.org/issue6407 created masher 2to3: Local package import 07/03/09 CLOSED http://bugs.python.org/issue6408 created alejolp 2to3 generates malformed diffs 07/03/09 http://bugs.python.org/issue6409 created exarkun patch Issues Now Closed (79) ______________________ Timeout in CGIXMLRPCRequestHandler under IIS 641 days http://bugs.python.org/issue1214 alexandre.vassalotti patch eliminate recursion in pickling 460 days http://bugs.python.org/issue2480 alexandre.vassalotti optparse quirks 236 days http://bugs.python.org/issue4278 amaury.forgeotdarc 3.0 file.read dreadfully slow 210 days http://bugs.python.org/issue4533 pitrou patch SystemError if invalid arguments passed to range() and step=-1 207 days http://bugs.python.org/issue4536 georg.brandl patch, needs review directory permission error with make install in 3.0 206 days http://bugs.python.org/issue4601 amaury.forgeotdarc patch print_function and unicode_literals don't work together 204 days http://bugs.python.org/issue4618 benjamin.peterson Issue with RotatingFileHandler logging handler on Windows 23 days http://bugs.python.org/issue4749 eantelman Remove checks for win NT 0 days http://bugs.python.org/issue4856 ocean-city patch Multiprocessing.Queue created by sub-process fails when used in 146 days http://bugs.python.org/issue5155 jnoller Formatting error in "findertools" header 135 days http://bugs.python.org/issue5254 ezio.melotti multiprocessing.process using os.close(sys.stdin.fileno) instead 131 days http://bugs.python.org/issue5313 jnoller patch multiprocessing hangs when Pool used within Process 130 days http://bugs.python.org/issue5331 jnoller Allow intermixing of keyword arguments and vargarg arguments 124 days http://bugs.python.org/issue5383 amaury.forgeotdarc Item 'Python x.x.x' in Add/Remove Programs list still lacks an i 115 days http://bugs.python.org/issue5390 loewis uuid module generates incorrect values for uuid3 (and possibly u 119 days http://bugs.python.org/issue5415 amaury.forgeotdarc multiprocessing/connection.py wrong pipe name under win32 104 days http://bugs.python.org/issue5503 jnoller optparse: clarify option concatenation in docs 99 days http://bugs.python.org/issue5555 ajs multiprocessing.connection.Client API documentation incorrect 78 days http://bugs.python.org/issue5740 jnoller needs review timeit documentation 57 days http://bugs.python.org/issue5896 ezio.melotti Risk of confusion in multiprocessing module - daemonic processes 58 days http://bugs.python.org/issue5906 jnoller re-usable generators / generator expressions should return itera 52 days http://bugs.python.org/issue5973 Jae Fix O(n**2) performance problem in socket._fileobject 33 days http://bugs.python.org/issue6117 krisvale patch, patch, easy, needs review Confusing DeprecationWarning 37 days http://bugs.python.org/issue6119 benjamin.peterson Parallel regression testing 30 days http://bugs.python.org/issue6152 pitrou patch Python 3.1 rc1 will not build on OS X 10.5.7 with macports libin 30 days http://bugs.python.org/issue6154 ygale patch [AIX] Patch to correct the AIX C/C++ linker argument used for 'r 27 days http://bugs.python.org/issue6164 tarek patch Mac 'make frameworkinstall' error: [...]/Python.framework/Versio 25 days http://bugs.python.org/issue6170 srid add disable_nagle_algorithm to SocketServer.TCPServer 21 days http://bugs.python.org/issue6192 krisvale patch, patch, easy TCP Sockets not closed by TCPServer and StreamRequestHandler 19 days http://bugs.python.org/issue6248 krisvale patch, patch, needs review distutils upload command doesn't work with http proxy 13 days http://bugs.python.org/issue6286 tarek Tix needs TCL package require statement 8 days http://bugs.python.org/issue6309 eantelman Add a "swap" method to list 8 days http://bugs.python.org/issue6326 pitrou patch, patch mmap.read() crashes when passed a negative argument 4 days http://bugs.python.org/issue6344 ocean-city patch Rstrip Incorrectly Strips Some Strings 0 days http://bugs.python.org/issue6346 ezio.melotti hpux11.00-parisc: dtoa.c: "Failed to find an exact-width 32-bit 4 days http://bugs.python.org/issue6347 marketdickinson patch email.Maildir cannot roundtrip messages. 1 days http://bugs.python.org/issue6349 r.david.murray Example at end of HTMLParser documentation uses old-style print 0 days http://bugs.python.org/issue6350 ezio.melotti "L" integer suffix in Python 3.1 tutorial 3 days http://bugs.python.org/issue6353 ezio.melotti patch Old floating point representation in 3.1 tutorial 1 days http://bugs.python.org/issue6354 marketdickinson bogus NULL check in PyCapsule 0 days http://bugs.python.org/issue6355 benjamin.peterson patch Segfault of sys.setrecursionlimit if limit exceeds the value 180 0 days http://bugs.python.org/issue6356 loewis tempfile.NamedTemporaryFile does not accept the delete= paramete 1 days http://bugs.python.org/issue6357 stutzbach __future__ statements break doctest 4 days http://bugs.python.org/issue6363 benjamin.peterson distutils duplicates package directory for C extensions in 3.1 f 0 days http://bugs.python.org/issue6365 tarek Change the instruction pointer in python 1 days http://bugs.python.org/issue6367 vinoth Fix unused variable warning introduced by commit r73603. 0 days http://bugs.python.org/issue6368 ocean-city patch binhex buggy in py3k 2 days http://bugs.python.org/issue6369 pitrou patch Bad performance of colllections.Counter at initialisation from a 0 days http://bugs.python.org/issue6370 rhettinger Minor internal link errors in Optparse doc 1 days http://bugs.python.org/issue6371 georg.brandl Make all switches keyword-only 0 days http://bugs.python.org/issue6372 ptn SystemError in encoder 0 days http://bugs.python.org/issue6373 amaury.forgeotdarc patch Confused by subprocess API documentation 1 days http://bugs.python.org/issue6374 georg.brandl patch description of Decinal.logical_invert is incorrect 1 days http://bugs.python.org/issue6376 georg.brandl Add first() function that would take first matching case from an 0 days http://bugs.python.org/issue6379 rhettinger error in unicodedata.numeric(u"\u2187") and 2188 1 days http://bugs.python.org/issue6383 amaury.forgeotdarc patch, needs review Permalink to built-in exception class hierarchy documentation 1 days http://bugs.python.org/issue6384 georg.brandl two lines of typographical inconsistency in doc of exception cla 0 days http://bugs.python.org/issue6385 georg.brandl floor division gives different result for int versus float. 0 days http://bugs.python.org/issue6387 r.david.murray platform.python_implementation does not work on IronPython 0 days http://bugs.python.org/issue6388 benjamin.peterson os.chmod() documentation refers to non-existent documentation in 2 days http://bugs.python.org/issue6389 r.david.murray easy File reads past EOF in "w+b" mode 0 days http://bugs.python.org/issue6390 amaury.forgeotdarc Incorrect description of unittest.TestCase.run 0 days http://bugs.python.org/issue6391 ezio.melotti IDLE 3.1 will not open 0 days http://bugs.python.org/issue6392 amaury.forgeotdarc incorrect commands import 0 days http://bugs.python.org/issue6400 alejolp Unexpected module garbage collection 0 days http://bugs.python.org/issue6401 pitrou Crash after attempt to set the error indicator via PyErr_SetStri 0 days http://bugs.python.org/issue6402 benjamin.peterson distutils builds extension modules to root package directory 1 days http://bugs.python.org/issue6403 tarek 3.1 NEWS.txt points to alpha 2 file 0 days http://bugs.python.org/issue6404 benjamin.peterson Redundant redeclarations in descrobject.h 0 days http://bugs.python.org/issue6405 benjamin.peterson patch NameError on 2to3 tool 0 days http://bugs.python.org/issue6406 alejolp patch 2to3: Local package import 0 days http://bugs.python.org/issue6408 alejolp Tk.quit leads to crash in python.exe 2169 days http://bugs.python.org/issue775544 gpolo fcntl.ioctl have a bit problem. 1413 days http://bugs.python.org/issue1262856 amaury.forgeotdarc traceback module can return undecodable byte strings 1392 days http://bugs.python.org/issue1284496 amaury.forgeotdarc Trailing slash redirection for SimpleHTTPServer 1364 days http://bugs.python.org/issue1314572 josiahcarlson patch, easy Polymorphic getters / setters 1285 days http://bugs.python.org/issue1388872 amaury.forgeotdarc Stepping into a generator throw does not work 962 days http://bugs.python.org/issue1592241 amaury.forgeotdarc pickle - cannot unpickle circular deps with custom __hash__ 704 days http://bugs.python.org/issue1761028 alexandre.vassalotti Top Issues Most Discussed (10) ______________________________ 7 getppid support in os module on Windows 2 days open http://bugs.python.org/issue6394 7 error in unicodedata.numeric(u"\u2187") and 2188 1 days closed http://bugs.python.org/issue6383 7 Green-box doc glitch: winhelp version only 126 days open http://bugs.python.org/issue5388 7 Remove checks for win NT 0 days closed http://bugs.python.org/issue4856 6 incorrect commands import 0 days closed http://bugs.python.org/issue6400 6 tempfile.NamedTemporaryFile does not accept the delete= paramet 1 days closed http://bugs.python.org/issue6357 6 Cumulative patcc:h to http and xmlrpc 22 days open http://bugs.python.org/issue6267 5 floor division gives different result for int versus float. 0 days closed http://bugs.python.org/issue6387 5 SystemError in encoder 0 days closed http://bugs.python.org/issue6373 5 Make all switches keyword-only 0 days closed http://bugs.python.org/issue6372 From p.f.moore at gmail.com Fri Jul 3 18:45:07 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 3 Jul 2009 17:45:07 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610907030744p6cc1fbbavd5fcc723c5a8917b@mail.gmail.com> References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com> <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com> <4A4E1572.3000801@gmail.com> <94bdd2610907030744p6cc1fbbavd5fcc723c5a8917b@mail.gmail.com> Message-ID: <79990c6b0907030945u1ed6e358o7847a08e39fa0c58@mail.gmail.com> 2009/7/3 Tarek Ziad? : > On Fri, Jul 3, 2009 at 4:28 PM, Nick Coghlan wrote: >> I will note (and I believe this is also the main point that Lukasz was >> making) that having the distribution metadata outside the distribution >> as currently proposed in PEP 376 is going to make any eventual PEP 302 >> integration much harder - 302 importers only provide a mechanism for >> accessing files inside the distribution, not "adjacent" to them, so the >> mechanism in the PEP doesn't generalise properly. >> >> I suspect this limitation of the PEP 302 APIs is the origin of the >> setuptools format that embeds the metadata inside the distribution - it >> lets you get at the metadata without having to assume that it exists >> directly on the filesystem anywhere. > > Maybe we could rework the pkgutil classes for PEP 376 so they look > like an implementation > of the PEP 302 protocol for directories and zip files, with an extra > "get_metadata()" API > and state that it could be an extension for PEP 302 later. That sounds like an excellent approach And given that PEP 302 is nothing more than an API spec, it's nice and easy to extend PEP 302 to say that importers can/should implement this (after all, the only 2 "real" cases in the wild are done, via PEP 376). Warning: I've not thought all this through fully, so I may be missing some subtleties! The ultimate spec needs to be clearly worded, and as a consequence of this discussion it occurs to me that the current PEP 376 is very far from clear in defining exactly what the naming, layout and location of egg-info directories is - it makes a lot of assumptions that may break for more general path importers I'll do some thinking, and maybe come up with some examples of PEP 302 edge cases, so that we can at least be sure that they've not been ignored (rejected as to stupid to care about, on the other hand, I'm happy with :-)) Paul. From pje at telecommunity.com Fri Jul 3 19:02:40 2009 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 03 Jul 2009 13:02:40 -0400 Subject: [Python-Dev] PEP 376 In-Reply-To: <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.co m> References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com> <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com> Message-ID: <20090703165942.38CFA3A4109@sparrow.telecommunity.com> At 02:54 PM 7/3/2009 +0100, Paul Moore wrote: >Eggs are fundamentally a PEP 302 zip file format. There are some >extra bits of metadata for setuptools/easy_install in there (as I >understand things) but essentially they are zip files. When you say >"decoupling the egg format", I assume you mean "decoupling the egg >metadata" - which is fine, but to properly decouple, you need API >level access to the metadata. PEP 376 offers read-only access, but >as you rightly point out, it is only for filesystem data (and some >form of zip file, which appears to be limited in some way, as it >isn't PEP 302 based, and the actual format isn't defined anywhere). >The basic point here is that PEP 376 needs to define precisely how >pkgutil.get_distributions() scans sys.path looking for ".egg-info >directories". What does it do for sys.path entries that don't >correspond to filesystem directories? (Note - these may or may not >be zip files. Even if they are zip files, an earlier entry on >sys.path_hooks could have taken precedence. At the very least, you >should only process path entries as zip files if their importer - in >sys.path_importer_cache or via an explicit path hook scan - is a >zipimporter object.). To be honest, this is a major can of worms. >But if PEP 376 is not going to support PEP 302, then it must state >that fact explicitly, to avoid giving people false expectations - >particularly with Brett's importlib in Python 3.1, which will make >it far easier for people to experiment with new packaging formats >such as the ones Lukasz mentions above. And it MUST fail gracefully >in the face of unsupported importer types. Well, we could always resurrect PEP 365, since pkg_resources already has documented extensible support for arbitrary importers. That solves backward *and* forward compatibility. Then PEP 376's uninstall facilities could be implemented using pkg_resources' existing metadata query features. The primary downside to that, of course, is that it brings in the matter of version specifications and dependencies... which appear to be a contentious topic. (Note that Tarek is proposing to drop the PEP 386 proposal to standardize a much more restrictive scheme than seutptools' version parser, precisely because of the controversy.) From p.f.moore at gmail.com Fri Jul 3 19:01:05 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 3 Jul 2009 18:01:05 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <4A4E1572.3000801@gmail.com> References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com> <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com> <4A4E1572.3000801@gmail.com> Message-ID: <79990c6b0907031001h39caea63j98f500afdcc05fbf@mail.gmail.com> 2009/7/3 Nick Coghlan : > I agree that even if the reference implementation of PEP 376 only > handles normal directories and zipfiles, the PEP itself needs to explain > how the developer of a custom PEP 302 importer or loader can hook into > the mechanism in order to provide metadata that distutils can > understand. Or, as you suggest, state explicitly that the proposal at > this stage is specifically limited to filesystem and zipfile packages > and defer extension to arbitrary PEP 302 importers and loaders to a > later PEP. > > To be honest, I'd personally be OK with the latter strategy - while > other PEP 302 loaders and importers do exist (as Lukasz'z post shows), > it would be unfortunate to unduly hold up improved installation metadata > for the vast majority of typical use cases while we try to figure out > ways to support the more esoteric use cases. Supporting both would > obviously be better, but given the choice between the status quo and > partial support, I believe this is a case where the partial support > would still be worthwhile. You put it more clearly than I did. That's basically what I think, with the one proviso that we should make sure that PEP 376 doesn't specify something that out and out breaks the more esoteric PEP 302 cases. When Just and I were developing PEP 302, we found that the best way to do that was to leave anything that didn't need to be specified, as unspecified (hence the fact that there's so little defined in PEP 302!). It's easier to add things later than to remove or change them. That's why I was recommending to Tarek that he take out of the PEP any details about classes or APIs that couldn't be directly obtained from the core API. The same applies here (just talking in terms of a duck-typed notional Distribution class, allows people to implement their own URLDistribution or SqliteDistribution, or whatever, and not have to emulate any more of a filesystem than necessary). > I will note (and I believe this is also the main point that Lukasz was > making) that having the distribution metadata outside the distribution > as currently proposed in PEP 376 is going to make any eventual PEP 302 > integration much harder - 302 importers only provide a mechanism for > accessing files inside the distribution, not "adjacent" to them, so the > mechanism in the PEP doesn't generalise properly. > > I suspect this limitation of the PEP 302 APIs is the origin of the > setuptools format that embeds the metadata inside the distribution - it > lets you get at the metadata without having to assume that it exists > directly on the filesystem anywhere. Again, agreed. But remember that PEP 302 is driven by looking up a module or package name. PEP 376 is looking up a *distribution* name. The docutils example in the PEP shows this: - docutils/ - roman.py - docutils-0.5-py2.6.egg-info/ There are 3 things here: a package (docutils), a module (roman) and a distribution (also named docutils, but could be called George for all it matters). So none of the PEP 302 lookup mechanisms (which work on package/module name) apply. The need for a separate concept (a "distribution") is unfortunate, as it adds complexity, but there are enough real life cases that make it clear that it's necessary. Hmm, I suspect that the implication here is that PEP 302 could do with an overhaul, to extend it to encompass the concept of a distribution. I'd be willing to have a look at that. Paul. From p.f.moore at gmail.com Fri Jul 3 19:06:55 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 3 Jul 2009 18:06:55 +0100 Subject: [Python-Dev] PEP 376 In-Reply-To: <20090703165942.38CFA3A4109@sparrow.telecommunity.com> References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com> <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com> <20090703165942.38CFA3A4109@sparrow.telecommunity.com> Message-ID: <79990c6b0907031006rd0c6265hf27a3f1488ba3af2@mail.gmail.com> 2009/7/3 P.J. Eby : > Well, we could always resurrect PEP 365, since pkg_resources already has > documented extensible support for arbitrary importers. ?That solves backward > *and* forward compatibility. ?Then PEP 376's uninstall facilities could be > implemented using pkg_resources' existing metadata query features. It would be easier to comment on that suggestion if the PEP documented the reasons why it was rejected. Sadly, that never got added to the PEP at the time. Do you recall the reasons, and if so would you be willing to update the PEP with them? Paul. From pje at telecommunity.com Fri Jul 3 19:10:48 2009 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 03 Jul 2009 13:10:48 -0400 Subject: [Python-Dev] PEP 376 In-Reply-To: <4A4E1572.3000801@gmail.com> References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com> <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com> <4A4E1572.3000801@gmail.com> Message-ID: <20090703170748.818B73A4109@sparrow.telecommunity.com> At 12:28 AM 7/4/2009 +1000, Nick Coghlan wrote: >I suspect this limitation of the PEP 302 APIs is the origin of the >setuptools format that embeds the metadata inside the distribution - it >lets you get at the metadata without having to assume that it exists >directly on the filesystem anywhere. I think you have this backwards; it's setuptools that doesn't care where (or whether) the metadata exists on the file system; it delegates metadata operations to a "metadata provider" that's usually an adapter over a PEP 302 "loader". See http://peak.telecommunity.com/DevCenter/PkgResources#supporting-custom-importers for the API details of how to register support for arbitrary PEP 302 importers and loaders. (Which presumably, Lukasz is using. I didn't know that anybody was actually using it, but it's nice to know that the documentation is apparently sufficient for *some* people. ;-) ) From brett at python.org Fri Jul 3 20:04:16 2009 From: brett at python.org (Brett Cannon) Date: Fri, 3 Jul 2009 11:04:16 -0700 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: On Thu, Jul 2, 2009 at 13:42, Dirkjan Ochtman wrote: > In response to some rumblings on python-committers and just to request > more feedback, a progress report. I know it's long, I've tried to put > to keep it concise and chunked, though. > > - First of all, I've got the basic conversion down, I've done it a few > times now, with progressively better results. You can view some > results at http://hg.python.org/, which has a preliminary cpython > repository. *** The changeset hashes for that repo will change, so you > won't be able to commit or pull from it in the future.*** > > - Second of all, some planning. I've thought about it a bit, and I > think we should aim for going live with hg on August 1. Given that I'm > on vacation from 8-18 July (and I'm not sure whether I'll be able to > actually work on it during that time, though I imagine I'll be able to > spend some time on it at least), that's quite ambitious, so I'm going > to say it's okay if it slips by a few days. Putting a deadline out > there is a good thing, anyway. > Fine by me as long as people realize that if anything is questionable then the switch will not happen. Getting this right takes precedence over any deadline. And obviously we will need to do at least one live conversion on python.org hardware to make sure everything will work smoothly. > > - Third of all, to make this possible, it would be helpful if I got > more feedback on the PEP. Last time I raised it, there was virtually > nothing. This time, I'll include it inline so there's hopefully less > of a barrier to reviewing it. > > - Fourth, Mercurial 1.3 was just released! Bet you didn't see that > coming. It's looking like a pretty good release, with an experimental > version of the much-coveted subrepository support (like > svn:externals). This also means that the latest version of > hgsubversion, the tool I used for the conversion, will be more > accessible for converting other projects. You know you want to! > And will make the idea of splitting out the standard library and tests a reasonable thing to do. > > - Fifth, here's a list of things, off the top of my head, that still need > doing: > > * Get agreement on branch strategy and branch processing (list of > branches + proposed handling at > http://hg.python.org/pymigr/file/tip/all-branches.txt) <--- PLEASE > REVIEW > * Get agreement on tag processing (first come up with a list) > * Set up hg-ssh infra (should be easy) > * Set up hooks (should be mostly straightforward) > * Set up roundup integration (should be made easier by quick revision > map hgweb extension) > * Write docs > > - Sixth (this is the good part), less obvious things that have been > done or don't need doing: > > * .hgignore generation (I've been convinced it's too hard, the current > version will do) Yeah, we can do this manually. > > * revlog reordering (it's painless and a big win) > > I'll get through all of these myself, but obviously any help would be > welcome. For any hg users, writing docs should be an easy start. For > others, please review the PEP (below), the branch map in > http://hg.python.org/pymigr/file/tip/all-branches.txt and the author > map at http://hg.python.org/pymigr/file/tip/author-map (not that much > has changed since the start, so if you've looked at it already, feel > free to skip this part). Right now I'm a little stuck on branch > processing, because it's a long running script that needs a bunch of > debugging, but I'll get going on that again. > > I think that's all I can think of for now, I'll update the PEP with > new bits soon. Here it is, ready for your review: > > ============================================================== > > Motivation > > After having decided to switch to the Mercurial DVCS, the actual > migration still has to be performed. In the case of an important piece > of infrastructure like the version control system for a large, > distributed project like Python, this is a significant effort. This > PEP is an attempt to describe the steps that must be taken for further > discussion. It's somewhat similar to PEP 347 [1], which discussed the > migration to SVN. > > To make the most of hg, I (Dirkjan) would like to make a high-fidelity > conversion, such that (a) as much of the svn metadata as possible is > retained, and (b) all metadata is converted to formats that are common > in Mercurial. This way, tools written for Mercurial can be optimally > used. In order to do this, I want to use the hgsubversion [2] software > to do an initial conversion. This hg extension is focused on providing > high-quality conversion from Subversion to Mercurial for use in > two-way correspondence, meaning it doesn't throw away as much > available metadata as other solutions. > > Such a conversion also seems like a good time to reconsider the > contents of the repository and determine if some things are still > valuable. In this spirit, the following sections also propose > discarding some of the older metadata. > Timeline > > TBD; needs fully working hgsubversion and consensus on this document. > Transition plan > Branch strategy > > Mercurial has two basic ways of using branches: cloned branches, where > each branch is kept in a separate repository, and named branches, > where each revision keeps metadata to note on which branch it belongs. > The former makes it easier to distinguish branches, at the expense of > requiring more disk space on the client. The latter makes it a little > easier to switch between branches, but often has somewhat unintuitive > results for people (though this has been getting better in recent > versions of Mercurial). > > I'm still a bit on the fence about whether Python should adopt cloned > branches and named branches. Since it usually makes more sense to tag > releases on the maintenance branch, for example, mainline history > would not contain release tags if we used cloned branches. Also, > Mercurial 1.2 and 1.3 have the necessary tools to make named branches > less painful (because they can be properly closed and closed heads are > no longer considered in relevant cases). > > A disadvantage might be that the used clones will be a good bit larger > (since they essentially contain all other branches as well). This can > me mitigated by keeping non-release (feature) branches in separate > clones. Also note that it's still possible to clone a single named > branch from a combined clone, by specifying the branch as in hg clone > http://hg.python.org/main/#2.6-maint. Keeping the py3k history in a > separate clone problably also makes sense. > While I really like the idea of using named branches for each release so that there is a single py3k branch that contains all relevant history for every release, I think we should start simple and go with cloned branches. That way the workflow does not radically shift from what we do now for svn to start. Once the conversion is done and people are comfortable with hg we can then discuss moving towards a named branch approach. > > XXX To do: size comparison for selected separation scenarios. > Converting branches > > There are quite a lot of branches in SVN's branches directory. I > propose to clean this up a bit, by employing the following the > strategy: > > * Keep all release (maintenance) branches > * Discard branches that haven't been touched in 18 months, unless > somone indicates there's still interest in such a branch > * Keep branches that have been touched in the last 18 months, > unless someone indicates the branch can be deprecated > Sounds reasonable to me. We can just make a list and send it to python-committers to make final decisions of what should stick around. > > Converting tags > > The SVN tags directory contains a lot of old stuff. Some of these are > not, in fact, full tags, but contain only a smaller subset of the > repository. I think we should keep all release tags, and consider > other tags for inclusion based on requests from the developer > community. I'd like to consider unifying the release tag naming scheme > to make some things more consistent, if people feel that won't create > too many problems. For example, Mercurial itself just uses '1.2.1' as > a tag, where CPython would currently use r121. I don't use tags so I don't really care, but in the name of easy transition I say we don't change the naming scheme (although I have no issue dropping obscure tags). > > Author map > > In order to provide user names the way they are common in hg (in the > 'First Last ' format), we need an author map to map > cvs and svn user names to real names and their email addresses. I have > a complete version of such a map in my migration tools repository [3]. > The email addresses in it might be out of date; that's bound to > happen, although it would be nice to try and have as many people as > possible review it for addresses that are out of date. The current > version also still seems to contain some encoding problems. Something else that can go out to python-committers before the switch. > > Generating .hgignore > > The .hgignore file can be used in Mercurial repositories to help > ignore files that are not eligible for version control. It does this > by employing several possible forms of pattern matching. The current > Python repository already includes a rudimentary .hgignore file to > help with using the hg mirrors. > > It might be useful to have the .hgignore be generated automatically > from svn:ignore properties. This would make sure all historic > revisions also have useful ignore information (though one could argue > ignoring isn't really relevant to just checking out an old revision). Don't bother with anything automatic. We can change the .hgignore file by hand. We all know glob and regex syntax. =) > > Revlog reordering > > As an optional optimization technique, we should consider trying a > reordering pass on the revlogs (internal Mercurial files) resulting > from the conversion. In some cases this results in dramatic decreases > in on-disk repository size. Fine by me. > > Other repositories > > Richard Tew has indicated that he'd like the Stackless repository to > also be converted. What other projects in the svn.python.org > repository should be converted? Do we want to convert the peps > repository? distutils? others? I don't think there is a single project we host -- all two of them -- that have not said they want to convert. So I say convert everything and let's turn off the svn server by the end of the year. > > Infrastructure > hg-ssh > > Developers should access the repositories through ssh, similar to the > current setup. Public keys can be used to grant people access to a > shared hg@ account. A hgwebdir instance should also be set up for easy > browsing and read-only access. If we're using ssh, developers should > trivially be able to start new clones (for longer-term features that > profit from a separate branch). > Hooks > > A number of hooks is currently in use. The hg equivalents for these > should be developed and deployed. The following hooks are being used: > > * check whitespace: a hook to reject commits in case the > whitespace doesn't match the rules for the Python codebase. Should be > straightforward to re-implement from the current version. We can also > offer a whitespace hook for use with client-side repositories that > people can use; it could either warn about whitespace issues and/or > truncate trailing whitespace from changed lines. Open issue: do we > check only the tip after each push, or do we check every commit in a > changegroup? > * commit mails: we can leverage the notify extension for this > * buildbots: both the regular and the community build masters must > be notified. Fortunately buildbot includes support for hg. I've also > implemented this for Mercurial itself, so I don't expect problems > here. > * check contributors: in the current setup, all changesets bear > the username of committers, who must have signed the contributor > agreement. In a DVCS, the committers are not necessarily the same > people who push, and so we can't check if the committer is a > contributor. We could use a hook to check if the committer is a > contributor if we keep a list of registered contributors. Can we check these scripts into the repository itself? That way there is a chance of reuse as hg commands, e.g. ``hg pydev-ci`` as a replacement for ``make patchcheck``. > > > hgwebdir > > A more or less stock hgwebdir installation should be set up. We might > want to come up with a style to match the Python website. It may also > be useful to build a quick extension to augment the URL rev parser so > that it can also take r[0-9]+ args and come up with the matching hg > revision. > After migration > Where to get code > > It needs to be decided where the hg repositories will live. I'd like > to propose to keep the hgwebdir instance at hg.python.org. This is an > accepted standard for many organizations, and an easy parallel to > svn.python.org. The 2.7 (trunk) repo might live at > http://hg.python.org/main/, for example, with py3k at > http://hg.python.org/py3k/. For write access, developers will have to > use ssh, which could be ssh://hg at hg.python.org/main/. A demo > installation will be set up with a preliminary conversion so people > can experiment and review; it can live at > http://hg.python.org/example/. > > code.python.org was also proposed as the hostname. Personally, I think > that using the VCS name in the hostname is good because it prevents > confusion: it should be clear that you can't use svn or bzr for > hg.python.org. > How about hg.python.org for the official branches and we keep code.python.org for personal branches of the developers like we have done with the bzr experiments? > > hgwebdir can already provide tarballs for every changeset. I think > this obviates the need for daily snapshots; we can just point users to > tip.tar.gz instead, meaning they will get the latest. If desired, we > could even use buildbot results to point to the last good changeset. I like the stable buildbot tarball idea. > > Python-specific documentation > > hg comes with good built-in documentation (available through hg help) > and a wiki [4] that's full of useful information and recipes. In > addition to that, the parts of the developer FAQ [5] concerning > version control will gain a section on using hg for Python > development. Some of the text will be dependent on the outcome of > debate about this PEP (for example, the branching strategy). > > Think first, commit later? > > In recent history, old versions of Python have been maintained by a > select group of people backporting patches from trunk to release > branches. While this may not scale so well as the development pace > grows, it also runs into some problems with the current crop of > distributed versioning tools. These tools (I believe similar problems > would exist for either git, bzr, or hg, though some may cope better > than others) are based on the idea of a Directed Acyclic Graph (or > DAG), meaning they keep track of relations of changesets. > > Mercurial itself has a stable branch which is a ''strict'' subset of > the unstable branch. This means that generally all fixes for the > stable branch get committed against the tip of the stable branch, then > they get merged into the unstable branch (which already contains the > parent of the new cset). This provides a largely frictionless > environment for moving changes from stable to unstable branches. > Mistakes, where a change that should go on stable goes on unstable > first, do happen, but they're usually easy to fix. That can be done by > copying the change over to the stable branch, then trivial-merging > with unstable -- meaning the merge in fact ignores the parent from the > stable branch). > > This strategy means a little more work for regular committers, because > they have to think about whether their change should go on stable or > unstable; they may even have to ask someone else (the RM) before > committing. But it also relieves a dedicated group of committers of > regular backporting duty, in addition to making it easier to work with > the tool. > > Now would be a good time to consider changing strategies in this > regard, although it would be relatively easy to switch to such a model > later on. As I have said, we should change our workflow habits after the switch and people are comfortable with hg. > > The future of Subversion > > What happens to the Subversion repositories after the migration? Since > the svn server contains a bunch of repositories, not just the CPython > one, it will probably live on for a bit as not every project may want > to migrate or it takes longer for other projects to migrate. To > prevent people from staying behind, we may want to remove migrated > projects from the repository. > Build identification > > Python currently provides the sys.subversion tuple to allow Python > code to find out exactly what version of Python it's running against. > The current version looks something like this: > > * ('CPython', 'tags/r262', '71600') > * ('CPython', 'trunk', '73128M') > > Another value is returned from Py_GetBuildInfo() in the C API, and > available to Python code as part of sys.version: > > * 'r262:71600, Jun 2 2009, 09:58:33' > * 'trunk:73128M, Jun 2 2009, 01:24:14' > > I propose that the revision identifier will be the short version of > hg's revision hash, for example 'dd3ebf81af43', augmented with '+' > (instead of 'M') if the working directory from which it was built was > modified. This mirrors the output of the hg id command, which is > intended for this kind of usage. > > For the tag/branch identifier, I propose that hg will check for tags > on the currently checked out revision, use the tag if there is one > ('tip' doesn't count), and uses the branch name otherwise. > sys.subversion becomes > > * ('CPython', '2.6.2', 'dd3ebf81af43') > * ('CPython', 'default', 'af694c6a888c+') > > and the build info string becomes > > * '2.6.2:dd3ebf81af43, Jun 2 2009, 09:58:33' > * 'default:af694c6a888c+, Jun 2 2009, 01:24:14' > > This reflects that the default branch in hg is called 'default' > instead of Subversion's 'trunk', and reflects the proposed new tag > format. Should we consider adding a sys.revision attribute and begin the deprecation of sys.subversion? > > References > [1] http://www.python.org/dev/peps/pep-0347/ > [2] http://bitbucket.org/durin42/hgsubversion/ > [3] http://hg.xavamedia.nl/cpython/pymigr/ > [4] http://www.selenic.com/mercurial/wiki/ > [5] http://www.python.org/dev/faq/#version-control > > ===================================================== > > Cheers, > > Dirkjan > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Jul 3 20:22:57 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 03 Jul 2009 14:22:57 -0400 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: Dirkjan Ochtman wrote: > It needs to be decided where the hg repositories will live. I'd like > to propose to keep the hgwebdir instance at hg.python.org. This is an > accepted standard for many organizations, and an easy parallel to > svn.python.org. The 2.7 (trunk) repo might live at > http://hg.python.org/main/, for example, with py3k at > http://hg.python.org/py3k/. I would very much like the 'k' dropped from the py3 name. It was a funny joke when py3 was vaporware, now it is excess baggage which only puzzles non-insiders and newcomers. I think the two repos should be either symmetrically named hg.python.org/py2 hg.python.org/py3 If one must be designated 'main', it should be py3. Continuing to call py2 'main' will continue to discourage use of py3. Terry Jan Reedy From python at mrabarnett.plus.com Fri Jul 3 20:41:00 2009 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 03 Jul 2009 19:41:00 +0100 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: <4A4E50BC.8060201@mrabarnett.plus.com> Terry Reedy wrote: > Dirkjan Ochtman wrote: > >> It needs to be decided where the hg repositories will live. I'd like >> to propose to keep the hgwebdir instance at hg.python.org. This is an >> accepted standard for many organizations, and an easy parallel to >> svn.python.org. The 2.7 (trunk) repo might live at >> http://hg.python.org/main/, for example, with py3k at >> http://hg.python.org/py3k/. > > I would very much like the 'k' dropped from the py3 name. It was a funny > joke when py3 was vaporware, now it is excess baggage which only puzzles > non-insiders and newcomers. > > I think the two repos should be either symmetrically named > > hg.python.org/py2 > hg.python.org/py3 > > If one must be designated 'main', it should be py3. > > Continuing to call py2 'main' will continue to discourage use of py3. > We could regard py3k as the phase from the original concept of Python 3 to its 'prototype', Python 3.0. Python 3.1 would be the first 'real/usable' version. From aahz at pythoncraft.com Fri Jul 3 21:06:53 2009 From: aahz at pythoncraft.com (Aahz) Date: Fri, 3 Jul 2009 12:06:53 -0700 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: <20090703190653.GA7213@panix.com> On Fri, Jul 03, 2009, Brett Cannon wrote: > > Should we consider adding a sys.revision attribute and begin the deprecation > of sys.subversion? +1 -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "as long as we like the same operating system, things are cool." --piranha From p.f.moore at gmail.com Fri Jul 3 21:29:57 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 3 Jul 2009 20:29:57 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata Message-ID: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> This post basically follows on from the previous thread about PEP 376 (Changing the .egg-info structure) where it was pointed out that the PEP doesn't cover PEP 302 import hooks (except in the explicit special case of zip files). This is likely to be a fairly long posting, but I'd like to try and cover the subtleties a little, so that people can comment without having to refer back to too many documents. Comments are most definitely welcome! While I know PEP 302 reasonably well, and the zip importer implementation, I'm not an expert on the egg-info structure, or the practical drivers behind it, so if I miss key issues because of a too-theoretical approach, I'd be grateful for corrections. The basic structure of PEP 302 imports is as follows: - scan sys.meta_path checking each finder in turn - if nothing found, scan sys.path and for each entry pass it to each element of sys.path_hooks, to get a finder to check - do the default filesystem processing on path items for which no path hook applies The first finder (if any) to recognise the module name wins (and returns a loader responsible for creating the actual module object). So it's the finders that are responsible for scanning the "filesystem" (more accurately, the namespace that finder manages). PEP 302 is entirely couched in terms of modules, packages and module names. There is no concept of a "distribution" in PEP 376 terms. This is entirely understandable, as imports don't know about distributions (see the docutils example in PEP 376 - the docutils *distribution* contains the *module* roman, but there's nothing about importing roman that requires you to know that it came from the docutils distribution). So we need to introduce a new concept, that of a distribution, into PEP 302 loaders. And the place it should be located is in the finder (which handles the "filesystem" aspects of the protocol. So we have the methods: Finder.list_distributions() Returns a list of all distribution names in the "filesystem" managed by the finder (usually one zip file, path directory, salite database, or whatever) Finder.get_metadata(distribution_name) Returns a metadata object for the given distribution (this is the PEP 376 Distribution object). One note here - get_egginfo_file should be specified as returning a *file-like object* rather than a file instance. Another note - get_egginfo_file and get_egginfo_files could just as easily be named get_metadata_file and get_metadata_files - just as meaningful, and less tied to the egg format (as well as avoiding the "egg" name, which I personally dislike :-)) Finder.uninstall(distribution_name, filter=callable, installer=name) Uninstall the given distribution. It's likely that many finders will be read-only. In that case, this function should return None. Otherwise, return a list of the "files" removed. (This may need some clarification, as many finders won't have a concept of a "file name"). I don't think anything else is needed to support PEP 376. The prototype implementation of PEP 376 could be extended to work with finders (doing the relevant sys.meta_path and sys.path_hooks searches). For the final implementation, the special casing of zip files would be replaced by an implementation of the extended finder protocol in the zipimporter object. There's almost certainly aspects missing from the above proposal. But it does have some definite advantages, above and beyond simply allowing PEP 302 importers to participate in the PEP 376 protocol. Setuptools-style eggs could be handled simply by creating a specialised finder (IIUC, they currently just use the standard zipimporter - the specialised version could subclass this) to override the metadata methods so as to cater for their specialised egg-info format. Other formats could be handled similarly. Does this sound sensible? Tarek, would you be OK with me attempting to modify your prototype to support this protocol? Are there any tests for PEP 376, so that I can confirm I haven't completely broken something? If I can, I'll knock up some simple prototype importers for non-standard examples, and see how they work with all this. Paul. From ziade.tarek at gmail.com Fri Jul 3 21:46:24 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 3 Jul 2009 21:46:24 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> Message-ID: <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> 2009/7/3 Paul Moore : > Does this sound sensible? Tarek, would you be OK with me attempting to > modify your prototype to support this protocol? Are there any tests > for PEP 376, so that I can confirm I haven't completely broken > something? If I can, I'll knock up some simple prototype importers for > non-standard examples, and see how they work with all this. Yes that's exactly what I was thinking about after the discussion we had in the other thread. This change would allow much more flexibility. Pleas go ahead, it's located here : http://bitbucket.org/tarek/pep376/ You can give me a bitbucket account so I can give you write access to the repo, There are tests as long as you install Nose. Tarek From ziade.tarek at gmail.com Fri Jul 3 22:14:27 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 3 Jul 2009 22:14:27 +0200 Subject: [Python-Dev] PEP 376 In-Reply-To: <20090703165942.38CFA3A4109@sparrow.telecommunity.com> References: <94bdd2610906220623k301e7924i5abc0a97d54741ba@mail.gmail.com> <79990c6b0907030654w5698e609qaf10a13eb237a430@mail.gmail.com> <20090703165942.38CFA3A4109@sparrow.telecommunity.com> Message-ID: <94bdd2610907031314p64df2581wae746bc595ef86f3@mail.gmail.com> On Fri, Jul 3, 2009 at 7:02 PM, P.J. Eby wrote: > At 02:54 PM 7/3/2009 +0100, Paul Moore wrote: >> >> Eggs are fundamentally a PEP 302 zip file format. There are some extra >> bits of metadata for setuptools/easy_install in there (as I understand >> things) but essentially they are zip files. When you say "decoupling the egg >> format", I assume you mean "decoupling the egg metadata" - which is fine, >> but to properly decouple, you need API level access to the metadata. PEP 376 >> offers read-only access, but as you rightly point out, it is only for >> filesystem data (and some form of zip file, which appears to be limited in >> some way, as it isn't PEP 302 based, and the actual format isn't defined >> anywhere). The basic point here is that PEP 376 needs to define precisely >> how pkgutil.get_distributions() scans sys.path looking for ".egg-info >> directories". What does it do for sys.path entries that don't correspond to >> filesystem directories? (Note - these may or may not be zip files. Even if >> they are zip files, an earlier entry on sys.path_hooks could have taken >> precedence. At the very least, you should only process path entries as zip >> files if their importer - in sys.path_importer_cache or via an explicit path >> hook scan - is a zipimporter object.). To be honest, this is a major can of >> worms. But if PEP 376 is not going to support PEP 302, then it must state >> that fact explicitly, to avoid giving people false expectations - >> particularly with Brett's importlib in Python 3.1, which will make it far >> easier for people to experiment with new packaging formats such as the ones >> Lukasz mentions above. And it MUST fail gracefully in the face of >> unsupported importer types. > > Well, we could always resurrect PEP 365, since pkg_resources already has > documented extensible support for arbitrary importers. ?That solves backward > *and* forward compatibility. ?Then PEP 376's uninstall facilities could be > implemented using pkg_resources' existing metadata query features. > > The primary downside to that, of course, is that it brings in the matter of > version specifications and dependencies... ?which appear to be a contentious > topic. ?(Note that Tarek is proposing to drop the PEP 386 proposal to > standardize a much more restrictive scheme than seutptools' version parser, > precisely because of the controversy.) I don't think we can add pkg_resources as-is because it does much more than providing the query features, which is beyond distutils scope. I think that part of the PEP 376 code at the end will be very similar to the subpart of pkg_resources that has the query features (as it is now with the current prototype) And what is nice with Paul's proposal on changing PEP 302 is that setuptools will be able to propose its EggFormats on the top of the pkgutils/distutils changes and make its pkg_resource code lighter. From martin at v.loewis.de Fri Jul 3 22:36:26 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 03 Jul 2009 22:36:26 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E0825.5020407@skippinet.com.au> References: <4A4D3C44.60504@gmail.com> <4A4E0825.5020407@skippinet.com.au> Message-ID: <4A4E6BCA.2090103@v.loewis.de> > So we must work without effective EOL support? If that's the case (i.e. no effective EOL support, the way svn supported it), then I think the PEP should make that clear (e.g. in a discussion section). For the server-side hooks: it would be good to know exactly what they check, wrt. line endings. To find out what files should not be stored with LF line endings, do "svn pg -R svn:eol-style .|grep CRLF". For win32text, it would probably be good if the FAQ would provide the relevant configuration instructions; it would be really helpful if somebody familiar with Windows and hg could provide detailed instructions well in advance of August 1. If we don't have anybody familiar with Windows and hg, we have a really serious problem. Regards, Martin From martin at v.loewis.de Fri Jul 3 22:38:32 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 03 Jul 2009 22:38:32 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A4D3C44.60504@gmail.com> <4A4E0825.5020407@skippinet.com.au> Message-ID: <4A4E6C48.7070408@v.loewis.de> >> This is exactly why I was asking for your advice - I can't work out how to >> work effectively with win32text as it stands myself, so remain stuck >> accidently checking in files with inappropriate line endings and stuck >> working out how to move pywin32's CVS repo with abandoning the very >> primitive EOL safety it offers... > > As long as the difference between \r\n- and \n-based files is clear > and can be reasoned about, I don't see why having some of both (I'm > assuming an overwhelming majority will have one, and only a few the > other) is a big problem. But feel free to enlighten me! If "both" means "both the server side test, and win32text", then Mark already gave the answer: he cannot use win32test because he does not know how to operate it (and doesn't bother studying its source code to understand it in detail). Regards, Martin From tjreedy at udel.edu Fri Jul 3 22:45:42 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 03 Jul 2009 16:45:42 -0400 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E50BC.8060201@mrabarnett.plus.com> References: <4A4E50BC.8060201@mrabarnett.plus.com> Message-ID: MRAB wrote: > Terry Reedy wrote: >> Dirkjan Ochtman wrote: >> >>> It needs to be decided where the hg repositories will live. I'd like >>> to propose to keep the hgwebdir instance at hg.python.org. This is an >>> accepted standard for many organizations, and an easy parallel to >>> svn.python.org. The 2.7 (trunk) repo might live at >>> http://hg.python.org/main/, for example, with py3k at >>> http://hg.python.org/py3k/. >> >> I would very much like the 'k' dropped from the py3 name. It was a >> funny joke when py3 was vaporware, now it is excess baggage which only >> puzzles non-insiders and newcomers. >> >> I think the two repos should be either symmetrically named >> >> hg.python.org/py2 >> hg.python.org/py3 >> >> If one must be designated 'main', it should be py3. >> >> Continuing to call py2 'main' will continue to discourage use of py3. >> > We could regard py3k as the phase from the original concept of Python 3 > to its 'prototype', Python 3.0. Right. And that phase is over, especially with Barry posting today on python-list that there will be no more 3.0.x releases ever. > Python 3.1 would be the first 'real/usable' version. Right. 'is'. as Barry also posted. tjr From martin at v.loewis.de Fri Jul 3 22:49:57 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 03 Jul 2009 22:49:57 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4A4E6EF5.4000901@v.loewis.de> > It will handle it, for sure, but I think it would all go easier if we > could work with stricter subset branches (and leave the effective > cherrypicking for the occasional problem). So I think the PEP should propose a workflow (or: merge flow) if you think we would be better off with a different one. In proposing such a workflow, consider these requirements: - we current have four active "maintenance" branches (i.e. where the entire code basis evolves): trunk, 3k, 2.6, and 3.1 (3.0 also until this morning). - in addition, we have two security branches currently: 2.4 and 2.5, although 2.4 will be closed soon. - our committers consistently refuse to merge changes across branches themselves, and likely continue to do so unless there is some feature of hg that I missed (e.g. one were merging would happen without any user specifically asking for it) Regards, Martin From martin at v.loewis.de Fri Jul 3 23:00:35 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 03 Jul 2009 23:00:35 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: <4A4E7173.1040507@v.loewis.de> > Should we consider adding a sys.revision attribute and begin the > deprecation of sys.subversion? I wouldn't mind killing sys.subversion "right away" (i.e. in trunk and 3k - obviously it has to stay in 2.6 and 3.1, and all the older branches). I'm -1 on calling it "sys.revision", as this makes it difficult to tell what the actual versioning system was, and hence how the data should be interpreted. It will already be a problem for 2.6, when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43', which will surely crash existing applications. I'm not sure what the motivation for a sys.revision is; it's probably similar to the desire of calling the machine code.python.org (instead of hg.python.org). It gives the illusion of being agnostic of the actual RCS being used. However, this is a complete illusion: anybody using it (either code.python.org, or sys.revision), *cannot* be agnostic of the specific technology. Regards, Martin From martin at v.loewis.de Fri Jul 3 23:25:15 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 03 Jul 2009 23:25:15 +0200 Subject: [Python-Dev] 3.1 buildbots Message-ID: <4A4E773B.10901@v.loewis.de> I have now turned the 3.0 buildbots off, and created new builders for 3.1. Regards, Martin From brett at python.org Fri Jul 3 23:39:14 2009 From: brett at python.org (Brett Cannon) Date: Fri, 3 Jul 2009 14:39:14 -0700 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E7173.1040507@v.loewis.de> References: <4A4E7173.1040507@v.loewis.de> Message-ID: On Fri, Jul 3, 2009 at 14:00, "Martin v. L?wis" wrote: > > Should we consider adding a sys.revision attribute and begin the > > deprecation of sys.subversion? > > I wouldn't mind killing sys.subversion "right away" (i.e. in trunk > and 3k - obviously it has to stay in 2.6 and 3.1, and all the older > branches). > > I'm -1 on calling it "sys.revision", as this makes it difficult to > tell what the actual versioning system was, and hence how the > data should be interpreted. It will already be a problem for 2.6, > when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43', > which will surely crash existing applications. > > I'm not sure what the motivation for a sys.revision is; it's > probably similar to the desire of calling the machine code.python.org > (instead of hg.python.org). It gives the illusion of being agnostic > of the actual RCS being used. However, this is a complete illusion: > anybody using it (either code.python.org, or sys.revision), *cannot* > be agnostic of the specific technology. We could add another value in the tuple that specifies the VCS: ('CPython', 'branches/release25-maint', '61464', 'svn'). I agree that VCSs are not universally the same, but the concept of a revision is universal. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Jul 3 23:41:56 2009 From: brett at python.org (Brett Cannon) Date: Fri, 3 Jul 2009 14:41:56 -0700 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: On Fri, Jul 3, 2009 at 11:22, Terry Reedy wrote: > Dirkjan Ochtman wrote: > > It needs to be decided where the hg repositories will live. I'd like >> to propose to keep the hgwebdir instance at hg.python.org. This is an >> accepted standard for many organizations, and an easy parallel to >> svn.python.org. The 2.7 (trunk) repo might live at >> http://hg.python.org/main/, for example, with py3k at >> http://hg.python.org/py3k/. >> > > I would very much like the 'k' dropped from the py3 name. It was a funny > joke when py3 was vaporware, now it is excess baggage which only puzzles > non-insiders and newcomers. > Is it really that confusing? I have never heard of anyone asking "what is py3k?" Plus I like keeping that bit of Python history around. I know I still use py3k as shorthand for Python 3.x. And we are not that serious of a bunch. =) > > I think the two repos should be either symmetrically named > > hg.python.org/py2 > hg.python.org/py3 > If we make it universal I say it should be '2.x' and '3.x'. The whole 'py' prefix is redundant. > > If one must be designated 'main', it should be py3. > > Continuing to call py2 'main' will continue to discourage use of py3. Yeah, 2.x shouldn't be special anymore. -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Fri Jul 3 23:52:39 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 03 Jul 2009 23:52:39 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A4E7173.1040507@v.loewis.de> Message-ID: <4A4E7DA7.2030402@v.loewis.de> > We could add another value in the tuple that specifies the VCS: > ('CPython', 'branches/release25-maint', '61464', 'svn'). I agree that > VCSs are not universally the same, but the concept of a revision is > universal. Actually, I think that's not the case. For bzr, the usual way of identifying a revision is by revision number, which, however, is not unique within a project, as each branch will use contiguous integers for numbers. There are also unique identifications - so a bzr revision has actually two numbers. More general, in a DVCS, it is not possible to access the revision being referred to by such a tuple. For sys.subversion, if [0]=='CPython', then you could go to svn.python.org. For a DVCS, the revision being identified may not be publically available, or may not live on a host that you can infer from your proposed sys.revision. For cloned branches, I wonder how sys.revision[1] would be computed. Regards, Martin From brett at python.org Fri Jul 3 23:53:52 2009 From: brett at python.org (Brett Cannon) Date: Fri, 3 Jul 2009 14:53:52 -0700 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E7DA7.2030402@v.loewis.de> References: <4A4E7173.1040507@v.loewis.de> <4A4E7DA7.2030402@v.loewis.de> Message-ID: On Fri, Jul 3, 2009 at 14:52, "Martin v. L?wis" wrote: > > We could add another value in the tuple that specifies the VCS: > > ('CPython', 'branches/release25-maint', '61464', 'svn'). I agree that > > VCSs are not universally the same, but the concept of a revision is > > universal. > > Actually, I think that's not the case. For bzr, the usual way of > identifying a revision is by revision number, which, however, is not > unique within a project, as each branch will use contiguous integers > for numbers. There are also unique identifications - so a bzr revision > has actually two numbers. > > More general, in a DVCS, it is not possible to access the revision being > referred to by such a tuple. For sys.subversion, if [0]=='CPython', then > you could go to svn.python.org. For a DVCS, the revision being > identified may not be publically available, or may not live on a host > that you can infer from your proposed sys.revision. > > For cloned branches, I wonder how sys.revision[1] would be computed. So are you saying we should drop the idea of a revision value altogether, or just embrace the differences and add a sys.mercurial attribute? -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Fri Jul 3 23:59:14 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 03 Jul 2009 23:59:14 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A4E7173.1040507@v.loewis.de> <4A4E7DA7.2030402@v.loewis.de> Message-ID: <4A4E7F32.7080309@v.loewis.de> > So are you saying we should drop the idea of a revision value > altogether, or just embrace the differences and add a sys.mercurial > attribute? That's what I would propose. It should be a best-effort(*) approach at providing all information that is needed to really find the source used for the specific version. Regards, Martin (*) even for svn it was best-effort only in case there were local modifications. From dirkjan at ochtman.nl Sat Jul 4 00:00:28 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sat, 4 Jul 2009 00:00:28 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: On Fri, Jul 3, 2009 at 23:41, Brett Cannon wrote: > If we make it universal I say it should be '2.x' and '3.x'. The whole 'py' > prefix is redundant. Right, I was aiming for /python/2.x and /python/3.x as well. Actually, I currently have /cpython to also make CPython less special among it's peers, but that idea was met with some resistance on #python-dev. Cheers, Dirkjan From brett at python.org Sat Jul 4 00:02:08 2009 From: brett at python.org (Brett Cannon) Date: Fri, 3 Jul 2009 15:02:08 -0700 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: On Fri, Jul 3, 2009 at 15:00, Dirkjan Ochtman wrote: > On Fri, Jul 3, 2009 at 23:41, Brett Cannon wrote: > > If we make it universal I say it should be '2.x' and '3.x'. The whole > 'py' > > prefix is redundant. > > Right, I was aiming for /python/2.x and /python/3.x as well. > > Actually, I currently have /cpython to also make CPython less special > among it's peers, but that idea was met with some resistance on > #python-dev. Don't worry about doing that right now. When the stdlib gets separated out we can revisit this. But for now just leave it as python. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Sat Jul 4 00:03:09 2009 From: barry at python.org (Barry Warsaw) Date: Fri, 3 Jul 2009 18:03:09 -0400 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E7173.1040507@v.loewis.de> References: <4A4E7173.1040507@v.loewis.de> Message-ID: On Jul 3, 2009, at 5:00 PM, Martin v. L?wis wrote: > I'm -1 on calling it "sys.revision", as this makes it difficult to > tell what the actual versioning system was, and hence how the > data should be interpreted. It will already be a problem for 2.6, > when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43', > which will surely crash existing applications. I can release a 2.6.3 right before the cut-over (well, just about any time between now and August 1st). Should we just plan now for a 2.6.3 on say July 24th, with a release candidate on July 20th? > I'm not sure what the motivation for a sys.revision is; it's > probably similar to the desire of calling the machine code.python.org > (instead of hg.python.org). It gives the illusion of being agnostic > of the actual RCS being used. However, this is a complete illusion: > anybody using it (either code.python.org, or sys.revision), *cannot* > be agnostic of the specific technology. Agreed. I originally chose code.python.org because I didn't want to be biased (maybe I should have been :). +1 for hg.python.org. I'd prefer to spell out sys.mercurial_revision. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From martin at v.loewis.de Sat Jul 4 00:09:52 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Jul 2009 00:09:52 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: <4A4E81B0.6010700@v.loewis.de> > - First of all, I've got the basic conversion down, I've done it a few > times now, with progressively better results. You can view some > results at http://hg.python.org/, which has a preliminary cpython > repository. *** The changeset hashes for that repo will change, so you > won't be able to commit or pull from it in the future.*** I'd like to see as much infrastructure going "life" well in advance of the switch over. Ideally, when we do switch, there should be no surprises (i.e. all problems should be known, and, ideally, be resolved). > * Set up hg-ssh infra (should be easy) > * Set up hooks (should be mostly straightforward) > * Set up roundup integration (should be made easier by quick revision > map hgweb extension) > * Write docs These are the ones that I was thinking of (and mentioned before, so they got onto your list :-) I would drop the roundup integration from the things that need to be done pre-migration - there currently is no svn integration, so not having it for hg is not a step backwards. > * buildbots: both the regular and the community build masters must > be notified. Fortunately buildbot includes support for hg. I've also > implemented this for Mercurial itself, so I don't expect problems > here. This should also be setup before the switchover, with a few test buildslaves (say, a Unix one and a Windows one). > In a DVCS, the committers are not necessarily the same > people who push, and so we can't check if the committer is a > contributor. We could use a hook to check if the committer is a > contributor if we keep a list of registered contributors. In the first sentence, you say that it can't actually work - so I think we should drop the test. > I propose that the revision identifier will be the short version of > hg's revision hash, for example 'dd3ebf81af43', augmented with '+' > (instead of 'M') if the working directory from which it was built was > modified. This mirrors the output of the hg id command, which is > intended for this kind of usage. I would like to see this well before the switch also. It could be a patch (unified diff) stored in the tracker, or it could be an actual branch that then needs to get merged with all active branches (IIUC). Regards, Martin From martin at v.loewis.de Sat Jul 4 00:15:54 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Jul 2009 00:15:54 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A4E7173.1040507@v.loewis.de> Message-ID: <4A4E831A.6090404@v.loewis.de> Barry Warsaw wrote: > On Jul 3, 2009, at 5:00 PM, Martin v. L?wis wrote: > >> I'm -1 on calling it "sys.revision", as this makes it difficult to >> tell what the actual versioning system was, and hence how the >> data should be interpreted. It will already be a problem for 2.6, >> when 2.6.3 will currently have a sys.subversion[2] of 'dd3ebf81af43', >> which will surely crash existing applications. > > I can release a 2.6.3 right before the cut-over (well, just about any > time between now and August 1st). Should we just plan now for a 2.6.3 > on say July 24th, with a release candidate on July 20th? I'm fine with that plan - but the original problem remains. We will surely release 2.6.4 at some point, and it will have a different version identification (based on hg rev ids). So those existing applications (which are probably few) will then crash for 2.6.4, unless we continue maintaining 2.6 in subversion, or just arrange to fake sys.subversion somehow (e.g. freezing it on the last subversion revision - which might still break applications that insist on accessing the revision mentioned - not sure whether such applications actually exist). Regards, Martin From barry at python.org Sat Jul 4 00:37:19 2009 From: barry at python.org (Barry Warsaw) Date: Fri, 3 Jul 2009 18:37:19 -0400 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E831A.6090404@v.loewis.de> References: <4A4E7173.1040507@v.loewis.de> <4A4E831A.6090404@v.loewis.de> Message-ID: On Jul 3, 2009, at 6:15 PM, Martin v. L?wis wrote: > I'm fine with that plan - but the original problem remains. We will > surely release 2.6.4 at some point, and it will have a different > version > identification (based on hg rev ids). > > So those existing applications (which are probably few) will then > crash > for 2.6.4, unless we continue maintaining 2.6 in subversion, or just > arrange to fake sys.subversion somehow (e.g. freezing it on the last > subversion revision - which might still break applications that insist > on accessing the revision mentioned - not sure whether such > applications > actually exist). Doesn't Mercurial support an Subversion bridge? Would it be possible to provide a /read-only/ copy of the hg branches for 2.4 (maybe), 2.5, 2.6, and 3.1? If so, then the release managers would simply have to cut their releases from the svn copy instead of the hg master. /All/ other work would be done from the hg master. This shouldn't be too much of a burden since it's done so rarely and would end with the EOL of each of those branches. It would mean maintaining the bridge until all currently released versions are EOL. If that's not possible or feasible, then given the documented sys.subversion semantics, I think we should just freeze the tuple at e.g. ('CPython', 'branches/release26-maint', None). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 832 bytes Desc: This is a digitally signed message part URL: From mhammond at skippinet.com.au Sat Jul 4 01:30:28 2009 From: mhammond at skippinet.com.au (Mark Hammond) Date: Sat, 04 Jul 2009 09:30:28 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E1009.4050901@gmail.com> References: <4A4D3C44.60504@gmail.com> <4A4E0825.5020407@skippinet.com.au> <4A4E1009.4050901@gmail.com> Message-ID: <4A4E9494.2040406@skippinet.com.au> On 4/07/2009 12:04 AM, Nick Coghlan wrote: > However, I expect that would still be painful to work with for Windows > developers, even if it prevented any line ending problems from making > their way into the main repository. I believe that is where the > win32text extensions can help. Looking at the Wiki page for win32text > [1], I believe it would be a matter of configuring the extension to > encode and decode all files with the extensions: > > *.py > *.pyw > *.h > *.c > *.in > *.rst > *.asdl > > That said, I don't see a way to tell win32text to also translate files > which don't have an extension at all (e.g. NEWS or ACKS), and there > doesn't seem to be a way to tell it to *skip* files matching certain > patterns (if there was, we could just tell it to ignore extensions like > .bat, .sln, .vcproj, .vsprops, .dps, .dsw, .wse, .ico, .bmp and convert > everything else) The big problem I have with win32text is that the rules are not versioned, meaning we will rely on every (windows only?) developer manually enabling the win32text extension, then manually adding these rules to an unversioned hgrc file. Should the rules change over time, we will need to communicate that to every developer. Contrast this with, say, SVN, where the rule is a versioned property on the file. Further, if a repo *already* has mixed line endings and the file doesn't appear in the rules, win32text makes *lots* of noise regarding existing line endings and to "review your rules before you commit." In effect, it seems to me that win32text will only work effectively if *all* developers using a repo are using it, or none are, or other steps have been put in place to ensure \r\n line endings in the repo are otherwise impossible. I've completely failed to use win32text on the mozilla hg repo for exactly this reason. Cheers, Mark From scott+python-dev at scottdial.com Sat Jul 4 01:11:03 2009 From: scott+python-dev at scottdial.com (Scott Dial) Date: Fri, 03 Jul 2009 19:11:03 -0400 Subject: [Python-Dev] PEP 376 In-Reply-To: <94bdd2610907020201w5f93b6a1j70fa599d6a1011a8@mail.gmail.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com> <877hyrwxxi.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610907020201w5f93b6a1j70fa599d6a1011a8@mail.gmail.com> Message-ID: <4A4E9007.1050908@scottdial.com> Tarek Ziad? wrote: > On Thu, Jul 2, 2009 at 5:44 AM, Stephen J. Turnbull wrote: >> Another general principle: even in the draft PEP, say "is", not "will >> be". > > Ok I'll fix that. That's a French stuff : in french, "will be" isn't > speculative at all. > I don't think "will be" is necessarily speculative in English either. I think the issue is that after the PEP is implemented, the document lives on. And when one reads, "X will be done." If "X" is not done in the current implementation, it is unclear whether that is an error or a promise that at some point in the future the implementation will be changed to do "X". In other words, the PEP will live on long after you have completed the implementation and it's at that point that occurrences of "will be" in the PEP become speculative. Someone feel free to correct me if I am incorrect about the desired tone and use of the document.. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From mhammond at skippinet.com.au Sat Jul 4 01:37:29 2009 From: mhammond at skippinet.com.au (Mark Hammond) Date: Sat, 04 Jul 2009 09:37:29 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A4D3C44.60504@gmail.com> <4A4E0825.5020407@skippinet.com.au> Message-ID: <4A4E9639.3010608@skippinet.com.au> On 3/07/2009 11:43 PM, Dirkjan Ochtman wrote: > On Fri, Jul 3, 2009 at 15:31, Mark Hammond wrote: >> So we must work without effective EOL support? I fear we will end up like >> the mozilla hg repo with some files in windows line endings and some with >> linux. While my editing tools are good enough to preserve existing EOL >> styles, I've found myself accidentally checking in new \r\n terminated files >> in a repo which otherwise uses \n line endings. IMO, SVN's EOL support was >> better than no EOL support. > > This is why we'll have hooks -- to prevent you from pushing changesets > with inappropriate, to say the least, and, if you're willing to do a > little bit of extra work, to prevent you from committing them. > >> This is exactly why I was asking for your advice - I can't work out how to >> work effectively with win32text as it stands myself, so remain stuck >> accidently checking in files with inappropriate line endings and stuck >> working out how to move pywin32's CVS repo with abandoning the very >> primitive EOL safety it offers... > > As long as the difference between \r\n- and \n-based files is clear > and can be reasoned about, I don't see why having some of both (I'm > assuming an overwhelming majority will have one, and only a few the > other) is a big problem. But feel free to enlighten me! I'm surprised it isn't obvious. Think about it this way: CVS had basic EOL support and SVN gave slightly better support, and we leveraged this support in both those tools. The end result is a very clean repo with consistent line endings and a distinct lack of developer friction between the platforms. Your position seems to be that we *never* needed EOL conversion, especially in SVN which offered decent hooks, and that having a source repo with a mismatch and unpredictable line endings isn't really a problem at all. Even just at face value, it seems clear we have taken full advantage of that feature for around 1.5 decades, so to suggest we never actually needed it, or at least don't need it now for some reason, seems slightly disingenuous... Cheers, Mark From martin at v.loewis.de Sat Jul 4 01:47:39 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Jul 2009 01:47:39 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A4E7173.1040507@v.loewis.de> <4A4E831A.6090404@v.loewis.de> Message-ID: <4A4E989B.9010806@v.loewis.de> > Doesn't Mercurial support an Subversion bridge? Would it be possible to > provide a /read-only/ copy of the hg branches for 2.4 (maybe), 2.5, 2.6, > and 3.1? If so, then the release managers would simply have to cut > their releases from the svn copy instead of the hg master. /All/ other > work would be done from the hg master. This shouldn't be too much of a > burden since it's done so rarely and would end with the EOL of each of > those branches. > > It would mean maintaining the bridge until all currently released > versions are EOL. That sounds very complicated (but Dirkjan should correct me if I'm wrong). I have already mentally prepared to port welease to hg, to create the missing 2.4 and 2.5 releases. > If that's not possible or feasible, then given the documented > sys.subversion semantics, I think we should just freeze the tuple at > e.g. ('CPython', 'branches/release26-maint', None). That would work for me (I had to re-read the documentation to see that None is a valid, documented value for version, and allowed if "the tree was exported"). Interesting to notice that sys.subversion didn't even last the life time of one feature release :-) Regards, Martin From ncoghlan at gmail.com Sat Jul 4 02:47:05 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 04 Jul 2009 10:47:05 +1000 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> Message-ID: <4A4EA689.1010704@gmail.com> Tarek Ziad? wrote: > 2009/7/3 Paul Moore : >> Does this sound sensible? Tarek, would you be OK with me attempting to >> modify your prototype to support this protocol? Are there any tests >> for PEP 376, so that I can confirm I haven't completely broken >> something? If I can, I'll knock up some simple prototype importers for >> non-standard examples, and see how they work with all this. > > Yes that's exactly what I was thinking about after the discussion we > had in the other > thread. This change would allow much more flexibility. It's also very similar to the ideas that started to tick over in the back of my head after the previous discussion (only far more fleshed out that anything I got to - fortunately I read Paul's message before spending too much time on it!) Specifically regarding this comment from Paul: > Finder.uninstall(distribution_name, filter=callable, installer=name) > Uninstall the given distribution. It's likely that many finders will > be read-only. In that case, this function should return None. > Otherwise, return a list of the "files" removed. (This may need some > clarification, as many finders won't have a concept of a "file name"). PEP 302 finders and loaders actually *are* expected to have a concept of a "file name". For Python modules they they are expected to have it so they can populate __file__ correctly (runpy is also a lot happier with them when they expose get_filename(), a relatively recent addition to the PEP 302 API). For other resources they're expected to have it so the get_data() API can work correctly. One other thing to consider: it may be desirable to have hooks for meta_path and path_hooks in distutils that are checked *before* the full import hooks in the sys module. The reason I say this is that it would allow someone to override just the metadata retrieval (e.g. to pick up the extra dependency information saved by setuptools and friends) without having to worry about breaking the actual imports from those locations. Cheers, Nick. P.S. +lots on using 'metadata' in the PEP 376 method names rather than the jargon 'egginfo'. Jargon isn't always bad, but using it seems fairly gratuitous in this case. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sat Jul 4 03:20:58 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 04 Jul 2009 11:20:58 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E9494.2040406@skippinet.com.au> References: <4A4D3C44.60504@gmail.com> <4A4E0825.5020407@skippinet.com.au> <4A4E1009.4050901@gmail.com> <4A4E9494.2040406@skippinet.com.au> Message-ID: <4A4EAE7A.5090406@gmail.com> Mark Hammond wrote: > On 4/07/2009 12:04 AM, Nick Coghlan wrote: > The big problem I have with win32text is that the rules are not > versioned, meaning we will rely on every (windows only?) developer > manually enabling the win32text extension, then manually adding these > rules to an unversioned hgrc file. Should the rules change over time, > we will need to communicate that to every developer. Contrast this > with, say, SVN, where the rule is a versioned property on the file. > > Further, if a repo *already* has mixed line endings and the file doesn't > appear in the rules, win32text makes *lots* of noise regarding existing > line endings and to "review your rules before you commit." In effect, > it seems to me that win32text will only work effectively if *all* > developers using a repo are using it, or none are, or other steps have > been put in place to ensure \r\n line endings in the repo are otherwise > impossible. I've completely failed to use win32text on the mozilla hg > repo for exactly this reason. I didn't realise Mercurial didn't have a way for a repository to provide versioned extension settings, but in this case I think using our own server side hook can deal with the problem (either adding it to the existing whitespace hook that checks for tab/space sanity in Python files, or else as a separate hook that just checks line endings). Such a hook should be able to base the determination of which files it checks for \r\n line endings on a normal versioned file in the repository. For everything that *isn't* mentioned in that file, the hook would check that it had \n line endings. I believe that would address both of your concerns: 1. A server side hook means that the rules are enforced on everyone regardless of whether they use win32text or not (e.g no Linux developers corrupting a .dsp file by opening it and saving it with \n line endings) 2. The server side hooks rules will be versioned, allowing people to copy them to their hgrc file so that win32text will correctly handle the line endings for them. Also, on another read of the win32text instructions, I believe the configuration actually *does* support negative filters, it just doesn't explain them very well (it only mentions them in the context of overriding default settings in the config file for a specific repository). I think a .hgrc file along the following lines might do what we want (I'm not developing on Windows though, so I haven't actually tested it): ============= [extensions] hgext.win32text= [encode] # Encode files that don't contain NUL characters. ** = cleverencode: # But leave Windows specific and known binary files alone **.bat = ! **.sln = ! **.vcproj = ! **.vsprops = ! **.dps = ! **.dsw = ! **.wse = ! **.ico = ! **.bmp = ! [decode] # Decode files that don't contain NUL characters. ** = cleverdecode: # But leave Windows specific and known binary files alone **.bat = ! **.sln = ! **.vcproj = ! **.vsprops = ! **.dps = ! **.dsw = ! **.wse = ! **.ico = ! **.bmp = ! ============= Assuming we can find settings that do what we want then a sample HGRC file could easily be provided in the Misc directory alongside things like python-mode.el. An alternative to the above would be to store *everything* (even the Windows specific files) in the repository with \n line endings (still enforced by a server side hook), and then *require* that Windows users run win32text with settings like the following: ============= [extensions] hgext.win32text= [encode] # Encode files that don't contain NUL characters. ** = cleverencode: # But leave known binary files alone **.ico = ! **.bmp = ! [decode] # Decode files that don't contain NUL characters. ** = cleverdecode: # But leave known binary files alone **.ico = ! **.bmp = ! ============= Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From mhammond at skippinet.com.au Sat Jul 4 03:20:29 2009 From: mhammond at skippinet.com.au (Mark Hammond) Date: Sat, 04 Jul 2009 11:20:29 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A4D3C44.60504@gmail.com> <4A4E0825.5020407@skippinet.com.au> Message-ID: <4A4EAE5D.6000908@skippinet.com.au> On 3/07/2009 11:43 PM, Dirkjan Ochtman wrote: > As long as the difference between \r\n- and \n-based files is clear > and can be reasoned about, I don't see why having some of both (I'm > assuming an overwhelming majority will have one, and only a few the > other) is a big problem. But feel free to enlighten me! In that case, and given that I expect more Windows users to be clueless about EOL issues than users of other operating systems, I propose we check in all files initially with Windows line endings Mark From ncoghlan at gmail.com Sat Jul 4 03:33:18 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 04 Jul 2009 11:33:18 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: <4A4EB15E.3080808@gmail.com> Brett Cannon wrote: > On Fri, Jul 3, 2009 at 15:00, Dirkjan Ochtman > wrote: > Actually, I currently have /cpython to also make CPython less special > among it's peers, but that idea was met with some resistance on > #python-dev. > > Don't worry about doing that right now. When the stdlib gets separated > out we can revisit this. But for now just leave it as python. Yes, if we ever do that split we can leave the common code (stdlib, test suite) under /python and move the CPython specific stuff to /cpython. +1 on 2.x and 3.x for the main development branches (although that does lead me to wonder if we should just make the maintenance branches 2.6, 3.1, etc). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From mhammond at skippinet.com.au Sat Jul 4 03:33:06 2009 From: mhammond at skippinet.com.au (Mark Hammond) Date: Sat, 04 Jul 2009 11:33:06 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4EAE7A.5090406@gmail.com> References: <4A4D3C44.60504@gmail.com> <4A4E0825.5020407@skippinet.com.au> <4A4E1009.4050901@gmail.com> <4A4E9494.2040406@skippinet.com.au> <4A4EAE7A.5090406@gmail.com> Message-ID: <4A4EB152.6040406@skippinet.com.au> On 4/07/2009 11:20 AM, Nick Coghlan wrote: > > I didn't realise Mercurial didn't have a way for a repository to provide > versioned extension settings, but in this case I think using our own > server side hook can deal with the problem (either adding it to the > existing whitespace hook that checks for tab/space sanity in Python > files, or else as a separate hook that just checks line endings). > > Such a hook should be able to base the determination of which files it > checks for \r\n line endings on a normal versioned file in the > repository. For everything that *isn't* mentioned in that file, the hook > would check that it had \n line endings. > > I believe that would address both of your concerns: > 1. A server side hook means that the rules are enforced on everyone > regardless of whether they use win32text or not (e.g no Linux developers > corrupting a .dsp file by opening it and saving it with \n line endings) > 2. The server side hooks rules will be versioned, allowing people to > copy them to their hgrc file so that win32text will correctly handle the > line endings for them. That sounds like it should work - presumably mention of this should go into the PEP. (While that is great for Python, it is a shame I probably can't use the same strategy for other hg repos I care about as I don't have access to server side hooks - but that if off-topic for this list.) > Also, on another read of the win32text instructions, I believe the > configuration actually *does* support negative filters, it just doesn't > explain them very well (it only mentions them in the context of > overriding default settings in the config file for a specific > repository). I think a .hgrc file along the following lines might do > what we want (I'm not developing on Windows though, so I haven't > actually tested it): This *appears* to be correct at first glance, but in practice it doesn't interact well with wildcards specifications - particularly '**=cleverencode'. I started a thread on the ML and submitted a patch a few months ago to fix this and it was accepted, but sadly it seemed to have dropped off the queue somewhere. The patch now conflicts, and I've promised to resubmit it when I find time. But even with that in place it doesn't address the more general problem that when only *some* developers use the extension - mixed mode files are still very possible, at which point win32text starts reporting enough spurious conflicts to become unusable for me. Eg, doing a clean checkout of mozilla with win32text enabled results in a working tree with hundreds of files reporting every line in the file has changed. It is worth noting that win32text also works on Linux, and that in theory, win32text should be able to allow Linux users to work seamlessly with a \r\n repo, in the same way it promises to allow Windows users to work with \n repos. But in practice, it is simply never used by Linux developers, even in the hg core dev team. Patches or comments regarding win32text are always met with "but no one here ever uses it". If they did, and some of the core hg team tried to experiment with win32text and mixed line ending repos the problems would, I believe, become self-evident... Cheers, Mark From ncoghlan at gmail.com Sat Jul 4 03:37:50 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 04 Jul 2009 11:37:50 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E81B0.6010700@v.loewis.de> References: <4A4E81B0.6010700@v.loewis.de> Message-ID: <4A4EB26E.3050706@gmail.com> Martin v. L?wis wrote: > I would drop the roundup integration from the things that need to > be done pre-migration - there currently is no svn integration, so > not having it for hg is not a step backwards. That's not quite true - the tracker turns text like "r64537" into a link to the appropriate changeset on svn.python.org. It would be unfortunate to lose that easy ability to indicate when a bug was fixed (or introduced). This is more something on the roundup side though: recognising the presence of a hg id in a comment and turning it into a link to hg.python.org. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sat Jul 4 03:44:17 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 04 Jul 2009 11:44:17 +1000 Subject: [Python-Dev] PEP 376 In-Reply-To: <4A4E9007.1050908@scottdial.com> References: <94bdd2610906291057i3d2f533dqcfb3c986a5946835@mail.gmail.com> <4A4927D5.4040001@v.loewis.de> <4A4A05DE.2040804@gmail.com> <79990c6b0906301137r3728c86atcca4cf2da61cdf7b@mail.gmail.com> <94bdd2610906301228s4ac0fb20id44bd144b82af600@mail.gmail.com> <79990c6b0906301311l4fd029dcq7ec2b4f427eb6bfb@mail.gmail.com> <79990c6b0907010444v4d7c8823iee12e3d05b3696bc@mail.gmail.com> <94bdd2610907010729k6b34ac9k77368710470903b1@mail.gmail.com> <877hyrwxxi.fsf@uwakimon.sk.tsukuba.ac.jp> <94bdd2610907020201w5f93b6a1j70fa599d6a1011a8@mail.gmail.com> <4A4E9007.1050908@scottdial.com> Message-ID: <4A4EB3F1.6090705@gmail.com> Scott Dial wrote: > Someone feel free to correct me if I am incorrect about the desired tone > and use of the document.. I don't think we're particularly consistent. For example, I actually ending up adding a note to PEP 343 indicating that people shouldn't read too much into the verb tense in that document because we were working on it for over a year and things that were in the future when we started were well in the past by the time we finished. In general, I agree that using present tense ("is") in specifications and documentation such as PEPs ends up standing the test of time better than using future tense ("will be"). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sat Jul 4 04:30:22 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 04 Jul 2009 12:30:22 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4EB152.6040406@skippinet.com.au> References: <4A4D3C44.60504@gmail.com> <4A4E0825.5020407@skippinet.com.au> <4A4E1009.4050901@gmail.com> <4A4E9494.2040406@skippinet.com.au> <4A4EAE7A.5090406@gmail.com> <4A4EB152.6040406@skippinet.com.au> Message-ID: <4A4EBEBE.7040604@gmail.com> Mark Hammond wrote: > This *appears* to be correct at first glance, but in practice it doesn't > interact well with wildcards specifications - particularly > '**=cleverencode'. I started a thread on the ML and submitted a patch a > few months ago to fix this and it was accepted, but sadly it seemed to > have dropped off the queue somewhere. The patch now conflicts, and I've > promised to resubmit it when I find time. But even with that in place > it doesn't address the more general problem that when only *some* > developers use the extension - mixed mode files are still very possible, > at which point win32text starts reporting enough spurious conflicts to > become unusable for me. Eg, doing a clean checkout of mozilla with > win32text enabled results in a working tree with hundreds of files > reporting every line in the file has changed. Ah, the scope of the issues begin to become clear... While a server side hook should be able to deal with the mixed mode file problem, I'm not sure what can be done about the problems with properly configuring win32text. If wildcard specifications don't interact properly with filtered negative specifications then that would appear to rule out that approach. The presence of text files without extensions like NEWS and ACKS in the Python repository appears to rule out the use of positive filters to select only the files that are stored in the repository with \n line endings. I spent some more time exploring the approach recommended on the win32text documentation page as a possible way of handling the situation: 1. Store all text files (even Windows specific ones) in \n format in the repository 2. Apply the win32text precommit hook to forbid the introduction of \r\n line endings 3. Use the recommended settings from the win32text documentation: =============================== [extensions] hgext.win32text= [encode] # Encode files that don't contain NUL characters. ** = cleverencode: [decode] # Decode files that don't contain NUL characters. ** = cleverdecode: =============================== This would be the equivalent of setting "svn:eol-style native" on every file in the repository. However, in running Martin's query (svn pg -R svn:eol-style .|grep CRLF) over 2.x and 3.x checkouts, I found that \r\n line endings are currently enforced for: - .bat files under Tools/buildbot - .dsp and .dsw files under PC/VC6 - Lib/email/test/data/msg_26.txt I believe the rationale for the first two is to allow a source tarball to be prepared on Linux but still be usable on Windows (e.g. see [1]). I'm not clear on the rationale for the explicit CRLF line ending on the email test message, but I would guess it is to ensure that the email module can handle CRLF line endings correctly regardless of platform. Only VC6 files appear on the list because later versions of Visual Studio actually tolerate \n line endings in their project files. Mercurial's heuristic handling of text vs binary and expected line endings fails completely for these use cases. I think there needs to be a solid answer in place for these use cases before the actual migration to Mercurial takes place. A hand-waving "use win32text" isn't enough - it needs to be "use win32text with these exact settings" (with server side hook support to enforce the rules). And since Mercurial doesn't even allow us to say "this is a binary file" the way CVS used to I'm currently not seeing any way for that to happen except for win32text to be updated to correctly handle wild cards in combination with negative filters. Cheers, Nick. [1] http://mail.python.org/pipermail/python-dev/2006-March/062225.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From stephen at xemacs.org Sat Jul 4 05:20:04 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 04 Jul 2009 12:20:04 +0900 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87prchuobf.fsf@uwakimon.sk.tsukuba.ac.jp> Dirkjan Ochtman writes: > On Fri, Jul 3, 2009 at 15:29, Stephen J. Turnbull wrote: > > I'll have to try them again now that 1.3 is out, but I found Mercurial > > named branches fundamentally unsuited to release management. > > Can you explain why, please? It's not clear from what you say > below. Well, the main problem I had was one that you say has improved: the various issues of confusion due to the presence of multiple active heads in a single repository. IME, Mercurial strongly encourages a non-branching style. Although I can't fully explain in concrete terms what makes me feel that way, it's certainly consistent with your own inclination to advise "subset branches". Part of it comes from the fact that you can't have a single revision on two branches. I would really like the node of a release branch to be on both the branch and the mainline so that the tag appears in the history of both, but that's not possible. Another issue (which you say has improved) is the handling of multiple heads. With unnamed heads, it's just too painful to leave them hanging around, so you merge them immediately (in fact, all XEmacs committers currently active use either "pull -u" or the fetch extension). With named branches, the additional heads do tend to hang around. I found that named branches tended to get inadvertantly pushed, and worse, they'd end up being the tip, which Mercurial treats specially. In one case a completely private branch inadvertantly got pushed into my "pristine" clone of the public repo, and from there into the public repo, where (since it had been the tip in my private workspace) it ended up as the tip in public. Embarrassing, to say the least. Fortuntely, the branch was more or less ready to be pushed anyway, but one of my colleagues ended up working on that branch without realizing that he wasn't on (his) mainline any more, and wondered why some previously done work suddenly disappeared. A good time was had by nobody involved. I don't know if that has been fixed in hg; the experience was painful enough that my workflow adjusted immediately. > > Ditto named branches. ?The problem is that (unless the internal > > implementation has changed very recently) a Mercurial revision can be > > on exactly one named branch (or on the trunk). > > That's still true. > > > Which defeats the purpose of having named branches, really. ?(I mean > > the version control purpose; obviously it still can save disk space.) > > Why does it defeat the purpose? What, in your opinion, is the > purpose? I use named branches to collect a sequence of revisions as a named object, for viewing and manipulation, as differentiated from some other sequence, for *various* values of "some other sequence". The problem is that suppose you have a branch A off the trunk, and you then (several revisions down the line of development) branch B off A. Now A meanders off and runs into "not ready for prime time" problems, while B just swims along. Problem is, you can't easily find the history of B relative to the trunk because much of its history (since forking from the *trunk*) is labeled A. > > Unless you're really short on space, though, that's not a big deal. > > What would be more important to me (not that I matter for the purpose > > of Python, but in XEmacs -- also a Mercurial shop -- I do :-) would be > > the other way around: pulling an external branch into a named branch. > > I have a feeling that working with such a repository with others would > > be a little difficult. > > Can you give an example? No, I haven't tried it. What concerns me is that I suspect that the branch name will end up as part of the revision's internal identifier, and that means that if you and I separately create the same named branch we have to choose exactly the same name or the branches won't be recognized as the same, resulting in the mother of all spurious merge conflicts. > > As others (MvL, I think) have commented, this isn't really relevant to > > Python which generally has four mainlines going at once. ?I don't see > > why the requirements are going to change with the shift to hg, and I > > see no reason why hg won't handle the existing workflow just fine. > > It will handle it, for sure, but I think it would all go easier if we > could work with stricter subset branches (and leave the effective > cherrypicking for the occasional problem). Sure, but what do you propose? That we nuke Python 3.1, 3.2, and 2.6? They're all pretty divergent from 2.7 by now, as well as from each other. Not to mention the "security branches" that are still around. And individual developers are for sure going to do interesting things in their private workspaces. I see that George Brandl and Martin van Loewis seem to be accomodating your viewpoint, but I don't get the impression that either you (as the hg migration proponent) nor they (as core committers) realize how far apart your assumptions are. You are talking about the Mercurial project, which has *one* line of development. There are many such projects; the ones I'm most familiar with are XEmacs and Scheme48, which have adopted a subset branch approach different the Mercurial project's (XEmacs is modeled on Scheme48). Works well enough (when I'm wearing my release manager hat; it's a little constraining as a developer used to the insane flexibility of git). But Python is *not* such a project. The problem is not coordinating concurrent development of a closeknit group of committers all working on the same mainline. In Python, there are *four* mainlines with rather different purposes, and a diverse group of developers, some who work on only one line of development, others who work on several, and a few who accept the role of coordinating across them. From martin at v.loewis.de Sat Jul 4 05:21:46 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Jul 2009 05:21:46 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4EB26E.3050706@gmail.com> References: <4A4E81B0.6010700@v.loewis.de> <4A4EB26E.3050706@gmail.com> Message-ID: <4A4ECACA.5080508@v.loewis.de> Nick Coghlan wrote: > Martin v. L?wis wrote: >> I would drop the roundup integration from the things that need to >> be done pre-migration - there currently is no svn integration, so >> not having it for hg is not a step backwards. > > That's not quite true - the tracker turns text like "r64537" into a link > to the appropriate changeset on svn.python.org. It would be unfortunate > to lose that easy ability to indicate when a bug was fixed (or introduced). > > This is more something on the roundup side though: recognising the > presence of a hg id in a comment and turning it into a link to > hg.python.org. Ah, right. That must be done, of course (although I suppose there is little hope to have the existing references continue to work). Regards, Martin From steve at pearwood.info Sat Jul 4 05:36:49 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 4 Jul 2009 13:36:49 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: <200907041336.49868.steve@pearwood.info> On Sat, 4 Jul 2009 04:22:57 am Terry Reedy wrote: > I would very much like the 'k' dropped from the py3 name. It was a > funny joke when py3 was vaporware, now it is excess baggage which > only puzzles non-insiders and newcomers. +1 Some day we'll be using Python3.7 and wondering what the "k" means. Alpha, beta, .... kappa version? -- Steven D'Aprano From skippy.hammond at gmail.com Sat Jul 4 06:03:30 2009 From: skippy.hammond at gmail.com (Mark Hammond) Date: Sat, 04 Jul 2009 14:03:30 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4EBEBE.7040604@gmail.com> References: <4A4D3C44.60504@gmail.com> <4A4E0825.5020407@skippinet.com.au> <4A4E1009.4050901@gmail.com> <4A4E9494.2040406@skippinet.com.au> <4A4EAE7A.5090406@gmail.com> <4A4EB152.6040406@skippinet.com.au> <4A4EBEBE.7040604@gmail.com> Message-ID: <4A4ED492.1050307@gmail.com> On 4/07/2009 12:30 PM, Nick Coghlan wrote: ... > I think there needs to be a solid answer in place for these use cases > before the actual migration to Mercurial takes place. A hand-waving "use > win32text" isn't enough - it needs to be "use win32text with these exact > settings" (with server side hook support to enforce the rules). > > And since Mercurial doesn't even allow us to say "this is a binary file" > the way CVS used to I'm currently not seeing any way for that to happen > except for win32text to be updated to correctly handle wild cards in > combination with negative filters. I agree with your conclusion. My ruminating on this over the last few months leaves me thinking this would involve: * my older 'accepted but then lost' hg patch to allow an explicit 'none' rule for a single file to override wildcards. * win32text be enhanced to use a normal versioned file in the root of the repo, much like hgingore, where a project can maintain project wide rules. * win32text be enhanced such that all python developers, regardless of platform, are willing to use this extension, even if the majority of files happen to use their native line ending (sauce for the goose is sauce for the gander, and all that...) * commit hooks be implemented to enforce this - but this should not be necessary if the above was implemented and socially enforced. Cheers, Mark > > Cheers, > Nick. > > [1] http://mail.python.org/pipermail/python-dev/2006-March/062225.html > From stephen at xemacs.org Sat Jul 4 06:08:36 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 04 Jul 2009 13:08:36 +0900 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4EBEBE.7040604@gmail.com> References: <4A4D3C44.60504@gmail.com> <4A4E0825.5020407@skippinet.com.au> <4A4E1009.4050901@gmail.com> <4A4E9494.2040406@skippinet.com.au> <4A4EAE7A.5090406@gmail.com> <4A4EB152.6040406@skippinet.com.au> <4A4EBEBE.7040604@gmail.com> Message-ID: <87ocs1um2j.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > I'm not clear on the rationale for the explicit CRLF line ending on the > email test message, but I would guess it is to ensure that the email > module can handle CRLF line endings correctly regardless of platform. IIRC, that's just the standard for text email. Lines must be less than 1000 characters long, should be no more than 78 characters long, and are terminated by CRLF. As they say, "honored more in the breach than the observance", but that's what the RFCs say. From pje at telecommunity.com Sat Jul 4 06:31:54 2009 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 04 Jul 2009 00:31:54 -0400 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <87prchuobf.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp> <87prchuobf.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20090704042935.949263A40D7@sparrow.telecommunity.com> At 12:20 PM 7/4/2009 +0900, Stephen J. Turnbull wrote: >IME, Mercurial strongly encourages a non-branching style. Although I >can't fully explain in concrete terms what makes me feel that way, >it's certainly consistent with your own inclination to advise "subset >branches". Part of it comes from the fact that you can't have a >single revision on two branches. I would really like the node of a >release branch to be on both the branch and the mainline so that the >tag appears in the history of both, but that's not possible. I'm a total newbie to Mercurial - I've basically done little more than the tutorial, in fact. So somebody correct me if what I'm about to say is completely stupid. Wouldn't the simple thing to do in Mercurial, just be to use different repositories for long-lived branches? I mean, if you're not merging them that much anyway, what's the point? Isn't it just as easy to pull from another repository, as from another branch within that repository? From ncoghlan at gmail.com Sat Jul 4 07:13:52 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 04 Jul 2009 15:13:52 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4ECACA.5080508@v.loewis.de> References: <4A4E81B0.6010700@v.loewis.de> <4A4EB26E.3050706@gmail.com> <4A4ECACA.5080508@v.loewis.de> Message-ID: <4A4EE510.7030409@gmail.com> Martin v. L?wis wrote: > Ah, right. That must be done, of course (although I suppose there is > little hope to have the existing references continue to work). I'd guess that the only way to keep those functional is to keep svn.python.org around in read-only mode. Although I'm not entirely sure about that - would it be possible to build up a mapping from svn revision numbers to hg repository IDs as the history information is being migrated? If that wasn't too difficult, it might be worth doing. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From stephen at xemacs.org Sat Jul 4 07:50:33 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 04 Jul 2009 14:50:33 +0900 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <20090704042935.949263A40D7@sparrow.telecommunity.com> References: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp> <87prchuobf.fsf@uwakimon.sk.tsukuba.ac.jp> <20090704042935.949263A40D7@sparrow.telecommunity.com> Message-ID: <87ljn5uhcm.fsf@uwakimon.sk.tsukuba.ac.jp> P.J. Eby writes: > Wouldn't the simple thing to do in Mercurial, just be to use > different repositories for long-lived branches? I mean, if you're > not merging them that much anyway, what's the point? I basically agree with that, and so does Dirkjan, I think. I'm not sure why he brought up the idea of using named branches. > Isn't it just as easy to pull from another repository, as from > another branch within that repository? Not if the other repository is remote and offline, and pulling from a remote does take more time, as do things like diffs against that branch. The temptation is to leave the remote repos on the public host, rather than put in the effort to make sure your local clones are up to date. If you do, there are minor annoyances like that. So people who actually work on more than one line of development learn pretty soon to keep a local clone for each repo they're interested in. Problem solved, IMO, but some people worry about the extra space taken up. From dirkjan at ochtman.nl Sat Jul 4 10:03:32 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sat, 4 Jul 2009 10:03:32 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: On Fri, Jul 3, 2009 at 20:04, Brett Cannon wrote: > Fine by me as long as people realize that if anything is questionable then > the switch will not happen. Getting this right takes precedence over any > deadline. And obviously we will need to do at least one live conversion on > python.org hardware to make sure everything will work smoothly. I'm not sure I see the need to do a (live? what does that mean in this context) on python.org hardware. Why exactly is that better than me doing it on one of my boxes, as long as all the necessary tools and an idea of how to do it are publically available (from the pymigr repo, for example)? > And will make the idea of splitting out the standard library and tests a > reasonable thing to do. In due time, yes. > While I really like the idea of using named branches for each release so > that there is a single py3k branch that contains all relevant history for > every release, I think we should start simple and go with cloned branches. > That way the workflow does not radically shift from what we do now for svn > to start. Once the conversion is done and people are comfortable with hg we > can then discuss moving towards a named branch approach. I don't think the cloned branches is much simpler than the named branches approach, in several ways. For example, populating the branch part of a sys.whatever value is significantly harder. Also, if you follow a useful tagging approach, doing cloned branches means that release tags aren't available on trunk/main/default. That seems like a step backwards. > Sounds reasonable to me. We can just make a list and send it to > python-committers to make final decisions of what should stick around. This list exists and has been referenced in my email a few times. > I don't use tags so I don't really care, but in the name of easy transition > I say we don't change the naming scheme (although I have no issue dropping > obscure tags). The current proposal is to clean up old tags to agree with the current naming scheme (and dropping obscure and partial tags). > Something else that can go out to python-committers before the switch. This should just be done ASAP, it helps with a smooth conversion process. > I don't think there is a single project we host -- all two of them -- that > have not said they want to convert. So I say convert everything and let's > turn off the svn server by the end of the year. I say we tackle each one as we go. I say doing a good conversion job is valuable, and we should take as much time as we need (though not more). You advocate something similar below for the Python conversion. > Can we check these scripts into the repository itself? That way there is a > chance of reuse as hg commands, e.g. ``hg pydev-ci`` as a replacement for > ``make patchcheck``. I'm not sure there's an easy way to make them into commands (although I guess you could make an extension to that effect), but hooks would be very easy. > How about hg.python.org for the official branches and we keep > code.python.org for personal branches of the developers like we have done > with the bzr experiments? I think that's just confusing. Most people seem to like hg.python.org, and it's obvious that hg goes to hg.p.o. Dividing up the namespace only makes it harder to find things. > As I have said, we should change our workflow habits after the switch and > people are comfortable with hg. Well, not everyone agrees, and although I think it doesn't matter much for the conversion itself, it may affect the branching strategy discussion. (sys.revision discussion elided because some others have already been bikeshedding on it.) Cheers, Dirkjan From dirkjan at ochtman.nl Sat Jul 4 10:05:26 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sat, 4 Jul 2009 10:05:26 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4EE510.7030409@gmail.com> References: <4A4E81B0.6010700@v.loewis.de> <4A4EB26E.3050706@gmail.com> <4A4ECACA.5080508@v.loewis.de> <4A4EE510.7030409@gmail.com> Message-ID: On Sat, Jul 4, 2009 at 07:13, Nick Coghlan wrote: > I'd guess that the only way to keep those functional is to keep > svn.python.org around in read-only mode. No, actually: the idea (I think I floated it in the PEP, as well), is that I can write a simple extension for hgweb that knows the mapping of SVN rev to hg rev and so can make hg.python.org/python/2.x/rev/r32542 come out to the changeset that resulted from converting that revision. Cheers, Dirkjan From dirkjan at ochtman.nl Sat Jul 4 10:12:04 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sat, 4 Jul 2009 10:12:04 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: On Fri, Jul 3, 2009 at 17:17, Georg Brandl wrote: > Do you have a key to the second column in that file? E.g. the difference > between "strip" and "discard" is not clear to me. "strip partial"? strip == discard. strip = remove, merged should be obvious, keep-clone means we'll keep the branch, in a separate clone, and keep-named means we'll keep the branch as a named branch in one of the other clones. > Why are there branch names starting with "../"? Because some people in the history put their branch at /python/branchname instead of /python/branches/branchname. That part will get cleaned up. [Regarding clone#branch syntax:] > * Does it work with "hg pull" etc. too, afterwards? Yes. > * Does it support more than one branch? You can only reference one branch at a time using the fragment specifier. > I would just kill all feature branches unless someone indicates it is > still used. ?There are very few active feature branches. > > (I guess in the case a branch gets killed erroneously it could still be > re-created after the conversion?) Re-creating it after the conversion could be done, I guess, but I'd prefer to get it right, the first time. As the branch map says, I'm mostly removing all the feature branches anyway. > As long as it can send diffs... It can. > That gets very ugly as soon as you start pulling from repos that just > fix a small typo or so. Okay, so we do no automatic checking. We can always write a script that walks the repository and finds all significant contributors. Cheers, Dirkjan From dirkjan at ochtman.nl Sat Jul 4 10:18:42 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sat, 4 Jul 2009 10:18:42 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E81B0.6010700@v.loewis.de> References: <4A4E81B0.6010700@v.loewis.de> Message-ID: On Sat, Jul 4, 2009 at 00:09, "Martin v. L?wis" wrote: > I would drop the roundup integration from the things that need to > be done pre-migration - there currently is no svn integration, so > not having it for hg is not a step backwards. Yeah, I mean just the linking here. > In the first sentence, you say that it can't actually work - so I > think we should drop the test. Okay. > I would like to see this well before the switch also. It could be > a patch (unified diff) stored in the tracker, or it could be an actual > branch that then needs to get merged with all active branches (IIUC). Yeah. Some help here would be welcome, too, as C is not a language I use often. Cheers, Dirkjan From dirkjan at ochtman.nl Sat Jul 4 10:20:38 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Sat, 4 Jul 2009 10:20:38 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A4E7173.1040507@v.loewis.de> <4A4E831A.6090404@v.loewis.de> Message-ID: On Sat, Jul 4, 2009 at 00:37, Barry Warsaw wrote: > Doesn't Mercurial support an Subversion bridge? ?Would it be possible to > provide a /read-only/ copy of the hg branches for 2.4 (maybe), 2.5, 2.6, and > 3.1? ?If so, then the release managers would simply have to cut their > releases from the svn copy instead of the hg master. ?/All/ other work would > be done from the hg master. ?This shouldn't be too much of a burden since > it's done so rarely and would end with the EOL of each of those branches. There is some push support in hgsubversion, but it doesn't do tags at this time, for example. I think this setup would be needlessly complicated (and tools will need to learn the new revision specifier anyway, so why not have it learn them sooner rather than later)? Cheers, Dirkjan From tjreedy at udel.edu Sat Jul 4 10:55:44 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 04 Jul 2009 04:55:44 -0400 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: Brett Cannon wrote: > I would very much like the 'k' dropped from the py3 name. It was a > funny joke when py3 was vaporware, now it is excess baggage which > only puzzles non-insiders and newcomers. > > > Is it really that confusing? I have never heard of anyone asking "what > is py3k?" Do you read python-list? It has been asked. Also, some people seem to think that py3k is different from python 3. tjr From martin at v.loewis.de Sat Jul 4 11:28:09 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Jul 2009 11:28:09 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <87prchuobf.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp> <87prchuobf.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4A4F20A9.5070200@v.loewis.de> > I see that George Brandl and Martin van Loewis seem to be accomodating > your viewpoint, but I don't get the impression that either you (as the > hg migration proponent) nor they (as core committers) realize how far > apart your assumptions are. Actually, I (probably) don't agree to a merge flow different from the one that we currently have. All I asked that *if* Dirkjan is in favor of a change, he should propose it explicitly, and fleshed-out, in the PEP, so that people can tear it down. I know that several committers are unhappy with the current merge flow (Georg in particular), so this is important to discuss. Regards, Martin From ncoghlan at gmail.com Sat Jul 4 12:36:20 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 04 Jul 2009 20:36:20 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A4E81B0.6010700@v.loewis.de> <4A4EB26E.3050706@gmail.com> <4A4ECACA.5080508@v.loewis.de> <4A4EE510.7030409@gmail.com> Message-ID: <4A4F30A4.5030405@gmail.com> Dirkjan Ochtman wrote: > On Sat, Jul 4, 2009 at 07:13, Nick Coghlan wrote: >> I'd guess that the only way to keep those functional is to keep >> svn.python.org around in read-only mode. > > No, actually: the idea (I think I floated it in the PEP, as well), is > that I can write a simple extension for hgweb that knows the mapping > of SVN rev to hg rev and so can make > hg.python.org/python/2.x/rev/r32542 come out to the changeset that > resulted from converting that revision. Ah, that's an excellent solution - good to hear it :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From p.f.moore at gmail.com Sat Jul 4 14:48:13 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 4 Jul 2009 13:48:13 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> Message-ID: <79990c6b0907040548j2266f0a4y4d0288dc0b32c1c3@mail.gmail.com> 2009/7/3 Tarek Ziad? : > You can give me a bitbucket account so I can give you write access to the repo, > There are tests as long as you install Nose. How do I get the tests to work? Just running nosetests gives an error (probably because pkgutil is being imported from the stdlib, rather than from this directory). If I set PYTHONPATH=. then I get errors. I suspect path normalisation (for backslashes) in the zipfile handling. Yes, it appears to be caused by the fact that Distribution uses os.path.join to construct self.pkg_info_path (and self.record_path) and yet zipfiles expect '/' separated pathnames. Paul. Error log: >\Apps\Python26\Scripts\nosetests-2.6.exe -v test_pkgutil.test_distributions ... ok test_pkgutil.test_distribution ... FAIL test_pkgutil.test_directory ... ok test_pkgutil.test_get_file_users ... ok test_pkgutil.test_egginfo_dirname ... ok test_pkgutil.test_zipped_directory ... ERROR test_pkgutil.test_zipped_distribution ... ERROR ====================================================================== ERROR: test_pkgutil.test_zipped_directory ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Apps\Python26\Lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Users\Gustav\Data\pep376\test_pkgutil.py", line 80, in test_zipped_directory dir = ZippedDistributionDir(SITE_PKG+'.zip') File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 324, in __init__ self.add(ZippedDistribution(self._zip_file, paths[0])) File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 259, in __init__ super(ZippedDistribution, self).__init__(path) File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 154, in __init__ pkginfo = self._open_pkginfo() File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 264, in _open_pkginfo return self.zipfile.open(self.pkg_info_path) File "C:\Apps\Python26\lib\zipfile.py", line 859, in open zinfo = self.getinfo(name) File "C:\Apps\Python26\lib\zipfile.py", line 826, in getinfo 'There is no item named %r in the archive' % name) KeyError: "There is no item named 'mercurial-1.0.1.egg-info\\\\PKG-INFO' in the archive" ====================================================================== ERROR: test_pkgutil.test_zipped_distribution ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Apps\Python26\Lib\site-packages\nose\case.py", line 253, in setUp try_run(self.test, names) File "C:\Apps\Python26\Lib\site-packages\nose\util.py", line 487, in try_run return func() File "C:\Users\Gustav\Data\pep376\test_pkgutil.py", line 90, in setup_zip pkgutil._dist_dirs.load(SITE_PKG+'.zip') File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 380, in load dist_dir = ZippedDistributionDir(path) File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 324, in __init__ self.add(ZippedDistribution(self._zip_file, paths[0])) File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 259, in __init__ super(ZippedDistribution, self).__init__(path) File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 154, in __init__ pkginfo = self._open_pkginfo() File "C:\Users\Gustav\Data\pep376\pkgutil.py", line 264, in _open_pkginfo return self.zipfile.open(self.pkg_info_path) File "C:\Apps\Python26\lib\zipfile.py", line 859, in open zinfo = self.getinfo(name) File "C:\Apps\Python26\lib\zipfile.py", line 826, in getinfo 'There is no item named %r in the archive' % name) KeyError: "There is no item named 'mercurial-1.0.1.egg-info\\\\PKG-INFO' in the archive" ====================================================================== FAIL: test_pkgutil.test_distribution ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Apps\Python26\Lib\site-packages\nose\case.py", line 183, in runTest self.test(*self.arg) File "C:\Users\Gustav\Data\pep376\test_pkgutil.py", line 60, in test_distribution os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/RECORD')]) AssertionError: ['C:\\Users\\Gustav\\Data\\pep376\\site-packages\\mercurial-1.0.1.egg-info\\PKG_INFO', 'C:\\Users\\ Gustav\\Data\\pep376\\site-packages\\mercurial-1.0.1.egg-info\\RECORD'] != ['C:\\Users\\Gustav\\Data\\pep376\\site- packages\\mercurial-1.0.1.egg-info/PKG_INFO', 'C:\\Users\\Gustav\\Data\\pep376\\site-packages\\mercurial-1.0.1.egg- info/RECORD'] ---------------------------------------------------------------------- Ran 7 tests in 0.019s FAILED (errors=2, failures=1) 13:40 C:\Users\Gustav\Data\pep376 >unzip -l site-packages.zip Archive: site-packages.zip Length EAs ACLs Date Time Name -------- --- ---- ---- ---- ---- 2531 0 0 19/06/09 17:14 mercurial/filelog.py 3449 0 0 19/06/09 17:14 mercurial/filelog.pyc 234 0 0 19/06/09 17:14 mercurial-1.0.1.egg-info/PKG-INFO 162 0 0 19/06/09 17:14 mercurial-1.0.1.egg-info/RECORD 0 0 0 19/06/09 17:14 processing/__init__.py 3482 0 0 19/06/09 17:14 processing-0.52.egg-info/PKG-INFO 165 0 0 19/06/09 17:14 processing-0.52.egg-info/RECORD -------- ----- ----- ------- 10023 0 0 7 files Paul. From p.f.moore at gmail.com Sat Jul 4 15:04:12 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 4 Jul 2009 14:04:12 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> Message-ID: <79990c6b0907040604p1f40aaffv150a037c4f1defa5@mail.gmail.com> 2009/7/3 Tarek Ziad? : > 2009/7/3 Paul Moore : >> Does this sound sensible? Tarek, would you be OK with me attempting to >> modify your prototype to support this protocol? Are there any tests >> for PEP 376, so that I can confirm I haven't completely broken >> something? If I can, I'll knock up some simple prototype importers for >> non-standard examples, and see how they work with all this. > > Yes that's exactly what I was thinking about after the discussion we > had in the other thread. This change would allow much more flexibility. One important note - I plan on using the fact that DistributionDirMap is not public, and hacking it about drastically, or possibly even removing it. (After all, the likes of the load method don't make sense when you've got sys.meta_path, sys.path_importer_cache and the like to consider). If the loss of any of the "internal" classes is an issue, say so now! Paul. From p.f.moore at gmail.com Sat Jul 4 16:31:40 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 4 Jul 2009 15:31:40 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907040548j2266f0a4y4d0288dc0b32c1c3@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <79990c6b0907040548j2266f0a4y4d0288dc0b32c1c3@mail.gmail.com> Message-ID: <79990c6b0907040731l255d2df1kf1ac63cb069621e5@mail.gmail.com> 2009/7/4 Paul Moore : > 2009/7/3 Tarek Ziad? : >> You can give me a bitbucket account so I can give you write access to the repo, >> There are tests as long as you install Nose. > > How do I get the tests to work? Just running nosetests gives an error > (probably because pkgutil is being imported from the stdlib, rather > than from this directory). > > If I set PYTHONPATH=. then I get errors. I suspect path normalisation > (for backslashes) in the zipfile handling. Actually, the test assert_equals(list(dist.get_egginfo_files(local=True)), [os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/PKG_INFO'), os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/RECORD')]) is broken, because the expected value uses slashes, which are *not* the local separator on win32. I've attached a patch. But there's 2 comments I'd make (one minor, one major) Minor one: The tests often seem to be exercising the internal classes, not so much the public API, so many of them will probably not be of much use to me :-( Major one: I'm not entirely sure what the purpose of the "local" flag is. With local=True, you are supposed to get a filesystem path in the local form. But that makes no sense for non-filesystem based loaders (e.g. zipfiles). So how is a user expected to use the value? With local=false, you get slash-separated files (which is good, because it's consistent) which are either absolute (see my earlier comments about how broken this is in practice) or relative to the "directory" containing the egg-info file (where this is of course not necessarily a real filesystem location). But these aren't useful as filesystem names, because they aren't local format. So what is the point of the get_installed_files, uses, and get_egginfo_files functions? Who will use the returned values, and what for? Where will a user get a value to pass to uses()? Given that get_egginfo_file takes names relative to the egginfo dir, shouldn't get_egginfo_files *return* values relative to the same location, so that they can be used there? I guess the same type of argument applies with get_inetalled_files() and uses() - the values returned by get_installed_files can be seen as a set of opaque tokens (OK, realistically they will be "filenames" in some sense) which can be passed to uses() - but if they aren't absolute filesystem paths, you can't compare them. Example: distribution a contains file p.py distribution b contains file p.py pkgutil.get_distribution(a).get_installed_files contains 'p.py' pkgutil.get_distribution(b).uses('p.py') is True What does this mean? If the 2 distributions are in the same sys.path element, there's a clash. If not, is there a clash? What if they are in 2 different directories? What if they are in the same directory, but it's repeated under different names (e.g. symlinks) on sys.path? What if one is in a directory and the other in a zip file? This isn't even related to non-filesystem loaders. It's a fundamental issue with the whole API. What you seem to want is actually a canonical "name" for a file object - for filesystem files, os.path.normcase(os.path.normpath(filename)) is probably good enough, although it doesn't deal with symlinks. For other types of loader, you have to rely on the loader itself - loader.get_fullname() is probably OK. But even then, it's not clear how user code would actually *use* these APIs. I think you need some real-world use cases, with actual sample (pseudo-)code, to validate the design here. As things stand, it's both confusing and (I suspect) unusable in practice. Sorry, I know that sounds negative, but if this isn't to be a source of subtle bugs for years to come, it needs to be clarified now. PEP 302 is still hitting this type of issue - runpy and importlib have brought out errors and holes in the protocol quite recently - even though Just and I went to great lengths to try to tease out hidden assumptions up front. Concrete proposal: get_metadata_files() - returns slash-separated names, relative to the egginfo dir get_metadata_file(path) - path must be slash-separated, relative to the egginfo dir get_installed_files - returns the contents of RECORD unaltered uses(path) - checks if path is in RECORD The latter 2 are not very useful in practice - you can't say anything about entries in different RECORD files, which is likely the real use case you want. Maybe RECORD could have an extra "Location" entry, which determines where it exists globally (this would be the directory to which the filenames were relative, in the case of filesystem-based distributions) and RECORD entries are comparable if the Location values in the 2 RECORD files match. That's a lot more complex - but depending on what use people expect to make of these 2 APIs, it may be justified. Paul. -------------- next part -------------- A non-text attachment was scrubbed... Name: test-win32.patch Type: text/x-patch Size: 666 bytes Desc: not available URL: From brett at python.org Sat Jul 4 21:28:31 2009 From: brett at python.org (Brett Cannon) Date: Sat, 4 Jul 2009 12:28:31 -0700 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: On Sat, Jul 4, 2009 at 01:03, Dirkjan Ochtman wrote: > On Fri, Jul 3, 2009 at 20:04, Brett Cannon wrote: > > Fine by me as long as people realize that if anything is questionable > then > > the switch will not happen. Getting this right takes precedence over any > > deadline. And obviously we will need to do at least one live conversion > on > > python.org hardware to make sure everything will work smoothly. > > I'm not sure I see the need to do a (live? what does that mean in this > context) on python.org hardware. "Live" as in run through all the steps on python.org hardware w/o hitting any final switch that turns off svn. > Why exactly is that better than me > doing it on one of my boxes, as long as all the necessary tools and an > idea of how to do it are publically available (from the pymigr repo, > for example)? Because there are different OSs, installed software, etc. Basically because you just never know. =) Plus it will make Martin sleep better. > > > > And will make the idea of splitting out the standard library and tests a > > reasonable thing to do. > > In due time, yes. > > > While I really like the idea of using named branches for each release so > > that there is a single py3k branch that contains all relevant history for > > every release, I think we should start simple and go with cloned > branches. > > That way the workflow does not radically shift from what we do now for > svn > > to start. Once the conversion is done and people are comfortable with hg > we > > can then discuss moving towards a named branch approach. > > I don't think the cloned branches is much simpler than the named > branches approach, in several ways. For example, populating the branch > part of a sys.whatever value is significantly harder. Also, if you > follow a useful tagging approach, doing cloned branches means that > release tags aren't available on trunk/main/default. That seems like a > step backwards. > I personally prefer named branches, but that's just me and I am not about to force my preferences on everyone. Guess we just have to see if others have opinions against named branches. > > > Sounds reasonable to me. We can just make a list and send it to > > python-committers to make final decisions of what should stick around. > > This list exists and has been referenced in my email a few times. > Sure, but as inlining the PEP in this email thread has shown, not making people click a link helps. =) Plus a separate email makes it very obvious that people need to check their email instead of making it a bullet point in a much larger email. > > > I don't use tags so I don't really care, but in the name of easy > transition > > I say we don't change the naming scheme (although I have no issue > dropping > > obscure tags). > > The current proposal is to clean up old tags to agree with the current > naming scheme (and dropping obscure and partial tags). > Fine by me. > > > Something else that can go out to python-committers before the switch. > > This should just be done ASAP, it helps with a smooth conversion process. > > > I don't think there is a single project we host -- all two of them -- > that > > have not said they want to convert. So I say convert everything and let's > > turn off the svn server by the end of the year. > > I say we tackle each one as we go. I say doing a good conversion job > is valuable, and we should take as much time as we need (though not > more). You advocate something similar below for the Python conversion. > I am not suggesting we do all conversions on the same day, just that everything should eventually be converted. > > > Can we check these scripts into the repository itself? That way there is > a > > chance of reuse as hg commands, e.g. ``hg pydev-ci`` as a replacement for > > ``make patchcheck``. > > I'm not sure there's an easy way to make them into commands (although > I guess you could make an extension to that effect), but hooks would > be very easy. > OK. I was just hoping we could factor the code in such a way as to share the basic steps the hooks were checking so as to reuse them in a command. > > > How about hg.python.org for the official branches and we keep > > code.python.org for personal branches of the developers like we have > done > > with the bzr experiments? > > I think that's just confusing. Most people seem to like hg.python.org, > and it's obvious that hg goes to hg.p.o. Dividing up the namespace > only makes it harder to find things. > Yeah, I realize I have lost this battle. > > > As I have said, we should change our workflow habits after the switch and > > people are comfortable with hg. > > Well, not everyone agrees, and although I think it doesn't matter much > for the conversion itself, it may affect the branching strategy > discussion. > Sure, to an extent. > > (sys.revision discussion elided because some others have already been > bikeshedding on it.) I think it has been answered; sys.subversion goes away and sys.mercurial or sys.mercurial_revision comes into existence. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jul 4 21:29:45 2009 From: brett at python.org (Brett Cannon) Date: Sat, 4 Jul 2009 12:29:45 -0700 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: On Sat, Jul 4, 2009 at 01:55, Terry Reedy wrote: > Brett Cannon wrote: > > I would very much like the 'k' dropped from the py3 name. It was a >> funny joke when py3 was vaporware, now it is excess baggage which >> only puzzles non-insiders and newcomers. >> >> >> Is it really that confusing? I have never heard of anyone asking "what is >> py3k?" >> > > Do you read python-list? No as it would take up so much of my Python time I wouldn't be able to code anymore. > It has been asked. Also, some people seem to think that py3k is different > from python 3. > Well, I still would not like to lose the py3k label. But if we do I still say 2.x/3.x instead of py2/py3. -Brett > > tjr > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Jul 4 21:54:58 2009 From: brett at python.org (Brett Cannon) Date: Sat, 4 Jul 2009 12:54:58 -0700 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <4A4EA689.1010704@gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> Message-ID: On Fri, Jul 3, 2009 at 17:47, Nick Coghlan wrote: > Tarek Ziad? wrote: > > 2009/7/3 Paul Moore : > >> Does this sound sensible? Tarek, would you be OK with me attempting to > >> modify your prototype to support this protocol? Are there any tests > >> for PEP 376, so that I can confirm I haven't completely broken > >> something? If I can, I'll knock up some simple prototype importers for > >> non-standard examples, and see how they work with all this. > > > > Yes that's exactly what I was thinking about after the discussion we > > had in the other > > thread. This change would allow much more flexibility. > > It's also very similar to the ideas that started to tick over in the > back of my head after the previous discussion (only far more fleshed out > that anything I got to - fortunately I read Paul's message before > spending too much time on it!) > > Specifically regarding this comment from Paul: > > > Finder.uninstall(distribution_name, filter=callable, installer=name) > > Uninstall the given distribution. It's likely that many finders will > > be read-only. In that case, this function should return None. > > Otherwise, return a list of the "files" removed. (This may need some > > clarification, as many finders won't have a concept of a "file name"). > > PEP 302 finders and loaders actually *are* expected to have a concept of > a "file name". For Python modules they they are expected to have it so > they can populate __file__ correctly Not only that, lots of code out there expects values from __path__ to contain os.path.sep on top of __file__, so the idea of file paths is already hard-coded into how import works, whether we like it or not. > (runpy is also a lot happier with > them when they expose get_filename(), a relatively recent addition to > the PEP 302 API). An addition I was not even aware of. Looks like importlib needs a little updating. > For other resources they're expected to have it so the > get_data() API can work correctly. > > One other thing to consider: it may be desirable to have hooks for > meta_path and path_hooks in distutils that are checked *before* the full > import hooks in the sys module. The reason I say this is that it would > allow someone to override just the metadata retrieval (e.g. to pick up > the extra dependency information saved by setuptools and friends) > without having to worry about breaking the actual imports from those > locations. So as the guy who plans to try to clean up import, I just want to say that implicit finders will eventually go away. Do not start off making them implicit, but you can have them installed on sys.meta_path or sys.path_hooks by default. > > > Cheers, > Nick. > > P.S. +lots on using 'metadata' in the PEP 376 method names rather than > the jargon 'egginfo'. Jargon isn't always bad, but using it seems fairly > gratuitous in this case. Ditto from here. Plus I have an aversion to terminology that goes down the reptile route instead of the Monty Python route. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Sat Jul 4 23:05:56 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 4 Jul 2009 17:05:56 -0400 (EDT) Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: On Sat, 4 Jul 2009 at 12:28, Brett Cannon wrote: > On Sat, Jul 4, 2009 at 01:03, Dirkjan Ochtman wrote: > >> On Fri, Jul 3, 2009 at 20:04, Brett Cannon wrote: >>> Fine by me as long as people realize that if anything is questionable >> then >>> the switch will not happen. Getting this right takes precedence over any >>> deadline. And obviously we will need to do at least one live conversion >> on >>> python.org hardware to make sure everything will work smoothly. >> >> I'm not sure I see the need to do a (live? what does that mean in this >> context) on python.org hardware. > > > "Live" as in run through all the steps on python.org hardware w/o hitting > any final switch that turns off svn. "Live" should also (and I think this is critical) include a period of at least a week where we tell python-dev/python-committers "hg.python.org looks like it is going to look at cutover, please try out the workflow". The idea would be to have committers actually exercise the workflow on their platform for at least one patch, including whatever replaces the svnmerge step. I can almost guarantee we will find some issues that need to be resolved, and that people won't try it out thoroughly until it is "live". --David From p.f.moore at gmail.com Sun Jul 5 00:13:14 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 4 Jul 2009 23:13:14 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> Message-ID: <79990c6b0907041513h6b08e6ccse07b4bd6c7a62301@mail.gmail.com> 2009/7/4 Brett Cannon : >> > Finder.uninstall(distribution_name, filter=callable, installer=name) >> > Uninstall the given distribution. It's likely that many finders will >> > be read-only. In that case, this function should return None. >> > Otherwise, return a list of the "files" removed. (This may need some >> > clarification, as many finders won't have a concept of a "file name"). >> >> PEP 302 finders and loaders actually *are* expected to have a concept of >> a "file name". For Python modules they they are expected to have it so >> they can populate __file__ correctly > > Not only that, lots of code out there expects values from __path__ to > contain os.path.sep on top of __file__, so the idea of file paths is already > hard-coded into how import works, whether we like it or not. Yes, consider me duly reprimanded :-) They aren't *file* names, but they still have to look similar enough to satisfy code that uses things like filename manipulation functions on them. >> (runpy is also a lot happier with >> them when they expose get_filename(), a relatively recent addition to >> the PEP 302 API). > > An addition I was not even aware of. Looks like importlib needs a little > updating. Yes, I didn't notice that sneak into PEP 302, either :-) >> For other resources they're expected to have it so the >> get_data() API can work correctly. >> >> One other thing to consider: it may be desirable to have hooks for >> meta_path and path_hooks in distutils that are checked *before* the full >> import hooks in the sys module. The reason I say this is that it would >> allow someone to override just the metadata retrieval (e.g. to pick up >> the extra dependency information saved by setuptools and friends) >> without having to worry about breaking the actual imports from those >> locations. > > So as the guy who plans to try to clean up import, I just want to say that > implicit finders will eventually go away. Do not start off making them > implicit, but you can have them installed on sys.meta_path or sys.path_hooks > by default. I'm against having distutils-specific hooks. I don't see a genuine need, and it adds complexity so let's wait for a proper use case before adding that. With some suitably ugly hacking, you could probably add a finder to sys.meta_path which never actually found a module, but which reported one or more distributions for the distutils metadata APIs. Playing around with that idea could likely give you any flexibility you needed (at the hopefully minor cost of an extra dummy meta_path call on each import). As far as the currently implicit filesystem path handling is concerned, I'm aware that the intention is for importlib to move these into an explicit importer at some point. For my prototype changes to Tarek's prototype, I'm creating a dummy filesystem finder with just the new methods needed for the distribution metadata. It will be easy to move these into a real filesystem finder when one emerges. Paul. From ncoghlan at gmail.com Sun Jul 5 00:42:49 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 05 Jul 2009 08:42:49 +1000 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907041513h6b08e6ccse07b4bd6c7a62301@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <79990c6b0907041513h6b08e6ccse07b4bd6c7a62301@mail.gmail.com> Message-ID: <4A4FDAE9.20205@gmail.com> Paul Moore wrote: > 2009/7/4 Brett Cannon : >>> (runpy is also a lot happier with >>> them when they expose get_filename(), a relatively recent addition to >>> the PEP 302 API). >> An addition I was not even aware of. Looks like importlib needs a little >> updating. > > Yes, I didn't notice that sneak into PEP 302, either :-) I probably should have been noisier about that when I added it. I'm pretty sure it did come up on this list, but it would have been somewhere in the middle of a runpy discussion: runpy was stuck because it only uses PEP 302 to *find* the modules it needs, but not to actually load them. In the original version of PEP 302 the only way to get a loader to tell you the filename was to load the module and see what it set __file__ to, which wasn't useful in the runpy case. If I recall correctly, at the time when PJE was rationalising the code duplication between runpy and pkglib the comment was also made that runpy's get_filename() optional loader extension should be mentioned in PEP 302. runpy still works for loaders that don't provide it, it just can't set __file__ or sys.argv[0] correctly in those cases. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From brett at python.org Sun Jul 5 00:56:02 2009 From: brett at python.org (Brett Cannon) Date: Sat, 4 Jul 2009 15:56:02 -0700 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <4A4FDAE9.20205@gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <79990c6b0907041513h6b08e6ccse07b4bd6c7a62301@mail.gmail.com> <4A4FDAE9.20205@gmail.com> Message-ID: I'm not complaining, just saying I didn't notice. I essentially did the same with get_source/get_bytecode so as to be able to set __file__. When I get around to it I will create a RunpyLoader in importlib.abc and add the method in the proper machinery loaders. On Jul 4, 2009 3:43 PM, "Nick Coghlan" wrote: Paul Moore wrote: > 2009/7/4 Brett Cannon : >>> (runpy is also a lot happier with >>> them when they expose get_filename(), a relatively recent ... I probably should have been noisier about that when I added it. I'm pretty sure it did come up on this list, but it would have been somewhere in the middle of a runpy discussion: runpy was stuck because it only uses PEP 302 to *find* the modules it needs, but not to actually load them. In the original version of PEP 302 the only way to get a loader to tell you the filename was to load the module and see what it set __file__ to, which wasn't useful in the runpy case. If I recall correctly, at the time when PJE was rationalising the code duplication between runpy and pkglib the comment was also made that runpy's get_filename() optional loader extension should be mentioned in PEP 302. runpy still works for loaders that don't provide it, it just can't set __file__ or sys.argv[0] correctly in those cases. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Sun Jul 5 13:10:14 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 05 Jul 2009 13:10:14 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E7DA7.2030402@v.loewis.de> References: <4A4E7173.1040507@v.loewis.de> <4A4E7DA7.2030402@v.loewis.de> Message-ID: Martin v. L?wis schrieb: >> We could add another value in the tuple that specifies the VCS: >> ('CPython', 'branches/release25-maint', '61464', 'svn'). I agree that >> VCSs are not universally the same, but the concept of a revision is >> universal. > > Actually, I think that's not the case. For bzr, the usual way of > identifying a revision is by revision number, which, however, is not > unique within a project, as each branch will use contiguous integers > for numbers. There are also unique identifications - so a bzr revision > has actually two numbers. > > More general, in a DVCS, it is not possible to access the revision being > referred to by such a tuple. For sys.subversion, if [0]=='CPython', then > you could go to svn.python.org. For a DVCS, the revision being > identified may not be publically available, or may not live on a host > that you can infer from your proposed sys.revision. At least you can tell that if the given hash is not present in the mainline repo, the build contains something that doesn't come from python.org. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Sun Jul 5 13:17:31 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 05 Jul 2009 13:17:31 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4E6EF5.4000901@v.loewis.de> References: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp> <4A4E6EF5.4000901@v.loewis.de> Message-ID: Martin v. L?wis schrieb: >> It will handle it, for sure, but I think it would all go easier if we >> could work with stricter subset branches (and leave the effective >> cherrypicking for the occasional problem). > > So I think the PEP should propose a workflow (or: merge flow) if you > think we would be better off with a different one. > > In proposing such a workflow, consider these requirements: > - we current have four active "maintenance" branches (i.e. where > the entire code basis evolves): trunk, 3k, 2.6, and 3.1 (3.0 > also until this morning). It seems that there is a consensus to separate the 2.x and 3.x repos, and that also makes sense to me. Using named branches for the maintenance branches should be possible, but I'm not opposed to using cloned branches either. What I really want to see is the common-subset approach for maintenance branches. Changesets still have to be transplanted from 2.x to 3.x or the other way round. > - in addition, we have two security branches currently: 2.4 and > 2.5, although 2.4 will be closed soon. > - our committers consistently refuse to merge changes across > branches themselves, and likely continue to do so unless there > is some feature of hg that I missed (e.g. one were merging > would happen without any user specifically asking for it) If the checkin is done in the proper (the maint) branch, at least merging of that change is automatic whenever someone does a hg merge. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From ziade.tarek at gmail.com Sun Jul 5 15:02:49 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 5 Jul 2009 15:02:49 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907040731l255d2df1kf1ac63cb069621e5@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <79990c6b0907040548j2266f0a4y4d0288dc0b32c1c3@mail.gmail.com> <79990c6b0907040731l255d2df1kf1ac63cb069621e5@mail.gmail.com> Message-ID: <94bdd2610907050602y2a56fe5cofddf40c80ce221e1@mail.gmail.com> 2009/7/4 Paul Moore : > 2009/7/4 Paul Moore : >> 2009/7/3 Tarek Ziad? : >>> You can give me a bitbucket account so I can give you write access to the repo, >>> There are tests as long as you install Nose. >> >> How do I get the tests to work? Just running nosetests gives an error >> (probably because pkgutil is being imported from the stdlib, rather >> than from this directory). >> I just run them from within the directory >> If I set PYTHONPATH=. then I get errors. I suspect path normalisation >> (for backslashes) in the zipfile handling. > > Actually, the test > > ? ?assert_equals(list(dist.get_egginfo_files(local=True)), > ? ? ? ? ? ? ? ? ?[os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/PKG_INFO'), > ? ? ? ? ? ? ? ? ? os.path.join(SITE_PKG, 'mercurial-1.0.1.egg-info/RECORD')]) > > is broken, because the expected value uses slashes, which are *not* > the local separator on win32. > > I've attached a patch. Applied, thanks (I didn't run them under win32 yet) > > But there's 2 comments I'd make (one minor, one major) > > Minor one: The tests often seem to be exercising the internal classes, > not so much the public API, so many of them will probably not be of > much use to me :-( I'll add some more tests then, or even user stories. > I think you need some real-world use cases, with actual sample > (pseudo-)code, to validate the design here. As things stand, it's both > confusing and (I suspect) unusable in practice. Sorry, I know that > sounds negative, but if this isn't to be a source of subtle bugs for > years to come, it needs to be clarified now. PEP 302 is still hitting > this type of issue - runpy and importlib have brought out errors and > holes in the protocol quite recently - even though Just and I went to > great lengths to try to tease out hidden assumptions up front. Agreed, the zip case was added afterwards, but in practice, the APIs are still dealing with the files are *filesystem files* located in a container (eg a directory or a zip file) located somewhere on the filesystem. "local" in that case is a flag that means "translate a file path expressed in the local filesystem" which make no sense anymore with zip files. But the goal really, is to be able to point out that two distributions are using the very same file. Right now PEP 376 and the prototype code handle these two real world use cases: - browsing regular site-packages-like directories - browsing site-packages-like directories, that are zipped. For example: - I have a "packages.zip" file in /var/, wich is also in my sys.path. It contains a distribution "foo-1.0" that has the "roman.py" file in its root. So the RECORD file located in "foo-1.0.egg-info" has a line starting with "roman.py,..." - Then if I install docutils 0.5 as a regular filesystem distribution, "roman.py" will be added in Python's site-packages. and docutils-0.5.egg-info/RECORD will contain "roman.py,..." with the same hash. The local flag will return these paths: - /var/packages.zip/roman.py <--- not a "real" path - /usr/local/lib/python2.6/site-packages/roman.py So removing the docutils distribution will be doable, because these paths are different. > > Concrete proposal: > > get_metadata_files() - returns slash-separated names, relative to the > egginfo dir > get_metadata_file(path) - path must be slash-separated, relative to > the egginfo dir > > get_installed_files - returns the contents of RECORD unaltered > uses(path) - checks if path is in RECORD > > The latter 2 are not very useful in practice - you can't say anything > about entries in different RECORD files, which is likely the real use > case you want. Maybe RECORD could have an extra "Location" entry, > which determines where it exists globally (this would be the directory > to which the filenames were relative, in the case of filesystem-based > distributions) and RECORD entries are comparable if the Location > values in the 2 RECORD files match. That's a lot more complex - but > depending on what use people expect to make of these 2 APIs, it may be > justified. Yes, In practice, if you look at my previous example, even if "/var/packages.zip/roman.py" isn't a real path, it's enough to compare RECORD entries globally. The "Location" entry you are proposing in that case, would be "/var/packages.zip". But do we really need to store it the RECORD ? Or can't we define an API that returns two elements : - the path to the location (in the example: /var/packages.zip or /usr/local/lib/python2.6/site-packages) - the path within the location itself (in the example: roman.py) A concrete proposal would be to take back your proposal, but return tuples with the location as the first member. e.g. "(location, relative path[s])" The code that is comparing paths to see if they are the same can join location+relative path[s], while we can provide in a dedicated function something to read the content of the file (that would be get_data I guess, if I refer to PEP 302) Tarek From ziade.tarek at gmail.com Sun Jul 5 15:05:10 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 5 Jul 2009 15:05:10 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907040604p1f40aaffv150a037c4f1defa5@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <79990c6b0907040604p1f40aaffv150a037c4f1defa5@mail.gmail.com> Message-ID: <94bdd2610907050605v522070eduda2fb1957a27264@mail.gmail.com> On Sat, Jul 4, 2009 at 3:04 PM, Paul Moore wrote: > 2009/7/3 Tarek Ziad? : >> 2009/7/3 Paul Moore : >>> Does this sound sensible? Tarek, would you be OK with me attempting to >>> modify your prototype to support this protocol? Are there any tests >>> for PEP 376, so that I can confirm I haven't completely broken >>> something? If I can, I'll knock up some simple prototype importers for >>> non-standard examples, and see how they work with all this. >> >> Yes that's exactly what I was thinking about after the discussion we >> had in the other thread. This change would allow much more flexibility. > > One important note - I plan on using the fact that DistributionDirMap > is not public, and hacking it about drastically, or possibly even > removing it. (After all, the likes of the load method don't make sense > when you've got sys.meta_path, sys.path_importer_cache and the like to > consider). > > If the loss of any of the "internal" classes is an issue, say so now! Go for it please, and let me know if you set a bitbucket account, so you can push your commits in there directly > > Paul. > -- Tarek Ziad? | http://ziade.org From p.f.moore at gmail.com Sun Jul 5 15:13:02 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Jul 2009 14:13:02 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <94bdd2610907050605v522070eduda2fb1957a27264@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <79990c6b0907040604p1f40aaffv150a037c4f1defa5@mail.gmail.com> <94bdd2610907050605v522070eduda2fb1957a27264@mail.gmail.com> Message-ID: <79990c6b0907050613g316a45e7ye00f0775b5c5299c@mail.gmail.com> 2009/7/5 Tarek Ziad? : > Go for it please, and let me know if you set a bitbucket account, so > you can push your commits in there directly My bitbucket account is 'pmoore'. Paul. From ziade.tarek at gmail.com Sun Jul 5 15:13:13 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 5 Jul 2009 15:13:13 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> Message-ID: <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> 2009/7/4 Brett Cannon : >> >> P.S. +lots on using 'metadata' in the PEP 376 method names rather than >> the jargon 'egginfo'. Jargon isn't always bad, but using it seems fairly >> gratuitous in this case. > > Ditto from here. Plus I have an aversion to terminology that goes down the > reptile route instead of the Monty Python route. If it turns out that we use PEP 302-like loaders, I am also suggesting that the default metadata directory name used in Distutils is changed to "DIST_NAME.metadata". The loader would still work with "DIST_NAME.egg-info" directories for compatibility with existing format in the query APIs, but the Distutils install command would rather create "DIST_NAME.metadata" From ziade.tarek at gmail.com Sun Jul 5 15:14:04 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 5 Jul 2009 15:14:04 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907050613g316a45e7ye00f0775b5c5299c@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <79990c6b0907040604p1f40aaffv150a037c4f1defa5@mail.gmail.com> <94bdd2610907050605v522070eduda2fb1957a27264@mail.gmail.com> <79990c6b0907050613g316a45e7ye00f0775b5c5299c@mail.gmail.com> Message-ID: <94bdd2610907050614r568c8ca9qa681b03e3a4fcb8b@mail.gmail.com> On Sun, Jul 5, 2009 at 3:13 PM, Paul Moore wrote: > 2009/7/5 Tarek Ziad? : >> Go for it please, and let me know if you set a bitbucket account, so >> you can push your commits in there directly > > My bitbucket account is 'pmoore'. You're set. From p.f.moore at gmail.com Sun Jul 5 15:27:31 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Jul 2009 14:27:31 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <94bdd2610907050602y2a56fe5cofddf40c80ce221e1@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <79990c6b0907040548j2266f0a4y4d0288dc0b32c1c3@mail.gmail.com> <79990c6b0907040731l255d2df1kf1ac63cb069621e5@mail.gmail.com> <94bdd2610907050602y2a56fe5cofddf40c80ce221e1@mail.gmail.com> Message-ID: <79990c6b0907050627q4afcf0e1u54aaa363e2a5e3ee@mail.gmail.com> 2009/7/5 Tarek Ziad? : > Agreed, the zip case was added afterwards, but in practice, the APIs are still > dealing with the files are *filesystem files* located in a container (eg a directory > or a zip file) located somewhere on the filesystem. > > "local" in that case is a flag that means "translate a file path expressed in the > local filesystem" which make no sense anymore with zip files. But the goal really, > is to be able to point out that two distributions are using the very same file. > > Right now PEP 376 and the prototype code handle these two real world use cases: > > - browsing regular site-packages-like directories > - browsing site-packages-like directories, that are zipped. > > For example: > > - I have a ?"packages.zip" file in /var/, wich is also in my sys.path. > It contains a distribution "foo-1.0" that has the "roman.py" file in > its root. ?So the RECORD file located in "foo-1.0.egg-info" has a line > starting with "roman.py,..." > > - Then if I install docutils 0.5 as a regular filesystem distribution, > "roman.py" will be added in Python's site-packages. > ?and docutils-0.5.egg-info/RECORD will contain "roman.py,..." with > the same hash. > > The local flag will return these paths: > > - /var/packages.zip/roman.py ? <--- not a "real" path > - /usr/local/lib/python2.6/site-packages/roman.py > > So removing the docutils distribution will be doable, because these > paths are different. > >> >> Concrete proposal: >> >> get_metadata_files() - returns slash-separated names, relative to the >> egginfo dir >> get_metadata_file(path) - path must be slash-separated, relative to >> the egginfo dir >> >> get_installed_files - returns the contents of RECORD unaltered >> uses(path) - checks if path is in RECORD >> >> The latter 2 are not very useful in practice - you can't say anything >> about entries in different RECORD files, which is likely the real use >> case you want. Maybe RECORD could have an extra "Location" entry, >> which determines where it exists globally (this would be the directory >> to which the filenames were relative, in the case of filesystem-based >> distributions) and RECORD entries are comparable if the Location >> values in the 2 RECORD files match. That's a lot more complex - but >> depending on what use people expect to make of these 2 APIs, it may be >> justified. > > Yes, > In practice, if you look at my previous example, even if > "/var/packages.zip/roman.py" isn't a > real path, it's enough to compare RECORD entries globally. > > The "Location" entry you are proposing in that case, would be > "/var/packages.zip". > > But do we really need to store it the RECORD ?? Or can't we define an > API that returns > two elements : > > - the path to the location (in the example: /var/packages.zip or > /usr/local/lib/python2.6/site-packages) > - the path within the location itself (in the example: roman.py) > > A concrete proposal would be to take back your proposal, but return > tuples with the location as the first member. > e.g. "(location, relative path[s])" That sounds reasonable. So we can forget the "local" parameter, and return a tuple: - absolute location of the container (directory, zipfile or whatever containing the egginfo file) as a filesystem path in canonical native form (where it's filesystem based) or as an opaque token for the odd cases (frozen modules, for example) where a filesystem location isn't available. - entry from the RECORD file, as a slash-separated filename relative to the root of the container. > The code that is comparing paths to see if they are the same can join > location+relative path[s], while we can provide in a dedicated function > something to read the content of the file (that would be get_data I guess, > if I refer to PEP 302) Unfortunately, get_data loads data files located within a *package*, using a name relative to the package directory. You can't get at the metadata of a *distribution* like that. But if you're using get_installed_files(), why would you then want to read the files? What exactly would you *use* get_installed_files for which would then leave you needing to read the files? If it's to check they haven't changed (by comparing md5 values) you're doing that to uninstall, so that's the responsibility of the uninstall function. Again, it's a question of what is a public API, and what is the use case it's designed for. I'm currently writing a SQLite importer, which will allow me to store "files" in any sort of database tables I want, so I can build in some nice pathological behaviour. That should tease out some awkward corner cases :-) Paul From ziade.tarek at gmail.com Sun Jul 5 15:48:50 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 5 Jul 2009 15:48:50 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907050627q4afcf0e1u54aaa363e2a5e3ee@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <79990c6b0907040548j2266f0a4y4d0288dc0b32c1c3@mail.gmail.com> <79990c6b0907040731l255d2df1kf1ac63cb069621e5@mail.gmail.com> <94bdd2610907050602y2a56fe5cofddf40c80ce221e1@mail.gmail.com> <79990c6b0907050627q4afcf0e1u54aaa363e2a5e3ee@mail.gmail.com> Message-ID: <94bdd2610907050648sd5335aqaa3466b5fb9d05cc@mail.gmail.com> On Sun, Jul 5, 2009 at 3:27 PM, Paul Moore wrote: >> >> A concrete proposal would be to take back your proposal, but return >> tuples with the location as the first member. >> e.g. "(location, relative path[s])" > > That sounds reasonable. So we can forget the "local" parameter, and > return a tuple: > > - absolute location of the container (directory, zipfile or whatever > containing the egginfo file) as a filesystem path in canonical native > form (where it's filesystem based) or as an opaque token for the odd > cases (frozen modules, for example) where a filesystem location isn't > available. > - entry from the RECORD file, as a slash-separated filename relative > to the root of the container. exactly, > > But if you're using get_installed_files(), why would you then want to > read the files? What exactly would you *use* get_installed_files for > which would then leave you needing to read the files? If it's to check > they haven't changed (by comparing md5 values) you're doing that to > uninstall, so that's the responsibility of the uninstall function. > > Again, it's a question of what is a public API, and what is the use > case it's designed for. Right. These APIs were created for third-party package managers. One use case of a package manager is the uninstallation, but I have no other use case in mind. > > I'm currently writing a SQLite importer, which will allow me to store > "files" in any sort of database tables I want, so I can build in some > nice pathological behaviour. That should tease out some awkward corner > cases :-) Sounds good. Semi-related: even if the files themselves are in the filesystem, having a sqlite db to index the list of installed distributions makes a good cache solution to reduce the disk I/O and speed up the query functions. So maybe we could use a disk-based cache for site-packages-like directories in the form of a sqlite db. That's what I am experimenting on my side. > > Paul > -- Tarek Ziad? | http://ziade.org From stephen at xemacs.org Sun Jul 5 16:42:23 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 05 Jul 2009 23:42:23 +0900 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp> <4A4E6EF5.4000901@v.loewis.de> Message-ID: <87hbxrur74.fsf@uwakimon.sk.tsukuba.ac.jp> Georg Brandl writes: > What I really want to see is the common-subset approach for > maintenance branches. IMO, this unfortunately unlikely to work as you seem to expect, for a two technical reasons and for a social reason. The first technical reason is that a maintenance branch really is a branch, not a subset. The fix that is appropriate for a maintenance branch is often inappropriate in detail for the mainline, and vice versa. But Mercurial doesn't care about "in detail" vs. "in design"; both will result in a conflict the first time that branch is merged. Second, some fixes for the maintenance branch will simply not be appropriate for the development branch, as the problem has already been fixed "en passant" by some other change. This can probably be handled by doing what git calls an "ours" merge to make it look like the unnecessary patch is an ancestor of the tip, even though no code was actually applied to the mainline. However, this kind of operation is some what delicate, and even if it's mostly scripted, it's likely to be somewhat unreliable for people who don't use the script very often ... which leads to the social problem: > > - our committers consistently refuse to merge changes across > > branches themselves, and likely continue to do so unless there > > is some feature of hg that I missed (e.g. one were merging > > would happen without any user specifically asking for it) > > If the checkin is done in the proper (the maint) branch, at least merging > of that change is automatic whenever someone does a hg merge. Maybe. But I see two problematic sides to this from the social point of view, which is the same old problem really. First, to the extent that it doesn't run into the technical problems, it encourages people to *not* review patches for each branch they are committed to. "It will get automerged anyway." Anything that discourages review is a bad thing. It will cause the development branch to "age" faster because of accumulation of crufty patches that are good enough as minimally invasive fixes for bugs in a maintenance branch, but which should be more robust for the development branch. I think you will also find that some people are not particularly interested in fixing the maintenance branch for some of their patches for exactly the same reasons they currently don't, and they will continue to refuse to do the work to commit in the maintenance branch first. Especially after the first time they run into one of the technical problems described above. In the end, any policy to encourage a "subset" policy is likely to end up as a burden on the same people who currently do the cross branch merging. From p.f.moore at gmail.com Sun Jul 5 18:26:07 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Jul 2009 17:26:07 +0100 Subject: [Python-Dev] PEP 376 - get_egginfo_files Message-ID: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com> The PEP says: """ get_egginfo_files(local=False) -> iterator of paths Iterates over the RECORD entries and return paths for each line if the path is pointing a file located in the .egg-info directory or one of its subdirectory. """ Should this method really only return filenames noted in the RECORD file? Would it not be better for it to iterate over *all* files in the .egg-info directory? I understand that there shouldn't, in practice, be any files in that directory *not* mentioned in the RECORD file, but given that we already have get_installed_files to read the RECORD file, I would imagine it's better for this file to so something more than filter the return values from get_installed_files. Actually, on that note, consider the pkgutil functions: def get_distribution(name): for d in get_distributions(): if d.name == name: return d return None def get_file_users(path): for d in get_distributions(): if d.uses(path): yield d These don't actually add much to the API. While I can see the advantage of having them as convenience methods, it might be worth pointing out in the PEP that that's all they are. Similarly, how valuable is Distribution.name, given that it's the same as Distribution.metadata.name? I'm probably just going to make it a property - @property def name(self): return self.metadata.name but that's actually slower than just using self.metadata.name directly, so it's a bit of an attractive nuisance, and I'd prefer it if it wasn't present. (For the PEP 302 stuff, I'm making metadata a cached property, so name *has* to be a property to ensure that the metadata cache is managed properly...) Paul. From jpellerin at gmail.com Sun Jul 5 19:46:40 2009 From: jpellerin at gmail.com (jason pellerin) Date: Sun, 5 Jul 2009 13:46:40 -0400 Subject: [Python-Dev] 2.6.3 unittest change breaks nose (issue 6418) Message-ID: <3bb02d620907051046k454d8612ic3edd168205b3465@mail.gmail.com> Bringing python-dev into the discussion at Barry's request. The summary is that a recent change to unittest.TestProgram breaks nose by moving self.testRunner initialization from it's old home in TestProgram.runTests to TestProgram.__init__. The very small patch attached to the ticket moves it back to runTests. Here's the ticket: http://bugs.python.org/issue6418 And a link to the testing in python list discussion: http://lists.idyll.org/pipermail/testing-in-python/2009-July/002032.html JP (primary author of nose) -------------- next part -------------- A non-text attachment was scrubbed... Name: PGP.sig Type: application/pgp-signature Size: 831 bytes Desc: not available URL: From martin at v.loewis.de Sun Jul 5 20:00:14 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 05 Jul 2009 20:00:14 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp> <4A4E6EF5.4000901@v.loewis.de> Message-ID: <4A50EA2E.8010401@v.loewis.de> >> In proposing such a workflow, consider these requirements: > > It seems that there is a consensus to separate the 2.x and 3.x repos, > and that also makes sense to me. (I think) I wasn't primarily talking about the representation of branches in hg - to that, I fully trust recommendations from hg users and experts. What will need debate and discussion in the PEP is the workflow, ie. the order in which changes should be applied to the branches. > If the checkin is done in the proper (the maint) branch, at least merging > of that change is automatic whenever someone does a hg merge. I probably don't fully understand, but that seems to imply a workflow were all changes made to one branch should also automatically be integrated into a different branch. I'm curious as to how such a mechanism can apply to Python. Regards, Martin From pje at telecommunity.com Sun Jul 5 20:32:48 2009 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 05 Jul 2009 14:32:48 -0400 Subject: [Python-Dev] PEP 376 - get_egginfo_files In-Reply-To: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.co m> References: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com> Message-ID: <20090705182953.90F7D3A4099@sparrow.telecommunity.com> At 05:26 PM 7/5/2009 +0100, Paul Moore wrote: >def get_distribution(name): > for d in get_distributions(): > if d.name == name: > return d > return None Btw, this is broken code anyway, because it's not handling case-insensitivity or name canonicalization. (I've mentioned these issue previously on the distutils-sig.) From ziade.tarek at gmail.com Sun Jul 5 20:33:22 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 5 Jul 2009 20:33:22 +0200 Subject: [Python-Dev] PEP 376 - get_egginfo_files In-Reply-To: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com> References: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com> Message-ID: <94bdd2610907051133v5605814bo998a3d25a25f78ff@mail.gmail.com> 2009/7/5 Paul Moore : > The PEP says: > > """ > get_egginfo_files(local=False) -> iterator of paths > > Iterates over the RECORD entries and return paths for each line if the > path is pointing a file located in the .egg-info directory or one of > its subdirectory. > """ > > Should this method really only return filenames noted in the RECORD > file? Would it not be better for it to iterate over *all* files in the > .egg-info directory? > I understand that there shouldn't, in practice, > be any files in that directory *not* mentioned in the RECORD file, but > given that we already have get_installed_files to read the RECORD > file, I would imagine it's better for this file to so something more > than filter the return values from get_installed_files. I don't see a use case for having more out of get_egginfo_files. I still find it useful because to iterate over metadata files. Maybe we could remove it and add a filter option for get_installed_files. A callable that gets each visited file and returns True or False to filter them out: get_installed_files(path, filter=callable) And then provide a "egginfo_files" callable to get what we have with get_egginfo_files : get_installed_files(path, filter=egginfo_files) > > Actually, on that note, consider the pkgutil functions: > > def get_distribution(name): > ? ?for d in get_distributions(): > ? ? ? ?if d.name == name: > ? ? ? ? ? ?return d > ? ?return None > > def get_file_users(path): > ? ?for d in get_distributions(): > ? ? ? ?if d.uses(path): > ? ? ? ? ? ?yield d > > These don't actually add much to the API. While I can see the > advantage of having them as convenience methods, it might be worth > pointing out in the PEP that that's all they are. Sure, > > Similarly, how valuable is Distribution.name, given that it's the same > as Distribution.metadata.name? I'm probably just going to make it a > property - It's just for conveniency, since this metadata field is also the identifier of the distribution. > > @property > def name(self): > ? ?return self.metadata.name I don't think this adds any value, since self.metadata is a read-only instance, that gets loaded once when the Distribution object is created. From pje at telecommunity.com Sun Jul 5 20:36:27 2009 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 05 Jul 2009 14:36:27 -0400 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.co m> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> Message-ID: <20090705183326.097CF3A4099@sparrow.telecommunity.com> At 03:13 PM 7/5/2009 +0200, Tarek Ziad? wrote: >The loader would still work with "DIST_NAME.egg-info" directories for >compatibility with >existing format in the query APIs, but the Distutils install command >would rather create "DIST_NAME.metadata" Note that this would then break setuptools without adding any benefit; ".metadata" is less precise and less unique than '.egg-info'. If you want a clearer name, '.pydist' or some such would at least be reasonably specific. (It'd still have a backward compatibility problem, but at least then there'd be some benefit to the name change.) From ziade.tarek at gmail.com Sun Jul 5 20:37:45 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 5 Jul 2009 20:37:45 +0200 Subject: [Python-Dev] PEP 376 - get_egginfo_files In-Reply-To: <20090705182953.90F7D3A4099@sparrow.telecommunity.com> References: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com> <20090705182953.90F7D3A4099@sparrow.telecommunity.com> Message-ID: <94bdd2610907051137q16912234h7ce39c0f262783c9@mail.gmail.com> 2009/7/5 P.J. Eby : > At 05:26 PM 7/5/2009 +0100, Paul Moore wrote: >> >> def get_distribution(name): >> ? ?for d in get_distributions(): >> ? ? ? ?if d.name == name: >> ? ? ? ? ? ?return d >> ? ?return None > > Btw, this is broken code anyway, because it's not handling > case-insensitivity or name canonicalization. ?(I've mentioned these issue > previously on the distutils-sig.) Yes thanks, we need to fix that, the case-insensitivity or name canonicalization functions are present, just to be used in that function too > > -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Sun Jul 5 20:43:20 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 5 Jul 2009 20:43:20 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <20090705183326.097CF3A4099@sparrow.telecommunity.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> Message-ID: <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> 2009/7/5 P.J. Eby : > At 03:13 PM 7/5/2009 +0200, Tarek Ziad? wrote: >> >> The loader would still work with "DIST_NAME.egg-info" directories for >> compatibility with >> existing format in the query APIs, but the Distutils install command >> would rather create ?"DIST_NAME.metadata" > > Note that this would then break setuptools without adding any benefit; > ".metadata" is less precise and less unique than '.egg-info'. But if it's based on PEP 302 protocols and if the pkgutil code works with the sys.meta_path hook, setuptools could then provide its loader, based on its EggFormats and act as a provider without being broken. In that case, Distutils could provide a standard loader, with the change I mentioned. >?If you want a > clearer name, '.pydist' or some such would at least be reasonably specific. > ?(It'd still have a backward compatibility problem, but at least then > there'd be some benefit to the name change.) I do find "DIST_NAME.metadata" well-named and specific. But I guess that's just bikeshedding :) From p.f.moore at gmail.com Sun Jul 5 20:46:58 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Jul 2009 19:46:58 +0100 Subject: [Python-Dev] PEP 376 - get_egginfo_files In-Reply-To: <20090705182953.90F7D3A4099@sparrow.telecommunity.com> References: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com> <20090705182953.90F7D3A4099@sparrow.telecommunity.com> Message-ID: <79990c6b0907051146o7c62ac97kffdafa1ab95040f8@mail.gmail.com> 2009/7/5 P.J. Eby : > At 05:26 PM 7/5/2009 +0100, Paul Moore wrote: >> >> def get_distribution(name): >> ? ?for d in get_distributions(): >> ? ? ? ?if d.name == name: >> ? ? ? ? ? ?return d >> ? ?return None > > Btw, this is broken code anyway, because it's not handling > case-insensitivity or name canonicalization. ?(I've mentioned these issue > previously on the distutils-sig.) Fair point. (Although I don't recall your distutils-sig posting, so I'm not sure what you mean by "name canonicalisation"). Note that even on case insensitive filesystems, module/package names are handled case sensitively. I would be happy to see distribution names handled the same (although I have no vested interest either way). Is there code around to handle filename matching based on the case sensitivity of the filesystem? (My understanding is that there isn't, and programs like Mercurial play fancy games to determine if a filesystem is case sensitive before doing tests). Of course, if you're OK with an inaccurate but simple choice based on OS, it would probably be OK to use os.path.normcase. Paul. From p.f.moore at gmail.com Sun Jul 5 21:05:30 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Jul 2009 20:05:30 +0100 Subject: [Python-Dev] PEP 376 - get_egginfo_files In-Reply-To: <94bdd2610907051133v5605814bo998a3d25a25f78ff@mail.gmail.com> References: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com> <94bdd2610907051133v5605814bo998a3d25a25f78ff@mail.gmail.com> Message-ID: <79990c6b0907051205t7458f693u493d6905ec5df749@mail.gmail.com> 2009/7/5 Tarek Ziad? : > 2009/7/5 Paul Moore : >> The PEP says: >> >> """ >> get_egginfo_files(local=False) -> iterator of paths >> >> Iterates over the RECORD entries and return paths for each line if the >> path is pointing a file located in the .egg-info directory or one of >> its subdirectory. >> """ >> >> Should this method really only return filenames noted in the RECORD >> file? Would it not be better for it to iterate over *all* files in the >> .egg-info directory? >> I understand that there shouldn't, in practice, >> be any files in that directory *not* mentioned in the RECORD file, but >> given that we already have get_installed_files to read the RECORD >> file, I would imagine it's better for this file to so something more >> than filter the return values from get_installed_files. > > I don't see a use case for having more out of get_egginfo_files. > I still find it useful because to iterate over metadata files. > > Maybe we could remove it and add a filter option for get_installed_files. > A callable that gets each visited file and returns True or False to > filter them out: > > ?get_installed_files(path, filter=callable) > > And then provide a "egginfo_files" callable to get what we have with > get_egginfo_files : > > ?get_installed_files(path, filter=egginfo_files) -1. Unnecessary generalisation. Let's stick with the 2 functions as documented. [...] >> Similarly, how valuable is Distribution.name, given that it's the same >> as Distribution.metadata.name? I'm probably just going to make it a >> property - > > It's just for conveniency, since this metadata field is also the > identifier of the distribution. > >> >> @property >> def name(self): >> ? ?return self.metadata.name > > I don't think this adds any value, since self.metadata is a read-only instance, > that gets loaded once when the Distribution object is created. ... not any more :-) Your zipfile handling was horribly broken on Windows, thanks to the usual slash/backslash confusion. The sanest way to fix it seemed to me to be to load the metadata lazily, rather than in the __init__ (as otherwise, zipfile and filesystem implementation end up not being able to share any code). Once that's done, the name attribute has to *also* handle lazy-loading of the metadata, and the above property is the easiest way to do this. Actually, my implementation is looking less and less like yours, and ultimately any implementation questions are irrelevant until you see my code and spot all the errors :-) I'm trying to get it into a postable state as fast as I can. (At last count, I've replaced about 140 lines of code with 70, and it now includes PEP 302 support all the (non-internal) tests still pass. So it's looking OK...) Paul. From dreamingforward at gmail.com Sun Jul 5 21:36:20 2009 From: dreamingforward at gmail.com (average) Date: Sun, 5 Jul 2009 12:36:20 -0700 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) Message-ID: <913f9f570907051236i68aee3f8p8278e8f3699d32f1@mail.gmail.com> >> Is it really that confusing? I have never heard of anyone asking "what >> is py3k?" > > Do you read python-list? It has been asked. Also, some people seem to > think that py3k is different from python 3. Personally, I vote for keeping the "3k" for 3000 (or is it 3072?). I believe that py3k represents a ideal that hasn't been reached, despite being hoped for in python3. By keeping it, it confers the idea continual evolution *within* the language until that hypothetical ideal is reached. Clearly, there are times when a language reaches only a local maximum, and must depart from itself to arrive at a more global optimum (an annealing problem in the minimization of frustration energy). If py3k wasn't kept, another term would eventually need to be invented. marcos From benjamin at python.org Sun Jul 5 21:48:01 2009 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 5 Jul 2009 14:48:01 -0500 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <913f9f570907051236i68aee3f8p8278e8f3699d32f1@mail.gmail.com> References: <913f9f570907051236i68aee3f8p8278e8f3699d32f1@mail.gmail.com> Message-ID: <1afaf6160907051248s4745cc27sc9d9f89c47b0c124@mail.gmail.com> 2009/7/5 average : >>> Is it really that confusing? I have never heard of anyone asking "what >>> is py3k?" >> >> Do you read python-list? It has been asked. Also, some people seem to >> think that py3k is different from python 3. > > Personally, I vote for keeping the "3k" for 3000 (or is it 3072?). ?I > believe that py3k represents a ideal that hasn't been reached, despite > being hoped for in python3. ?By keeping it, it confers the idea > continual evolution *within* the language until that hypothetical > ideal is reached. ?Clearly, there are times when a language reaches > only a local maximum, and must depart from itself to arrive at a more > global optimum (an annealing problem in the minimization of > frustration energy). ?If py3k wasn't kept, another term would > eventually need to be invented. And that's why we're already fantasizing about Py4k. :) -- Regards, Benjamin From martin at v.loewis.de Sun Jul 5 22:37:56 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 05 Jul 2009 22:37:56 +0200 Subject: [Python-Dev] Mercurial migration: help needed Message-ID: <4A510F24.1060004@v.loewis.de> In this thread, I'd like to collect things that ought to be done but where Dirkjan has indicated that he would prefer if somebody else did it. So far, I have only one item: build identification. If you want to work on this, please either provide a patch (for trunk and/or py3k), or (if you are a committer) create a subversion branch. It seems that Barry and I agree that for the maintenance branches, sys.subversion should be frozen, so we need actually two sets of patches: one that removes sys.subversion entirely, and the other that freezes the branch to the respective one, and freezes the subversion revision to None. Of course, it seems that the actual representation of branches hasn't been determined yet, so the build process integration may need to be changed if named branches aren't going to be used in the end. Anybody working on this should have good knowledge of the Python source code, Mercurial, and either autoconf or Visual Studio (preferably both). Regards, Martin From ajaksu at gmail.com Sun Jul 5 22:45:13 2009 From: ajaksu at gmail.com (Daniel Diniz) Date: Sun, 5 Jul 2009 17:45:13 -0300 Subject: [Python-Dev] Mercurial migration: help needed In-Reply-To: <4A510F24.1060004@v.loewis.de> References: <4A510F24.1060004@v.loewis.de> Message-ID: <2d75d7660907051345i1a9e839dgbf1362ba2e9b22bc@mail.gmail.com> Hi Martin, "Martin v. L?wis" wrote: > In this thread, I'd like to collect things that ought to be done > but where Dirkjan has indicated that he would prefer if somebody else > did it. > > So far, I have only one item: build identification. If you want to work > on this, please either provide a patch (for trunk and/or py3k), or > (if you are a committer) create a subversion branch. I do want to help, but I believe I'll only have time a week from now. If we need/want Roundup tweaks to go with Mercurial, I can work on that (keep in mind we have a cool GSoC student working on Mercurial-Roundup integration, and I'm willing to work on our needs with him). For build identification, I'd only be able to do the C/Python side of things and (with luck) autoconf. Cheers, Daniel From ncoghlan at gmail.com Sun Jul 5 22:58:50 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 06 Jul 2009 06:58:50 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A50EA2E.8010401@v.loewis.de> References: <87r5wxvqre.fsf@uwakimon.sk.tsukuba.ac.jp> <4A4E6EF5.4000901@v.loewis.de> <4A50EA2E.8010401@v.loewis.de> Message-ID: <4A51140A.9010801@gmail.com> Martin v. L?wis wrote: > What will need debate and discussion in the PEP is the workflow, ie. > the order in which changes should be applied to the branches. Particularly since there isn't even "one true workflow" *now*. I see three main variants go by on Python-checkins: - svnmerge based - commit to 2.x - backport to 2.6 with svnmerge - forward port to 3.x with svnmerge - backport to 3.0 (now 3.1) with svnmerge - manual port based - as above, but without using svnmerge - Py3k focused - commit to 3.x - manual backport to 2.x - possible svnmerge block of the backported 2.x commit - possible svnmerge based or manual backport to 2.6/3.1 While it would obviously be *nice* if every committer maintained 4 checkouts and either blocked or committed each change on the appropriate branches, I think actually *requiring* a specific workflow has the potential to cost us commits. Even if one workflow is designated the 'preferred' approach, the others still need to be supported to handle various possibilities for the "first" commit for a given change: - forward port only - commit to 2.6 - forward port to 2.x - forward port to 3.1 - forward port to 3.x - backport only - commit to 3.x - backport to 3.1 - backport to 2.x - backport to 2.6 - mixed, starting with 2.x (aka current svnmerge workflow) - commit to 2.x - backport to 2.6 - forward port to 3.x - backport to 3.1 - mixed, starting with 3.1 - commit to 3.1 - forward port to 3.x - backport to 2.6 - forward port or backport to 2.x Note that there are actual multiple variations even of the above workflows, based on the *source* of the various forward ports and backports. Also, each "forward port" or "backport" can be replaced by blocking the merge rather than applying it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Sun Jul 5 23:10:52 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 06 Jul 2009 07:10:52 +1000 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> Message-ID: <4A5116DC.1070109@gmail.com> Tarek Ziad? wrote: > 2009/7/5 P.J. Eby : >> If you want a >> clearer name, '.pydist' or some such would at least be reasonably specific. >> (It'd still have a backward compatibility problem, but at least then >> there'd be some benefit to the name change.) > > I do find "DIST_NAME.metadata" well-named and specific. But I guess that's just > bikeshedding :) pydist has the advantage of both being more intuitive than 'egginfo' while still making it clear that this is Python related metadata rather than, say, something added by an OS packaging utility. So no, I don't think it's bikeshedding: 'metadata': accurate but generic. Not clear that it relates to Python specifically (except that it happens to stored in a Python related directory) 'egginfo': accurate, specific and serves as a good mnemonic. However, use of the 'egg' jargon means that someone is unlikely to guess what it means without being told, and it's less obvious that this is Python related 'pydist': accurate, specific and without the disadvantages of the 'egg' jargon PJE points out that existing tools (setuptools, pip, etc) won't be compatible with the new format at all if it uses a new name, but I am having trouble seeing that as a *bad* thing. By using a new name for the directory we *guarantee* that old packaging utilities won't get confused by the new format (they simply won't acknowledge its existence). So +1 for pydist as the directory extension in PEP 376 (for the above reasons). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From p.f.moore at gmail.com Sun Jul 5 23:25:34 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Jul 2009 22:25:34 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <20090705183326.097CF3A4099@sparrow.telecommunity.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> Message-ID: <79990c6b0907051425t175fbe5tb700b52da7a8d33e@mail.gmail.com> 2009/7/5 P.J. Eby : > At 03:13 PM 7/5/2009 +0200, Tarek Ziad? wrote: >> >> The loader would still work with "DIST_NAME.egg-info" directories for >> compatibility with >> existing format in the query APIs, but the Distutils install command >> would rather create ?"DIST_NAME.metadata" > > Note that this would then break setuptools without adding any benefit; > ".metadata" is less precise and less unique than '.egg-info'. ?If you want a > clearer name, '.pydist' or some such would at least be reasonably specific. > ?(It'd still have a backward compatibility problem, but at least then > there'd be some benefit to the name change.) Personally, the filename doesn't bother me. It's the API name that I'd like changed (particularly as the contraction "egginfo" is clumsy in function names...) Paul. From ben+python at benfinney.id.au Mon Jul 6 00:21:06 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Mon, 06 Jul 2009 08:21:06 +1000 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <4A5116DC.1070109@gmail.com> Message-ID: <87ljn2rctp.fsf@benfinney.id.au> Nick Coghlan writes: > 'pydist': accurate, specific and without the disadvantages of the 'egg' > jargon > > PJE points out that existing tools (setuptools, pip, etc) won't be > compatible with the new format at all if it uses a new name, but I am > having trouble seeing that as a *bad* thing. By using a new name for the > directory we *guarantee* that old packaging utilities won't get confused > by the new format (they simply won't acknowledge its existence). > > So +1 for pydist as the directory extension in PEP 376 (for the above > reasons). +1 for the same reasons. Thank you for expressing them. -- \ ?We are not gonna be great; we are not gonna be amazing; we are | `\ gonna be *amazingly* amazing!? ?Zaphod Beeblebrox, _The | _o__) Hitch-Hiker's Guide To The Galaxy_, Douglas Adams | Ben Finney From pje at telecommunity.com Mon Jul 6 03:04:08 2009 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 05 Jul 2009 21:04:08 -0400 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com > References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> Message-ID: <20090706010108.CCEC83A4099@sparrow.telecommunity.com> At 08:43 PM 7/5/2009 +0200, Tarek Ziad? wrote: >But if it's based on PEP 302 protocols and if the pkgutil code works >with the sys.meta_path hook, >setuptools could then provide its loader, based on its EggFormats and >act as a provider without being broken. You misunderstand me. The whole point of putting .egg-info in distutils in the first place was to enable setuptools to detect the presence of disutils-installed packages. That's what's broken by changing the name. From pje at telecommunity.com Mon Jul 6 03:06:20 2009 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 05 Jul 2009 21:06:20 -0400 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <4A5116DC.1070109@gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <4A5116DC.1070109@gmail.com> Message-ID: <20090706010320.3220A3A4099@sparrow.telecommunity.com> At 07:10 AM 7/6/2009 +1000, Nick Coghlan wrote: >By using a new name for the >directory we *guarantee* that old packaging utilities won't get confused >by the new format (they simply won't acknowledge its existence). This is incorrect; they will get confused because they will think that the relevant package is *not* installed, and proceed to install a duplicate. That's why .egg-info was added to the stdlib in the first place. From martin at v.loewis.de Mon Jul 6 07:07:30 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 06 Jul 2009 07:07:30 +0200 Subject: [Python-Dev] Mercurial migration: help needed In-Reply-To: <2d75d7660907051345i1a9e839dgbf1362ba2e9b22bc@mail.gmail.com> References: <4A510F24.1060004@v.loewis.de> <2d75d7660907051345i1a9e839dgbf1362ba2e9b22bc@mail.gmail.com> Message-ID: <4A518692.6020605@v.loewis.de> > I do want to help, but I believe I'll only have time a week from now. > If we need/want Roundup tweaks to go with Mercurial, I can work on > that (keep in mind we have a cool GSoC student working on > Mercurial-Roundup integration, and I'm willing to work on our needs > with him). I think that's straight-forward. Once we know what URLs to link to, we just need to fix the regexps. > For build identification, I'd only be able to do the C/Python side of > things and (with luck) autoconf. Ok, we'll see in a week from now whether anybody had volunteered. Regards, Martin From ziade.tarek at gmail.com Mon Jul 6 08:55:14 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 6 Jul 2009 08:55:14 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <20090706010320.3220A3A4099@sparrow.telecommunity.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <4A5116DC.1070109@gmail.com> <20090706010320.3220A3A4099@sparrow.telecommunity.com> Message-ID: <94bdd2610907052355w6d69ee53nc4e85cf1e2a5e748@mail.gmail.com> But as I said earlier, if we use a PEP 302-like loader, distutils will be able to consume several loaders, so setuptools will be able to provide its storage strategy (naming and egg dir locations) So I don't understand why you are saying that it will be incompatible or get confused. 2009/7/6 P.J. Eby : > At 07:10 AM 7/6/2009 +1000, Nick Coghlan wrote: >> >> By using a new name for the >> directory we *guarantee* that old packaging utilities won't get confused >> by the new format (they simply won't acknowledge its existence). > > This is incorrect; they will get confused because they will think that the > relevant package is *not* installed, and proceed to install a duplicate. > ?That's why .egg-info was added to the stdlib in the first place. > > -- Tarek Ziad? | http://ziade.org From p.f.moore at gmail.com Mon Jul 6 09:15:05 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Jul 2009 08:15:05 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <20090706010108.CCEC83A4099@sparrow.telecommunity.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> Message-ID: <79990c6b0907060015u71d07defyc846e88adf48c3c1@mail.gmail.com> 2009/7/6 P.J. Eby : > At 08:43 PM 7/5/2009 +0200, Tarek Ziad? wrote: >> >> But if it's based on PEP 302 protocols and if the pkgutil code works >> with the sys.meta_path hook, >> setuptools could then provide its loader, based on its EggFormats and >> act as a provider without being broken. > > You misunderstand me. ?The whole point of putting .egg-info in distutils in > the first place was to enable setuptools to detect the presence of > disutils-installed packages. ?That's what's broken by changing the name. This implies that there is no possibility that setuptools will be changed to support the new standard. That's patently false. Whether you *want* to change setuptools in such a way is up to you. And it's worth a note in the PEP if this change is made, that it will require a change to setuptools if that package is still to recognise distutils-installed packages. Paul. From ronaldoussoren at mac.com Mon Jul 6 08:16:08 2009 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 06 Jul 2009 08:16:08 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <4A5116DC.1070109@gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <4A5116DC.1070109@gmail.com> Message-ID: <31B6EE4E-8D3A-4BBC-ABAD-074DBEEC60F9@mac.com> I'm -1 on changing the name. For better or worse setuptools is the elephant in the room w.r.t. package management and it would IMHO be better to stay compatible (even if the stdlib only implements a subset of setuptools/pkg_resources) Ronald On 5 jul 2009, at 23:10, Nick Coghlan wrote: > Tarek Ziad? wrote: >> 2009/7/5 P.J. Eby : >>> If you want a >>> clearer name, '.pydist' or some such would at least be reasonably >>> specific. >>> (It'd still have a backward compatibility problem, but at least then >>> there'd be some benefit to the name change.) >> >> I do find "DIST_NAME.metadata" well-named and specific. But I guess >> that's just >> bikeshedding :) > > pydist has the advantage of both being more intuitive than 'egginfo' > while still making it clear that this is Python related metadata > rather > than, say, something added by an OS packaging utility. So no, I don't > think it's bikeshedding: > > 'metadata': accurate but generic. Not clear that it relates to Python > specifically (except that it happens to stored in a Python related > directory) > > 'egginfo': accurate, specific and serves as a good mnemonic. However, > use of the 'egg' jargon means that someone is unlikely to guess what > it > means without being told, and it's less obvious that this is Python > related > > 'pydist': accurate, specific and without the disadvantages of the > 'egg' > jargon > > PJE points out that existing tools (setuptools, pip, etc) won't be > compatible with the new format at all if it uses a new name, but I am > having trouble seeing that as a *bad* thing. By using a new name for > the > directory we *guarantee* that old packaging utilities won't get > confused > by the new format (they simply won't acknowledge its existence). > > So +1 for pydist as the directory extension in PEP 376 (for the above > reasons). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > --------------------------------------------------------------- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com From ziade.tarek at gmail.com Mon Jul 6 09:50:09 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 6 Jul 2009 09:50:09 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907060015u71d07defyc846e88adf48c3c1@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <79990c6b0907060015u71d07defyc846e88adf48c3c1@mail.gmail.com> Message-ID: <94bdd2610907060050h307669e6r600692c1ed6fcaf0@mail.gmail.com> 2009/7/6 Paul Moore : > 2009/7/6 P.J. Eby : >> At 08:43 PM 7/5/2009 +0200, Tarek Ziad? wrote: >>> >>> But if it's based on PEP 302 protocols and if the pkgutil code works >>> with the sys.meta_path hook, >>> setuptools could then provide its loader, based on its EggFormats and >>> act as a provider without being broken. >> >> You misunderstand me. ?The whole point of putting .egg-info in distutils in >> the first place was to enable setuptools to detect the presence of >> disutils-installed packages. ?That's what's broken by changing the name. > Forget about my previous mail, I didn't see that answer, > This implies that there is no possibility that setuptools will be > changed to support the new standard. That's patently false. Whether > you *want* to change setuptools in such a way is up to you. And it's > worth a note in the PEP if this change is made, that it will require a > change to setuptools if that package is still to recognise > distutils-installed packages. +1 setuptools is built on the top of distutils, not the contrary. And if setuptools wants to query installed distribution, it will have to be changed to use the query functions added in pkgutil. Tarek From ziade.tarek at gmail.com Mon Jul 6 09:53:21 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 6 Jul 2009 09:53:21 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <31B6EE4E-8D3A-4BBC-ABAD-074DBEEC60F9@mac.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <4A5116DC.1070109@gmail.com> <31B6EE4E-8D3A-4BBC-ABAD-074DBEEC60F9@mac.com> Message-ID: <94bdd2610907060053x58d9b191wac7bfda8743c7fad@mail.gmail.com> 2009/7/6 Ronald Oussoren : > I'm -1 on changing the name. For better or worse setuptools is the elephant > in the room w.r.t. package management and it would IMHO be better to stay > compatible (even if the stdlib only implements a subset of > setuptools/pkg_resources) > I'd rather see the elephant evolves. I don't see why we should bend a standard we want to introduce in the stdlib, for a third-party package that is able to evolve to stick to a new standard without any problem. From p.f.moore at gmail.com Mon Jul 6 11:32:28 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Jul 2009 10:32:28 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <94bdd2610907060053x58d9b191wac7bfda8743c7fad@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <4A5116DC.1070109@gmail.com> <31B6EE4E-8D3A-4BBC-ABAD-074DBEEC60F9@mac.com> <94bdd2610907060053x58d9b191wac7bfda8743c7fad@mail.gmail.com> Message-ID: <79990c6b0907060232j526f293dv35ba9ee327e912ae@mail.gmail.com> 2009/7/6 Tarek Ziad? : > 2009/7/6 Ronald Oussoren : >> I'm -1 on changing the name. For better or worse setuptools is the elephant >> in the room w.r.t. package management and it would IMHO be better to stay >> compatible (even if the stdlib only implements a subset of >> setuptools/pkg_resources) >> > > I'd rather see the elephant evolves. > > I don't see why we should bend a standard we want to introduce in the stdlib, > for a third-party package that is able to evolve to stick to a new standard > without any problem. I agree with this. The big problem is that setuptools development seems to have essentially stagnated, and while that shouldn't be something we care about, it does need to be considered. Although my instinct is to fight against "do it our way because we won't change to do it your way" arguments, I have to ask whether the (purely internal) name of the metadata directory is really worth fighting over. I'm +0 on changing the name, as long as it's the *only* "do it this way because setuptools isn't going to change" issue. If more such issues come up, I'll become +1 on a clean break, to avoid endless such arguments and restrictions. Paul. From ncoghlan at gmail.com Mon Jul 6 16:36:15 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 07 Jul 2009 00:36:15 +1000 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <20090706010108.CCEC83A4099@sparrow.telecommunity.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> Message-ID: <4A520BDF.80503@gmail.com> P.J. Eby wrote: > At 08:43 PM 7/5/2009 +0200, Tarek Ziad? wrote: >> But if it's based on PEP 302 protocols and if the pkgutil code works >> with the sys.meta_path hook, >> setuptools could then provide its loader, based on its EggFormats and >> act as a provider without being broken. > > You misunderstand me. The whole point of putting .egg-info in distutils > in the first place was to enable setuptools to detect the presence of > disutils-installed packages. That's what's broken by changing the name. How much information does setuptools actually need in order to tell that a distribution is already present? Presumably the existing .egginfo files generated by distutils are sufficient for that task? If so, I'd still prefer to keep the new metadata safely out of reach of the legacy package management systems that don't understand it, while having distutils retain the ability to generate a legacy ".egginfo" file to warn off the existing package management tools from installing the same distribution again. So instead of ".egginfo becomes a directory" as in the current version of PEP 376, the existing distutils .egginfo file generation would instead remain unchanged with a ".pydist" directory being added alongside to hold all the new goodies. That way, the legacy formats (EGG-INFO, .egg-info file, .egg-info directory) continue to play as well together as they do now, while the new distutils system will understand .pydist directories natively and allow the injection of hook implementations by the third party package management tools in order to understand the old formats. That seems like it will give us the best of both worlds - a clean base to work from in the future, but without breaking the existing interoperability mechanisms (at least in the short term - maybe by the time 3.3 rolls around we will be able to start thinking about deprecating the legacy .egginfo formats, but even considering that question is a long way off). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From mal at egenix.com Mon Jul 6 16:51:47 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 06 Jul 2009 16:51:47 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: <4A520F83.7020300@egenix.com> Dirkjan Ochtman wrote: > In response to some rumblings on python-committers and just to request > more feedback, a progress report. I know it's long, I've tried to put > to keep it concise and chunked, though. Two things: * We need some form of documentation of how committers are expected to work with the hg repo. (This is also missing for the Subversion repo, which due to the 3.x branch has gotten somewhat unclear - at least for me) It is currently not clear where to check in patches, whether and where to backport or forward-patch, which branches to consider closed, etc. E.g. if I check in something in trunk/ (Python 2.7), do I have to forward patch this change to the 3.0 branch (guess not), the py3k/ branch (Python 3.1), or will someone else take care of this, so that it's better not to interfere by doing it myself ? * Our tracker, checkin messages, online documentation and lots of Python resources on the web are full of references to the Subversion rXYZ notation of changesets. The PEP has to provide some way to gracefully provide a redirect from those URLs to the new ones (both for the source code browser and the bug search tool on python.org). The PEP will also have to address the same issue for checkin messages. Perhaps by converting the rXYZ notation to the hash based values during migration or by adding the has value after the rXYZ value in parens ?! -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 06 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From p.f.moore at gmail.com Mon Jul 6 17:14:08 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Jul 2009 16:14:08 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <4A520BDF.80503@gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> Message-ID: <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> 2009/7/6 Nick Coghlan : > P.J. Eby wrote: >> At 08:43 PM 7/5/2009 +0200, Tarek Ziad? wrote: >>> But if it's based on PEP 302 protocols and if the pkgutil code works >>> with the sys.meta_path hook, >>> setuptools could then provide its loader, based on its EggFormats and >>> act as a provider without being broken. >> >> You misunderstand me. ?The whole point of putting .egg-info in distutils >> in the first place was to enable setuptools to detect the presence of >> disutils-installed packages. ?That's what's broken by changing the name. This is getting confusing. Is Phillip saying that setuptools will cope with the file changing to a directory without modification, but it can't handle a change in the name? My site-packages has a confusing mix of egginfo directories and files. Note that I NEVER use setuptools other than where an existing package's setup.py requires it. In that case, I still only do python setup.py bdist_wininst and install the generated installer. So is PEP 376 going to be able to cope with the stuff I have installed at the moment? If not, what's the point??? Example: I have the "coverage" package installed. It does not use setuptools at runtime - I could happily install it from a bdist_wininst on a machine without setuptools installed, and it would run fine. (In fact, that's what I did!) It has created a coverage-3.0-py2.6.egg-info *directory* in my site-packages, with no RECORD file - presumably, this is in a setuptools format. If setuptools is not going to change to conform to PEP 376, then any tools built using PEP 376 will fail to recognise my coverage install. Which, as far as I am concerned, is a completely standard bdist_wininst installer. In fact, the above strongly suggests to me that PEP 376 must NOT follow setuptools conventions - otherwise, it will end up clashing with the files setuptools is putting on my system. > How much information does setuptools actually need in order to tell that > a distribution is already present? Presumably the existing .egginfo > files generated by distutils are sufficient for that task? It appears so, but setuptools doesn't use (or create!!!) those files in its own installer formats. > If so, I'd still prefer to keep the new metadata safely out of reach of > the legacy package management systems that don't understand it, while > having distutils retain the ability to generate a legacy ".egginfo" file > to warn off the existing package management tools from installing the > same distribution again. So for a minimal case of a single .py file packaged up as a distribution, we now have the .py file, a legacy .egginfo file, a new .pydist directory containing RECORD and PKG-INFO files? That's getting silly. > > So instead of ".egginfo becomes a directory" as in the current version > of PEP 376, the existing distutils .egginfo file generation would > instead remain unchanged with a ".pydist" directory being added > alongside to hold all the new goodies. That way, the legacy formats > (EGG-INFO, .egg-info file, .egg-info directory) continue to play as well > together as they do now, while the new distutils system will understand > .pydist directories natively and allow the injection of hook > implementations by the third party package management tools in order to > understand the old formats. > > That seems like it will give us the best of both worlds - a clean base > to work from in the future, but without breaking the existing > interoperability mechanisms (at least in the short term - maybe by the > time 3.3 rolls around we will be able to start thinking about > deprecating the legacy .egginfo formats, but even considering that > question is a long way off). And yet, given that PEP 376 is explicitly being designed with a goal being to act as the common standard that *all* package management formats can use, is it not the whole point that once it's defined and we have achieved consensus, existing package managers will switch to using it rather than retaining a range of custom formats? I honestly fail to see why we are catering to this "setuptools cannot change" view. Usually, it's the standard library that cannot change, because of the longer release cycle of the core. If a standard is agreed, and setuptools won't conform to it, maybe it's time for someone to fork setuptools (unilaterally, if Phillip can't agree to an amicable transfer). Otherwise setuptools will remain a roadblock for any development of distutils. Frustrated-ly y'rs Paul. From ziade.tarek at gmail.com Mon Jul 6 17:40:43 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 6 Jul 2009 17:40:43 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> Message-ID: <94bdd2610907060840p3d0c82eakbeb85f742f6979a@mail.gmail.com> On Mon, Jul 6, 2009 at 5:14 PM, Paul Moore wrote: > [...] > And yet, given that PEP 376 is explicitly being designed with a goal > being to act as the common standard that *all* package management > formats can use, is it not the whole point that once it's defined and > we have achieved consensus, existing package managers will switch to > using it rather than retaining a range of custom formats? That's the goal from the very beginning. > > I honestly fail to see why we are catering to this "setuptools cannot > change" view. Usually, it's the standard library that cannot change, > because of the longer release cycle of the core. If a standard is > agreed, and setuptools won't conform to it, maybe it's time for > someone to fork setuptools (unilaterally, if Phillip can't agree to an > amicable transfer). Otherwise setuptools will remain a roadblock for > any development of distutils. > > Frustrated-ly y'rs +1 why can't we just go ahead and continue the work as we started with PEP 376, introducing your work on PEP 302-like behavior. Then if we get a consensus on this PEP and introduce it in 2.7/3.2, setuptools will have to follow this consensus. FWIW I am also frustrated because the setuptools development has been stopped 9 months ago, and *doesn't work anymore with the current distutils trunk*. There are a lot of patches waiting in the setuptools tracker people wrote. I hate to say this but this project needs an active maintainer or it's going to die. So I've said in distutils-sig that I would start my own branch of setuptools before 2.7/3.2 is out for the people in the community that relies on this package. Not to maintain the project because I have a lot to do already on distutils side. But at least to make sure everyone can work with the upcoming releases of Python 2.x and 3.X without having their setup.py broken (and switch eventually to a plain distutils solution maybe) Tarek From p.f.moore at gmail.com Mon Jul 6 17:53:01 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Jul 2009 16:53:01 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <94bdd2610907060840p3d0c82eakbeb85f742f6979a@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <94bdd2610907060840p3d0c82eakbeb85f742f6979a@mail.gmail.com> Message-ID: <79990c6b0907060853o22d29679kd702c88e5c3ff46f@mail.gmail.com> 2009/7/6 Tarek Ziad? : > why can't we just go ahead and continue the work as we started with PEP 376, > introducing your work on PEP 302-like behavior. > > Then if we get a consensus on this PEP and introduce it in 2.7/3.2, > setuptools will have to follow this consensus. Essentially, because when I ask questions, responses along the lines of "you have to do it like X because setuptools does that" come back, and (not being a setuptools user) I can't tell whether there's a valid reason in there. I'm uncomfortable assuming that setuptools experience is irrelevant, but I can't distinguish between valid arguments and "setuptools can't change" arguments. I need to write another email with a list of outstanding open issues. If we can thrash them out *without* getting bogged down in setuptools compatibility questions, then I think we can move forward again. I'll do that this evening. Paul. From ziade.tarek at gmail.com Mon Jul 6 18:05:49 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 6 Jul 2009 18:05:49 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907060853o22d29679kd702c88e5c3ff46f@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <94bdd2610907060840p3d0c82eakbeb85f742f6979a@mail.gmail.com> <79990c6b0907060853o22d29679kd702c88e5c3ff46f@mail.gmail.com> Message-ID: <94bdd2610907060905w32026c2t9bc5f76b93233f33@mail.gmail.com> On Mon, Jul 6, 2009 at 5:53 PM, Paul Moore wrote: > 2009/7/6 Tarek Ziad? : >> why can't we just go ahead and continue the work as we started with PEP 376, >> introducing your work on PEP 302-like behavior. >> >> Then if we get a consensus on this PEP and introduce it in 2.7/3.2, >> setuptools will have to follow this consensus. > > Essentially, because when I ask questions, responses along the lines > of "you have to do it like X because setuptools does that" come back, > and (not being a setuptools user) I can't tell whether there's a valid > reason in there. Notice that we created PEP 376 in distutils-SIG with most of the valid reasons/use cases setuptools had, with the help of Phillip, before I brought it up again on python-dev. > > I'm uncomfortable assuming that setuptools experience is irrelevant, > but I can't distinguish between valid arguments and "setuptools can't > change" arguments. > > I need to write another email with a list of outstanding open issues. > If we can thrash them out *without* getting bogged down in setuptools > compatibility questions, then I think we can move forward again. I'll > do that this evening. Ok, I'll wait for your work to work on the PEP again then, and your push in the hg repo as well. From pje at telecommunity.com Mon Jul 6 18:51:40 2009 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 06 Jul 2009 12:51:40 -0400 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907060232j526f293dv35ba9ee327e912ae@mail.gmail.co m> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <4A5116DC.1070109@gmail.com> <31B6EE4E-8D3A-4BBC-ABAD-074DBEEC60F9@mac.com> <94bdd2610907060053x58d9b191wac7bfda8743c7fad@mail.gmail.com> <79990c6b0907060232j526f293dv35ba9ee327e912ae@mail.gmail.com> Message-ID: <20090706164840.9E8763A407B@sparrow.telecommunity.com> At 10:32 AM 7/6/2009 +0100, Paul Moore wrote: >I'm +0 on changing the name, as long as it's the *only* "do it this >way because setuptools isn't going to change" issue. Please note that I never said that. I was the one who suggested ".pydist", remember? I just don't want to have to complicate the pkg_resources directory scan to support a stupid name like "metadata", no offense to Tarek. ;-) I'd sooner not make the change at all, just because I doubt I'm the only person who has code that depends on the old name. (Consider the Linux OS packagers who treat .egg-info files specially at the moment.) From pje at telecommunity.com Mon Jul 6 18:58:47 2009 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 06 Jul 2009 12:58:47 -0400 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.co m> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> Message-ID: <20090706165546.25CCB3A407B@sparrow.telecommunity.com> At 04:14 PM 7/6/2009 +0100, Paul Moore wrote: >2009/7/6 Nick Coghlan : > > P.J. Eby wrote: > >> At 08:43 PM 7/5/2009 +0200, Tarek Ziad? wrote: > >>> But if it's based on PEP 302 protocols and if the pkgutil code works > >>> with the sys.meta_path hook, > >>> setuptools could then provide its loader, based on its EggFormats and > >>> act as a provider without being broken. > >> > >> You misunderstand me. The whole point of putting .egg-info in distutils > >> in the first place was to enable setuptools to detect the presence of > >> disutils-installed packages. That's what's broken by changing the name. > >This is getting confusing. Is Phillip saying that setuptools will cope >with the file changing to a directory without modification, but it >can't handle a change in the name? The existing versions of setuptools will read a file or a directory with no problem; it's the name change that will require a code change, and it's a rather more complex issue than just one name change, because it'll need to support both names. What's more, on the build/install side, it'll have to figure out whether to use the new name or the old name when creating a project's metadata for installation in single-version mode. In other words, this will likely affect pip as well, or at least the parts of setuptools that pip uses. >My site-packages has a confusing mix of egginfo directories and files. >Note that I NEVER use setuptools other than where an existing >package's setup.py requires it. In that case, I still only do python >setup.py bdist_wininst and install the generated installer. > >So is PEP 376 going to be able to cope with the stuff I have installed >at the moment? If not, what's the point??? If I understand Tarek's proposal correctly, then no, it will not cope. >If setuptools is not going to change to conform to PEP 376, then any >tools built using PEP 376 will fail to recognise my coverage install. I'm all in favor of adding RECORD support to setuptools; it was in fact my idea to have the file there in the first place. Adding a RECORD file doesn't introduce any new and weird name migration requirements, which is why I'd rather not change the extension if we can avoid it. Reading both names is painful, writing both is more so, and I'm not sure how many tools/users *besides* setuptools will be affected by a name change. > > How much information does setuptools actually need in order to tell that > > a distribution is already present? Presumably the existing .egginfo > > files generated by distutils are sufficient for that task? > >It appears so, but setuptools doesn't use (or create!!!) those files >in its own installer formats. Setuptools treats an .egg-info file as if it were a PKG-INFO file contained in an .egg-info directory. This allows it to treat distutils-supplied .egg-info files as if they were setuptools-supplied .egg-info directories containing exactly one file. From hnassrat at gmail.com Mon Jul 6 18:16:38 2009 From: hnassrat at gmail.com (Hatem Nassrat) Date: Mon, 6 Jul 2009 13:16:38 -0300 Subject: [Python-Dev] Any updates on this subprocess/signal bug/issue (Re: subprocess and EINTR errnos) In-Reply-To: Message-ID: <20090706161638.GA16498@gmail.com> On Wed, 17 Nov 2004, Peter Astrand wrote, > I've made a patch (attached) to subprocess.py (and test_subprocess.py) > that should guard against EINTR, but I haven't committed it yet. It's > quite large. > > Are Python modules supposed to handle EINTR? Why not let the C code handle > this? Or, perhaps the signal module should provide a sigaction function, > so that users can use SA_RESTART. This is a snippet from a email sent in 2004, I was wondering if there was any update on this issue. Are these issues supposed to be handled on a per application basis, or will a fix go into Python in the near future? -- Hatem Nassrat Refrences: http://mail.python.org/pipermail/python-dev/2004-November/049983.html From ziade.tarek at gmail.com Mon Jul 6 19:18:44 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 6 Jul 2009 19:18:44 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <20090706165546.25CCB3A407B@sparrow.telecommunity.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <20090706165546.25CCB3A407B@sparrow.telecommunity.com> Message-ID: <94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com> On Mon, Jul 6, 2009 at 6:58 PM, P.J. Eby wrote: >> My site-packages has a confusing mix of egginfo directories and files. >> Note that I NEVER use setuptools other than where an existing >> package's setup.py requires it. In that case, I still only do python >> setup.py bdist_wininst and install the generated installer. If you end up with confusing mix of egginfo directories and files, that's because for the packages that uses setuptools in their setup.py, it patches distutils and to creates its own egg-info format. >> So is PEP 376 going to be able to cope with the stuff I have installed >> at the moment? If not, what's the point??? > > If I understand Tarek's proposal correctly, then no, it will not cope. Why that ? Can you detail ? On a system that uses only plain distutils distributions, it'll work. From pje at telecommunity.com Mon Jul 6 19:40:31 2009 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 06 Jul 2009 13:40:31 -0400 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.co m> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <20090706165546.25CCB3A407B@sparrow.telecommunity.com> <94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com> Message-ID: <20090706173730.A09F13A407B@sparrow.telecommunity.com> At 07:18 PM 7/6/2009 +0200, Tarek Ziad? wrote: > >> So is PEP 376 going to be able to cope with the stuff I have installed > >> at the moment? If not, what's the point??? > > > > If I understand Tarek's proposal correctly, then no, it will not cope. > >Why that ? Can you detail ? On a system that uses only plain distutils >distributions, it'll work. Paul's question, as I understood it, was whether pkgutil would handle the existing .egg-info's. Your proposal, as I understood it, was that pkgutil would not detect .egg-info's without a plugin of some sort... and you've not proposed a way of loading such a thing, which means that to use pkgutil with that plugin, an application would have to know that the plugin exists and import it. Unless you're planning to have a plugin registration facility (ala entry points) so that plugins can be used without each application needing to discover them on its own? From p.f.moore at gmail.com Mon Jul 6 19:57:52 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Jul 2009 18:57:52 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <20090706165546.25CCB3A407B@sparrow.telecommunity.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <20090706165546.25CCB3A407B@sparrow.telecommunity.com> Message-ID: <79990c6b0907061057h66a82f04ke6f4971209752255@mail.gmail.com> 2009/7/6 P.J. Eby : > At 04:14 PM 7/6/2009 +0100, Paul Moore wrote: >> This is getting confusing. Is Phillip saying that setuptools will cope >> with the file changing to a directory without modification, but it >> can't handle a change in the name? > > The existing versions of setuptools will read a file or a directory with no > problem; it's the name change that will require a code change, and it's a > rather more complex issue than just one name change, because it'll need to > support both names. If PEP 376 is the standard, setuptools can just use that. Whether you have to handle backward compatibility issues so that newer versions of setuptools work with older metadata layouts is largely an internal setuptools issue. While I don't want to gratuitously break setuptools, it already supports something like 3 formats, as I understand it, so adding a 4th shouldn't be the end of the world. > What's more, on the build/install side, it'll have to figure out whether to > use the new name or the old name when creating a project's metadata for > installation in single-version mode. ?In other words, this will likely > affect pip as well, or at least the parts of setuptools that pip uses. And yet, there have been no adverse comments from users of pip, or buildout, or any of the other tools that this PEP is targeted at. So either there's a huge chunk of the affected community that's unaware of this discussion (which I doubt), or they have no problem with this, or their interaction is via setuptools, and so as long as setuptools is changed, they are fine. (The other benefit of the PEP is that if setuptools *doesn't* change, they have alternative tools to build their own interface, although they may not want to be forced to do this...) >> If setuptools is not going to change to conform to PEP 376, then any >> tools built using PEP 376 will fail to recognise my coverage install. > > I'm all in favor of adding RECORD support to setuptools; it was in fact my > idea to have the file there in the first place. > > Adding a RECORD file doesn't introduce any new and weird name migration > requirements, which is why I'd rather not change the extension if we can > avoid it. You don't give any reasons why you pick and choose what changes you're happy with - "weird name migration requirements" isn't very specific. > Reading both names is painful, writing both is more so, and I'm not sure how > many tools/users *besides* setuptools will be affected by a name change. Don't forget - I made that point when I said I wasn't too bothered about what was an internal name change. >> > How much information does setuptools actually need in order to tell that >> > a distribution is already present? Presumably the existing .egginfo >> > files generated by distutils are sufficient for that task? >> >> It appears so, but setuptools doesn't use (or create!!!) those files >> in its own installer formats. > > Setuptools treats an .egg-info file as if it were a PKG-INFO file contained > in an .egg-info directory. ?This allows it to treat distutils-supplied > .egg-info files as if they were setuptools-supplied .egg-info directories > containing exactly one file. Hey, look - you get to *remove* some compatibility code! Paul. From brett at python.org Mon Jul 6 19:49:53 2009 From: brett at python.org (Brett Cannon) Date: Mon, 6 Jul 2009 10:49:53 -0700 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A520F83.7020300@egenix.com> References: <4A520F83.7020300@egenix.com> Message-ID: On Mon, Jul 6, 2009 at 07:51, M.-A. Lemburg wrote: > Dirkjan Ochtman wrote: > > In response to some rumblings on python-committers and just to request > > more feedback, a progress report. I know it's long, I've tried to put > > to keep it concise and chunked, though. > > Two things: > > * We need some form of documentation of how committers are expected > to work with the hg repo. (This is also missing for the Subversion > repo, which due to the 3.x branch has gotten somewhat unclear - at > least for me) > I am planning to get a version of the dev FAQ written up that covers most of what it already does now for svn. > > It is currently not clear where to check in patches, whether and > where to backport or forward-patch, which branches to consider > closed, etc. > > E.g. if I check in something in trunk/ (Python 2.7), do I have to > forward patch this change to the 3.0 branch (guess not), the py3k/ > branch (Python 3.1), or will someone else take care of this, so that > it's better not to interfere by doing it myself ? > This question is partially answered by http://www.python.org/dev/faq/#how-do-i-merge-between-branches, but I agree that we should have either this spelled out in the FAQ or a committer-specific doc at www.python.org/dev/ that makes this all very obvious. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Jul 6 20:15:52 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Jul 2009 19:15:52 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <20090706173730.A09F13A407B@sparrow.telecommunity.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <20090706165546.25CCB3A407B@sparrow.telecommunity.com> <94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com> <20090706173730.A09F13A407B@sparrow.telecommunity.com> Message-ID: <79990c6b0907061115t7e674e23t367deed496890b86@mail.gmail.com> 2009/7/6 P.J. Eby : > At 07:18 PM 7/6/2009 +0200, Tarek Ziad? wrote: >> >> >> So is PEP 376 going to be able to cope with the stuff I have installed >> >> at the moment? If not, what's the point??? >> > >> > If I understand Tarek's proposal correctly, then no, it will not cope. >> >> Why that ? Can you detail ? On a system that uses only plain distutils >> distributions, it'll work. > > Paul's question, as I understood it, was whether pkgutil would handle the > existing .egg-info's. ?Your proposal, as I understood it, was that pkgutil > would not detect .egg-info's without a plugin of some sort... ?and you've > not proposed a way of loading such a thing, which means that to use pkgutil > with that plugin, an application would have to know that the plugin exists > and import it. Close. My point was that distributions which use setuptools in their setup.py, *even if there's no runtime dependency on setuptools* end up with non-standard .egg-info's. There is no good reason for this, from my POV as a package user. So if setuptools is breaking otherwise entirely standard packages, so that they won't work with core distutils mechanisms, that's wrong. Setuptools is refusing to play nicely with other packaging systems. Now at the moment, this is a moot point, because the stdlib doesn't have any package discovery mechanisms. But when they *do* appear, I expect setuptools to work with them. So, when PEP 376 is accepted and implemented, I contend that, in order to be a well-behaved packaging tool, setuptools must: - Generate PEP 376 compatible metadata - Recognise PEP 376 compatible metadata - Unless installing a custom package format (such as eggs), which has its own PEP 302 installer, refrain from installing hooks that alter the way the standard distutils commands handle metadata. That's my view as an end user of Python packages. It means that if I choose not to use setuptools/easy_install to manage my packages, I shouldn't need them so that distutils or other core utilities (such as bdist_wininst) can do so. > Unless you're planning to have a plugin registration facility (ala entry > points) so that plugins can be used without each application needing to > discover them on its own? My understanding was that package management tools provide their own end-user level install and uninstall features (the PEP 376 uninstall is just for filesystems, to correspond to the distutils install command). PEP 376 is just standardising the format of metadata, and the APIs to allow read access. The PEP 302 hooks I'm writing provide the "plugin" ability for alternative storage formats. But there's no provision for alternative *metadata* formats. What's the point of a standard if people don't follow it? Paul. From p.f.moore at gmail.com Mon Jul 6 20:38:24 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Jul 2009 19:38:24 +0100 Subject: [Python-Dev] PEP 376 - Open questions Message-ID: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> As promised, here are some open questions on PEP 376. - Will the public API names be changed from *egginfo* to *metadata*? - What precisely are the use cases for absolute path names? Concrete examples are needed. With the current spec, some things can go wrong (e.g., see below), so we need real use cases to know how to address this. - How will bdist_wininst/bdist_msi/bdist_rpm be updated? - How will the RECORD file be managed? (Particularly for the case of bdist_xxx) [1] - Can distutils be made to install files in places the current RECORD file spec can't handle? (I think the answer is "yes"). What happens then? - Should distribution names be case insensitive on case insensitive filesystems? For comparison, module/package names are always case sensitive even on case insensitive systems. - What will happen with the md5 hash? Are more types of hash going to be supported? What's the default? (Actually, the PEP doesn't need to care about the default, as the PEP says nothing about how RECORD files are written). [1] Note - the idea of using $EXEC_PREFIX / $PREFIX implies that the RECORD file is intended to be relocatable. Which is worrying, and also implies that an individual Distribution class must be able to handle filesystem files as well as whatever else it handles (consider mylib.zip on sys.path, containing a distribution which installed some files in $PREFIX). If this isn't possible, it should be clearly stated that it isn't possible. If it is, the ramifications are complex... I'm still unsure how the local vs relative, slash-separated filename formats should be handled. I don't actually think there's any real benefit in having 2 formats. I propose: - get_installed_files(), uses(), get_file_users() - always use local format absolute pathnames (for zipfiles and the like, these may not be "real" filenames, but they will be in "real filename" format, so other code will be able to manipulate them as filenames). - get_egginfo_files, get_egginfo_file - always use slash-separated forms, relative to the egginfo directory (so the name of the RECORD file is just 'RECORD') But there could be uses I haven't thought about, so this still counts as an open question at the moment (i.e., I'm reluctant to implement things this way until I've had some feedback). Paul. From hnassrat at gmail.com Mon Jul 6 20:18:40 2009 From: hnassrat at gmail.com (Hatem Nassrat) Date: Mon, 6 Jul 2009 15:18:40 -0300 Subject: [Python-Dev] Any updates on this subprocess/signal bug/issue (Re: subprocess and EINTR errnos) In-Reply-To: Message-ID: <20090706181840.GA19211@gmail.com> On Wed, 17 Nov 2004, Peter Astrand wrote, > I've made a patch (attached) to subprocess.py (and test_subprocess.py) > that should guard against EINTR, but I haven't committed it yet. It's > quite large. > > Are Python modules supposed to handle EINTR? Why not let the C code handle > this? Or, perhaps the signal module should provide a sigaction function, > so that users can use SA_RESTART. This is a snippet from a email sent in 2004, I was wondering if there was any update on this issue. Are these issues supposed to be handled on a per application basis, or will a fix go into Python in the near future? -- Hatem Nassrat Refrences: http://mail.python.org/pipermail/python-dev/2004-November/049983.html From brett at python.org Mon Jul 6 19:33:25 2009 From: brett at python.org (Brett Cannon) Date: Mon, 6 Jul 2009 10:33:25 -0700 Subject: [Python-Dev] semi-regular check that all committers are subscribed to python-committers In-Reply-To: References: Message-ID: Since no one objected, I will give Dirkjan commit privs so he can work on the hg migration. Now I just need you, Dirkjan, to get a contributor agreement sent in and then send python-dev your SSH2 key and we will get you set up. On Thu, Jul 2, 2009 at 13:08, Brett Cannon wrote: > > > On Thu, Jul 2, 2009 at 12:56, Dirkjan Ochtman wrote: > >> On Thu, Jul 2, 2009 at 21:04, Brett Cannon wrote: >> > If you have commit privileges on Python but are not subscribed to >> > python-committers, please let me know and I will subscribe you. >> Membership >> > on python-committers is mandatory as that's where we announce branch >> > freezes, etc. >> >> Would it be helpful if I was subscribed? I'm not strictly a committer, >> but what with the Mercurial migration I imagine it won't be long >> before the first cset bearing my name appears in the Python repository >> (plus I just saw there was some discussion about the hg migration >> there which I could weigh in on). > > > Maybe you should just get the commit privileges now. Any objections? > > -Brett > -------------- next part -------------- An HTML attachment was scrubbed... URL: From aahz at pythoncraft.com Mon Jul 6 21:29:45 2009 From: aahz at pythoncraft.com (Aahz) Date: Mon, 6 Jul 2009 12:29:45 -0700 Subject: [Python-Dev] Any updates on this subprocess/signal bug/issue (Re: subprocess and EINTR errnos) In-Reply-To: <20090706161638.GA16498@gmail.com> References: <20090706161638.GA16498@gmail.com> Message-ID: <20090706192945.GA17222@panix.com> On Mon, Jul 06, 2009, Hatem Nassrat wrote: > On Wed, 17 Nov 2004, Peter Astrand wrote, >> >> I've made a patch (attached) to subprocess.py (and test_subprocess.py) >> that should guard against EINTR, but I haven't committed it yet. It's >> quite large. >> >> Are Python modules supposed to handle EINTR? Why not let the C code handle >> this? Or, perhaps the signal module should provide a sigaction function, >> so that users can use SA_RESTART. > > This is a snippet from a email sent in 2004, I was wondering if there > was any update on this issue. Are these issues supposed to be handled on > a per application basis, or will a fix go into Python in the near > future? For starters, if there is to be any progress, there needs to be an open issue on bugs.python.org -- have you searched to see if one already exists and created one if it doesn't? -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "as long as we like the same operating system, things are cool." --piranha From pje at telecommunity.com Mon Jul 6 22:55:38 2009 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 06 Jul 2009 16:55:38 -0400 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907061115t7e674e23t367deed496890b86@mail.gmail.co m> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <20090706165546.25CCB3A407B@sparrow.telecommunity.com> <94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com> <20090706173730.A09F13A407B@sparrow.telecommunity.com> <79990c6b0907061115t7e674e23t367deed496890b86@mail.gmail.com> Message-ID: <20090706205237.014663A407B@sparrow.telecommunity.com> At 07:15 PM 7/6/2009 +0100, Paul Moore wrote: >My point was that distributions which use setuptools in their >setup.py, *even if there's no runtime dependency on setuptools* end up >with non-standard .egg-info's. There is no good reason for this, from >my POV as a package user. So if setuptools is breaking otherwise >entirely standard packages, so that they won't work with core >distutils mechanisms, that's wrong. Setuptools is refusing to play >nicely with other packaging systems. > >Now at the moment, this is a moot point, because the stdlib doesn't >have any package discovery mechanisms. Not so; it has .egg-info. (See footnote below.) >The PEP 302 hooks I'm writing provide the "plugin" ability for >alternative storage formats. But there's no provision for alternative >*metadata* formats. What's the point of a standard if people don't >follow it? I'm confused. Are you saying pkgutil will recognize .egg-info, or not? The only practical difference between the existing formats and the proposed new one (AFAICR) is the extension of the directory name, and the presence or absence of the RECORD file. My POV is that it would be a shame to discard compatibility with existing installed software, just for a bikeshedding of the directory name. Especially if it means that any tools that currently generate .egg-info's (and I'm not 100% positive setuptools is the only case) will need to grow options for whether to generate the old name, possibly based on what Python version... except you can install new distutils into old Python... etc. If this were strictly a matter of, "from Python 3.x on, generate .pydist", it's a lot less of a problem for me and everybody else, because Python 3 is effectively going to force a fork (or at least a branch) of setuptools anyway. For Python 2.x, having *two* names for .egg-info (which is what I understand the proposal to effectively be) is just asking for trouble, IMO. And my concern is that currently these compatibility issues are being handwaved rather than considered up front. To reiterate, summarize, and attempt to make as concrete as I can: I'm fine with a rename, *if and only if* it can be done in a way that never requires a tool to generate BOTH files for compatibility reasons. AFAICT, this can only be done by making the extension dependent upon the Python version, and this version distinction must be made *even if* the distutils themselves are backported to an earlier Python version. (Alternatively, this could be resolved by *not* backporting the PEP 376-compatible distutils/pkgutil to 2.x.) Currently, there is no official Python 3 release of setuptools, so compatibility can (IMO) be considered moot there. What's more, it is easy to conditionally determine whether to use '.egg-info' or '.pydist' based on Python version. What is not easy and likely to be confusing for everyone, is having the possibility for *both* to exist in the same system, *without* also having bidirectional interoperability. So, I'm +1 for no change (obviously), and +0 for "always .pydist in 3.x and always .egg-info in 2.x", whether the latter part is achieved by making distutils/pkgutil use a version-dependent extension, or by refusing to backport distutils/pkgutil to 2.x. I'm -1 for having .pydist ever appear on a 2.x Python, either via stdlib or backporting. (These comments also apply for any name other than '.pydist'.) Footnote: there has been some discussion that implies that somehow .egg-info was hijacked into the stdlib, and that it should therefore not be supported. This is incorrect: at Pycon the year this happened (2005 IIRC), Guido *blessed setuptools for inclusion in the stdlib* in his keynote. Yes, *all* of setuptools, including easy_install, eggs, the whole nine yards. A few months later, when I actually checked in the code, there were protests from a few people on Python-Dev, some of which I agreed were valid. I reverted the checkin, floating instead a compromise proposal to add .egg-info support to distutils (i.e. the install_egg_info command) to allow setuptools and other 3rd party tools to detect the presence of distutils-installed packages. At the time, the assumption was that eggs and setuptools would still, eventually, make it into the stdlib -- at the time, Guido had not yet changed his mind about that. So the ".egg-info" name was not controversial, to the best of my recollection, and the inclusion did not violate any of the expressed concerns of the three persons who objected to including setuptools whole-hog. So, from my POV at least, dropping .egg-info is dropping backward compatibility on a feature that was *intentionally* added to the stdlib -- with approval from Guido and non-protest from Python-Dev -- specifically to provide that compatibility. That doesn't obligate it to stay... but I suggest that it *does* mean Python 3, not 2, is the appropriate place for dropping it (if it is indeed to be dropped)... and that it is not appropriate to do so in 2.x on merely a handwave by persons not involved in the previous discussions. (And of course Guido can Pronounce otherwise; I'm just saying I don't think anything less than such a pronouncement is required to remove it from 2.x. And again, I'm fine with making a clean break of it in 3.x, since setuptools is already broken there anyway.) From pje at telecommunity.com Mon Jul 6 23:07:05 2009 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 06 Jul 2009 17:07:05 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.co m> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> Message-ID: <20090706210405.4BFD63A407B@sparrow.telecommunity.com> At 07:38 PM 7/6/2009 +0100, Paul Moore wrote: >As promised, here are some open questions on PEP 376. > >- Will the public API names be changed from *egginfo* to *metadata*? +1 (FWIW, 'metadata' is what pkg_resources API refers to this kind of stuff as.) >- What precisely are the use cases for absolute path names? Concrete >examples are needed. With the current spec, some things can go wrong >(e.g., see below), so we need real use cases to know how to address >this. The purpose is to be able to remove scripts, data files, documentation, etc. that's stored in some place that's not package-relative, and survive a move of the package directory itself. For example, suppose I install with --prefix=~, and have files in ~/bin and ~/lib. I then move ~/lib/python2.7 to ~/pylib. If the scripts were installed with a relative path, this would break the link to the script still living in ~/bin. Now that I'm thinking about this, it occurs to me that there is another way to address this besides mixing relative and absolute paths in RECORD, and that is to make *all* the paths relative, and include an original installation location in INSTALLER. Then, it would be possible to detect whether the installation location had been moved and handle it accordingly... for some definition of "accordingly". I agree that further discussion is probably needed to resolve this. >- How will bdist_wininst/bdist_msi/bdist_rpm be updated? bdist_wininst, bdist_dumb, and various others use 'install --root' pattern to generate files for installation, which means that they would transparently end up writing a correct RECORD file, except for the inclusion of incorrect absolute paths for non-libdir-relative files. However, if we used the "all relative in RECORD, with a base in INSTALLER", these cases could transparently be treated as another instance of install directory relocation. I don't know if bdist_msi does a --root install before generating the .msi; if it does, then it should work the same way. >- Can distutils be made to install files in places the current RECORD >file spec can't handle? (I think the answer is "yes"). What happens >then? Surely an absolute path is sufficient? >- Should distribution names be case insensitive on case insensitive >filesystems? For comparison, module/package names are always case >sensitive even on case insensitive systems. IMO, they should be case-insensitive *regardless* of filesystem case, and I believe PyPI has already begun disallowing packages with only case-distinct names (e.g. 'Zope' vs. 'ZoPe') to be registered. From ncoghlan at gmail.com Mon Jul 6 23:22:25 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 07 Jul 2009 07:22:25 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A520F83.7020300@egenix.com> References: <4A520F83.7020300@egenix.com> Message-ID: <4A526B11.7060308@gmail.com> M.-A. Lemburg wrote: > Dirkjan Ochtman wrote: > * Our tracker, checkin messages, online documentation and lots of > Python resources on the web are full of references to the > Subversion rXYZ notation of changesets. > > The PEP has to provide some way to gracefully provide a redirect > from those URLs to the new ones (both for the source code browser > and the bug search tool on python.org). > > The PEP will also have to address the same issue for checkin > messages. Perhaps by converting the rXYZ notation to the hash > based values during migration or by adding the has value after > the rXYZ value in parens ?! I've asked this question before: Dirkjan indicated that he will be writing a web redirection filter that translates from the SVN rXYZ revision numbers to the Hg hash value that corresponds to that revision in the history conversion. I suggest that we also run a version of that redirection filter to remap the old svn.python.org links in addition to making them accessible through hg.python.org as Dirkjan proposed. Cheers, Nick. [1] http://mail.python.org/pipermail/python-dev/2009-July/090397.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Mon Jul 6 23:43:12 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 07 Jul 2009 07:43:12 +1000 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <20090706205237.014663A407B@sparrow.telecommunity.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <20090706165546.25CCB3A407B@sparrow.telecommunity.com> <94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com> <20090706173730.A09F13A407B@sparrow.telecommunity.com> <79990c6b0907061115t7e674e23t367deed496890b86@mail.gmail.com> <20090706205237.014663A407B@sparrow.telecommunity.com> Message-ID: <4A526FF0.10204@gmail.com> P.J. Eby wrote: > So, I'm +1 for no change (obviously), and +0 for "always .pydist in 3.x > and always .egg-info in 2.x", whether the latter part is achieved by > making distutils/pkgutil use a version-dependent extension, or by > refusing to backport distutils/pkgutil to 2.x. I'm -1 for having > .pydist ever appear on a 2.x Python, either via stdlib or backporting. > > (These comments also apply for any name other than '.pydist'.) Thanks for the explanation - Before this message I didn't fully grasp *how* installing a directory containing files that the other packages might not understand would help with the backwards compatibility problem. Now I understand that aspect better (i.e. it isn't a matter of "setuptools won't change" but "the user might already have an old version of setuptools/pip/etc on their system, or be installing a package that uses an old version of one of those libraries for its installer"), I no longer think it is worth the hassle of trying to use a "prettier" name for the metadata directory. So +1 on keeping the same extension name and -1 on the idea of changing it to anything other than the existing ".egg-info". I believe the idea of having different names in 2.x and 3.x would likely cause too many problems for simple bdist_* installers of modules that use only the common 2.x/3.x language subset, so I'm also -1 on that idea. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Mon Jul 6 23:54:58 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 07 Jul 2009 07:54:58 +1000 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> Message-ID: <4A5272B2.8000202@gmail.com> (I cancelled sending this the first time, so apologies if a half-written version turns up) Paul Moore wrote: > As promised, here are some open questions on PEP 376. I'd add one more question to the list: is allowing backslash separated names in the RECORD file actually a good idea, or would it be better to always use forward slashes? My reason for this question is what happens if (for example) a bdist_* installer is generated on Linux and then used on Windows or vice-versa? If the expected RECORD format is different on the two platforms, won't it get things wrong? For the other questions, I don't have anything much to add to PJE's comments, except that the "all relative" paths idea won't work due to the Windows drive letter issue (i.e. if an installer puts files in C:\Program Files, there is no guarantee that a relative path between site-packages and Program Files even exists if Python is installed on a different drive). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Mon Jul 6 23:57:55 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 06 Jul 2009 23:57:55 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A526B11.7060308@gmail.com> References: <4A520F83.7020300@egenix.com> <4A526B11.7060308@gmail.com> Message-ID: <4A527363.7070305@v.loewis.de> > I suggest that we also run a version of that redirection filter to remap > the old svn.python.org links in addition to making them accessible > through hg.python.org as Dirkjan proposed. Not sure what links you are talking about, but we also need to keep the current svn.python.org up as-is, for all the stuff that won't be migrated. Regards, Martin From ncoghlan at gmail.com Tue Jul 7 00:05:08 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 07 Jul 2009 08:05:08 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A527363.7070305@v.loewis.de> References: <4A520F83.7020300@egenix.com> <4A526B11.7060308@gmail.com> <4A527363.7070305@v.loewis.de> Message-ID: <4A527514.5040709@gmail.com> Martin v. L?wis wrote: >> I suggest that we also run a version of that redirection filter to remap >> the old svn.python.org links in addition to making them accessible >> through hg.python.org as Dirkjan proposed. > > Not sure what links you are talking about, but we also need to keep the > current svn.python.org up as-is, for all the stuff that won't be migrated. In that case, no need to redirect anything (although it will still be possible to look up the old SVN revisions in the Hg web view). The previous discussion was based on the assumption that it was going to be possible to switch the SVN server off at some point rather than just disallowing commits to the projects that had migrated to Mercurial. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From p.f.moore at gmail.com Tue Jul 7 00:16:56 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Jul 2009 23:16:56 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <4A526FF0.10204@gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <20090706165546.25CCB3A407B@sparrow.telecommunity.com> <94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com> <20090706173730.A09F13A407B@sparrow.telecommunity.com> <79990c6b0907061115t7e674e23t367deed496890b86@mail.gmail.com> <20090706205237.014663A407B@sparrow.telecommunity.com> <4A526FF0.10204@gmail.com> Message-ID: <79990c6b0907061516h4a3fe11epc7793cbcd93222b2@mail.gmail.com> 2009/7/6 Nick Coghlan : > P.J. Eby wrote: >> So, I'm +1 for no change (obviously), and +0 for "always .pydist in 3.x >> and always .egg-info in 2.x", whether the latter part is achieved by >> making distutils/pkgutil use a version-dependent extension, or by >> refusing to backport distutils/pkgutil to 2.x. ?I'm -1 for having >> .pydist ever appear on a 2.x Python, either via stdlib or backporting. >> >> (These comments also apply for any name other than '.pydist'.) > > Thanks for the explanation - Before this message I didn't fully grasp > *how* installing a directory containing files that the other packages > might not understand would help with the backwards compatibility problem. > > Now I understand that aspect better (i.e. it isn't a matter of > "setuptools won't change" but "the user might already have an old > version of setuptools/pip/etc on their system, or be installing a > package that uses an old version of one of those libraries for its > installer"), I no longer think it is worth the hassle of trying to use a > "prettier" name for the metadata directory. So +1 on keeping the same > extension name and -1 on the idea of changing it to anything other than > the existing ".egg-info". > > I believe the idea of having different names in 2.x and 3.x would likely > cause too many problems for simple bdist_* installers of modules that > use only the common 2.x/3.x language subset, so I'm also -1 on that idea. That suits me too. I'm happy for the name to stay as it is. Paul. From p.f.moore at gmail.com Tue Jul 7 00:36:08 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Jul 2009 23:36:08 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <20090706210405.4BFD63A407B@sparrow.telecommunity.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <20090706210405.4BFD63A407B@sparrow.telecommunity.com> Message-ID: <79990c6b0907061536q418fec7ep1d4e555f0ee6266f@mail.gmail.com> 2009/7/6 P.J. Eby : > At 07:38 PM 7/6/2009 +0100, Paul Moore wrote: >> >> As promised, here are some open questions on PEP 376. >> >> - Will the public API names be changed from *egginfo* to *metadata*? > > +1 (FWIW, 'metadata' is what pkg_resources API refers to this kind of stuff > as.) Thanks. I think this one is pretty non-controversial. >> - What precisely are the use cases for absolute path names? Concrete >> examples are needed. With the current spec, some things can go wrong >> (e.g., see below), so we need real use cases to know how to address >> this. > > The purpose is to be able to remove scripts, data files, documentation, etc. > that's stored in some place that's not package-relative, and survive a move > of the package directory itself. > > For example, suppose I install with --prefix=~, and have files in ~/bin and > ~/lib. ?I then move ~/lib/python2.7 to ~/pylib. ?If the scripts were > installed with a relative path, this would break the link to the script > still living in ~/bin. > > Now that I'm thinking about this, it occurs to me that there is another way > to address this besides mixing relative and absolute paths in RECORD, and > that is to make *all* the paths relative, and include an original > installation location in INSTALLER. ?Then, it would be possible to detect > whether the installation location had been moved and handle it > accordingly... ?for some definition of "accordingly". > > I agree that further discussion is probably needed to resolve this. I'm not 100% sure I follow your logic here. If you're renaming directories after install, without involving the package manager, I think you probably deserve whatever you get :-) Having said that, your proposal (all paths relative, plus an install location) is similar to something I mentioned earlier (but better explained :-)) The only case it doesn't cater for is when an installed file can't be expressed as relative to the "install location" (which would involve being on a different drive on Windows). That's likely to never work, so I doubt it counts in any case. I've got some concerns about the possibility of having locations "above" the install location - something like "../../scripts/driver.bat". I'm not sure if such things will work (consider if the install location is a zipfile or egg) but nor am I sure anything generates such names. As you say, more discussion (and use cases!) would help here. >> - How will bdist_wininst/bdist_msi/bdist_rpm be updated? > > bdist_wininst, bdist_dumb, and various others use 'install --root' pattern > to generate files for installation, which means that they would > transparently end up writing a correct RECORD file, except for the inclusion > of incorrect absolute paths for non-libdir-relative files. ?However, if we > used the "all relative in RECORD, with a base in INSTALLER", these cases > could transparently be treated as another instance of install directory > relocation. > > I don't know if bdist_msi does a --root install before generating the .msi; > if it does, then it should work the same way. Yes, that sounds like a plausible approach. >> - Can distutils be made to install files in places the current RECORD >> file spec can't handle? (I think the answer is "yes"). What happens >> then? > > Surely an absolute path is sufficient? Current spec says relative or $PREFIX/... or $EXEC_PREFIX/... I think that was based on earlier discussion, but it's hampered by the fact that nobody's pointing at a real case of a distribution that actually tries to *install* files in odd places. (Actually, something that installs a driver script, like twisted's twistd or nose's nosetests, would do - I'll see what I can find tomorrow). >> - Should distribution names be case insensitive on case insensitive >> filesystems? For comparison, module/package names are always case >> sensitive even on case insensitive systems. > > IMO, they should be case-insensitive *regardless* of filesystem case, and I > believe PyPI has already begun disallowing packages with only case-distinct > names (e.g. 'Zope' vs. 'ZoPe') to be registered. OK, that's another option :-) It's not hard to change the code, so as long as a final decision is reached at some point, we're OK here... Thanks for the comments, Paul. From p.f.moore at gmail.com Tue Jul 7 00:44:09 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Jul 2009 23:44:09 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A5272B2.8000202@gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> Message-ID: <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> 2009/7/6 Nick Coghlan : > I'd add one more question to the list: is allowing backslash separated > names in the RECORD file actually a good idea, or would it be better to > always use forward slashes? They do always use forward slashes. > For the other questions, I don't have anything much to add to PJE's > comments, except that the "all relative" paths idea won't work due to > the Windows drive letter issue (i.e. if an installer puts files in > C:\Program Files, there is no guarantee that a relative path between > site-packages and Program Files even exists if Python is installed on a > different drive). The big question, though, is can an installer actually *do* that in practical terms? - There are *no* guaranteed absolute locations on Windows, so any such oddly-located file would require user interaction to work. Certainly bdist_wininst and bdist_msi don't do that. - My experiments indicate that bdist_{wininst,msi} are broken with respect to absolute paths anyway: they do a --root install to a temporary directory (and the absolute paths don't end up in there) and then package up that temporary directory. I still want to see a real life example that demonstrates that there is a genuine issue here. We're spending a lot of energy and complexity trying to design a solution to a problem that actually doesn't appear to exist in practice... (To be honest, I'd be fairly confident in saying that absolute paths can be ignored on Windows, subject to some corner cases that I haven't thought through yet. My worry is that I don't know what Unix and Mac users might do, so I can't just wish away the issue because it can't arise on Windows. Can a Unix/Mac user offer a real-world example on their own system?) Paul. From ben+python at benfinney.id.au Tue Jul 7 03:45:19 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 07 Jul 2009 11:45:19 +1000 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> Message-ID: <87ws6lp8pc.fsf@benfinney.id.au> Paul Moore writes: > In fact, the above strongly suggests to me that PEP 376 must NOT > follow setuptools conventions - otherwise, it will end up clashing > with the files setuptools is putting on my system. I think the argument for a separate ?.pydist? metadata directory, normative in PEP 376, is a good one. > > If so, I'd still prefer to keep the new metadata safely out of reach > > of the legacy package management systems that don't understand it, > > while having distutils retain the ability to generate a legacy > > ".egginfo" file to warn off the existing package management tools > > from installing the same distribution again. > > So for a minimal case of a single .py file packaged up as a > distribution, we now have the .py file, a legacy .egginfo file, a new > .pydist directory containing RECORD and PKG-INFO files? > > That's getting silly. The burden of switching is that we need a compatibility bridge. If that means that a tool must support the old while also supporting the new, that's not silliness in the tool, but acknowledgement of the silliness we're migrating away from. That doesn't mean, though, that PEP 376 needs to mention the old format at all; just that distutils should support it, deprecated, until a time when it's deemed safe to remove. In fact, I think it's best that PEP 376 discuss *only* the format we're trying to migrate to, and support for existing formats is firmly outside its scope. > And yet, given that PEP 376 is explicitly being designed with a goal > being to act as the common standard that *all* package management > formats can use, is it not the whole point that once it's defined and > we have achieved consensus, existing package managers will switch to > using it rather than retaining a range of custom formats? Again, that needs to be a gradual process. Supporting a new format as defined in PEP 376 needs not (indeed, I would argue *should* not) mean the tool drop support for old formats immediately. There needs to be a compatibility chain to allow for fast migration *and* slow migration, since there will be a broad mix of both. Let PEP 376 discuss a format that is clean and new, but let discussions leading to that format acknowledge that it must allow (*not* require) parallel support of old formats for some time. -- \ ?I love to go down to the schoolyard and watch all the little | `\ children jump up and down and run around yelling and screaming. | _o__) They don't know I'm only using blanks.? ?Emo Philips | Ben Finney From p.f.moore at gmail.com Tue Jul 7 09:15:19 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Jul 2009 08:15:19 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <87ws6lp8pc.fsf@benfinney.id.au> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <87ws6lp8pc.fsf@benfinney.id.au> Message-ID: <79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com> 2009/7/7 Ben Finney : > Paul Moore writes: > >> In fact, the above strongly suggests to me that PEP 376 must NOT >> follow setuptools conventions - otherwise, it will end up clashing >> with the files setuptools is putting on my system. > > I think the argument for a separate ?.pydist? metadata directory, > normative in PEP 376, is a good one. I've been convinced otherwise. There's no long-term benefit (the directory layout and format is the same, so it's a purely cosmetic change) and we're doing no favours to anyone just adding a new format for the sake of it. Paul. From ziade.tarek at gmail.com Tue Jul 7 09:21:39 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Jul 2009 09:21:39 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907061516h4a3fe11epc7793cbcd93222b2@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <20090706165546.25CCB3A407B@sparrow.telecommunity.com> <94bdd2610907061018r2041b96asb06bcc191e505e8f@mail.gmail.com> <20090706173730.A09F13A407B@sparrow.telecommunity.com> <79990c6b0907061115t7e674e23t367deed496890b86@mail.gmail.com> <20090706205237.014663A407B@sparrow.telecommunity.com> <4A526FF0.10204@gmail.com> <79990c6b0907061516h4a3fe11epc7793cbcd93222b2@mail.gmail.com> Message-ID: <94bdd2610907070021s4d90dd4fw55e5961e597b1afb@mail.gmail.com> On Tue, Jul 7, 2009 at 12:16 AM, Paul Moore wrote: >> >> I believe the idea of having different names in 2.x and 3.x would likely >> cause too many problems for simple bdist_* installers of modules that >> use only the common 2.x/3.x language subset, so I'm also -1 on that idea. > > That suits me too. I'm happy for the name to stay as it is. While I'd rather like having an "egg"-free standard, it's a detail compared to the rest, so I'm 0+ From ziade.tarek at gmail.com Tue Jul 7 09:30:39 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Jul 2009 09:30:39 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907061536q418fec7ep1d4e555f0ee6266f@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <20090706210405.4BFD63A407B@sparrow.telecommunity.com> <79990c6b0907061536q418fec7ep1d4e555f0ee6266f@mail.gmail.com> Message-ID: <94bdd2610907070030vfe98bc8jeb4503a6f40c5bc1@mail.gmail.com> 2009/7/7 Paul Moore : > 2009/7/6 P.J. Eby : >> At 07:38 PM 7/6/2009 +0100, Paul Moore wrote: >>> >>> As promised, here are some open questions on PEP 376. >>> >>> - Will the public API names be changed from *egginfo* to *metadata*? >> >> +1 (FWIW, 'metadata' is what pkg_resources API refers to this kind of stuff >> as.) > > Thanks. I think this one is pretty non-controversial. +1, I'll change the PEP accordingly; >>> - How will bdist_wininst/bdist_msi/bdist_rpm be updated? >> >> bdist_wininst, bdist_dumb, and various others use 'install --root' pattern >> to generate files for installation, which means that they would >> transparently end up writing a correct RECORD file, except for the inclusion >> of incorrect absolute paths for non-libdir-relative files. ?However, if we >> used the "all relative in RECORD, with a base in INSTALLER", these cases >> could transparently be treated as another instance of install directory >> relocation. >> >> I don't know if bdist_msi does a --root install before generating the .msi; >> if it does, then it should work the same way. > > Yes, that sounds like a plausible approach. As I said earlier in the previous thread, all these commands rely on a call to the install command, so bdist_msi does too. From mal at egenix.com Tue Jul 7 09:50:34 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 07 Jul 2009 09:50:34 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <87ws6lp8pc.fsf@benfinney.id.au> <79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com> Message-ID: <4A52FE4A.6020204@egenix.com> Paul Moore wrote: > 2009/7/7 Ben Finney : >> Paul Moore writes: >> >>> In fact, the above strongly suggests to me that PEP 376 must NOT >>> follow setuptools conventions - otherwise, it will end up clashing >>> with the files setuptools is putting on my system. >> I think the argument for a separate ?.pydist? metadata directory, >> normative in PEP 376, is a good one. > > I've been convinced otherwise. There's no long-term benefit (the > directory layout and format is the same, so it's a purely cosmetic > change) and we're doing no favours to anyone just adding a new format > for the sake of it. The PEP should take the chance to define not only the directory, but also the complete set of files in there and their format. A typical setuptools dir looks like this: dependency_links.txt namespace_packages.txt not-zip-safe PKG-INFO requires.txt SOURCES.txt top_level.txt Ie. a complete mess of upper-case/lower-case names, names with underscores or hyphens as word separator, files with extensions, files without them. This needs some serious cleanup. A new dir name will allow to do this, so +1 on a new dir name. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 07 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ziade.tarek at gmail.com Tue Jul 7 09:50:50 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Jul 2009 09:50:50 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> Message-ID: <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> 2009/7/7 Paul Moore : > 2009/7/6 Nick Coghlan : >> I'd add one more question to the list: is allowing backslash separated >> names in the RECORD file actually a good idea, or would it be better to >> always use forward slashes? > > They do always use forward slashes. > >> For the other questions, I don't have anything much to add to PJE's >> comments, except that the "all relative" paths idea won't work due to >> the Windows drive letter issue (i.e. if an installer puts files in >> C:\Program Files, there is no guarantee that a relative path between >> site-packages and Program Files even exists if Python is installed on a >> different drive). > > The big question, though, is can an installer actually *do* that in > practical terms? > > - There are *no* guaranteed absolute locations on Windows, so any such > oddly-located file would require user interaction to work. Certainly > bdist_wininst and bdist_msi don't do that. > - My experiments indicate that bdist_{wininst,msi} are broken with > respect to absolute paths anyway: they do a --root install to a > temporary directory (and the absolute paths don't end up in there) and > then package up that temporary directory. yes that's unfortunately the case for all windows-based installation. wether it's a bdist call to the install command, to create a binary package, wether it's an installation. c:\something or d:\something will be installed in sys.prefix\something. I will add an issue for distutils for this, probably ending up raising an exception when we hit this case, because I don't see how these paths can work. Unless we define a "drive that contains the python installation" maybe, or the "Program Files" directory would that make sense from a win32 point of view ? > > I still want to see a real life example that demonstrates that there > is a genuine issue here. We're spending a lot of energy and complexity > trying to design a solution to a problem that actually doesn't appear > to exist in practice... > > (To be honest, I'd be fairly confident in saying that absolute paths > can be ignored on Windows, subject to some corner cases that I haven't > thought through yet. My worry is that I don't know what Unix and Mac > users might do, so I can't just wish away the issue because it can't > arise on Windows. Can a Unix/Mac user offer a real-world example on > their own system?) > I know some people are writing to /etc to add their configuration file on the system, So a real-world example under linux would be: setup(..., data_files=[('/etc', ['myconf.cfg'])], ...) That is basically how the examples are shown at: http://docs.python.org/distutils/setupscript.html#installing-additional-files But this is already os-specific, and exists because distutils doesn't have a way (yet) to express systems locations independantly from their physical location, like what the RPM system does with %VARIABLES. So another way to handle this maybe, like I have added with $PREFIX and $EXEC_PREFIX would be to nominate a list of variables that every python environment has (querying modules like sys) and let the developers use them as root locations for some files. - sys.prefix - sys.exec_prefix - some elements returned by distutils.sysconfig.get_config_vars() (distutils.sysconfig queries the python Makefile amongst others) - ... Semi-related: distutils.sysconfig should be removed from distutils, and be a standalone module in the sdtlib for example. Regards Tarek From mal at egenix.com Tue Jul 7 10:16:19 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 07 Jul 2009 10:16:19 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A527514.5040709@gmail.com> References: <4A520F83.7020300@egenix.com> <4A526B11.7060308@gmail.com> <4A527363.7070305@v.loewis.de> <4A527514.5040709@gmail.com> Message-ID: <4A530453.10005@egenix.com> Nick Coghlan wrote: > Martin v. L?wis wrote: >>> I suggest that we also run a version of that redirection filter to remap >>> the old svn.python.org links in addition to making them accessible >>> through hg.python.org as Dirkjan proposed. >> Not sure what links you are talking about, but we also need to keep the >> current svn.python.org up as-is, for all the stuff that won't be migrated. Good point. So the old svn revision links will continue to work. > In that case, no need to redirect anything (although it will still be > possible to look up the old SVN revisions in the Hg web view). > > The previous discussion was based on the assumption that it was going to > be possible to switch the SVN server off at some point rather than just > disallowing commits to the projects that had migrated to Mercurial. Is there a standard notation for hg revisions that roundup could detect and turn into links (much like it does for svn) ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 07 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ronaldoussoren at mac.com Tue Jul 7 10:17:17 2009 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 07 Jul 2009 10:17:17 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <94bdd2610907060053x58d9b191wac7bfda8743c7fad@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907031246t636c7049y98b2f45b273791bb@mail.gmail.com> <4A4EA689.1010704@gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <4A5116DC.1070109@gmail.com> <31B6EE4E-8D3A-4BBC-ABAD-074DBEEC60F9@mac.com> <94bdd2610907060053x58d9b191wac7bfda8743c7fad@mail.gmail.com> Message-ID: <18234706-FD64-453F-BFBD-D084D5CC0B1C@mac.com> On 6 Jul, 2009, at 9:53, Tarek Ziad? wrote: > 2009/7/6 Ronald Oussoren : >> I'm -1 on changing the name. For better or worse setuptools is the >> elephant >> in the room w.r.t. package management and it would IMHO be better >> to stay >> compatible (even if the stdlib only implements a subset of >> setuptools/pkg_resources) >> > > I'd rather see the elephant evolves. > > I don't see why we should bend a standard we want to introduce in > the stdlib, > for a third-party package that is able to evolve to stick to a new > standard > without any problem. But why break existing code without having any other benifits? If I read the discussion correctly the name would be changed without any changes to the contents of the metadata directory. I would be more inclined to be in favour if the name change had a sound technical reason, such as a change of the contents of the directory which would make setuptools "egg-info" directories incompatible with the PEP376 ones. Ronald From dirkjan at ochtman.nl Tue Jul 7 10:30:30 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 7 Jul 2009 10:30:30 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A530453.10005@egenix.com> References: <4A520F83.7020300@egenix.com> <4A526B11.7060308@gmail.com> <4A527363.7070305@v.loewis.de> <4A527514.5040709@gmail.com> <4A530453.10005@egenix.com> Message-ID: On Tue, Jul 7, 2009 at 10:16, M.-A. Lemburg wrote: > Is there a standard notation for hg revisions that roundup could > detect and turn into links (much like it does for svn) ? [a-f0-9]{12} should mostly do. (Sorry for my absence from the discussion for some time. I'll try to update the PEP and summarize ongoing discussions here soon.) Cheers, Dirkjan From p.f.moore at gmail.com Tue Jul 7 10:33:55 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Jul 2009 09:33:55 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> Message-ID: <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> 2009/7/7 Tarek Ziad? : > Unless we define a "drive that contains the python installation" maybe, or > the "Program Files" directory > > would that ?make sense from a win32 point of view ? I can't imagine that it would be useful in practice. > I know some people are writing to /etc to add their configuration file > on the system, > > So a real-world example under linux would be: > > setup(..., data_files=[('/etc', ['myconf.cfg'])], ...) > > > That is basically how the examples are shown at: > > http://docs.python.org/distutils/setupscript.html#installing-additional-files Thanks. Yes, that makes for a good example. > But this is already os-specific, and exists because distutils doesn't > have a way (yet) > to express systems locations independantly from their physical location, > like what the RPM system does with %VARIABLES. > > So another way to handle this maybe, like I have added with $PREFIX > and $EXEC_PREFIX would be to nominate a list of variables that every > python environment has (querying modules like sys) and let the developers > use them as root locations for some files. > > - sys.prefix > - sys.exec_prefix > - some elements returned by distutils.sysconfig.get_config_vars() > (distutils.sysconfig queries the python Makefile amongst others) > - ... This adds extra complexity to the RECORD format, for little practical benefit that I can see. >From the POV of core distutils: - Windows users use bdist_wininst and/or bdist_msi, and absolute locations are a lost cause. - Presumably, some people use bdist_rpm (I don't know if there are other ways of creating RPMs). - Everyone else uses setup.py install or a 3rd party tool. - PEP 302 style loaders aren't relevant as the core only uses the filesystem (not even zip files). - Only 3rd party tools will consume this data. So, we need input from developers of 3rd party tools here. Phillip has stated the case for setuptools, from his POV having everything relative to the "install location" which is stored elsewhere (in the installer file) is fine. I'd like to know whether he needs "upwards-pointing" relative paths like ../../../../xx.py, but that's a small detail. So - do any other potential users of the PEP 376 metadata want to speak up? At the moment, it feels like we're designing things more or less in a vacuum. Paul. From ziade.tarek at gmail.com Tue Jul 7 10:40:00 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Jul 2009 10:40:00 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> Message-ID: <94bdd2610907070140g73e787a7rd171a6d828621efa@mail.gmail.com> On Tue, Jul 7, 2009 at 10:33 AM, Paul Moore wrote: > 2009/7/7 Tarek Ziad? : >> Unless we define a "drive that contains the python installation" maybe, or >> the "Program Files" directory >> >> would that ?make sense from a win32 point of view ? > > I can't imagine that it would be useful in practice. > >> I know some people are writing to /etc to add their configuration file >> on the system, >> >> So a real-world example under linux would be: >> >> setup(..., data_files=[('/etc', ['myconf.cfg'])], ...) >> >> >> That is basically how the examples are shown at: >> >> http://docs.python.org/distutils/setupscript.html#installing-additional-files > > Thanks. Yes, that makes for a good example. > >> But this is already os-specific, and exists because distutils doesn't >> have a way (yet) >> to express systems locations independantly from their physical location, >> like what the RPM system does with %VARIABLES. >> >> So another way to handle this maybe, like I have added with $PREFIX >> and $EXEC_PREFIX would be to nominate a list of variables that every >> python environment has (querying modules like sys) and let the developers >> use them as root locations for some files. >> >> - sys.prefix >> - sys.exec_prefix >> - some elements returned by distutils.sysconfig.get_config_vars() >> (distutils.sysconfig queries the python Makefile amongst others) >> - ... > > This adds extra complexity to the RECORD format, for little practical > benefit that I can see. > > From the POV of core distutils: > - Windows users use bdist_wininst and/or bdist_msi, and absolute > locations are a lost cause. > - Presumably, some people use bdist_rpm (I don't know if there are > other ways of creating RPMs). They are othe ways to generate RPMs. There are also debian packaging scripts > - Everyone else uses setup.py install or a 3rd party tool. > - PEP 302 style loaders aren't relevant as the core only uses the > filesystem (not even zip files). > - Only 3rd party tools will consume this data. > > So, we need input from developers of 3rd party tools here. Phillip has > stated the case for setuptools, from his POV having everything > relative to the "install location" which is stored elsewhere (in the > installer file) is fine. I'd like to know whether he needs > "upwards-pointing" relative paths like ../../../../xx.py, but that's a > small detail. > > So - do any other potential users of the PEP 376 metadata want to speak up? > > At the moment, it feels like we're designing things more or less in a vacuum. I am CC'ing the people that worked with us for the versionning matters, they can speak from a Fedora and Ubuntu POV, I am also adding Jim for zc.buildout, he can provide precious input. (I am sorry if some people get the mail twice) From ronaldoussoren at mac.com Tue Jul 7 10:40:12 2009 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 07 Jul 2009 10:40:12 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> Message-ID: <4B578D30-0B20-461F-AFEB-7B23710FE14D@mac.com> On 6 Jul, 2009, at 20:38, Paul Moore wrote: > > - Should distribution names be case insensitive on case insensitive > filesystems? For comparison, module/package names are always case > sensitive even on case insensitive systems. I'd then go for case sensitive names for distributions as well, just to be consistent. But isn't the actual name in the PKG-INFO file anyway? Do you mean really case insensive filesystems like DOS' FAT, or case preserving (HFS+ on OSX, NTFS on Windows)? For the latter you can at least get the original name back using os.readdir. BTW. Actually determining if you are working with a case-sensitive fileystem requires filesystem access, os.path.normcase is hopelessly naieve in that respect. I'm reguarly dealing with NFS mounted filesystems on OSX systems, which means that parts of a path are case preserving (the HFS+ filesystem upto the mount point) and other parts are truly case sensitive (paths on the NFS mounted filesystem originating on a Linux server). Ronald From mal at egenix.com Tue Jul 7 10:57:33 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 07 Jul 2009 10:57:33 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> Message-ID: <4A530DFD.40802@egenix.com> Paul Moore wrote: >> I know some people are writing to /etc to add their configuration file >> on the system, >> >> So a real-world example under linux would be: >> >> setup(..., data_files=[('/etc', ['myconf.cfg'])], ...) >> >> >> That is basically how the examples are shown at: >> >> http://docs.python.org/distutils/setupscript.html#installing-additional-files > > Thanks. Yes, that makes for a good example. > >> But this is already os-specific, and exists because distutils doesn't >> have a way (yet) >> to express systems locations independantly from their physical location, >> like what the RPM system does with %VARIABLES. >> >> So another way to handle this maybe, like I have added with $PREFIX >> and $EXEC_PREFIX would be to nominate a list of variables that every >> python environment has (querying modules like sys) and let the developers >> use them as root locations for some files. I think you have to differentiate between packages and applications. Packages will usually have their own way of getting configured, either by passing parameters via some API or pointing the package to a configuration file. They may come with some example config files, but should normally don't interfere with the system configuration. I.e. putting a Python package configuration into /etc does not really sound like a good idea. Applications tend to ship everything needed to run the application together with the installer and typically use a system-dependent installer rather than a distutils based approach. These then place config files in the usual default dirs of the system. This is out-of-scope for PEP 376. Then you have tools like zc.buildout which basically build an application in some directory at "install" time. Since only these tools know what they are doing, the whole "uninstall" mechanism also lies in their hands. Again, PEP 376 would only apply to the dynamic package installation part, but not to the complete application build. Another aspect to consider is that config files should normally not be uninstalled automatically. The user should either be asked whether she wants to keep the files or the uninstaller should leave them untouched and issue a warning that certain files were not uninstalled. Summarizing, I think it's better not to record config files and other user-edited files in the RECORD file. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 07 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From p.f.moore at gmail.com Tue Jul 7 11:04:04 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Jul 2009 10:04:04 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <4A52FE4A.6020204@egenix.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <87ws6lp8pc.fsf@benfinney.id.au> <79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com> <4A52FE4A.6020204@egenix.com> Message-ID: <79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com> 2009/7/7 M.-A. Lemburg : > The PEP should take the chance to define not only the > directory, but also the complete set of files in there and > their format. > > A typical setuptools dir looks like this: > > dependency_links.txt > namespace_packages.txt > not-zip-safe > PKG-INFO > requires.txt > SOURCES.txt > top_level.txt > > Ie. a complete mess of upper-case/lower-case names, names with > underscores or hyphens as word separator, files with extensions, > files without them. > > This needs some serious cleanup. > > A new dir name will allow to do this, so +1 on a new dir name. I agree the current situation is a mess. I believe that everything you mention is related to setuptools - so essentially, we have the situation: - the existing setuptools format is a mess (at least in the opinion of MAL and me :-)) - PEP 376 is an opportunity to consider cleanup - there are some strong supporters of keeping things setuptools-compatible - nobody has come forward with any other real-world use case than setuptools - consequently, those of us arguing for cleanup are talking theoretically :-( - Do you (MAL) have any real use for this data? Any non-setuptools use case, no matter how small, would really help here. I suspect practicality will beat purity here. Which would be fine, if the "practical" cases weren't just setuptools. I understood that there was a lot of interest in a "distutils cleanup" from the packaging community. And yet so far in this discussion, the main participants have seemed to be (apologies to anyone if I've misunderstood their position): - Tarek, trying to get the proposal he thrashed out in the distutils SIG agreed - Me, advocating Windows issues and PEP 302 integration (to cover eggs and general flexibility) - Phillip (and occasional others), representing setuptools POV - Interested python-dev participants Nobody representing any other 3rd party packaging solution than setuptools. Much as I'd like to, I can't really argue the case for breaking setuptools compatibility without practical reasons. And if we're not going to do that, PEP 376 reduces to a further stage of the incremental move of setuptools into distutils core (by removing the partial solution of a .egg-info file, and replacing it with a full setuptools .egg-info directory, plus a few introspection APIs as a sweetener). 2009/7/7 Ronald Oussoren : > But why break existing code without having any other benifits? If I read > the discussion correctly the name would be changed without any changes to > the contents of the metadata directory. I would be more inclined to be in > favour if the name change had a sound technical reason, such as a change of > the contents of the directory which would make setuptools "egg-info" > directories incompatible with the PEP376 ones. See MAL's comments above, and my response. If (and only if!) his proposal is accepted, then a name change starts to be worth discussing further. As things stand at the moment, the structure appears to be setuptools-compatibile (other than the new RECORD file, which Phillip has committed to adding) so a name change isn't worth it. I can't comment myself on setuptools compatibility, so I'm assuming here that Phillip will speak up if the proposed format is *not* compatible... Paul. From mal at egenix.com Tue Jul 7 11:28:10 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 07 Jul 2009 11:28:10 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <87ws6lp8pc.fsf@benfinney.id.au> <79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com> <4A52FE4A.6020204@egenix.com> <79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com> Message-ID: <4A53152A.8020003@egenix.com> Paul Moore wrote: > 2009/7/7 M.-A. Lemburg : >> The PEP should take the chance to define not only the >> directory, but also the complete set of files in there and >> their format. >> >> A typical setuptools dir looks like this: >> >> dependency_links.txt >> namespace_packages.txt >> not-zip-safe >> PKG-INFO >> requires.txt >> SOURCES.txt >> top_level.txt >> >> Ie. a complete mess of upper-case/lower-case names, names with >> underscores or hyphens as word separator, files with extensions, >> files without them. >> >> This needs some serious cleanup. >> >> A new dir name will allow to do this, so +1 on a new dir name. > > I agree the current situation is a mess. I believe that everything you > mention is related to setuptools - so essentially, we have the > situation: > > - the existing setuptools format is a mess (at least in the opinion of > MAL and me :-)) > - PEP 376 is an opportunity to consider cleanup > - there are some strong supporters of keeping things setuptools-compatible > - nobody has come forward with any other real-world use case than setuptools > - consequently, those of us arguing for cleanup are talking theoretically :-( > - Do you (MAL) have any real use for this data? Any non-setuptools use > case, no matter how small, would really help here. No, we currently don't have a package installer that would implement what setuptools does on the user side, but without all the black magic stuff. What we do have is a somewhat different approach to packaging binary packages: we call them pre-built archives. The idea is that the developer runs a complete build command on the target platform, the bdist_prebuilt command then packages up everything (including the current state of distutils) and the user then runs the usual python setup.py install on her machine to complete the setup. The major advantage here is that the user actually has full control over where things are installed and how. So whatever distutils implements as part of the install command, this format will also support. We also provide an uninstall command, that basically runs an installation in reverse. Something else we've done recently is write a bdist_egg command that creates .egg files without relying on setuptools. It's fairly easy to do and if there's interest, I can add that to core distutils. Changing the layout of the EGG-INFO dir embedded in those .egg files is pretty easy to do, so I don't really see much of a problem with changing it or renaming the dir. > I suspect practicality will beat purity here. Which would be fine, if > the "practical" cases weren't just setuptools. > > I understood that there was a lot of interest in a "distutils cleanup" > from the packaging community. And yet so far in this discussion, the > main participants have seemed to be (apologies to anyone if I've > misunderstood their position): > > - Tarek, trying to get the proposal he thrashed out in the distutils SIG agreed > - Me, advocating Windows issues and PEP 302 integration (to cover eggs > and general flexibility) > - Phillip (and occasional others), representing setuptools POV > - Interested python-dev participants > > Nobody representing any other 3rd party packaging solution than setuptools. Well I've added another one above, the pre-built approach :-) Writing an uninstall command really isn't all that hard, provided you stick with the standard distutils "python setup.py install" dance. The whole packaging approach only complicates things, IMHO. > Much as I'd like to, I can't really argue the case for breaking > setuptools compatibility without practical reasons. And if we're not > going to do that, PEP 376 reduces to a further stage of the > incremental move of setuptools into distutils core (by removing the > partial solution of a .egg-info file, and replacing it with a full > setuptools .egg-info directory, plus a few introspection APIs as a > sweetener). Uhm, we are talking about a Python standard here and since there is no prior implementation in Python stdlib, we're not breaking anything and are free to design whatever we like. Of course, it does make a lot of sense to build upon things that have been used in the past, so looking at setuptools is certainly a useful strategy. However, let's learn from their mistakes and take the change to clean up things a bit. There are good reasons to do so: * the naming scheme should be concise * the meta-data directory should get a name that doesn't imply any specific distribution form... Calling the directory .egg-info is natural if you're only dealing with eggs. It is not when running standard "python setup.py install" or having a mixed setup using different packaging mechanisms. > 2009/7/7 Ronald Oussoren : >> But why break existing code without having any other benifits? If I read >> the discussion correctly the name would be changed without any changes to >> the contents of the metadata directory. I would be more inclined to be in >> favour if the name change had a sound technical reason, such as a change of >> the contents of the directory which would make setuptools "egg-info" >> directories incompatible with the PEP376 ones. > > See MAL's comments above, and my response. If (and only if!) his > proposal is accepted, then a name change starts to be worth discussing > further. As things stand at the moment, the structure appears to be > setuptools-compatibile (other than the new RECORD file, which Phillip > has committed to adding) so a name change isn't worth it. > > I can't comment myself on setuptools compatibility, so I'm assuming > here that Phillip will speak up if the proposed format is *not* > compatible... -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 07 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ziade.tarek at gmail.com Tue Jul 7 11:33:48 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Jul 2009 11:33:48 +0200 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <4A53152A.8020003@egenix.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <87ws6lp8pc.fsf@benfinney.id.au> <79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com> <4A52FE4A.6020204@egenix.com> <79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com> <4A53152A.8020003@egenix.com> Message-ID: <94bdd2610907070233m4b0bd28cie264c42cf228abe5@mail.gmail.com> On Tue, Jul 7, 2009 at 11:28 AM, M.-A. Lemburg wrote: > > Writing an uninstall command really isn't all that hard, provided > you stick with the standard distutils "python setup.py install" > dance. The whole packaging approach only complicates things, IMHO. You can't expect people to keep the distribution they used to install in a corner of their hard drive, to make sure they will be able to uninstall it (or to ask them to download it again) That's all about the RECORD files and the asymetric uninstall script described in PEP 376. having an uninstall command on the top of the uninstall script is a nice shortcut though. From p.f.moore at gmail.com Tue Jul 7 11:44:45 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Jul 2009 10:44:45 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A530DFD.40802@egenix.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> <4A530DFD.40802@egenix.com> Message-ID: <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> 2009/7/7 M.-A. Lemburg : > I think you have to differentiate between packages and applications. Agreed. I believe that only packages should be considered here. Applications are the focus of tools like py2exe on Windows, and (AIUI) things like workingenv. These tools should (will) have their own approaches. The only cases I know where there is reason for a package to store paths outside the package directory are: - executable scripts, which go in sys.prefix/Scripts on Windows, and ??? on Unix/MAC OS - supporting files (MoinMoin puts its HTML documents etc in sys.prefix/share on Windows, cx_Oracle puts its documentation in sys.prefix/cx_Oracle-doc) Executable files could probably be superseded by using "python -m", but compatibility and users' preference for having a "real script" probably means they aren't going away in the near future. Support files are getting put in the package directory more often these days, but stuff that needs to be found by non-Python tools is arguably still better in a more discoverable location. For Windows, having a few distinguished locations under the installation root (scripts, share, doc) would probably do. For Unix and Mac OS, I have no opinion (but I suspect that absolute paths like /usr/local/bin might be the norm there). [...] > Summarizing, I think it's better not to record config files > and other user-edited files in the RECORD file. The RECORD file should contain precisely those files that are created as part of the install. That's ultimately the point of the file (for ownership queries and uninstallation). Hmm, on the other hand, if foo.py is in the RECORD file, the uninstaller should uninstall foo.pyc and foo.pyo as well. And a query as to whether the distribution owns foo.pyc should return True. How will this be handled? Paul. From p.f.moore at gmail.com Tue Jul 7 11:52:25 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Jul 2009 10:52:25 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <4A53152A.8020003@egenix.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <87ws6lp8pc.fsf@benfinney.id.au> <79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com> <4A52FE4A.6020204@egenix.com> <79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com> <4A53152A.8020003@egenix.com> Message-ID: <79990c6b0907070252i2e1d945fw9a5d8920915c3d39@mail.gmail.com> 2009/7/7 M.-A. Lemburg : > Well I've added another one above, the pre-built approach :-) Thanks! That's a lot of help. Are there any public examples, or is your format only used internally? Would you expect to use any of the new pkgutil APIs - for example, to check if a distribution is installed, and/or the installed version? Or to check you're not overwriting a file owned by another distribution? Or would you get any value from storing your own format-specific metadata files in the egg-info directory? If all you care about is that setup.py install continues to work, my suspicion is that PEP 376 won't have much to offer you, whatever happens. Paul. From ncoghlan at gmail.com Tue Jul 7 13:38:34 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 07 Jul 2009 21:38:34 +1000 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> Message-ID: <4A5333BA.3030806@gmail.com> Paul Moore wrote: > 2009/7/6 Nick Coghlan : > - There are *no* guaranteed absolute locations on Windows, so any such > oddly-located file would require user interaction to work. Certainly > bdist_wininst and bdist_msi don't do that. > - My experiments indicate that bdist_{wininst,msi} are broken with > respect to absolute paths anyway: they do a --root install to a > temporary directory (and the absolute paths don't end up in there) and > then package up that temporary directory. > > I still want to see a real life example that demonstrates that there > is a genuine issue here. We're spending a lot of energy and complexity > trying to design a solution to a problem that actually doesn't appear > to exist in practice... > > (To be honest, I'd be fairly confident in saying that absolute paths > can be ignored on Windows, subject to some corner cases that I haven't > thought through yet. My worry is that I don't know what Unix and Mac > users might do, so I can't just wish away the issue because it can't > arise on Windows. Can a Unix/Mac user offer a real-world example on > their own system?) I thought installing pywin32 based COM objects still involved messing with the Windows directory, but MS may have improved that in more recent OS versions. It's been years since I played with win32com objects, and even then it was just idle experimentation that didn't get very far so I could easily be misremembering. For *nix, the obvious use case is installing scripts somewhere like /usr/bin or /usr/local/bin. One option is to punt on this whole issue and say if people want to install stuff outside the Python module heirarchy they should create their own OS-specific package to manage it (i.e. leave the non-relative paths to be managed by APT or a Windows installer or whatever). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ben+python at benfinney.id.au Tue Jul 7 13:52:38 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 07 Jul 2009 21:52:38 +1000 Subject: [Python-Dev] PEP 376 - Open questions References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> Message-ID: <87k52kpv5l.fsf@benfinney.id.au> Paul Moore writes: > The only cases I know where there is reason for a package to store > paths outside the package directory are: I think the *only* files that actually belong in the package directory are the Python modules inside that package. Other files need to be easily, and automatically, separable for purposes of installation. > - executable scripts, which go in sys.prefix/Scripts on Windows os.path.join(sys.prefix, 'Scripts') > and ??? on Unix/MAC OS Depending on whether the developer designates the programs for sysadmin-only use or not: os.path.join(sys.prefix, 'bin') os.path.join(sys.prefix, 'sbin') > - supporting files (MoinMoin puts its HTML documents etc in > sys.prefix/share on Windows, cx_Oracle puts its documentation in > sys.prefix/cx_Oracle-doc) This category, it seems to me, needs to be expanded with metadata that allows ?purpose?-based tagging, so that platform-specific standards can be applied using those purposes to determine the correct location for these non-Python-module files. > Support files are getting put in the package directory more often > these days Only, IMO, because there's no way of flagging it as anything but arbitrary ?data?. Examples of purpose-based classifications that need to be distinctly declarable: executable program, importable module source, platform-independent compiled module, platform-dependent compiled module, documentation, run-time variable data, static data, configuration, and so on. All of these (and others I've forgotten) should be possible for the developer to declare in distribution metadata, and the installer can then use those declarations to make the files go to platform-specific locations, not the Python package directories. > For Windows, having a few distinguished locations under the > installation root (scripts, share, doc) would probably do. For Unix > and Mac OS, I have no opinion (but I suspect that absolute paths like > /usr/local/bin might be the norm there). The lines are drawn in different places on each platform; we don't want to have the union of all these different platform-specific locations in the standard, and likewise we don't want to leave any of them with second-class support. I think it's not the developer's burden to decide *where* such files go; rather, they should be declaring only the *purpose* of these files in the distribution metadata, and it's up to the site-specific installer (possibly as configured by the installing user) to decide the location of each file by its declared purpose. -- \ ?I can picture in my mind a world without war, a world without | `\ hate. And I can picture us attacking that world, because they'd | _o__) never expect it.? ?Jack Handey | Ben Finney From ben+python at benfinney.id.au Tue Jul 7 13:58:51 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 07 Jul 2009 21:58:51 +1000 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907050613q59d45e69g4b34fc40ff9ca100@mail.gmail.com> <20090705183326.097CF3A4099@sparrow.telecommunity.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <87ws6lp8pc.fsf@benfinney.id.au> <79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com> <4A52FE4A.6020204@egenix.com> <79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com> Message-ID: <87fxd8puv8.fsf@benfinney.id.au> Paul Moore writes: > I agree the current situation is a mess. I believe that everything you > mention is related to setuptools - so essentially, we have the > situation: > > - the existing setuptools format is a mess (at least in the opinion of > MAL and me :-)) No argument from me on this point. > - PEP 376 is an opportunity to consider cleanup > - there are some strong supporters of keeping things > setuptools-compatible Your implication seems to be that these two are incompatible. It should be noted that it's possible to clean up, by having a separate (e.g. ?.pydist?) directory as the new standard, with ?cleaned up? standard content, and continue backward compatibility in the actual tools while deprecating the old ?.egg-info? format for some time until finally removing that support. > Much as I'd like to, I can't really argue the case for breaking > setuptools compatibility without practical reasons. Agreed. I think we should be making it clear that ?.egg-info? is not going to evolve any further, and that it will eventually go away; but not remove support for it until there's a clearly established replacement. -- \ ?I got up the other day, and everything in my apartment has | `\ been stolen and replaced with an exact replica.? ?Steven Wright | _o__) | Ben Finney From p.f.moore at gmail.com Tue Jul 7 14:05:19 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Jul 2009 13:05:19 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <87k52kpv5l.fsf@benfinney.id.au> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> Message-ID: <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> 2009/7/7 Ben Finney : [... lots of interesting stuff deleted ...] > I think it's not the developer's burden to decide *where* such files go; > rather, they should be declaring only the *purpose* of these files in > the distribution metadata, and it's up to the site-specific installer > (possibly as configured by the installing user) to decide the location > of each file by its declared purpose. That's a whole different PEP, though. Paul. From p.f.moore at gmail.com Tue Jul 7 14:11:50 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 Jul 2009 13:11:50 +0100 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <87fxd8puv8.fsf@benfinney.id.au> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <87ws6lp8pc.fsf@benfinney.id.au> <79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com> <4A52FE4A.6020204@egenix.com> <79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com> <87fxd8puv8.fsf@benfinney.id.au> Message-ID: <79990c6b0907070511h6facd39di64ba9ae0cab58e4b@mail.gmail.com> 2009/7/7 Ben Finney : >> - PEP 376 is an opportunity to consider cleanup >> - there are some strong supporters of keeping things >> setuptools-compatible > > Your implication seems to be that these two are incompatible. It should > be noted that it's possible to clean up, by having a separate (e.g. > ?.pydist?) directory as the new standard, with ?cleaned up? standard > content, and continue backward compatibility in the actual tools while > deprecating the old ?.egg-info? format for some time until finally > removing that support. Phillip argued strongly against this. I'm not going to repeat his arguments - they seemed to make sense to me at the time, but I haven't internalised them well enough that I'd be willing to try to repeat them here. The crux of all this seems to be setuptools' backwards compatibility issues, As someone who has a very strong dislike of setuptools (the implementation, not the ideas), I'm trying very, very hard to leave it to someone else to say "stuff setuptools, this is a core distutils proposal, setuptools can keep up or die" :-) Paul. From rdmurray at bitdance.com Tue Jul 7 14:52:41 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 7 Jul 2009 08:52:41 -0400 (EDT) Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> Message-ID: On Tue, 7 Jul 2009 at 13:05, Paul Moore wrote: > 2009/7/7 Ben Finney : > [... lots of interesting stuff deleted ...] >> I think it's not the developer's burden to decide *where* such files go; >> rather, they should be declaring only the *purpose* of these files in >> the distribution metadata, and it's up to the site-specific installer >> (possibly as configured by the installing user) to decide the location >> of each file by its declared purpose. > > That's a whole different PEP, though. Which one? It seems to me that supporting this is implicit in the language summit goals of (1) having distutils be better support infrastructure for system packaging utilities and (2) needing a way to deal with resource files "that might be installed in a specific place on the target system by the system packager". I'll grant that I'm reading between the lines, it isn't an explicitly stated goal. But it was the direction my mind went when I read Tarek's notes, given that the first stated goal is "standardize more metadata". But I'm not one of the people involved in system packaging tools, so I'll leave it to them to say how useful/important this is. --David From ziade.tarek at gmail.com Tue Jul 7 15:23:01 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Jul 2009 15:23:01 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> Message-ID: <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> On Tue, Jul 7, 2009 at 2:52 PM, R. David Murray wrote: > On Tue, 7 Jul 2009 at 13:05, Paul Moore wrote: >> >> 2009/7/7 Ben Finney : >> [... lots of interesting stuff deleted ...] >>> >>> I think it's not the developer's burden to decide *where* such files go; >>> rather, they should be declaring only the *purpose* of these files in >>> the distribution metadata, and it's up to the site-specific installer >>> (possibly as configured by the installing user) to decide the location >>> of each file by its declared purpose. >> >> That's a whole different PEP, though. > > Which one? ?It seems to me that supporting this is implicit in > the language summit goals of (1) having distutils be better support > infrastructure for system packaging utilities and (2) needing a way to > deal with resource files "that might be installed in a specific place on > the target system by the system packager". ?I'll grant that I'm reading > between the lines, it isn't an explicitly stated goal. ?But it was the > direction my mind went when I read Tarek's notes, given that the first > stated goal is "standardize more metadata". Yes but the topic is so wide that it has to be cut in several PEP, and things have to be done gradually So far: - PEP 376 : standard for the metadata format and location + query APIs - PEP 345 : standard for the metadata *content* - work in progress too (there's a branch with new fields waiting) - PEP 386 : standard for version comparisons topics that are not yet in PEP are grouped on the wiki page (under "current work") with notes : http://wiki.python.org/moin/Distutils When I started to work on this I didn't realize the gigantic amount of work and coordination it requires, and I do understand now the current state. At first I was trying to coordinate interested people to work on each topic mentioned there in parallel. (like we did a bit after the summit) But at the end, it seems that having everyone interested in packaging matters focusing on the less number of possible topics pays more. PEP 376 is just a piece of the puzzle but I am confident it will speed up other tasks since it raises our common ground knowledge. From mal at egenix.com Tue Jul 7 15:26:20 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 07 Jul 2009 15:26:20 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A520F83.7020300@egenix.com> Message-ID: <4A534CFC.30909@egenix.com> Brett Cannon wrote: > On Mon, Jul 6, 2009 at 07:51, M.-A. Lemburg wrote: > >> Dirkjan Ochtman wrote: >>> In response to some rumblings on python-committers and just to request >>> more feedback, a progress report. I know it's long, I've tried to put >>> to keep it concise and chunked, though. >> Two things: >> >> * We need some form of documentation of how committers are expected >> to work with the hg repo. (This is also missing for the Subversion >> repo, which due to the 3.x branch has gotten somewhat unclear - at >> least for me) >> > > I am planning to get a version of the dev FAQ written up that covers most of > what it already does now for svn. > > >> It is currently not clear where to check in patches, whether and >> where to backport or forward-patch, which branches to consider >> closed, etc. >> >> E.g. if I check in something in trunk/ (Python 2.7), do I have to >> forward patch this change to the 3.0 branch (guess not), the py3k/ >> branch (Python 3.1), or will someone else take care of this, so that >> it's better not to interfere by doing it myself ? >> > > This question is partially answered by > http://www.python.org/dev/faq/#how-do-i-merge-between-branches, but I agree > that we should have either this spelled out in the FAQ or a > committer-specific doc at www.python.org/dev/ that makes this all very > obvious. The merge process itself is more or less clear. What I'm missing is the agreed upon strategy for applying the patches to the various branches. I've seen a few discussions about this, but no final statement of what strategy to follow and whether hg makes this easier (AFAIR, that was the main argument for switching to hg). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 07 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Tue Jul 7 15:32:46 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 07 Jul 2009 15:32:46 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A520F83.7020300@egenix.com> <4A526B11.7060308@gmail.com> <4A527363.7070305@v.loewis.de> <4A527514.5040709@gmail.com> <4A530453.10005@egenix.com> Message-ID: <4A534E7E.7010700@egenix.com> Dirkjan Ochtman wrote: > On Tue, Jul 7, 2009 at 10:16, M.-A. Lemburg wrote: >> Is there a standard notation for hg revisions that roundup could >> detect and turn into links (much like it does for svn) ? > > [a-f0-9]{12} should mostly do. Hmm, no prefix or suffix ? So we'll always have to write "see deadbeefdeadbeefff for details" or "Reverting f00fl33df00fl33d00 after problems on Pentium CPUs" ?! Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 07 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From dirkjan at ochtman.nl Tue Jul 7 15:40:24 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Tue, 7 Jul 2009 15:40:24 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A534E7E.7010700@egenix.com> References: <4A520F83.7020300@egenix.com> <4A526B11.7060308@gmail.com> <4A527363.7070305@v.loewis.de> <4A527514.5040709@gmail.com> <4A530453.10005@egenix.com> <4A534E7E.7010700@egenix.com> Message-ID: On Tue, Jul 7, 2009 at 15:32, M.-A. Lemburg wrote: > Hmm, no prefix or suffix ? No, not really. hg often shows revision integers as well, but as they aren't globally consistent, they're of little value in communicating changeset pointers. > So we'll always have to write "see deadbeefdeadbeefff for details" > or "Reverting f00fl33df00fl33d00 after problems on Pentium CPUs" ?! Yes. And it's really not that bad. Cheers, Dirkjan From rdmurray at bitdance.com Tue Jul 7 16:17:21 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 7 Jul 2009 10:17:21 -0400 (EDT) Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A534CFC.30909@egenix.com> References: <4A520F83.7020300@egenix.com> <4A534CFC.30909@egenix.com> Message-ID: On Tue, 7 Jul 2009 at 15:26, M.-A. Lemburg wrote: > The merge process itself is more or less clear. What I'm missing > is the agreed upon strategy for applying the patches to the various > branches. > > I've seen a few discussions about this, but no final statement > of what strategy to follow and whether hg makes this easier (AFAIR, > that was the main argument for switching to hg). I think the main reason for switching was that it would make it easier for non-core-committers to maintain branches and submit patches (as changesets core committers can pull). I don't think it was ever clear that the merge workflow would in fact get easier, except insofar as hg's merge support is better than SVN's (at least, I believe people have said that last is true). There is _hope_ that it will be easier, but I think it remains to be proven/worked out. And I believe there is no tool like svnmerge for tracking changesets to be merged, which could be an issue that needs a resolution. IIUC, the discussion about named versus cloned branches is part of figuring out the workflow.... --David From ideasman42 at gmail.com Tue Jul 7 19:38:05 2009 From: ideasman42 at gmail.com (Campbell Barton) Date: Tue, 7 Jul 2009 10:38:05 -0700 Subject: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ? Message-ID: <7c1ab96d0907071038p4ce29935j2580420de05eaf4b@mail.gmail.com> In Python3 PySys_SetArgv takes (wchar_t **) for the argv, I looked into converting a (char **) into a (wchar_t **) and while its possible its lengthy enough not to be trivial, see python.c:18 - char2wcharm(), its 102 lines with ifdef's and goto's, not including the loop lower down that loops over the argv. Looking further python has a very similar function for ./PC/bdist_wininst/install.c, for do_run_installscript - widen_string(). Since many C applications take argv as a (char **) it seems reasonable for python to keep a function for the C api that accepts a (char **) argument for argv. otherwise each C applications that embeds python will need to write their own conversion function. also noticed Demo/embed/demo.c is out of date, giving (char **) to PySys_SetArgv(). -- - Campbell From pje at telecommunity.com Tue Jul 7 22:30:05 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 07 Jul 2009 16:30:05 -0400 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files Message-ID: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> Just an idea... suppose that instead of using "real" absolute paths in the RECORD file for non-local files (scripts, data, etc) we changed the format to include a "prefix" field, containing something like LIBDIR, SHARE, SCRIPTS, etc., ala bdist_wininst internals? Also, we could include a separate (optional) PREFIXES file defining what those locations translated to at install time. Dumb bdists would still have their build paths there (or delete the file before packaging, or use real paths instead of the --root prefixed versions). Upside: relocation can be detected and handled, RECORD remains cross-platform, and bdists are ok. Downside: more complex API required to read/manipulate paths and delete files, since you need to be able to check that you have the right prefixes, and may have to ask the user for a prefix you don't recognize, if the default in PREFIXES doesn't match. Thoughts? (Also, as a special case, any file that's actually installed to LIBDIR or a subdirectory thereof (even if it's technically a "data" file or script), will probably need to be designated under LIBDIR prefix to prevent runtime breakage in the event any system package maintainers are tempted to use RECORD files as a way of forcing a HFS conformance. These are installation prefixes, *not* content types.) From pje at telecommunity.com Tue Jul 7 22:31:52 2009 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 07 Jul 2009 16:31:52 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.co m> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> Message-ID: <20090707202851.7DBB93A410E@sparrow.telecommunity.com> At 03:23 PM 7/7/2009 +0200, Tarek Ziad? wrote: >When I started to work on this I didn't realize the gigantic amount of >work and coordination it requires No one expects the package inquisition. ;-) From ncoghlan at gmail.com Tue Jul 7 23:02:52 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 08 Jul 2009 07:02:52 +1000 Subject: [Python-Dev] PEP 376 and PEP 302 - allowing import hooks to provide distribution metadata In-Reply-To: <79990c6b0907070511h6facd39di64ba9ae0cab58e4b@mail.gmail.com> References: <79990c6b0907031229w2554a3c7re937040ababad730@mail.gmail.com> <94bdd2610907051143hf917169m267d89cc6375b1fa@mail.gmail.com> <20090706010108.CCEC83A4099@sparrow.telecommunity.com> <4A520BDF.80503@gmail.com> <79990c6b0907060814j6e4fa3dev8283ff94bd653792@mail.gmail.com> <87ws6lp8pc.fsf@benfinney.id.au> <79990c6b0907070015q142f17f4h659052f4de3f8e2a@mail.gmail.com> <4A52FE4A.6020204@egenix.com> <79990c6b0907070204p41420022mc866ed052026dc75@mail.gmail.com> <87fxd8puv8.fsf@benfinney.id.au> <79990c6b0907070511h6facd39di64ba9ae0cab58e4b@mail.gmail.com> Message-ID: <4A53B7FC.6090500@gmail.com> Paul Moore wrote: > 2009/7/7 Ben Finney : >>> - PEP 376 is an opportunity to consider cleanup >>> - there are some strong supporters of keeping things >>> setuptools-compatible >> Your implication seems to be that these two are incompatible. It should >> be noted that it's possible to clean up, by having a separate (e.g. >> ?.pydist?) directory as the new standard, with ?cleaned up? standard >> content, and continue backward compatibility in the actual tools while >> deprecating the old ?.egg-info? format for some time until finally >> removing that support. > > Phillip argued strongly against this. I'm not going to repeat his > arguments - they seemed to make sense to me at the time, but I haven't > internalised them well enough that I'd be willing to try to repeat > them here. > > The crux of all this seems to be setuptools' backwards compatibility > issues, As someone who has a very strong dislike of setuptools (the > implementation, not the ideas), I'm trying very, very hard to leave it > to someone else to say "stuff setuptools, this is a core distutils > proposal, setuptools can keep up or die" :-) Actually, the approach Ben suggests is the same as the one I suggested for dealing with the backwards compatibility problem (i.e. distutils would install both the existing .egg-info for detection by existing packaging tools, while putting the new stuff in a .pydist directory). The argument against *that* approach to backwards compatibility is the accumulation of cruft in site-packages (and other module installation directories) since even a simple installation of a single module or small package would now be generating at least *three* entries in the relevant directory (.egg-info, .pydist and the module or package itself). Retaining the .egg-info naming provides detection of installed distributions by old packaging systems with a minimum of cruft on the end user's system. The only possible advantage changing the extension name might give us is making it fractionally easier to clean up the contents of the .egg-info directory, but I don't think it will actually make that much difference (i.e. it really shouldn't be that hard to keep the new PEP 376 based filenames all UPPERCASE and distinct from any filenames currently stored in the directory by setuptools or other packaging utilities) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ziade.tarek at gmail.com Tue Jul 7 23:21:23 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Jul 2009 23:21:23 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> Message-ID: <94bdd2610907071421r70daf38bt91fb29fb18db46f2@mail.gmail.com> 2009/7/7 Paul Moore : > > The RECORD file should contain precisely those files that are created > as part of the install. That's ultimately the point of the file (for > ownership queries and uninstallation). > > Hmm, on the other hand, if foo.py is in the RECORD file, the > uninstaller should uninstall foo.pyc and foo.pyo as well. And a query > as to whether the distribution owns foo.pyc should return True. How > will this be handled? It's planned to list them as well in RECORD, since install calls a sub command that build thems (install_lib), So the same rules apply than the .py ones. But there's a special case : if the --no-compile or the --no-optimize option is used, then the pyc|pyo files are not added. Which means they will be created afterwards when the module is used on the target system. So the pyc|pyo files could be removed when they are present besides the py file that is being removed. Although, it will still be required to write them in the RECORD file by the install command when the --no-compile or the --no-optimize options are *not* used. So they are properly detected and removed when the py files are not distributed in binary distributions, but just pyc files. From ziade.tarek at gmail.com Tue Jul 7 23:21:25 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Jul 2009 23:21:25 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A5333BA.3030806@gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A5272B2.8000202@gmail.com> <79990c6b0907061544g1f32d82cva647c68f401f150a@mail.gmail.com> <4A5333BA.3030806@gmail.com> Message-ID: <94bdd2610907071421l3d7a5122n31e1901e2baf9f34@mail.gmail.com> 2009/7/7 Nick Coghlan : > For *nix, the obvious use case is installing scripts somewhere like > /usr/bin or /usr/local/bin. Using distutils' scripts option, they will end up in : sys.exec_path/bin Another use case I've found in a distro I've installed this afternoon : setup(..., data_files=[('/usr/share/man/man1/', ['SOMEFILE'])], ...) That's not the most elegant way to add a man page, but for someone who doesn't bather with APT or whatever, it works to build a binary distribution. > > One option is to punt on this whole issue and say if people want to > install stuff outside the Python module heirarchy they should create > their own OS-specific package to manage it (i.e. leave the non-relative > paths to be managed by APT or a Windows installer or whatever). If so, what do we do with the "data_files" option in distutils ? If it's used with absolute paths, files can be installed anywhere on the system, and we want to track them. Even if we don't uninstall them automatically, they should be tracked so a third-party uninstaller can deal with them properly. Or do we change this distutils feature and state that the directories used in "data_files" will always be relative to sys.prefix ? That would bring us back to three cases in the RECORD: - files located under sys.prefix, but not located under site-packages - files located under sys.exec_prefix, but not located under site-packages - files located under site-packages Where "site-packages" is the directory that contains .egg-info directory of the distribution (that's basically the current PEP state, beside the absolute paths case we would need to remove) From ziade.tarek at gmail.com Tue Jul 7 23:21:27 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Jul 2009 23:21:27 +0200 Subject: [Python-Dev] PEP 376 - RECORD format proposal Message-ID: <94bdd2610907071421i3e1dc6b4y3ba3125d0f72bfb8@mail.gmail.com> Hello, Let's state that we don't allow absolute paths in RECORD file and see what we can do with other paths. First as a reminder, here's the full list of directories used by Distutils at installation time. The "install" command uses five options to decide where to install the files. The default values are: Unix: install_purelib: sys.prefix/lib/python2.6/site-packages install_platlib: sys.exec_prefix/lib/python2.6/site-packages install_headers: sys.prefix/include/python2.6/docutils install_scripts: sys.prefix/bin install_data : sys.prefix Unix, using the home option, instead of sys.prefix/sys.exec_prefix: install_purelib: home/lib/python install_platlib: home/lib/python install_headers: home/include/python/$dist_name install_scripts: home/bin install_data : home Windows: install_purelib: sys.prefix/Lib/site-packages install_platlib: sys.prefix/Lib/site-packages install_headers: sys.prefix/Include/python2.6/docutils install_scripts: sys.prefix/bin install_data : sys.prefix There are also some old scheme for os.name in ('mac', 'os2') I need to cleanup asap in there, but I won't mention them. Last, there's now the new per user site-package that adds more potential locations. e.g. replaces sys.prefix and sys.exec_prefix with two user directories. So they are all located under two root locations at the max. But since these are all overridable options (in pydistutils.cfg, distutils.cfg, etc) files can be located under *five* different roots. So my proposal is to write those five roots in a header line in the RECORD file, and use them as prefixes to locate the files that are not under the directory where the egg-info dir is located (which will be kept : relative paths) purelib,xxx,headers,xxx,scripts,xxx,install_data,xxx ... $purelib/some/file ... How does that sounds ? Tarek. -- Tarek Ziad? | http://ziade.org From martin at v.loewis.de Tue Jul 7 23:29:47 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 07 Jul 2009 23:29:47 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A534E7E.7010700@egenix.com> References: <4A520F83.7020300@egenix.com> <4A526B11.7060308@gmail.com> <4A527363.7070305@v.loewis.de> <4A527514.5040709@gmail.com> <4A530453.10005@egenix.com> <4A534E7E.7010700@egenix.com> Message-ID: <4A53BE4B.6000604@v.loewis.de> M.-A. Lemburg wrote: > Dirkjan Ochtman wrote: >> On Tue, Jul 7, 2009 at 10:16, M.-A. Lemburg wrote: >>> Is there a standard notation for hg revisions that roundup could >>> detect and turn into links (much like it does for svn) ? >> [a-f0-9]{12} should mostly do. > > Hmm, no prefix or suffix ? > > So we'll always have to write "see deadbeefdeadbeefff for details" > or "Reverting f00fl33df00fl33d00 after problems on Pentium CPUs" ?! No, rather "see deadbeefdead for details" and "Reverting f00fl33df00fl33d00 after problems on Pentium CPUs". In practice, rather 30ba63d28b1b or 12fb3b32d75d (from html5lib, fwiw). Dirkjan meant "[a-f0-9]{12}" literally. Regards, Martin From ziade.tarek at gmail.com Tue Jul 7 23:30:16 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 7 Jul 2009 23:30:16 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <20090707202851.7DBB93A410E@sparrow.telecommunity.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> Message-ID: <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby wrote: > At 03:23 PM 7/7/2009 +0200, Tarek Ziad? wrote: >> >> When I started to work on this I didn't realize the gigantic amount of >> work and coordination it requires > > No one expects the package inquisition. ?;-) > > Sorry, i've looked in the english dictionary but I don't get this one. what do you mean ? From martin at v.loewis.de Tue Jul 7 23:32:16 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 07 Jul 2009 23:32:16 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A520F83.7020300@egenix.com> <4A526B11.7060308@gmail.com> <4A527363.7070305@v.loewis.de> <4A527514.5040709@gmail.com> <4A530453.10005@egenix.com> <4A534E7E.7010700@egenix.com> Message-ID: <4A53BEE0.3000703@v.loewis.de> >> Hmm, no prefix or suffix ? > > No, not really. hg often shows revision integers as well, but as they > aren't globally consistent, they're of little value in communicating > changeset pointers. I think MAL wasn't asking for a "1354:" prefix, but for a, say, "r" or "R" prefix, or perhaps "V" or "merc:". I think you are proposing that there is no prefix because the chance for a misidentification of some word as a hg revision number is small, as it has to be exactly 12 hex digits. Regards, Martin From rdmurray at bitdance.com Tue Jul 7 23:40:06 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 7 Jul 2009 17:40:06 -0400 (EDT) Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> Message-ID: On Tue, 7 Jul 2009 at 23:30, Tarek Ziad? wrote: > On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby wrote: >> At 03:23 PM 7/7/2009 +0200, Tarek Ziad? wrote: >>> >>> When I started to work on this I didn't realize the gigantic amount of >>> work and coordination it requires >> >> No one expects the package inquisition. ?;-) >> >> > > Sorry, i've looked in the english dictionary but I don't get this one. > what do you mean ? See the recent thread on python-committers about mandatory Monty Python cultural education for committers.... :) See also http://www.youtube.com/watch?v=gldlyTjXk9A --David From ncoghlan at gmail.com Tue Jul 7 23:41:14 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 08 Jul 2009 07:41:14 +1000 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> Message-ID: <4A53C0FA.8020304@gmail.com> Tarek Ziad? wrote: > On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby wrote: >> At 03:23 PM 7/7/2009 +0200, Tarek Ziad? wrote: >>> When I started to work on this I didn't realize the gigantic amount of >>> work and coordination it requires >> No one expects the package inquisition. ;-) >> >> > > Sorry, i've looked in the english dictionary but I don't get this one. > what do you mean ? It's a Monty Python reference: http://en.wikipedia.org/wiki/The_Spanish_Inquisition_(Monty_Python) http://www.youtube.com/watch?v=gldlyTjXk9A Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Tue Jul 7 23:42:56 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 07 Jul 2009 23:42:56 +0200 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A520F83.7020300@egenix.com> <4A534CFC.30909@egenix.com> Message-ID: <4A53C160.6040400@v.loewis.de> >> I've seen a few discussions about this, but no final statement >> of what strategy to follow and whether hg makes this easier (AFAIR, >> that was the main argument for switching to hg). > > I think the main reason for switching was that it would make it easier > for non-core-committers to maintain branches and submit patches (as > changesets core committers can pull). Indeed, it's all in PEP 374: - motivation 1: give direct tool support to non-committers - motivation 2: allow disconnected (off-line) access to the repository - motivation 3: allow revisions of a patch - motivation 4: allow tracking the mainline while working on a patch - motivation 5: avoid using arcane third-party tools for merge tracking So merge tracking was not the main argument, but the fifth. Regards, Martin From ncoghlan at gmail.com Tue Jul 7 23:43:56 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 08 Jul 2009 07:43:56 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A53BEE0.3000703@v.loewis.de> References: <4A520F83.7020300@egenix.com> <4A526B11.7060308@gmail.com> <4A527363.7070305@v.loewis.de> <4A527514.5040709@gmail.com> <4A530453.10005@egenix.com> <4A534E7E.7010700@egenix.com> <4A53BEE0.3000703@v.loewis.de> Message-ID: <4A53C19C.6040502@gmail.com> Martin v. L?wis wrote: > I think you are proposing that there is no prefix because the chance > for a misidentification of some word as a hg revision number is small, > as it has to be exactly 12 hex digits. So triggering it accidentally would require a 12 letter word or object name that used only the letter a-f? Agreed that seems very unlikely. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Tue Jul 7 23:46:51 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 07 Jul 2009 23:46:51 +0200 Subject: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ? In-Reply-To: <7c1ab96d0907071038p4ce29935j2580420de05eaf4b@mail.gmail.com> References: <7c1ab96d0907071038p4ce29935j2580420de05eaf4b@mail.gmail.com> Message-ID: <4A53C24B.7080202@v.loewis.de> > Since many C applications take argv as a (char **) it seems reasonable > for python to keep a function for the C api that accepts a (char **) > argument for argv. I'm not quite sure what you are suggesting: either that there is a function in the C API that accepts a (char**), or that PySys_SetArgv be that function. The latter alternative is out of question, as it breaks the Windows port of Python - there is simply no way of representing argv with char** in a meaningful way. It also breaks backwards compatibility. As for creating another function: please provide a patch to bugs.python.org. I believe this patch is *very* difficult to implement, unless the function can accept some severe limitations. Regards, Martin From dalcinl at gmail.com Wed Jul 8 00:18:51 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 7 Jul 2009 19:18:51 -0300 Subject: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ? In-Reply-To: References: <7c1ab96d0907071038p4ce29935j2580420de05eaf4b@mail.gmail.com> <4A53C24B.7080202@v.loewis.de> Message-ID: Sorry... ---------- Forwarded message ---------- From: Lisandro Dalcin Date: Tue, Jul 7, 2009 at 7:16 PM Subject: Re: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ? To: "Martin v. L?wis" On Tue, Jul 7, 2009 at 6:46 PM, "Martin v. L?wis" wrote: >> Since many C applications take argv as a (char **) it seems reasonable >> for python to keep a function for the C api that accepts a ?(char **) >> argument for argv. > I've run in the same issue, and basically had to copy&paste from Python sources... > > I'm not quite sure what you are suggesting: either that there is a > function in the C API that accepts a (char**), or that PySys_SetArgv > be that function. > > The latter alternative is out of question, as it breaks the Windows > port of Python - there is simply no way of representing argv with > char** in a meaningful way. It also breaks backwards compatibility. > Indeed... > As for creating another function: please provide a patch to > bugs.python.org. I believe this patch is *very* difficult to > implement, unless the function can accept some severe limitations. > What about the Python lib exposing a public PyOS_char2wchar() ? No idea how to write a Windows implementation, though.. -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From stephen at xemacs.org Wed Jul 8 03:55:27 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 08 Jul 2009 10:55:27 +0900 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> Message-ID: <873a98ueeo.fsf@uwakimon.sk.tsukuba.ac.jp> Tarek Ziad? writes: > On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby wrote: > > No one expects the package inquisition. ?;-) > Sorry, i've looked in the english dictionary but I don't get this one. I think that far more important than PEP 385 conversion of Roundup and other utilities to recognize Mercurial revision identifiers is a filter for Python channels that recognizes Monty Python references and automatically inserts YouTube video URLs for this essential background. From stephen at xemacs.org Wed Jul 8 04:19:39 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 08 Jul 2009 11:19:39 +0900 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: <4A520F83.7020300@egenix.com> <4A534CFC.30909@egenix.com> Message-ID: <871vorvrus.fsf@uwakimon.sk.tsukuba.ac.jp> R. David Murray writes: > On Tue, 7 Jul 2009 at 15:26, M.-A. Lemburg wrote: > > I've seen a few discussions about this, but no final statement > > of what strategy to follow and whether hg makes this easier (AFAIR, > > that was the main argument for switching to hg). Yes, yes, yes, and no. In reverse order, no: the main argument for switching to hg is that it makes private branching easier. Yes: hg will make public branching easier, too, but that can't be proved until the workflow adjustments get worked out. For that reason, it is essential that the current workflow be supportable essentially without change, and it is. With respect to "how", I'm not a Mercurial geek, but I have been working with Mercurial queues a bit recently in another area, and I think they have some promise for helping organize the workflow. (Although by themselves they're clearly not sufficient, since they're oriented to a single branch.) Yes: there has been no final statement of what strategy to follow because opinions are extremely varied, even as to what is feasible with Mercurial. For example, Dirkjan and Georg want a workflow that makes moving patches among the public branches worry-free Mercurial merges. I believe that means (to the extent it is implemented) essentially gutting the current strategy of cutting maintenance branches and simply lagging the maintenance branches relative to the dev branches, and that it's infeasible for py3k vs. py2. I can't substantiate that; maybe the patch flow would support what they want, I'm not that familiar with how much patches currently morph across branches. And yes: there are a few inconclusive discussions. That's why PEP 374 was written consciously with the intent of postponing the hard issues of workflow across the public branches in favor of picking off the low hanging fruit of private branching and disconnected version control. > I think the main reason for switching was that it would make it easier > for non-core-committers to maintain branches and submit patches (as > changesets core committers can pull). Patches or "bundles" aka merge directives. "Pulling" submissions is probably not going to happen; that's a practice that is common with highly distributed workflows, but Python has a fairly centralized workflow. > but I think it remains to be proven/worked out. And I believe there is > no tool like svnmerge for tracking changesets to be merged, which could > be an issue that needs a resolution. I think that Mercurial queues or some related extension can be adapted to this, but I can't say for sure (after all, I was the git person on the PEP 374 team :-). > IIUC, the discussion about named versus cloned branches is part of > figuring out the workflow.... Peripherally. But actually it's not really relevant to workflow. Anything that can be done with named branches can be done with cloned branches, possibly requiring substantially more space. That discussion really is about whether anything is *lost* by using named branches. I worry that something is lost, but the discussion so far has been inconclusive. From ideasman42 at gmail.com Wed Jul 8 06:06:07 2009 From: ideasman42 at gmail.com (Campbell Barton) Date: Tue, 7 Jul 2009 21:06:07 -0700 Subject: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ? In-Reply-To: References: <7c1ab96d0907071038p4ce29935j2580420de05eaf4b@mail.gmail.com> <4A53C24B.7080202@v.loewis.de> Message-ID: <7c1ab96d0907072106j747dcd5au7f5813af452df3c6@mail.gmail.com> The reason I was a big vague is that I'm not really bothered HOW its done, I'd just like there to be some way to use PySys_SetArgv with a (char **) Definitely not suggesting that PySys_SetArgv be reverted to the way it worked in py2.x, if there was 2 versions of PySys_SetArgv that would be acceptable to me. PySys_SetArgvASCII() maybe? Alternately a function to convert the char **argv to wchar_t **argv could be ok too. PyOS_char2wchar would be useful though it still means you need to manually copy the argv, convert and free it. On Tue, Jul 7, 2009 at 3:18 PM, Lisandro Dalcin wrote: > Sorry... > > ---------- Forwarded message ---------- > From: Lisandro Dalcin > Date: Tue, Jul 7, 2009 at 7:16 PM > Subject: Re: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ? > To: "Martin v. L?wis" > > > On Tue, Jul 7, 2009 at 6:46 PM, "Martin v. L?wis" wrote: >>> Since many C applications take argv as a (char **) it seems reasonable >>> for python to keep a function for the C api that accepts a ?(char **) >>> argument for argv. >> > > I've run in the same issue, and basically had to copy&paste from > Python sources... > >> >> I'm not quite sure what you are suggesting: either that there is a >> function in the C API that accepts a (char**), or that PySys_SetArgv >> be that function. >> >> The latter alternative is out of question, as it breaks the Windows >> port of Python - there is simply no way of representing argv with >> char** in a meaningful way. It also breaks backwards compatibility. >> > > Indeed... > >> As for creating another function: please provide a patch to >> bugs.python.org. I believe this patch is *very* difficult to >> implement, unless the function can accept some severe limitations. >> > > What about the Python lib exposing a public PyOS_char2wchar() ? > > No idea how to write a Windows implementation, though.. > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > > > > -- > Lisandro Dalc?n > --------------- > Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) > Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) > Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) > PTLC - G?emes 3450, (3000) Santa Fe, Argentina > Tel/Fax: +54-(0)342-451.1594 > -- - Campbell From pje at telecommunity.com Wed Jul 8 07:09:09 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 08 Jul 2009 01:09:09 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com > References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <94bdd2610907070050o281a7822x816eececdce8e159@mail.gmail.com> <79990c6b0907070133v16fb264cw30ccf42783fb97e7@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> Message-ID: <20090708050608.AC2C63A407B@sparrow.telecommunity.com> At 11:30 PM 7/7/2009 +0200, Tarek Ziad? wrote: >On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby wrote: > > At 03:23 PM 7/7/2009 +0200, Tarek Ziad? wrote: > >> > >> When I started to work on this I didn't realize the gigantic amount of > >> work and coordination it requires > > > > No one expects the package inquisition. ;-) > > > > > >Sorry, i've looked in the english dictionary but I don't get this one. >what do you mean ? It's a Monty Python reference, i.e. "no one expects the Spanish Inquisition". In the Spanish Inquisition sketch, a character is asked several questions he can't answer, and then complains that he wasn't expecting some kind of Spanish inquisition, at which point three scarlet-robed cardinals burst into the room and exclaim, "NO ONE expects the Spanish Inquisition!" So I punned on that in order to imply that nobody who takes on the job of packaging expects to be asked the kind of detailed, unanswerable questions that come with the territory of packaging, or to be subjected to torture by third parties merely for trying to do something useful. And, by further implication, I was thus expressing camaraderie with you on your initiation into the wonderful world of trying to satisfy everyone in Pythonland, and our shared experience in the area of "if I'd really known what I was getting into, I probably wouldn't have". ;-) From martin at v.loewis.de Wed Jul 8 08:13:18 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 08 Jul 2009 08:13:18 +0200 Subject: [Python-Dev] Suggested PySys_SetArgv use with a (char **) argv ? In-Reply-To: <7c1ab96d0907072106j747dcd5au7f5813af452df3c6@mail.gmail.com> References: <7c1ab96d0907071038p4ce29935j2580420de05eaf4b@mail.gmail.com> <4A53C24B.7080202@v.loewis.de> <7c1ab96d0907072106j747dcd5au7f5813af452df3c6@mail.gmail.com> Message-ID: <4A5438FE.3050608@v.loewis.de> > The reason I was a big vague is that I'm not really bothered HOW its > done, I'd just like there to be some way to use PySys_SetArgv with a > (char **) Ok, that's easy: void PySys_SetArgvChar(int argc, char** argv) { PySys_SetArgv(0, NULL); } So you don't bother *how* it's done as long as you can pass char** in some way. > Alternately a function to convert the char **argv to wchar_t **argv > could be ok too. Convert in what manner? What is the encoding of your char**? Regards, Martin From ziade.tarek at gmail.com Wed Jul 8 08:26:45 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 8 Jul 2009 08:26:45 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <20090708050608.AC2C63A407B@sparrow.telecommunity.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> <20090708050608.AC2C63A407B@sparrow.telecommunity.com> Message-ID: <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> On Wed, Jul 8, 2009 at 7:09 AM, P.J. Eby wrote: > At 11:30 PM 7/7/2009 +0200, Tarek Ziad? wrote: >> >> On Tue, Jul 7, 2009 at 10:31 PM, P.J. Eby wrote: >> > At 03:23 PM 7/7/2009 +0200, Tarek Ziad? wrote: >> >> >> >> When I started to work on this I didn't realize the gigantic amount of >> >> work and coordination it requires >> > >> > No one expects the package inquisition. ?;-) >> > >> > >> >> Sorry, i've looked in the english dictionary but I don't get this one. >> what do you mean ? > > It's a Monty Python reference, i.e. "no one expects the Spanish > Inquisition". > > In the Spanish Inquisition sketch, a character is asked several questions he > can't answer, and then complains that he wasn't expecting some kind of > Spanish inquisition, at which point three scarlet-robed cardinals burst into > the room and exclaim, "NO ONE expects the Spanish Inquisition!" > > So I punned on that in order to imply that nobody who takes on the job of > packaging expects to be asked the kind of detailed, unanswerable questions > that come with the territory of packaging, or to be subjected to torture by > third parties merely for trying to do something useful. > > And, by further implication, I was thus expressing camaraderie with you on > your initiation into the wonderful world of trying to satisfy everyone in > Pythonland, and our shared experience in the area of "if I'd really known > what I was getting into, I probably wouldn't have". ?;-) Hehe very nice, thanks for the explanation :) (thanks to the others too) > > -- Tarek Ziad? | http://ziade.org From ncoghlan at gmail.com Wed Jul 8 12:41:26 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 08 Jul 2009 20:41:26 +1000 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> <20090708050608.AC2C63A407B@sparrow.telecommunity.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> Message-ID: <4A5477D6.1080708@gmail.com> Tarek Ziad? wrote: > On Wed, Jul 8, 2009 at 7:09 AM, P.J. Eby wrote: >> So I punned on that in order to imply that nobody who takes on the job of >> packaging expects to be asked the kind of detailed, unanswerable questions >> that come with the territory of packaging, or to be subjected to torture by >> third parties merely for trying to do something useful. >> >> And, by further implication, I was thus expressing camaraderie with you on >> your initiation into the wonderful world of trying to satisfy everyone in >> Pythonland, and our shared experience in the area of "if I'd really known >> what I was getting into, I probably wouldn't have". ;-) > > Hehe very nice, thanks for the explanation :) > > (thanks to the others too) I think is exchange actually a good reminder that the bar for acceptance of PEP 376 shouldn't be "Addresses every possible packaging issue we can come up with". That would be setting expectations far too high because packaging cross-platform is so messy. A more appropriate bar is probably: a) Better that what we have now; and b) Provides possible avenues for even more improvement in the future For example, I quite like the concept behind the various ideas for "location" or "prefix" definitions either in the RECORD file itself or in a separate PREFIXES file, since such approaches feeds directly in to part b) above. Once some kind prefix definition mechanism is in place in the metadata, it becomes possible to leverage it to categorise files for the benefit of system packagers while still allowing simple internal-use-only packages to just bundle everything together. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From solipsis at pitrou.net Wed Jul 8 12:50:59 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 8 Jul 2009 10:50:59 +0000 (UTC) Subject: [Python-Dev] PEP 376 - Open questions References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> <20090708050608.AC2C63A407B@sparrow.telecommunity.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> <4A5477D6.1080708@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > > I think is exchange actually a good reminder that the bar for acceptance > of PEP 376 shouldn't be "Addresses every possible packaging issue we can > come up with". That would be setting expectations far too high because > packaging cross-platform is so messy. +1 > For example, I quite like the concept behind the various ideas for > "location" or "prefix" definitions either in the RECORD file itself or > in a separate PREFIXES file, since such approaches feeds directly in to > part b) above. I don't understand the point. Since RECORD is specific to a particular machine and installation (it is generated at install time), what would you gain by adding a fake genericity to its entries? Regards Antoine. From ncoghlan at gmail.com Wed Jul 8 13:08:16 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 08 Jul 2009 21:08:16 +1000 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> <20090708050608.AC2C63A407B@sparrow.telecommunity.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> <4A5477D6.1080708@gmail.com> Message-ID: <4A547E20.5030606@gmail.com> Antoine Pitrou wrote: > Nick Coghlan gmail.com> writes: >> For example, I quite like the concept behind the various ideas for >> "location" or "prefix" definitions either in the RECORD file itself or >> in a separate PREFIXES file, since such approaches feeds directly in to >> part b) above. > > I don't understand the point. Since RECORD is specific to a particular machine > and installation (it is generated at install time), what would you gain by > adding a fake genericity to its entries? My understanding is that RECORD isn't generated at install time for the bdist_* commands -it's generated at the time the binary distribution is created. Hence the problems raised with the idea of having absolute paths in RECORD - if they were written directly into RECORD they would reflect where the files were when the distribution was created, not where they end up when it is installed. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From p.f.moore at gmail.com Wed Jul 8 14:13:55 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Jul 2009 13:13:55 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A547E20.5030606@gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> <20090708050608.AC2C63A407B@sparrow.telecommunity.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> <4A5477D6.1080708@gmail.com> <4A547E20.5030606@gmail.com> Message-ID: <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> 2009/7/8 Nick Coghlan : > Antoine Pitrou wrote: >> Nick Coghlan gmail.com> writes: >>> For example, I quite like the concept behind the various ideas for >>> "location" or "prefix" definitions either in the RECORD file itself or >>> in a separate PREFIXES file, since such approaches feeds directly in to >>> part b) above. >> >> I don't understand the point. Since RECORD is specific to a particular machine >> and installation (it is generated at install time), what would you gain by >> adding a fake genericity to its entries? > > My understanding is that RECORD isn't generated at install time for the > bdist_* commands -it's generated at the time the binary distribution is > created. Hence the problems raised with the idea of having absolute > paths in RECORD - if they were written directly into RECORD they would > reflect where the files were when the distribution was created, not > where they end up when it is installed. This is a key open question, IMHO. No-one has done any work on updating the core bdist_xxx commands to the new structure, and the PEP is silent on the matter, so it's still all speculation. The current "best guess" assumption is that, given that bdist_xxx uses a "setup.py install --root=xxx" incantation to generate the structure to package, the RECORD can be picked up from this. There are a number of issues with this assumption, though - it doesn't handle absolute locations well (there are existing bugs), paths may be affected by the environment at the time the installer is built, and it requires RECORD to be relocatable. An alternative solution would be for the bdist_xxx commands to ignore the RECORD file generated in the temp area, and build its own on the target machine when the installer is run. This is conceptuially far cleaner, it's in line with the implicit assumptions in the PEP, and it avoids a lot of the problems with path formats (the filenames in RECORD could be absolute local pathnames, and that's the end of the story). But - and it's a big but - this dumps RECORD file handling on the bdist_xxx command, and they all need to implement the same code, resulting in a lot of duplication and opportunity for inconsistency. I have no answer here. But I think we need feedback from the people who will ultimately be building bdist_xxx formats - Debian packagers, people wrting alternate RPM builders, etc. There's also Phillip commenting from the setuptools POV and MAL who has mentioned his build process. Bottom line - Is RECORD to be created on the target machine at install time, or must it be relocatable? Subsequent question - if it's to be relocatable, how will that work? Paul. From solipsis at pitrou.net Wed Jul 8 14:24:34 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 8 Jul 2009 12:24:34 +0000 (UTC) Subject: [Python-Dev] PEP 376 - Open questions References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> <20090708050608.AC2C63A407B@sparrow.telecommunity.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> <4A5477D6.1080708@gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> Message-ID: Paul Moore gmail.com> writes: > > I have no answer here. But I think we need feedback from the people > who will ultimately be building bdist_xxx formats - Debian packagers, > people wrting alternate RPM builders, etc. I think any bdist_xxx command which targets externally handled package formats (rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or dpkg, you will uninstall with the same tool, won't you? > Bottom line - Is RECORD to be created on the target machine at install > time, or must it be relocatable? This is quite an uninformed opinion, but a relocatable RECORD looks like a can of worms to me. As for creating RECORD at install time, I don't understand the argument about duplicate code. Distutils can provide useful primitives for generating RECORD, so that interested subcommands don't have a lot of additional work to do. From p.f.moore at gmail.com Wed Jul 8 14:54:49 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Jul 2009 13:54:49 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> <20090708050608.AC2C63A407B@sparrow.telecommunity.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> <4A5477D6.1080708@gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> Message-ID: <79990c6b0907080554t49de466cpde75258fa8e02bee@mail.gmail.com> 2009/7/8 Antoine Pitrou : > Paul Moore gmail.com> writes: >> >> I have no answer here. But I think we need feedback from the people >> who will ultimately be building bdist_xxx formats - Debian packagers, >> people wrting alternate RPM builders, etc. > > I think any bdist_xxx command which targets externally handled package formats > (rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or > dpkg, you will uninstall with the same tool, won't you? I thought the purpose of RECORD was (also) so that *other* installs could check who "owned" an existing file, before potentially overwriting it. That seems like it's the reasoning behind the uses() function. But the PEP is very light on practical applications, so it's hard to tell what the methods are intended to be used for. >> Bottom line - Is RECORD to be created on the target machine at install >> time, or must it be relocatable? > > This is quite an uninformed opinion, but a relocatable RECORD looks like a can > of worms to me. > > As for creating RECORD at install time, I don't understand the argument about > duplicate code. Distutils can provide useful primitives for generating RECORD, > so that interested subcommands don't have a lot of additional work to do. The bdist_wininst code is in C, and the bdist_msi code is driven by the MSI data format. I don't know what scripting options exist for .deb and .rpm files. Different languages, so no code sharing seems possible. Maybe I'm wrong, and something could be done along the lines of a "pkgutil.generate_record_file()" function which could be run from a post-install script written in Python, but I don't know how plausible or robust that would be... Paul. From eric at trueblade.com Wed Jul 8 14:59:57 2009 From: eric at trueblade.com (Eric Smith) Date: Wed, 08 Jul 2009 08:59:57 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> <20090708050608.AC2C63A407B@sparrow.telecommunity.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> <4A5477D6.1080708@gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> Message-ID: <4A54984D.4020305@trueblade.com> Antoine Pitrou wrote: > Paul Moore gmail.com> writes: >> I have no answer here. But I think we need feedback from the people >> who will ultimately be building bdist_xxx formats - Debian packagers, >> people wrting alternate RPM builders, etc. > > I think any bdist_xxx command which targets externally handled package formats > (rpm, deb, msi, etc.) shouldn't generate RECORD. If you install with rpm or > dpkg, you will uninstall with the same tool, won't you? I agree with this. For RPM's, there's a whole other database of what files were installed. Is it really the intent that a file will be managed by multiple different installers? That I can install with RPM but remove with some python-installer (or other) tool? That way lies madness. In fact, I see RECORD as an installer-specific detail that doesn't need to be standardized at all. For bdist_rpm (which I'm contemplating taking over, but no promises), the only thing that needs to be done with the metadata is be able to build a .spec file. Once the .spec file is generated, rpmbuild (or equivalent) will produce the .rpm files. I image that .deb's are similar, but I'm no expert. Which brings up the "static metadata" argument I was making at PyCon. I want there to be enough static metadata that the new bdist_rpm can read so that it produces a well-behaved .spec file. I need to know the intent of some of the files (which are documentation, which are config files, which are scripts, which are libraries, etc.) and not much else. I envision the new bdist_rpm as a standalone tool, not a distutils extension. >> Bottom line - Is RECORD to be created on the target machine at install >> time, or must it be relocatable? > > This is quite an uninformed opinion, but a relocatable RECORD looks like a can > of worms to me. Agreed. > As for creating RECORD at install time, I don't understand the argument about > duplicate code. Distutils can provide useful primitives for generating RECORD, > so that interested subcommands don't have a lot of additional work to do. I envision a "distlib" that contains new support code that's not part of the current distutils morass. From erik at cq2.nl Wed Jul 8 14:43:29 2009 From: erik at cq2.nl (Erik Groeneveld) Date: Wed, 8 Jul 2009 14:43:29 +0200 Subject: [Python-Dev] Define metatype and a type that uses it Message-ID: L.S. I am working on an extension that allows one to open Java .jar files and use the objects defined in it (via CNI or JNI) in Python. It works like a charm, but I am stuck with the definition of a metatype for the wrappers. I have to do this in Python now, with a helper class. I did find examples of how to define metatypes, but I can't find anything on how to declare one type using this metatype. Could anyone point me to examples or documentation or help me fix my code below? What I have is a type JObject that has a metatype JObjectMeta, but the objects created from it seem not properly initialized. If I call PyString_Str(self) in JObjectMeta_init, it seg faults. De Python code that initiates this is: class MyJObject(JObject): pass Below is the complete, contained code of the extension. Thanks in advance, Erik Groeneveld #include /*************** JObjectMeta *******************/ typedef struct { PyObject_HEAD } JObjectMeta; PyTypeObject JObjectMetaType = { PyObject_HEAD_INIT(NULL) }; int JObjectMeta_init(PyObject* self, PyObject* args, PyObject *kwargs) { printf("before str\n"); PyObject_Str(self); printf("after str\n"); return 0; } PyObject* JObjectMeta_getattro(PyObject *self, PyObject *name) { return PyBaseObject_Type.tp_getattro(self, name); } /*************** JObject ***********************/ typedef struct { PyObject_HEAD } JObject; PyTypeObject JObjectType = { PyObject_HEAD_INIT(&JObjectMetaType) }; int JObject_init(PyObject* self, PyObject* args, PyObject *kwargs) { return 0; } static PyMethodDef methods[] = { {NULL, NULL, 0, NULL} }; /*************** INIT *******************/ PyMODINIT_FUNC initmetaclass(void) { JObjectMetaType.tp_name = "metaclass.JObjectMeta"; JObjectMetaType.tp_basicsize = sizeof(JObjectMeta); JObjectMetaType.tp_flags = Py_TPFLAGS_DEFAULT; JObjectMetaType.tp_doc = "JObject objects"; JObjectMetaType.tp_init = JObjectMeta_init; JObjectMetaType.tp_base = &PyType_Type; // must ? JObjectMetaType.tp_new = PyType_GenericNew; JObjectMetaType.tp_getattro = JObjectMeta_getattro; JObjectType.tp_name = "metaclass.JObject"; JObjectType.tp_basicsize = sizeof(JObject); JObjectType.tp_flags = Py_TPFLAGS_DEFAULT|Py_TPFLAGS_BASETYPE; JObjectType.tp_doc = "JObjectMeta objects"; JObjectType.tp_init = JObject_init; JObjectType.tp_base = &JObjectMetaType; // must ? JObjectType.tp_new = PyType_GenericNew; PyObject* module = Py_InitModule3("metaclass", methods, "Metaclass try out."); if (module == NULL) { PyErr_Print(); exit(-1); } if (PyType_Ready(&JObjectMetaType) < 0) { PyErr_Print(); exit(-1); } if (PyModule_AddObject(module, "JObjectMeta", (PyObject*) &JObjectMetaType) < 0) { PyErr_Print(); exit(-1); } if (PyType_Ready(&JObjectType) < 0) { PyErr_Print(); exit(-1); } if (PyModule_AddObject(module, "JObject", (PyObject*) &JObjectType) < 0 ) { PyErr_Print(); exit(-1); } } From lists at cheimes.de Wed Jul 8 15:47:45 2009 From: lists at cheimes.de (Christian Heimes) Date: Wed, 08 Jul 2009 15:47:45 +0200 Subject: [Python-Dev] Define metatype and a type that uses it In-Reply-To: References: Message-ID: Erik Groeneveld wrote: > L.S. > > I am working on an extension that allows one to open Java .jar files > and use the objects defined in it (via CNI or JNI) in Python. It > works like a charm, but I am stuck with the definition of a metatype > for the wrappers. I have to do this in Python now, with a helper > class. Did you know that Andi Vajda has already created a fantastic wrapper for Java called JCC? http://pypi.python.org/pypi/JCC/2.3 Contrary to your project it doesn't create the wrappers in runtime. It analyzes the jar file(s) with JNI, created C++ templates and wraps the C++ code in Python wrappers. JCC even allows you to implement native methods of Java classes in Python. Christian From p.f.moore at gmail.com Wed Jul 8 16:57:08 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Jul 2009 15:57:08 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A54AF4F.9080800@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> <4A5477D6.1080708@gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> Message-ID: <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> 2009/7/8 Eric Smith : > [Any reason you didn't respond to the list? I'd like to continue the > discussion there. But I'm currently replying just to you.] No, I just clicked the wrong button. Sorry. Redirecting back to the list. > Paul Moore wrote: >> >> 2009/7/8 Eric Smith : >>> >>> I agree with this. For RPM's, there's a whole other database of what >>> files >>> were installed. Is it really the intent that a file will be managed by >>> multiple different installers? That I can install with RPM but remove >>> with >>> some python-installer (or other) tool? That way lies madness. In fact, I >>> see >>> RECORD as an installer-specific detail that doesn't need to be >>> standardized >>> at all. >> >> I thought RECORD was to ensure that you can install with RPM, and >> *not* have any other random uninstaller remove your files (because you >> populate RECORD, and the other uninstaller uses >> pkgutil.get_file_users(path) to find out that you care about the file, >> and hence avoid deleting it). > > I really don't think this is a use case worth supporting. A single file > managed by multiple installers is a mess waiting to happen. RPM won't let > you do it. About the only use case I can think of is namespace packages, and > that can be handled by factoring out the __init__.py file into it's own > install package. Personally, I think you're right. But I have essentially zero experience with the non-Windows use cases, so I don't feel qualified to judge. It seems to me that a lot of the fancy detecting what other people do stuff is motivated by setuptools. I'm not going to comment on that, other than to say that it may be that setuptools is generating more problems than it's solving... >> Basically, other install methods can't read your (RPM) database, and >> you can't read their database. RECORD is a simple common format, >> providing just enough information for everyone to communicate the >> message "I manage this file". > > The RPM uninstaller won't look at the RECORD file. I think the best we can > say is that _if_ an installer wants to share state about what's installed > with other installers, this is how it's done. OK. I'd be stunned if bdist_wininst or bdist_msi ends up checking RECORD. So that's 3 "not interested" votes. Who's going to use the APIs based around the RECORD file? Anyone? Seriously - who is actually going to USE any of the APIs being proposed in PEP 376? If no-one's interested, we should stop right now. Do any of the people from the summit read this list? If not, can anyone contact them and get some feedback? Paul. From pje at telecommunity.com Wed Jul 8 17:11:44 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 08 Jul 2009 11:11:44 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A54984D.4020305@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> <20090708050608.AC2C63A407B@sparrow.telecommunity.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> <4A5477D6.1080708@gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> Message-ID: <20090708150844.EC3843A407B@sparrow.telecommunity.com> At 01:13 PM 7/8/2009 +0100, Paul Moore wrote: >An alternative solution would be for the bdist_xxx commands to ignore >the RECORD file generated in the temp area, and build its own on the >target machine when the installer is run. This is conceptuially far >cleaner, it's in line with the implicit assumptions in the PEP, and it >avoids a lot of the problems with path formats (the filenames in >RECORD could be absolute local pathnames, and that's the end of the >story). But - and it's a big but - this dumps RECORD file handling on >the bdist_xxx command, and they all need to implement the same code, >resulting in a lot of duplication and opportunity for inconsistency. The PREFIXES approach I suggested doesn't have these problems; the bdist installer only needs to write a PREFIXES file, and doesn't need to touch RECORD (which would be generated by the install --root). In fact, since PREFIXES is strictly advisory, it can be generated by the install --root and left alone at installation time. And, if it's generated without the --root prefix included, RPMs would have a correct PREFIXES file without further manipulation. Plus, the bdist_wininst .exe installer can easily write a PREFIXES file, since it has to know the prefixes in order to do an installation in the first place. (We still need to hash out an initial list of prefixes, of course.) At 08:59 AM 7/8/2009 -0400, Eric Smith wrote: >I agree with this. For RPM's, there's a whole other database of what >files were installed. Is it really the intent that a file will be >managed by multiple different installers? That I can install with >RPM but remove with some python-installer (or other) tool? That way >lies madness. In fact, I see RECORD as an installer-specific detail >that doesn't need to be standardized at all. This is a misunderstanding. The purpose is to let an *installer* (like easy_install) know that these files belong to some other installer, and not allow overwriting them. That's why there's also an INSTALLER file. From pje at telecommunity.com Wed Jul 8 17:19:48 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 08 Jul 2009 11:19:48 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.co m> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> <4A5477D6.1080708@gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> Message-ID: <20090708151647.9E0703A407B@sparrow.telecommunity.com> At 03:57 PM 7/8/2009 +0100, Paul Moore wrote: >Who's going to use the APIs based around the RECORD file? Anyone? The distutils uninstall facility, for starters. easy_install and pip also will, eventually. In all three cases, the use will be in order to avoid overwriting files belonging to a system package manager, and in order to uninstall/upgrade their own files. (I can't personally commit that pip will need this; but it's currently the only installer besides the distutils that installs things in this format, and would therefore need this to prevent overwriting things.) Setuptools will also need to generate RECORD, INSTALLER, and PREFIXES appropriately for both development and actual installations. >Seriously - who is actually going to USE any of the APIs being >proposed in PEP 376? If no-one's interested, we should stop right now. >Do any of the people from the summit read this list? If not, can >anyone contact them and get some feedback? Package management people aren't likely to be doing anything with this stuff directly; if they need a list of installed files they're probably using --root or --record at package build time, not needing to sniff stuff and installation time, and they already have installation facilities. Their only real need in relation to this will be to set an option to generate an INSTALLER file that says you can't uninstall their packages manually, thereby preventing the package from being uninstalled by the distutils utility, or any other well-behaved PEP 376-based package management tool. From amauryfa at gmail.com Wed Jul 8 17:35:21 2009 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 8 Jul 2009 17:35:21 +0200 Subject: [Python-Dev] Define metatype and a type that uses it In-Reply-To: References: Message-ID: Hello, 2009/7/8 Erik Groeneveld : > I am working on an extension that allows one to open Java .jar files > and use the objects defined in it (via CNI or JNI) in Python. ?It > works like a charm, but I am stuck with the definition of a metatype > for the wrappers. ?I have to do this in Python now, with a helper > class. > > I did find examples of how to define metatypes, but I can't find > anything on how to declare one type using this metatype. > > Could anyone point me to examples or documentation or help me fix my code below? [...] There are several changes to do in your code: - Don't define a JObjectMeta struct, use JObjectType directly instead. An instance of the metatype is the type itself! - Don't set JObjectMetaType.tp_basicsize, let it inherit from the base type (the correct value would be sizeof(PyHeapTypeObject), this is important in order to create derived classes in python) - Don't set JObjectMetaType.tp_new, let it inherit from the base type (PyType_GenericNew just allocates memory; types are more complex...) - JObjectType.tp_base = &JObjectMetaType" is wrong, it should be "JObjectType.ob_type = &JObjectMetaType;", or better to be compatible with python3: "Py_TYPE(JObjectType) = &JObjectMetaType;" In summary, I made your example work with: JObjectMetaType.tp_name = "metaclass.JObjectMeta"; JObjectMetaType.tp_flags = Py_TPFLAGS_DEFAULT; JObjectMetaType.tp_doc = "JObjectMeta objects"; JObjectMetaType.tp_init = JObjectMeta_init; JObjectMetaType.tp_base = &PyType_Type; JObjectMetaType.tp_getattro = JObjectMeta_getattro; Py_TYPE(&JObjectType) = &JObjectMetaType; JObjectType.tp_name = "metaclass.JObject"; JObjectType.tp_basicsize = sizeof(JObject); JObjectType.tp_flags = Py_TPFLAGS_DEFAULT|Py_TPFLAGS_BASETYPE; JObjectType.tp_doc = "JObject objects"; JObjectType.tp_init = JObject_init; -- Amaury Forgeot d'Arc From ziade.tarek at gmail.com Wed Jul 8 17:36:36 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 8 Jul 2009 17:36:36 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <20090708150844.EC3843A407B@sparrow.telecommunity.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <20090708050608.AC2C63A407B@sparrow.telecommunity.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> <4A5477D6.1080708@gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <20090708150844.EC3843A407B@sparrow.telecommunity.com> Message-ID: <94bdd2610907080836q55d68de4od7a8b55d1ee1ae0a@mail.gmail.com> On Wed, Jul 8, 2009 at 5:11 PM, P.J. Eby wrote: > At 01:13 PM 7/8/2009 +0100, Paul Moore wrote: >> >> An alternative solution would be for the bdist_xxx commands to ignore >> the RECORD file generated in the temp area, and build its own on the >> target machine when the installer is run. This is conceptuially far >> cleaner, it's in line with the implicit assumptions in the PEP, and it >> avoids a lot of the problems with path formats (the filenames in >> RECORD could be absolute local pathnames, and that's the end of the >> story). But - and it's a big but - this dumps RECORD file handling on >> the bdist_xxx command, and they all need to implement the same code, >> resulting in a lot of duplication and opportunity for inconsistency. > > The PREFIXES approach I suggested doesn't have these problems; the bdist > installer only needs to write a PREFIXES file, and doesn't need to touch > RECORD (which would be generated by the install --root). ?In fact, since > PREFIXES is strictly advisory, it can be generated by the install --root and > left alone at installation time. ?And, if it's generated without the --root > prefix included, RPMs would have a correct PREFIXES file without further > manipulation. ?Plus, the bdist_wininst .exe installer can easily write a > PREFIXES file, since it has to know the prefixes in order to do an > installation in the first place. > > (We still need to hash out an initial list of prefixes, of course.) Instead of building this list of prefixes from scratch, let's use the ones that are *actually* used when the install command is called, have a look at the list I've suggested : http://mail.python.org/pipermail/python-dev/2009-July/090514.html And let's drop other absolute paths, From p.f.moore at gmail.com Wed Jul 8 18:22:52 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Jul 2009 17:22:52 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <20090708151647.9E0703A407B@sparrow.telecommunity.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> Message-ID: <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> 2009/7/8 P.J. Eby : > At 03:57 PM 7/8/2009 +0100, Paul Moore wrote: >> >> Who's going to use the APIs based around the RECORD file? Anyone? > > The distutils uninstall facility, for starters. ?easy_install and pip also > will, eventually. Is pip getting this via setuptools, or will it be independently changing its code? (Who's the pip contact who could confirm this directly?) > In all three cases, the use will be in order to avoid overwriting files > belonging to a system package manager, and in order to uninstall/upgrade > their own files. To uninstall/upgrade their own files counts as "an installer-specific detail" in the sense that Eric stated it. To avoid overwriting files belonging to a system package manager: That sounds like we're looking at a use case where people use a mixture of tools to install packages into the system Python. Personally, I never do this (I use bdist_wininst exclusively). The cases I can imagine are: - no OS-specific package available, and the user can't build his own, so he uses setup.py install. - user using easy_install as well as the OS package In case (1), I'd imagine the user is careful enough (given that he's doing everything manually) to avoid installing something that's already present. Case (2) is setuptools-related (see below for setuptools-related comments). > (I can't personally commit that pip will need this; but it's currently the > only installer besides the distutils that installs things in this format, > and would therefore need this to prevent overwriting things.) Not the only one - MAL mentioned his (internal use?) packaging process. > Setuptools will also need to generate RECORD, INSTALLER, and PREFIXES > appropriately for both development and actual installations. If no-one else is using them, that's an internal setuptools detail again. Or it's for interoperability between setuptools and distutils (and other packagers don't benefit unless they are using PEP 376 APIs). >> Seriously - who is actually going to USE any of the APIs being >> proposed in PEP 376? If no-one's interested, we should stop right now. >> Do any of the people from the summit read this list? If not, can >> anyone contact them and get some feedback? > > Package management people aren't likely to be doing anything with this stuff > directly; if they need a list of installed files they're probably using > --root or --record at package build time, not needing to sniff stuff and > installation time, and they already have installation facilities. ?Their > only real need in relation to this will be to set an option to generate an > INSTALLER file that says you can't uninstall their packages manually, > thereby preventing the package from being uninstalled by the distutils > utility, or any other well-behaved PEP 376-based package management tool. So if I follow this, there's a use for this in making sure the new distutils uninstall option doesn't break something it shouldn't, but as the uninstall uses RECORD, it won't be (accidentally) usable on distributions that didn't follow PEP 376 and create a RECORD file. There's also a use for this in terms of setuptools/easy_install interoperability. I'm now going to take of my "PEP supporter" hat, and put on my "End user" hat. Please understand that the following is *not* unbiased, whereas the rest of what I've said *is* intended to be reasonably impartial. If the only driver for this PEP is setuptools, then I'm -1 on it. Unless someone working on a packaging tool *other* than setuptools (or setuptools-derived projects) speaks up and says "I have code of my own which uses distutils, and I would benefit as follows from PEP 376 and I will be changing my code to conform to PEP 376" I think the PEP should be rejected, and the energy put into some other PEP which will benefit the wider (non-setuptools) community. OK, end of the opinion bit. I'll go back to trying to keep things impartial in future mails. Paul. PS Note - I like the idea of the PEP in theory. It's the practical benefits that I'm trying to establish. From eric at trueblade.com Wed Jul 8 19:58:30 2009 From: eric at trueblade.com (Eric Smith) Date: Wed, 08 Jul 2009 13:58:30 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <20090708150844.EC3843A407B@sparrow.telecommunity.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> <20090708050608.AC2C63A407B@sparrow.telecommunity.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> <4A5477D6.1080708@gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <20090708150844.EC3843A407B@sparrow.telecommunity.com> Message-ID: <4A54DE46.4030404@trueblade.com> P.J. Eby wrote: > At 08:59 AM 7/8/2009 -0400, Eric Smith wrote: >> I agree with this. For RPM's, there's a whole other database of what >> files were installed. Is it really the intent that a file will be >> managed by multiple different installers? That I can install with RPM >> but remove with some python-installer (or other) tool? That way lies >> madness. In fact, I see RECORD as an installer-specific detail that >> doesn't need to be standardized at all. > > This is a misunderstanding. The purpose is to let an *installer* (like > easy_install) know that these files belong to some other installer, and > not allow overwriting them. That's why there's also an INSTALLER file. I really don't see this use case. Supporting multiple installers for the same file (or even just trying to prevent two installers from writing the same file)? Wouldn't you be better off just saying an installer can't overwrite any existing file? Likewise, I don't see a use case for installing with one installer and uninstalling with another. Put those two together, and the mechanism that an installer uses to record what files it installed is a private implementation detail. Eric. From Scott.Daniels at Acm.Org Wed Jul 8 20:10:52 2009 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Wed, 08 Jul 2009 11:10:52 -0700 Subject: [Python-Dev] Current patch process question Message-ID: I am confused by the current hydra-like development. I've recently submitted a change (issue6422 -- a feature request) which is currently moving through the process. I am not particularly asking for special attention on that issue, but I am confused about the process I should be using (in the face of two trunks). There seem to be two places to plausibly introduce a change: py3k and trunk. Should I submit changes for trunk, py3k, or both? I tried asking in the initial feature request, but nobody replied, and I finally decided that I'd go for patching trunk in the interim. For myself, I'd like to know if I should add a patch file for the py3k branch, or will that simply make more work for someone else? Certainly it will be easier for me to do this sooner rather than later. For the wider group, how should I have known the answer to these questions? Is there some page I overlooked? Is there a better forum in which to ask this question? --Scott David Daniels Scott.Daniels at Acm.Org From solipsis at pitrou.net Wed Jul 8 20:12:41 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 8 Jul 2009 18:12:41 +0000 (UTC) Subject: [Python-Dev] Current patch process question References: Message-ID: Hi, > There seem to be two places to plausibly introduce a change: py3k and > trunk. Should I submit changes for trunk, py3k, or both? I tried > asking in the initial feature request, but nobody replied, and I finally > decided that I'd go for patching trunk in the interim. Patches are usually first checked into trunk, and then svnmerge'd to py3k. So it makes more sense to submit your patch against trunk. > For myself, I'd like to know if I should add a patch file for the py3k > branch, or will that simply make more work for someone else? If porting the patch to py3k is trivial, you shouldn't care. svnmerge will take care of it. If, on the other hand, porting your patch requires many changes, or delicate ones, providing a patch against py3k can make the committer's life easier. Regards Antoine. From pje at telecommunity.com Wed Jul 8 20:33:02 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 08 Jul 2009 14:33:02 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.co m> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> Message-ID: <20090708183001.0A8A53A407B@sparrow.telecommunity.com> At 05:22 PM 7/8/2009 +0100, Paul Moore wrote: >2009/7/8 P.J. Eby : > > At 03:57 PM 7/8/2009 +0100, Paul Moore wrote: > >> > >> Who's going to use the APIs based around the RECORD file? Anyone? > > > > The distutils uninstall facility, for starters. easy_install and pip also > > will, eventually. > >Is pip getting this via setuptools, or will it be independently >changing its code? (Who's the pip contact who could confirm this >directly?) It would likely be changing its code directly, with respect to supporting overwrite-avoidance and uninstall-before-upgrade scenarios. (Generation support would likely come from setuptools, dependent on who implemented first.) Ian Bicking is pip's author. >In case (1), I'd imagine the user is careful enough (given that he's >doing everything manually) to avoid installing something that's >already present. Obviously, you've never tried doing this yourself. ;-) My guess would actually be that in Unixy environments these days, people either use virtualenv or buildouts to manage such things. Buildout handles uninstall-before-update (for eggs, anyway, I'm not sure about other things) and virtualenv makes it easy to create throwaway Python installs. >So if I follow this, there's a use for this in making sure the new >distutils uninstall option doesn't break something it shouldn't, but >as the uninstall uses RECORD, it won't be (accidentally) usable on >distributions that didn't follow PEP 376 and create a RECORD file. That's an interesting point. But for non-distutils tools, it'd be nice to be able to find out whether another package manager is administering some particular file. For that matter, the distutils themselves should (ultimately) support uninstall-before-upgrade and avoid overwriting another package manager's files. >If the only driver for this PEP is setuptools, then I'm -1 on it. If it were being driven by setuptools, I'd have just implemented it myself and presented it as a fait accompli. I can't speak to Tarek's motives, but I assume that, as stated in the PEP, the primary driver is supporting the distutils being able to uninstall things, and secondarily to allow other tools to be built on top of the API. >a packaging tool *other* than setuptools (or setuptools-derived projects) Is there really such a thing? ;-) AFAIK, every published tool for managing Python projects is either distutils-based or setuptools-based. (Things like scons and pymake and various other project build tools don't seem to fall under "packaging tool" in this sense.) From pje at telecommunity.com Wed Jul 8 20:42:32 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 08 Jul 2009 14:42:32 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A54DE46.4030404@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> <20090708050608.AC2C63A407B@sparrow.telecommunity.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> <4A5477D6.1080708@gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <20090708150844.EC3843A407B@sparrow.telecommunity.com> <4A54DE46.4030404@trueblade.com> Message-ID: <20090708183930.E138B3A407B@sparrow.telecommunity.com> At 01:58 PM 7/8/2009 -0400, Eric Smith wrote: >I really don't see this use case. Supporting multiple installers for >the same file (or even just trying to prevent two installers from >writing the same file)? Wouldn't you be better off just saying an >installer can't overwrite any existing file? > >Likewise, I don't see a use case for installing with one installer >and uninstalling with another. > >Put those two together, and the mechanism that an installer uses to >record what files it installed is a private implementation detail. You'll have to ask Tarek about that. As I've said in another email, if I were doing this for setuptools I'd just go and do it; it's already on my backlog of things to do. The point of the PEP is to provide an uninstall facility for the distutils, and to expose those records via an API so that other tools can be built on top of it. Whether anything other than distutils-based tools will actually use the format directly is entirely irrelevant to the PEP's merits, since bdist_* tools and platform-specific distutils installers may need to read or write the format. Hence, a PEP is appropriate to describe a format and API for the distutils family of tools to operate on. The specific proposed format is based on an existing de facto standard for installed package metadata, so that the many existing packaging tools based on that other format and existing APIs can easily read or write this data, and so that the API can also be used as a way to detect the presence of already-installed distributions of software -- even ones installed other ways. From martin at v.loewis.de Wed Jul 8 21:41:10 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 08 Jul 2009 21:41:10 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A5477D6.1080708@gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A530DFD.40802@egenix.com> <79990c6b0907070244p61a83145vb96b8d691ddc9c0b@mail.gmail.com> <87k52kpv5l.fsf@benfinney.id.au> <79990c6b0907070505h6fca1252wd4246ebf87b23442@mail.gmail.com> <94bdd2610907070623v6602fd15m30176a4db5ea7672@mail.gmail.com> <20090707202851.7DBB93A410E@sparrow.telecommunity.com> <94bdd2610907071430sfaf667as9b8eb5d77181293c@mail.gmail.com> <20090708050608.AC2C63A407B@sparrow.telecommunity.com> <94bdd2610907072326p74dc1a56qf980edde10b953e2@mail.gmail.com> <4A5477D6.1080708@gmail.com> Message-ID: <4A54F656.8040006@v.loewis.de> >> Hehe very nice, thanks for the explanation :) >> >> (thanks to the others too) > > I think is exchange actually a good reminder that the bar for acceptance > of PEP 376 shouldn't be "Addresses every possible packaging issue we can > come up with". That would be setting expectations far too high because > packaging cross-platform is so messy. OTOH, it *is* expected that the PEP explicitly lists aspects that it didn't resolve, so that any reader can find out whether something was discussed and rejected, or not discussed. Regards, Martin From martin at v.loewis.de Wed Jul 8 21:44:26 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 08 Jul 2009 21:44:26 +0200 Subject: [Python-Dev] Current patch process question In-Reply-To: References: Message-ID: <4A54F71A.90906@v.loewis.de> >> There seem to be two places to plausibly introduce a change: py3k and >> trunk. Should I submit changes for trunk, py3k, or both? I tried >> asking in the initial feature request, but nobody replied, and I finally >> decided that I'd go for patching trunk in the interim. > > Patches are usually first checked into trunk, and then svnmerge'd to py3k. > So it makes more sense to submit your patch against trunk. > >> For myself, I'd like to know if I should add a patch file for the py3k >> branch, or will that simply make more work for someone else? > > If porting the patch to py3k is trivial, you shouldn't care. svnmerge will take > care of it. > If, on the other hand, porting your patch requires many changes, or delicate > ones, providing a patch against py3k can make the committer's life easier. I usually abstain from "me too" messages, but only this time: as Antoine says. Regards, Martin From Scott.Daniels at Acm.Org Wed Jul 8 21:57:14 2009 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Wed, 08 Jul 2009 12:57:14 -0700 Subject: [Python-Dev] Current patch process question In-Reply-To: <4A54F71A.90906@v.loewis.de> References: <4A54F71A.90906@v.loewis.de> Message-ID: Martin v. L?wis wrote: >>> There seem to be two places to plausibly introduce a change: py3k and >>> trunk. Should I submit changes for trunk, py3k, or both? .... >> Patches are usually first checked into trunk, and then svnmerge'd to py3k. >> So it makes more sense to submit your patch against trunk. >> >>> For myself, I'd like to know if I should add a patch file for the py3k >>> branch, or will that simply make more work for someone else? >> If porting the patch to py3k is trivial, you shouldn't care. svnmerge will take >> care of it. >> If, on the other hand, porting your patch requires many changes, or delicate >> ones, providing a patch against py3k can make the committer's life easier. > > I usually abstain from "me too" messages, but only this time: > as Antoine says. Thanks to both of you for the quick response. In this case, the port should be dirt simple, so I'll consider myself done until comments come back on the patch. --Scott David Daniels Scott.Daniels at Acm.Org From p.f.moore at gmail.com Wed Jul 8 23:56:15 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Jul 2009 22:56:15 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <20090708183001.0A8A53A407B@sparrow.telecommunity.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> Message-ID: <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> 2009/7/8 P.J. Eby : > If it were being driven by setuptools, I'd have just implemented it myself > and presented it as a fait accompli. ?I can't speak to Tarek's motives, but > I assume that, as stated in the PEP, the primary driver is supporting the > distutils being able to uninstall things, and secondarily to allow other > tools to be built on top of the API. My understanding is that all of the various distutils PEPs were driven by the "packaging summit" ay PyCon. The struggle here seems to be to find *anyone* from that summit who will now comment on the discussion :-( >> a packaging tool *other* than setuptools (or setuptools-derived projects) > > Is there really such a thing? ?;-) I wonder... :-) > AFAIK, every published tool for managing Python projects is either > distutils-based or setuptools-based. ?(Things like scons and pymake and > various other project build tools don't seem to fall under "packaging tool" > in this sense.) The big driver seems to be things that take distutils packages and build Linux packages (debs, rpms, ...) from them. Unfortunately, I know nothing about these tools, and no-one seems to be speaking up on the subject. Maybe this was all thrashed out on the distutils SIG, and consensus reached there (I only skim that list). If so, maybe this list is the wrong one to be discussing changes to the PEP on. Paul. From eric at trueblade.com Thu Jul 9 00:07:21 2009 From: eric at trueblade.com (Eric Smith) Date: Wed, 08 Jul 2009 18:07:21 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> Message-ID: <4A551899.1080105@trueblade.com> Paul Moore wrote: > 2009/7/8 P.J. Eby : >> If it were being driven by setuptools, I'd have just implemented it myself >> and presented it as a fait accompli. I can't speak to Tarek's motives, but >> I assume that, as stated in the PEP, the primary driver is supporting the >> distutils being able to uninstall things, and secondarily to allow other >> tools to be built on top of the API. > > My understanding is that all of the various distutils PEPs were driven > by the "packaging summit" ay PyCon. The struggle here seems to be to > find *anyone* from that summit who will now comment on the discussion > :-( I was there, and I've been commenting! There might have been more discussion after the language summit and the one open space event I went to. But the focus as I recall was static metadata and version specification. When I originally brought up static metadata at the summit, I meant metadata describing the sources in the distribution, so that we can get rid of setup.py's. From that metadata, I want to be able to generate .debs, .rpms, .eggs, etc. But I think we've veered into metadata that describes what has been installed. I don't think that's so useful. As I've said, this is private to the installers. If 2 installers want to communicate with each other about what they've installed, then they can agree on that data. I just don't find it generally useful for all installers, and therefore not useful for distutils. I'd like to get back to the metadata that describes the source files. That's where the real value lies, in my opinion. I'll try and work on a post to distutils-sig explaining my thinking. From p.f.moore at gmail.com Thu Jul 9 01:07:29 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 9 Jul 2009 00:07:29 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A551899.1080105@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> Message-ID: <79990c6b0907081607s10cfd89dy43cd11b78881f5b9@mail.gmail.com> 2009/7/8 Eric Smith : > I was there, and I've been commenting! Sorry, I hadn't realised that. Thanks for the correction. > There might have been more discussion after the language summit and the one > open space event I went to. But the focus as I recall was static metadata > and version specification. When I originally brought up static metadata at > the summit, I meant metadata describing the sources in the distribution, so > that we can get rid of setup.py's. From that metadata, I want to be able to > generate .debs, .rpms, .eggs, etc. > > But I think we've veered into metadata that describes what has been > installed. I don't think that's so useful. As I've said, this is private to > the installers. If 2 installers want to communicate with each other about > what they've installed, then they can agree on that data. I just don't find > it generally useful for all installers, and therefore not useful for > distutils. > > I'd like to get back to the metadata that describes the source files. That's > where the real value lies, in my opinion. I'll try and work on a post to > distutils-sig explaining my thinking. OK, that helps a lot. I see how your postings fit into things a little better now. Paul. From cournape at gmail.com Thu Jul 9 01:42:01 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 9 Jul 2009 08:42:01 +0900 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A551899.1080105@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> Message-ID: <5b8d13220907081642i4c8f4af2u8a59cfb332379cf6@mail.gmail.com> On Thu, Jul 9, 2009 at 7:07 AM, Eric Smith wrote: > Paul Moore wrote: >> >> 2009/7/8 P.J. Eby : >>> >>> If it were being driven by setuptools, I'd have just implemented it >>> myself >>> and presented it as a fait accompli. ?I can't speak to Tarek's motives, >>> but >>> I assume that, as stated in the PEP, the primary driver is supporting the >>> distutils being able to uninstall things, and secondarily to allow other >>> tools to be built on top of the API. >> >> My understanding is that all of the various distutils PEPs were driven >> by the "packaging summit" ay PyCon. The struggle here seems to be to >> find *anyone* from that summit who will now comment on the discussion >> :-( > > I was there, and I've been commenting! > > There might have been more discussion after the language summit and the one > open space event I went to. But the focus as I recall was static metadata > and version specification. When I originally brought up static metadata at > the summit, I meant metadata describing the sources in the distribution, so > that we can get rid of setup.py's. From that metadata, I want to be able to > generate .debs, .rpms, .eggs, etc. I agree wholeheartedly. Getting rid of setup.py for most packages should be a goal IMHO. Most packages don't need anything fancy, and static metadata are so much easier to use compared to setup.py/distutils for 3rd party interop. There was a discussion about how to describe/find the list of files to form a distribution (for the different sdist/bdist_* commands), but no agreement was reached. Some people strongly defend the setuptools feature to get the list of files from the source control system, in particular. http://mail.python.org/pipermail/distutils-sig/2009-April/011226.html David From stephen at xemacs.org Thu Jul 9 02:38:46 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 09 Jul 2009 09:38:46 +0900 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A551899.1080105@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> Message-ID: <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> Eric Smith writes: > But I think we've veered into metadata that describes what has been > installed. I don't think that's so useful. As I've said, this is private > to the installers. If 2 installers want to communicate with each other > about what they've installed, then they can agree on that data. I just > don't find it generally useful for all installers, and therefore not > useful for distutils. ISTM that the problem that it solves is uninstall in the absence of the original installer. Am I to understand that you don't think that use case is important? Or that there's another way to do this? From steve at pearwood.info Thu Jul 9 02:51:02 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 9 Jul 2009 10:51:02 +1000 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A551899.1080105@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> Message-ID: <200907091051.02623.steve@pearwood.info> On Thu, 9 Jul 2009 08:07:21 am Eric Smith wrote: > But I think we've veered into metadata that describes what has been > installed. I don't think that's so useful. As I've said, this is > private to the installers. If 2 installers want to communicate with > each other about what they've installed, then they can agree on that > data. I just don't find it generally useful for all installers, and > therefore not useful for distutils. But doesn't this metadata give any two installers a common language to use to communicate, instead of having every pair of installers create their own private communication method? Personally, I like to be able to look at a package and say "What did that install?" Or contrary-wise, look at a file and say "What package installed that?" There are few things worse than discovering a bunch of mysterious executable files on your system that you don't remember installing, and then spending a few paranoid hours trying to determine whether you've stumbled across a root kit or virus or whether they have a legitimate reason to be there. -- Steven D'Aprano From eric at trueblade.com Thu Jul 9 03:03:38 2009 From: eric at trueblade.com (Eric Smith) Date: Wed, 08 Jul 2009 21:03:38 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4A5541EA.3020503@trueblade.com> Stephen J. Turnbull wrote: > Eric Smith writes: > > > But I think we've veered into metadata that describes what has been > > installed. I don't think that's so useful. As I've said, this is private > > to the installers. If 2 installers want to communicate with each other > > about what they've installed, then they can agree on that data. I just > > don't find it generally useful for all installers, and therefore not > > useful for distutils. > > ISTM that the problem that it solves is uninstall in the absence of > the original installer. Am I to understand that you don't think that > use case is important? Or that there's another way to do this? Yes, I think that's a use case that doesn't need to be supported by this PEP. It's an issue for a single installer, or a group of installers. It's not something that _all_ installers need to care about. From pje at telecommunity.com Thu Jul 9 03:07:48 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 08 Jul 2009 21:07:48 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20090709010447.2A86D3A407B@sparrow.telecommunity.com> At 09:38 AM 7/9/2009 +0900, Stephen J. Turnbull wrote: >Eric Smith writes: > > > But I think we've veered into metadata that describes what has been > > installed. I don't think that's so useful. As I've said, this is private > > to the installers. If 2 installers want to communicate with each other > > about what they've installed, then they can agree on that data. I just > > don't find it generally useful for all installers, and therefore not > > useful for distutils. > >ISTM that the problem that it solves is uninstall in the absence of >the original installer. Or uninstall where the installer is "setup.py install", actually. > Am I to understand that you don't think that >use case is important? Or that there's another way to do this? From SridharR at activestate.com Thu Jul 9 03:08:31 2009 From: SridharR at activestate.com (Sridhar Ratnakumar) Date: Wed, 08 Jul 2009 18:08:31 -0700 Subject: [Python-Dev] PEP 376 - get_egginfo_files In-Reply-To: <79990c6b0907051146o7c62ac97kffdafa1ab95040f8@mail.gmail.com> References: <79990c6b0907050926w64436c2ct33b218f6d94c061b@mail.gmail.com> <20090705182953.90F7D3A4099@sparrow.telecommunity.com> <79990c6b0907051146o7c62ac97kffdafa1ab95040f8@mail.gmail.com> Message-ID: On Sun, 05 Jul 2009 11:46:58 -0700, Paul Moore wrote: > 2009/7/5 P.J. Eby : >> At 05:26 PM 7/5/2009 +0100, Paul Moore wrote: >>> >>> def get_distribution(name): >>> for d in get_distributions(): >>> if d.name == name: >>> return d >>> return None >> >> Btw, this is broken code anyway, because it's not handling >> case-insensitivity or name canonicalization. (I've mentioned these >> issue >> previously on the distutils-sig.) > Fair point. (Although I don't recall your distutils-sig posting, so > I'm not sure what you mean by "name canonicalisation"). Correct me if I'm wrong - I believe he was referring to the `safe_name` function: http://mail.python.org/pipermail/distutils-sig/2005-September/005120.html -srid From SridharR at activestate.com Thu Jul 9 03:42:43 2009 From: SridharR at activestate.com (Sridhar Ratnakumar) Date: Wed, 08 Jul 2009 18:42:43 -0700 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> Message-ID: On Wed, 08 Jul 2009 09:22:52 -0700, Paul Moore wrote: > If the only driver for this PEP is setuptools, then I'm -1 on it. > Unless someone working on a packaging tool *other* than setuptools (or > setuptools-derived projects) speaks up and says "I have code of my own > which uses distutils, and I would benefit as follows from PEP 376 and > I will be changing my code to conform to PEP 376" I think the PEP > should be rejected, and the energy put into some other PEP which will > benefit the wider (non-setuptools) community. Other than easy_install/pip, there is also PyPM which is being developed at ActiveState. PyPM is the Python package manager much like what ppm is for ActivePerl. We build packages using 'setup.py install --root', but when it comes installation on client .. there is a separate database of installed packages that has the following information for each package installed: - metadata (name, version, author, etc..) - installed_files (list of files installed on client) So "pypm remove Foo" will simply remove all the 'installed_files' of Foo and remove Foo from the package database. The package database is currently an sqlite-based one. PEP 376 definitely will benefit PyPM besides benefiting setuptools/pip. If all package managers rely on common API to 1) retrieve metadata of installed packages 2) write metadata into site-packages), then it is very possible that one package manager does not mess up with the installation of others. For example: $ pip install IPython $ pypm install ipython Without PEP 376, pypm will overwrite pip's installation. But with PEP 376, as pip writes the RECORD file, pypm will reuse it (via the API) and co-operate with it. Eventually, I'd like PEP 376 to support system packagers too. So for example, if you did "apt-get install python-pyqt4", then running "pip install python-pyqt4" should return without installing anything .. as RECORD will be part of the .deb previously installed. As for generating the RECORD file, I vote for generating it during install time (w/ absolute paths). -srid From SridharR at activestate.com Thu Jul 9 03:51:25 2009 From: SridharR at activestate.com (Sridhar Ratnakumar) Date: Wed, 08 Jul 2009 18:51:25 -0700 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> Message-ID: Is there any reason why RECORD file can't be generated at runtime? Also, why should the RECORD file be generated at all by bdist* commands? A .deb file contains "data.tar.gz" that is simply extracted over "/". There is no need for RECORD inside a .deb file because it is implicit in the contents on data.tar.gz. -srid On Tue, 07 Jul 2009 13:30:05 -0700, P.J. Eby wrote: > Just an idea... suppose that instead of using "real" absolute paths in > the RECORD file for non-local files (scripts, data, etc) we changed the > format to include a "prefix" field, containing something like LIBDIR, > SHARE, SCRIPTS, etc., ala bdist_wininst internals? > > Also, we could include a separate (optional) PREFIXES file defining what > those locations translated to at install time. Dumb bdists would still > have their build paths there (or delete the file before packaging, or > use real paths instead of the --root prefixed versions). > > Upside: relocation can be detected and handled, RECORD remains > cross-platform, and bdists are ok. > > Downside: more complex API required to read/manipulate paths and delete > files, since you need to be able to check that you have the right > prefixes, and may have to ask the user for a prefix you don't recognize, > if the default in PREFIXES doesn't match. > > Thoughts? > > (Also, as a special case, any file that's actually installed to LIBDIR > or a subdirectory thereof (even if it's technically a "data" file or > script), will probably need to be designated under LIBDIR prefix to > prevent runtime breakage in the event any system package maintainers are > tempted to use RECORD files as a way of forcing a HFS conformance. > These are installation prefixes, *not* content types.) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/sridharr%40activestate.com From pje at telecommunity.com Thu Jul 9 04:12:39 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 08 Jul 2009 22:12:39 -0400 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> Message-ID: <20090709020938.5C0A83A407B@sparrow.telecommunity.com> At 06:51 PM 7/8/2009 -0700, Sridhar Ratnakumar wrote: >Is there any reason why RECORD file can't be generated at runtime? Applications can be relocatable, yet require plugins and libraries to be upgraded, installed, uninstalled, etc. (Also, RECORD is cross-platform to allow cross-platform installation/testing of cross-platform packages; that's why I propose keeping the prefix information separate.) > Also, >why should the RECORD file be generated at all by bdist* commands? bdist commands that use "install --root" will get it as a side effect, so they don't have to do anything special. They needn't even generate the PREFIXES, if PREFIXES is also generated by "install --root", and omits the root from the prefixes written. In other words, most bdist tools wouldn't change. bdist_wininst and bdist_msi might need to regenerate PREFIXES at actual installation time, but AFAIK those are the only ones. (And the spec would allow for the possibility that a PREFIXES file can be incorrect.) From eric at trueblade.com Thu Jul 9 04:55:14 2009 From: eric at trueblade.com (Eric Smith) Date: Wed, 08 Jul 2009 22:55:14 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <200907091051.02623.steve@pearwood.info> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <200907091051.02623.steve@pearwood.info> Message-ID: <4A555C12.7000205@trueblade.com> Steven D'Aprano wrote: > On Thu, 9 Jul 2009 08:07:21 am Eric Smith wrote: >> But I think we've veered into metadata that describes what has been >> installed. I don't think that's so useful. As I've said, this is >> private to the installers. If 2 installers want to communicate with >> each other about what they've installed, then they can agree on that >> data. I just don't find it generally useful for all installers, and >> therefore not useful for distutils. > > But doesn't this metadata give any two installers a common language to > use to communicate, instead of having every pair of installers create > their own private communication method? I really don't get this use case of multiple installers trying to install the same package. There's just no way that running "yum install twisted" and "apt-get install twisted" and "pip install twisted" are going to coexist with each other. The best they can do is say "a file I'm trying to install already exists". Why try for anything else? > Personally, I like to be able to look at a package and say "What did > that install?" Or contrary-wise, look at a file and say "What package > installed that?" I completely support that. All of the system installers I know of support this (at least on Linux; I'm familiar with the msi database layout (as exposed by Orca) on Windows, but I've never looked to see if they record the results of installing an msi). I just see this as an issue within each installer. Why would the installers need to communicate this with each other? > There are few things worse than discovering a bunch of > mysterious executable files on your system that you don't remember > installing, and then spending a few paranoid hours trying to determine > whether you've stumbled across a root kit or virus or whether they have > a legitimate reason to be there. We can't solve the problem of "here's an arbitrary file on the system, where did it come from?". There are many, many installers outside of our control that might have installed something. Ruby has installers, Perl has installers, I'm sure there are hundreds of such installers. The user might have even used wget to install something. From eric at trueblade.com Thu Jul 9 05:20:59 2009 From: eric at trueblade.com (Eric Smith) Date: Wed, 08 Jul 2009 23:20:59 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <20090709010447.2A86D3A407B@sparrow.telecommunity.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> <20090709010447.2A86D3A407B@sparrow.telecommunity.com> Message-ID: <4A55621B.5070904@trueblade.com> P.J. Eby wrote: >> ISTM that the problem that it solves is uninstall in the absence of >> the original installer. > > Or uninstall where the installer is "setup.py install", actually. I think we need to move away from "setup.py install". It's the antithesis of static metadata. setup.py needs to go away. If we have a static metadata file called, say "setup.info", then I'd like to see: "bdist_rpm setup.info" produce a .rpm "bdist_msi setup.info" produce a .msi "bdist_deb setup.info" produce a .deb "bdist_egg setup.info" produce a .egg and so on. There might be a library (and I call dibs on the name "distlib" :) that provides support routines to parse setup.info, but there's no framework involved. And no need for a plugin system. There might be support routines used to build setup.info. For those who think it's reasonable that the list of source files comes from your version control system, build a small command that produces setup.info from your vcs. If you need the version number to come from some other source, write a small command that reads that and sticks it into a setup.info. But the source distribution would be defined by a (possibly generated) setup.info. The metadata in setup.info would be extensible. If setup.info contains information that's needed to build a .rpm that's not needed for a .deb, then that's fine. We can define the minimum information needed and an extension mechanism. As time goes by, some of the extensions might become standardized. Then for each binary format, there's an installer (or more than one!), and it might handle dependency detection and automatic fetching, or it might not. Some of these exist (apt-get, yum, msi), some might need some tweaking (setuptools, pip) and others might need to be written from scratch. But the installer, its database of what it has installed, and how it handles uninstalls (if it does) is really not the issue. It's the conflation of "build a package" with "install a package" that distutils and setuptools has given us (via setup.py) that leads to us never making progress. Let's just focus on "describe a package" and let the packagers handle "build a package" and let the installers handle "install and uninstall a package". Then the packagers (and their associated installers) can evolve independently of each other. From eric at trueblade.com Thu Jul 9 05:28:53 2009 From: eric at trueblade.com (Eric Smith) Date: Wed, 08 Jul 2009 23:28:53 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> Message-ID: <4A5563F5.3020707@trueblade.com> > Eventually, I'd like PEP 376 to support system packagers too. So for > example, if you did "apt-get install python-pyqt4", then running "pip > install python-pyqt4" should return without installing anything .. as > RECORD will be part of the .deb previously installed. As for generating > the RECORD file, I vote for generating it during install time (w/ > absolute paths). I think we should explicitly not support this. What if pip and apt-get (or rpm, or others) install the same package in different places because of system conventions (/usr vs. /usr/local vs. /opt, say)? There's no way we're ever going to get this right, and it's not worth complicating our lives over it. Seriously: Is there some real world use case I'm missing? Does any existing install system support this? From pje at telecommunity.com Thu Jul 9 05:38:53 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 08 Jul 2009 23:38:53 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A55621B.5070904@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> <20090709010447.2A86D3A407B@sparrow.telecommunity.com> <4A55621B.5070904@trueblade.com> Message-ID: <20090709033551.AA35D3A407B@sparrow.telecommunity.com> At 11:20 PM 7/8/2009 -0400, Eric Smith wrote: >P.J. Eby wrote: >>>ISTM that the problem that it solves is uninstall in the absence of >>>the original installer. >>Or uninstall where the installer is "setup.py install", actually. > >I think we need to move away from "setup.py install". It's the >antithesis of static metadata. Please note that that's entirely out of scope for the PEP at hand. That being said, the rest of your proposal is strikingly similar to a proposal I previously floated on the distutils-sig for a concept called BUILDS. The main difference is that I suggested that the spec should include a standard interface for running build operations that would produce the manifest (equivalent to your setup.info), and that distutils and setuptools should provide setup.py commands to generate said manifest, to allow for a seamless transition. There was very little comment on the proposal, perhaps because it involves a lot of work that most people are sane enough not to sign up for. ;-) (Or more to the point, it's the sort of thing that never gets off the ground for design by consensus. The best way to make something like your proposal to happen is to go off and build it, and get it to take over. If it can't win substantial market share on its own merits, it probably doesn't deserve to be blessed as a standard.) From pje at telecommunity.com Thu Jul 9 05:40:53 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 08 Jul 2009 23:40:53 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A5563F5.3020707@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <4A5563F5.3020707@trueblade.com> Message-ID: <20090709033751.638213A407B@sparrow.telecommunity.com> At 11:28 PM 7/8/2009 -0400, Eric Smith wrote: >>Eventually, I'd like PEP 376 to support system packagers too. So >>for example, if you did "apt-get install python-pyqt4", then >>running "pip install python-pyqt4" should return without installing >>anything .. as RECORD will be part of the .deb previously >>installed. As for generating the RECORD file, I vote for generating >>it during install time (w/ absolute paths). > >I think we should explicitly not support this. What if pip and >apt-get (or rpm, or others) install the same package in different >places because of system conventions (/usr vs. /usr/local vs. /opt, >say)? There's no way we're ever going to get this right, and it's >not worth complicating our lives over it. > >Seriously: Is there some real world use case I'm missing? Does any >existing install system support this? The use case described above is supported right now by easy_install and pip; it doesn't require a RECORD file, though. An .egg-info file or an .egg-info directory with a PKG-INFO is enough to prevent duplicate installation. I don't know how pip handles file overwrites; easy_install OTOH never overwrites anything that's not in an .egg or a .pth. From glyph at divmod.com Thu Jul 9 05:46:41 2009 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 09 Jul 2009 03:46:41 -0000 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A555C12.7000205@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <200907091051.02623.steve@pearwood.info> <4A555C12.7000205@trueblade.com> Message-ID: <20090709034641.27019.110061093.divmod.xquotient.141@weber.divmod.com> On 02:55 am, eric at trueblade.com wrote: >I really don't get this use case of multiple installers trying to >install the same package. There's just no way that running "yum install >twisted" and "apt-get install twisted" and "pip install twisted" are >going to coexist with each other. The best they can do is say "a file >I'm trying to install already exists". Why try for anything else? Here are some use-cases. You do "apt-get install twisted", or, rather, Twisted comes preinstalled on your OS. Later, you do "pip install py_awesome_network_thing_system", which depends on twisted. You've configured pip to install files into ~/.local, though, and it's not sure if Twisted is installed or not. So thanks to automatic dependency resolution, it downloads and installs another copy. But that's not what you want: you want it to use the system-installed Twisted (and zope.interface, and pycrypto, and pytz, etc etc) and just install PANTS into your .local environment. No files will conflict, but it would be helpful for the package installation tool to have an API to inspect both the system package installation and any other places where packages have been installed in order to make a decision about how it needs to satisfy its dependencies. If you're not as interested as I am in per-user installation of packages, then consider the fact that Ubuntu stores Python packages in /usr/lib/pythonX.Y/dist-packages now. Python package installers should be able to query that before putting things into site-packages. (which I *believe* is now reserved for non-OS-installed packages, but it's very hard to find documentation on the motivation behind debian's manipulations of distutils. In any event, I digress.) You're correct that worrying about 'yum install twisted' and 'apt-get install twisted' is way out of scope for any Python installation system; unfortunately, yum and apt are likely as not to have giant, opaque makefiles or shell scripts attached to these packages for obscure linux-y reasons that (hopefully) we don't need to care about, but would fight with each other if you tried to combine them, totally out of scope with distutils. One hopes that regardless of whatever else they're doing, these packages *are* providing correct installation metadata. (Unfortunately, I can bet you a nickel they will screw it up in the first few python packages that include it; see, for example, . This spec should try to gracefully address what to do if system packagers have done something unusual or wrong.) So, the filesystem and the OS package database and the Python package "database" (such as it is) are definitely three distinct things, but they have some overlap, and there should be APIs for querying each. If for no other reason so you know when Python packages shouldn't stomp on OS packages, and vice versa. From glyph at divmod.com Thu Jul 9 05:52:43 2009 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 09 Jul 2009 03:52:43 -0000 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A5563F5.3020707@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A547E20.5030606@gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <4A5563F5.3020707@trueblade.com> Message-ID: <20090709035243.27019.863612690.divmod.xquotient.150@weber.divmod.com> On 03:28 am, eric at trueblade.com wrote: >>Eventually, I'd like PEP 376 to support system packagers too. So for >>example, if you did "apt-get install python-pyqt4", then running "pip >>install python-pyqt4" should return without installing anything .. as >>RECORD will be part of the .deb previously installed. As for >>generating the RECORD file, I vote for generating it during install >>time (w/ absolute paths). > >I think we should explicitly not support this. What if pip and apt-get >(or rpm, or others) install the same package in different places >because of system conventions (/usr vs. /usr/local vs. /opt, say)? >There's no way we're ever going to get this right, and it's not worth >complicating our lives over it. As I understand it, the point of the PEP is just to *provide the information* that the package installers need to get it right, not to actually implement the interaction with system package managers. >Seriously: Is there some real world use case I'm missing? Does any >existing install system support this? System package managers already provide .egg-info metadata, so this is hardly unprecedented. The antecedent of your "this" is somewhat ambiguous, so I'm not sure if any package manager provides the exact feature that you have in mind. If you're talking about different package managers installing stuff on the same system, you can look a bit farther afield: 'alien' allows you to install RPM and debian packages on the same system, with some reconciliation of dependencies and other metadata. From p.f.moore at gmail.com Thu Jul 9 09:18:18 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 9 Jul 2009 08:18:18 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A55621B.5070904@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> <20090709010447.2A86D3A407B@sparrow.telecommunity.com> <4A55621B.5070904@trueblade.com> Message-ID: <79990c6b0907090018v65359b0fjc7c055bc7ec1a04b@mail.gmail.com> 2009/7/9 Eric Smith : > P.J. Eby wrote: >>> >>> ISTM that the problem that it solves is uninstall in the absence of >>> the original installer. >> >> Or uninstall where the installer is "setup.py install", actually. > > I think we need to move away from "setup.py install". It's the antithesis of > static metadata. setup.py needs to go away. > > If we have a static metadata file called, say "setup.info", then I'd like to > see: > "bdist_rpm setup.info" produce a .rpm > "bdist_msi setup.info" produce a .msi > "bdist_deb setup.info" produce a .deb > "bdist_egg setup.info" produce a .egg > and so on. > > There might be a library (and I call dibs on the name "distlib" :) that > provides support routines to parse setup.info, but there's no framework > involved. And no need for a plugin system. +1. Now who's going to design & write it? > It's the conflation of "build a package" with "install a package" that > distutils and setuptools has given us (via setup.py) that leads to us never > making progress. Let's just focus on "describe a package" and let the > packagers handle "build a package" and let the installers handle "install > and uninstall a package". Then the packagers (and their associated > installers) can evolve independently of each other. There still needs to be some sort of package build system. I do *not* want to see a situation where users can't easily build their own packages. Not all systems have all Python packages packaged up by "official" system packagers... Paul From ziade.tarek at gmail.com Thu Jul 9 09:53:25 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 9 Jul 2009 09:53:25 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A551899.1080105@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> Message-ID: <94bdd2610907090053sf80135gd87b8e60453ccb5f@mail.gmail.com> On Thu, Jul 9, 2009 at 12:07 AM, Eric Smith wrote: > Paul Moore wrote: >> >> 2009/7/8 P.J. Eby : >>> >>> If it were being driven by setuptools, I'd have just implemented it >>> myself >>> and presented it as a fait accompli. ?I can't speak to Tarek's motives, >>> but >>> I assume that, as stated in the PEP, the primary driver is supporting the >>> distutils being able to uninstall things, and secondarily to allow other >>> tools to be built on top of the API. >> >> My understanding is that all of the various distutils PEPs were driven >> by the "packaging summit" ay PyCon. The struggle here seems to be to >> find *anyone* from that summit who will now comment on the discussion >> :-( > > I was there, and I've been commenting! Yes, so Ian Bicking was, and Jim Fulton, and we worked from there on Distutils-SIG. To answer to Paul's question on my motivations for PEP 376, Phillip gave the answers I was going to give. But as Eric says, that was one topic out of others. > > There might have been more discussion after the language summit and the one > open space event I went to. But the focus as I recall was static metadata > and version specification. When I originally brought up static metadata at > the summit, I meant metadata describing the sources in the distribution, so > that we can get rid of setup.py's. From that metadata, I want to be able to > generate .debs, .rpms, .eggs, etc. > > But I think we've veered into metadata that describes what has been > installed. I don't think that's so useful. As I've said, this is private to > the installers. If 2 installers want to communicate with each other about > what they've installed, then they can agree on that data. I just don't find > it generally useful for all installers, and therefore not useful for > distutils. > > I'd like to get back to the metadata that describes the source files. That's > where the real value lies, in my opinion. I'll try and work on a post to > distutils-sig explaining my thinking. Yes please do so, and have a look at Tres and Matthias notes from the summit http://wiki.python.org/moin/Distutils/StaticMetadata but that's another topic, that will deserve a PEP on its own Tarek From ziade.tarek at gmail.com Thu Jul 9 09:57:53 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 9 Jul 2009 09:57:53 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <20090709010447.2A86D3A407B@sparrow.telecommunity.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> <20090709010447.2A86D3A407B@sparrow.telecommunity.com> Message-ID: <94bdd2610907090057o3fb6dd1ch55df608de23435de@mail.gmail.com> On Thu, Jul 9, 2009 at 3:07 AM, P.J. Eby wrote: > At 09:38 AM 7/9/2009 +0900, Stephen J. Turnbull wrote: >> >> Eric Smith writes: >> >> ?> But I think we've veered into metadata that describes what has been >> ?> installed. I don't think that's so useful. As I've said, this is >> private >> ?> to the installers. If 2 installers want to communicate with each other >> ?> about what they've installed, then they can agree on that data. I just >> ?> don't find it generally useful for all installers, and therefore not >> ?> useful for distutils. >> >> ISTM that the problem that it solves is uninstall in the absence of >> the original installer. > > Or uninstall where the installer is "setup.py install", actually. > Right, and for instance, Fedora packagers are using it to re-package eggs http://fedoraproject.org/wiki/Packaging/Python/Eggs#Providing_Eggs_using_Setuptools with the --root option, so they will get a RECORD file and will be able to set a value for the INSTALLER one, From ziade.tarek at gmail.com Thu Jul 9 10:15:13 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 9 Jul 2009 10:15:13 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907090018v65359b0fjc7c055bc7ec1a04b@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> <20090709010447.2A86D3A407B@sparrow.telecommunity.com> <4A55621B.5070904@trueblade.com> <79990c6b0907090018v65359b0fjc7c055bc7ec1a04b@mail.gmail.com> Message-ID: <94bdd2610907090115h2d4340fajb1309e9c1a9abf8b@mail.gmail.com> On Thu, Jul 9, 2009 at 9:18 AM, Paul Moore wrote: > 2009/7/9 Eric Smith : >> P.J. Eby wrote: >>>> >>>> ISTM that the problem that it solves is uninstall in the absence of >>>> the original installer. >>> >>> Or uninstall where the installer is "setup.py install", actually. >> >> I think we need to move away from "setup.py install". It's the antithesis of >> static metadata. setup.py needs to go away. >> >> If we have a static metadata file called, say "setup.info", then I'd like to >> see: >> "bdist_rpm setup.info" produce a .rpm >> "bdist_msi setup.info" produce a .msi >> "bdist_deb setup.info" produce a .deb >> "bdist_egg setup.info" produce a .egg >> and so on. >> >> There might be a library (and I call dibs on the name "distlib" :) that >> provides support routines to parse setup.info, but there's no framework >> involved. And no need for a plugin system. > > +1. Now who's going to design & write it? Some work has been started on that at the summit too on that, under the "static metadata" name. (see the wiki) Can we move that discussion to distutils-SIG and focus on PEP 376 in this thread ? I am scared to see the discussion going in too many directions, Regards Tarek From ziade.tarek at gmail.com Thu Jul 9 10:22:19 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 9 Jul 2009 10:22:19 +0200 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> Message-ID: <94bdd2610907090122x12a3f25s13bba018240f331d@mail.gmail.com> On Thu, Jul 9, 2009 at 3:42 AM, Sridhar Ratnakumar wrote: > Other than easy_install/pip, there is also PyPM which is being developed at > ActiveState. PyPM is the Python package manager much like what ppm is for > ActivePerl. > Great ! besides the RECORD file, anything remark on the PEP from a PyPM point of view ? Once Paul has finished to work on the PEP 302 part of the prototype, maybe we could try it out on PyPM to see how it fits ? > Eventually, I'd like PEP 376 to support system packagers too. So for > example, if you did "apt-get install python-pyqt4", then running "pip > install python-pyqt4" should return without installing anything .. as RECORD > will be part of the .deb previously installed. As for generating the RECORD > file, I vote for generating it during install time (w/ absolute paths). That should be possible for all "python setup.py install"-based systems e.g. stating that if python-pyqt4 is already installed and if the INSTALLER file contains another name than "pip", pip should not try to do anything with it. From p.f.moore at gmail.com Thu Jul 9 10:32:52 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 9 Jul 2009 09:32:52 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <94bdd2610907090122x12a3f25s13bba018240f331d@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <94bdd2610907090122x12a3f25s13bba018240f331d@mail.gmail.com> Message-ID: <79990c6b0907090132m52db8771p463e00ec4d8db04a@mail.gmail.com> 2009/7/9 Tarek Ziad? : > On Thu, Jul 9, 2009 at 3:42 AM, Sridhar > Ratnakumar wrote: >> Other than easy_install/pip, there is also PyPM which is being developed at >> ActiveState. PyPM is the Python package manager much like what ppm is for >> ActivePerl. >> > > Great ! besides the RECORD file, anything remark on the PEP from a > PyPM point of view ? > Once Paul has finished to work on the PEP 302 part of the prototype, > maybe we could try it out > on PyPM to see how it fits ? Note: the "pkgutil2" implementation I pushed to Tarek's bitbucket repository a day or so ago includes PEP 302 support (using a few new finder methods, as proposed). It's ready to go now. It's as compliant with the PEP as possible given the outstanding questions (I've implemented some things on the basis of what I thought was consensus here, even though the PEP has not changed yet to reflect that - using "metadata" rather than "egginfo" in method names, and removing the local= parameter from some of the methods, being the 2 I can recall immediately). I'd like to add a test case using a non-standard PEP 302 importer, but that's a small detail. There are bound to be holes in the implementation, especially around the areas of the open questions that started this thread. I'd appreciate feedback, but I'm not expecting to make code changes until the PEP is changed - it's too much like trying to hit a moving target at the moment! Paul. PS I've added distutils-sig back into the cc list. Please keep "PEP 376" in the subject of any subsequent threads on that list if you want my input, as I don't follow all the traffic on there. From ncoghlan at gmail.com Thu Jul 9 10:37:47 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 09 Jul 2009 18:37:47 +1000 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A555C12.7000205@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <200907091051.02623.steve@pearwood.info> <4A555C12.7000205@trueblade.com> Message-ID: <4A55AC5B.5020405@gmail.com> Eric Smith wrote: > I really don't get this use case of multiple installers trying to > install the same package. There's just no way that running "yum install > twisted" and "apt-get install twisted" and "pip install twisted" are > going to coexist with each other. The best they can do is say "a file > I'm trying to install already exists". Why try for anything else? If that use case doesn't click for you, just focus on the distutils.uninstall use case. At the moment, there is no automatic method for uninstalling a distutils installed package. This PEP is mainly about fixing that situation. However, if we're going to go to all the effort of having an index of distutils-installed packages and the metadata needed to uninstall them, it makes sense to have an official API for querying that datastore. The fact that that API then provides support for "Did someone else install this already?" is really a sideshow. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Jul 9 10:50:12 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 09 Jul 2009 18:50:12 +1000 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: <20090709020938.5C0A83A407B@sparrow.telecommunity.com> References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> <20090709020938.5C0A83A407B@sparrow.telecommunity.com> Message-ID: <4A55AF44.4090508@gmail.com> P.J. Eby wrote: >> Also, >> why should the RECORD file be generated at all by bdist* commands? > > bdist commands that use "install --root" will get it as a side effect, > so they don't have to do anything special. They needn't even generate > the PREFIXES, if PREFIXES is also generated by "install --root", and > omits the root from the prefixes written. In other words, most bdist > tools wouldn't change. bdist_wininst and bdist_msi might need to > regenerate PREFIXES at actual installation time, but AFAIK those are the > only ones. (And the spec would allow for the possibility that a > PREFIXES file can be incorrect.) I haven't seen any suggestions for handling non-relative paths that are cleaner than this. The distutils based list that Tarek posted also gives a good "starter set" of supported entries in the PREFIXES file: PURELIB= PLATLIB= HEADERS= SCRIPTS= DATA= The question then would be whether relative paths should be handled directly in the RECORD file (as in the current PEP) or by writing a "." into PREFIXES to indicate relative paths. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From cournape at gmail.com Thu Jul 9 11:26:19 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 9 Jul 2009 18:26:19 +0900 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907090018v65359b0fjc7c055bc7ec1a04b@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> <20090709010447.2A86D3A407B@sparrow.telecommunity.com> <4A55621B.5070904@trueblade.com> <79990c6b0907090018v65359b0fjc7c055bc7ec1a04b@mail.gmail.com> Message-ID: <5b8d13220907090226r466f28d3y9af85848fe4c4f7a@mail.gmail.com> On Thu, Jul 9, 2009 at 4:18 PM, Paul Moore wrote: >> >> There might be a library (and I call dibs on the name "distlib" :) that >> provides support routines to parse setup.info, but there's no framework >> involved. And no need for a plugin system. > > +1. Now who's going to design & write it? I started a new thread on distutils-sig ("setup.py needs to go away") to avoid jeopardizing this thread. I added the context as well as my own suggestions for such a design. David From p.f.moore at gmail.com Thu Jul 9 15:45:22 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 9 Jul 2009 14:45:22 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907090132m52db8771p463e00ec4d8db04a@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <94bdd2610907090122x12a3f25s13bba018240f331d@mail.gmail.com> <79990c6b0907090132m52db8771p463e00ec4d8db04a@mail.gmail.com> Message-ID: <79990c6b0907090645q7468f0dj4b32e89f70306022@mail.gmail.com> 2009/7/9 Paul Moore : > I'd like to add a test case using a non-standard PEP 302 importer, but > that's a small detail. I've just pushed a PEP 302 importer test case. The level of boilerplate needed is a bit painful, but it's fine as a prototype. When we get consensus on some of the remaining details, I'll tidy up the implementation and streamline the importer requirements at that time. Paul. From eric at trueblade.com Thu Jul 9 15:53:56 2009 From: eric at trueblade.com (Eric Smith) Date: Thu, 09 Jul 2009 09:53:56 -0400 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <20090709033551.AA35D3A407B@sparrow.telecommunity.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> <20090709010447.2A86D3A407B@sparrow.telecommunity.com> <4A55621B.5070904@trueblade.com> <20090709033551.AA35D3A407B@sparrow.telecommunity.com> Message-ID: <4A55F674.6090200@trueblade.com> P.J. Eby wrote: > At 11:20 PM 7/8/2009 -0400, Eric Smith wrote: >> P.J. Eby wrote: >>>> ISTM that the problem that it solves is uninstall in the absence of >>>> the original installer. >>> Or uninstall where the installer is "setup.py install", actually. >> >> I think we need to move away from "setup.py install". It's the >> antithesis of static metadata. > > Please note that that's entirely out of scope for the PEP at hand. Yes. I'm just trying to point out that the information in the PEP is applicable only to the set of installers that want to work together in some integrated way. It doesn't apply to all installers, and I think that's the bigger problem. But I grant you that the rest of my mail is veering off-topic, and there's a more appropriate thread now on distutils-sig. Eric. From p.f.moore at gmail.com Thu Jul 9 16:22:45 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 9 Jul 2009 15:22:45 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A55F674.6090200@trueblade.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> <20090709010447.2A86D3A407B@sparrow.telecommunity.com> <4A55621B.5070904@trueblade.com> <20090709033551.AA35D3A407B@sparrow.telecommunity.com> <4A55F674.6090200@trueblade.com> Message-ID: <79990c6b0907090722o78973109s536c6734a870ae96@mail.gmail.com> 2009/7/9 Eric Smith : > Yes. I'm just trying to point out that the information in the PEP is > applicable only to the set of installers that want to work together in some > integrated way. It doesn't apply to all installers, and I think that's the > bigger problem. Interoperability standards are only as useful as the number of applications that conform to them. The set of installers that will *not* provide PEP 376 compatible metadata is an important measure of the usefulness of the PEP. Some metadata (the INSTALLER file and the proposed PREFIXES file) must be created by the installer for the PEP 376 APIs to work. Simple cases (relocatable packages with all package data maintained inside the package directory) will work fine even without installer co-operation, as long as the installer copies the whole structure unchanged. But: - some cases are not simple, and it's not clear to me how useful "nearly always accurate" data will be - I understand that some packagers deliberately pick apart Python distributions to conform to OS filesystem layout standards Paul. From tseaver at palladion.com Thu Jul 9 20:46:18 2009 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 09 Jul 2009 14:46:18 -0400 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: <4A55AF44.4090508@gmail.com> References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> <20090709020938.5C0A83A407B@sparrow.telecommunity.com> <4A55AF44.4090508@gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Nick Coghlan wrote: > P.J. Eby wrote: >>> Also, >>> why should the RECORD file be generated at all by bdist* commands? >> bdist commands that use "install --root" will get it as a side effect, >> so they don't have to do anything special. They needn't even generate >> the PREFIXES, if PREFIXES is also generated by "install --root", and >> omits the root from the prefixes written. In other words, most bdist >> tools wouldn't change. bdist_wininst and bdist_msi might need to >> regenerate PREFIXES at actual installation time, but AFAIK those are the >> only ones. (And the spec would allow for the possibility that a >> PREFIXES file can be incorrect.) > > I haven't seen any suggestions for handling non-relative paths that are > cleaner than this. The distutils based list that Tarek posted also gives > a good "starter set" of supported entries in the PREFIXES file: > > PURELIB= > PLATLIB= > HEADERS= > SCRIPTS= > DATA= > > The question then would be whether relative paths should be handled > directly in the RECORD file (as in the current PEP) or by writing a "." > into PREFIXES to indicate relative paths. Per discussion with Matthias Klose, it would be useful to split out the following prefixes for "special" kinds of data: - - docs - - i18n / locales Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKVjr6+gerLs4ltQ4RAqwTAKCoohHwkbON3Gg6gmJ4AmZUcPSRdQCgyeZi nBJVZQziPYnWN0cI3B2i4A0= =bSHL -----END PGP SIGNATURE----- From pje at telecommunity.com Thu Jul 9 21:09:33 2009 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 09 Jul 2009 15:09:33 -0400 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> <20090709020938.5C0A83A407B@sparrow.telecommunity.com> <4A55AF44.4090508@gmail.com> Message-ID: <20090709190631.CE40E3A407B@sparrow.telecommunity.com> At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote: >-----BEGIN PGP SIGNED MESSAGE----- >Hash: SHA1 > >Nick Coghlan wrote: > > P.J. Eby wrote: > >>> Also, > >>> why should the RECORD file be generated at all by bdist* commands? > >> bdist commands that use "install --root" will get it as a side effect, > >> so they don't have to do anything special. They needn't even generate > >> the PREFIXES, if PREFIXES is also generated by "install --root", and > >> omits the root from the prefixes written. In other words, most bdist > >> tools wouldn't change. bdist_wininst and bdist_msi might need to > >> regenerate PREFIXES at actual installation time, but AFAIK those are the > >> only ones. (And the spec would allow for the possibility that a > >> PREFIXES file can be incorrect.) > > > > I haven't seen any suggestions for handling non-relative paths that are > > cleaner than this. The distutils based list that Tarek posted also gives > > a good "starter set" of supported entries in the PREFIXES file: > > > > PURELIB= > > PLATLIB= > > HEADERS= > > SCRIPTS= > > DATA= > > > > The question then would be whether relative paths should be handled > > directly in the RECORD file (as in the current PEP) or by writing a "." > > into PREFIXES to indicate relative paths. > >Per discussion with Matthias Klose, it would be useful to split out the >following prefixes for "special" kinds of data: > >- - docs >- - i18n / locales Unfortunately, the distutils don't currently have a way to specify these. I think we'll need to use only the ones that are distutils install targets at first, but of course the format is extensible. OTOH, one supposed that at least the 'data_files' setup argument could be enhanced to allow a data type flag, that would then be used as a prefix name for those files, with a fallback to the DATA prefix if there isn't one specified. But that's something that will require tagging support in the install_data command. As it is, refactoring the distutils to track prefixes for the RECORD file is probably going to be... challenging. ;-) From ncoghlan at gmail.com Thu Jul 9 21:53:13 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Jul 2009 05:53:13 +1000 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> <20090709020938.5C0A83A407B@sparrow.telecommunity.com> <4A55AF44.4090508@gmail.com> Message-ID: <4A564AA9.3000305@gmail.com> Tres Seaver wrote: > Nick Coghlan wrote: >> The question then would be whether relative paths should be handled >> directly in the RECORD file (as in the current PEP) or by writing a "." >> into PREFIXES to indicate relative paths. > > Per discussion with Matthias Klose, it would be useful to split out the > following prefixes for "special" kinds of data: > > - docs > - i18n / locales There are probably all sorts of other useful distinctions which could be made. However, that would be a topic for a different PEP - as Tarek pointed out, improving distutils is such a huge can of worms that trying to tackle all the issues in a single PEP would most likely just result in developer burnout rather than improvement. The basic PREFIXES discussion is at least somewhat relevant to PEP 376, since the distutils uninstaller needs *some* way for the bdist_* installers to tell it where the non-relative files ended up. Since we need a mechanism for it, it might as well be one that should help with the system package builders needs in the future. Actually adding new categories, while I agree it would be useful, isn't something that is needed to support uninstallation. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Jul 9 22:00:19 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Jul 2009 06:00:19 +1000 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907090722o78973109s536c6734a870ae96@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <20090708183001.0A8A53A407B@sparrow.telecommunity.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> <20090709010447.2A86D3A407B@sparrow.telecommunity.com> <4A55621B.5070904@trueblade.com> <20090709033551.AA35D3A407B@sparrow.telecommunity.com> <4A55F674.6090200@trueblade.com> <79990c6b0907090722o78973109s536c6734a870ae96@mail.gmail.com> Message-ID: <4A564C53.4080102@gmail.com> Paul Moore wrote: > - some cases are not simple, and it's not clear to me how useful > "nearly always accurate" data will be Since the accuracy can always be checked against the filesystem, I think the metadata is useful even if not 100% reliable. Applications that need the extra assurance (such as the distutils uninstaller itself) can check the filesytem and complain when it differs from what it is claimed in the package metadata. One thing that did occur to me based on this - do we want the format to support designation of files (such as config files) that *shouldn't* be uninstalled along with everything else? Or are we happy with not mentioning the file in RECORD at all as the means of supporting that use case? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From fdrake at acm.org Thu Jul 9 22:00:41 2009 From: fdrake at acm.org (Fred Drake) Date: Thu, 09 Jul 2009 16:00:41 -0400 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: <20090709190631.CE40E3A407B@sparrow.telecommunity.com> References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> <20090709020938.5C0A83A407B@sparrow.telecommunity.com> <4A55AF44.4090508@gmail.com> <20090709190631.CE40E3A407B@sparrow.telecommunity.com> Message-ID: <4F597EB9-7612-44DD-B3FA-BD4EFEF51E6C@acm.org> At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote: > - - docs > - - i18n / locales On Jul 9, 2009, at 3:09 PM, P.J. Eby wrote: > Unfortunately, the distutils don't currently have a way to specify > these. There actually is a way to identify documentation files via setup.cfg, but I don't know if the RPM packagers (or others) use that information. -Fred -- Fred Drake From david.lyon at preisshare.net Fri Jul 10 01:29:32 2009 From: david.lyon at preisshare.net (David Lyon) Date: Thu, 09 Jul 2009 19:29:32 -0400 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: <4A564AA9.3000305@gmail.com> References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> <20090709020938.5C0A83A407B@sparrow.telecommunity.com> <4A55AF44.4090508@gmail.com> <4A564AA9.3000305@gmail.com> Message-ID: <52d9d4853c80afe4dd049dc0f2bb4057@preisshare.net> On Fri, 10 Jul 2009 05:53:13 +1000, Nick Coghlan wrote: > There are probably all sorts of other useful distinctions which could be > made. However, that would be a topic for a different PEP - as Tarek > pointed out, improving distutils is such a huge can of worms that trying > to tackle all the issues in a single PEP would most likely just result > in developer burnout rather than improvement. All it requires is a few word changes to the documentation to tell programmers to put your documentation in "docs"... If that is going to burn out the existing developers, it might suggest that it is time to get some fresh developers onboard.... Thats not even a programmatic fix... Regards David From david.lyon at preisshare.net Fri Jul 10 01:22:57 2009 From: david.lyon at preisshare.net (David Lyon) Date: Thu, 09 Jul 2009 19:22:57 -0400 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: <4F597EB9-7612-44DD-B3FA-BD4EFEF51E6C@acm.org> References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> <20090709020938.5C0A83A407B@sparrow.telecommunity.com> <4A55AF44.4090508@gmail.com> <20090709190631.CE40E3A407B@sparrow.telecommunity.com> <4F597EB9-7612-44DD-B3FA-BD4EFEF51E6C@acm.org> Message-ID: On Thu, 09 Jul 2009 16:00:41 -0400, Fred Drake wrote: > At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote: >> - - docs >> - - i18n / locales > > > On Jul 9, 2009, at 3:09 PM, P.J. Eby wrote: >> Unfortunately, the distutils don't currently have a way to specify >> these. > > > There actually is a way to identify documentation files via setup.cfg, > but I don't know if the RPM packagers (or others) use that information. Hi, Perphaps the RPM packagers dont use it but the Python Package Manager GUI (http://sourceforge.net/projects/pythonpkgmgr) does. With that, if there is any package documentation in a docs directory the user can click on it and display it. Regards David From ncoghlan at gmail.com Fri Jul 10 03:48:48 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Jul 2009 11:48:48 +1000 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: <52d9d4853c80afe4dd049dc0f2bb4057@preisshare.net> References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> <20090709020938.5C0A83A407B@sparrow.telecommunity.com> <4A55AF44.4090508@gmail.com> <4A564AA9.3000305@gmail.com> <52d9d4853c80afe4dd049dc0f2bb4057@preisshare.net> Message-ID: <4A569E00.4080509@gmail.com> David Lyon wrote: > All it requires is a few word changes to the documentation to tell > programmers to put your documentation in "docs"... There is no current distutils setting to indicate that a particular data file is actually documentation rather than something else - hence such a thing *isn't* just a matter of updating the documentation. If you're actually saying that "hey, that metadata shouldn't be in the code, it should just be a file naming convention" then that *in itself* is a design decision that needs to be discussed and agreed via a PEP (a 'standard' that nobody follows or has even agreed to follow is a useless standard). > If that is going to burn out the existing developers, it might > suggest that it is time to get some fresh developers onboard.... No, tackling *all* of the distutils issues (supporting uninstallation, moving from setup.py to static metadata for simple cases, providing more fine-grained *programmatic* file categorisation, anything else I've missed) at once would probably burn out anyone that tried to do it. The coding in all this really isn't that hard. It's the gathering of feedback, assessment of use cases and the constant balancing act between retaining enough flexibility to allow for future extensions without a complete rebuild of the design without making the current design so complex as to be unusable. If it was just a matter of throwing some code in the standard library or some words in the documentation and declaring "this is how it shall be from now on" it would have been done years ago :P Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From tjreedy at udel.edu Fri Jul 10 05:34:58 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 09 Jul 2009 23:34:58 -0400 Subject: [Python-Dev] Search error on tracker? Message-ID: Before posting http://bugs.python.org/issue6453 I searched the tracker for 'bool TypeError' to avoid making a duplicate report. A few irrelevant hits were return but not the important one http://bugs.python.org/issue6428 which was the one I needed to find. Going back and searching titles for 'TypeError', cut and pasted from the title of my report, I get 6 hits but neither of the two above. Is there some sort of delay in updating the search indexes? Terry Jan Reedy From martin at v.loewis.de Fri Jul 10 06:44:11 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 10 Jul 2009 06:44:11 +0200 Subject: [Python-Dev] Search error on tracker? In-Reply-To: References: Message-ID: <4A56C71B.5060006@v.loewis.de> Terry Reedy wrote: > Before posting http://bugs.python.org/issue6453 > I searched the tracker for 'bool TypeError' to avoid making a duplicate > report. A few irrelevant hits were return but not the important one > http://bugs.python.org/issue6428 > which was the one I needed to find. When I search for 'bool TypeError', #6428 shows up. Are you sure you had been searching for closed reports as well? > Going back and searching titles for 'TypeError', cut and pasted from the > title of my report, I get 6 hits but neither of the two above. Again, did you search for closed reports also? > Is there some sort of delay in updating the search indexes? No, it's immediate. Regards, Martin From p.f.moore at gmail.com Fri Jul 10 10:56:36 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 10 Jul 2009 09:56:36 +0100 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <4A564C53.4080102@gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> <20090709010447.2A86D3A407B@sparrow.telecommunity.com> <4A55621B.5070904@trueblade.com> <20090709033551.AA35D3A407B@sparrow.telecommunity.com> <4A55F674.6090200@trueblade.com> <79990c6b0907090722o78973109s536c6734a870ae96@mail.gmail.com> <4A564C53.4080102@gmail.com> Message-ID: <79990c6b0907100156i72ceaf2bvdf14edbaa3b3b707@mail.gmail.com> 2009/7/9 Nick Coghlan : > Paul Moore wrote: >> - some cases are not simple, and it's not clear to me how useful >> "nearly always accurate" data will be > > Since the accuracy can always be checked against the filesystem, I think > the metadata is useful even if not 100% reliable. Applications that need > the extra assurance (such as the distutils uninstaller itself) can check > the filesytem and complain when it differs from what it is claimed in > the package metadata. The inaccuracy I was thinking of was that if distribution X's RECORD doesn't mention file Y, the implication is that X doesn't "own" Y. There's no way of checking this against the filesystem - RECORD is the only place that says that X owns Y. I was thinking in terms of "does anyone else mind if I delete/overwrite this file?" queries, not uninstallation. > One thing that did occur to me based on this - do we want the format to > support designation of files (such as config files) that *shouldn't* be > uninstalled along with everything else? Or are we happy with not > mentioning the file in RECORD at all as the means of supporting that use > case? No. In my view, the uninstall feature is a relatively minor issue - people who want uninstall facilities generally use system packages (or complain that easy_install doesn't support uninstallation :-)). They aren't using setup.py install. (Of course, my experience is minimal in this regard, so maybe I'm wrong). Remember that the PEP says "Distutils will provide a **very basic** uninstall function" (my emphasis). Unless I'm misunderstanding, no-one is intending to use the new distutils uninstall feature in their own package manager (bdist_wininst, RPM, apt, ...) Paul. From ziade.tarek at gmail.com Fri Jul 10 15:31:10 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 10 Jul 2009 15:31:10 +0200 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: <20090709190631.CE40E3A407B@sparrow.telecommunity.com> References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> <20090709020938.5C0A83A407B@sparrow.telecommunity.com> <4A55AF44.4090508@gmail.com> <20090709190631.CE40E3A407B@sparrow.telecommunity.com> Message-ID: <94bdd2610907100631j5d5d6befrbcb31627c34a27bf@mail.gmail.com> On Thu, Jul 9, 2009 at 9:09 PM, P.J. Eby wrote: > At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote: >> >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Nick Coghlan wrote: >> > P.J. Eby wrote: >> >>> ?Also, >> >>> why should the RECORD file be generated at all by bdist* commands? >> >> bdist commands that use "install --root" will get it as a side effect, >> >> so they don't have to do anything special. ?They needn't even generate >> >> the PREFIXES, if PREFIXES is also generated by "install --root", and >> >> omits the root from the prefixes written. ?In other words, most bdist >> >> tools wouldn't change. ?bdist_wininst and bdist_msi might need to >> >> regenerate PREFIXES at actual installation time, but AFAIK those are >> >> the >> >> only ones. ?(And the spec would allow for the possibility that a >> >> PREFIXES file can be incorrect.) >> > >> > I haven't seen any suggestions for handling non-relative paths that are >> > cleaner than this. The distutils based list that Tarek posted also gives >> > a good "starter set" of supported entries in the PREFIXES file: >> > >> > PURELIB= >> > PLATLIB= >> > HEADERS= >> > SCRIPTS= >> > DATA= >> > >> > The question then would be whether relative paths should be handled >> > directly in the RECORD file (as in the current PEP) or by writing a "." >> > into PREFIXES to indicate relative paths. I like the idea of a separated PREFIXES file, then using variables in the RECORD file. While this doesn't change anything code-wise, we can keep that way two csv-browsable files. >> >> Per discussion with Matthias Klose, it would be useful to split out the >> following prefixes for "special" kinds of data: >> >> - - docs >> - - i18n / locales > > Unfortunately, the distutils don't currently have a way to specify these. ?I > think we'll need to use only the ones that are distutils install targets at > first, but of course the format is extensible. > > OTOH, one supposed that at least the 'data_files' setup argument could be > enhanced to allow a data type flag, that would then be used as a prefix name > for those files, with a fallback to the DATA prefix if there isn't one > specified. ?But that's something that will require tagging support in the > install_data command. ?As it is, refactoring the distutils to track prefixes > for the RECORD file is probably going to be... ?challenging. ?;-) yes, if we want to provide extensibility, a new prefix that is used by a developer in its distribution will have to be provided for all platforms, so distutils can build the proper RECORD file by querying the path that corresponds to the platform it runs on. In other words, if a data type flag is provided in data_files, the list of absolute paths per key in distutils.command.install.INSTALL_SCHEMES should be provided with it. This could be done with a new 'extra_prefixes' argument for example, provided with the 'data_files', that would let distutils extend INSTALL_SCHEMES on the fly setup( ... data_files={'i18n': ['en.po', 'fr.po']}, extra_prefixes={'i18n': {'unix_prefix': '/somewhere', 'mac': '/Application/right/here'} ... ) But then data_files could maybe be left alone (that's specific to 'data' after all) and a new option created. In any case that would be a great solution to avoid 'wild absolute path' and be sure that the developer thinks about where each fil of his distrbution goes, depending on the plaftorm. While debian packagers and al might not use it to re-package python distributions in debian packages, I am sure they will like it very much because it helps identifiying what the developer wanted to do with these "data" files. From asmodai at in-nomine.org Fri Jul 10 15:58:28 2009 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Fri, 10 Jul 2009 15:58:28 +0200 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: <94bdd2610907100631j5d5d6befrbcb31627c34a27bf@mail.gmail.com> References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> <20090709020938.5C0A83A407B@sparrow.telecommunity.com> <4A55AF44.4090508@gmail.com> <20090709190631.CE40E3A407B@sparrow.telecommunity.com> <94bdd2610907100631j5d5d6befrbcb31627c34a27bf@mail.gmail.com> Message-ID: <20090710135828.GS65921@nexus.in-nomine.org> -On [20090710 15:32], Tarek Ziad? (ziade.tarek at gmail.com) wrote: >setup( > ... > data_files={'i18n': ['en.po', 'fr.po']}, > extra_prefixes={'i18n': {'unix_prefix': '/somewhere', > 'mac': '/Application/right/here'} > ... > ) As an aside: Where i18n above would sooner be called 'locale' rather than i18n, since i18n is only providing software functionality for supporting localization. So in that sense L10n would be better, but locale is quite a defacto standard in many projects. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B In every colour there's the Light... From status at bugs.python.org Fri Jul 10 18:07:30 2009 From: status at bugs.python.org (Python tracker) Date: Fri, 10 Jul 2009 18:07:30 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20090710160730.01D44785FE@psf.upfronthosting.co.za> ACTIVITY SUMMARY (07/03/09 - 07/10/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2245 open (+32) / 16035 closed (+17) / 18280 total (+49) Open issues with patches: 894 Average duration of open issues: 660 days. Median duration of open issues: 414 days. Open Issues Breakdown open 2215 (+32) pending 30 ( +0) Issues Created Or Reopened (52) _______________________________ multiprocessing: BaseManager.from_address documented but doesn't 07/09/09 http://bugs.python.org/issue3518 reopened jnoller logging.basicConfig(level='DEBUG', ... and setLevel("DEBUG") res 07/06/09 http://bugs.python.org/issue6314 reopened r.david.murray patch distutils compiler switch ignored 07/08/09 http://bugs.python.org/issue6377 reopened tarek patch Dictionaries should support __add__ 07/04/09 CLOSED http://bugs.python.org/issue6410 reopened ezio.melotti 3.1 not functional 07/03/09 CLOSED http://bugs.python.org/issue6411 created element.effect Titlecase as defined in Unicode Case Mappings not followed 07/03/09 http://bugs.python.org/issue6412 created christoph patch incorrect log level in distutlis.dist for announce 07/04/09 CLOSED http://bugs.python.org/issue6413 created tarek struct module : processor endianness descriptions misleading 07/04/09 http://bugs.python.org/issue6414 created kmag easy warnings.warn segfaults on bad formatted string 07/04/09 http://bugs.python.org/issue6415 created wiesenth patch Failed to compile selectmodule.c on windows (trunk) 07/04/09 CLOSED http://bugs.python.org/issue6416 created ocean-city easy multiprocessing Process examples: print and getppid 07/04/09 http://bugs.python.org/issue6417 created mnewman unittest.TestProgram change breaks nose 07/04/09 http://bugs.python.org/issue6418 created jpellerin patch Broken test_kqueue.py on OpenBSD 07/04/09 http://bugs.python.org/issue6419 created henry.precheur Fix warning in nismodule.c on OpenBSD 07/04/09 CLOSED http://bugs.python.org/issue6420 created henry.precheur patch errors in docs re module initialization vs self arg to functions 07/04/09 http://bugs.python.org/issue6421 created aleax easy timeit called from within Python should allow autoranging 07/05/09 http://bugs.python.org/issue6422 created scott_daniels patch The cgi docs should advertize using "in" instead of "has_key" 07/05/09 http://bugs.python.org/issue6423 created cito patch Idle3 crashing when using completion 07/05/09 CLOSED http://bugs.python.org/issue6424 created tcourbon imaplib.IMAP4.fetch() is missing documentation for message_set p 07/06/09 CLOSED http://bugs.python.org/issue6425 created Sjoerder imaplib.IMAP4 "command illegal in this state" is unhelpful error 07/06/09 http://bugs.python.org/issue6426 created Sjoerder easy Rename float*.[ch] to double.[ch] 07/06/09 CLOSED http://bugs.python.org/issue6427 created jackdied TypeError: __bool__ should return bool or int, returned int 07/06/09 CLOSED http://bugs.python.org/issue6428 created SilentGhost patch 2to3: fix_future conflicts with fix_print 07/06/09 http://bugs.python.org/issue6429 created alexandre.vassalotti array.array falsely advertises support for 'w' in documentation 07/06/09 http://bugs.python.org/issue6430 created alexandre.vassalotti Fraction fails equality test with a user-defined type 07/07/09 http://bugs.python.org/issue6431 created casevh patch ImageTk.PhotoImage 07/07/09 CLOSED http://bugs.python.org/issue6432 created jpmieville multiprocessing: pool.map hangs on empty list 07/07/09 http://bugs.python.org/issue6433 created ede patch buffer overflow in Zipfile when wrinting more than 2gig file 07/07/09 http://bugs.python.org/issue6434 created segfault42 logging: cache the traceback text on formatter, instead of recor 07/08/09 CLOSED http://bugs.python.org/issue6435 created srid trace module doesn't seem to produce .cover files for Py3 (but d 07/08/09 http://bugs.python.org/issue6436 created mark Personal invitation from Manas 07/08/09 CLOSED http://bugs.python.org/issue6437 created gravitywarrior1 cygwincompiler regular expressions broken 07/08/09 http://bugs.python.org/issue6438 created tarek Demo/embed/demo.c use of PySys_SetArgv() is invalid 07/08/09 http://bugs.python.org/issue6439 created ideasman42 2to3: convert deprecated string.maketrans to str.maketrans 07/08/09 CLOSED http://bugs.python.org/issue6440 created ede Tkinter cannot find *64 bit* Tcl/Tk on Mac OS X 07/08/09 http://bugs.python.org/issue6441 created keflavich Backwards __contains__/has_key in xml/sax/xmlreader.py 07/08/09 http://bugs.python.org/issue6442 created MostAwesomeDude patch _winreg.QueryValueEx confused on double null terminated strings 07/09/09 http://bugs.python.org/issue6443 created AndrewZiem multiline exception logging via syslog handler 07/09/09 http://bugs.python.org/issue6444 created LwarX Add check parameter to subprocess.Popen.communicate 07/09/09 http://bugs.python.org/issue6445 created OG7 import_spam() in extending python can fail without setting the e 07/09/09 http://bugs.python.org/issue6446 created jak PATCH: typo (English) in subprocess module 07/09/09 http://bugs.python.org/issue6447 created Yinon patch imp.find_module() -- be explicity that 'path' must be a list 07/09/09 http://bugs.python.org/issue6448 created jordanb Improve/update python.org/dev/ 07/09/09 http://bugs.python.org/issue6449 created ezio.melotti easy normpath() sometimes maps unicode to str 07/09/09 http://bugs.python.org/issue6450 created sandberg 2to3 adds spare brackets to print 07/09/09 CLOSED http://bugs.python.org/issue6451 created bazooka urllib2.Request() will not work with long authorization headers 07/09/09 http://bugs.python.org/issue6452 created jamal Improve bool TypeError message 07/10/09 CLOSED http://bugs.python.org/issue6453 reopened tjreedy easy Add "example" keyword argument to optparse constructor 07/09/09 http://bugs.python.org/issue6454 created waylonis Lib/distutils/tests/test_build_ext.py fails on windows 07/10/09 CLOSED http://bugs.python.org/issue6455 created ocean-city patch locale.D_* and .T_* are int, not string 07/10/09 CLOSED http://bugs.python.org/issue6456 created dauerbaustelle subprocess.Popen.communicate can lose data from output/error str 07/10/09 http://bugs.python.org/issue6457 created dwalczak With logging.config.fileConfig can't create logger without handl 07/10/09 http://bugs.python.org/issue6458 created sean Issues Now Closed (40) ______________________ xdrlib fails to detect overflow (struct bug?) 586 days http://bugs.python.org/issue1523 marketdickinson python2.6 -3 gives "warning: callable() not supported in 3.x" on 509 days http://bugs.python.org/issue2060 ezio.melotti operator.{isCallable,sequenceIncludes} needs a fixer 474 days http://bugs.python.org/issue2370 alexandre.vassalotti patch, 26backport, needs review Regression for executing packages 432 days http://bugs.python.org/issue2751 ncoghlan truncation of text in tables in Library Reference PDF 407 days http://bugs.python.org/issue2977 georg.brandl datetime.time does not support arithmetic 370 days http://bugs.python.org/issue3250 lemburg patch os.chdir() et al: is the path str or bytes? 298 days http://bugs.python.org/issue3810 r.david.murray pydoc in web server mode tails at initial request 277 days http://bugs.python.org/issue4005 alexandre.vassalotti patch, needs review Tkinter cannot find Tcl/Tk on Mac OS X 279 days http://bugs.python.org/issue4017 ronaldoussoren Partial function application 'from the right' 150 days http://bugs.python.org/issue5191 amaury.forgeotdarc patch Remove deprecated features from struct module 116 days http://bugs.python.org/issue5463 marketdickinson patch SocketServer: TypeError: waitpid() takes no keyword arguments 76 days http://bugs.python.org/issue5814 r.david.murray format(1234.5, '.4') gives misleading result 70 days http://bugs.python.org/issue5864 eric.smith patch Windows Installer Error 1722 when opting for compilation at inst 57 days http://bugs.python.org/issue5960 keldonin Py3.1 pdb doesn't deal well with syntax errors 17 days http://bugs.python.org/issue6323 amaury.forgeotdarc patch, needs review logging: ValueError: I/O operation on closed file 16 days http://bugs.python.org/issue6333 hpk "L" integer suffix in Python 3.1 tutorial 5 days http://bugs.python.org/issue6353 ezio.melotti patch README typo 2 days http://bugs.python.org/issue6398 ezio.melotti patch Error reporting by logging.config.fileConfig() 7 days http://bugs.python.org/issue6399 vsajip Dictionaries should support __add__ 2 days http://bugs.python.org/issue6410 exarkun 3.1 not functional 0 days http://bugs.python.org/issue6411 element.effect incorrect log level in distutlis.dist for announce 0 days http://bugs.python.org/issue6413 tarek Failed to compile selectmodule.c on windows (trunk) 5 days http://bugs.python.org/issue6416 amaury.forgeotdarc easy Fix warning in nismodule.c on OpenBSD 2 days http://bugs.python.org/issue6420 amaury.forgeotdarc patch Idle3 crashing when using completion 0 days http://bugs.python.org/issue6424 amaury.forgeotdarc imaplib.IMAP4.fetch() is missing documentation for message_set p 2 days http://bugs.python.org/issue6425 Sjoerder Rename float*.[ch] to double.[ch] 0 days http://bugs.python.org/issue6427 loewis TypeError: __bool__ should return bool or int, returned int 0 days http://bugs.python.org/issue6428 amaury.forgeotdarc patch ImageTk.PhotoImage 0 days http://bugs.python.org/issue6432 jpmieville logging: cache the traceback text on formatter, instead of recor 2 days http://bugs.python.org/issue6435 vsajip Personal invitation from Manas 0 days http://bugs.python.org/issue6437 marketdickinson 2to3: convert deprecated string.maketrans to str.maketrans 0 days http://bugs.python.org/issue6440 r.david.murray 2to3 adds spare brackets to print 0 days http://bugs.python.org/issue6451 r.david.murray Improve bool TypeError message 0 days http://bugs.python.org/issue6453 amaury.forgeotdarc easy Lib/distutils/tests/test_build_ext.py fails on windows 0 days http://bugs.python.org/issue6455 tarek patch locale.D_* and .T_* are int, not string 0 days http://bugs.python.org/issue6456 amaury.forgeotdarc win32 os.path.normpath not correct for leading slash cases 2371 days http://bugs.python.org/issue665336 amaury.forgeotdarc ntpath.expandvars doesn't expand Windows-style variables. 2091 days http://bugs.python.org/issue824371 amaury.forgeotdarc easy struct.pack raises TypeError where it used to convert 1073 days http://bugs.python.org/issue1530559 marketdickinson patch struct.pack("I", "foo"); struct.pack("L", "foo") should fail 745 days http://bugs.python.org/issue1741130 marketdickinson patch Top Issues Most Discussed (10) ______________________________ 23 distutils compiler switch ignored 2 days open http://bugs.python.org/issue6377 18 logging: ValueError: I/O operation on closed file 16 days closed http://bugs.python.org/issue6333 14 Dictionaries should support __add__ 2 days closed http://bugs.python.org/issue6410 12 platform.version() don't work as expected in Vista in portugues 357 days open http://bugs.python.org/issue3410 10 OS X: python3 from python-3.1.dmg crashes at startup 9 days open http://bugs.python.org/issue6393 9 logging: cache the traceback text on formatter, instead of reco 2 days closed http://bugs.python.org/issue6435 7 _winreg.QueryValueEx confused on double null terminated strings 2 days open http://bugs.python.org/issue6443 7 test_socketserver fails on trunk in test_ForkingTCPServer 11 days open http://bugs.python.org/issue6382 6 3.1 not functional 0 days closed http://bugs.python.org/issue6411 5 Fraction fails equality test with a user-defined type 3 days open http://bugs.python.org/issue6431 From pje at telecommunity.com Fri Jul 10 20:17:40 2009 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 10 Jul 2009 14:17:40 -0400 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: <94bdd2610907100631j5d5d6befrbcb31627c34a27bf@mail.gmail.co m> References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> <20090709020938.5C0A83A407B@sparrow.telecommunity.com> <4A55AF44.4090508@gmail.com> <20090709190631.CE40E3A407B@sparrow.telecommunity.com> <94bdd2610907100631j5d5d6befrbcb31627c34a27bf@mail.gmail.com> Message-ID: <20090710181438.E1CFE3A4752@sparrow.telecommunity.com> At 03:31 PM 7/10/2009 +0200, Tarek Ziad? wrote: >On Thu, Jul 9, 2009 at 9:09 PM, P.J. Eby wrote: > > At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote: > >> > >> -----BEGIN PGP SIGNED MESSAGE----- > >> Hash: SHA1 > >> > >> Nick Coghlan wrote: > >> > P.J. Eby wrote: > >> >>> Also, > >> >>> why should the RECORD file be generated at all by bdist* commands? > >> >> bdist commands that use "install --root" will get it as a side effect, > >> >> so they don't have to do anything special. They needn't even generate > >> >> the PREFIXES, if PREFIXES is also generated by "install --root", and > >> >> omits the root from the prefixes written. In other words, most bdist > >> >> tools wouldn't change. bdist_wininst and bdist_msi might need to > >> >> regenerate PREFIXES at actual installation time, but AFAIK those are > >> >> the > >> >> only ones. (And the spec would allow for the possibility that a > >> >> PREFIXES file can be incorrect.) > >> > > >> > I haven't seen any suggestions for handling non-relative paths that are > >> > cleaner than this. The distutils based list that Tarek posted also gives > >> > a good "starter set" of supported entries in the PREFIXES file: > >> > > >> > PURELIB= > >> > PLATLIB= > >> > HEADERS= > >> > SCRIPTS= > >> > DATA= > >> > > >> > The question then would be whether relative paths should be handled > >> > directly in the RECORD file (as in the current PEP) or by writing a "." > >> > into PREFIXES to indicate relative paths. > >I like the idea of a separated PREFIXES file, then using variables in >the RECORD file. Please note that my proposal intentionally omitted "variables", but instead used a separate column to designate the prefix. This allows some guarantees regarding path sanity that could otherwise be abused, like combining prefixes or inserting them in the middle of paths. In this way, the formal conversion of a path would be os.path.join(PREFIX, *relative_path.split('/')), rather than a variable expansion. From tjreedy at udel.edu Fri Jul 10 20:26:43 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 10 Jul 2009 14:26:43 -0400 Subject: [Python-Dev] Search error on tracker? In-Reply-To: <4A56C71B.5060006@v.loewis.de> References: <4A56C71B.5060006@v.loewis.de> Message-ID: Martin v. L?wis wrote: > Terry Reedy wrote: >> Before posting http://bugs.python.org/issue6453 >> I searched the tracker for 'bool TypeError' to avoid making a duplicate >> report. A few irrelevant hits were return but not the important one >> http://bugs.python.org/issue6428 >> which was the one I needed to find. > > When I search for 'bool TypeError', #6428 shows up. Are you sure you had > been searching for closed reports as well? Un no. Thanks. Perhaps that field should default to 'don't care' like all the others. Sorry for the noise. Terry From martin at v.loewis.de Sat Jul 11 00:50:29 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 11 Jul 2009 00:50:29 +0200 Subject: [Python-Dev] Mercurial: tag generation incorrect Message-ID: <4A57C5B5.5070000@v.loewis.de> According to http://hg.python.org/cpython/tags?revcount=50 the tag r301 belongs to revision be32850b093f. However, this is really r69557, which was not tagged. be32850b093f is listed as having a child revision, 52b0a279fec6, and ISTM that *this* should be the revision that got tagged. Regards, Martin From hfuerstenau at gmx.net Sat Jul 11 07:06:22 2009 From: hfuerstenau at gmx.net (=?ISO-8859-1?Q?Hagen_F=FCrstenau?=) Date: Sat, 11 Jul 2009 07:06:22 +0200 Subject: [Python-Dev] Mercurial: tag generation incorrect In-Reply-To: <4A57C5B5.5070000@v.loewis.de> References: <4A57C5B5.5070000@v.loewis.de> Message-ID: <4A581DCE.4050307@gmx.net> > be32850b093f is listed > as having a child revision, 52b0a279fec6, and ISTM that *this* > should be the revision that got tagged. I think the tag is correct. Note that the concept of tagging is different in Mercurial, where a tag can only refer to a revision previous to the one where it is inserted in .hgtags. If I understand correctly, all relevant tagging revisions from SVN are replaced by Mercurial revisions setting tags, which then refer to their immediate predecessors. - Hagen From ziade.tarek at gmail.com Sat Jul 11 12:28:57 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 11 Jul 2009 12:28:57 +0200 Subject: [Python-Dev] "Absolute" paths in PEP 376 RECORD files In-Reply-To: <20090710181438.E1CFE3A4752@sparrow.telecommunity.com> References: <20090707202705.3C7D23A40D9@sparrow.telecommunity.com> <20090709020938.5C0A83A407B@sparrow.telecommunity.com> <4A55AF44.4090508@gmail.com> <20090709190631.CE40E3A407B@sparrow.telecommunity.com> <94bdd2610907100631j5d5d6befrbcb31627c34a27bf@mail.gmail.com> <20090710181438.E1CFE3A4752@sparrow.telecommunity.com> Message-ID: <94bdd2610907110328x5d0bd8c7w25f44460d55cc7de@mail.gmail.com> On Fri, Jul 10, 2009 at 8:17 PM, P.J. Eby wrote: > At 03:31 PM 7/10/2009 +0200, Tarek Ziad? wrote: >> >> On Thu, Jul 9, 2009 at 9:09 PM, P.J. Eby wrote: >> > At 02:46 PM 7/9/2009 -0400, Tres Seaver wrote: >> >> >> >> -----BEGIN PGP SIGNED MESSAGE----- >> >> Hash: SHA1 >> >> >> >> Nick Coghlan wrote: >> >> > P.J. Eby wrote: >> >> >>> ?Also, >> >> >>> why should the RECORD file be generated at all by bdist* commands? >> >> >> bdist commands that use "install --root" will get it as a side >> >> >> effect, >> >> >> so they don't have to do anything special. ?They needn't even >> >> >> generate >> >> >> the PREFIXES, if PREFIXES is also generated by "install --root", and >> >> >> omits the root from the prefixes written. ?In other words, most >> >> >> bdist >> >> >> tools wouldn't change. ?bdist_wininst and bdist_msi might need to >> >> >> regenerate PREFIXES at actual installation time, but AFAIK those are >> >> >> the >> >> >> only ones. ?(And the spec would allow for the possibility that a >> >> >> PREFIXES file can be incorrect.) >> >> > >> >> > I haven't seen any suggestions for handling non-relative paths that >> >> > are >> >> > cleaner than this. The distutils based list that Tarek posted also >> >> > gives >> >> > a good "starter set" of supported entries in the PREFIXES file: >> >> > >> >> > PURELIB= >> >> > PLATLIB= >> >> > HEADERS= >> >> > SCRIPTS= >> >> > DATA= >> >> > >> >> > The question then would be whether relative paths should be handled >> >> > directly in the RECORD file (as in the current PEP) or by writing a >> >> > "." >> >> > into PREFIXES to indicate relative paths. >> >> I like the idea of a separated PREFIXES file, then using variables in >> the RECORD file. > > Please note that my proposal intentionally omitted "variables", but instead > used a separate column to designate the prefix. ?This allows some guarantees > regarding path sanity that could otherwise be abused, like combining > prefixes or inserting them in the middle of paths. ?In this way, the formal > conversion of a path would be os.path.join(PREFIX, > *relative_path.split('/')), rather than a variable expansion. > Let's build a detailed example on real distributions then. I propose docutils because it has scripts, a module and a package. Could you build its files so we can see ? I'll try to look for other real world examples. Regards Tarek > -- Tarek Ziad? | http://ziade.org From solipsis at pitrou.net Sun Jul 12 15:10:25 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 12 Jul 2009 13:10:25 +0000 (UTC) Subject: [Python-Dev] hg mirror for 3.1 maintenance branch Message-ID: Hi, I've added the release31-maint branch to the temporary Mercurial mirrors: http://code.python.org/hg Regards Antoine. From erik at cq2.nl Mon Jul 13 12:08:29 2009 From: erik at cq2.nl (Erik Groeneveld) Date: Mon, 13 Jul 2009 12:08:29 +0200 Subject: [Python-Dev] Define metatype and a type that uses it In-Reply-To: References: Message-ID: Amaury, Thank you very much for your detailed explanation. It helps to understand it better, and it mostly works now. There is one thing however: On Wed, Jul 8, 2009 at 17:35, Amaury Forgeot d'Arc wrote: > - Don't define a JObjectMeta struct, use JObjectType directly instead. > An instance of the metatype is the type itself! > - Don't set JObjectMetaType.tp_basicsize, let it inherit from the base > type (the correct value would be sizeof(PyHeapTypeObject), this is > important in order to create derived classes in python) I'd like to add a C pointer field to the metatype instance (the type). So, contrary to your advice, I have defined: typedef struct { PyHeapTypeObject x; void* p; } JObjectMeta; This seems the way to do it for objects, but for types, it doesn't seem right, as the p member turns out to be overwritten unexpectedly at runtime. Reading Python's object.h file it turns out that PyHeapTypeObject 'extends' PyTypeObject, which in turn has a PyObject_VAR_HEAD init macro. So mustn't: PyTypeObject JObjectMetaType = { PyObject_HEAD_INIT(NULL) }; actually be: PyTypeObject JObjectMetaType = { PyVarObject_HEAD_INIT(NULL, 1) }; with: JObjectMetaType.tp_basic_size = sizeof(JObjectMeta); JObjectMetaType.tp_itemsize = sizeof(void*); I tried this, but it doesn't keep my app from dumping core on an overwritten 'p'. My question basically is: how can I define a pointer for each type created with this metatype, such as is intended by the JObjectMeta struct? Best regards Erik Groeneveld From amauryfa at gmail.com Mon Jul 13 13:35:22 2009 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 13 Jul 2009 13:35:22 +0200 Subject: [Python-Dev] Define metatype and a type that uses it In-Reply-To: References: Message-ID: Hi, 2009/7/13 Erik Groeneveld : > Amaury, > > Thank you very much for your detailed explanation. ?It helps to > understand it better, and it mostly works now. ?There is one thing > however: > > On Wed, Jul 8, 2009 at 17:35, Amaury Forgeot d'Arc wrote: >> - Don't define a JObjectMeta struct, use JObjectType directly instead. >> An instance of the metatype is the type itself! >> - Don't set JObjectMetaType.tp_basicsize, let it inherit from the base >> type (the correct value would be sizeof(PyHeapTypeObject), this is >> important in order to create derived classes in python) > > I'd like to add a C pointer field to the metatype instance (the type). > So, contrary to your advice, I have defined: > > typedef struct { > ? ?PyHeapTypeObject x; > ? ?void* p; > } JObjectMeta; > > This seems the way to do it for objects, but for types, it doesn't > seem right, as the p member turns out to be overwritten unexpectedly > at runtime. > > Reading Python's object.h file it turns out that PyHeapTypeObject > 'extends' PyTypeObject, which in turn has a PyObject_VAR_HEAD init > macro. ?So mustn't: > > PyTypeObject JObjectMetaType = { > ? ?PyObject_HEAD_INIT(NULL) > }; > > actually be: > > PyTypeObject JObjectMetaType = { > ? ?PyVarObject_HEAD_INIT(NULL, 1) > }; > > with: > > JObjectMetaType.tp_basic_size = sizeof(JObjectMeta); > JObjectMetaType.tp_itemsize = sizeof(void*); > > I tried this, but it doesn't keep my app from dumping core on an > overwritten 'p'. > > My question basically is: how can I define a pointer for each type > created with this metatype, such as is intended by the JObjectMeta > struct? The best is probably to store it in the class dictionary: PyObject_SetAttrString(self, "_javatype_", PyLong_FromVoidPtr(p)); -- Amaury Forgeot d'Arc From fuzzyman at voidspace.org.uk Mon Jul 13 16:48:19 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 13 Jul 2009 15:48:19 +0100 Subject: [Python-Dev] 2.6.3 unittest change breaks nose (issue 6418) In-Reply-To: <3bb02d620907051046k454d8612ic3edd168205b3465@mail.gmail.com> References: <3bb02d620907051046k454d8612ic3edd168205b3465@mail.gmail.com> Message-ID: <4A5B4933.2060801@voidspace.org.uk> jason pellerin wrote: > Bringing python-dev into the discussion at Barry's request. The > summary is that a recent change to unittest.TestProgram breaks nose by > moving self.testRunner initialization from it's old home in > TestProgram.runTests to TestProgram.__init__. The very small patch > attached to the ticket moves it back to runTests. > > Here's the ticket: http://bugs.python.org/issue6418 > And a link to the testing in python list discussion: > > http://lists.idyll.org/pipermail/testing-in-python/2009-July/002032.html > I can look at this - no reason for it not to be included in 2.6.3 and ported to trunk. Sorry for the late reply - I've been on holiday. Michael > JP (primary author of nose) > > ------------------------------------------------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From eric.pruitt at gmail.com Mon Jul 13 17:46:43 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Mon, 13 Jul 2009 10:46:43 -0500 Subject: [Python-Dev] Windows Toolchain Message-ID: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> Hello, I am trying to build Python 3.1 on Windows XP Pro (32 bit) using Microsoft Visual C++ Express Edition in order to test some modifications I made to the _subprocess.c file as part of my Google Summer of Code proposal. I cannot seem to figure out how to compile Python on Windows and could use some guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not work when I attempt to open and no error message is given when clicked. When opened from the Visual C++, it says "Solution folders are not supported in this version of the application." Any advice is greatly appreciated, Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pruitt at gmail.com Mon Jul 13 17:53:07 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Mon, 13 Jul 2009 10:53:07 -0500 Subject: [Python-Dev] Windows Toolchain In-Reply-To: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> Message-ID: <171e8a410907130853s61f29beew8b0ed96502b08e0b@mail.gmail.com> Sorry, forgot to include my build log: Command Lines: http://pastebin.com/f25614b01 Output Window: > Compiling... > python.c > Compiling resources... > Microsoft (R) Windows (R) Resource Compiler Version 6.1.6723.1 > Copyright (C) Microsoft Corporation. All rights reserved. > Linking... > LINK : fatal error LNK1181: cannot open input file '.\python31_d.lib' > Creating browse information file... > Microsoft Browse Information Maintenance Utility Version 9.00.30729 > Copyright (C) Microsoft Corporation. All rights reserved. On Mon, Jul 13, 2009 at 10:46, Eric Pruitt wrote: > Hello, > > I am trying to build Python 3.1 on Windows XP Pro (32 bit) using Microsoft > Visual C++ Express Edition in order to test some modifications I made to the > _subprocess.c file as part of my Google Summer of Code proposal. I cannot > seem to figure out how to compile Python on Windows and could use some > guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not > work when I attempt to open and no error message is given when clicked. When > opened from the Visual C++, it says "Solution folders are not supported in > this version of the application." > > Any advice is greatly appreciated, > Eric > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric.pruitt at gmail.com Mon Jul 13 18:32:49 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Mon, 13 Jul 2009 11:32:49 -0500 Subject: [Python-Dev] Windows Toolchain In-Reply-To: References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> Message-ID: <171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> I opened the solution, hit Ctrl+F5 and it began compiling but it fails with this error: LINK :fatal error LNK1181: cannot open input file '.\python31_d.lib' How do I specify that "pythoncore" be built? When I follow your instructions, right clicking on the "python" project in the "Solution Explorer" panel, I get the exact same link error as above. If I hit F7, the build log changes completely. The last lines of output with F7 are as follows: > 21>Build log was saved at "file://C:\Documents and Settings\James\Desktop\Python-3.1\PCbuild\Win32-temp-Debug\select\BuildLog.htm" > 21>select - 1 error(s), 0 warning(s) > ========== Build: 0 succeeded, 20 failed, 4 up-to-date, 1 skipped ========== On Mon, Jul 13, 2009 at 11:03, Amaury Forgeot d'Arc wrote: > Hi, > > 2009/7/13 Eric Pruitt : > > Hello, > > > > I am trying to build Python 3.1 on Windows XP Pro (32 bit) using > Microsoft > > Visual C++ Express Edition in order to test some modifications I made to > the > > _subprocess.c file as part of my Google Summer of Code proposal. I cannot > > seem to figure out how to compile Python on Windows and could use some > > guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not > > work when I attempt to open and no error message is given when clicked. > > Is Visual C++ correctly installed? > > > When > > opened from the Visual C++, it says "Solution folders are not supported > in > > this version of the application." > > This is a warning from the Express Edition, which has absolutely no > impact on the build. You can safely ignore it and continue. > > What steps did you perform exactly? From your build log the > "pythoncore" project was not built. Why? > > The following has always worked for me: In the "Solution explorer" > panel, right-click on the "python" project, and click "Build". This > should build the project and all its dependencies. > Or just hit the F7 key and watch the whole solution build. > > -- > Amaury Forgeot d'Arc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Mon Jul 13 18:37:57 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 13 Jul 2009 17:37:57 +0100 Subject: [Python-Dev] 2.6.3 unittest change breaks nose (issue 6418) In-Reply-To: <3bb02d620907051046k454d8612ic3edd168205b3465@mail.gmail.com> References: <3bb02d620907051046k454d8612ic3edd168205b3465@mail.gmail.com> Message-ID: <4A5B62E5.1090700@voidspace.org.uk> jason pellerin wrote: > Bringing python-dev into the discussion at Barry's request. The > summary is that a recent change to unittest.TestProgram breaks nose by > moving self.testRunner initialization from it's old home in > TestProgram.runTests to TestProgram.__init__. The very small patch > attached to the ticket moves it back to runTests. > > Here's the ticket: http://bugs.python.org/issue6418 > And a link to the testing in python list discussion: > > http://lists.idyll.org/pipermail/testing-in-python/2009-July/002032.html > > This is fixed on the Python 2.6 maintenance branch (revision 73997) but needs fixing on trunk. No problem applying the fix but I'm struggling to test this without causing spurious extra output on stderr. The style of dependency injection used means that merely overriding sys.stderr does nothing and a custom stream can't be used as we want to test the default initialisation behavior. Michael Foord > JP (primary author of nose) > > ------------------------------------------------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From eric.pruitt at gmail.com Mon Jul 13 19:00:34 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Mon, 13 Jul 2009 12:00:34 -0500 Subject: [Python-Dev] Windows Toolchain In-Reply-To: References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> <171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> Message-ID: <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to be Unix style but after fixing them, I still have the same problem. On Mon, Jul 13, 2009 at 11:43, Amaury Forgeot d'Arc wrote: > 2009/7/13 Eric Pruitt : > > I opened the solution, hit Ctrl+F5 and it began compiling but it fails > with > > this error: > > LINK :fatal error LNK1181: cannot open input file '.\python31_d.lib' > > > > How do I specify that "pythoncore" be built? When I follow your > > instructions, right clicking on the "python" project in the "Solution > > Explorer" panel, I get the exact same link error as above. If I hit F7, > the > > build log changes completely. The last lines of output with F7 are as > > follows: > > > >> 21>Build log was saved at "file://C:\Documents and > >> > Settings\James\Desktop\Python-3.1\PCbuild\Win32-temp-Debug\select\BuildLog.htm" > >> 21>select - 1 error(s), 0 warning(s) > >> ========== Build: 0 succeeded, 20 failed, 4 up-to-date, 1 skipped > >> ========== > > > > Did you open the sub-project "python.vcproj" by mistake? > You must open the solution file: "pcbuild.sln". > > And if pcbuild.sln does not load correctly, check that it has correct > line endings: it is supposed to have DOS line endings. > > -- > Amaury Forgeot d'Arc > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tlesher at gmail.com Mon Jul 13 19:36:08 2009 From: tlesher at gmail.com (Tim Lesher) Date: Mon, 13 Jul 2009 13:36:08 -0400 Subject: [Python-Dev] Windows Toolchain In-Reply-To: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> Message-ID: <9613db600907131036n12f92e07m89b36404a6e2476@mail.gmail.com> On Mon, Jul 13, 2009 at 11:46, Eric Pruitt wrote: > Hello, > > I am trying to build Python 3.1 on Windows XP Pro (32 bit) using Microsoft > Visual C++ Express Edition in order to test some modifications I made to the > _subprocess.c file as part of my Google Summer of Code proposal. I cannot > seem to figure out how to compile Python on Windows and could use some > guidance. The Visual Studio Solution file ../PCbuild/pcbuild.sln does not > work when I attempt to open and no error message is given when clicked. When > opened from the Visual C++, it says "Solution folders are not supported in > this version of the application." What version of Visual C++ Express are you using? This sounds suspiciously like a version issue. -- Tim Lesher From p.f.moore at gmail.com Mon Jul 13 20:09:45 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 13 Jul 2009 19:09:45 +0100 Subject: [Python-Dev] Windows Toolchain In-Reply-To: <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> <171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> Message-ID: <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> 2009/7/13 Eric Pruitt : > It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to be > Unix style but after fixing them, I still have the same problem. Where did you get your copy of the Python source from? If it's from Subversion, I'm surprised the line endings are wrong. If it's Mercurial, then this may be indicative of the still somewhat open question of how line endings will be handled in Mercurial (which doesn't have a server-side way of saying that a file should use platform-native line endings). If you still have the same problem after fixing the file, this may indicate further damage, or maybe the fix isn't doing the right thing as the file has already been opened (and mangled) by Visual Studio, I'd suggest trying to reproduce your issue with a clean checkout from Subversion. It would also help if you could describe precisely the steps you took to download the sources and build them (I appreciate that this may not be easy now, after you've tried various things). Paul. From eric.pruitt at gmail.com Mon Jul 13 20:21:42 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Mon, 13 Jul 2009 13:21:42 -0500 Subject: [Python-Dev] Windows Toolchain In-Reply-To: <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> <171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> Message-ID: <171e8a410907131121y354c5fbfn7cdbb993474760ff@mail.gmail.com> The code I downloaded was the official Python 3.1 release tarball. I only checked the line endings in the pcbuild.sln file so I am not sure about the other files. I will attempt to build the Python 3000 SVN trunk shortly and report back. On Mon, Jul 13, 2009 at 13:09, Paul Moore wrote: > 2009/7/13 Eric Pruitt : > > It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to > be > > Unix style but after fixing them, I still have the same problem. > > Where did you get your copy of the Python source from? If it's from > Subversion, I'm surprised the line endings are wrong. If it's > Mercurial, then this may be indicative of the still somewhat open > question of how line endings will be handled in Mercurial (which > doesn't have a server-side way of saying that a file should use > platform-native line endings). > > If you still have the same problem after fixing the file, this may > indicate further damage, or maybe the fix isn't doing the right thing > as the file has already been opened (and mangled) by Visual Studio, > > I'd suggest trying to reproduce your issue with a clean checkout from > Subversion. > > It would also help if you could describe precisely the steps you took > to download the sources and build them (I appreciate that this may not > be easy now, after you've tried various things). > > Paul. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Jul 13 20:42:33 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 13 Jul 2009 19:42:33 +0100 Subject: [Python-Dev] Windows Toolchain In-Reply-To: <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> <171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> Message-ID: <79990c6b0907131142l61f151cexf3a4ddd713fbb5c9@mail.gmail.com> 2009/7/13 Paul Moore : > 2009/7/13 Eric Pruitt : >> It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to be >> Unix style but after fixing them, I still have the same problem. [...] > I'd suggest trying to reproduce your issue with a clean checkout from > Subversion. FWIW, I did: svn co http://svn.python.org/projects/python/tags/r31/ py31 Open pcbuild in Visual C++ 2008 Express. OK on the warning about solution folders. Right click "python" and choose "Build". Works OK here. Also works if I download Python-3.1.tgz from python.org and unpack it using (command line) bsdtar. I wonder - did you use WinZip to unpack the archive? That has some sort of brain-damaged "Smart" CRLF conversion for tar files, which may have screwed up your files. Paul. From eric.pruitt at gmail.com Mon Jul 13 22:14:46 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Mon, 13 Jul 2009 15:14:46 -0500 Subject: [Python-Dev] Windows Toolchain In-Reply-To: <79990c6b0907131142l61f151cexf3a4ddd713fbb5c9@mail.gmail.com> References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> <171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> <79990c6b0907131142l61f151cexf3a4ddd713fbb5c9@mail.gmail.com> Message-ID: <171e8a410907131314p328abe3h6ae86a61b07966b4@mail.gmail.com> Well, I narrowed the culprit down to Windows DEP. I had been having trouble with it for some time now, with it raising an error for nearly every program I ran so I just disabled it completely and Python now compiles fine. Thank you all for the assistance. On Mon, Jul 13, 2009 at 13:42, Paul Moore wrote: > 2009/7/13 Paul Moore : > > 2009/7/13 Eric Pruitt : > >> It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to > be > >> Unix style but after fixing them, I still have the same problem. > [...] > > I'd suggest trying to reproduce your issue with a clean checkout from > > Subversion. > > FWIW, I did: > > svn co http://svn.python.org/projects/python/tags/r31/ py31 > > Open pcbuild in Visual C++ 2008 Express. OK on the warning about > solution folders. > > Right click "python" and choose "Build". > > Works OK here. Also works if I download Python-3.1.tgz from python.org > and unpack it using (command line) bsdtar. I wonder - did you use > WinZip to unpack the archive? That has some sort of brain-damaged > "Smart" CRLF conversion for tar files, which may have screwed up your > files. > > Paul. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From SridharR at activestate.com Mon Jul 13 22:40:25 2009 From: SridharR at activestate.com (Sridhar Ratnakumar) Date: Mon, 13 Jul 2009 13:40:25 -0700 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <79990c6b0907100156i72ceaf2bvdf14edbaa3b3b707@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907081456q375d8f2cn5ceb00ec329bf934@mail.gmail.com> <4A551899.1080105@trueblade.com> <87r5wqu1ux.fsf@uwakimon.sk.tsukuba.ac.jp> <20090709010447.2A86D3A407B@sparrow.telecommunity.com> <4A55621B.5070904@trueblade.com> <20090709033551.AA35D3A407B@sparrow.telecommunity.com> <4A55F674.6090200@trueblade.com> <79990c6b0907090722o78973109s536c6734a870ae96@mail.gmail.com> <4A564C53.4080102@gmail.com> <79990c6b0907100156i72ceaf2bvdf14edbaa3b3b707@mail.gmail.com> Message-ID: On Fri, 10 Jul 2009 01:56:36 -0700, Paul Moore wrote: >> One thing that did occur to me based on this - do we want the format to >> support designation of files (such as config files) that *shouldn't* be >> uninstalled along with everything else? Or are we happy with not >> mentioning the file in RECORD at all as the means of supporting that use >> case? > No. In my view, the uninstall feature is a relatively minor issue - > people who want uninstall facilities generally use system packages (or > complain that easy_install doesn't support uninstallation :-)). They > aren't using setup.py install. (Of course, my experience is minimal in > this regard, so maybe I'm wrong). > Remember that the PEP says "Distutils will provide a **very basic** > uninstall function" (my emphasis). > Unless I'm misunderstanding, no-one is intending to use the new > distutils uninstall feature in their own package manager > (bdist_wininst, RPM, apt, ...) Perhaps distutils can also provide an `uninstall` hook that application can use to do some pre-processing? >> import distutils.installer >> distutils.installer.register_uninstall_hoook(my_uninstall_hook) I imagine that packages like pywin32 may have to register DLLs (install time) and unregister them (uninstall time). Just a wild idea .. I understand that this may not be related to the current PEP. :-) -srid From ncoghlan at gmail.com Mon Jul 13 23:04:34 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Jul 2009 07:04:34 +1000 Subject: [Python-Dev] Windows Toolchain In-Reply-To: <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> <171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> Message-ID: <4A5BA162.4010804@gmail.com> Paul Moore wrote: > 2009/7/13 Eric Pruitt : >> It is indeed the file ../PCBuild/pcbuild.sln. The line endings appear to be >> Unix style but after fixing them, I still have the same problem. > > Where did you get your copy of the Python source from? If it's from > Subversion, I'm surprised the line endings are wrong. If it's > Mercurial, then this may be indicative of the still somewhat open > question of how line endings will be handled in Mercurial (which > doesn't have a server-side way of saying that a file should use > platform-native line endings). For the record, we only have SVN set to force Windows line endings for the old VC6 project files (.dsp and .dsw). The newer versions of Visual Studio can handle Unix line endings just fine so the .sln/.vcprops/etc files are left with native line endings and hence end up with Unix line endings in the source tarball. Cheers, Nick. P.S. I must admit I've never really understood how Data Execution Prevention is ever going to work in a world with interpreted languages and just in time compilers... the line between data and code is fuzzier than one might think for a lot of programs... -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Mon Jul 13 23:09:07 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Jul 2009 07:09:07 +1000 Subject: [Python-Dev] Windows Toolchain In-Reply-To: <4A5BA162.4010804@gmail.com> References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> <171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> <4A5BA162.4010804@gmail.com> Message-ID: <4A5BA273.3010907@gmail.com> Nick Coghlan wrote: > For the record, we only have SVN set to force Windows line endings for > the old VC6 project files (.dsp and .dsw). The newer versions of Visual > Studio can handle Unix line endings just fine so the .sln/.vcprops/etc > files are left with native line endings and hence end up with Unix line > endings in the source tarball. It would appear that Amaury just changed that. I'm not sure that was actually necessary though (since nobody else has complained and the original poster of this thread discovered the line endings weren't the problem after all). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From martin at v.loewis.de Mon Jul 13 23:24:01 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 13 Jul 2009 23:24:01 +0200 Subject: [Python-Dev] Windows Toolchain In-Reply-To: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> Message-ID: <4A5BA5F1.7080504@v.loewis.de> > I am trying to build Python 3.1 on Windows XP Pro (32 bit) using > Microsoft Visual C++ Express Edition in order to test some modifications > I made to the _subprocess.c file as part of my Google Summer of Code > proposal. As a posting guideline, please be careful to not post too many messages in quick succession. If you run into a problem, try to solve it for yourself. If you fail, consult your mentor and/or wait a couple of hours. Then, when posting, summarize your findings and the alternatives that you tried unsuccessfully. If people learn that you post while being in the process of working on a problem, they will learn that it is better to wait a day or two before responding, in case you figure it out on your own. Regards, Martin From amauryfa at gmail.com Mon Jul 13 23:40:30 2009 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 13 Jul 2009 23:40:30 +0200 Subject: [Python-Dev] Windows Toolchain In-Reply-To: <4A5BA273.3010907@gmail.com> References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> <171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> <4A5BA162.4010804@gmail.com> <4A5BA273.3010907@gmail.com> Message-ID: 2009/7/13 Nick Coghlan : > Nick Coghlan wrote: >> For the record, we only have SVN set to force Windows line endings for >> the old VC6 project files (.dsp and .dsw). The newer versions of Visual >> Studio can handle Unix line endings just fine so the .sln/.vcprops/etc >> files are left with native line endings and hence end up with Unix line >> endings in the source tarball. > > It would appear that Amaury just changed that. I'm not sure that was > actually necessary though (since nobody else has complained and the > original poster of this thread discovered the line endings weren't the > problem after all). Well, I was caught several times by this line ending problem. And the first time, I did not even understand what was going on... -- Amaury Forgeot d'Arc From curt at hagenlocher.org Mon Jul 13 23:50:48 2009 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Mon, 13 Jul 2009 14:50:48 -0700 Subject: [Python-Dev] Windows Toolchain In-Reply-To: <4A5BA162.4010804@gmail.com> References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> <171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> <4A5BA162.4010804@gmail.com> Message-ID: On Mon, Jul 13, 2009 at 2:04 PM, Nick Coghlan wrote: > > P.S. I must admit I've never really understood how Data Execution > Prevention is ever going to work in a world with interpreted languages > and just in time compilers... the line between data and code is fuzzier > than one might think for a lot of programs... The OS has to provide a mechanism to enable execution for a particular region of memory. Under Windows, this is done by the VirtualProtect function. -- Curt Hagenlocher curt at hagenlocher.org From solipsis at pitrou.net Mon Jul 13 23:55:11 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 13 Jul 2009 21:55:11 +0000 (UTC) Subject: [Python-Dev] Windows Toolchain References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> <171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> <4A5BA162.4010804@gmail.com> Message-ID: Curt Hagenlocher hagenlocher.org> writes: > > The OS has to provide a mechanism to enable execution for a particular > region of memory. Under Windows, this is done by the VirtualProtect > function. More surprising is that Microsoft didn't whitelist their own toolchain. From curt at hagenlocher.org Tue Jul 14 00:07:45 2009 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Mon, 13 Jul 2009 15:07:45 -0700 Subject: [Python-Dev] Windows Toolchain In-Reply-To: References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> <171e8a410907130932p4dc34a4fga82392ae9f29ff87@mail.gmail.com> <171e8a410907131000u5fa50032l68e3abdf16ff5cf2@mail.gmail.com> <79990c6b0907131109s6dccb046hdd6cfe44b2c2ad@mail.gmail.com> <4A5BA162.4010804@gmail.com> Message-ID: On Mon, Jul 13, 2009 at 2:55 PM, Antoine Pitrou wrote: > > Curt Hagenlocher hagenlocher.org> writes: >> >> The OS has to provide a mechanism to enable execution for a particular >> region of memory. Under Windows, this is done by the VirtualProtect >> function. > > More surprising is that Microsoft didn't whitelist their own toolchain. But dev tools don't actually execute the code they create in-process; they write the executable code out to disk and a new process is spun up from that image. So there should be no need to disable DEP for anything that the toolchain creates directly. Given that this was affecting multiple programs, the most likely culprit is either an anti-virus program or a virus -- though you probably can't rule out some kind of glitch in the OS itself. :/ -- Curt Hagenlocher curt at hagenlocher.org From sridharr at activestate.com Tue Jul 14 02:07:02 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Mon, 13 Jul 2009 17:07:02 -0700 Subject: [Python-Dev] PEP 376 - Open questions In-Reply-To: <94bdd2610907090122x12a3f25s13bba018240f331d@mail.gmail.com> References: <79990c6b0907061138o3fe12709vc39876256f38e6fe@mail.gmail.com> <79990c6b0907080513g6f9c9318i92d3e7624a9c388f@mail.gmail.com> <4A54984D.4020305@trueblade.com> <79990c6b0907080628r50e569aaue0261c422d2710c@mail.gmail.com> <4A54AF4F.9080800@trueblade.com> <79990c6b0907080757y576599bav3aa8c93904722e07@mail.gmail.com> <20090708151647.9E0703A407B@sparrow.telecommunity.com> <79990c6b0907080922v4242672fg1abb3eb02a8004d9@mail.gmail.com> <94bdd2610907090122x12a3f25s13bba018240f331d@mail.gmail.com> Message-ID: On Thu, 09 Jul 2009 01:22:19 -0700, Tarek Ziad? wrote: > On Thu, Jul 9, 2009 at 3:42 AM, Sridhar > Ratnakumar wrote: >> Other than easy_install/pip, there is also PyPM which is being >> developed at >> ActiveState. PyPM is the Python package manager much like what ppm is >> for >> ActivePerl. >> > > Great ! besides the RECORD file, anything remark on the PEP from a > PyPM point of view ? I'll shortly send an email I composed regarding this. > Once Paul has finished to work on the PEP 302 part of the prototype, > maybe we could try it out > on PyPM to see how it fits ? I took a look at PEP 302 and Paul's new implemention in pkgutil2.py http://bitbucket.org/tarek/pep376/src/tip/pkgutil2.py And yes, I'd happy to try the new pkgutil API on PyPM .. keeping interoperability with pip/easy_install installed packages in mind. -srid From sridharr at activestate.com Tue Jul 14 02:12:32 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Mon, 13 Jul 2009 17:12:32 -0700 Subject: [Python-Dev] PEP 376 - from PyPM's point of view Message-ID: Here are my comments regarding PEP 376 with respect to PyPM (the Python package manager being developd at ActiveState) Multiple versions: I understand that the PEP does not support installation (thus uninstallation) of multiple versions of the same package. Should this be explicitly mentioned in the PEP -- as `get_distribution` API accepts only `name` argument, and not a `version` argument? > get_distribution(name) -> Distribution or None. >Scans all elements in sys.path and looks for all directories ending > with .egg-info. Returns a Distribution corresponding to the .egg-info > directory that contains a PKG-INFO that matches name for the name > metadata. >Notice that there should be at most one result. The first result > founded is returned. If the directory is not found, returns None. Some packages have package names with mixed case. Example: ConfigObj .. as registered in setup.py. However, other packages such as turbogears specifies "configobj" (lowercase) in their install_requires. Is `get_distribution(name)` supposed to handle mixed cases? Will it match both 'ConfigObj' and 'configobj'? > get_installed_files(local=False) -> iterator of (path, md5, size) Will this also return the directories /created/ during the installation? For example, will it also contain the entry "docutils" .. along with "docutils/__init__.py"? If not, how is the installer (pip, pypm, etc..) supposed to know which directories to remove (docutils/) and which directories not to remove (site-packages/, bin/, etc..)? > The new version of PEP 345 (XXX work in progress) extends the Metadata > standard and fullfills the requirements described in PEP 262, like the > REQUIRES section. Can you tell more about this? I see that PEP 262 allows both distributions names ('docutils') and modules/packages ('roman.py') in the 'Requires:' section. Is this how the new PEP is going to adhere to? Or, is it going to adhere to PEP 345's way of allowing *only* modules/packages? In practice, I noticed that packages usually specify distribution names in their 'Requires:' file (or install_requires.txt in the case of setuptools). Hence, PyPM *assumes* the install requirements to be distribution name. But then .. most distributions have the same name as their primary module/package. Ok, so PEP 345 also specifies the 'Provides:' header. Does easy_install/pip make use 'Provides:' at all when resolving dependencies? For example, does 'pip install sphinx' go look for all distributions that 'provides' the 'docutils' provision.. or does it simply get the distribution named 'docutils'? -srid From fuzzyman at voidspace.org.uk Tue Jul 14 03:15:22 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 14 Jul 2009 02:15:22 +0100 Subject: [Python-Dev] PEP 376 - from PyPM's point of view In-Reply-To: References: Message-ID: -- http://www.ironpythoninaction.com On 14 Jul 2009, at 01:12, "Sridhar Ratnakumar" wrote: > Here are my comments regarding PEP 376 with respect to PyPM (the > Python > package manager being developd at ActiveState) > > > Multiple versions: I understand that the PEP does not support > installation (thus uninstallation) of multiple versions of the same > package. Should this be explicitly mentioned in the PEP -- as > `get_distribution` API accepts only `name` argument, and not a > `version` > argument? > >> get_distribution(name) -> Distribution or None. >> Scans all elements in sys.path and looks for all directories ending >> with .egg-info. Returns a Distribution corresponding to the .egg-info >> directory that contains a PKG-INFO that matches name for the name >> metadata. >> Notice that there should be at most one result. The first result >> founded is returned. If the directory is not found, returns None. > > Some packages have package names with mixed case. Example: ConfigObj > .. as registered in setup.py. However, other packages such as > turbogears > specifies "configobj" (lowercase) in their install_requires. > > Is `get_distribution(name)` supposed to handle mixed cases? Will it > match both 'ConfigObj' and 'configobj'? > An abomination for which I am truly sorry - however to be precise I'm pretty sure the setup.py specifies configobj and it is only registered on PyPI with mixed case (which I don't believe I can change). Michael >> get_installed_files(local=False) -> iterator of (path, md5, size) > > Will this also return the directories /created/ during the > installation? > For example, will it also contain the entry "docutils" .. along with > "docutils/__init__.py"? > > If not, how is the installer (pip, pypm, etc..) supposed to know which > directories to remove (docutils/) and which directories not to remove > (site-packages/, bin/, etc..)? > >> The new version of PEP 345 (XXX work in progress) extends the >> Metadata >> standard and fullfills the requirements described in PEP 262, like >> the >> REQUIRES section. > > Can you tell more about this? > > I see that PEP 262 allows both distributions names ('docutils') and > modules/packages ('roman.py') in the 'Requires:' section. Is this how > the new PEP is going to adhere to? Or, is it going to adhere to PEP > 345's way of allowing *only* modules/packages? > > In practice, I noticed that packages usually specify distribution > names > in their 'Requires:' file (or install_requires.txt in the case of > setuptools). Hence, PyPM *assumes* the install requirements to be > distribution name. But then .. most distributions have the same name > as > their primary module/package. > > Ok, so PEP 345 also specifies the 'Provides:' header. Does > easy_install/pip make use 'Provides:' at all when resolving > dependencies? For example, does 'pip install sphinx' go look for all > distributions that 'provides' the 'docutils' provision.. or does it > simply get the distribution named 'docutils'? > > -srid > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk From eric.pruitt at gmail.com Tue Jul 14 19:42:09 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Tue, 14 Jul 2009 12:42:09 -0500 Subject: [Python-Dev] Windows Toolchain In-Reply-To: <4A5BA5F1.7080504@v.loewis.de> References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> <4A5BA5F1.7080504@v.loewis.de> Message-ID: <171e8a410907141042y3c2ce93bwfc38b23db4ce02f@mail.gmail.com> I will keep a grace period in mind when I am posting. In my defense, however, I had been working on the problem for some time before posting this and my messages in quick succession were in response to the suggestions people offered. The problem has arisen again but I will work on it some more and will also post a detailed post on my GSoC blog of the process I went through when I encountered the issue. On Mon, Jul 13, 2009 at 16:24, "Martin v. L?wis" wrote: > > I am trying to build Python 3.1 on Windows XP Pro (32 bit) using > > Microsoft Visual C++ Express Edition in order to test some modifications > > I made to the _subprocess.c file as part of my Google Summer of Code > > proposal. > > As a posting guideline, please be careful to not post too many messages > in quick succession. If you run into a problem, try to solve it for > yourself. If you fail, consult your mentor and/or wait a couple of > hours. Then, when posting, summarize your findings and the alternatives > that you tried unsuccessfully. > > If people learn that you post while being in the process of working > on a problem, they will learn that it is better to wait a day or two > before responding, in case you figure it out on your own. > > Regards, > Martin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Tue Jul 14 22:16:10 2009 From: dinov at microsoft.com (Dino Viehland) Date: Tue, 14 Jul 2009 20:16:10 +0000 Subject: [Python-Dev] 2.6 object.__init__ & deling __new__ Message-ID: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com> I'm updating IronPython to match CPython's behavior w/ for this issue: http://bugs.python.org/issue1683368 One thing that I've noticed is that this doesn't seem to be respecting the deletion of attributes (on 2.6.2): class x(object): pass x().__init__(2,3,4) # throws - seems right class x(object): def __new__(cls, *args): return object.__new__(cls) x().__init__(2,3,4) # doesn't throw - seems right del x.__new__ x().__init__(2,3,4) # doesn't throw - I would expect this to throw. Is this just a bug in CPython not updating whether __new__ has been defined? Or is there something that makes this behavior expected which I'm just missing? -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Tue Jul 14 23:11:07 2009 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 14 Jul 2009 16:11:07 -0500 Subject: [Python-Dev] 2.6 object.__init__ & deling __new__ In-Reply-To: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com> References: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com> Message-ID: <1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com> 2009/7/14 Dino Viehland : > Is this just a bug in CPython not updating whether __new__ has been > defined?? Or is there something that makes this behavior expected which I?m > just missing? There's a wonderful comment about this in typeobject.c: /* You may wonder why object.__new__() only complains about arguments when object.__init__() is not overridden, and vice versa. Consider the use cases: 1. When neither is overridden, we want to hear complaints about excess (i.e., any) arguments, since their presence could indicate there's a bug. 2. When defining an Immutable type, we are likely to override only __new__(), since __init__() is called too late to initialize an Immutable object. Since __new__() defines the signature for the type, it would be a pain to have to override __init__() just to stop it from complaining about excess arguments. 3. When defining a Mutable type, we are likely to override only __init__(). So here the converse reasoning applies: we don't want to have to override __new__() just to stop it from complaining. 4. When __init__() is overridden, and the subclass __init__() calls object.__init__(), the latter should complain about excess arguments; ditto for __new__(). Use cases 2 and 3 make it unattractive to unconditionally check for excess arguments. The best solution that addresses all four use cases is as follows: __init__() complains about excess arguments unless __new__() is overridden and __init__() is not overridden (IOW, if __init__() is overridden or __new__() is not overridden); symmetrically, __new__() complains about excess arguments unless __init__() is overridden and __new__() is not overridden (IOW, if __new__() is overridden or __init__() is not overridden). However, for backwards compatibility, this breaks too much code. Therefore, in 2.6, we'll *warn* about excess arguments when both methods are overridden; for all other cases we'll use the above rules. */ -- Regards, Benjamin From eric.pruitt at gmail.com Tue Jul 14 23:14:58 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Tue, 14 Jul 2009 16:14:58 -0500 Subject: [Python-Dev] Windows Toolchain In-Reply-To: <4A5BA5F1.7080504@v.loewis.de> References: <171e8a410907130846id402867k7444bcb346d8546e@mail.gmail.com> <4A5BA5F1.7080504@v.loewis.de> Message-ID: <171e8a410907141414t49d3369v213a8021ec49efbc@mail.gmail.com> The problem appears resolved again and I have two posts on the issue on my blog located at http://subdev.blogspot.com/. I was missing an MSVC++ run-time DLL and re-installing Visual C++ Express fixed the problem. A bit of a somewhat random note -- looking through some of the buildbot output for Python 3.1, I noticed that at one point, the bot compilations had the same issue; Google Cache . On Mon, Jul 13, 2009 at 16:24, "Martin v. L?wis" wrote: > > I am trying to build Python 3.1 on Windows XP Pro (32 bit) using > > Microsoft Visual C++ Express Edition in order to test some modifications > > I made to the _subprocess.c file as part of my Google Summer of Code > > proposal. > > As a posting guideline, please be careful to not post too many messages > in quick succession. If you run into a problem, try to solve it for > yourself. If you fail, consult your mentor and/or wait a couple of > hours. Then, when posting, summarize your findings and the alternatives > that you tried unsuccessfully. > > If people learn that you post while being in the process of working > on a problem, they will learn that it is better to wait a day or two > before responding, in case you figure it out on your own. > > Regards, > Martin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dinov at microsoft.com Tue Jul 14 23:25:34 2009 From: dinov at microsoft.com (Dino Viehland) Date: Tue, 14 Jul 2009 21:25:34 +0000 Subject: [Python-Dev] 2.6 object.__init__ & deling __new__ In-Reply-To: <1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com> References: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com> <1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com> Message-ID: <1A472770E042064698CB5ADC83A12ACD03471396@TK5EX14MBXC116.redmond.corp.microsoft.com> Benjamin wrote: > There's a wonderful comment about this in typeobject.c: > This is basically the same what I've gathered from the issue description which was quite helpful. But in this case we're dealing with mutating the type object and changing whether __new__ or __init__ exist at all at runtime - and object.__new__/__init__ don't seem to be picking up on the change. I believe that the comments here w.r.t. mutability/immutability are more about whether the instances are immutable and not the type objects themselves - for example list vs tuple where tuple has __new__ but no __init__. Based upon the behavior I'm seeing it seems to me that the presence of __new__ / __init__ must be getting cached somewhere and the deletion isn't updating the cache and that's specifically what struck me as odd here. From pje at telecommunity.com Wed Jul 15 06:25:02 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Jul 2009 00:25:02 -0400 Subject: [Python-Dev] 2.6 object.__init__ & deling __new__ In-Reply-To: <1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.co m> References: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com> <1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com> Message-ID: <20090715042200.946A13A410E@sparrow.telecommunity.com> At 04:11 PM 7/14/2009 -0500, Benjamin Peterson wrote: >4. When __init__() is overridden, and the subclass __init__() >calls object.__init__(), the latter should complain about >excess arguments; ditto for __new__(). Actually, this rule is a PITA, because there's no good way to get rid of the warnings when you're trying to write MRO-agnostic mixins. In other words, it negates many of the gains that were obtained by having new-style MROs, since you can no-longer write pass-through constructors that leave their arguments untouched. Instead, every class must know ahead of time whether it will be the "last" class before 'object' -- thereby making it impossible to slip other mixins into the chain below them. In effect, 2.6 forces you to have a common known base class *other* than 'object' in order to write co-operative classes. :-( From brett at python.org Wed Jul 15 06:26:36 2009 From: brett at python.org (Brett Cannon) Date: Tue, 14 Jul 2009 21:26:36 -0700 Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy? Message-ID: I implemented get_filename() as specified in PEP 302 for importlib's source and bytecode loaders and I was starting to create the ABC for importlib.abc, but then I realized that perhaps the loader should live in runpy instead of importlib. Putting the new ABC in importlib keeps all PEP 302 ABCs in a single spot. But this addition to the PEP 302 protocols is very specific to runpy and is not directly required to make imports work (the implementation was just a reshuffling of pre-existing code from one method to another). I am tempted to leave the ABC out of importlib and stop at implementing get_filename() for importlib.abc.PyLoader and PyPycLoader. Any opinions? -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Jul 15 09:29:27 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 15 Jul 2009 19:29:27 +1200 Subject: [Python-Dev] 2.6 object.__init__ & deling __new__ In-Reply-To: <20090715042200.946A13A410E@sparrow.telecommunity.com> References: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com> <1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com> <20090715042200.946A13A410E@sparrow.telecommunity.com> Message-ID: <4A5D8557.90703@canterbury.ac.nz> P.J. Eby wrote: > In effect, 2.6 forces you to have a common known base class *other* than > 'object' in order to write co-operative classes. :-( You have to do that anyway if you want to make cooperative calls to any method *other* that __init__. -- Greg From ziade.tarek at gmail.com Wed Jul 15 11:01:24 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 15 Jul 2009 11:01:24 +0200 Subject: [Python-Dev] PEP 376 - from PyPM's point of view In-Reply-To: References: Message-ID: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> On Tue, Jul 14, 2009 at 2:12 AM, Sridhar Ratnakumar wrote: > Here are my comments regarding PEP 376 with respect to PyPM (the Python > package manager being developd at ActiveState) > > > Multiple versions: I understand that the PEP does not support > installation (thus uninstallation) of multiple versions of the same > package. Should this be explicitly mentioned in the PEP -- as > `get_distribution` API accepts only `name` argument, and not a `version` > argument? That's another can of worms ;) Before I answer here's a bit of background, i's a bit long but required, sorry For people that don't want to read the rest, here's the idea : multiple version support imho should be introduced later, if they are to be introduced, by extending PEP 302 protocol. The long explanantion now: given a "foo" package, containing a "bar" module, multiple versions support implies to do one of these: 1 - a custom PEP 302-like loader/importer that picks a version of "foo" when the code imports the "bar" module. this works if the "foo" package is not directly available in sys.path, and if the custom loader/importer is put in sys.meta_path for example. If "foo 0.9" is located in /var/packages/foo/0.9 and if "foo 1.0" is in /var/packages/foo/1.0, The loader will select the right foo package to load and return with through a loader that scans /var/packages/foo/* To make it work it requires 2 things : a/ a version comparison system (see PEP 386) that will make the loader pick the "latest" version by default b/ an API that will force the loader to pick one particular version 2 - changing the paths in sys.path to include the path containing the right version, and let the existing importer/loader do the work. That's what setuptools does with its multiple version system: an API called "require" will let you change sys.path on the fly >>> from pkg_resources import require >>> require('docutils==0.4') <--- looks for a docutils egg distribution and adds it in the path So if we support multiple versions in Python, (I'd love too). PEP 376 would need to be able to find the various versions of each distribution, not by scanning sys.path but rather by scanning a arbitrary collection of directories, then publishing the right ones in sys.path (with a PEP 302 loader, or ala setuptools by inserting them in sys.path) In other words this would require changing the way the distributions are stored. e.g. in self-contained eggs or in a brand-new storage tree. (I am currently experimenting this with "virtual site-packages" see http://bitbucket.org/tarek/vsp/src/tip/README.txt) But as we said earlier, people might want to store their modules anywhere (on a sql database, on Mars, etc.) and provide a PEP 302-like loader for them. PJE has "eggs" but John Doe might want to store its packages differently and provide an importer/loader for them. So each one of them provides a "package manager", which should composed of : A- a loader/importer system B- an installation system (that is easy_install -m for setuptools) C- query APIs D- a version comparison system E- an uninstaller So the real solution is to work with PEP 302 importers/loaders (A) (e.g. "package managers") Which means that PEP 302 need to be changed to become 'distribution-aware' as Paul said. Because we would then be able to browse distributions (C) that are not already loaded in sys.path, so work on two versions of the same distribution. but some open questions remains: It also implies that each package manager provides its installer (B) and a version comparison system (D) I'm not sure about the way package installers could be declared. Plus, how people would deal with several installers ? For the version comparison system I am not sure either, but it would require to have one global version comparison system to rule them all otherwise some conflicts may occur. So there's no plan to support multiple versions yet, because that requires another PEP imho. Since distutils is a package manager in some ways (it provides an installer, and stores distributions that are made available in sys.path) My feeling is that we need first to finish what's missing to make it fully usable (eg query apis + uninstaller) Then maybe think about extending PEP 302 > >> get_distribution(name) -> Distribution or None. >> Scans all elements in sys.path and looks for all directories ending >> with .egg-info. Returns a Distribution corresponding to the .egg-info >> directory that contains a PKG-INFO that matches name for the name >> metadata. >> Notice that there should be at most one result. The first result >> founded is returned. If the directory is not found, returns None. > > Some packages have package names with mixed case. Example: ConfigObj > .. as registered in setup.py. However, other packages such as turbogears > specifies "configobj" (lowercase) in their install_requires. > > Is `get_distribution(name)` supposed to handle mixed cases? Will it > match both 'ConfigObj' and 'configobj'? As PJE said, we need normalization here yes. Right now PyPI is case insensitive for its index: http://pypi.python.org/simple/ConfigObj == http://pypi.python.org/simple/configobj But in the meantime, IIRC, the XML-RPC apis are case sensitive, and so the html browsing. easy_install is case insensitive though, because it uses the index. So we should be case-insensitive everywhere, so in PEP 376 too. > >> get_installed_files(local=False) -> iterator of (path, md5, size) > > Will this also return the directories /created/ during the installation? > For example, will it also contain the entry "docutils" .. along with > "docutils/__init__.py"? I don't think it's necessary to add "docutils" if "docutils/__init__.py" is present But for empty directories added during installation we should add the I guess. So, I'll add a note. > > If not, how is the installer (pip, pypm, etc..) supposed to know which > directories to remove (docutils/) and which directories not to remove > (site-packages/, bin/, etc..)? > >> The new version of PEP 345 (XXX work in progress) extends the Metadata >> standard and fullfills the requirements described in PEP 262, like the >> REQUIRES section. > > Can you tell more about this? > > I see that PEP 262 allows both distributions names ('docutils') and > modules/packages ('roman.py') in the 'Requires:' section. Is this how > the new PEP is going to adhere to? Or, is it going to adhere to PEP > 345's way of allowing *only* modules/packages? The plan is to add what setuptools called "installed_requires", so you can tell which *distributions* should be installed, no matter if they are composed of a single module, or many packages. > > In practice, I noticed that packages usually specify distribution names > in their 'Requires:' file (or install_requires.txt in the case of > setuptools). Hence, PyPM *assumes* the install requirements to be > distribution name. But then .. most distributions have the same name as > their primary module/package. That's it yes: it will be distribution aware. If a module or package has the same name than the distribution name, it will make no difference. > > Ok, so PEP 345 also specifies the 'Provides:' header. Does > easy_install/pip make use 'Provides:' at all when resolving > dependencies? For example, does 'pip install sphinx' go look for all > distributions that 'provides' the 'docutils' provision.. or does it > simply get the distribution named 'docutils'? setuptools doesn't. I don't think pip does. btw: is PyPM a public project ? Regards Tarek -- Tarek Ziad? | http://ziade.org From p.f.moore at gmail.com Wed Jul 15 12:10:13 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 Jul 2009 11:10:13 +0100 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> Message-ID: <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> 2009/7/15 Tarek Ziad? : > On Tue, Jul 14, 2009 at 2:12 AM, Sridhar > Ratnakumar wrote: >> Here are my comments regarding PEP 376 with respect to PyPM (the Python >> package manager being developd at ActiveState) >> >> >> Multiple versions: I understand that the PEP does not support >> installation (thus uninstallation) of multiple versions of the same >> package. Should this be explicitly mentioned in the PEP -- as >> `get_distribution` API accepts only `name` argument, and not a `version` >> argument? > > That's another can of worms ;) :-) > Before I answer here's a bit of background, i's a bit long but required, sorry > > For people that don't want to read the rest, here's the idea : > multiple version support imho should > be introduced later, if they are to be introduced, by extending PEP > 302 protocol. Disclaimer: I've only read the short version, so if some of this is covered in the longer explanation, I apologise now. -1. In my view, multiple version support is not at all related to PEP 302 - or to core Python in general. The import statement has no concept of versions, any version handling is done by explicit user manipulation of sys.path. PEP 302 is currently purely an import protocol. As such, it only cares about locating the correct code to run to populate sys.modules['foo']. Once the code has been located, there are a number of other details that might be useful, hence the extensions like get_data, get_filename, etc. But note that these are all *loader* extensions - their starting point is an imported module. The PEP 376 support I've just added is a *finder* extension, which is working alongside the scanning of the container - but rather than looking for modules, it's looking for distributions. Disappointingly (for me) it turned out not to give much opportunity to share code - the finder extensions could just as easily have been a completely independent protocol. PEP 376 support has added a requirement for 3 additional methods to the existing 1 finder method in PEP 302. That's already a 300% increase in complexity. I'm against adding any further complexity to PEP 302 - in all honesty, I'd rather move towards PEP 376 defining its *own* locator protocol and avoid any extra burden on PEP 302. I'm not sure implementers of PEP 302 importers will even provide the current PEP 376 extensions. I propose that before the current prototype is turned into a final (spec and) implementation, the PEP 302 extensions are extracted and documented as an independent protocol, purely part of PEP 376. (This *helps* implementers, as they can write support for, for example, eggs, without needing to modify the existing egg importer). I'll volunteer to do that work - but I won't start until the latest iteration of questions and discussions has settled down and PEP 376 has achieved a stable form with the known issues addressed. Of course, this is moving more and more towards saying that the design of setuptools, with its generic means for locating distributions, etc etc, is the right approach. We're reinventing the wheel here. But the problem is that too many people dislike setuptools as it stands for it to gain support. My understanding is that the current set of PEPs were intended to be a stripped down, more generally acceptable subset of setuptools. Let's keep them that way (and omit the complexities of multi-version support). If you want setuptools, you know where to get it... Paul. From fuzzyman at voidspace.org.uk Wed Jul 15 12:17:38 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 15 Jul 2009 11:17:38 +0100 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> Message-ID: <4A5DACC2.5030609@voidspace.org.uk> Paul Moore wrote: > 2009/7/15 Tarek Ziad? : > >> On Tue, Jul 14, 2009 at 2:12 AM, Sridhar >> Ratnakumar wrote: >> >>> Here are my comments regarding PEP 376 with respect to PyPM (the Python >>> package manager being developd at ActiveState) >>> >>> >>> Multiple versions: I understand that the PEP does not support >>> installation (thus uninstallation) of multiple versions of the same >>> package. Should this be explicitly mentioned in the PEP -- as >>> `get_distribution` API accepts only `name` argument, and not a `version` >>> argument? >>> >> That's another can of worms ;) >> > > :-) > > >> Before I answer here's a bit of background, i's a bit long but required, sorry >> >> For people that don't want to read the rest, here's the idea : >> multiple version support imho should >> be introduced later, if they are to be introduced, by extending PEP >> 302 protocol. >> > > Disclaimer: I've only read the short version, so if some of this is > covered in the longer explanation, I apologise now. > > -1. > I agree. People with versioning issues *should* be using virtualenv. Michael Foord > In my view, multiple version support is not at all related to PEP 302 > - or to core Python in general. The import statement has no concept of > versions, any version handling is done by explicit user manipulation > of sys.path. > > PEP 302 is currently purely an import protocol. As such, it only cares > about locating the correct code to run to populate sys.modules['foo']. > Once the code has been located, there are a number of other details > that might be useful, hence the extensions like get_data, > get_filename, etc. But note that these are all *loader* extensions - > their starting point is an imported module. > > The PEP 376 support I've just added is a *finder* extension, which is > working alongside the scanning of the container - but rather than > looking for modules, it's looking for distributions. Disappointingly > (for me) it turned out not to give much opportunity to share code - > the finder extensions could just as easily have been a completely > independent protocol. > > PEP 376 support has added a requirement for 3 additional methods to > the existing 1 finder method in PEP 302. That's already a 300% > increase in complexity. I'm against adding any further complexity to > PEP 302 - in all honesty, I'd rather move towards PEP 376 defining its > *own* locator protocol and avoid any extra burden on PEP 302. I'm not > sure implementers of PEP 302 importers will even provide the current > PEP 376 extensions. > > I propose that before the current prototype is turned into a final > (spec and) implementation, the PEP 302 extensions are extracted and > documented as an independent protocol, purely part of PEP 376. (This > *helps* implementers, as they can write support for, for example, > eggs, without needing to modify the existing egg importer). I'll > volunteer to do that work - but I won't start until the latest > iteration of questions and discussions has settled down and PEP 376 > has achieved a stable form with the known issues addressed. > > Of course, this is moving more and more towards saying that the design > of setuptools, with its generic means for locating distributions, etc > etc, is the right approach. We're reinventing the wheel here. But the > problem is that too many people dislike setuptools as it stands for it > to gain support. My understanding is that the current set of PEPs were > intended to be a stripped down, more generally acceptable subset of > setuptools. Let's keep them that way (and omit the complexities of > multi-version support). > > If you want setuptools, you know where to get it... > > Paul. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ From solipsis at pitrou.net Wed Jul 15 12:27:58 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 15 Jul 2009 10:27:58 +0000 (UTC) Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy? References: Message-ID: Brett Cannon python.org> writes: > > > I implemented get_filename() as specified in PEP 302 for importlib's source and bytecode loaders and I was starting to create the ABC for importlib.abc, but then I realized that perhaps the loader should live in runpy instead of importlib. > > Putting the new ABC in importlib keeps all PEP 302 ABCs in a single spot. But this addition to the PEP 302 protocols is very specific to runpy and is not directly required to make imports work (the implementation was just a reshuffling of pre-existing code from one method to another). I am tempted to leave the ABC out of importlib and stop at implementing get_filename() for importlib.abc.PyLoader and PyPycLoader. I am not sure when this discussion started. Are you replying to a 3 month-old message of yours? :) In any case, keeping all import-related ABCs in a single place sounds like a good idea. Alternatively, if the method is runpy specific, it doesn't have its place in an ABC, does it? Antoine. From p.f.moore at gmail.com Wed Jul 15 14:39:31 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 Jul 2009 13:39:31 +0100 Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy? In-Reply-To: References: Message-ID: <79990c6b0907150539p33def8aid70f64704bcece6d@mail.gmail.com> 2009/7/15 Brett Cannon : > I implemented get_filename() as specified in PEP 302 for importlib's source > and bytecode loaders and I was starting to create the ABC for importlib.abc, > but then I realized that perhaps the loader should live in runpy instead of > importlib. > Putting the new ABC in importlib keeps all PEP 302 ABCs in a single spot. > But this addition to the PEP 302 protocols is very specific to runpy and is > not directly required to make imports work (the implementation was just a > reshuffling of pre-existing code from one method to another). I am tempted > to leave the ABC out of importlib and stop at implementing get_filename() > for importlib.abc.PyLoader and PyPycLoader. > Any opinions? importlib. In my view, all of the interfaces defined by PEP 302 should be available from one place. And importlib is that place. Paul. From ncoghlan at gmail.com Wed Jul 15 15:41:40 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Jul 2009 23:41:40 +1000 Subject: [Python-Dev] 2.6 object.__init__ & deling __new__ In-Reply-To: <1A472770E042064698CB5ADC83A12ACD03471396@TK5EX14MBXC116.redmond.corp.microsoft.com> References: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com> <1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com> <1A472770E042064698CB5ADC83A12ACD03471396@TK5EX14MBXC116.redmond.corp.microsoft.com> Message-ID: <4A5DDC94.60509@gmail.com> Dino Viehland wrote: > Based upon the behavior I'm seeing it seems to me that the > presence of __new__ / __init__ must be getting cached somewhere > and the deletion isn't updating the cache and that's specifically > what struck me as odd here. Digging through typeobject.c, it isn't clear to me why this wouldn't be getting picked up. If all is working correctly: 1. When the class is created, tp_new and tp_init are copied from the base class (see inherit_special and inherit_slots) 2. When object_new and object_init are called they compare the values stored in the tp_new and tp_init slots with their own function addresses to decide whether or not to raise the warning 3. When either of those is deleted, the update_slots call in type_setattro should fix up the slot inheritance, and hence affect future calls to __new__ and __init__ That said, the following comments in update_one_slot() are leading me to wonder if Guido might owe Dino a beer: { /* The __new__ wrapper is not a wrapper descriptor, so must be special-cased differently. If we don't do this, creating an instance will always use slot_tp_new which will look up __new__ in the MRO which will call tp_new_wrapper which will look through the base classes looking for a static base and call its tp_new (usually PyType_GenericNew), after performing various sanity checks and constructing a new argument list. Cut all that nonsense short -- this speeds up instance creation tremendously. */ specific = (void *)type->tp_new; /* XXX I'm not 100% sure that there isn't a hole in this reasoning that requires additional sanity checks. I'll buy the first person to point out a bug in this reasoning a beer. */ } I *think* this logic may be getting confused when update_one_slot() is called immediately after the "del x.__new__" call has stuffed a NULL into the tp_new slot. In the normal class creation case the tp_new method will have been copied down from the parent class so the conditions when update_one_slot() gets called are different (ditto for when you actually *set* x.__new__ rather than delete it). If nobody else gets anywhere with this I should have a chance to actually play with it (rather than just reading code) tomorrow evening. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Wed Jul 15 15:51:39 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 15 Jul 2009 23:51:39 +1000 Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy? In-Reply-To: References: Message-ID: <4A5DDEEB.4040700@gmail.com> Antoine Pitrou wrote: > I am not sure when this discussion started. Are you replying to a 3 month-old > message of yours? :) That depends on how you define the beginning of the discussion... However, the fact that importlib doesn't implement the comparatively recent get_filename() optional extension documented in PEP 302 came up in one of the PEP 376 threads within the last week or so. > In any case, keeping all import-related ABCs in a single place sounds like a > good idea. > Alternatively, if the method is runpy specific, it doesn't have its place in an > ABC, does it? While runpy is the only client in the standard library for the get_filename() method, the method is still a PEP 302 extension. I documented the extension in the PEP as loaders are the only things reliably in a position to provide the filename details that runpy needs to set __file__ and sys.argv[0] correctly and until importlib came along PEP 302 itself was the only real documentation of that API. Since importlib is now the "go-to" location for people that want to write their own PEP 302 importers and loaders, I would say that it is also the right place for the new ExecutionLoader ABC. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ziade.tarek at gmail.com Wed Jul 15 16:00:27 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 15 Jul 2009 16:00:27 +0200 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> Message-ID: <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> On Wed, Jul 15, 2009 at 12:10 PM, Paul Moore wrote: > > Disclaimer: I've only read the short version, so if some of this is > covered in the longer explanation, I apologise now. Next time I won't put a short version ;) > > PEP 376 support has added a requirement for 3 additional methods to > the existing 1 finder method in PEP 302. That's already a 300% > increase in complexity. I'm against adding any further complexity to > PEP 302 - in all honesty, I'd rather move towards PEP 376 defining its > *own* locator protocol and avoid any extra burden on PEP 302. I'm not > sure implementers of PEP 302 importers will even provide the current > PEP 376 extensions. > > I propose that before the current prototype is turned into a final > (spec and) implementation, the PEP 302 extensions are extracted and > documented as an independent protocol, purely part of PEP 376. (This > *helps* implementers, as they can write support for, for example, > eggs, without needing to modify the existing egg importer). I'll > volunteer to do that work - but I won't start until the latest > iteration of questions and discussions has settled down and PEP 376 > has achieved a stable form with the known issues addressed. Sure that makes sense. I am all for having these 302 extensions flipped on PEP 376 side, then think about the "locator" protocol. I am lagging a bit in the discussions, I have 10 messages left to read or so, but the known issues I've listed so far are about the RECORD file and absolute paths, I am waiting for PJE example on the syntax he proposed for prefixes, on the docutils example. > Of course, this is moving more and more towards saying that the design > of setuptools, with its generic means for locating distributions, etc > etc, is the right approach. > We're reinventing the wheel here. But the > problem is that too many people dislike setuptools as it stands for it > to gain support. I don't think it's about setuptools design. I think it's more likely to be about the fact that there's no way in Python to install two different versions of the same distribution without "hiding" one from each other, using setuptools, virtualenv or zc.buildout. "installing" a distribution in Python means that its activated globally, whereas people need it locally at the application level. > My understanding is that the current set of PEPs were > intended to be a stripped down, more generally acceptable subset of > setuptools. Let's keep them that way (and omit the complexities of > multi-version support). > > If you want setuptools, you know where to get it... Sure, but let's not forget that the multiple-version issue is a global issue OS packagers also meet. (setuptools is not the problem) : - application Foo uses docutils 0.4 and doesn't work with docutils 0.5 - application Bar uses docutils 0.5 if docutils 0.5 is installed, Foo is broken, unless docutils 0.4 is shipped with it. So right now application developers must use tools to isolate their dependencies from the rest of the system, using virtualenv or zc.buildout, and ship the dependencies with them. So they are sure that their applications are not broken by the system. This is not optimal of course, because you end up with several occurences of the same versions sometimes. And can be a nightmare for os packagers and maintainers. And virtualenv and such tools are now required when you develop applications (mid-size and large ones) and the "good practice" is to avoid any installation of any distributions in Python itself. So basically "site-packages" is a distribution location that is avoided by everyone because it doesn't know how to handle multiple versions. If we had a multi-versions support protocol, that would help os packagers and application developers to be friends again imho ;) Regards Tarek -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Wed Jul 15 16:03:08 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 15 Jul 2009 16:03:08 +0200 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <4A5DACC2.5030609@voidspace.org.uk> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <4A5DACC2.5030609@voidspace.org.uk> Message-ID: <94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com> On Wed, Jul 15, 2009 at 12:17 PM, Michael Foord wrote: >> Disclaimer: I've only read the short version, so if some of this is >> covered in the longer explanation, I apologise now. >> >> -1. >> > > I agree. People with versioning issues *should* be using virtualenv. > Let's remove site-packages from Python then. From peter.hanecak at alcatel-lucent.sk Wed Jul 15 15:46:59 2009 From: peter.hanecak at alcatel-lucent.sk (Peter Hanecak) Date: Wed, 15 Jul 2009 15:46:59 +0200 Subject: [Python-Dev] 64-bit values in XML RPC: OverflowError: int exceeds XML-RPC limits Message-ID: <4A5DDDD3.4070900@alcatel-lucent.sk> Hello, when I'm trying to use 64-bit integer values with SimpleXMLRPCServer, I'm getting "OverflowError: int exceeds XML-RPC limits" error each time I use an integer with value greater or equal to 2^31. I googled this: http://bugs.python.org/issue2985 So, my question is: In which Python release has been this fix distributed? Thank you in advance for information. Btw, I've made some dummy scripts to demonstrate the issue: - server.py: from SimpleXMLRPCServer import SimpleXMLRPCServer def dummy(number): return number server = SimpleXMLRPCServer(("localhost", 8000)) server.register_function(dummy, 'dummy') server.serve_forever() - client.py import xmlrpclib proxy = xmlrpclib.ServerProxy("http://localhost:8000/") print proxy.dummy(0x7FFFFFFF) print proxy.dummy(0x80000000) The output from client is: localhost.localdomain - - [15/Jul/2009 15:24:12] "POST / HTTP/1.0" 200 - 2147483647 Traceback (most recent call last): File "./client.py", line 7, in print proxy.dummy(0x80000000) File "/usr/lib64/python2.6/xmlrpclib.py", line 1199, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python2.6/xmlrpclib.py", line 1483, in __request allow_none=self.__allow_none) File "/usr/lib64/python2.6/xmlrpclib.py", line 1132, in dumps data = m.dumps(params) File "/usr/lib64/python2.6/xmlrpclib.py", line 677, in dumps dump(v, write) File "/usr/lib64/python2.6/xmlrpclib.py", line 699, in __dump f(self, value, write) File "/usr/lib64/python2.6/xmlrpclib.py", line 710, in dump_int raise OverflowError, "int exceeds XML-RPC limits" OverflowError: int exceeds XML-RPC limits Have a nice day. Peter p.s.: I'm not subscribed to the list, se please keep me in CC when replying. Thank you. -- Peter Hanecak ePC Developer | Alcatel-Lucent Apollo BC II - B block | Prievozska 4/A 11 | Bratislava | Slovak Republic email: peter.hanecak at alcatel-lucent.com phone: +421 (0)2 49264 857 From benjamin at python.org Wed Jul 15 16:29:02 2009 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 15 Jul 2009 09:29:02 -0500 Subject: [Python-Dev] 64-bit values in XML RPC: OverflowError: int exceeds XML-RPC limits In-Reply-To: <4A5DDDD3.4070900@alcatel-lucent.sk> References: <4A5DDDD3.4070900@alcatel-lucent.sk> Message-ID: <1afaf6160907150729l5afa9612l87f86ce636ff0a3d@mail.gmail.com> 2009/7/15 Peter Hanecak : > So, my question is: In which Python release has been this fix distributed? Python 2.6 and above. -- Regards, Benjamin From rdmurray at bitdance.com Wed Jul 15 16:39:44 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 15 Jul 2009 10:39:44 -0400 (EDT) Subject: [Python-Dev] 64-bit values in XML RPC: OverflowError: int exceeds XML-RPC limits In-Reply-To: <1afaf6160907150729l5afa9612l87f86ce636ff0a3d@mail.gmail.com> References: <4A5DDDD3.4070900@alcatel-lucent.sk> <1afaf6160907150729l5afa9612l87f86ce636ff0a3d@mail.gmail.com> Message-ID: On Wed, 15 Jul 2009 at 09:29, Benjamin Peterson wrote: > 2009/7/15 Peter Hanecak : >> So, my question is: In which Python release has been this fix distributed? > > Python 2.6 and above. But it doesn't solve your problem, since the ticket says it only fixes reading long ints, not writing them. --David From peter.hanecak at alcatel-lucent.sk Wed Jul 15 16:42:20 2009 From: peter.hanecak at alcatel-lucent.sk (Peter Hanecak) Date: Wed, 15 Jul 2009 16:42:20 +0200 Subject: [Python-Dev] 64-bit values in XML RPC: OverflowError: int exceeds XML-RPC limits In-Reply-To: References: <4A5DDDD3.4070900@alcatel-lucent.sk> <1afaf6160907150729l5afa9612l87f86ce636ff0a3d@mail.gmail.com> Message-ID: <4A5DEACC.5040502@alcatel-lucent.sk> Hello, thank you David and Benjamin for quick response. So my subsequent question is: What can help me solve the "writing" part? Sincerely Peter On 07/15/2009 04:39 PM, R. David Murray wrote: > On Wed, 15 Jul 2009 at 09:29, Benjamin Peterson wrote: >> 2009/7/15 Peter Hanecak : >>> So, my question is: In which Python release has been this fix >>> distributed? >> >> Python 2.6 and above. > > But it doesn't solve your problem, since the ticket says it only fixes > reading long ints, not writing them. > > --David -- Peter Hanecak ePC Developer | Alcatel-Lucent Apollo BC II - B block | Prievozska 4/A 11 | Bratislava | Slovak Republic email: peter.hanecak at alcatel-lucent.com phone: +421 (0)2 49264 857 From ziade.tarek at gmail.com Wed Jul 15 16:46:51 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 15 Jul 2009 16:46:51 +0200 Subject: [Python-Dev] 64-bit values in XML RPC: OverflowError: int exceeds XML-RPC limits In-Reply-To: <4A5DEACC.5040502@alcatel-lucent.sk> References: <4A5DDDD3.4070900@alcatel-lucent.sk> <1afaf6160907150729l5afa9612l87f86ce636ff0a3d@mail.gmail.com> <4A5DEACC.5040502@alcatel-lucent.sk> Message-ID: <94bdd2610907150746v7b16021dk82c6a8331f9072b6@mail.gmail.com> On Wed, Jul 15, 2009 at 4:42 PM, Peter Hanecak wrote: > Hello, > > thank you David and Benjamin for quick response. > > So my subsequent question is: What can help me solve the "writing" part? Use strings. Send str(0x7FFFFFFF) from the client for example , and get back your number on server side with a long(number) call. That's what I do. From dirkjan at ochtman.nl Wed Jul 15 16:46:59 2009 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 15 Jul 2009 16:46:59 +0200 Subject: [Python-Dev] 64-bit values in XML RPC: OverflowError: int exceeds XML-RPC limits In-Reply-To: <4A5DEACC.5040502@alcatel-lucent.sk> References: <4A5DDDD3.4070900@alcatel-lucent.sk> <1afaf6160907150729l5afa9612l87f86ce636ff0a3d@mail.gmail.com> <4A5DEACC.5040502@alcatel-lucent.sk> Message-ID: On Wed, Jul 15, 2009 at 16:42, Peter Hanecak wrote: > So my subsequent question is: What can help me solve the "writing" part? The XML-RPC protocol, as specified at [1], doesn't support integers with more than 32 bits (in fact, the i4 alias can be used to make the use of 4 bytes explicit). So, either don't use XML-RPC, or encode it in some other type (e.g. a string or base64). Cheers, Dirkjan [1] http://www.xmlrpc.com/spec From p.f.moore at gmail.com Wed Jul 15 17:14:30 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 Jul 2009 16:14:30 +0100 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <4A5DACC2.5030609@voidspace.org.uk> <94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com> Message-ID: <79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com> 2009/7/15 Tarek Ziad? : > On Wed, Jul 15, 2009 at 12:17 PM, Michael > Foord wrote: >>> Disclaimer: I've only read the short version, so if some of this is >>> covered in the longer explanation, I apologise now. >>> >>> -1. >>> >> >> I agree. People with versioning issues *should* be using virtualenv. >> > > Let's remove site-packages from Python then. If virtualenv/py2exe/cx_Freeze/py2app don't offer a solution, then maybe you're right. For me, py2exe (and a clean virtual machine if I require an absolutely pristine environment, I guess virtualenv would help here too) does what I need for application packaging. But I'll freely admit that my needs are minimal. Bluntly, as Python stands, import and sys.path do not offer any core support for multiple versions. Custom solutions can be built on top of that - that's what setuptools does. But they are precisely that - custom solutions, and should be supported as such, outside the core (and stdlib). If standard Python support for multi-version imports is required (it's not by me, but I accept that some people want it), then it should be designed in thoughout: - how do I import version 0.7.1 of package foo, rather than 0.7.2? - how do I use foo 0.8 at my interactive prompt, and import bar 0.2 which relies on foo 0.7.1? - what happens if I import foo 2.0 (which requires baz 0.1) and bar 1.5 (which requires baz 0.2)? - what does "import foo" (without a version number) mean? Is it different if it's at the command line or in bar 0.5 (which explicitly declares a dependency on foo 0.7 in its setup.py)? Does the answer to that mean that imports always need to read dependency information? - does your head hurt yet? I could go on... Any PEP dealing with multi versions should address these issues. It's a big area, and I have no interest in it myself, but I do have an interest in avoiding partial solutions which only look at some of the questions that might arise. Look - I really, really don't mind if people use setuptools. Honest. But I do mind if core python gets changed to support little bits of what setuptools does, adding complexity to deal with issues that setuptools handles, but without making it possible to avoid using setuptools. Where's the benefit to anyone then? Paul. From p.f.moore at gmail.com Wed Jul 15 17:21:12 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 Jul 2009 16:21:12 +0100 Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy? In-Reply-To: <4A5DDEEB.4040700@gmail.com> References: <4A5DDEEB.4040700@gmail.com> Message-ID: <79990c6b0907150821r6488759dr4cd49c382f2daf79@mail.gmail.com> 2009/7/15 Nick Coghlan : > While runpy is the only client in the standard library for the > get_filename() method, the method is still a PEP 302 extension. I > documented the extension in the PEP as loaders are the only things > reliably in a position to provide the filename details that runpy needs > to set __file__ and sys.argv[0] correctly and until importlib came along > PEP 302 itself was the only real documentation of that API. Yes, sorry about that :-) > Since importlib is now the "go-to" location for people that want to > write their own PEP 302 importers and loaders, I would say that it is > also the right place for the new ExecutionLoader ABC. Agreed. (But I said that before...) Paul. From him at online.de Wed Jul 15 17:16:41 2009 From: him at online.de (=?ISO-8859-1?Q?Joachim_K=F6nig?=) Date: Wed, 15 Jul 2009 17:16:41 +0200 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> Message-ID: <4A5DF2D9.6050403@online.de> Tarek Ziad? wrote: > So basically "site-packages" is a distribution location that is > avoided by everyone because it doesn't > know how to handle multiple versions. I think you overrate the importance of having multiple versions of a package available for the same python interpreter. If you have m different versions of n packages then you could have n**m different combinations for an application so you need a possiblilty to select one combination from n**m possible ones at application startup time. Is this really worth it? > If we had a multi-versions > support protocol, that would > help os packagers and application developers to be friends again imho ;) > > Let's remove site-packages from Python then. The _one_ site-packages folder stands for _one_ python interpreter. All the clever efforts to provide a set of package versions at runtime to an application (that uses the singleton python interpreter) do logically create a new python interpreter with a site-packages folder that contains just the versions of the packages the application needs, unfortunately by mucking with PYTHONPATH and .pth, site.py etc making it very difficult to understand what is happening for the joe average python developer. From ziade.tarek at gmail.com Wed Jul 15 17:51:13 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 15 Jul 2009 17:51:13 +0200 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <4A5DACC2.5030609@voidspace.org.uk> <94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com> <79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com> Message-ID: <94bdd2610907150851u1c022d10i5bc6b6d9b581bc17@mail.gmail.com> On Wed, Jul 15, 2009 at 5:14 PM, Paul Moore wrote: > 2009/7/15 Tarek Ziad? : >> On Wed, Jul 15, 2009 at 12:17 PM, Michael >> Foord wrote: >>>> Disclaimer: I've only read the short version, so if some of this is >>>> covered in the longer explanation, I apologise now. >>>> >>>> -1. >>>> >>> >>> I agree. People with versioning issues *should* be using virtualenv. >>> >> >> Let's remove site-packages from Python then. > > If virtualenv/py2exe/cx_Freeze/py2app don't offer a solution, then > maybe you're right. They do offer a solution, but these solution are hard to maintain from a OS packager point of view. In any case I don't see any use case to have a "site-packages" remaining in Python itself. > If standard Python support for multi-version imports is required (it's > not by me, but I accept that some people want it), then it should be > designed in thoughout: > [..] > > Any PEP dealing with multi versions should address these issues. It's > a big area, and I have no interest in it myself, but I do have an > interest in avoiding partial solutions which only look at some of the > questions that might arise. > > Look - I really, really don't mind if people use setuptools. Honest. > But I do mind if core python gets changed to support little bits of > what setuptools does, adding complexity to deal with issues that > setuptools handles, but without making it possible to avoid using > setuptools. Where's the benefit to anyone then? I totally agree. But I think that the boundary between what Python+stdlib should provide feature-wise and what third party packages provides is still fuzzy and should be made clearer. At some point, third party projects are trying hard to isolate dependencies because they can't do it with the execution environment Python sets by default when you launch a script or an interpreter. site.py loads site-packages and user site-packages at startup basically, and you can use PYTHONPATH / sys.path to add more, but that's partially shared by all apps. To address this issued, a project like zc.buildout will create a local repository of distributions, and will change sys.path on the fly so the local repository is used. virtualenv on its side creates an isolated Python installation for your application, So If the stdlib doesn't provide a standard protocol that goes further than that, people that have this need will continue to use zc.buildout or virtualenv and install many times the same libs on a system. It's perfectly fine from an application developer PoV, but it make site-packages obsoletes and every application installed that way looks like a black box for os packagers. They don't want that. At the end, application developers have to care about the way their applications dependencies are installed, were they should just give the list of those dependencies, and let any package manager project install them. If these matters are not adressed by Python sdtlib, then we should remove the loading of site-packages at Python startup completely, and make it cristal clear that it's not the core/stdlib problem. Otherwise we should think hard about how to let os packagers manage a single repository of Python distributions (and multiple versions) and how application developers could use them from within their applications. This protocol imho could be in the sdtlib even if the implementation is outside, but that's just me. In any case, these issues can be postponed after PEP 376, because that's a another (bigger) part of the puzzle. Regards Tarek From cournape at gmail.com Wed Jul 15 17:22:03 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 16 Jul 2009 00:22:03 +0900 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> Message-ID: <5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com> On Wed, Jul 15, 2009 at 11:00 PM, Tarek Ziad? wrote: > On Wed, Jul 15, 2009 at 12:10 PM, Paul Moore wrote: >> >> Disclaimer: I've only read the short version, so if some of this is >> covered in the longer explanation, I apologise now. > > Next time I won't put a short version ;) > > >> >> PEP 376 support has added a requirement for 3 additional methods to >> the existing 1 finder method in PEP 302. That's already a 300% >> increase in complexity. I'm against adding any further complexity to >> PEP 302 - in all honesty, I'd rather move towards PEP 376 defining its >> *own* locator protocol and avoid any extra burden on PEP 302. I'm not >> sure implementers of PEP 302 importers will even provide the current >> PEP 376 extensions. >> >> I propose that before the current prototype is turned into a final >> (spec and) implementation, the PEP 302 extensions are extracted and >> documented as an independent protocol, purely part of PEP 376. (This >> *helps* implementers, as they can write support for, for example, >> eggs, without needing to modify the existing egg importer). I'll >> volunteer to do that work - but I won't start until the latest >> iteration of questions and discussions has settled down and PEP 376 >> has achieved a stable form with the known issues addressed. > > Sure that makes sense. I am all for having these 302 extensions > flipped on PEP 376 > side, then think about the "locator" protocol. > > I am lagging a bit in the discussions, I have 10 messages left to read or so, > but the known issues I've listed so far are about the RECORD file and > absolute paths, > I am waiting for PJE example on the syntax he proposed for prefixes, > on the docutils example. > >> Of course, this is moving more and more towards saying that the design >> of setuptools, with its generic means for locating distributions, etc >> etc, is the right approach. >> We're reinventing the wheel here. But the >> problem is that too many people dislike setuptools as it stands for it >> to gain support. > > I don't think it's about setuptools design. I think it's more likely > to be about the fact > that there's no way in Python to install two different versions of the > same distribution > without "hiding" one from each other, using setuptools, virtualenv or > zc.buildout. > > "installing" a distribution in Python means that its activated > globally, whereas people > need it locally at the application level. > >> My understanding is that the current set of PEPs were >> intended to be a stripped down, more generally acceptable subset of >> setuptools. Let's keep them that way (and omit the complexities of >> multi-version support). >> >> If you want setuptools, you know where to get it... > > Sure, but let's not forget that the multiple-version issue is a global > issue OS packagers > also meet. (setuptools is not the problem) : > > - application Foo uses docutils 0.4 and doesn't work with docutils 0.5 > - application Bar uses docutils 0.5 > > if docutils 0.5 is installed, Foo is broken, unless docutils 0.4 is > shipped with it. As was stated by Debian packagers on the distutils ML, the problem is that docutils 0.5 breaks packages which work with docutils 0.4 in the first place. http://www.mail-archive.com/distutils-sig at python.org/msg05775.html And current hacks to work around lack of explicit version handling for module import is a maintenance burden: http://www.mail-archive.com/distutils-sig at python.org/msg05742.html setuptools has given the incentive to use versioning as a workaround for API/ABI compatibility. That's the core problem, and most problems brought by setuptools (sys.path and .pth hacks with the unreliability which ensued) are consequences of this. I don't see how virtualenv solves anything in that regard for deployment issues. I doubt using things like virtualenv will make OS packagers happy. David From pje at telecommunity.com Wed Jul 15 17:58:42 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Jul 2009 11:58:42 -0400 Subject: [Python-Dev] 2.6 object.__init__ & deling __new__ In-Reply-To: <4A5D8557.90703@canterbury.ac.nz> References: <1A472770E042064698CB5ADC83A12ACD03470DD9@TK5EX14MBXC116.redmond.corp.microsoft.com> <1afaf6160907141411o54670699x2cbbe0b6426b25e0@mail.gmail.com> <20090715042200.946A13A410E@sparrow.telecommunity.com> <4A5D8557.90703@canterbury.ac.nz> Message-ID: <20090715155538.A77C03A4099@sparrow.telecommunity.com> At 07:29 PM 7/15/2009 +1200, Greg Ewing wrote: >P.J. Eby wrote: > >>In effect, 2.6 forces you to have a common known base class *other* >>than 'object' in order to write co-operative classes. :-( > >You have to do that anyway if you want to make cooperative >calls to any method *other* that __init__. I haven't found that to be an issue, actually, since there's usually a base where the "bottom" version of those methods live, and anything implementing those methods is going to inherit it via the mixin. The problem is that for at least __init__ and __new__, 'object' *is* the "bottom" -- and there may be more than one family of mixins that need to call it. From p.f.moore at gmail.com Wed Jul 15 17:59:56 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 Jul 2009 16:59:56 +0100 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> <5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com> Message-ID: <79990c6b0907150859h6d7ad65g67e179c31aaa81f9@mail.gmail.com> 2009/7/15 David Cournapeau : > As was stated by Debian packagers on the distutils ML, the problem is > that docutils 0.5 breaks packages which work with docutils 0.4 in the > first place. > > http://www.mail-archive.com/distutils-sig at python.org/msg05775.html > > And current hacks to work around lack of explicit version handling for > module import is a maintenance burden: > > http://www.mail-archive.com/distutils-sig at python.org/msg05742.html > > setuptools has given the incentive to use versioning as a workaround > for API/ABI compatibility. That's the core problem, and most problems > brought by setuptools (sys.path and .pth hacks with the unreliability > which ensued) are consequences of this. I don't see how virtualenv > solves anything in that regard for deployment issues. I doubt using > things like virtualenv will make OS packagers happy. So, I think what you're saying is: - The real issues is packages not maintaining backward compatibility (I agree) - Setuptools is a workaround (I agree, at least it's *one* workaround) - Virtualenv isn't a workaround (I don't know virtualenv, I'll take your word for it) Three points: - When building *applications*, bundling everything (py2exe-style) is an alternative workaround - universal on Windows, apparently uncommon on Unix/Linux. - When managing multiple packages in a "toolkit" style interactive Python installation, I'm not aware of a good solution (other than avoiding core that hits this issue in the first place). - I do not believe that it's clear that sanctioning the setuptools workaround as the "right" approach by building it into the Python core/stdlib is the right thing to do. Paul. From ziade.tarek at gmail.com Wed Jul 15 18:00:48 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 15 Jul 2009 18:00:48 +0200 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <4A5DF2D9.6050403@online.de> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> <4A5DF2D9.6050403@online.de> Message-ID: <94bdd2610907150900j62a249fwcf88a188d71c5a51@mail.gmail.com> On Wed, Jul 15, 2009 at 5:16 PM, Joachim K?nig wrote: > Tarek Ziad? wrote: >> >> So basically "site-packages" is a distribution location that is >> avoided by everyone because it doesn't >> know how to handle multiple versions. > > I think you overrate the importance of having multiple versions of a package > available > for the same python interpreter. If you have m different versions of n > packages then > you could have n**m different combinations for an application so you need a > possiblilty to select one combination from n**m possible ones at application > startup time. Is this really worth it? When you create an application that uses several libs, and when you create your distribution, you end up pinning a version for each one of your dependency to avoid any problems. While it's workable in a small application to list the dependencies in a text file, and let your users install them manually, it's impossible in bigger applications. Applications based on Zope for example have **hundreds** of dependencies you need to have installed, in very specific versions. If you install a Zope 2 app on your system, and a Zope 3 one, you have good chances of breaking them because the "zope.schema" distribution is incompatible. So either each application ships its own collection of dependencies and ignore site-packages (that's what zope-based applications does with zc.buildout), either they have a way to pick the right version of each package. > >> If we had a multi-versions >> support protocol, that would >> help os packagers and application developers to be friends again imho ;) >> ?Let's remove site-packages from Python then. > > The _one_ site-packages folder stands for _one_ python interpreter. All the > clever > efforts to provide a set of package versions at runtime to an application > (that uses > the singleton python interpreter) do logically create a new python > interpreter with > a site-packages folder that contains just the versions of the packages the > application > needs, unfortunately by mucking with PYTHONPATH and .pth, site.py > etc > making it very difficult to understand what is happening for the joe average > python developer. That's what we have nowadays. python packages installed in different places, and scripts tweaking the path to launch an application with them. From pje at telecommunity.com Wed Jul 15 18:21:32 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Jul 2009 12:21:32 -0400 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.co m> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> Message-ID: <20090715161829.E45BE3A4116@sparrow.telecommunity.com> At 11:10 AM 7/15/2009 +0100, Paul Moore wrote: >I propose that before the current prototype is turned into a final >(spec and) implementation, the PEP 302 extensions are extracted and >documented as an independent protocol, purely part of PEP 376. (This >*helps* implementers, as they can write support for, for example, >eggs, without needing to modify the existing egg importer). Btw, this is why setuptools chose to implement these things as adapters or generic functions in pkg_resources (and in the vestigial bits that were added to python 2.5's pkgutil). So as you can see, trying to solve these particular problems tends to lead to reinventing setuptools or at least portions thereof. ;-) >But the >problem is that too many people dislike setuptools as it stands for it >to gain support. My understanding is that the current set of PEPs were >intended to be a stripped down, more generally acceptable subset of >setuptools. Let's keep them that way (and omit the complexities of >multi-version support). Even without multi-version support, the parts of PEP 376 that aren't about uninstallation are still reinventing chunks of pkg_resources. Had pkg_resources been in the stdlib a couple years back (note that bugs and changes in it are still quite rare), the PEP 376 bits for pkgutil could have been focused strictly on uninstallation, and just reused pkg_resources API for finding distributions, reading metadata, getting version info, etc. etc. All that stuff is extremely stable code, very widely used for a very long time. If politics is the only thing keeping it from being used, then that's a pretty sad statement about the community, given that pkg_resources is not to blame for 99% of what people complain about in relation to setuptools. All pkg_resources does is find stuff (on request) and maybe add it to sys.path (on request), pull out data and metadata from distributions (on request), or set up namespace packages (on request). So if politics demands that it be rejected by association with "setuptools", then just search-and-replace the API, PEP 8-ify it, call it something different, and lie to everyone about where it came from. I won't tell if you won't. ;-) From pje at telecommunity.com Wed Jul 15 18:27:11 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Jul 2009 12:27:11 -0400 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.co m> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <4A5DACC2.5030609@voidspace.org.uk> <94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com> <79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com> Message-ID: <20090715162407.AD8723A4116@sparrow.telecommunity.com> At 04:14 PM 7/15/2009 +0100, Paul Moore wrote: >Look - I really, really don't mind if people use setuptools. Honest. >But I do mind if core python gets changed to support little bits of >what setuptools does, adding complexity to deal with issues that >setuptools handles, but without making it possible to avoid using >setuptools. I don't see how that's being proposed. Nothing in PEP 376 requires setuptools or any portion thereof, AFAICT. From pje at telecommunity.com Wed Jul 15 18:30:39 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Jul 2009 12:30:39 -0400 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <4A5DF2D9.6050403@online.de> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> <4A5DF2D9.6050403@online.de> Message-ID: <20090715162736.B54D33A4116@sparrow.telecommunity.com> At 05:16 PM 7/15/2009 +0200, Joachim K?nig wrote: >f you have m different versions of n packages then >you could have n**m different combinations for an application so you need a >possiblilty to select one combination from n**m possible ones at application >startup time. Is this really worth it? Obviously yes, as neither buildout nor setuptools would exist otherwise. ;-) Nor would Fedora be packaging certain library versions as eggs specifically to get certain multi-version scenarios to work. The specific solutions for handling n*m problems aren't fantastic, but they are clearly needed. (Buildout, btw, actually hardwires the n*m choice at install time, which is probably better for production situations than setuptools' dynamic approach.) From pje at telecommunity.com Wed Jul 15 18:34:00 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Jul 2009 12:34:00 -0400 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <79990c6b0907150859h6d7ad65g67e179c31aaa81f9@mail.gmail.com > References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> <5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com> <79990c6b0907150859h6d7ad65g67e179c31aaa81f9@mail.gmail.com> Message-ID: <20090715163056.B34B63A4116@sparrow.telecommunity.com> At 04:59 PM 7/15/2009 +0100, Paul Moore wrote: >- Virtualenv isn't a workaround (I don't know virtualenv, I'll take >your word for it) It's not one for system package maintainers because it would effectively be managing multiple instances of 'python'. Really not a suitable solution. >- I do not believe that it's clear that sanctioning the setuptools >workaround as the "right" approach by building it into the Python >core/stdlib is the right thing to do. I still don't understand how we're doing that. From rdmurray at bitdance.com Wed Jul 15 18:49:57 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 15 Jul 2009 12:49:57 -0400 (EDT) Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <4A5DACC2.5030609@voidspace.org.uk> <94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com> <79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com> Message-ID: On Wed, 15 Jul 2009 at 16:14, Paul Moore wrote: > Bluntly, as Python stands, import and sys.path do not offer any core > support for multiple versions. Custom solutions can be built on top of > that - that's what setuptools does. But they are precisely that - > custom solutions, and should be supported as such, outside the core > (and stdlib). > > If standard Python support for multi-version imports is required (it's > not by me, but I accept that some people want it), then it should be > designed in thoughout: Isn't this problem directly analogous to the '.so version' (*) problem? Can we learn anything from the solution to that problem? Does not the fact that a (standard) solution to that problem was required indicate that a similar solution needs to be provided by core Python? (And yes, it's out of scope for PEP 376). --David (*) or, for our Windows users, DLL Hell and the Side By Side Component Sharing solution... From ziade.tarek at gmail.com Wed Jul 15 19:07:34 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 15 Jul 2009 19:07:34 +0200 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <20090715161829.E45BE3A4116@sparrow.telecommunity.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <20090715161829.E45BE3A4116@sparrow.telecommunity.com> Message-ID: <94bdd2610907151007p7cd5d6bof32612bec2908e43@mail.gmail.com> 2009/7/15 P.J. Eby : [...] > So if politics demands that it be rejected by association with "setuptools", > then just search-and-replace the API, PEP 8-ify it, call it something > different, and lie to everyone about where it came from. ?I won't tell if > you won't. ?;-) While setuptools did address some issues, we are now adressing in PEP 376, it does a *lot* of other things and it's impossible to extract the features we would want to re-use. e.g. what I would call the good bits. The prototype code we are building on bitbucket is clear, easy to understand, read and adopt and already PEP8-fied (which is required for sdtlib inclusion) Reading and understanding pkg_resources requires a lot of time. A side note on setuptools, I don't mean to be rude here or to tell you what you should do, really, but: While I am very glad that you are part of PEP 376 discussions, and that you helped on almost every section of it, I really don't understand why you keep on pushing setuptools *code* here and at distutils-SIG since you don't maintain it anymore (the last commit is > 9 months) Part of the rejection of setuptools is because of that and because you don't bless anyone to maintain the project code base, or do releases, neither to communicate clearly on what's its roadmap. Regards Tarek From SridharR at activestate.com Wed Jul 15 19:38:21 2009 From: SridharR at activestate.com (Sridhar Ratnakumar) Date: Wed, 15 Jul 2009 10:38:21 -0700 Subject: [Python-Dev] PEP 376 - from PyPM's point of view In-Reply-To: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> Message-ID: On Wed, 15 Jul 2009 02:01:24 -0700, Tarek Ziad? wrote: >>> get_installed_files(local=False) -> iterator of (path, md5, size) >> >> Will this also return the directories /created/ during the installation? >> For example, will it also contain the entry "docutils" .. along with >> "docutils/__init__.py"? > > I don't think it's necessary to add "docutils" if > "docutils/__init__.py" is present > > But for empty directories added during installation we should add the I > guess. > > So, I'll add a note. It seems that you overlooked the below paragraph. >> If not, how is the installer (pip, pypm, etc..) supposed to know which >> directories to remove (docutils/) and which directories not to remove >> (site-packages/, bin/, etc..)? Quoting from the PEP: [quote]'(...)uninstall uses the APIs described earlier and remove all unique files, as long as their hash didn't change. Then it removes empty directories left behind.'[endquote] Let's assume that site-packages/ contained only one package 'Foo'. Will uninstall('Foo') remove the site-packages/ directory just because it turned out to be empty after removing 'Foo'? To explain, let's assume the RECORD of 'Foo' contains: $ cat RECORD Foo/__init__.py Foo/bar/__init__.py Foo/bar/test.py and according to what you wrote in the PEP ("it removes empty directories left behind"): $ python -m distutils.util.uninstall Foo rm /.../site-packages/Foo/__init__.py rm /.../site-packages/Foo/bar/__init__.py rm /.../site-packages/Foo/bar/test.py rm empty dir /.../site-packages/Foo/bar rm empty dir /.../site-packages/Foo/ rm empty dir /.../site-packages/ # !!!!! it also remove the site-packages directory! Then there is ~/python26/bin, ~/python26/include, ~/python26/etc, etc.. Do you see my point? > btw: is PyPM a public project ? If by 'public', you meant open source, then no. -srid From p.f.moore at gmail.com Wed Jul 15 19:40:28 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 Jul 2009 18:40:28 +0100 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <20090715161829.E45BE3A4116@sparrow.telecommunity.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <20090715161829.E45BE3A4116@sparrow.telecommunity.com> Message-ID: <79990c6b0907151040h5d37348cn8ee09755ba17f9cd@mail.gmail.com> 2009/7/15 P.J. Eby : > At 11:10 AM 7/15/2009 +0100, Paul Moore wrote: >> >> I propose that before the current prototype is turned into a final >> (spec and) implementation, the PEP 302 extensions are extracted and >> documented as an independent protocol, purely part of PEP 376. (This >> *helps* implementers, as they can write support for, for example, >> eggs, without needing to modify the existing egg importer). > > Btw, this is why setuptools chose to implement these things as adapters or > generic functions in pkg_resources (and in the vestigial bits that were > added to python 2.5's pkgutil). > > So as you can see, trying to solve these particular problems tends to lead > to reinventing setuptools or at least portions thereof. ?;-) Yes, that's the impression I'm getting :-) And yet you seem resistant to reading any message into the fact that people find it easier to solve the problems from scratch than to extract the relevant parts of the existing code. To be explicit - it's too flipping hard to identify, extract and tidy up the appropriate bits of the setuptools code! (Disclaimer: I haven't looked at the code for some time - if the style has changed utterly in the last 2 years or so, so that it no longer resembles the style of the pkgutil code you contributed to the stdlib, I'll retract that criticism of the setuptools codebase). >> But the >> problem is that too many people dislike setuptools as it stands for it >> to gain support. My understanding is that the current set of PEPs were >> intended to be a stripped down, more generally acceptable subset of >> setuptools. Let's keep them that way (and omit the complexities of >> multi-version support). > > Even without multi-version support, the parts of PEP 376 that aren't about > uninstallation are still reinventing chunks of pkg_resources. ?Had > pkg_resources been in the stdlib a couple years back (note that bugs and > changes in it are still quite rare), the PEP 376 bits for pkgutil could have > been focused strictly on uninstallation, and just reused pkg_resources API > for finding distributions, reading metadata, getting version info, etc. etc. > > All that stuff is extremely stable code, very widely used for a very long > time. ?If politics is the only thing keeping it from being used, then that's > a pretty sad statement about the community, given that pkg_resources is not > to blame for 99% of what people complain about in relation to setuptools. > ?All pkg_resources does is find stuff (on request) and maybe add it to > sys.path (on request), pull out data and metadata from distributions (on > request), or set up namespace packages (on request). I don't think it's quite "politics" as you describe it. Maybe "marketing", if you want a simple tag for it. The problem is that the setuptools "brand name" is associated with a lot of baggage, that maybe it doesn't deserve, but equally no-one seems interested in addressing. It's certainly not a technical argument at this point. And yet you counter criticisms with technical responses. But if I recall the history correctly, one of the initial goals of setuptools was to become a de-facto standard precisely by marketing and filling the ecological niche - a goal it has achieved fantastically well. The fact that in doing so it alienated certain groups is a more or less inevitable consequence. My argument - which you may well disagree with - is that the "setuptools ecosystem" (by which I mean setuptools/easy_install/pkg_resources/eggs and the various subsystems around these) is documented in a way which is: - confusing to the casual user - not at all modular (ie, it's impossible to easily pick out individual "bits") - full of new concepts, so it's hard to find an easy place to start from There's not much technically at issue here, it's all about presentation. > So if politics demands that it be rejected by association with "setuptools", > then just search-and-replace the API, PEP 8-ify it, call it something > different, and lie to everyone about where it came from. ?I won't tell if > you won't. ?;-) Unless the documentation and code is cleaned up, it'll still be obvious from the style where it came from - that's the problem. Who's going to maintain the code within the standard library, and are they happy with the style of the existing code - even with the serial numbers filed off? And of course, someone has to do the clean-up. It seems to me that the fact that people are more inclined to reinvent the code than to try to understand the existing codebase and pick out the relevant bits, says something important about how easy it would be to maintain the existing code within the Python core... Paul. From tjreedy at udel.edu Wed Jul 15 19:43:25 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 15 Jul 2009 13:43:25 -0400 Subject: [Python-Dev] PEP 376 - from PyPM's point of view In-Reply-To: <94bdd2610907150851u1c022d10i5bc6b6d9b581bc17@mail.gmail.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <4A5DACC2.5030609@voidspace.org.uk> <94bdd2610907150703i7af22a1eq28e0617c6abd735f@mail.gmail.com> <79990c6b0907150814r293abb92k7aa15ba0a70a7931@mail.gmail.com> <94bdd2610907150851u1c022d10i5bc6b6d9b581bc17@mail.gmail.com> Message-ID: Tarek Ziad? wrote: > In any case I don't see any use case to have a "site-packages" > remaining in Python itself. I have and am using it. Where else would you have me put library packages meant to be accessible by any Python program? Terry Jan Reedy From SridharR at activestate.com Wed Jul 15 19:52:09 2009 From: SridharR at activestate.com (Sridhar Ratnakumar) Date: Wed, 15 Jul 2009 10:52:09 -0700 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> <5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com> Message-ID: On Wed, 15 Jul 2009 08:22:03 -0700, David Cournapeau wrote: >> if docutils 0.5 is installed, Foo is broken, unless docutils 0.4 is >> shipped with it. > As was stated by Debian packagers on the distutils ML, the problem is > that docutils 0.5 breaks packages which work with docutils 0.4 in the > first place. Thus I am -1 on multi-version support .. and would rather have the python developers make their packages backward compatible just like what Armin did with Jinja and Jinja2. Debian at the moment has only one package "python-docutils" which is 0.5. How is a debian application supposed to depend upon 0.4? With Jinja, there is no problem .. there are 'python-jinja' and 'python-jinja2'. Similarly, CherryPy should have gone with renaming their package names to CherryPy2 and CherryPy3. -srid PS: Quoting from a fellow developer: > > [...] it sounds like CherryPy 3.x is really an incompatible > > module that just kept the old name. That is rather unfortunate, but can > > sometimes happen. > I have never seen a Python package changing its name (when significantly > changing API, design, etc..). The only exception that comes to my mind > is Jinja2 (renamed from 'Jinja'): > [Armin] (...) Because we love backwards compatibility these changes will > go into a package called "jinja2" > Well, congrats to the Jinja team then! The others will eventually learn... Mixing incompatible APIs in a single namespace and using a non-standardized version numbering system to keep things apart creates a world of pain! > The challenge however is in compartmentalizing versions according to > their major release numbers. Versioning in the Python world is already a > mess which we are beginning to sort out: > http://wiki.python.org/moin/Distutils/VersionComparison (PyPM relies on > this standard and the ongoing implementation - verlib.py) How embarrassing for a cult that prides itself on having only one way for everything they do... CPAN versions numbers are just as much if not more of a mess, but at least you can argue that it is the price for there being "more than one way to do it". From pje at telecommunity.com Wed Jul 15 20:12:44 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Jul 2009 14:12:44 -0400 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <94bdd2610907151007p7cd5d6bof32612bec2908e43@mail.gmail.com > References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <20090715161829.E45BE3A4116@sparrow.telecommunity.com> <94bdd2610907151007p7cd5d6bof32612bec2908e43@mail.gmail.com> Message-ID: <20090715180942.407DB3A403A@sparrow.telecommunity.com> At 07:07 PM 7/15/2009 +0200, Tarek Ziad? wrote: > Part of the rejection of setuptools is because of that and > because you don't > bless anyone to maintain the project code base, or do releases, >neither to communicate clearly > on what's its roadmap. Jim Fulton and Ian Bicking have been officially blessed for *years*, and I've posted plenty about what's on its roadmap. >I really don't understand why you keep on pushing setuptools *code* pkg_resources is not setuptools; its stability and revision history are very different, as it addresses different problems. From pje at telecommunity.com Wed Jul 15 20:20:10 2009 From: pje at telecommunity.com (P.J. Eby) Date: Wed, 15 Jul 2009 14:20:10 -0400 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <79990c6b0907151040h5d37348cn8ee09755ba17f9cd@mail.gmail.co m> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <20090715161829.E45BE3A4116@sparrow.telecommunity.com> <79990c6b0907151040h5d37348cn8ee09755ba17f9cd@mail.gmail.com> Message-ID: <20090715181705.E22823A403A@sparrow.telecommunity.com> At 06:40 PM 7/15/2009 +0100, Paul Moore wrote: >And of course, someone has to do the clean-up. It seems to me that the >fact that people are more inclined to reinvent the code than to try to >understand the existing codebase and pick out the relevant bits, says >something important about how easy it would be to maintain the >existing code within the Python core... That's normal for any code that contains "legacy" issues, which is why people always prefer rewriting code they don't understand: it's more fun to write than to read. However, as Joel Spolsky has well explained, rewriting such code inevitably means that you must re-learn the lessons that were learned by the original author. It seems like a waste, but then, I suppose those lessons must be relearned *some* way. I just think it'd be better if, having re-learned most of the lessons by trying to rewrite, one could then go back and learn the rest from the code. ;-) That having been said, it's obviously a dead parrot... one which I will now cease attempting to revive. From chrism at plope.com Wed Jul 15 19:47:35 2009 From: chrism at plope.com (Chris McDonough) Date: Wed, 15 Jul 2009 13:47:35 -0400 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <20090715163056.B34B63A4116@sparrow.telecommunity.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> <5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com> <79990c6b0907150859h6d7ad65g67e179c31aaa81f9@mail.gmail.com> <20090715163056.B34B63A4116@sparrow.telecommunity.com> Message-ID: <4A5E1637.30109@plope.com> I've been trying to follow this discussion now for weeks. The signal to noise ratio is pretty low. I'd love to have an stdlib solution for distribution packaging and installation. But I think we might as well pack it up and go home if the folks whom are contributing to the discussion "recreationally" (whom are not themselves implementers and potential implementers or spec writers or potential spec writers of packaging systems) continue to chime in on *every single issue*, contributing only stop energy. It's just completely pointless. - C On 7/15/09 12:34 PM, P.J. Eby wrote: > At 04:59 PM 7/15/2009 +0100, Paul Moore wrote: >> - Virtualenv isn't a workaround (I don't know virtualenv, I'll take >> your word for it) > > It's not one for system package maintainers because it would effectively > be managing multiple instances of 'python'. Really not a suitable solution. > > >> - I do not believe that it's clear that sanctioning the setuptools >> workaround as the "right" approach by building it into the Python >> core/stdlib is the right thing to do. > > I still don't understand how we're doing that. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/lists%40plope.com > From brett at python.org Wed Jul 15 20:38:22 2009 From: brett at python.org (Brett Cannon) Date: Wed, 15 Jul 2009 11:38:22 -0700 Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy? In-Reply-To: <4A5DDEEB.4040700@gmail.com> References: <4A5DDEEB.4040700@gmail.com> Message-ID: On Wed, Jul 15, 2009 at 06:51, Nick Coghlan wrote: > Antoine Pitrou wrote: > > I am not sure when this discussion started. Are you replying to a 3 > month-old > > message of yours? :) > > That depends on how you define the beginning of the discussion... > Especially since I was offlist three months ago. =) > > However, the fact that importlib doesn't implement the comparatively > recent get_filename() optional extension documented in PEP 302 came up > in one of the PEP 376 threads within the last week or so. > I have been ignoring the PEP 372 discussions since I came into it late and I don't deal with packaging enough to want to influence the discussion, so I missed that point being made. > > > In any case, keeping all import-related ABCs in a single place sounds > like a > > good idea. > > Alternatively, if the method is runpy specific, it doesn't have its place > in an > > ABC, does it? > > While runpy is the only client in the standard library for the > get_filename() method, the method is still a PEP 302 extension. I > documented the extension in the PEP as loaders are the only things > reliably in a position to provide the filename details that runpy needs > to set __file__ and sys.argv[0] correctly and until importlib came along > PEP 302 itself was the only real documentation of that API. > > Since importlib is now the "go-to" location for people that want to > write their own PEP 302 importers and loaders, I would say that it is > also the right place for the new ExecutionLoader ABC. OK. I will add an ExecutionLoader ABC to importlib that simply subclasses InspectLoader and adds the get_filename() abstract method. Thanks to everyone for their feedback. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Jul 15 21:26:38 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 15 Jul 2009 21:26:38 +0200 Subject: [Python-Dev] Mercurial: tag generation incorrect In-Reply-To: <4A581DCE.4050307@gmx.net> References: <4A57C5B5.5070000@v.loewis.de> <4A581DCE.4050307@gmx.net> Message-ID: <4A5E2D6E.6040200@v.loewis.de> Hagen F?rstenau wrote: >> be32850b093f is listed >> as having a child revision, 52b0a279fec6, and ISTM that *this* >> should be the revision that got tagged. > > I think the tag is correct. Note that the concept of tagging is > different in Mercurial, where a tag can only refer to a revision > previous to the one where it is inserted in .hgtags. If I understand > correctly, all relevant tagging revisions from SVN are replaced by > Mercurial revisions setting tags, which then refer to their immediate > predecessors. Ah, ok. Thanks for the explanation. Regards, Martin From ncoghlan at gmail.com Wed Jul 15 23:56:46 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 16 Jul 2009 07:56:46 +1000 Subject: [Python-Dev] Add an ExecutionLoader abc to importlib or to runpy? In-Reply-To: References: <4A5DDEEB.4040700@gmail.com> Message-ID: <4A5E509E.4010900@gmail.com> Brett Cannon wrote: > On Wed, Jul 15, 2009 at 06:51, Nick Coghlan > wrote: > However, the fact that importlib doesn't implement the comparatively > recent get_filename() optional extension documented in PEP 302 came up > in one of the PEP 376 threads within the last week or so. > > I have been ignoring the PEP 372 discussions since I came into it late > and I don't deal with packaging enough to want to influence the > discussion, so I missed that point being made. Since I'm pretty sure you were the one that mentioned it, it's possible I just misremembered which thread it came up in. The various discussions can blur together sometimes! Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From p.f.moore at gmail.com Thu Jul 16 00:27:00 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 Jul 2009 23:27:00 +0100 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <4A5E1637.30109@plope.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> <5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com> <79990c6b0907150859h6d7ad65g67e179c31aaa81f9@mail.gmail.com> <20090715163056.B34B63A4116@sparrow.telecommunity.com> <4A5E1637.30109@plope.com> Message-ID: <79990c6b0907151527k1d8b8889j9adc7f4b1edc4e57@mail.gmail.com> 2009/7/15 Chris McDonough : > I've been trying to follow this discussion now for weeks. The signal to > noise ratio is pretty low. I agree :-( > I'd love to have an stdlib solution for distribution packaging and > installation. ?But I think we might as well pack it up and go home if the > folks whom are contributing to the discussion "recreationally" (whom are not > themselves implementers and potential implementers or spec writers or > potential spec writers of packaging systems) continue to chime in on *every > single issue*, contributing only stop energy. ?It's just completely > pointless. I thought that as someone who is writing code for Tarek's PEP 376 prototype implementation, I counted as offering useful input. Thinking further on what I've said, I'm no longer sure that's true. I do think that at *some* point, the feedback from the non-packagers, negative or irrelevant though it might be, needs to be requested and considered. Maybe now isn't the right time, but it needs to be done at some point. Until then, I'll try to avoid adding to the noise. Paul. From david.lyon at preisshare.net Thu Jul 16 01:39:40 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 15 Jul 2009 19:39:40 -0400 Subject: [Python-Dev] PEP 376 - from pythonpkgmgr point of view In-Reply-To: <4A5E1637.30109@plope.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> <5b8d13220907150822lee247csfd12e1a204f37931@mail.gmail.com> <79990c6b0907150859h6d7ad65g67e179c31aaa81f9@mail.gmail.com> <20090715163056.B34B63A4116@sparrow.telecommunity.com> <4A5E1637.30109@plope.com> Message-ID: <7cc3a3a3ff2f5a17c81060e6155c88c2@preisshare.net> On Wed, 15 Jul 2009 13:47:35 -0400, Chris McDonough wrote: > I've been trying to follow this discussion now for weeks. The signal to > noise ratio is pretty low. I'm -1 on that.. As a relative newcomer to python packaging I'm finding all these discussions very informative. :-) > I'd love to have an stdlib solution for distribution packaging and > installation. That's right. I'd love to find out what the process is for submitting my pythonpkgmgr project for consideration into the stdlib. As it fills a huge gap that you've just identified that currently exists within the python that is being shipped today. That is, allowing users to easily install packages from pypi. > But I think we might as well pack it up and go home if the folks whom are > contributing to the discussion "recreationally" (whom are not themselves > implementers and potential implementers or spec writers or potential spec > writers of packaging systems) continue to chime in on *every single issue*, > > contributing only stop energy. It's just completely pointless. We can all work in different ways.. I'm not writing PEPS and don't care too much about how the internals of a package work. As long as there are APIs for such stuff I'm happy. What I'm working on is to try to improve the user experience which isn't so handled so easily by a design by commitee process. Packaging and Distutils, need to be looked at wholistically as well. It's not just about the api's, but how well they work. It's not noise.... it's just the process.... ps: (http://peak.telecommunity.com/DevCenter/EasyInstall) "Easy Install ... Please share your experiences with us! If you encounter difficulty installing a package, please contact us via the distutils mailing list. (Note: please DO NOT send private email directly to the author of setuptools; it will be discarded. " So what are we to do if we find issues with setuptools other than to post to distutils? David From ziade.tarek at gmail.com Thu Jul 16 02:43:50 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 16 Jul 2009 02:43:50 +0200 Subject: [Python-Dev] PEP 376 - from PyPM's point of view In-Reply-To: References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> Message-ID: <94bdd2610907151743m619f91b9n9cc1dc31a86d2729@mail.gmail.com> 2009/7/15 Sridhar Ratnakumar : > On Wed, 15 Jul 2009 02:01:24 -0700, Tarek Ziad? > wrote: > >>>> get_installed_files(local=False) -> iterator of (path, md5, size) >>> >>> Will this also return the directories /created/ during the installation? >>> For example, will it also contain the entry "docutils" .. along with >>> "docutils/__init__.py"? >> >> I don't think it's necessary to add "docutils" if >> "docutils/__init__.py" is present >> >> But for empty directories added during installation we should add the I >> guess. >> >> So, I'll add a note. > > It seems that you overlooked the below paragraph. > >>> If not, how is the installer (pip, pypm, etc..) supposed to know which >>> directories to remove (docutils/) and which directories not to remove >>> (site-packages/, bin/, etc..)? > > Quoting from the PEP: > > [quote]'(...)uninstall uses the APIs described earlier and remove all unique > files, as long as their hash didn't change. Then it removes empty > directories left behind.'[endquote] > > Let's assume that site-packages/ contained only one package 'Foo'. Will > uninstall('Foo') remove the site-packages/ directory just because it turned > out to be empty after removing 'Foo'? To explain, let's assume the RECORD of > 'Foo' contains: > > $ cat RECORD > Foo/__init__.py > Foo/bar/__init__.py > Foo/bar/test.py > > and according to what you wrote in the PEP ("it removes empty directories > left behind"): > > $ python -m distutils.util.uninstall Foo > rm /.../site-packages/Foo/__init__.py > rm /.../site-packages/Foo/bar/__init__.py > rm /.../site-packages/Foo/bar/test.py > rm empty dir /.../site-packages/Foo/bar > rm empty dir /.../site-packages/Foo/ > rm empty dir /.../site-packages/ ? ? ? ?# !!!!! > > it also remove the site-packages directory! > > Then there is ~/python26/bin, ~/python26/include, ~/python26/etc, etc.. Do > you see my point? I didn't mean that of course. While we can avoid your example for the code, by removing only packages that are fully emptied, and are alongside the egg-info directory, we might not be able to do it properly for the data. So let's add the directories as well From him at online.de Thu Jul 16 14:26:00 2009 From: him at online.de (=?ISO-8859-1?Q?Joachim_K=F6nig?=) Date: Thu, 16 Jul 2009 14:26:00 +0200 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <20090715162736.B54D33A4116@sparrow.telecommunity.com> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> <4A5DF2D9.6050403@online.de> <20090715162736.B54D33A4116@sparrow.telecommunity.com> Message-ID: <4A5F1C58.3030003@online.de> P.J. Eby wrote: > At 05:16 PM 7/15/2009 +0200, Joachim K?nig wrote: >> f you have m different versions of n packages then >> you could have n**m different combinations for an application so you >> need a >> possiblilty to select one combination from n**m possible ones at >> application >> startup time. Is this really worth it? > > Obviously yes, as neither buildout nor setuptools would exist > otherwise. ;-) > Nor would Fedora be packaging certain library versions as eggs > specifically to get certain multi-version scenarios to work. > > The specific solutions for handling n*m problems aren't fantastic, but > they are clearly needed. I still do not see the need. IMO the whole obfuscation comes from fact that all versions of all packages are installed into one location where python automaticallly looks for packages and then with a lot of magic the packages are hidden from the interpreter and only specific requested versions are made "visible" to the interpreter at runtime. Why do the package have to be installed there at the first place? For an application it would be enough to have an additional directory on its PYTHONPATH where the packages required for this application would be installed. So a package could be installed either to the common directory ("site-packages") or an application specific directory (e.g. something like "app-packages//"). This approach has been used by Zope2 with its "private" lib/python directory for years. So one would have to set up the application specific packages before running the application, but the whole clutter of uncounted versions of the same package in one directory could go away. The "drawback" of this approach would be, that the same version of a package would have to be installed multiple times if needed by different applications. From thomas at python.org Thu Jul 16 22:08:59 2009 From: thomas at python.org (Thomas Wouters) Date: Thu, 16 Jul 2009 22:08:59 +0200 Subject: [Python-Dev] pthreads, fork, import, and execvp In-Reply-To: <446CB69E.4090609@v.loewis.de> References: <44684958.9010206@gmail.com> <446973F0.3010007@v.loewis.de> <44699B47.4030504@gmail.com> <446A4202.6070608@v.loewis.de> <446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com> <446CB69E.4090609@v.loewis.de> Message-ID: <9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com> On Thu, May 18, 2006 at 20:02, "Martin v. L?wis" wrote: > Nick Coghlan wrote: > > And if I understand it correctly, it falls under the category that > > waiting for another thread while holding the import lock is a *really* > > bad idea from a thread safety point of view. > > > > The thing with the import-after-fork deadlock is that you can trigger it > > without even doing anything that's known not to be thread-safe. > > Right. With some googling, I found that one solution is pthread_atexit: > a pthread_atexit handler is a triple (before, in_parent, in_child). > You set it to (acquire, release, release). When somebody forks, > the pthread library will first acquire the import lock in the thread > that wants to fork. Then the fork occurs, and the import lock gets > then released both in the parent and in the child. > > I would like to see this approach implemented, but I agree with you > that a test case should be created first. > Picking up a rather old discussion... We encountered this bug at Google and I'm now "incentivized" to fix it. For a short recap: Python has an import lock that prevents more than one thread from doing an import at any given time. However, unlike most of the locks we have lying around, we don't clear that lock in the child after an os.fork(). That means that doing an os.fork() during an import means the child process can't do any other imports. It also means that doing an os.fork() *while another thread is doing an import* means the child process can't do any other imports. Since this three-year-old discussion we've added a couple of post-fork-cleanups to CPython (the TLS, the threading module's idea of active threads, see Modules/signalmodule.c:PyOS_AfterFork) and we already do simply discard the memory for other locks held during fork (the GIL, see Python/ceval.c:PyEval_ReInitThreads, and the TLS lock in Python/thread.c:PyThread_ReInitTLS) -- but not so with the import lock, except when the platform is AIX. I don't see any particular reason why we aren't doing the same thing to the import lock that we do to the other locks, on all platforms. It's a quick fix for a real problem (see http://bugs.python.org/issue1590864 and http://bugs.python.org/issue1404925 for two bugreports that seem to be this very issue.) It also seems to me, since we have two or three (depending on how you count) uses for it, that we should either add a way to free the old locks (to the various threading implementations) or add a way to use pthread_atexit portably -- or at least portably across systems with fork(). I don't think we should wait with fixing the big issue (making threads and fork() being unnecessarily flakey) until we have a good fix for the small issue (a tiny bit of a memory leak after a fork() -- in the child process only.) Especially since the good fix for the small issue might require co-ordination between all the threading implementations we have and nobody knows about. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jul 16 23:07:22 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 17 Jul 2009 07:07:22 +1000 Subject: [Python-Dev] pthreads, fork, import, and execvp In-Reply-To: <9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com> References: <44684958.9010206@gmail.com> <446973F0.3010007@v.loewis.de> <44699B47.4030504@gmail.com> <446A4202.6070608@v.loewis.de> <446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com> <446CB69E.4090609@v.loewis.de> <9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com> Message-ID: <4A5F968A.4060706@gmail.com> Thomas Wouters wrote: > Picking up a rather old discussion... We encountered this bug at Google > and I'm now "incentivized" to fix it. > > For a short recap: Python has an import lock that prevents more than one > thread from doing an import at any given time. However, unlike most of > the locks we have lying around, we don't clear that lock in the child > after an os.fork(). That means that doing an os.fork() during an import > means the child process can't do any other imports. It also means that > doing an os.fork() *while another thread is doing an import* means the > child process can't do any other imports. > > Since this three-year-old discussion we've added a couple of > post-fork-cleanups to CPython (the TLS, the threading module's idea of > active threads, see Modules/signalmodule.c:PyOS_AfterFork) and we > already do simply discard the memory for other locks held during fork > (the GIL, see Python/ceval.c:PyEval_ReInitThreads, and the TLS lock in > Python/thread.c:PyThread_ReInitTLS) -- but not so with the import lock, > except when the platform is AIX. I don't see any particular reason why > we aren't doing the same thing to the import lock that we do to the > other locks, on all platforms. It's a quick fix for a real problem (see > http://bugs.python.org/issue1590864 and > http://bugs.python.org/issue1404925 for two bugreports that seem to be > this very issue.) +1. Leaving deadlock issues around that people can run into without doing anything wrong in their application is unfriendly. > It also seems to me, since we have two or three (depending on how you > count) uses for it, that we should either add a way to free the old > locks (to the various threading implementations) or add a way to use > pthread_atexit portably -- or at least portably across systems with > fork(). I don't think we should wait with fixing the big issue (making > threads and fork() being unnecessarily flakey) until we have a good fix > for the small issue (a tiny bit of a memory leak after a fork() -- in > the child process only.) Especially since the good fix for the small > issue might require co-ordination between all the threading > implementations we have and nobody knows about. One-off memory leaks for relatively rare operations like fork() don't bother me all that much, so I agree that having fork() leak another lock in the child process is preferable to it having a chance to mysteriously deadlock. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From dalcinl at gmail.com Thu Jul 16 23:17:12 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 16 Jul 2009 18:17:12 -0300 Subject: [Python-Dev] pthreads, fork, import, and execvp In-Reply-To: <446CB69E.4090609@v.loewis.de> References: <44684958.9010206@gmail.com> <446973F0.3010007@v.loewis.de> <44699B47.4030504@gmail.com> <446A4202.6070608@v.loewis.de> <446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com> <446CB69E.4090609@v.loewis.de> Message-ID: On Thu, May 18, 2006 at 3:02 PM, "Martin v. L?wis" wrote: > > Right. With some googling, I found that one solution is pthread_atexit: > a pthread_atexit handler is a triple (before, in_parent, in_child). > You set it to (acquire, release, release). Did you mean pthread_atfork() ? > When somebody forks, > the pthread library will first acquire the import lock in the thread > that wants to fork. Then the fork occurs, and the import lock gets > then released both in the parent and in the child. > > I would like to see this approach implemented, but I agree with you > that a test case should be created first. > Whould some new C-API functions do the trick? PyOS_AtForkPrepare() # acquire import lock pid = fork() if (pid == 0) PyOS_AtForkChild() # current PyOS_AfterFork() + release import lock else PyOS_AtForkParent() # release import lock -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From ncoghlan at gmail.com Thu Jul 16 23:20:49 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 17 Jul 2009 07:20:49 +1000 Subject: [Python-Dev] [Distutils] PEP 376 - from PyPM's point of view In-Reply-To: <4A5F1C58.3030003@online.de> References: <94bdd2610907150201y2204d4f6pbf57f3cd8e3d45f9@mail.gmail.com> <79990c6b0907150310r318a6d9ep22e84f6c98355969@mail.gmail.com> <94bdd2610907150700y55eb26d6qda2ebc4f702fb0af@mail.gmail.com> <4A5DF2D9.6050403@online.de> <20090715162736.B54D33A4116@sparrow.telecommunity.com> <4A5F1C58.3030003@online.de> Message-ID: <4A5F99B1.6000006@gmail.com> Joachim K?nig wrote: > So one would have to set up the application specific packages before > running the application, but the > whole clutter of uncounted versions of the same package in one directory > could go away. The > "drawback" of this approach would be, that the same version of a package > would have to be installed > multiple times if needed by different applications. While this is a very common practice in the Windows world, it is far less common in the *nix world of vendor managed packaging systems. As for why it can be a problem, it (bundling libraries with applications) makes security vulnerability management a *lot* more difficult for system administrators. If a bug is found in a key library (e.g. openssl) a dependency based system just needs to update the single shared copy of that library. In a bundling system, you first have to work out which of your applications contain an instance of that library and then see if the application vendors have provided a security patch. If any one of them hasn't released a patch and you can't patch it yourself, then you either have to stop using that application or else accept remaining exposed to the vulnerability. The bundling approach also leads to applications being much bigger than they need to be. That isn't much of a problem for desktop or server systems these days, but can still be an issue in the embedded world. As far as the idea of making bundling easier goes, we already implemented that in 2.6 and 3.0. It's the whole reason that zip files and directories are directly executable now: the named zip file or directory itself is automatically added to sys.path, so the top level "__main__.py" in that location can freely import any other co-located modules and packages. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From status at bugs.python.org Fri Jul 17 18:07:04 2009 From: status at bugs.python.org (Python tracker) Date: Fri, 17 Jul 2009 18:07:04 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20090717160704.57CCA7861B@psf.upfronthosting.co.za> ACTIVITY SUMMARY (07/10/09 - 07/17/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2255 open (+32) / 16072 closed (+15) / 18327 total (+47) Open issues with patches: 892 Average duration of open issues: 662 days. Median duration of open issues: 414 days. Open Issues Breakdown open 2222 (+31) pending 33 ( +1) Issues Created Or Reopened (48) _______________________________ locale.D_* and .T_* are int, not string 07/10/09 CLOSED http://bugs.python.org/issue6456 reopened r.david.murray distutils.command.build_ext.get_export_symbols should use the "P 07/11/09 CLOSED http://bugs.python.org/issue6459 reopened tarek patch test failure in test_xmlrpc on Gentoo in trunk 07/11/09 http://bugs.python.org/issue6460 created r.david.murray multiprocessing: freezing apps on Windows 07/11/09 http://bugs.python.org/issue6461 created sgm bsddb3 intermittent test failures 07/11/09 http://bugs.python.org/issue6462 created r.david.murray IDLE with Tk-Cocoa: Edit, format menus hang 07/11/09 http://bugs.python.org/issue6463 created wordtech test_normalization failures due to truncated NormalizationTest.t 07/11/09 CLOSED http://bugs.python.org/issue6464 created r.david.murray easy email.feedparser regular expression bug (NLCRE_crack) 07/11/09 http://bugs.python.org/issue6465 created jkg patch duplicate get_version() code between cygwinccompiler and emxccom 07/12/09 http://bugs.python.org/issue6466 created tarek raw_input() doesn't work as expected when it gets multiple ^D 07/12/09 http://bugs.python.org/issue6467 created lucaspmelo __missing__ not completely implemented in dictobject.c 07/12/09 CLOSED http://bugs.python.org/issue6468 created bvukov Function definition expressions feature 07/12/09 CLOSED http://bugs.python.org/issue6469 created grammati patch Tkinter import fails when running Python.exe from a network shar 07/12/09 http://bugs.python.org/issue6470 created cgohlke patch, needs review errno and strerror attributes incorrectly set on socket errors w 07/12/09 http://bugs.python.org/issue6471 created ezio.melotti Inconsistent use of "iterator" in ElementTree doc & diff between 07/13/09 http://bugs.python.org/issue6472 created MLModel hmac sha384/sha512 fails test vectors 07/13/09 CLOSED http://bugs.python.org/issue6473 created iwade Inconsistent TypeError message on function calls with wrong numb 07/13/09 http://bugs.python.org/issue6474 created hagen Documentation > Tutorial > List Comprehensions 07/13/09 CLOSED http://bugs.python.org/issue6475 created Retro MSVCRT's spawnve/spawnvpe are not thread safe 07/13/09 http://bugs.python.org/issue6476 created jfonseca Pickling of NoneType raises PicklingError 07/13/09 http://bugs.python.org/issue6477 created July time.tzset does not reset _strptime's locale time cache 07/13/09 http://bugs.python.org/issue6478 created mibanescu platform.py writes to hard coded platform dependant "dev/null" 07/13/09 CLOSED http://bugs.python.org/issue6479 created john.burnett patch code.runsource() parsing bug 07/14/09 http://bugs.python.org/issue6480 created smcallis PATCH: typo in subprocess documentation 07/14/09 CLOSED http://bugs.python.org/issue6481 created Yinon patch PATCH: tiny simplification for subprocess 07/14/09 CLOSED http://bugs.python.org/issue6482 created Yinon patch Modules are not deallocated correctly if m_size = -1 07/14/09 http://bugs.python.org/issue6483 created jak No unit test for mailcap module 07/14/09 http://bugs.python.org/issue6484 created gnofi is_finished not exported by zlib 07/14/09 CLOSED http://bugs.python.org/issue6485 created solinym Library doc introduction strangely points to "Built-in Objects" 07/14/09 CLOSED http://bugs.python.org/issue6486 created MLModel Some index entries appear in black 07/14/09 CLOSED http://bugs.python.org/issue6487 created MLModel ElementTree documentation refers to "path" with no explanation, 07/15/09 http://bugs.python.org/issue6488 created MLModel Documentation of ElementTree.Element.getiterator implies element 07/15/09 http://bugs.python.org/issue6489 created MLModel os.popen documentation in 2.6 is probably wrong 07/15/09 http://bugs.python.org/issue6490 created krawyoti Improve --with-dbmliborder option 07/15/09 http://bugs.python.org/issue6491 created Arfrever patch xml.etree does not escape CR, LF and TAB characters within attri 07/15/09 http://bugs.python.org/issue6492 created devon Can not set value for structure members larger than 32 bits 07/16/09 http://bugs.python.org/issue6493 created higstar patch xmlrpc client does not support HTTPS 07/16/09 CLOSED http://bugs.python.org/issue6494 created lister171254 particular variable's name case exception attributeError 07/16/09 CLOSED http://bugs.python.org/issue6495 created tq0fqeu 2to3 generates "from urllib.parse import pathname2url" 07/16/09 http://bugs.python.org/issue6496 created srid patch Support for digital Cinema/film DPX and Kodak Cineon image file 07/16/09 http://bugs.python.org/issue6497 created waltermb patch Py_Main() does not return on SystemExit 07/16/09 http://bugs.python.org/issue6498 created Rogi Can't import xmlrpclib, DocXMLRPCServer and SimpleXMLRPCServer w 07/17/09 http://bugs.python.org/issue6499 created ezio.melotti urllib2 maximum recursion depth exceeded 07/17/09 http://bugs.python.org/issue6500 created nako521 Fatal LookupError: unknown encoding: cp0 on Windows embedded sta 07/17/09 http://bugs.python.org/issue6501 created grahamd documentation error: missing comma between kwonlyargcount & nloc 07/17/09 http://bugs.python.org/issue6502 created kaizhu Python Docs: 6.6.4. Mutable Sequence Types 07/17/09 CLOSED http://bugs.python.org/issue6503 created mkesper infinite recursion from calling builtins.open() 07/17/09 http://bugs.python.org/issue6504 created kaizhu Minor typos in tutorial (i/o chapter) 07/17/09 http://bugs.python.org/issue6505 created csmayfield Issues Now Closed (43) ______________________ Doc changes left over after mega-merge from trunk 622 days http://bugs.python.org/issue1370 amaury.forgeotdarc Confusing Descrintro example 534 days http://bugs.python.org/issue1921 benjamin.peterson easy Array pickling exposes internal memory representation of element 484 days http://bugs.python.org/issue2389 alexandre.vassalotti patch Check 2to3 for support of print function. 481 days http://bugs.python.org/issue2412 benjamin.peterson patch Import errors in email.message.py 461 days http://bugs.python.org/issue2622 amaury.forgeotdarc patch "Suggest a change" link 370 days http://bugs.python.org/issue3341 georg.brandl typo in a module describes utf-8 as uft-8 223 days http://bugs.python.org/issue4540 amaury.forgeotdarc patch, needs review kqueue for more than one event is broken. 74 days http://bugs.python.org/issue5910 georg.brandl patch Missing title for configDialog.py 56 days http://bugs.python.org/issue6076 amaury.forgeotdarc patch os.popen exit code inconsistent 13 days http://bugs.python.org/issue6358 amaury.forgeotdarc Dictionaries should support __add__ 6 days http://bugs.python.org/issue6410 tjreedy warnings.warn segfaults on bad formatted string 13 days http://bugs.python.org/issue6415 ocean-city patch unittest.TestProgram change breaks nose 10 days http://bugs.python.org/issue6418 michael.foord patch errors in docs re module initialization vs self arg to functions 6 days http://bugs.python.org/issue6421 georg.brandl easy 2to3: fix_future conflicts with fix_print 6 days http://bugs.python.org/issue6429 benjamin.peterson array.array falsely advertises support for 'w' in documentation 5 days http://bugs.python.org/issue6430 georg.brandl ImageTk.PhotoImage 7 days http://bugs.python.org/issue6432 srid multiprocessing: pool.map hangs on empty list 9 days http://bugs.python.org/issue6433 jnoller patch cygwinccompiler regular expressions broken 4 days http://bugs.python.org/issue6438 tarek 2to3: convert deprecated string.maketrans to str.maketrans 3 days http://bugs.python.org/issue6440 benjamin.peterson import_spam() in extending python can fail without setting the e 2 days http://bugs.python.org/issue6446 georg.brandl PATCH: typo (English) in subprocess module 1 days http://bugs.python.org/issue6447 amaury.forgeotdarc patch imp.find_module() -- be explicity that 'path' must be a list 2 days http://bugs.python.org/issue6448 georg.brandl Improve/update python.org/dev/ 8 days http://bugs.python.org/issue6449 ezio.melotti easy Improve bool TypeError message 1 days http://bugs.python.org/issue6453 tjreedy easy locale.D_* and .T_* are int, not string 1 days http://bugs.python.org/issue6456 georg.brandl With logging.config.fileConfig can't create logger without handl 3 days http://bugs.python.org/issue6458 vsajip distutils.command.build_ext.get_export_symbols should use the "P 0 days http://bugs.python.org/issue6459 tarek patch test_normalization failures due to truncated NormalizationTest.t 0 days http://bugs.python.org/issue6464 benjamin.peterson easy __missing__ not completely implemented in dictobject.c 0 days http://bugs.python.org/issue6468 benjamin.peterson Function definition expressions feature 1 days http://bugs.python.org/issue6469 r.david.murray patch hmac sha384/sha512 fails test vectors 4 days http://bugs.python.org/issue6473 iwade Documentation > Tutorial > List Comprehensions 0 days http://bugs.python.org/issue6475 georg.brandl platform.py writes to hard coded platform dependant "dev/null" 0 days http://bugs.python.org/issue6479 john.burnett patch PATCH: typo in subprocess documentation 2 days http://bugs.python.org/issue6481 georg.brandl patch PATCH: tiny simplification for subprocess 3 days http://bugs.python.org/issue6482 georg.brandl patch is_finished not exported by zlib 0 days http://bugs.python.org/issue6485 ezio.melotti Library doc introduction strangely points to "Built-in Objects" 1 days http://bugs.python.org/issue6486 georg.brandl Some index entries appear in black 2 days http://bugs.python.org/issue6487 georg.brandl xmlrpc client does not support HTTPS 0 days http://bugs.python.org/issue6494 loewis particular variable's name case exception attributeError 0 days http://bugs.python.org/issue6495 amaury.forgeotdarc Python Docs: 6.6.4. Mutable Sequence Types 0 days http://bugs.python.org/issue6503 ezio.melotti cp720 encoding map 940 days http://bugs.python.org/issue1616979 amaury.forgeotdarc patch, needs review Top Issues Most Discussed (10) ______________________________ 10 distutils.command.build_ext.get_export_symbols should use the " 0 days closed http://bugs.python.org/issue6459 7 multiline exception logging via syslog handler 8 days open http://bugs.python.org/issue6444 6 os.popen documentation in 2.6 is probably wrong 2 days open http://bugs.python.org/issue6490 6 test failure in test_xmlrpc on Gentoo in trunk 7 days open http://bugs.python.org/issue6460 6 Fraction fails equality test with a user-defined type 10 days open http://bugs.python.org/issue6431 6 OS X: python3 from python-3.1.dmg crashes at startup 16 days open http://bugs.python.org/issue6393 6 CVE-2008-5983 python: untrusted python modules search path 94 days open http://bugs.python.org/issue5753 5 cp720 encoding map 940 days closed http://bugs.python.org/issue1616979 4 PATCH: tiny simplification for subprocess 3 days closed http://bugs.python.org/issue6482 4 platform.py writes to hard coded platform dependant "dev/null" 0 days closed http://bugs.python.org/issue6479 From jafo at tummy.com Sat Jul 18 02:28:36 2009 From: jafo at tummy.com (Sean Reifschneider) Date: Fri, 17 Jul 2009 18:28:36 -0600 Subject: [Python-Dev] Experiment: Adding "re" to string objects. Message-ID: <20090718002836.GA26311@tummy.com> I'm mailing this to python-dev because I'd like feedback on the idea of adding an "re" attribute to strings. I'm not sure if it's a good idea or not yet, but I figure it's worth discussion. The module mentioned here includes a class called "restr()" which allows you to play with "s.re". As some of you may recall, I'm not particularly fond of the recipe: m = re.match(r'whatever(.*)', s) if m: m.group(1) The other morning I came up on the idea of adding an "re" to strings, so you could do things like: if s.re.match(r'whatever(.*)'): print s.re.group(1) or: if (date.re.match(r'(?P\d\d\d\d)-(?P\d\d)' or date.re.match(r'(?P\d\d)-(?P\d\d\d\d)'): print date.re.groupdict('year') So I decided to try experimenting with it and see how I like it. I've also thrown a bunch of other stuff into it and made a module called "filtertools": http://pypi.python.org/pypi/filtertools/0.01 ftp://ftp.tummy.com/pub/tummy/Python/python-filtertools/ As the version number is meant to indicate, this is something that I'm still exploring whether it is the right thing done in the right way. Though at the moment the only thing I plan to change is that some of the iterators (having nothing to do with adding "re" to string objects) probably shouldn't consume the "barrier" such as the "dropwhile()" and "takewhile()". You might want to do something like: fp = filtertools.reopen('mailbox') for header in filtertools.takewhile([ r'^\S' ], fp.readlines()) : print 'HEADER:', header.rstrip() for continued in filtertools.takewhile([ r'^\s+\S' ], fp.readlines()) : print 'CONTINUED:', continued.rstrip() But, the "takewhile()" I will consume the first non-matching line. Anyway, I appreciate any feedback folks have. Thanks, Sean -- What we see depends on mainly what we look for. -- John Lubbock Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability From aahz at pythoncraft.com Sat Jul 18 02:48:41 2009 From: aahz at pythoncraft.com (Aahz) Date: Fri, 17 Jul 2009 17:48:41 -0700 Subject: [Python-Dev] Experiment: Adding "re" to string objects. In-Reply-To: <20090718002836.GA26311@tummy.com> References: <20090718002836.GA26311@tummy.com> Message-ID: <20090718004841.GB15099@panix.com> On Fri, Jul 17, 2009, Sean Reifschneider wrote: > > I'm mailing this to python-dev because I'd like feedback on the idea of > adding an "re" attribute to strings. I'm not sure if it's a good idea or > not yet, but I figure it's worth discussion. The module mentioned here > includes a class called "restr()" which allows you to play with "s.re". Ideas should go to python-ideas, please. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "If you think it's expensive to hire a professional to do the job, wait until you hire an amateur." --Red Adair From python at mrabarnett.plus.com Sat Jul 18 02:51:54 2009 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 18 Jul 2009 01:51:54 +0100 Subject: [Python-Dev] Experiment: Adding "re" to string objects. In-Reply-To: <20090718002836.GA26311@tummy.com> References: <20090718002836.GA26311@tummy.com> Message-ID: <4A611CAA.5020108@mrabarnett.plus.com> Sean Reifschneider wrote: > I'm mailing this to python-dev because I'd like feedback on the idea of > adding an "re" attribute to strings. I'm not sure if it's a good idea or > not yet, but I figure it's worth discussion. The module mentioned here > includes a class called "restr()" which allows you to play with "s.re". > > As some of you may recall, I'm not particularly fond of the recipe: > > m = re.match(r'whatever(.*)', s) > if m: > m.group(1) > > The other morning I came up on the idea of adding an "re" to strings, so > you could do things like: > > if s.re.match(r'whatever(.*)'): > print s.re.group(1) > > or: > > if (date.re.match(r'(?P\d\d\d\d)-(?P\d\d)' or > date.re.match(r'(?P\d\d)-(?P\d\d\d\d)'): > print date.re.groupdict('year') > [snip] Why not drop the ".re" part? You would, however, then need a new name for the re split, eg "re_split". Or you could make the string the pattern, eg r'whatever(.*)'.match(s). From p.f.moore at gmail.com Sat Jul 18 11:55:40 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 18 Jul 2009 10:55:40 +0100 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: References: Message-ID: <79990c6b0907180255q5c0482ds5a8d05c3babc7716@mail.gmail.com> 2009/7/4 Brett Cannon : >> > While I really like the idea of using named branches for each release so >> > that there is a single py3k branch that contains all relevant history >> > for >> > every release, I think we should start simple and go with cloned >> > branches. >> > That way the workflow does not radically shift from what we do now for >> > svn >> > to start. Once the conversion is done and people are comfortable with hg >> > we >> > can then discuss moving towards a named branch approach. >> >> I don't think the cloned branches is much simpler than the named >> branches approach, in several ways. For example, populating the branch >> part of a sys.whatever value is significantly harder. Also, if you >> follow a useful tagging approach, doing cloned branches means that >> release tags aren't available on trunk/main/default. That seems like a >> step backwards. > > I personally prefer named branches, but that's just me and I am not about to > force my preferences on everyone. Guess we just have to see if others have > opinions against named branches. Personally, I prefer clones, as it seems to me that Mercurial named branches are not quite what people generally think of when they think of "branches", in some subtle respects. I could be wrong, as I don't personally use named branches. However, there's been a small thread on distutils-sig recently with a new Mercurial user complaining that he's got confused and messed up a repository. Without digging particularly deeply, it appears that the problems were caused by confusion over named branches. FWIW, the Mercurial book (http://hgbook.red-bean.com/read/managing-releases-and-branchy-development.html) describes named branches with the comment "If you're more in the ?power user? category (*and* your collaborators are too)" (author's emphasis). I'm not sure we want to require contributors to be "power users" of Mercurial... Paul. PS Sorry for responding to an old thread - I couldn't locate more recent discussion, although I thought there had been some. From fuzzyman at voidspace.org.uk Sat Jul 18 13:21:11 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 18 Jul 2009 12:21:11 +0100 Subject: [Python-Dev] Experiment: Adding "re" to string objects. In-Reply-To: <4A611CAA.5020108@mrabarnett.plus.com> References: <20090718002836.GA26311@tummy.com> <4A611CAA.5020108@mrabarnett.plus.com> Message-ID: <4A61B027.4050106@voidspace.org.uk> MRAB wrote: > Sean Reifschneider wrote: >> I'm mailing this to python-dev because I'd like feedback on the idea of >> adding an "re" attribute to strings. I'm not sure if it's a good >> idea or >> not yet, but I figure it's worth discussion. The module mentioned here >> includes a class called "restr()" which allows you to play with "s.re". >> >> As some of you may recall, I'm not particularly fond of the recipe: >> >> m = re.match(r'whatever(.*)', s) >> if m: >> m.group(1) >> >> The other morning I came up on the idea of adding an "re" to strings, so >> you could do things like: >> >> if s.re.match(r'whatever(.*)'): >> print s.re.group(1) >> >> or: >> >> if (date.re.match(r'(?P\d\d\d\d)-(?P\d\d)' or >> date.re.match(r'(?P\d\d)-(?P\d\d\d\d)'): >> print date.re.groupdict('year') >> > [snip] > Why not drop the ".re" part? You would, however, then need a new name > for the re split, eg "re_split". > > Or you could make the string the pattern, eg r'whatever(.*)'.match(s). +1 for re support built-in to strings. Michael Foord > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From jimjjewett at gmail.com Sat Jul 18 22:07:39 2009 From: jimjjewett at gmail.com (Jim Jewett) Date: Sat, 18 Jul 2009 16:07:39 -0400 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from PyPM's point of view] Message-ID: Michael Foord wrote: >> I agree. People with versioning issues *should* be using virtualenv. Tarek Ziad? replied (and several people later agreed) > Let's remove site-packages from Python then. What about those people *without* versioning issues? I have no qualms suggesting that package management programs avoid the use of site-packages. Such programs do need to cater to edge cases. But a single drop-it-in directory works great for the vast majority of *my* needs; requiring me to drink the Kool-Aid from a specific package management system just to get access to any add-ons -- even those I wrote myself in pure python -- would be a huge step backwards. -jJ From pje at telecommunity.com Sat Jul 18 22:22:37 2009 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 18 Jul 2009 16:22:37 -0400 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from PyPM's point of view] In-Reply-To: References: Message-ID: <20090718201933.B6AE63A403A@sparrow.telecommunity.com> At 04:07 PM 7/18/2009 -0400, Jim Jewett wrote: >Michael Foord wrote: > >> I agree. People with versioning issues *should* be using virtualenv. > >Tarek Ziad? replied (and several people later agreed) > > Let's remove site-packages from Python then. > >What about those people *without* versioning issues? > >I have no qualms suggesting that package management programs avoid the >use of site-packages. Such programs do need to cater to edge cases. > >But a single drop-it-in directory works great for the vast majority of >*my* needs; requiring me to drink the Kool-Aid from a specific package >management system just to get access to any add-ons -- even those I >wrote myself in pure python -- would be a huge step backwards. IIUC, the new "user" directories supported by the 'site' module would fill this purpose. That is, rather than having a site-wide packages directory, there'd just be the user-specific package directories. (That having been said, I'm not actually in favor of the idea.) From david.lyon at preisshare.net Sun Jul 19 02:49:30 2009 From: david.lyon at preisshare.net (David Lyon) Date: Sat, 18 Jul 2009 20:49:30 -0400 Subject: [Python-Dev] =?utf-8?q?Remove_site-packages=3F!=3F_=5Bwas=3A_=5BD?= =?utf-8?q?istutils=5D_PEP_376_-_from_PyPM=27s_point_of_view=5D?= In-Reply-To: References: Message-ID: On Sat, 18 Jul 2009 16:07:39 -0400, Jim Jewett wrote: > What about those people *without* versioning issues? > > I have no qualms suggesting that package management programs avoid the > use of site-packages. Such programs do need to cater to edge cases. I'm just wondering how that is possible? Given that a package management system (pythonpkgmgr - for example) is a tool to assist in the management of packages in site-packages. If you take away the drop in directory (site-packages), effectively you're taking away the place for system drop in packages. I can't see the point in that. > But a single drop-it-in directory works great for the vast majority of > *my* needs; requiring me to drink the Kool-Aid from a specific package > management system just to get access to any add-ons -- even those I > wrote myself in pure python -- would be a huge step backwards. Where you keep your packages is entirely up to you. Even with the Python Package Manager, you can still have unversioned packages in project directories. That doesn't change. And you can still have unversioned packages in site-packages. So I can't really see that having a Package Manager program takes your use case scenerio backwards because it wouldn't really change it. (repeating) > But a single drop-it-in directory works great for the vast majority of > *my* needs; That's exactly what site-packages is..... So it isn't clear why you want to remove the thing that you are advocating works so great.... David From ncoghlan at gmail.com Sun Jul 19 08:07:28 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 19 Jul 2009 16:07:28 +1000 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from PyPM's point of view] In-Reply-To: References: Message-ID: <4A62B820.4030903@gmail.com> David Lyon wrote: > So it isn't clear why you want to remove the thing that you are > advocating works so great.... Jim was quoting someone *else* that had suggested removing it (I'm not sure how serious the original suggestion actually was though). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From rdmurray at bitdance.com Sun Jul 19 15:08:20 2009 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 19 Jul 2009 09:08:20 -0400 (EDT) Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from PyPM's point of view] In-Reply-To: <4A62B820.4030903@gmail.com> References: <4A62B820.4030903@gmail.com> Message-ID: On Sun, 19 Jul 2009 at 16:07, Nick Coghlan wrote: > David Lyon wrote: >> So it isn't clear why you want to remove the thing that you are >> advocating works so great.... > > Jim was quoting someone *else* that had suggested removing it (I'm not > sure how serious the original suggestion actually was though). It was Tarek, and I'm pretty sure it was in the nature of what rhetoricians (among others) call "reductio ad absurdum". But Tarek would have to let us know for sure, since it seems from subsequent discussion that the notion is not _completely_ absurd, just mostly :) --David From skippy.hammond at gmail.com Sun Jul 19 15:27:19 2009 From: skippy.hammond at gmail.com (Mark Hammond) Date: Sun, 19 Jul 2009 23:27:19 +1000 Subject: [Python-Dev] Mercurial migration: progress report (PEP 385) In-Reply-To: <4A4ED492.1050307@gmail.com> References: <4A4D3C44.60504@gmail.com> <4A4E0825.5020407@skippinet.com.au> <4A4E1009.4050901@gmail.com> <4A4E9494.2040406@skippinet.com.au> <4A4EAE7A.5090406@gmail.com> <4A4EB152.6040406@skippinet.com.au> <4A4EBEBE.7040604@gmail.com> <4A4ED492.1050307@gmail.com> Message-ID: <4A631F37.6080601@gmail.com> Sorry Dirkjan - I just noticed I didn't CC you on this mail originally. I'm wondering if you have any more thoughts on these EOL issues and if there is anything I can do to help? Cheers, Mark On 4/07/2009 2:03 PM, Mark Hammond wrote: > On 4/07/2009 12:30 PM, Nick Coghlan wrote: > ... >> I think there needs to be a solid answer in place for these use cases >> before the actual migration to Mercurial takes place. A hand-waving "use >> win32text" isn't enough - it needs to be "use win32text with these exact >> settings" (with server side hook support to enforce the rules). >> >> And since Mercurial doesn't even allow us to say "this is a binary file" >> the way CVS used to I'm currently not seeing any way for that to happen >> except for win32text to be updated to correctly handle wild cards in >> combination with negative filters. > > I agree with your conclusion. My ruminating on this over the last few > months leaves me thinking this would involve: > > * my older 'accepted but then lost' hg patch to allow an explicit 'none' > rule for a single file to override wildcards. > > * win32text be enhanced to use a normal versioned file in the root of > the repo, much like hgingore, where a project can maintain project wide > rules. > > * win32text be enhanced such that all python developers, regardless of > platform, are willing to use this extension, even if the majority of > files happen to use their native line ending (sauce for the goose is > sauce for the gander, and all that...) > > * commit hooks be implemented to enforce this - but this should not be > necessary if the above was implemented and socially enforced. > > Cheers, > > Mark >> >> Cheers, >> Nick. >> >> [1] http://mail.python.org/pipermail/python-dev/2006-March/062225.html >> > From ziade.tarek at gmail.com Sun Jul 19 17:15:44 2009 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 19 Jul 2009 17:15:44 +0200 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from PyPM's point of view] In-Reply-To: References: <4A62B820.4030903@gmail.com> Message-ID: <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> On Sun, Jul 19, 2009 at 3:08 PM, R. David Murray wrote: > On Sun, 19 Jul 2009 at 16:07, Nick Coghlan wrote: >> >> David Lyon wrote: >>> >>> So it isn't clear why you want to remove the thing that you are >>> advocating works so great.... >> >> Jim was quoting someone *else* that had suggested removing it (I'm not >> sure how serious the original suggestion actually was though). > > It was Tarek, and I'm pretty sure it was in the nature of what > rhetoricians (among others) call "reductio ad absurdum". ?But > Tarek would have to let us know for sure, since it seems from > subsequent discussion that the notion is not _completely_ absurd, > just mostly :) > Yes, well, maybe we should go back to Python-ideas to continue this discussion, :) What made me say that, is the unecessary complexity of the situation because we have right now : "Python the application" v.s. "Applications that uses Python the interpreter" So we have: A - a drop-it-in directory were we can put packages and modules B - installers that adds automatically packages and modules in the drop-it-in directory C - applications that uses these installers to install themselves and their dependencies in the drop-it-in directory. D - a tool like virtualenv that duplicates a Python installation so this drop-in-in directory is not shared globally C - installation documentation that tells you to use virtualenv to create a custom environment so you are sure not to break dependencies. (But you can break them nevertheless because using virtualenv is not enforced :) ) E - OS packagers that installs all distributions in the drop-it-in directory (or a similar place) So if you remove the global site-packages, "Python the application" dissapear in favor of "Python the interpreter", and people have to point a local, non-shared drop-it-in directory to run any python application that uses extra modules: 1- For Jim's case, the per-user site packages (PEP 370) can be used 2- For every application it can be a local directory loaded in sys.path at startup 3- For OS packagers a single directory where they maintain a collection of packages and modules for their system. But unfortunately, the problem remains because of (3) : OS packagers will fight against applications (2) that deal with their own dependencies to avoid having duplicate packages in their system. That's why a shared space, not loaded at startup by site.py, that knows how to handle multiple versions for the same distribution would be required. Every script (eg application) would decide which version to use, or use the default version by default. And a smart PEP 302 loader/importer would get the right versions. Regards Tarek From david.lyon at preisshare.net Mon Jul 20 01:13:30 2009 From: david.lyon at preisshare.net (David Lyon) Date: Sun, 19 Jul 2009 19:13:30 -0400 Subject: [Python-Dev] =?utf-8?q?Remove_site-packages=3F!=3F_=5Bwas=3A_=5BD?= =?utf-8?q?istutils=5D_PEP_376_-_from_pythonpkgmgr=27s_point_of_view=5D?= In-Reply-To: <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> Message-ID: <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> On Sun, 19 Jul 2009 17:15:44 +0200, Tarek Ziad? wrote: > What made me say that, is the unecessary complexity of the situation > because we have right now : > > .. > > So if you remove the global site-packages, "Python the application" > dissapear in favor of "Python the interpreter", > and people have to point a local, non-shared drop-it-in directory to > run any python application that uses extra modules: > > 1- For Jim's case, the per-user site packages (PEP 370) can be used > 2- For every application it can be a local directory loaded in > sys.path at startup > 3- For OS packagers a single directory where they maintain a > collection of packages and modules for their system. > > But unfortunately, the problem remains because of (3) : OS packagers > will fight against applications (2) that deal with their own > dependencies to avoid having duplicate packages in their system. Well now I completely understand the jist of what you are saying.. And I completely agree with the direction of your comments. This problem is being solved within the pythonpkgmgr project. As we are allowing a distinction between operating system packages and developer packages. I only have ubuntu, and not OS-X at the moment, but let me explain with that. The operating system can install packages into /usr/lib/pythonX.X/ site-packages. The develeloper can install packages into /usr/local/pythonX.x/ site-packages. Therefore providing two seperate areas for packages to reside and eliminating the chance for serious conflict. The main problem imho has been that easy_install by default has installed to /usr/lib/pythonX.X/site-packages when in fact that really is an operating system controlled directory. So naughty naughty to setuptools.... That's why in pythonpkgmgr, we're moving to an entirely different place where there is no chance of conflict with the os package manager and python packages can be installed without su privelages. If the user wishes to install o/s packages that's fine and if they wish to install developer packages that's fine too. The saving grace here is the gui that remembers the appropriate command line switches that are needed to direct this behaviour because it doesn't happen by default in easy_install. Under ubuntu, it seems to be /usr/local/pythonX.X/site-packages and I am checking other posix operating systems and trying to work out what the equivilents might be. When I have it working.. I will let you know.. I'm just trying to work out how to get the permissions going correctly. David From mike.klaas at gmail.com Mon Jul 20 20:26:50 2009 From: mike.klaas at gmail.com (Mike Klaas) Date: Mon, 20 Jul 2009 11:26:50 -0700 Subject: [Python-Dev] pthreads, fork, import, and execvp In-Reply-To: <9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com> References: <44684958.9010206@gmail.com> <446973F0.3010007@v.loewis.de> <44699B47.4030504@gmail.com> <446A4202.6070608@v.loewis.de> <446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com> <446CB69E.4090609@v.loewis.de> <9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com> Message-ID: <3d2ce8cb0907201126t2ce8866eq3580aed9e3e6bf2@mail.gmail.com> On Thu, Jul 16, 2009 at 1:08 PM, Thomas Wouters wrote: > > Picking up a rather old discussion... We encountered this bug at Google and > I'm now "incentivized" to fix it. > > For a short recap: Python has an import lock that prevents more than one > thread from doing an import at any given time. However, unlike most of the > locks we have lying around, we don't clear that lock in the child after an > os.fork(). That means that doing an os.fork() during an import means the > child process can't do any other imports. It also means that doing an > os.fork() *while another thread is doing an import* means the child process > can't do any other imports. > > Since this three-year-old discussion we've added a couple of > post-fork-cleanups to CPython (the TLS, the threading module's idea of > active threads, see Modules/signalmodule.c:PyOS_AfterFork) and we already do > simply discard the memory for other locks held during fork (the GIL, see > Python/ceval.c:PyEval_ReInitThreads, and the TLS lock in > Python/thread.c:PyThread_ReInitTLS) -- but not so with the import lock, > except when the platform is AIX. I don't see any particular reason why we > aren't doing the same thing to the import lock that we do to the other > locks, on all platforms. It's a quick fix for a real problem (see > http://bugs.python.org/issue1590864 and > http://bugs.python.org/issue1404925 for two bugreports that seem to be > this very issue.) > +1. We were also affected by this bug, getting sporatic deadlocks in a multi-threaded program that fork()s subprocesses to do processing. It took a while to figure out what was going on. -Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From thomas at python.org Tue Jul 21 02:16:30 2009 From: thomas at python.org (Thomas Wouters) Date: Mon, 20 Jul 2009 17:16:30 -0700 Subject: [Python-Dev] pthreads, fork, import, and execvp In-Reply-To: <3d2ce8cb0907201126t2ce8866eq3580aed9e3e6bf2@mail.gmail.com> References: <446973F0.3010007@v.loewis.de> <44699B47.4030504@gmail.com> <446A4202.6070608@v.loewis.de> <446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com> <446CB69E.4090609@v.loewis.de> <9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com> <3d2ce8cb0907201126t2ce8866eq3580aed9e3e6bf2@mail.gmail.com> Message-ID: <9e804ac0907201716v584e3337t3be9a7bf3311ff3b@mail.gmail.com> On Mon, Jul 20, 2009 at 11:26, Mike Klaas wrote: > > > On Thu, Jul 16, 2009 at 1:08 PM, Thomas Wouters wrote: > >> >> Picking up a rather old discussion... We encountered this bug at Google >> and I'm now "incentivized" to fix it. >> >> For a short recap: Python has an import lock that prevents more than one >> thread from doing an import at any given time. However, unlike most of the >> locks we have lying around, we don't clear that lock in the child after an >> os.fork(). That means that doing an os.fork() during an import means the >> child process can't do any other imports. It also means that doing an >> os.fork() *while another thread is doing an import* means the child process >> can't do any other imports. >> >> Since this three-year-old discussion we've added a couple of >> post-fork-cleanups to CPython (the TLS, the threading module's idea of >> active threads, see Modules/signalmodule.c:PyOS_AfterFork) and we already do >> simply discard the memory for other locks held during fork (the GIL, see >> Python/ceval.c:PyEval_ReInitThreads, and the TLS lock in >> Python/thread.c:PyThread_ReInitTLS) -- but not so with the import lock, >> except when the platform is AIX. I don't see any particular reason why we >> aren't doing the same thing to the import lock that we do to the other >> locks, on all platforms. It's a quick fix for a real problem (see >> http://bugs.python.org/issue1590864 and >> http://bugs.python.org/issue1404925 for two bugreports that seem to be >> this very issue.) >> > > +1. We were also affected by this bug, getting sporatic deadlocks in a > multi-threaded program that fork()s subprocesses to do processing. > It took a while to figure out what was going on. > Actually, after careful consideration I have come to realize that simply resetting the lock is exactly the wrong thing to do. The import lock exists to prevent threads from getting a half-initialized module *while another module is creating the module*. That is to say, if threads 'A' and 'B' both import 'mod', it is not only important that threads A and B don't *both* try to execute mod.py, but also that thread B doesn't get a half-initialized module object that thread A is busy populating. if the import lock is held by the thread doing fork(), this is not a problem: the module is still being imported in the child process, and the fork doesn't affect it. If the import lock is held by another thread than the one doing fork(), any partially imported modules would remain in sys.modules in the child process, without ever getting finished. So we need to actually acquire the import lock before forking. We can do that in os.fork() and os.forkpty(), but we can't fix third-party extension modules; we'll have to introduce a new set of API functions, for getting and releasing the import lock. I would suggest we don't expose it as that, but instead call it a fork lock or such, so we can add extra lock acquire/release pairs as necessary. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From lanyjie at yahoo.com Tue Jul 21 06:19:10 2009 From: lanyjie at yahoo.com (Yingjie Lan) Date: Mon, 20 Jul 2009 21:19:10 -0700 (PDT) Subject: [Python-Dev] expy: an expressway to extend Python Message-ID: <290588.53242.qm@web54205.mail.re2.yahoo.com> Hi All, This is to announce the initial release of expy 0.1.0. More details at http://expy.sourceforge.net/ Thanks! Yingjie From phd at phd.pp.ru Tue Jul 21 08:44:04 2009 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 21 Jul 2009 10:44:04 +0400 Subject: [Python-Dev] expy: an expressway to extend Python In-Reply-To: <290588.53242.qm@web54205.mail.re2.yahoo.com> References: <290588.53242.qm@web54205.mail.re2.yahoo.com> Message-ID: <20090721064404.GA3038@phd.pp.ru> On Mon, Jul 20, 2009 at 09:19:10PM -0700, Yingjie Lan wrote: > This is to announce the initial release of expy 0.1.0. > More details at http://expy.sourceforge.net/ What is it (the announcement is too brief and I'm not going to click on a link without a real need)? How does it help core developers to develop Python (if you posted this to the python-dev mailing list instead of c.l.p or c.l.p.a)? Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From john at szakmeister.net Tue Jul 21 11:44:03 2009 From: john at szakmeister.net (John Szakmeister) Date: Tue, 21 Jul 2009 05:44:03 -0400 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> Message-ID: On Sun, Jul 19, 2009 at 7:13 PM, David Lyon wrote: [snip] > The main problem imho has been that easy_install by default > has installed to /usr/lib/pythonX.X/site-packages when in > fact that really is an operating system controlled directory. > > So naughty naughty to setuptools.... I've witnessed this a few times... and I completely agree with that statement. :-) > That's why in pythonpkgmgr, we're moving to an entirely > different place where there is no chance of conflict with > the os package manager and python packages can be installed > without su privelages. [snip > Under ubuntu, it seems to be /usr/local/pythonX.X/site-packages > and I am checking other posix operating systems and trying to > work out what the equivilents might be. Perhaps I'm not reading what you're saying correctly. Doesn't /usr/local/pythonX.x/site-packages require su privileges to install? FWIW, the equivalent of /usr/local/pythonX.x/site-packages on Mac OS X seems to be /Library/Python/X.x/site-packages -John From tseaver at palladion.com Tue Jul 21 19:41:41 2009 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 21 Jul 2009 13:41:41 -0400 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 John Szakmeister wrote: > On Sun, Jul 19, 2009 at 7:13 PM, David Lyon wrote: > [snip] >> The main problem imho has been that easy_install by default >> has installed to /usr/lib/pythonX.X/site-packages when in >> fact that really is an operating system controlled directory. >> >> So naughty naughty to setuptools.... > > I've witnessed this a few times... and I completely agree with that > statement. :-) That behvior isn't setuptools' fault: it is the behavior of *any* distutils-derived app. "OS controlled directory" is not a concept which the distutils honors at all: $prefix/lib/pytoonX.Y/site-packages is the *intended* installation location for any modules not shipped with Python. Consider the case where the user has built a separate python (e.g., to avoid conflicting with the OS version) in /opt/PythonX.y: are you actually saying that distutils / setuptools should somehow know the difference betweeen /usr/lib/pythonX.y/site-packages and /opt/PythonX.Y/lib/pythonX.Y/site-packages? Or are you asserting that it is somehow invalid for distutils / setuptools to install things into /opt/PythonX.Y/lib/pythonX.y/site-packages, which is certainly not an "OS controlled directory." Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKZf3V+gerLs4ltQ4RAud7AJ9GG9NY4UaKpQ5bBIQ+N6tztPVJSgCdEEIa DUugfdzvBYgPtaas9c+fTTg= =gvLY -----END PGP SIGNATURE----- From lanyjie at yahoo.com Tue Jul 21 20:22:19 2009 From: lanyjie at yahoo.com (Yingjie Lan) Date: Tue, 21 Jul 2009 11:22:19 -0700 (PDT) Subject: [Python-Dev] expy: an expressway to extend Python Message-ID: <765806.59082.qm@web54202.mail.re2.yahoo.com> Hi, Here is a brief example on how to use expy to implement the math module: (for more details, see http://expy.sf.net/) """Python math module by expy-cxpy.""" from expy import * expymodule(__name__) #includes, defines, etc. @prologue def myprolog(): return """ #include """ @function(double) #return type: double def sqrt(x=double): #argument x: double """sqrt(x) --> the square root of x.""" return "sqrt(x)" #as an expression #a more terse way @function(double) def sin(x=double): """sin(x) --> the sin of x.""" pass #the deduced call: sin(x) #more functions ... expymodule(__name__) #end of module From steve at holdenweb.com Tue Jul 21 20:34:59 2009 From: steve at holdenweb.com (Steve Holden) Date: Tue, 21 Jul 2009 14:34:59 -0400 Subject: [Python-Dev] Support for Python/Windows Message-ID: <4A660A53.4050806@holdenweb.com> Devs: I've been in correspondence with Microsoft about the provision of software, and it transpires that if you want to support Windows better Microsoft will be quite liberal about licensing: they will *give* you a Microsoft Developer Network license. If you are interested in offering better Windows support then please read the email below (note: Windows buildbot support would be a qualifying activity) and let me have the required details. I will pass them to Tom in bulk to simplify the processing. Note that I'm not following python-dev right now due to pressure of work, so PLEASE EMAIL ME DIRECTLY (or Cc me on your list replies) to make sure I get your information. Thanks! regards Steve -------- Original Message -------- Subject: RE: Support for Python: Windows Buildbots Date: Tue, 7 Jul 2009 08:52:10 -0700 From: Tom Hanrahan For the purposes of providing MSDN licenses to an open source development community, I consider anyone who writes, builds, tests or documents software to be a "developer who contributes" to the project. (In fact, having started out as a test engineer, I would take exception to anyone who claimed only people who write code are "developers" :-) We do ask that requests are for people who are active contributors and not just minor/occasional participants. [...] Here's what we need for each request: First Name Last Name Email Address (the subscription will be sent here, and this will also be used to log into the MSDN site) Project/Company (Python Software Foundation) Complete Mailing Address (Street Address) (City), (State/Province) (Postal or Zip Code) (Country) Phone Number -- Tom -----Original Message----- From: Steve Holden [mailto:steve at holdenweb.com] Sent: Tuesday, July 07, 2009 6:01 AM To: Tom Hanrahan Cc: Anandeep Pannu; Pat Campbell; Python Board; Jim Hugunin Subject: Re: Support for Python: Windows Buildbots Tom: Further to Sam's email, in fact the original inquiry was instituted by the need of our part-time administrator to acquire an Office license. I am guessing she wouldn't qualify as an Open Source Developer, but that least naturally to the more interesting question of who would. A Bing search for "Microsoft Open Source Developer Program" didn't yield any usable hits, so it might be helpful if you could point me to some web resources that will help me make sense of what's available, who's eligible and how they apply for it. I will be happy to publicize the details to the development team. It's true, I believe, that most of the core Python developers use Linux, but both Tim Peters and I are primarily on the Windows platform. What's more, with the emergence of virtualization environments having Linux on your desktop is no hindrance to running Windows in a virtual machine (I run Linux on virtuals when appropriate). So let's take it from here and see where we go. regards Steve Sam Ramji wrote: > Steve: > > If the PSF's Windows users are developers who contribute to Python, we can offer them MSDN subscriptions as part of the Open Source Developer program. > > MSDN subscriptions include copies of most Microsoft products (including Office and Exchange) for use while developing and testing software. For more details, check here - we provide Visual Studio Pro with MSDN Premium under this program (http://msdn.microsoft.com/en-us/subscriptions/subscriptionschart.aspx). > > Tom Hanrahan is the Director of the Open Source Technology Center, and is the sponsor of the Open Source Developer program. I've copied him here - please contact him directly with the details of the people who would like to participate. He's at hanrahat at microsoft.com. > > We definitely want to make Windows a friendly place for Python developers! > > Cheers, > > Sam > [...] -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Watch PyCon on video now! http://pycon.blip.tv/ From phd at phd.pp.ru Tue Jul 21 20:52:03 2009 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 21 Jul 2009 22:52:03 +0400 Subject: [Python-Dev] expy: an expressway to extend Python In-Reply-To: <765806.59082.qm@web54202.mail.re2.yahoo.com> References: <765806.59082.qm@web54202.mail.re2.yahoo.com> Message-ID: <20090721185203.GA16546@phd.pp.ru> On Tue, Jul 21, 2009 at 11:22:19AM -0700, Yingjie Lan wrote: > @function(double) #return type: double > def sqrt(x=double): #argument x: double Python 3.0 has arguments and return value annotations: http://docs.python.org/3.0/whatsnew/3.0.html#new-syntax http://www.python.org/dev/peps/pep-3107/ Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From eric at ericentin.com Tue Jul 21 22:26:52 2009 From: eric at ericentin.com (Eric Entin) Date: Tue, 21 Jul 2009 16:26:52 -0400 Subject: [Python-Dev] expy: an expressway to extend Python In-Reply-To: <20090721185203.GA16546@phd.pp.ru> References: <765806.59082.qm@web54202.mail.re2.yahoo.com> <20090721185203.GA16546@phd.pp.ru> Message-ID: I think the point of his software is to make it easier to interface Python with C code, although his examples were probably not the best. Eric Entin On Tue, Jul 21, 2009 at 2:52 PM, Oleg Broytmann wrote: > On Tue, Jul 21, 2009 at 11:22:19AM -0700, Yingjie Lan wrote: > > @function(double) #return type: double > > def sqrt(x=double): #argument x: double > > Python 3.0 has arguments and return value annotations: > > http://docs.python.org/3.0/whatsnew/3.0.html#new-syntax > http://www.python.org/dev/peps/pep-3107/ > > Oleg. > -- > Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/eric%40ericentin.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phd.pp.ru Tue Jul 21 22:45:10 2009 From: phd at phd.pp.ru (Oleg Broytmann) Date: Wed, 22 Jul 2009 00:45:10 +0400 Subject: [Python-Dev] expy: an expressway to extend Python In-Reply-To: References: <765806.59082.qm@web54202.mail.re2.yahoo.com> <20090721185203.GA16546@phd.pp.ru> Message-ID: <20090721204510.GA22163@phd.pp.ru> On Tue, Jul 21, 2009 at 04:26:52PM -0400, Eric Entin wrote: > I think the point of his software is to make it easier to interface Python > with C code I think I understand that. And I think this > > > @function(double) #return type: double > > > def sqrt(x=double): #argument x: double is how C functions are declared in Python, so I think annotations is the way to go for such declarations. > > Python 3.0 has arguments and return value annotations: > > > > http://docs.python.org/3.0/whatsnew/3.0.html#new-syntax > > http://www.python.org/dev/peps/pep-3107/ Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From lists at cheimes.de Tue Jul 21 23:36:50 2009 From: lists at cheimes.de (Christian Heimes) Date: Tue, 21 Jul 2009 23:36:50 +0200 Subject: [Python-Dev] Support for Python/Windows In-Reply-To: <4A660A53.4050806@holdenweb.com> References: <4A660A53.4050806@holdenweb.com> Message-ID: <4A6634F2.1040003@cheimes.de> Steve Holden wrote: > Devs: > > I've been in correspondence with Microsoft about the provision of > software, and it transpires that if you want to support Windows better > Microsoft will be quite liberal about licensing: they will *give* you a > Microsoft Developer Network license. > > If you are interested in offering better Windows support then please > read the email below (note: Windows buildbot support would be a > qualifying activity) and let me have the required details. I will pass > them to Tom in bulk to simplify the processing. [...] >> MSDN subscriptions include copies of most Microsoft products (including Office and Exchange) for use while developing and testing software. For more details, check here - we provide Visual Studio Pro with MSDN Premium under this program (http://msdn.microsoft.com/en-us/subscriptions/subscriptionschart.aspx). Thanks you for getting in touch with Microsoft. The deal is worth a fortune for any Windows developer! Does the MSDN subscription also include the permission to create and release binaries? Sam Ramji wrote just "developing and testing". Me and probably all other subscribers like to use the MSDN subscription to build Windows binaries of Python and Python related extensions. Can you please verify that we are allowed to use the subscription for that purpose, too? Christian From curt at hagenlocher.org Tue Jul 21 23:53:20 2009 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Tue, 21 Jul 2009 14:53:20 -0700 Subject: [Python-Dev] Support for Python/Windows In-Reply-To: <4A6634F2.1040003@cheimes.de> References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de> Message-ID: On Tue, Jul 21, 2009 at 2:36 PM, Christian Heimes wrote: > > Does the MSDN subscription also include the permission to create and > release binaries? Sam Ramji wrote just "developing and testing". Me and > probably all other subscribers like to use the MSDN subscription to > build Windows binaries of Python and Python related extensions. Can you > please verify that we are allowed to use the subscription for that > purpose, too? Considering that the compilers are all freely downloadable as part of the Windows SDK, it would be weird if the subscription were *more* restrictive than what you can get without it. Disclaimer: I work for Microsoft, but eh, I'm just guessing. -- Curt Hagenlocher curt at hagenlocher.org From lists at cheimes.de Wed Jul 22 00:04:05 2009 From: lists at cheimes.de (Christian Heimes) Date: Wed, 22 Jul 2009 00:04:05 +0200 Subject: [Python-Dev] Support for Python/Windows In-Reply-To: References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de> Message-ID: <4A663B55.7000500@cheimes.de> Curt Hagenlocher wrote: > Considering that the compilers are all freely downloadable as part of > the Windows SDK, it would be weird if the subscription were *more* > restrictive than what you can get without it. > > Disclaimer: I work for Microsoft, but eh, I'm just guessing. Some features like PGO aren't available in VS Express Edition or Windows SDK. I'm not sure about all aspects of X86_64 builds, too. I'm prefer better safe than sorry. Disclaimer: I would never disclose that I'm a tiny bit paranoid ... :) Christian From steve at holdenweb.com Wed Jul 22 00:32:20 2009 From: steve at holdenweb.com (Steve Holden) Date: Tue, 21 Jul 2009 18:32:20 -0400 Subject: [Python-Dev] Support for Python/Windows In-Reply-To: <4A6634F2.1040003@cheimes.de> References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de> Message-ID: <4A6641F4.9000200@holdenweb.com> Christian Heimes wrote: > Steve Holden wrote: >> Devs: >> >> I've been in correspondence with Microsoft about the provision of >> software, and it transpires that if you want to support Windows better >> Microsoft will be quite liberal about licensing: they will *give* you a >> Microsoft Developer Network license. >> >> If you are interested in offering better Windows support then please >> read the email below (note: Windows buildbot support would be a >> qualifying activity) and let me have the required details. I will pass >> them to Tom in bulk to simplify the processing. > > [...] > >>> MSDN subscriptions include copies of most Microsoft products (including Office and Exchange) for use while developing and testing software. For more details, check here - we provide Visual Studio Pro with MSDN Premium under this program (http://msdn.microsoft.com/en-us/subscriptions/subscriptionschart.aspx). > > Thanks you for getting in touch with Microsoft. The deal is worth a > fortune for any Windows developer! > > Does the MSDN subscription also include the permission to create and > release binaries? Sam Ramji wrote just "developing and testing". Me and > probably all other subscribers like to use the MSDN subscription to > build Windows binaries of Python and Python related extensions. Can you > please verify that we are allowed to use the subscription for that > purpose, too? > I'll ask. I don't see why not (it would hardly be in Microsoft's interest to help us create unreleasable open source projects, would it?) regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ Watch PyCon on video now! http://pycon.blip.tv/ From curt at hagenlocher.org Wed Jul 22 01:08:34 2009 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Tue, 21 Jul 2009 16:08:34 -0700 Subject: [Python-Dev] Support for Python/Windows In-Reply-To: <4A663B55.7000500@cheimes.de> References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de> <4A663B55.7000500@cheimes.de> Message-ID: On Tue, Jul 21, 2009 at 3:04 PM, Christian Heimes wrote: > > Some features like PGO aren't available in VS Express Edition or Windows > SDK. I'm not sure about all aspects of X86_64 builds, too. I'm prefer > better safe than sorry. Ah, you're right -- the PGO bits probably need VS Pro. The 64-bit compilers should be in the Windows SDK, but it wouldn't surprise me if they were not included in Express. -- Curt Hagenlocher curt at hagenlocher.org From nyamatongwe at gmail.com Wed Jul 22 01:19:29 2009 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Wed, 22 Jul 2009 09:19:29 +1000 Subject: [Python-Dev] Support for Python/Windows In-Reply-To: References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de> <4A663B55.7000500@cheimes.de> Message-ID: <50862ebd0907211619p6c8b71edxcfd1307d4c83a835@mail.gmail.com> Curt Hagenlocher: > Ah, you're right -- the PGO bits probably need VS Pro. The 64-bit > compilers should be in the Windows SDK, but it wouldn't surprise me if > they were not included in Express. 64-bit isn't in Express and merging the 64 bit compiler from the SDK into Express may be possible but certainly isn't easy. I just use the command line compiler to check 64 bit issues. Neil From david.lyon at preisshare.net Wed Jul 22 01:38:20 2009 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 21 Jul 2009 19:38:20 -0400 Subject: [Python-Dev] =?utf-8?q?Remove_site-packages=3F!=3F_=5Bwas=3A_=5BD?= =?utf-8?q?istutils=5D_PEP_376_-_from_pythonpkgmgr=27s_point_of_view=5D?= In-Reply-To: References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> Message-ID: <742787a67a8690d390c83a11a10aeb2a@preisshare.net> On Tue, 21 Jul 2009 13:41:41 -0400, Tres Seaver wrote: >>> The main problem imho has been that easy_install by default >>> has installed to /usr/lib/pythonX.X/site-packages when in >>> fact that really is an operating system controlled directory. >>> >>> So naughty naughty to setuptools.... .. > That behvior isn't setuptools' fault: it is the behavior of *any* > distutils-derived app. "OS controlled directory" is not a concept which > the distutils honors at all: $prefix/lib/pytoonX.Y/site-packages is the > *intended* installation location for any modules not shipped with Python. Ok - so naughty distutils... > Consider the case where the user has built a separate python (e.g., to > avoid conflicting with the OS version) in /opt/PythonX.y: are you > actually saying that distutils / setuptools should somehow know the > difference betweeen /usr/lib/pythonX.y/site-packages and > /opt/PythonX.Y/lib/pythonX.Y/site-packages? It already does. If you run the python in /opt/pythonX.X you'll get a different value for sys.path than when you run the other python from /usr/lib/pythonX.X. Try it and see for yourself.. That's because sys.path is calculated relatively within site.py depending on where the interpretor is run from. > Or are you asserting that > it is somehow invalid for distutils / setuptools to install things into > /opt/PythonX.Y/lib/pythonX.y/site-packages, which is certainly not an > "OS controlled directory." Not at all. All I'm saying is that after so many years of python under linux and operating system we should have observed the common use cases and learned a thing or two. If you hadn't noticed, there is mayhem in package installation because the operating system tries to install its packages in the same place as setuptools does in development mode. That has got to be asking for trouble.. When I go into python on ubuntu I see there is /usr/local/pythonX.X/lib/ site-packages and I'm wondering why the hubba setuptools/distutils doesn't put packages there by default. That would solve a lot of problems. Just leave /usr/lib/pythonX.X//lib/site-packages to the O/S. Wrapping up.. most of the functionality that I'm asking for is already there. So I don't require anything extra. I don't want things taken out. All I'm suggesting is that we use what we already have a bit smarter. David From lanyjie at yahoo.com Wed Jul 22 03:15:45 2009 From: lanyjie at yahoo.com (Yingjie Lan) Date: Tue, 21 Jul 2009 18:15:45 -0700 (PDT) Subject: [Python-Dev] expy: an expressway to extend Python Message-ID: <728371.42493.qm@web54201.mail.re2.yahoo.com> --- On Wed, 7/22/09, Oleg Broytmann wrote: > From: Oleg Broytmann > Subject: Re: [Python-Dev] expy: an expressway to extend Python > To: python-dev at python.org > Date: Wednesday, July 22, 2009, 12:45 AM > On Tue, Jul 21, 2009 at 04:26:52PM > -0400, Eric Entin wrote: > > I think the point of his software is to make it easier > to interface Python > > with C code > > ???I think I understand that. And I think > this > > > > > @function(double) #return type: double > > > > def sqrt(x=double): #argument x: double > > ???is how C functions are declared in > Python, so I think annotations is the > way to go for such declarations. > > > >? ? Python 3.0 has arguments and return > value annotations: > > > > > > http://docs.python.org/3.0/whatsnew/3.0.html#new-syntax > > > http://www.python.org/dev/peps/pep-3107/ > Thanks, I think that is a brilliant suggestion, when expy is implemented for Python 3.0, this will greatly improve readability, and make it more like a natural part of Python. Yingjie From foom at fuhm.net Wed Jul 22 03:46:11 2009 From: foom at fuhm.net (James Y Knight) Date: Tue, 21 Jul 2009 21:46:11 -0400 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: <742787a67a8690d390c83a11a10aeb2a@preisshare.net> References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> Message-ID: On Jul 21, 2009, at 7:38 PM, David Lyon wrote: > When I go into python on ubuntu I see there is /usr/local/pythonX.X/ > lib/ > site-packages and I'm wondering why the hubba setuptools/distutils > doesn't put packages there by default. That would solve a lot of > problems. > > Just leave /usr/lib/pythonX.X//lib/site-packages to the O/S. Uh guys, I'm not sure if anyone here noticed, but Debian and Ubuntu have switched to install their distribution-supplied python libraries into: /usr/lib/pythonX.Y/lib/dist-packages and distutils by default will install into /usr/local/lib/pythonX.Y/dist-packages starting with python 2.6. See: http://lists.debian.org/debian-devel/2009/02/msg00431.html Since that email says "Discussed this with Barry Warsaw and Martin v. Loewis", I'd assume this change would be more widely known in the distutils/python-dev community, but apparently not?? James From david.lyon at preisshare.net Wed Jul 22 03:57:46 2009 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 21 Jul 2009 21:57:46 -0400 Subject: [Python-Dev] =?utf-8?q?Remove_site-packages=3F!=3F_=5Bwas=3A_=5BD?= =?utf-8?q?istutils=5D_PEP_376_-_from_pythonpkgmgr=27s_point_of_view=5D?= In-Reply-To: References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> Message-ID: On Tue, 21 Jul 2009 21:46:11 -0400, James Y Knight wrote: > Uh guys, I'm not sure if anyone here noticed, but Debian and Ubuntu > have switched to install their distribution-supplied python libraries > into: > /usr/lib/pythonX.Y/lib/dist-packages > and distutils by default will install into > /usr/local/lib/pythonX.Y/dist-packages > > starting with python 2.6. > .. > Since that email says "Discussed this with Barry Warsaw and Martin v. > Loewis", I'd assume this change would be more widely known in the > distutils/python-dev community, but apparently not?? Forgive my ignorance... I'm working with 2.5 on ubuntu and didn't think to try 2.6... It's good to know that the problem is fixed in 2.6 but I still need to make things work for pythonpkgmgr on ubuntu py2.5 as 2.5 is the default for ubuntu 8.10. Take care David From mal at egenix.com Wed Jul 22 10:49:20 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 22 Jul 2009 10:49:20 +0200 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> Message-ID: <4A66D290.9050103@egenix.com> James Y Knight wrote: > On Jul 21, 2009, at 7:38 PM, David Lyon wrote: >> When I go into python on ubuntu I see there is /usr/local/pythonX.X/lib/ >> site-packages and I'm wondering why the hubba setuptools/distutils >> doesn't put packages there by default. That would solve a lot of >> problems. >> >> Just leave /usr/lib/pythonX.X//lib/site-packages to the O/S. > > Uh guys, I'm not sure if anyone here noticed, but Debian and Ubuntu have > switched to install their distribution-supplied python libraries into: > /usr/lib/pythonX.Y/lib/dist-packages > and distutils by default will install into > /usr/local/lib/pythonX.Y/dist-packages > > starting with python 2.6. > > See: > http://lists.debian.org/debian-devel/2009/02/msg00431.html > > Since that email says "Discussed this with Barry Warsaw and Martin v. > Loewis", I'd assume this change would be more widely known in the > distutils/python-dev community, but apparently not?? Debian has a long history of doing this different, so it's not much of a surprise. They also apply such changes to Python packages. However, all of this is non-standard and will cause problems with tools that rely on the standard site-packages/ location. Such changes should be discouraged. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 22 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From olemis at gmail.com Wed Jul 22 15:48:04 2009 From: olemis at gmail.com (Olemis Lang) Date: Wed, 22 Jul 2009 08:48:04 -0500 Subject: [Python-Dev] Support for Python/Windows In-Reply-To: <4A6641F4.9000200@holdenweb.com> References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de> <4A6641F4.9000200@holdenweb.com> Message-ID: <24ea26600907220648mf2ce2b4nad05d2a8c9d06349@mail.gmail.com> Hi ! On Tue, Jul 21, 2009 at 5:32 PM, Steve Holden wrote: > Christian Heimes wrote: >> Steve Holden wrote: >>> Devs: >>> >>> If you are interested in offering better Windows support then please >>> read the email below >> >> [...] >> >>>> MSDN subscriptions include copies of most Microsoft products (including Office and Exchange) for use while developing and testing software. ?For more details, check here - we provide Visual Studio Pro with MSDN Premium under this program (http://msdn.microsoft.com/en-us/subscriptions/subscriptionschart.aspx). >> >> Thanks you for getting in touch with Microsoft. The deal is worth a >> fortune for any Windows developer! >> >> Does the MSDN subscription also include the permission to create and >> release binaries? [...] >> Can you >> please verify that we are allowed to use the subscription for that >> purpose, too? >> > I'll ask. I don't see why not (it would hardly be in Microsoft's > interest to help us create unreleasable open source projects, would it?) > When talking about MS + FOSS everything is possible :-/ My question is the following : - What are the implications for Py users ? I mean, even if somebody (not me but enterprises & organizations I work or may work for in the future ;o) decides to use Windows pay for that and everything else, I'd not like to qualify as a "pirate" (or alike) for using a Py distribution or app including MS Intelectual Property (MSIP) (and MS loves MSIP -even if nobody can see it- and all kind of legal issues, especially with FOSS) nor even have Py in the middle of a patent dispute or something ... And they have some "great" innovations [1]_ to ensure (sometimes, I know) that (some) apps (who decides ?) wont run on a Win host. I could mention a lot of snippets in that text (yes it's very "interesting" and "substantial", and "useful" ) here goes one of them : {{{ According to another aspect of the invention, the digest catalog includes, for each program file corresponding to an application or driver that should be executable by the computer, a digitally signed hash value that is generated from a hash function based on the corresponding program file. When attempting to load a particular file, the loader generates a hash value and compares it to the decrypted hash values in the digest catalog. If the comparison results in no matches, then the corresponding program file (and thus the application or driver) is not loaded. }}} OTOH : - What are the implications for other devs (not core ;o) who use to download sources and try new things, or perhaps use Py code the way they want to solve an specific issue, or modify it somehow to experiment or learn something, or whatever ? - Will that affect contributions from ?future or potential? devs ? - Will they also need an MS license to see or compile (or whatever) the changes contributed by Py devs ? - What about if for some reason, a idea or impl or alg or snippet (or whatever) is propagated to GNU/Linux distributions and it's MSIP? (considering former disputes like ?Linux kernel violates 42 of MS patents?) ? .. [1] Restricted software and hardware usage on a computer (http://patft.uspto.gov/netacgi/nph-Parser?patentnumber=7,536,726) PD: My question is not technical at all but at least for me is important (even if I'm not a lawyer, nor a core Py dev ;o) since I manage (and develop ;o) several Py-based apps running on Win hosts in different locations . Finally I clearly see that this msg is strongly influenced by my biases, paranoia, and maybe I'm overreacting ... but I prefer to ask before things actually happen (and MS has a long history specially with FOSS + patents + legal affaires). I apologize in advance if I'm being rude or na?ve or * -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From eric.pruitt at gmail.com Wed Jul 22 16:01:19 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Wed, 22 Jul 2009 09:01:19 -0500 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile Message-ID: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com> Hello, I have written replacements for a few of Mark Hammond's PyWin32 functions using ctypes to call upon the Windows kernel API. Code can be found here; http://pastebin.com/m46de01f . When calling ReadFile, it appears that the kernel API converts '/n' newlines '/r/n' newlines. I have not been able to find any information about this but as far as I can tell, there is nothing in my code that is causing the problem. Initially I suspected it related to newline translation but the function in subprocess.Popen for translating newlines only converts to '/n' newlines. When I replaced my ReadFile and WriteFile functions with the PyWin32 functions I was imitating, the problem still arose. I was hoping someone could confirm this problem for me from experience or by testing out my code. If you would like the see the functions used in full context, I have a Mercurial Google Code project that can be checked out at http://code.google.com/p/subprocdev/source/list(branch "python3k"). My work entails modifications to subprocess.py so when running the code, please update your PYTHONPATH variable so that my subprocess.py will be imported. Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Wed Jul 22 16:26:59 2009 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 22 Jul 2009 10:26:59 -0400 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: <742787a67a8690d390c83a11a10aeb2a@preisshare.net> References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 David Lyon wrote: > On Tue, 21 Jul 2009 13:41:41 -0400, Tres Seaver > wrote: >> Consider the case where the user has built a separate python (e.g., to >> avoid conflicting with the OS version) in /opt/PythonX.y: are you >> actually saying that distutils / setuptools should somehow know the >> difference betweeen /usr/lib/pythonX.y/site-packages and >> /opt/PythonX.Y/lib/pythonX.Y/site-packages? > > It already does. > > If you run the python in /opt/pythonX.X you'll get a different value > for sys.path than when you run the other python from /usr/lib/pythonX.X. > > Try it and see for yourself.. You missed my point: I know that sys.prefix etc. is set based on the location of the executable. You were complaining that setuptools / distutils wants to put files in an "OS-controlled directory": I was asking how Python was supposed to know the difference between /usr/lib/python2.x/site-packages (presumably "OS-controlled") and /opt/Python2.x/lib/python2.x/site-packages (definitely not "OS-controlled"). The solution is to avoid doing distutils / setuptools operations as root, *ever*, and instead use either an alternate install location (such as /opt/Python2.x) or use something like virtualenv, which provides a "safe" sandbox for installing stuff. > That's because sys.path is calculated relatively within site.py > depending on where the interpretor is run from. > >> Or are you asserting that >> it is somehow invalid for distutils / setuptools to install things into >> /opt/PythonX.Y/lib/pythonX.y/site-packages, which is certainly not an >> "OS controlled directory." > > Not at all. > > All I'm saying is that after so many years of python under linux and > operating system we should have observed the common use cases and > learned a thing or two. > > If you hadn't noticed, there is mayhem in package installation because > the operating system tries to install its packages in the same place > as setuptools does in development mode. > > That has got to be asking for trouble.. Users who install anything into /usr/lib/python2.x in "development" mode has already invited all the trouble they can handle into the house. > When I go into python on ubuntu I see there is /usr/local/pythonX.X/lib/ > site-packages and I'm wondering why the hubba setuptools/distutils > doesn't put packages there by default. That would solve a lot of > problems. > > Just leave /usr/lib/pythonX.X//lib/site-packages to the O/S. > > Wrapping up.. most of the functionality that I'm asking for is > already there. So I don't require anything extra. I don't want > things taken out. > > All I'm suggesting is that we use what we already have a bit smarter. That is precisely how I think of virtualenv: it sits on top of distutils / setuptools, and makes it sane to do Python development without compromising the core system. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFKZyGz+gerLs4ltQ4RAupxAJ95RZQJm6t1FtcFY+DMyHHmwRjmSACglo+W TehyHzxUwHQ5LfdibJsgUcA= =nf13 -----END PGP SIGNATURE----- From david.lyon at preisshare.net Wed Jul 22 16:54:16 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 22 Jul 2009 10:54:16 -0400 Subject: [Python-Dev] =?utf-8?q?Remove_site-packages=3F!=3F_=5Bwas=3A_=5BD?= =?utf-8?q?istutils=5D_PEP_376_-_from_pythonpkgmgr=27s_point_of_view=5D?= In-Reply-To: References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> Message-ID: On Wed, 22 Jul 2009 10:26:59 -0400, Tres Seaver wrote: > You were complaining that setuptools / > distutils wants to put files in an "OS-controlled directory": I complain about many things - so what. > I was > asking how Python was supposed to know the difference between > /usr/lib/python2.x/site-packages (presumably "OS-controlled") and > /opt/Python2.x/lib/python2.x/site-packages (definitely not > "OS-controlled"). Well, at an interpreter level it doesn't, and I don't expect it to. > The solution is to avoid doing distutils / setuptools operations as > root, *ever*, and instead use either an alternate install location (such > as /opt/Python2.x) Have you tried this ? When I try to run it without su privelages setuptools crashes... (ubuntu 8.10) ------------------------------------------------------------------------------------------------ Running installer ... /usr/bin/easy_install "/media/Elements/Pythn Packages/html5lib-0.10.zip" ERRORS: Traceback (most recent call last): File "/usr/bin/easy_install", line 5, in from pkg_resources import load_entry_point ImportError: No module named pkg_resources Error in sys.excepthook: Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/apport_python_hook.py", line 78, in apport_excepthook report_file = open(pr_filename, 'wt') IOError: [Errno 13] Permission denied: '/var/crash/_usr_bin_easy_install.1000.crash' Original exception was: Traceback (most recent call last): File "/usr/bin/easy_install", line 5, in from pkg_resources import load_entry_point ImportError: No module named pkg_resources ------------------------------------------------------------------------------------------------ I can only get it to work with sudo. > That is precisely how I think of virtualenv: it sits on top of distutils > / setuptools, and makes it sane to do Python development without > compromising the core system. For me I prefer using a tool with a GUI interface. Take care David From amauryfa at gmail.com Wed Jul 22 17:33:35 2009 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 22 Jul 2009 17:33:35 +0200 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com> References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com> Message-ID: Hi, 2009/7/22 Eric Pruitt : > Hello, > > I have written replacements for a few of Mark Hammond's PyWin32 functions > using ctypes to call upon the Windows kernel API. Code can be found here; > http://pastebin.com/m46de01f . When calling ReadFile, it appears that the > kernel API converts '/n' newlines '/r/n' newlines. I have not been able to > find any information about this but as far as I can tell, there is nothing > in my code that is causing the problem. Initially I suspected it related to > newline translation but the function in subprocess.Popen for translating > newlines only converts to '/n' newlines. When I replaced my ReadFile and > WriteFile functions with the PyWin32 functions I was imitating, the problem > still arose. I was hoping someone could confirm this problem for me from > experience or by testing out my code. If you would like the see the > functions used in full context, I have a Mercurial Google Code project that > can be checked out at http://code.google.com/p/subprocdev/source/list > (branch "python3k"). My work entails modifications to subprocess.py so when > running the code, please update your PYTHONPATH variable so that my > subprocess.py will be imported. These questions should be redirected to comp.lang.python. But as a quick response, the subprocess stdout is likely to be opened in text mode. So reading \r\n is not a surprise to me. -- Amaury Forgeot d'Arc From tismer at stackless.com Wed Jul 22 20:36:00 2009 From: tismer at stackless.com (Christian Tismer) Date: Wed, 22 Jul 2009 11:36:00 -0700 Subject: [Python-Dev] mingw32 and gc-header weirdness Message-ID: <4A675C10.6060200@stackless.com> Hi all, I was hacking to get mingw32 builds of psyco to work and found a pretty weird thing: I used mingw32 (stable distro) to build the psyco extension on top of standard python2.6, built with Visual Studio, and got weird crashes. The reason is in objimpl.h: typedef union _gc_head { struct { union _gc_head *gc_next; union _gc_head *gc_prev; Py_ssize_t gc_refs; } gc; long double dummy; /* force worst-case alignment */ } PyGC_Head; Mingw32 behaves funny here. The size of long double is 12 ! 8-) I happened to use the _PyObject_GC_UNTRACK macro in psyco, instead of the function, and that caused the errors, because psython and the extension disagreed on the location of the gc pointers... Using PyObject_GC_Untrack instead fixed the problem, but there is a bad feeling left. Question: Is that considered a mingw bug? Should we change the above union to a safer construct? Or maybe I just missed something obvious and made a fool out of me? cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From solipsis at pitrou.net Wed Jul 22 21:40:08 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 22 Jul 2009 19:40:08 +0000 (UTC) Subject: [Python-Dev] mingw32 and gc-header weirdness References: <4A675C10.6060200@stackless.com> Message-ID: Hi, > I used mingw32 (stable distro) to build the psyco extension > on top of standard python2.6, built with Visual Studio, > and got weird crashes. > > The reason is in objimpl.h: > > typedef union _gc_head { > struct { > union _gc_head *gc_next; > union _gc_head *gc_prev; > Py_ssize_t gc_refs; > } gc; > long double dummy; /* force worst-case alignment */ > } PyGC_Head; > > Mingw32 behaves funny here. The size of long double is 12 ! The size of long double is also 12 under 32-bit Linux. Perhaps mingw disagrees with Visual Studio on some ABI subtleties (is it expected? is mingw supposed to be ABI-compatible with Visual Studio? if yes, you may report a bug to them :-)). Regards Antoine. From martin at v.loewis.de Wed Jul 22 21:43:45 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 22 Jul 2009 21:43:45 +0200 Subject: [Python-Dev] Support for Python/Windows In-Reply-To: <24ea26600907220648mf2ce2b4nad05d2a8c9d06349@mail.gmail.com> References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de> <4A6641F4.9000200@holdenweb.com> <24ea26600907220648mf2ce2b4nad05d2a8c9d06349@mail.gmail.com> Message-ID: <4A676BF1.4000303@v.loewis.de> > My question is the following : > > - What are the implications for Py users ? After reading your message, I had a difficult time getting your point, or even finding out what your question is. So I stick with what you said is your question: What are the implications for Py users ? To this, the answer is mostly: none at all. There may be vague indirect effects (such as more Python software being available on Windows), but I doubt it will be significantly noticeable. > I mean, even if somebody (not me but enterprises & organizations I > work or may work for in the future ;o) decides to use Windows pay for > that and everything else, I'd not like to qualify as a "pirate" (or > alike) for using a Py distribution or app including MS Intelectual > Property (MSIP) (and MS loves MSIP -even if nobody can see it- and all > kind of legal issues, especially with FOSS) nor even have Py in the > middle of a patent dispute or something ... I can't follow here. Why would using a Py distribution or app that includes MSIP qualify you as a "pirate"? And why would Microsoft's making free (as in beer) tools available to Python contributors make you qualify more as a "pirate" than you are currently qualified as? > - What are the implications for other devs (not core ;o) who use to > download sources and try new things, or perhaps use Py code the way > they want to solve an specific issue, or modify it somehow to > experiment or learn something, or whatever ? They can now get tools for free that they previously had to buy. > - Will that affect contributions from ?future or potential? devs ? This is an indirect effect; I doubt there is any noticable change (in particular as VS Express is free (as in beer) already). > - Will they also need an MS license to see or compile (or whatever) > the changes contributed by Py devs ? Not more than currently already, no. You may not be aware that Python is *already* compiled by MSVC on Windows. > - What about if for some reason, a idea or impl or alg or snippet (or > whatever) is propagated to GNU/Linux distributions and it's MSIP? > (considering former disputes like ?Linux kernel violates 42 of MS > patents?) ? Python contributors should always have the copyright to all changes they contribute. They should not contribute code owned by Microsoft, nor should they contribute code owned by anybody else (except themselves). The latter is the bigger problem: sometimes people contribute code that is owned by their employer, without an agreement of the employer. > I apologize in advance if I'm being rude or na?ve or * I didn't consider your message rude. It is perhaps na?ve (apparently ignoring the status quo), but you don't have to apologize for that. Regards, Martin From glenn at zewt.org Wed Jul 22 21:45:29 2009 From: glenn at zewt.org (Glenn Maynard) Date: Wed, 22 Jul 2009 15:45:29 -0400 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> Message-ID: On Wed, Jul 22, 2009 at 10:26 AM, Tres Seaver wrote: > You missed my point: ?I know that sys.prefix etc. is set based on the > location of the executable. ?You were complaining that setuptools / > distutils wants to put files in an "OS-controlled directory": ?I was > asking how Python was supposed to know the difference between > /usr/lib/python2.x/site-packages (presumably "OS-controlled") and > /opt/Python2.x/lib/python2.x/site-packages (definitely not "OS-controlled"). autoconf--for all its warts--solved this ages ago. On installation, you specify what prefix you want to install stuff into; if you don't, you get a sensible default (/usr/local). If you're creating an OS package, or if installing directly into /usr is otherwise really what you really want to do, you specify --prefix=/usr. The Python core doesn't need to know anything about this; it just needs to have a sys.path that includes both /usr/lib/... and /usr/local/lib/.... This works for everything else--C headers, libraries, manpages, shared libraries. Why is Python special? > The solution is to avoid doing distutils / setuptools operations as > root, *ever*, and instead use either an alternate install location (such > as /opt/Python2.x) or use something like virtualenv, which provides a > "safe" sandbox for installing stuff. This is nasty. I should certainly be able to globally install packages on my own, without creating a whole new Python installation or other hacks, just like I can install anything else into /usr/local. -- Glenn Maynard From martin at v.loewis.de Wed Jul 22 21:55:24 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 22 Jul 2009 21:55:24 +0200 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A675C10.6060200@stackless.com> References: <4A675C10.6060200@stackless.com> Message-ID: <4A676EAC.7080206@v.loewis.de> > Is that considered a mingw bug? Perhaps. I can't test right now: *if* VS has a long double type that is smaller, it is apparently an ABI violation for mingw to not follow VS. It appears that other people also consider it a bug: http://www.velocityreviews.com/forums/showpost.php?p=1514923&postcount=2 OTOH, they probably cannot change it without causing > Should we change the above union to a safer construct? No. tim_one changed it to be long double in r25454 to support some system that Dave Abrahams uses, so it needs to stay that way :-) However, we can certainly acknowledge that this is a bug in MingW, and special case it. Either introduce a symbolic type gchead_align_t which gets defined to just double on MingW, or put the #ifdef right into the structure. It might also be useful to assert that sizeof(gchead_align_t) is 8 or 16, and reject 12 as a value. The point is that we need the maximum alignment, and that certainly shouldn't be 12. Regards, Martin From foom at fuhm.net Wed Jul 22 22:40:29 2009 From: foom at fuhm.net (James Y Knight) Date: Wed, 22 Jul 2009 16:40:29 -0400 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: <4A66D290.9050103@egenix.com> References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> <4A66D290.9050103@egenix.com> Message-ID: On Jul 22, 2009, at 4:49 AM, M.-A. Lemburg wrote: > Debian has a long history of doing this different, so it's > not much of a surprise. They also apply such changes to > Python packages. > > However, all of this is non-standard and will cause problems > with tools that rely on the standard site-packages/ location. Such > changes should be discouraged. And yet, the change seems to have some strong reasoning, solves the problem discussed in this thread, and was apparently discussed and approved of by some core python developers before being implemented. It seems a bit foolish to me to thus just dismiss it as "evil debian being different"... If anything it seems like it's a failure of the Python project to make easily deployable software, compounded with a failure of communication within the python community. James From mal at egenix.com Wed Jul 22 23:22:56 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 22 Jul 2009 23:22:56 +0200 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> <4A66D290.9050103@egenix.com> Message-ID: <4A678330.7010103@egenix.com> James Y Knight wrote: > > On Jul 22, 2009, at 4:49 AM, M.-A. Lemburg wrote: > >> Debian has a long history of doing this different, so it's >> not much of a surprise. They also apply such changes to >> Python packages. >> >> However, all of this is non-standard and will cause problems >> with tools that rely on the standard site-packages/ location. Such >> changes should be discouraged. > > And yet, the change seems to have some strong reasoning, solves the > problem discussed in this thread, and was apparently discussed and > approved of by some core python developers before being implemented. It > seems a bit foolish to me to thus just dismiss it as "evil debian being > different"... Maybe I've misunderstood some important detail, but how will their "change" help with anything other than making their distribution a non-standard Python installation ? distutils allows for a great deal of flexibility. For some reason, this does not appear to be known to a larger audience. I can only recommend reading Greg's great write-up about the end-user perspective of installing Python modules: http://docs.python.org/install/ A little known fact is that distutils can easily be customized using config files: http://docs.python.org/install/#distutils-configuration-files Together with the PYTHONHOME and PYTHONPATH environment variables setup in your .bashrc, this gives you full flexibility regarding the Python package setup and permits setting up private Python installations, parallel Python installations, Python installations sharing packages, etc. etc. It really depends a lot on what you want to achieve. There's no one fits all configuration. Using a non-standard Python installation as system Python is certainly not a good approach to solving anything. It only makes it harder for users having to fight problems arising out of this. You have the same problem in other areas as well. A prominent example being Microsoft's tendency (in the past) to tweak standards to better meet their requirements. > If anything it seems like it's a failure of the Python project to make > easily deployable software, compounded with a failure of communication > within the python community. I lost you there. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 22 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From glenn at zewt.org Thu Jul 23 01:08:26 2009 From: glenn at zewt.org (Glenn Maynard) Date: Wed, 22 Jul 2009 19:08:26 -0400 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: <4A678330.7010103@egenix.com> References: <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> <4A66D290.9050103@egenix.com> <4A678330.7010103@egenix.com> Message-ID: On Wed, Jul 22, 2009 at 5:22 PM, M.-A. Lemburg wrote: > Maybe I've misunderstood some important detail, but how will > their "change" help with anything other than making their > distribution a non-standard Python installation ? I think I'm a little confused, too, because Python supports the /usr|/usr/local separation just fine (setup.py install --prefix=/usr/local). It seems like it's also using "dist-packages" instead of "site-packages". That part, I don't understand at all--distribution packages should go in /usr/lib/pythonx.y/site-packages, and "site" packages go in /usr/local/pythonx.y/site-packages; /usr/local *itself* means "non-distribution site-installed stuff". If that's what you're referring to, then at least on first impression I agree. -- Glenn Maynard From david.lyon at preisshare.net Thu Jul 23 01:10:19 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 22 Jul 2009 19:10:19 -0400 Subject: [Python-Dev] =?utf-8?q?Remove_site-packages=3F!=3F_=5Bwas=3A_=5BD?= =?utf-8?q?istutils=5D_PEP_376_-=09from_pythonpkgmgr=27s_point_of_view=5D?= In-Reply-To: <4A678330.7010103@egenix.com> References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> <4A66D290.9050103@egenix.com> <4A678330.7010103@egenix.com> Message-ID: <423776adb90814245468ea5d7d809f26@preisshare.net> On Wed, 22 Jul 2009 23:22:56 +0200, "M.-A. Lemburg" wrote: > Maybe I've misunderstood some important detail, but how will > their "change" help with anything other than making their > distribution a non-standard Python installation ? The Debian/ubuntu distribution isn't non-standard. If anything I'd prefer to suggest that it is in many ways "a standard" Here's a sys.pth from a mac... ['', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload', '/Users/david/.local/lib/python2.6/site-packages', '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages'] You can see that there are many choices along sys.path for installing packages. > distutils allows for a great deal of flexibility. For some > reason, this does not appear to be known to a larger > audience. People forget command lines - that's why. > I can only recommend reading Greg's great write-up about the > end-user perspective of installing Python modules: > > http://docs.python.org/install/ It's good documentation of course. Cheers to Greg but the old method is so tedious. That really is the hard way. pythonpkgmgr offers a much easier solution by wrapping easy_install and/or pip. You just type in parts of the package name into a search box, click [search], a search of pypi is done, click [install] and your package is downloaded and installed. It's a much more modern way of doing package installation and requires absolutely no typing on a command line. > A little known fact is that distutils can easily be customized > using config files: > > http://docs.python.org/install/#distutils-configuration-files Perphaps. But it seems only for advanced users.. and I couldn't figure out on the face of it what advantage it would have. Take care David From bugtrack at roumenpetrov.info Thu Jul 23 01:45:51 2009 From: bugtrack at roumenpetrov.info (Roumen Petrov) Date: Thu, 23 Jul 2009 02:45:51 +0300 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A675C10.6060200@stackless.com> References: <4A675C10.6060200@stackless.com> Message-ID: <4A67A4AF.1060802@roumenpetrov.info> Christian Tismer wrote: > Hi all, > > I was hacking to get mingw32 builds of psyco to work > and found a pretty weird thing: > > I used mingw32 (stable distro) to build the psyco extension > on top of standard python2.6, built with Visual Studio, > and got weird crashes. > > The reason is in objimpl.h: > > typedef union _gc_head { > struct { > union _gc_head *gc_next; > union _gc_head *gc_prev; > Py_ssize_t gc_refs; > } gc; > long double dummy; /* force worst-case alignment */ > } PyGC_Head; > > Mingw32 behaves funny here. The size of long double is 12 ! 8-) No it is correct. GNU C compiler for windows support 80-bit extended precision. > I happened to use the _PyObject_GC_UNTRACK macro in psyco, instead > of the function, and that caused the errors, because psython > and the extension disagreed on the location of the gc pointers... > Using PyObject_GC_Untrack instead fixed the problem, > but there is a bad feeling left. > > Question: > Is that considered a mingw bug? No as sizeof(long double) = sizeof(double) is MSVC limitation. > Should we change the above union to a safer construct? > > Or maybe I just missed something obvious and made a fool out of me? > > cheers - chris > Regards, Roumen From bugtrack at roumenpetrov.info Thu Jul 23 01:56:01 2009 From: bugtrack at roumenpetrov.info (Roumen Petrov) Date: Thu, 23 Jul 2009 02:56:01 +0300 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A676EAC.7080206@v.loewis.de> References: <4A675C10.6060200@stackless.com> <4A676EAC.7080206@v.loewis.de> Message-ID: <4A67A711.7020006@roumenpetrov.info> Martin v. L?wis wrote: [SNIP] > No. tim_one changed it to be long double in r25454 to support some > system that Dave Abrahams uses, so it needs to stay that way :-) > > However, we can certainly acknowledge that this is a bug in MingW, > and special case it. Either introduce a symbolic type gchead_align_t > which gets defined to just double on MingW, or put the #ifdef right > into the structure. No this is not GCC bug. GCC support "hardware extended precision" as implement long double and mingw w32api implement long double functions. According to http://msdn.microsoft.com/en-us/library/9cx8xs15.aspx it is MSVC limitation. > It might also be useful to assert that sizeof(gchead_align_t) is > 8 or 16, and reject 12 as a value. The point is that we need the > maximum alignment, and that certainly shouldn't be 12. So look like python bug. > Regards, > Martin Regards, Roumen From david.lyon at preisshare.net Thu Jul 23 01:53:29 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 22 Jul 2009 19:53:29 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request Message-ID: Hi Greg, I'm on the python-dev mailing list and somebody gave me a link to a page that you have done: http://docs.python.org/install/ Can I ask that you also provide a link for windows users to my project: http://sourceforge.net/projects/pythonpkgmgr/ Our project provides an alternative to command line installation. And can save a lot of time for users when they wish to install packages. Thanks David From ben+python at benfinney.id.au Thu Jul 23 02:00:25 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 23 Jul 2009 10:00:25 +1000 Subject: [Python-Dev] Remove site-packages?!? References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> <4A66D290.9050103@egenix.com> <4A678330.7010103@egenix.com> Message-ID: <87k5206zfq.fsf@benfinney.id.au> "M.-A. Lemburg" writes: > James Y Knight wrote: > > And yet, the change seems to have some strong reasoning, solves the > > problem discussed in this thread, and was apparently discussed and > > approved of by some core python developers before being implemented. > > It seems a bit foolish to me to thus just dismiss it as "evil debian > > being different"... > > Maybe I've misunderstood some important detail, but how will > their "change" help with anything other than making their > distribution a non-standard Python installation ? Before this discussion descends further into circular speculation on the Debian Python developers's intentions, perhaps someone should ask them directly: . -- \ ?Let others praise ancient times; I am glad I was born in | `\ these.? ?Ovid (43 BCE?18 CE) | _o__) | Ben Finney From tismer at stackless.com Thu Jul 23 02:13:51 2009 From: tismer at stackless.com (Christian Tismer) Date: Wed, 22 Jul 2009 17:13:51 -0700 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A67A711.7020006@roumenpetrov.info> References: <4A675C10.6060200@stackless.com> <4A676EAC.7080206@v.loewis.de> <4A67A711.7020006@roumenpetrov.info> Message-ID: <4A67AB3F.2060405@stackless.com> On 7/22/09 4:56 PM, Roumen Petrov wrote: > Martin v. L?wis wrote: > [SNIP] >> No. tim_one changed it to be long double in r25454 to support some >> system that Dave Abrahams uses, so it needs to stay that way :-) >> >> However, we can certainly acknowledge that this is a bug in MingW, >> and special case it. Either introduce a symbolic type gchead_align_t >> which gets defined to just double on MingW, or put the #ifdef right >> into the structure. > > No this is not GCC bug. GCC support "hardware extended precision" as > implement long double and mingw w32api implement long double functions. > According to http://msdn.microsoft.com/en-us/library/9cx8xs15.aspx it is > MSVC limitation. > > >> It might also be useful to assert that sizeof(gchead_align_t) is >> 8 or 16, and reject 12 as a value. The point is that we need the >> maximum alignment, and that certainly shouldn't be 12. > > So look like python bug. The assumption is that the union with long double gives alignment to the largest possible integral type with a power of 2 size, which is then wrong, because of the unexpected size. What do you propose for doing proper alignment, then? I fear this needs to become yet another special case in pyconfig.h cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From cournape at gmail.com Thu Jul 23 02:17:41 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 23 Jul 2009 09:17:41 +0900 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: References: <4A675C10.6060200@stackless.com> Message-ID: <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> On Thu, Jul 23, 2009 at 4:40 AM, Antoine Pitrou wrote: > > The size of long double is also 12 under 32-bit Linux. Perhaps mingw disagrees > with Visual Studio Yes, mingw and VS do not have the same long double type. This has been the source of some problems in numpy as well, since mingw uses the MS runtime, and everything involving long double and the runtime is broken (printf, math library calls). I wish there was a way to disable this in mingw, but there isn't AFAIK. > on some ABI subtleties (is it expected? is mingw supposed to > be ABI-compatible with Visual Studio? if yes, you may report a bug to them :-)). I think mostly ABI compatible is the best description :) David From tismer at stackless.com Thu Jul 23 02:28:51 2009 From: tismer at stackless.com (Christian Tismer) Date: Wed, 22 Jul 2009 17:28:51 -0700 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> Message-ID: <4A67AEC3.6070000@stackless.com> On 7/22/09 5:17 PM, David Cournapeau wrote: > On Thu, Jul 23, 2009 at 4:40 AM, Antoine Pitrou wrote: > >> The size of long double is also 12 under 32-bit Linux. Perhaps mingw disagrees >> with Visual Studio > > Yes, mingw and VS do not have the same long double type. This has been > the source of some problems in numpy as well, since mingw uses the MS > runtime, and everything involving long double and the runtime is > broken (printf, math library calls). I wish there was a way to disable > this in mingw, but there isn't AFAIK. > >> on some ABI subtleties (is it expected? is mingw supposed to >> be ABI-compatible with Visual Studio? if yes, you may report a bug to them :-)). > > I think mostly ABI compatible is the best description :) Well, the source of my problems is simply that I tried to build an extension for a Visual Studio built Python, using mingw. Did that, because it seems to be common practice so far. Maybe the simple solution is to prevent building extensions with mingw, if the python executable was not also built with it? Then, all would be fine I guess. Despite the fact that Python probably has to be changed: If it is true then all the 32-bit Linux Pythons have a 12 byte GC head, IOW they are *all* badly aligned. If that matters, of course. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From songofacandy at gmail.com Thu Jul 23 03:15:16 2009 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 23 Jul 2009 10:15:16 +0900 Subject: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI In-Reply-To: <49C9454A.7090600@v.loewis.de> References: <2d75d7660903221530g2ad1590u32fb2843051cc930@mail.gmail.com> <49C70C6A.7070106@g.nevcal.com> <49C7FF1F.6080707@g.nevcal.com> <49C9454A.7090600@v.loewis.de> Message-ID: Hi. >> We have to use "IME" to input non-ascii charactor in Windows. >> When "> chcp 65001" in cmd.exe, we cannot use IME on cmd.exe. >> >> So setting codepage to 65001 make output universal but make input ascii-only. >> Sit!!! > > Is there a code page that still allows IME input, but supports all > of Unicode? I know there is GB18030 - is it any good? I found WriteConsoleW() API recently. This API can write utf16 string to console directly, without change OutputCodepage. example: http://bitbucket.org/methane/hg-fixutf8-jp/src/tip/win32helper.py#cl-42 I think this API is good for py3k. When stdout is console and not redirected to [pipe|file], sys.stdout.write(u"foo") can avoid encoding and use WriteConsoleW(L"foo") -- Naoki INADA From bugtrack at roumenpetrov.info Thu Jul 23 03:46:32 2009 From: bugtrack at roumenpetrov.info (Roumen Petrov) Date: Thu, 23 Jul 2009 04:46:32 +0300 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A67AB3F.2060405@stackless.com> References: <4A675C10.6060200@stackless.com> <4A676EAC.7080206@v.loewis.de> <4A67A711.7020006@roumenpetrov.info> <4A67AB3F.2060405@stackless.com> Message-ID: <4A67C0F8.6020902@roumenpetrov.info> Christian Tismer wrote: > On 7/22/09 4:56 PM, Roumen Petrov wrote: >> Martin v. L?wis wrote: >> [SNIP] >>> No. tim_one changed it to be long double in r25454 to support some >>> system that Dave Abrahams uses, so it needs to stay that way :-) >>> >>> However, we can certainly acknowledge that this is a bug in MingW, >>> and special case it. Either introduce a symbolic type gchead_align_t >>> which gets defined to just double on MingW, or put the #ifdef right >>> into the structure. >> >> No this is not GCC bug. GCC support "hardware extended precision" as >> implement long double and mingw w32api implement long double functions. >> According to http://msdn.microsoft.com/en-us/library/9cx8xs15.aspx it is >> MSVC limitation. >> >> >>> It might also be useful to assert that sizeof(gchead_align_t) is >>> 8 or 16, and reject 12 as a value. The point is that we need the >>> maximum alignment, and that certainly shouldn't be 12. >> >> So look like python bug. > > The assumption is that the union with long double gives alignment > to the largest possible integral type with a power of 2 size, > which is then wrong, because of the unexpected size. As is posted for GCC on linux(32-bit) size of structure is 12 = sizeof(struct in union) = sizeof(pointer)+sizeof(pointer)+sizeof(ssize_t). And sizeof(ssize_t) <= sizeof(long) <= sizeof(pointer) . > What do you propose for doing proper alignment, then? May be "void* dummy[4]" is better for force alignment ? What about alignment on platforms with pointers > 32 bit ? > I fear this needs to become yet another special case in pyconfig.h > > cheers - chris > Regards, Roumen From martin at v.loewis.de Thu Jul 23 05:21:09 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Jul 2009 05:21:09 +0200 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A67A711.7020006@roumenpetrov.info> References: <4A675C10.6060200@stackless.com> <4A676EAC.7080206@v.loewis.de> <4A67A711.7020006@roumenpetrov.info> Message-ID: <4A67D725.6060503@v.loewis.de> Roumen Petrov wrote: > Martin v. L?wis wrote: > [SNIP] >> No. tim_one changed it to be long double in r25454 to support some >> system that Dave Abrahams uses, so it needs to stay that way :-) >> >> However, we can certainly acknowledge that this is a bug in MingW, >> and special case it. Either introduce a symbolic type gchead_align_t >> which gets defined to just double on MingW, or put the #ifdef right >> into the structure. > > No this is not GCC bug. GCC support "hardware extended precision" as > implement long double and mingw w32api implement long double functions. And that exactly is the bug. GCC cannot simultaneously be Windows-compatible, and also support extended precision floating point under the name "long double". > According to http://msdn.microsoft.com/en-us/library/9cx8xs15.aspx it is > MSVC limitation. Therefore, mingw should also implement that limitation, to be binary-compatible with MSVC. Since binary compatibility is an explicit goal of mingw, failing to provide binary compatibility is a bug. Regards, Martin From dalcinl at gmail.com Thu Jul 23 05:23:56 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 23 Jul 2009 00:23:56 -0300 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com> Message-ID: On Wed, Jul 22, 2009 at 12:33 PM, Amaury Forgeot d'Arc wrote: > > These questions should be redirected to comp.lang.python. > Eric seems to be working on a GSoC for PFS related to improving subprocess. Even in such case this list is not the right place to make these posts?? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From martin at v.loewis.de Thu Jul 23 05:26:20 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Jul 2009 05:26:20 +0200 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A67C0F8.6020902@roumenpetrov.info> References: <4A675C10.6060200@stackless.com> <4A676EAC.7080206@v.loewis.de> <4A67A711.7020006@roumenpetrov.info> <4A67AB3F.2060405@stackless.com> <4A67C0F8.6020902@roumenpetrov.info> Message-ID: <4A67D85C.1010809@v.loewis.de> >> What do you propose for doing proper alignment, then? > > May be "void* dummy[4]" is better for force alignment ? That would have the same alignment as a single pointer. > What about alignment on platforms with pointers > 32 bit ? The C compiler will choose the alignment of the union to be the alignment of the union branch that requires that largest alignment. So if the largest alignment is the one that a pointer has, then alignof(PyGC_HEAD) == alignof(gc_next). The double is in the union only in case it has larger alignment than a pointer. Regards, Martin From martin at v.loewis.de Thu Jul 23 05:36:03 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Jul 2009 05:36:03 +0200 Subject: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI In-Reply-To: References: <2d75d7660903221530g2ad1590u32fb2843051cc930@mail.gmail.com> <49C70C6A.7070106@g.nevcal.com> <49C7FF1F.6080707@g.nevcal.com> <49C9454A.7090600@v.loewis.de> Message-ID: <4A67DAA3.1050909@v.loewis.de> > When stdout is console and not redirected to [pipe|file], > sys.stdout.write(u"foo") > can avoid encoding and use WriteConsoleW(L"foo") I think this would be fairly difficult to implement given the way the output libraries work. If you think it can be done, please provide a patch to bugs.python.org. Regards, Martin From martin at v.loewis.de Thu Jul 23 05:33:24 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Jul 2009 05:33:24 +0200 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: Message-ID: <4A67DA04.70305@v.loewis.de> > Can I ask that you also provide a link for windows users > to my project: > > http://sourceforge.net/projects/pythonpkgmgr/ I fail to see how this project is relevant in the context of explaining distutils. So you would have to come up with a proposal of specific wording that makes the relevance more apparent. Please provide a patch to bugs.python.org. Regards, Martin From ben+python at benfinney.id.au Thu Jul 23 09:03:00 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 23 Jul 2009 17:03:00 +1000 Subject: [Python-Dev] Update to Python Documentation Website Request References: Message-ID: <87bpnb7ufv.fsf@benfinney.id.au> David Lyon writes: > I'm on the python-dev mailing list and somebody gave me a link > to a page that you have done: > > http://docs.python.org/install/ That's a document describing how to use ?distutils?, which is what every recipient of Python will already have installed. > Can I ask that you also provide a link for windows users > to my project: > > http://sourceforge.net/projects/pythonpkgmgr/ That doesn't seem at all appropriate; promoting third-party packages isn't at all what the above document should be doing. > Our project provides an alternative to command line installation. If people want an alternative to the standard tools provided in Python, the documentation for Python itself is not the right place to be looking. The Python Package Index is the right place to be promoting (free-software) third-party tools. -- \ ?There's no excuse to be bored. Sad, yes. Angry, yes. | `\ Depressed, yes. Crazy, yes. But there's no excuse for boredom, | _o__) ever.? ?Viggo Mortensen | Ben Finney From david.lyon at preisshare.net Thu Jul 23 09:22:37 2009 From: david.lyon at preisshare.net (david.lyon at preisshare.net) Date: Thu, 23 Jul 2009 17:22:37 +1000 (EST) Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <87bpnb7ufv.fsf@benfinney.id.au> References: <87bpnb7ufv.fsf@benfinney.id.au> Message-ID: <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> > That's a document describing how to use ?distutils?, which is what > every > recipient of Python will already have installed. > >> Can I ask that you also provide a link for windows users >> to my project: >> >> http://sourceforge.net/projects/pythonpkgmgr/ > > That doesn't seem at all appropriate; promoting third-party packages > isn't at all what the above document should be doing. Distutils was once seperate and was then included in the standard python. So i guess that I am working with the same goal in mind. > If people want an alternative to the standard tools provided in Python, The Python Package Index is the > right place to be promoting (free-software) third-party tools. Well I can sure try that and thank you for the tip. Btw, at the moment there exists no inbuilt mechanism within python for retrieving packages from pypi. Imho this is an omission for a so called 'batteries included' language. Distutils is a builtin module for 'pushing' a developer package 'to' pypi. But there is no corresponging mechanise for a user to 'pull' packages back. Surely this is a gap in the standard distro? So it is not inappropriate for me to ask about this on this list. Take care David From aahz at pythoncraft.com Thu Jul 23 05:03:00 2009 From: aahz at pythoncraft.com (Aahz) Date: Wed, 22 Jul 2009 20:03:00 -0700 Subject: [Python-Dev] GSoC: Replace MS Windows Console with Unicode UI In-Reply-To: References: <2d75d7660903221530g2ad1590u32fb2843051cc930@mail.gmail.com> <49C70C6A.7070106@g.nevcal.com> <49C7FF1F.6080707@g.nevcal.com> <49C9454A.7090600@v.loewis.de> Message-ID: <20090723030300.GA22273@panix.com> On Thu, Jul 23, 2009, INADA Naoki wrote: > > I found WriteConsoleW() API recently. > This API can write utf16 string to console directly, without change > OutputCodepage. > > example: > http://bitbucket.org/methane/hg-fixutf8-jp/src/tip/win32helper.py#cl-42 > > I think this API is good for py3k. > When stdout is console and not redirected to [pipe|file], > sys.stdout.write(u"foo") > can avoid encoding and use WriteConsoleW(L"foo") Please submit a feature request to bugs.python.org -- with a patch would be even nicer, of course. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "The volume of a pizza of thickness 'a' and radius 'z' is given by pi*z*z*a" From ben+python at benfinney.id.au Thu Jul 23 10:30:58 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 23 Jul 2009 18:30:58 +1000 Subject: [Python-Dev] Update to Python Documentation Website Request References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> Message-ID: <873a8n7qd9.fsf@benfinney.id.au> david.lyon at preisshare.net writes: > >> Can I ask that you also provide a link for windows users > >> to my project: > >> > >> http://sourceforge.net/projects/pythonpkgmgr/ > > > > That doesn't seem at all appropriate; promoting third-party packages > > isn't at all what the above document should be doing. > > Distutils was once seperate and was then included in the standard > python. > > So i guess that I am working with the same goal in mind. In which case you should work to get it accepted into standard Python *before* asking for it to be promoted in the standard Python documentation. > So it is not inappropriate for me to ask about this on this list. Discussing it here is appropriate. The modification you request is not, IMO. -- \ ?It's easy to play any musical instrument: all you have to do | `\ is touch the right key at the right time and the instrument | _o__) will play itself.? ?Johann Sebastian Bach | Ben Finney From mal at egenix.com Thu Jul 23 10:34:30 2009 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 23 Jul 2009 10:34:30 +0200 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: <423776adb90814245468ea5d7d809f26@preisshare.net> References: <4A62B820.4030903@gmail.com> <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> <4A66D290.9050103@egenix.com> <4A678330.7010103@egenix.com> <423776adb90814245468ea5d7d809f26@preisshare.net> Message-ID: <4A682096.8010707@egenix.com> David Lyon wrote: > On Wed, 22 Jul 2009 23:22:56 +0200, "M.-A. Lemburg" wrote: >> Maybe I've misunderstood some important detail, but how will >> their "change" help with anything other than making their >> distribution a non-standard Python installation ? > > The Debian/ubuntu distribution isn't non-standard. If anything > I'd prefer to suggest that it is in many ways "a standard" It's not just Debian mangling their Python installation. Many other Linux distros do the same. Keeping up with all those changes has become so hard, that we have chosen to always build our projects with private (regular) Python installations on Linux and Macs. Disk space is cheap and so is bandwidth. Compared to the hours spent on resolving issues with system provided Python installations, it's a no-brainer. > Here's a sys.pth from a mac... > > ['', > '/Library/Frameworks/Python.framework/Versions/2.6/lib/python26.zip', > '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6', > '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-darwin', > > '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac', > > '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/plat-mac/lib-scriptpackages', > > '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-tk', > '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-old', > '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/lib-dynload', > > '/Users/david/.local/lib/python2.6/site-packages', > '/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages'] > > You can see that there are many choices along sys.path for installing > packages. No, not really. The above is perfectly standard. The directories on sys.path are setup by Python itself and make sure that it can find the stdlib modules. Those entries are not meant to be used as installation targets. Python 2.6 has two standard places for installing packages: 1. In the stdlib site-packages/ subdir 2. In the user home dir's .local/lib/python2.6/site-packages dir The second is a new feature in 2.6. >> distutils allows for a great deal of flexibility. For some >> reason, this does not appear to be known to a larger >> audience. > > People forget command lines - that's why. > >> I can only recommend reading Greg's great write-up about the >> end-user perspective of installing Python modules: >> >> http://docs.python.org/install/ > > It's good documentation of course. Cheers to Greg but the > old method is so tedious. That really is the hard way. I think that's the main problem in all this... people just keep on complaining regardless of what you give them. If people don't read documentation, are not willing to at least spend a few minutes on considering what they really want to do and then shout out loud when Python doesn't do what they thought it should do (without telling Python, of course), there's little hope. > pythonpkgmgr offers a much easier solution by wrapping > easy_install and/or pip. You just type in parts of the > package name into a search box, click [search], a search > of pypi is done, click [install] and your package is > downloaded and installed. > > It's a much more modern way of doing package installation > and requires absolutely no typing on a command line. > >> A little known fact is that distutils can easily be customized >> using config files: >> >> http://docs.python.org/install/#distutils-configuration-files > > Perphaps. > > But it seems only for advanced users.. and I couldn't figure > out on the face of it what advantage it would have. You define once and for all where you want your Python packages to install themselves when typing "python setup.py install". After that's done, you can forget about all this and move on to get some real work done ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 23 2009) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Thu Jul 23 10:58:52 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 23 Jul 2009 18:58:52 +1000 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> Message-ID: <4A68264C.9060700@gmail.com> david.lyon at preisshare.net wrote: > Distutils is a builtin module for 'pushing' a developer package 'to' pypi. > > But there is no corresponging mechanise for a user to 'pull' packages back. > > Surely this is a gap in the standard distro? > > So it is not inappropriate for me to ask about this on this list. Raising it without at least glancing at the list archives which hold copious amounts of virtual text on that topic is somewhat inappropriate though :) Cheers, Nick. P.S. http://www.google.com/search?q=site:mail.python.org+inurl:python-dev+easy_install -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From floris.bruynooghe at gmail.com Thu Jul 23 10:50:44 2009 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Thu, 23 Jul 2009 09:50:44 +0100 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: References: <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> <4A66D290.9050103@egenix.com> <4A678330.7010103@egenix.com> Message-ID: <20090723085044.GA3378@laurie.devork> On Wed, Jul 22, 2009 at 07:08:26PM -0400, Glenn Maynard wrote: > On Wed, Jul 22, 2009 at 5:22 PM, M.-A. Lemburg wrote: > > Maybe I've misunderstood some important detail, but how will > > their "change" help with anything other than making their > > distribution a non-standard Python installation ? > > I think I'm a little confused, too, because Python supports the > /usr|/usr/local separation just fine (setup.py install > --prefix=/usr/local). > > It seems like it's also using "dist-packages" instead of > "site-packages". That part, I don't understand at all--distribution > packages should go in /usr/lib/pythonx.y/site-packages, and "site" > packages go in /usr/local/pythonx.y/site-packages; /usr/local *itself* > means "non-distribution site-installed stuff". If that's what you're > referring to, then at least on first impression I agree. The way Debian does this isn't that stupid. It solves the problem of the sysadmin intalling Python distibutions for the system Python without writing in the system locations, which is a very reasonable thing to do. Before Debian used dist-packages, /usr/local/lib/pythonX.Y/site-packages was added to the sys.path of the system Python and this was the location where a sysadmin should add packages (although distutils did not default to that location, so it wasn so it was still easy to mess up the system package management). But this caused trouble for people who installed their own Python in /usr/local since now you have the same /usr/local/lib/pythonX.Y/site-packages shared by 2 interpreters. To solve this the system python moved to dist-packages instead, this allows the local admin location to use /usr/local/lib/pythonX.Y/dist-packages which does not conflict with locally installed interpreters. Together with this they changed distutils to install to /usr/local/lib/pythonX.Y/dist-packages by default, so that a sysadmin would have to go out of their way to mess up the system package management. This is a pretty neat solution to the problem. But the weird thing (IMHO) is that distutils-sig completely refuses to see this requirement. Generally it seems accepted that installing modules in the system location (/usr/) is a bad thing, but last time it got discussed there was a point blank refusal to accept that the local admin needs a location in to install packages in. I think it would be great if Python instead provided a general guideline or best practice for this situation which would be explained in the documentation. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From solipsis at pitrou.net Thu Jul 23 11:04:30 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 Jul 2009 09:04:30 +0000 (UTC) Subject: [Python-Dev] mingw32 and gc-header weirdness References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> Message-ID: Christian Tismer stackless.com> writes: > > Despite the fact that Python probably has to be changed: > If it is true then all the 32-bit Linux Pythons have a 12 > byte GC head, IOW they are *all* badly aligned. Why are they badly aligned? The fact that long double is 12 bytes long doesn't mean it will force a 12-byte alignment - just whatever alignment is enough for a long double on the target machine. This could be 4, 8 or 16 bytes. From david.lyon at preisshare.net Thu Jul 23 11:22:32 2009 From: david.lyon at preisshare.net (david.lyon at preisshare.net) Date: Thu, 23 Jul 2009 19:22:32 +1000 (EST) Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: <20090723085044.GA3378@laurie.devork> References: <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> <4A66D290.9050103@egenix.com> <4A678330.7010103@egenix.com> <20090723085044.GA3378@laurie.devork> Message-ID: <60480.203.171.195.84.1248340952.squirrel@syd-srv02.ezyreg.com> Hi Floris, That's exactly how I see it and i totally agree. My contribution is to make a Package Manager Gui that tries to be supportive of what you describe so well. If i have any complaint about the state of affairs it would only be that it takes a newcomer such a long time (months) to fully understand what should be a 5 minute thing as you so well describe below. > On Wed, Jul 22, 2009 at 07:08:26PM -0400, Glenn Maynard wrote: >> On Wed, Jul 22, 2009 at 5:22 PM, M.-A. Lemburg wrote: >> > Maybe I've misunderstood some important detail, but how will >> > their "change" help with anything other than making their >> > distribution a non-standard Python installation ? >> >> I think I'm a little confused, too, because Python supports the >> /usr|/usr/local separation just fine (setup.py install >> --prefix=/usr/local). >> >> It seems like it's also using "dist-packages" instead of >> "site-packages". That part, I don't understand at all--distribution >> packages should go in /usr/lib/pythonx.y/site-packages, and "site" >> packages go in /usr/local/pythonx.y/site-packages; /usr/local *itself* >> means "non-distribution site-installed stuff". If that's what you're >> referring to, then at least on first impression I agree. > > The way Debian does this isn't that stupid. It solves the problem of > the sysadmin intalling Python distibutions for the system Python > without writing in the system locations, which is a very reasonable > thing to do. > > Before Debian used dist-packages, > /usr/local/lib/pythonX.Y/site-packages was added to the sys.path of > the system Python and this was the location where a sysadmin should > add packages (although distutils did not default to that location, so > it wasn so it was still easy to mess up the system package > management). But this caused trouble for people who installed their > own Python in /usr/local since now you have the same > /usr/local/lib/pythonX.Y/site-packages shared by 2 interpreters. > > To solve this the system python moved to dist-packages instead, this > allows the local admin location to use > /usr/local/lib/pythonX.Y/dist-packages which does not conflict with > locally installed interpreters. Together with this they changed > distutils to install to /usr/local/lib/pythonX.Y/dist-packages by > default, so that a sysadmin would have to go out of their way to mess > up the system package management. > > This is a pretty neat solution to the problem. But the weird thing > (IMHO) is that distutils-sig completely refuses to see this > requirement. Generally it seems accepted that installing modules in > the system location (/usr/) is a bad thing, but last time it got > discussed there was a point blank refusal to accept that the local > admin needs a location in to install packages in. I think it would be > great if Python instead provided a general guideline or best practice > for this situation which would be explained in the documentation. > > > Regards > Floris > > > -- > Debian GNU/Linux -- The Power of Freedom > www.debian.org | www.gnu.org | www.kernel.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/david.lyon%40preisshare.net > From david.lyon at preisshare.net Thu Jul 23 11:43:14 2009 From: david.lyon at preisshare.net (david.lyon at preisshare.net) Date: Thu, 23 Jul 2009 19:43:14 +1000 (EST) Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <4A68264C.9060700@gmail.com> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <4A68264C.9060700@gmail.com> Message-ID: <36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com> > Raising it without at least glancing at the list archives which hold > copious amounts of virtual text on that topic is somewhat inappropriate > though :) Well I have consulted every available expert on the distutils list to the point where I feel 'up' with the issues at hand. They're great people.. And as helpful as they possibly can be.. But surely Python isn't only about archives and age old problems. Sure they might be there.. So lets get in and fix them.. Which is totally what i'm attempting to do at the moment even if packaging isnt perceived as being the most interesting part of python develepment going on today. Inapropriate or not, i want to donate my time to it.. Because i think we need 'fresh' thinking - not archive regurgitation. David From p.f.moore at gmail.com Thu Jul 23 11:49:03 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 23 Jul 2009 02:49:03 -0700 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A67AEC3.6070000@stackless.com> References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> Message-ID: <79990c6b0907230249i3b6d3659ud36797222f1639c3@mail.gmail.com> 2009/7/22 Christian Tismer : > Maybe the simple solution is to prevent building extensions > with mingw, if the python executable was not also built with it? > Then, all would be fine I guess. I have never had problems in practice with extensions built with mingw rather than MSVC - so while I'm not saying that the issue doesn't exist, it certainly doesn't affect all extensions, so disabling mingw support seems a bit of an extreme measure. Having a preprocessor check in the Python headers to check if this bug is triggered in the actual code, and raising a compile error in that case, may be an option (but I expect that ensuring such a check is precise enough would be hard). Now that VC Express is freely available, the importance of mingw support is less, but AFAIK many developers cross-compile Windows binaries on Linux using mingw, or develop on gcc and use mingw to build on Windows (or Wine) but wouldn't install VC. So removing mingw support in cases where it does work would cause definite issues for Windows users relying on prebuilt binaries. Paul. From floris.bruynooghe at gmail.com Thu Jul 23 12:15:27 2009 From: floris.bruynooghe at gmail.com (Floris Bruynooghe) Date: Thu, 23 Jul 2009 11:15:27 +0100 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: <4A682096.8010707@egenix.com> References: <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> <4A66D290.9050103@egenix.com> <4A678330.7010103@egenix.com> <423776adb90814245468ea5d7d809f26@preisshare.net> <4A682096.8010707@egenix.com> Message-ID: <20090723101527.GA4322@laurie.devork> On Thu, Jul 23, 2009 at 10:34:30AM +0200, M.-A. Lemburg wrote: > Python 2.6 has two standard places for installing packages: > > 1. In the stdlib site-packages/ subdir > > 2. In the user home dir's .local/lib/python2.6/site-packages dir And is missing a 3rd one. The sysadmin who wants to install packages globally has to use the first option here, but that conflicts with the system package management on UNIXes. Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From lists at cheimes.de Thu Jul 23 13:45:31 2009 From: lists at cheimes.de (Christian Heimes) Date: Thu, 23 Jul 2009 13:45:31 +0200 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com> Message-ID: <4A684D5B.5030104@cheimes.de> Lisandro Dalcin wrote: > Eric seems to be working on a GSoC for PFS related to improving > subprocess. Even in such case this list is not the right place to make > these posts?? Eric didn't say that he is working on a GSoC project for the PSF. Anyway the Python general mailing list might still be a better place. IMHO he can reach many more competent Python developers there who can help him with this particular problem. By the way I don't think that ctypes is the right way to go here. ctypes is very handy if you need a quick solution. However I wouldn't use it as a permanent solution for the subprocess module. It's tricky to get ctypes based solutions right on multiple platforms (32 vs 64bit). It's harder to debug a ctypes solutions rather than a C extension, too. I assume that calling overhead is greater than a pure C extension. Can ctypes release the GIL for a function call? Christian From amauryfa at gmail.com Thu Jul 23 14:02:27 2009 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 23 Jul 2009 14:02:27 +0200 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: <4A684D5B.5030104@cheimes.de> References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com> <4A684D5B.5030104@cheimes.de> Message-ID: Hi, 2009/7/23 Christian Heimes : > Lisandro Dalcin wrote: >> Eric seems to be working on a GSoC for PFS related to improving >> subprocess. Even in such case this list is not the right place to make >> these posts?? > > Eric didn't say that he is working on a GSoC project for the PSF. Anyway > the Python general mailing list might still be a better place. IMHO he > can reach many more competent Python developers there who can help him > with this particular problem. Furthermore it was more a Windows question than a Python one. Anyway we continued the discussion privately and I try to be helpful. > By the way I don't think that ctypes is the right way to go here. ctypes > is very handy if you need a quick solution. However I wouldn't use it as > a permanent solution for the subprocess module. It's tricky to get > ctypes based solutions right on multiple platforms (32 vs 64bit). It's > harder to debug a ctypes solutions rather than a C extension, too. I > assume that calling overhead is greater than a pure C extension. And some distributions may choose to not distribute the ctypes module. No stdlib module should require it. OTOH ctypes is the perfect tool for rapid development with the win32 api, specially when the compiler is far away. > Can ctypes release the GIL for a function call? Yes, see http://docs.python.org/library/ctypes.html#ctypes.CFUNCTYPE -- Amaury Forgeot d'Arc From theller at ctypes.org Thu Jul 23 14:04:44 2009 From: theller at ctypes.org (Thomas Heller) Date: Thu, 23 Jul 2009 14:04:44 +0200 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: <4A684D5B.5030104@cheimes.de> References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com> <4A684D5B.5030104@cheimes.de> Message-ID: Christian Heimes schrieb: > Can ctypes release the GIL for a function call? It will do that automatically, except for functions using the pythonapi callign convention. -- Thanks, Thomas From fuzzyman at voidspace.org.uk Thu Jul 23 14:10:46 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 23 Jul 2009 13:10:46 +0100 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com> <4A684D5B.5030104@cheimes.de> Message-ID: <4A685346.6000206@voidspace.org.uk> Amaury Forgeot d'Arc wrote: > Hi, > > 2009/7/23 Christian Heimes : > >> Lisandro Dalcin wrote: >> >>> Eric seems to be working on a GSoC for PFS related to improving >>> subprocess. Even in such case this list is not the right place to make >>> these posts?? >>> >> Eric didn't say that he is working on a GSoC project for the PSF. Anyway >> the Python general mailing list might still be a better place. IMHO he >> can reach many more competent Python developers there who can help him >> with this particular problem. >> > > Furthermore it was more a Windows question than a Python one. > Anyway we continued the discussion privately and I try to be helpful. > > >> By the way I don't think that ctypes is the right way to go here. ctypes >> is very handy if you need a quick solution. However I wouldn't use it as >> a permanent solution for the subprocess module. It's tricky to get >> ctypes based solutions right on multiple platforms (32 vs 64bit). It's >> harder to debug a ctypes solutions rather than a C extension, too. I >> assume that calling overhead is greater than a pure C extension. >> > > And some distributions may choose to not distribute the ctypes module. > No stdlib module should require it. > A big advantage of using ctypes is that it works cross-implementation - on IronPython and PyPy already and on Jython soon. I'd like to see more standard library modules use it. Distributions that choose not to include it are crippling their Python distribution. Michael Foord > OTOH ctypes is the perfect tool for rapid development with the win32 > api, specially when the compiler is far away. > > >> Can ctypes release the GIL for a function call? >> > > Yes, see http://docs.python.org/library/ctypes.html#ctypes.CFUNCTYPE > > -- http://www.ironpythoninaction.com/ From ncoghlan at gmail.com Thu Jul 23 14:13:28 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 23 Jul 2009 22:13:28 +1000 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: <4A684D5B.5030104@cheimes.de> References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com> <4A684D5B.5030104@cheimes.de> Message-ID: <4A6853E8.6060606@gmail.com> Christian Heimes wrote: > By the way I don't think that ctypes is the right way to go here. ctypes > is very handy if you need a quick solution. However I wouldn't use it as > a permanent solution for the subprocess module. It's tricky to get > ctypes based solutions right on multiple platforms (32 vs 64bit). It's > harder to debug a ctypes solutions rather than a C extension, too. I > assume that calling overhead is greater than a pure C extension. I see ctypes as largely useful when you want to call a native DLL but don't have any existing infrastructure for accessing native code from your project. A few lines of ctypes code is then a much better solution than adding a C or C++ compilation dependency just to access a couple of functions. Of course, that definitely isn't the case for CPython - we not only have plenty of existing C infrastructure, but in the specific case of subprocess on Windows we already have a dedicated extension module (PC/_subprocess.c). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Thu Jul 23 14:18:00 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 23 Jul 2009 22:18:00 +1000 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <79990c6b0907230249i3b6d3659ud36797222f1639c3@mail.gmail.com> References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <79990c6b0907230249i3b6d3659ud36797222f1639c3@mail.gmail.com> Message-ID: <4A6854F8.70308@gmail.com> Paul Moore wrote: > 2009/7/22 Christian Tismer : >> Maybe the simple solution is to prevent building extensions >> with mingw, if the python executable was not also built with it? >> Then, all would be fine I guess. > > I have never had problems in practice with extensions built with mingw > rather than MSVC - so while I'm not saying that the issue doesn't > exist, it certainly doesn't affect all extensions, so disabling mingw > support seems a bit of an extreme measure. For runtime compatibility purposes, mingw is no worse than using versions of MSVC other than the one that was used to build Python (i.e. primarily making sure you use the correct memory management APIs and not trying to call the FILE* APIs). If there are other things that cause mingw to get structure sizes wrong then I don't see a major problem with a config check or some #ifdef'ery to work around it. A patch added to a tracker item would be a necessary starting point for that idea though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From lists at cheimes.de Thu Jul 23 14:21:38 2009 From: lists at cheimes.de (Christian Heimes) Date: Thu, 23 Jul 2009 14:21:38 +0200 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: <4A685346.6000206@voidspace.org.uk> References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com> <4A684D5B.5030104@cheimes.de> <4A685346.6000206@voidspace.org.uk> Message-ID: <4A6855D2.3030009@cheimes.de> Michael Foord wrote: > A big advantage of using ctypes is that it works cross-implementation - > on IronPython and PyPy already and on Jython soon. I'd like to see more > standard library modules use it. Distributions that choose not to > include it are crippling their Python distribution. Interesting, I didn't know that IronPython supports ctypes, too. I still find ctypes a bit problematic because it doesn't us header files for its types, structs and function definitions. Christian PS: I'n still reading your IronPython book. I hope I'll have some spare time to write a review anytime soon. From cournape at gmail.com Thu Jul 23 14:22:27 2009 From: cournape at gmail.com (David Cournapeau) Date: Thu, 23 Jul 2009 21:22:27 +0900 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <79990c6b0907230249i3b6d3659ud36797222f1639c3@mail.gmail.com> References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <79990c6b0907230249i3b6d3659ud36797222f1639c3@mail.gmail.com> Message-ID: <5b8d13220907230522j7acfdd93lcab9d36058aa4258@mail.gmail.com> On Thu, Jul 23, 2009 at 6:49 PM, Paul Moore wrote: > 2009/7/22 Christian Tismer : >> Maybe the simple solution is to prevent building extensions >> with mingw, if the python executable was not also built with it? >> Then, all would be fine I guess. > > I have never had problems in practice with extensions built with mingw > rather than MSVC - so while I'm not saying that the issue doesn't > exist, it certainly doesn't affect all extensions, so disabling mingw > support seems a bit of an extreme measure. I am strongly against this as well. We build numpy with mingw on windows, and disabling it would make my life even more miserable on windows. One constant source of pain with MS compilers is when supporting different versions of python - 2.4, 2.5 and 2.6 require a different VS version (and free versions are available only for the last version of VS usually). I am far from a windows specialist, but I understand that quite a few problems with mingw-built extensions with python are caused by some Python decisions as well (the C API with runtime-dependent structures like FILE, etc...). So mingw is not the only to blame :) David From lists at cheimes.de Thu Jul 23 14:23:56 2009 From: lists at cheimes.de (Christian Heimes) Date: Thu, 23 Jul 2009 14:23:56 +0200 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: <4A6853E8.6060606@gmail.com> References: <171e8a410907220701p4c7ec414h856608630167ac77@mail.gmail.com> <4A684D5B.5030104@cheimes.de> <4A6853E8.6060606@gmail.com> Message-ID: <4A68565C.3020500@cheimes.de> Nick Coghlan wrote: > I see ctypes as largely useful when you want to call a native DLL but > don't have any existing infrastructure for accessing native code from > your project. A few lines of ctypes code is then a much better solution > than adding a C or C++ compilation dependency just to access a couple of > functions. > > Of course, that definitely isn't the case for CPython - we not only have > plenty of existing C infrastructure, but in the specific case of > subprocess on Windows we already have a dedicated extension module > (PC/_subprocess.c). You've hit the nail on the head! That's it. Christian From ncoghlan at gmail.com Thu Jul 23 14:30:34 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 23 Jul 2009 22:30:34 +1000 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <4A68264C.9060700@gmail.com> <36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com> Message-ID: <4A6857EA.4060002@gmail.com> david.lyon at preisshare.net wrote: >> Raising it without at least glancing at the list archives which hold >> copious amounts of virtual text on that topic is somewhat inappropriate >> though :) > > Well I have consulted every available expert on the distutils list to the > point where I feel 'up' with the issues at hand. If you're actually up to speed with the issues, then I apologise. It was just something of a novelty to see the topic brought up without easy_install and setuptools even getting a mention. However, the reason for the asymmetry has less to do with code (easy_install exists after all) and more to do with the complexities of system administration. Providing a native ability to download and install packages from PyPI is a major maintenance commitment due to a couple of major issues: 1. Providing an installation mechanism that is compatibility with a wide variety of package management systems across at least Windows, Mac OS X and the assorted flavours of *nix (Linux RPM, Linux APT, Solaris, *BSD, etc, etc). distutils cops a lot of heat already for not playing nicely with distro packages. easy_install is loathed even more by many system administrators (and that loathing often appears to flow over onto other parts of setuptools). 2. There are some serious security implications in providing a native mechanism for downloading, installing and running code in a non-sandboxed environment. The latter problem is probably the more technical of the two, but both pose fairly complex social issues as well in terms of getting agreement across disparate groups. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From exarkun at divmod.com Thu Jul 23 14:55:14 2009 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Thu, 23 Jul 2009 08:55:14 -0400 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: <4A6855D2.3030009@cheimes.de> Message-ID: <20090723125514.2543.1215672179.divmod.quotient.4682@henry.divmod.com> On Thu, 23 Jul 2009 14:21:38 +0200, Christian Heimes wrote: >Michael Foord wrote: >> A big advantage of using ctypes is that it works cross-implementation - >> on IronPython and PyPy already and on Jython soon. I'd like to see more >> standard library modules use it. Distributions that choose not to >> include it are crippling their Python distribution. > >Interesting, I didn't know that IronPython supports ctypes, too. I still >find ctypes a bit problematic because it doesn't us header files for its >types, structs and function definitions. This is indeed a big problem with ctypes. Fortunately, a project exists to correct it: http://pypi.python.org/pypi/ctypes_configure/0.1 Anyone writing code with ctypes should be looking at ctypes_configure as well. Jean-Paul From exarkun at divmod.com Thu Jul 23 14:57:33 2009 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Thu, 23 Jul 2009 08:57:33 -0400 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: <4A68565C.3020500@cheimes.de> Message-ID: <20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com> On Thu, 23 Jul 2009 14:23:56 +0200, Christian Heimes wrote: >Nick Coghlan wrote: >> I see ctypes as largely useful when you want to call a native DLL but >> don't have any existing infrastructure for accessing native code from >> your project. A few lines of ctypes code is then a much better solution >> than adding a C or C++ compilation dependency just to access a couple of >> functions. >> >> Of course, that definitely isn't the case for CPython - we not only have >> plenty of existing C infrastructure, but in the specific case of >> subprocess on Windows we already have a dedicated extension module >> (PC/_subprocess.c). > >You've hit the nail on the head! That's it. > True, CPython has C infrastructure. What about the other Python runtimes, though? At the language summit, there was a lot of discussion (and, I thought, agreement) about moving the standard library to be a collaborative project between several of the major runtime projects. A ctypes-based solution seems more aligned with this goal than more C code. Jean-Paul From jnoller at gmail.com Thu Jul 23 15:11:38 2009 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 23 Jul 2009 09:11:38 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <4A68264C.9060700@gmail.com> <36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com> Message-ID: <4222a8490907230611tfbb8952m73345e4244eac9a2@mail.gmail.com> On Thu, Jul 23, 2009 at 5:43 AM, wrote: > >> Raising it without at least glancing at the list archives which hold >> copious amounts of virtual text on that topic is somewhat inappropriate >> though :) > > Well I have consulted every available expert on the distutils list to the > point where I feel 'up' with the issues at hand. > > They're great people.. And as helpful as they possibly can be.. > > But surely Python isn't only about archives and age old problems. > > Sure they might be there.. So lets get in and fix them.. > > Which is totally what i'm attempting to do at the moment even if packaging > isnt perceived as being the most interesting part of python develepment > going on today. > > Inapropriate or not, i want to donate my time to it.. Because i think we > need 'fresh' thinking - not archive regurgitation. > > > David Then why not include pip, easy_install, and this bash script I use to install packages into core? The more the merrier, right? Answer: None of these are standards, and as nick points out, there's issues with sysadmins, security, and other things. Not to mention they're fundamentally not part of the language. At the language summit, this was hashed out quite a bit. I think most everyone agreed that tools like easy_install, pip, virtualenv, zc.buildout, etc simply do not belong in core python. The "installation" landscape varies from platform to platform, it can run full in the face of people who just want to use apt or yum, etc. What *does* belong in core (distutils) however, and is something Tarek has been working on, are APIs/Hooks/standards for tools like the ones I've mentioned, and yours, to be able to do their job better, cleaner and easier. Standards on metadata, uninstall hooks, etc are what the stdlib should provide, not applications and tools built on top of those things. IMHO, the only "binary" python-core itself should "ship" is the python binary itself (and pydoc, maybe :-)) - everything else should be built with the idea of making integration with internals easier and standard. That way OS package maintainers can hook into these APIs (because they don't want to use something "non standard" to them), people such as yourself can build GUI apps for downloading and managing things, etc. I know you want to help, and I don't think anyone is discouraging that - but I think instead of solely focusing on your application is a mistake. The bulk of the effort should be spent helping Tarek and others "fix" distutils and spraying down bikeshedders so progress can be made. jesse From jnoller at gmail.com Thu Jul 23 15:14:43 2009 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 23 Jul 2009 09:14:43 -0400 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: <20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com> References: <4A68565C.3020500@cheimes.de> <20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com> Message-ID: <4222a8490907230614v53d11614q6f059dadd0e35d70@mail.gmail.com> On Thu, Jul 23, 2009 at 8:57 AM, Jean-Paul Calderone wrote: > On Thu, 23 Jul 2009 14:23:56 +0200, Christian Heimes > wrote: >> >> Nick Coghlan wrote: >>> >>> I see ctypes as largely useful when you want to call a native DLL but >>> don't have any existing infrastructure for accessing native code from >>> your project. A few lines of ctypes code is then a much better solution >>> than adding a C or C++ compilation dependency just to access a couple of >>> functions. >>> >>> Of course, that definitely isn't the case for CPython - we not only have >>> plenty of existing C infrastructure, but in the specific case of >>> subprocess on Windows we already have a dedicated extension module >>> (PC/_subprocess.c). >> >> You've hit the nail on the head! That's it. >> > > True, CPython has C infrastructure. ?What about the other Python runtimes, > though? ?At the language summit, there was a lot of discussion (and, I > thought, agreement) about moving the standard library to be a collaborative > project between several of the major runtime projects. ?A ctypes-based > solution seems more aligned with this goal than more C code. > > Jean-Paul Yeah, I remember that too - all of us discussed "breaking out" the stdlib post mercurial migration to be shared amongst all of the implementations, marking CPython things as "CPython only" and so on. That way we all have a common base to work from. jesse From olemis at gmail.com Thu Jul 23 15:15:45 2009 From: olemis at gmail.com (Olemis Lang) Date: Thu, 23 Jul 2009 08:15:45 -0500 Subject: [Python-Dev] Support for Python/Windows In-Reply-To: <4A676BF1.4000303@v.loewis.de> References: <4A660A53.4050806@holdenweb.com> <4A6634F2.1040003@cheimes.de> <4A6641F4.9000200@holdenweb.com> <24ea26600907220648mf2ce2b4nad05d2a8c9d06349@mail.gmail.com> <4A676BF1.4000303@v.loewis.de> Message-ID: <24ea26600907230615s65500422kd9d722d2cffeb81c@mail.gmail.com> On Wed, Jul 22, 2009 at 2:43 PM, "Martin v. L?wis" wrote: >> My question is the following : >> >> - What are the implications for Py users ? > > So I stick with what you said is your question: What are the > implications for Py users ? > > To this, the answer is mostly: none at all. There may be vague indirect > effects (such as more Python software being available on Windows), > Well this being said, it seems to be cool. >> - What are the implications for other devs (not core ;o) who use to >> download sources and try new things, or perhaps use Py code the way >> they want to solve an specific issue, or modify it somehow to >> experiment or learn something, or whatever ? > > They can now get tools for free that they previously had to buy. > Well it seems that this applies to core devs and I was talking about people not being Py core devs. But anyway, if everybody can still compile Py sources without worrying about further licensing side-effects (i.e. more than we have today ;) then the storm is gone. >> - Will that affect contributions from ?future or potential? devs ? > > This is an indirect effect; I doubt there is any noticable change > (in particular as VS Express is free (as in beer) already). > Well my concern (and what I didnt understand) was that if some people, in this case core devs, (need | like to have) the license to do (use) something they cannot do (use) without the license then (possibly) everybody else (i.e. those not having the license) trying to reproduce what others did (e.g. compilation) had to purchase the license. If this is not the case, and non-core devs can do what they do the way they do it so far, then the storm is over ... >> - Will they also need an MS license to see or compile (or whatever) >> the changes contributed by Py devs ? > > Not more than currently already, no. ... and it seems that's the case ;o) > You may not be aware that Python > is *already* compiled by MSVC on Windows. > Yes I am, but since I'm a frustrated lawyer I didnt understand a few things (and I couldnt sleep yesterday because of that ... XD ...) >> I apologize in advance if I'm being rude or na?ve or * > > I didn't consider your message rude. It is perhaps na?ve (apparently > ignoring the status quo), but you don't have to apologize for that. > Well I wanted to avoid flamewars or unnecessary disputes or whatever (you know, this licensing and FOSS vs proprietary debates may be complicated and sometimes people get excited /me included, of course : I'm human ;o) Thnx ! -- Regards, Olemis. Blog ES: http://simelo-es.blogspot.com/ Blog EN: http://simelo-en.blogspot.com/ Featured article: From amauryfa at gmail.com Thu Jul 23 16:05:40 2009 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Thu, 23 Jul 2009 16:05:40 +0200 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <5b8d13220907230522j7acfdd93lcab9d36058aa4258@mail.gmail.com> References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <79990c6b0907230249i3b6d3659ud36797222f1639c3@mail.gmail.com> <5b8d13220907230522j7acfdd93lcab9d36058aa4258@mail.gmail.com> Message-ID: 2009/7/23 David Cournapeau : > On Thu, Jul 23, 2009 at 6:49 PM, Paul Moore wrote: >> 2009/7/22 Christian Tismer : >>> Maybe the simple solution is to prevent building extensions >>> with mingw, if the python executable was not also built with it? >>> Then, all would be fine I guess. >> >> I have never had problems in practice with extensions built with mingw >> rather than MSVC - so while I'm not saying that the issue doesn't >> exist, it certainly doesn't affect all extensions, so disabling mingw >> support seems a bit of an extreme measure. > > I am strongly against this as well. We build numpy with mingw on > windows, and disabling it would make my life even more miserable on > windows. One constant source of pain with MS compilers is when > supporting different versions of python - 2.4, 2.5 and 2.6 require a > different VS version (and free versions are available only for the > last version of VS usually). > > I am far from a windows specialist, but I understand that quite a few > problems with mingw-built extensions with python are caused by some > Python decisions as well (the C API with runtime-dependent structures > like FILE, etc...). So mingw is not the only to blame :) In this case, the OP tries to use an API that is explicitly documented as dangerous for extension modules. The recommended function is not sensitive to compiler differences. -- Amaury Forgeot d'Arc From stephen at xemacs.org Thu Jul 23 16:53:04 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 23 Jul 2009 23:53:04 +0900 Subject: [Python-Dev] Remove site-packages?!? [was: [Distutils] PEP 376 - from pythonpkgmgr's point of view] In-Reply-To: <20090723085044.GA3378@laurie.devork> References: <94bdd2610907190815m1a14c521v7e10fe00d4fe3b1f@mail.gmail.com> <40e590ca51381ba25fc8b6f2853d1325@preisshare.net> <742787a67a8690d390c83a11a10aeb2a@preisshare.net> <4A66D290.9050103@egenix.com> <4A678330.7010103@egenix.com> <20090723085044.GA3378@laurie.devork> Message-ID: <87tz13v4bz.fsf@uwakimon.sk.tsukuba.ac.jp> Floris Bruynooghe writes: > [dist-packages] is a pretty neat solution to the problem. To what problem? I admit I am no expert on Python packaging, but my experience with XEmacs suggests that this is the distro trying to help with a *set* of problems that the user/sysadmins really should be handling themselves. There are varied requirements that lead to a need for yet another place to install modules, but there's no guarantee that a given user/sysadmin has *just one*. With the Emacsen, often in trying to satisfy several requirements they end up with conflicts (different libraries in the site-lisp that are compiled for, or even require, different versions of (X)Emacs or even trying to deal with Emacs and XEmacs simultaneously). And it's a pain (as an XEmacs maintainer) to have to field questions about why things don't work in a set up I didn't design, and don't understand the rationale for. > Generally it seems accepted that installing modules in the system > location (/usr/) is a bad thing, but last time it got discussed > there was a point blank refusal to accept that the local admin > needs a location in to install packages in. I think it would be > great if Python instead provided a general guideline or best > practice for this situation which would be explained in the > documentation. Yeah, so do we all, but I suspect that for Python, like the Emacsen, there is *no* general guideline. You're just asking for various kinds of trouble, all of which require expert attention, by installing a command that is invoked as "python" (as opposed to "pythonX.Y") in /usr/local/bin unless you intend it to be a complete substitute for the system python. And that itself is a serious tour de force (as anybody who remembers Red Hat's love affair with Python 1.5.2 will agree, I suspect). As far as I can see, *in general* /usr/local/lib/python/site-packages is the right place to install local packages for use with the system python, and no other. That doesn't mean that there aren't *other* valid uses for that location, just that I see no general rule for supporting *all* of them at the same time. From theller at ctypes.org Thu Jul 23 17:35:12 2009 From: theller at ctypes.org (Thomas Heller) Date: Thu, 23 Jul 2009 17:35:12 +0200 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: <20090723125514.2543.1215672179.divmod.quotient.4682@henry.divmod.com> References: <4A6855D2.3030009@cheimes.de> <20090723125514.2543.1215672179.divmod.quotient.4682@henry.divmod.com> Message-ID: Jean-Paul Calderone schrieb: > On Thu, 23 Jul 2009 14:21:38 +0200, Christian Heimes wrote: >>Michael Foord wrote: >>> A big advantage of using ctypes is that it works cross-implementation - >>> on IronPython and PyPy already and on Jython soon. I'd like to see more >>> standard library modules use it. Distributions that choose not to >>> include it are crippling their Python distribution. >> >>Interesting, I didn't know that IronPython supports ctypes, too. I still >>find ctypes a bit problematic because it doesn't us header files for its >>types, structs and function definitions. > > This is indeed a big problem with ctypes. Fortunately, a project exists > to correct it: > > http://pypi.python.org/pypi/ctypes_configure/0.1 > > Anyone writing code with ctypes should be looking at ctypes_configure as > well. There is also another project that uses gccxml to parse the header files and generate ctypes-compatible code. http://web.archive.org/web/20080115092648/starship.python.net/crew/theller/wiki/CodeGenerator Especially well on Windows works the dynamic, on-demand code generation. http://web.archive.org/web/20080115092648/starship.python.net/crew/theller/wiki/CodeGenerator/DynamicModule I do all win32 api programming with this stuff. I should note that the ctypeslib project is not well maintained because I don't have the time for that any longer. -- Thanks, Thomas From kalman.gergely at duodecad.hu Thu Jul 23 17:34:47 2009 From: kalman.gergely at duodecad.hu (=?UTF-8?B?S8OhbG3DoW4gR2VyZ2VseQ==?=) Date: Thu, 23 Jul 2009 17:34:47 +0200 Subject: [Python-Dev] python sendmsg()/recvmsg() implementation In-Reply-To: <4A365B93.7020501@gmail.com> References: <20090609151026.22176.614294214.divmod.quotient.3795@henry.divmod.com> <4A3634EB.2020800@duodecad.hu> <4A365B93.7020501@gmail.com> Message-ID: <4A688317.9060600@duodecad.hu> Hello This is the rewritten-from-scratch implementation of the sendmsg()/recvmsg() methods. Any comments / suggestions / flames are very welcome. Currently it supports what I need and I'm only releasing it, because I don't have much time to develop it further in the forseeable future (1-2 months). It is rewritten from scratch, using the python c-api documents. I've tried my best, but I wouldn't bet that it works as it's supposed to. I'd be glad if someone could give me a review on what I've done wrong. The core parts are implemented correctly (I think), the features that are missing: - using scatter/gather - using it with non-stream oriented sockets (doesn't support addresses /msg_name/) These should be very easy to implement though. I will fix the errors that are present right now, and if no one takes up the task I will implement the missing features also. You might have to wait for it a little though. Thanks in advance Cheers, Kalman Gergely -------------- next part -------------- A non-text attachment was scrubbed... Name: srmsg.patch Type: text/x-diff Size: 8341 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sendrecvmsgtest.py Type: text/x-python Size: 769 bytes Desc: not available URL: From dalcinl at gmail.com Thu Jul 23 18:13:59 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Thu, 23 Jul 2009 13:13:59 -0300 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: <20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com> References: <4A68565C.3020500@cheimes.de> <20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com> Message-ID: On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderone wrote: > > True, CPython has C infrastructure. ?What about the other Python runtimes, > though? > Perhaps these other Python runtimes could implement the functionality of PC/_subprocess.c and use ctypes for that ? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From aahz at pythoncraft.com Thu Jul 23 19:37:39 2009 From: aahz at pythoncraft.com (Aahz) Date: Thu, 23 Jul 2009 10:37:39 -0700 Subject: [Python-Dev] python sendmsg()/recvmsg() implementation In-Reply-To: <4A688317.9060600@duodecad.hu> References: <20090609151026.22176.614294214.divmod.quotient.3795@henry.divmod.com> <4A3634EB.2020800@duodecad.hu> <4A365B93.7020501@gmail.com> <4A688317.9060600@duodecad.hu> Message-ID: <20090723173739.GB565@panix.com> On Thu, Jul 23, 2009, K?lm?n Gergely wrote: > > This is the rewritten-from-scratch implementation of the > sendmsg()/recvmsg() methods. Any comments / suggestions / flames are > very welcome. Currently it supports what I need and I'm only releasing > it, because I don't have much time to develop it further in the > forseeable future (1-2 months). It is rewritten from scratch, using > the python c-api documents. I've tried my best, but I wouldn't bet > that it works as it's supposed to. I'd be glad if someone could give > me a review on what I've done wrong. Please post this to bugs.python.org -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "The volume of a pizza of thickness 'a' and radius 'z' is given by pi*z*z*a" From g.brandl at gmx.net Thu Jul 23 20:59:42 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 23 Jul 2009 20:59:42 +0200 Subject: [Python-Dev] Document None values in sys.modules? Message-ID: Hi all, is the presence of None values in sys.modules considered an implementation detail? If not, it should be documented what the None values mean to the interpreter. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From fdrake at acm.org Thu Jul 23 21:09:10 2009 From: fdrake at acm.org (Fred Drake) Date: Thu, 23 Jul 2009 15:09:10 -0400 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: References: Message-ID: <66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> On Jul 23, 2009, at 2:59 PM, Georg Brandl wrote: > is the presence of None values in sys.modules considered an > implementation > detail? If not, it should be documented what the None values mean > to the > interpreter. As I recall, they're an optimization. But since sys.modules is itself documented, and many applications actually use it, I think it's worth explaining that the None values can reasonably be expected, and what they indicate. -Fred -- Fred Drake From guido at python.org Thu Jul 23 22:05:51 2009 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Jul 2009 13:05:51 -0700 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: <66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> References: <66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> Message-ID: On Thu, Jul 23, 2009 at 12:09 PM, Fred Drake wrote: > On Jul 23, 2009, at 2:59 PM, Georg Brandl wrote: >> >> is the presence of None values in sys.modules considered an implementation >> detail? ?If not, it should be documented what the None values mean to the >> interpreter. > > As I recall, they're an optimization. ?But since sys.modules is itself > documented, and many applications actually use it, I think it's worth > explaining that the None values can reasonably be expected, and what they > indicate. They should certainly be documented -- without them imports from inside package would be super expensive (at least for Python versions where implicit relative import exists). I'm somewhat surprised this isn't documented, I don't think I've tried to keep this usage hidden. I've also sometimes abused this to force some module to believe that a certain other module doesn't exist. OTOH in Py3k I'm not sure that we even *need* them any more, since there is no more implicit relative import... They would only speed up the raising of ImportError, not the finding of a similar-named module elsewhere in the hierarchy. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tjreedy at udel.edu Thu Jul 23 22:42:37 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 23 Jul 2009 16:42:37 -0400 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: References: <4A68565C.3020500@cheimes.de> <20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com> Message-ID: Lisandro Dalcin wrote: > On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderone wrote: > >> True, CPython has C infrastructure. What about the other Python runtimes, >> though? >> > > Perhaps these other Python runtimes could implement the functionality > of PC/_subprocess.c and use ctypes for that ? Is it sensibly possible to augment a C file with #ifdefs so that it can be compiled either as a directly importable module or as a ctypes-accessible shared library? tjr From tismer at stackless.com Thu Jul 23 23:14:35 2009 From: tismer at stackless.com (Christian Tismer) Date: Thu, 23 Jul 2009 14:14:35 -0700 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> Message-ID: <4A68D2BB.8060607@stackless.com> On 7/23/09 2:04 AM, Antoine Pitrou wrote: > Christian Tismer stackless.com> writes: >> Despite the fact that Python probably has to be changed: >> If it is true then all the 32-bit Linux Pythons have a 12 >> byte GC head, IOW they are *all* badly aligned. > > Why are they badly aligned? > The fact that long double is 12 bytes long doesn't mean it will force a 12-byte > alignment - just whatever alignment is enough for a long double on the target > machine. This could be 4, 8 or 16 bytes. Things are a bit different: Alignment is not the primary concern of the gc header structure. Note that all the objects are created by malloc (system or python's arena allocator), and therefore all objects are correctly aligned by construction. The point is: The GC header is a structure invisible to the "real" gc allocated objects. It is opaquely prepended to every gc aware object. Therefore, it *needs* to have the correct size, in order to propagate its (already correct) alignment to the real object. It appears that python, compiled with gcc for all x64 32bit Linuxen (and mingw32) produces a 12 byte GC header. Not relevant for the header itself, but all GC objects are misaligned. This may not be recognized so far, because there is no builtin GC-enabled type that embeds a double. But if you create such a type, then the double will be correctly aligned in your object's structure, but then, when the object gets allocated, it is misaligned, because the header size is not a multiple of 8. To Martin: So I disagree. The gc header is not responsible for alignment in the first place, but to propagate it, correctly. And this fails miserably (in principle) since years. Proposal: We should use a simple construct that makes the gc header size simply a multiple of 8 or 16, whatever needed. Even a byte array would be ok. But please no long double :-) cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From solipsis at pitrou.net Thu Jul 23 23:27:15 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 Jul 2009 21:27:15 +0000 (UTC) Subject: [Python-Dev] mingw32 and gc-header weirdness References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> Message-ID: Christian Tismer stackless.com> writes: > > The point is: The GC header is a structure invisible to the "real" > gc allocated objects. It is opaquely prepended to every gc aware > object. Therefore, it *needs* to have the correct size, in order > to propagate its (already correct) alignment to the real object. Indeed. > This may not be recognized so far, because there is no builtin > GC-enabled type that embeds a double. > But if you create such a type, then the double will be correctly > aligned in your object's structure, but then, when the object > gets allocated, it is misaligned, because the header size is not a > multiple of 8. I'm not sure a double aligned on a 4-byte boundary is "misaligned" on a x86 CPU. Alignment is primarily important to avoid access violations on CPUs and datatypes which don't support arbitrary alignment, although it can also be useful for performance reasons. In any case, you seem to be right on this particular point: the PyGC_Head union should probably contain a "double" alternative in addition to the "long double" (and perhaps even a "long long" one). Of course, it will also make memory consumption a tad bigger for GC-enabled objects (but GC-enabled objects are generally not that small anyway). (I disagree, however, that we should remove the "long double". After all, we also want alignment of PyObjects to allow inclusion of a "long double" in them) Feel free to propose a patch on bugs.python.org, by the way. Regards Antoine. From solipsis at pitrou.net Thu Jul 23 23:38:47 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 Jul 2009 21:38:47 +0000 (UTC) Subject: [Python-Dev] mingw32 and gc-header weirdness References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> Message-ID: Antoine Pitrou pitrou.net> writes: > > In any case, you seem to be right on this particular point: the PyGC_Head union > should probably contain a "double" alternative in addition to the "long double" > (and perhaps even a "long long" one). Sorry, I realize that this doesn't really address the point. In addition to that union, we should also have a particular mechanism to compute what the proper offset should be between the PyGC_Head and the PyObject. Probably something like: typedef struct { PyGC_Head head; union { /* ... similar union as in PyGC_head */ } body; } _PyGC_dummy; #define _PyGC_Head_OFFSET offsetof(_PyGC_dummy, body) #define _Py_AS_GC(o) ((PyGC_Head *) ((void *)(o) - _PyGC_Head_OFFSET)) Regards Antoine. From tismer at stackless.com Fri Jul 24 00:14:56 2009 From: tismer at stackless.com (Christian Tismer) Date: Thu, 23 Jul 2009 15:14:56 -0700 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> Message-ID: <4A68E0E0.7010508@stackless.com> On 7/23/09 2:27 PM, Antoine Pitrou wrote: > Christian Tismer stackless.com> writes: ... > I'm not sure a double aligned on a 4-byte boundary is "misaligned" on a x86 CPU. I'm also not sure. Anyway, the result was neither intended nor expected, I guess. > Alignment is primarily important to avoid access violations on CPUs and > datatypes which don't support arbitrary alignment, although it can also be > useful for performance reasons. Performance, performance, of course (that's my job, after all :-) ) ... > Of course, it will also make memory consumption a tad bigger for GC-enabled > objects (but GC-enabled objects are generally not that small anyway). For that reason, I don't like the addition of the opaque header too much. If there were an option to make the header explicit, we would not have to round up the object size to a multiple of (8, 16), but could arrange embedded doubles as they fit the best. > (I disagree, however, that we should remove the "long double". After all, we > also want alignment of PyObjects to allow inclusion of a "long double" in them) Well, I doubt that a 12 byte long double causes any other alignment but 4. -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From solipsis at pitrou.net Fri Jul 24 00:55:13 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 Jul 2009 22:55:13 +0000 (UTC) Subject: [Python-Dev] mingw32 and gc-header weirdness References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A68E0E0.7010508@stackless.com> Message-ID: Christian Tismer stackless.com> writes: > > Well, I doubt that a 12 byte long double causes any other > alignment but 4. In 32-bit mode, no. But under x86-64 Linux, a long double is 16 bytes (!!). Regards Antoine. From david.lyon at preisshare.net Fri Jul 24 01:01:31 2009 From: david.lyon at preisshare.net (David Lyon) Date: Thu, 23 Jul 2009 19:01:31 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <873a8n7qd9.fsf@benfinney.id.au> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <873a8n7qd9.fsf@benfinney.id.au> Message-ID: On Thu, 23 Jul 2009 18:30:58 +1000, Ben Finney wrote: > In which case you should work to get it accepted into standard Python > *before* asking for it to be promoted in the standard Python > documentation. I'm very interested in how I would go about doing that. Die-hard users probably know all the python issues relating to package installation and appear to have thick skins. An "easy-way" might not mean to much to them because they've settled down into their own comfort zone. But new users, who's first operating system comes complete with a polished GUI, have an expectation for having a polished GUI tool to download their python packages in. Under Windows, we get IDLE. And ActiveState as I understand are working on their own package manager for their own python distribution. So maybe it is appropriate to consider having a GUI Package Manager within at least the Windows distribution of Python. Yes, please tell me the process... I'm very interested. David From thomas at python.org Fri Jul 24 01:28:05 2009 From: thomas at python.org (Thomas Wouters) Date: Thu, 23 Jul 2009 16:28:05 -0700 Subject: [Python-Dev] pthreads, fork, import, and execvp In-Reply-To: <9e804ac0907201716v584e3337t3be9a7bf3311ff3b@mail.gmail.com> References: <44699B47.4030504@gmail.com> <446A4202.6070608@v.loewis.de> <446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com> <446CB69E.4090609@v.loewis.de> <9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com> <3d2ce8cb0907201126t2ce8866eq3580aed9e3e6bf2@mail.gmail.com> <9e804ac0907201716v584e3337t3be9a7bf3311ff3b@mail.gmail.com> Message-ID: <9e804ac0907231628vd9de823h95656a02ffc59bc4@mail.gmail.com> So attached (and at http://codereview.appspot.com/96125/show ) is a preliminary fix, correcting the problem with os.fork(), os.forkpty() and os.fork1(). This doesn't expose a general API for C code to use, for two reasons: it's not easy, and I need this fix more than I need the API change :-) (I actually need this fix myself for Python 2.4, but it applies fairly cleanly.) To fix the mutex-across-fork problem correctly we should really acquire three locks (the import lock, the GIL and the TLS lock, in that order.) The import lock is re-entrant, and the TLS lock is tightly confined to the thread-local-storage lookup functions, but the GIL is neither re-entrant nor inspectable. That means we can't use pthread_atfork (we can't tell whether we have the GIL already or not, right before the fork), nor can we provide a convenient API for extension modules to use, really. The best we can do is provide three functions, pthread_atfork-style: a 'prepare' function, an 'after-fork-in-child' function, and an 'after-fork-in-parent' function. The 'prepare' function would start by releasing the GIL, then acquire the import lock, the GIL and the TLS lock in that order. It would also record *somewhere* the thread_ident of the current thread. The 'in-parent' function would simply release the TLS lock and the import lock, and the 'in-child' would release those locks, call the threading._at_fork() function, and fix up the TLS data, using the recorded thread ident to do lookups. The 'in-child' function would replace the current PyOS_AfterFork() function (which blindly reinitializes the GIL and the TLS lock, and calls threading._at_fork().) Alternatively we could do what we do now, which is to ignore the fact that thread idents may change by fork() and thus that thread-local data may disappear, in which case the 'in-child' function could do a little less work. I'm suitably scared and disgusted of the TLS implementation, the threading implementations we support (including the pthread one) and the way we blindly type-pun a pthread_t to a long, that I'm not sure I want to try and build something correct and reliable on top of it. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: import_lock_fork_deadlock.diff Type: text/x-diff Size: 5418 bytes Desc: not available URL: From jafo at tummy.com Fri Jul 24 02:57:34 2009 From: jafo at tummy.com (Sean Reifschneider) Date: Thu, 23 Jul 2009 18:57:34 -0600 Subject: [Python-Dev] REVIEW: PyArg_ParseTuple with "s" format and NUL: Bogus TypeError detail string. Message-ID: <20090724005734.GA20019@tummy.com> Please review this, I'm worried that there are cases where convertitem() is returning a string that really should be overridden by the argument "help string". However, I'm worried that this change will get rid of useful messages (via the format "; help string"), when there otherwise wouldn't be. PyArg_ParseTuple when handling a string format "s", raises a TypeError when the passed string contains a NUL. However, the TypeError doesn't contain useful information. For example: syslog.syslog('hello\0there') TypeError: [priority,] message string This seems to be a thinko in Python/getargs.c at line 331: msg = convertitem(PyTuple_GET_ITEM(args, i), &format, p_va, flags, levels, msgbuf, sizeof(msgbuf), &freelist); if (msg) { seterror(i+1, msg, levels, fname, message); <<< Line 331 return cleanreturn(0, freelist); } This also applies to Python 3 trunk in line 390. I think that's supposed to be "msg" instead of "message" in the last argument. If I change it, I get: >>> import syslog; syslog.syslog('hello\0there') Traceback (most recent call last): File "", line 1, in TypeError: must be string without null bytes, not str I think it's safe to change "message" to "msg" because: "message" is the "help" string "[priority,] message string", but "msg" contains much more useful information. "msg" is in the "fall back if the last argument doesn't contain useful information" argument position, but "msg" is never NULL. "message" only is NULL if the format string doesn't contain ";". In every case I can find in the code, convertitem() is returning a much more useful string than the help string. Or perhaps it should do something like: if (msg) { seterror(i+1, msg, levels, fname, '%s (%s)' % ( msg, message )); Pardon my mixed C+Python, but you get the idea. Thoughts? Thanks, Sean -- [...] Premature optimization is the root of all evil. -- Donald Knuth Sean Reifschneider, Member of Technical Staff tummy.com, ltd. - Linux Consulting since 1995: Ask me about High Availability From lists at cheimes.de Fri Jul 24 03:23:57 2009 From: lists at cheimes.de (Christian Heimes) Date: Fri, 24 Jul 2009 03:23:57 +0200 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> Message-ID: david.lyon at preisshare.net wrote: >> That's a document describing how to use ?distutils?, which is what >> every >> recipient of Python will already have installed. >> >>> Can I ask that you also provide a link for windows users >>> to my project: >>> >>> http://sourceforge.net/projects/pythonpkgmgr/ >> That doesn't seem at all appropriate; promoting third-party packages >> isn't at all what the above document should be doing. > > Distutils was once seperate and was then included in the standard python. > > So i guess that I am working with the same goal in mind. I'm sorry to inform you that a wxWindows based solution has zero change to get into the Python standard library ever. We are not going to add another GUI toolkit to the core distribution. We might even remove TK from the core and ship it as a separate package like bsddb3. Christian From david.lyon at preisshare.net Fri Jul 24 03:55:18 2009 From: david.lyon at preisshare.net (David Lyon) Date: Thu, 23 Jul 2009 21:55:18 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> Message-ID: <5ecc64238b6e429c72441815bf818f48@preisshare.net> On Fri, 24 Jul 2009 03:23:57 +0200, Christian Heimes wrote: > I'm sorry to inform you that a wxWindows based solution has zero change > to get into the Python standard library ever. We are not going to add > another GUI toolkit to the core distribution. In executable form, the Package Manager does not require wxWidgets to be installed. There is no dependency for this to be installed. I'm not requesting that wxPython be added to the standard python distribution. > We might even remove TK > from the core and ship it as a separate package like bsddb3. That doesn't affect the Python Package Manager in any way. David From brett at python.org Fri Jul 24 04:31:24 2009 From: brett at python.org (Brett Cannon) Date: Thu, 23 Jul 2009 19:31:24 -0700 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: References: <66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> Message-ID: On Thu, Jul 23, 2009 at 13:05, Guido van Rossum wrote: > On Thu, Jul 23, 2009 at 12:09 PM, Fred Drake wrote: > > On Jul 23, 2009, at 2:59 PM, Georg Brandl wrote: > >> > >> is the presence of None values in sys.modules considered an > implementation > >> detail? If not, it should be documented what the None values mean to > the > >> interpreter. > > > > As I recall, they're an optimization. But since sys.modules is itself > > documented, and many applications actually use it, I think it's worth > > explaining that the None values can reasonably be expected, and what they > > indicate. > > They should certainly be documented -- without them imports from > inside package would be super expensive (at least for Python versions > where implicit relative import exists). I'm somewhat surprised this > isn't documented, I don't think I've tried to keep this usage hidden. > I've also sometimes abused this to force some module to believe that a > certain other module doesn't exist. > It has not been hidden, but the only place I ever came across it was in the original essay about packages. > > OTOH in Py3k I'm not sure that we even *need* them any more, since > there is no more implicit relative import... They would only speed up > the raising of ImportError, not the finding of a similar-named module > elsewhere in the hierarchy. None in Python 3.1 is really useless in terms of its semantics in relative imports; importlib doesn't support it and still passes as __import__ (at least last time I ran the test suite that way). I thought we had agreed a while back that supporting None was not warranted in Python 3.0? Otherwise I will do whatever work is necessary for this to happen. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Fri Jul 24 04:38:07 2009 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 23 Jul 2009 21:38:07 -0500 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: References: <66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> Message-ID: <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> 2009/7/23 Brett Cannon : > None in Python 3.1 is really useless in terms of its semantics in relative > imports; importlib doesn't support it and still passes as __import__ (at > least last time I ran the test suite that way). I thought we had agreed a > while back that supporting None was not warranted in Python 3.0? Otherwise I > will do whatever work is necessary for this to happen. I think it's still nice for the rare cases where you need to trick a module into thinking another one doesn't exist. -- Regards, Benjamin From brett at python.org Fri Jul 24 04:42:21 2009 From: brett at python.org (Brett Cannon) Date: Thu, 23 Jul 2009 19:42:21 -0700 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> References: <66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> Message-ID: On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson wrote: > 2009/7/23 Brett Cannon : > > None in Python 3.1 is really useless in terms of its semantics in > relative > > imports; importlib doesn't support it and still passes as __import__ (at > > least last time I ran the test suite that way). I thought we had agreed a > > while back that supporting None was not warranted in Python 3.0? > Otherwise I > > will do whatever work is necessary for this to happen. > > I think it's still nice for the rare cases where you need to trick a > module into thinking another one doesn't exist. But None does not strictly mean "I don't exist". None is supposed to trigger an another import attempt for the module with a top-level name. It's that extra import trigger that has no real use in 3.0 and just complicates import semantics (IMO) needlessly. If you want a module to not exist then you either stick something else in (e.g. '42') or we remove the special semantics for None (which I thought we had). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Fri Jul 24 04:48:05 2009 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 23 Jul 2009 21:48:05 -0500 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: References: <66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> Message-ID: <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> 2009/7/23 Brett Cannon : > > > On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson > wrote: >> >> 2009/7/23 Brett Cannon : >> > None in Python 3.1 is really useless in terms of its semantics in >> > relative >> > imports; importlib doesn't support it and still passes as __import__ (at >> > least last time I ran the test suite that way). I thought we had agreed >> > a >> > while back that supporting None was not warranted in Python 3.0? >> > Otherwise I >> > will do whatever work is necessary for this to happen. >> >> I think it's still nice for the rare cases where you need to trick a >> module into thinking another one doesn't exist. > > But None does not strictly mean "I don't exist". None is supposed to trigger > an another import attempt for the module with a top-level name. It's that > extra import trigger that has no real use in 3.0 and just complicates import > semantics (IMO) needlessly. If you want a module to not exist then you > either stick something else in (e.g. '42') or we remove the special > semantics for None (which I thought we had). I didn't realize None had other semantics attached to it. (Imagine that dealing with import!) +1 for making it simply indicate an ImportError. -- Regards, Benjamin From brett at python.org Fri Jul 24 04:50:32 2009 From: brett at python.org (Brett Cannon) Date: Thu, 23 Jul 2009 19:50:32 -0700 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> References: <66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> Message-ID: On Thu, Jul 23, 2009 at 19:48, Benjamin Peterson wrote: > 2009/7/23 Brett Cannon : > > > > > > On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson > > wrote: > >> > >> 2009/7/23 Brett Cannon : > >> > None in Python 3.1 is really useless in terms of its semantics in > >> > relative > >> > imports; importlib doesn't support it and still passes as __import__ > (at > >> > least last time I ran the test suite that way). I thought we had > agreed > >> > a > >> > while back that supporting None was not warranted in Python 3.0? > >> > Otherwise I > >> > will do whatever work is necessary for this to happen. > >> > >> I think it's still nice for the rare cases where you need to trick a > >> module into thinking another one doesn't exist. > > > > But None does not strictly mean "I don't exist". None is supposed to > trigger > > an another import attempt for the module with a top-level name. It's that > > extra import trigger that has no real use in 3.0 and just complicates > import > > semantics (IMO) needlessly. If you want a module to not exist then you > > either stick something else in (e.g. '42') or we remove the special > > semantics for None (which I thought we had). > > > I didn't realize None had other semantics attached to it. (Imagine > that dealing with import!) +1 for making it simply indicate an > ImportError. I'm +0 with having import raise ImportError if None is set in sys.modules as long as we don't suddenly expect loaders to trigger the same thing if they return None (actually, as of right now what loaders return count for nothing, but just want to be clear). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From dalcinl at gmail.com Fri Jul 24 05:18:17 2009 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Fri, 24 Jul 2009 00:18:17 -0300 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: References: <4A68565C.3020500@cheimes.de> <20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com> Message-ID: On Thu, Jul 23, 2009 at 5:42 PM, Terry Reedy wrote: > Lisandro Dalcin wrote: >> >> On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderone >> wrote: >> >>> True, CPython has C infrastructure. ?What about the other Python >>> runtimes, >>> though? >>> >> >> Perhaps these other Python runtimes could implement the functionality >> of PC/_subprocess.c and use ctypes for that ? > > Is it sensibly possible to augment a C file with #ifdefs so that it can be > compiled either as a directly importable module or as a ctypes-accessible > shared library? > Of course it is posible... Moreover, you could likely reuse 100 % of code intended for ctypes in implementing the extension module calls intended for Python. Mac OS X could have some issues though (IIUC, .so ext modules are not the same as .dylib dynamic libs, though perhaps ctypes can still dlopen() .so files?) However, how that would help these other Python runtimes without C infraestructure ?? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From guido at python.org Fri Jul 24 05:18:35 2009 From: guido at python.org (Guido van Rossum) Date: Thu, 23 Jul 2009 20:18:35 -0700 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: References: <66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> Message-ID: So, I guess, we'll live with it for a while longer. Given that it managed to evade our attention for so long, I think that's fine. I agree that there's no reason for a None result from loaders to be interpreted the same way, assuming that's not how it works ATM. And we can live with import and importlib differing on this in 3.1 (though you could call it a bug in importlib and fix it for 3.1.1 -- not sure if you were planning on that). --Guido On Thu, Jul 23, 2009 at 7:50 PM, Brett Cannon wrote: > > > On Thu, Jul 23, 2009 at 19:48, Benjamin Peterson > wrote: >> >> 2009/7/23 Brett Cannon : >> > >> > >> > On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson >> > wrote: >> >> >> >> 2009/7/23 Brett Cannon : >> >> > None in Python 3.1 is really useless in terms of its semantics in >> >> > relative >> >> > imports; importlib doesn't support it and still passes as __import__ >> >> > (at >> >> > least last time I ran the test suite that way). I thought we had >> >> > agreed >> >> > a >> >> > while back that supporting None was not warranted in Python 3.0? >> >> > Otherwise I >> >> > will do whatever work is necessary for this to happen. >> >> >> >> I think it's still nice for the rare cases where you need to trick a >> >> module into thinking another one doesn't exist. >> > >> > But None does not strictly mean "I don't exist". None is supposed to >> > trigger >> > an another import attempt for the module with a top-level name. It's >> > that >> > extra import trigger that has no real use in 3.0 and just complicates >> > import >> > semantics (IMO) needlessly. If you want a module to not exist then you >> > either stick something else in (e.g. '42') or we remove the special >> > semantics for None (which I thought we had). >> >> >> I didn't realize None had other semantics attached to it. (Imagine >> that dealing with import!) +1 for making it simply indicate an >> ImportError. > > I'm +0 with having import raise ImportError if None is set in sys.modules as > long as we don't suddenly expect loaders to trigger the same thing if they > return None (actually, as of right now what loaders return count for > nothing, but just want to be clear). > -Brett -- --Guido van Rossum (home page: http://www.python.org/~guido/) From brett at python.org Fri Jul 24 05:35:46 2009 From: brett at python.org (Brett Cannon) Date: Thu, 23 Jul 2009 20:35:46 -0700 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: References: <66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> Message-ID: On Thu, Jul 23, 2009 at 20:18, Guido van Rossum wrote: > So, I guess, we'll live with it for a while longer. Given that it > managed to evade our attention for so long, I think that's fine. > Can someone double-check me that the semantics can even be triggered in 3.1? I just tried and couldn't come up with anything. Heck, I quick search for a Py_None comparison in 3.1's import.c turned up nothing useful (other than mark_miss() is the function used to set None in sys.modules). We might have actually already removed it or made it so that the semantics can't be triggered. > > I agree that there's no reason for a None result from loaders to be > interpreted the same way, assuming that's not how it works ATM. > > And we can live with import and importlib differing on this in 3.1 > (though you could call it a bug in importlib and fix it for 3.1.1 -- > not sure if you were planning on that). > I can if people can trigger the semantics somehow so I have a test to go by. -Brett > > --Guido > > On Thu, Jul 23, 2009 at 7:50 PM, Brett Cannon wrote: > > > > > > On Thu, Jul 23, 2009 at 19:48, Benjamin Peterson > > wrote: > >> > >> 2009/7/23 Brett Cannon : > >> > > >> > > >> > On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson > > >> > wrote: > >> >> > >> >> 2009/7/23 Brett Cannon : > >> >> > None in Python 3.1 is really useless in terms of its semantics in > >> >> > relative > >> >> > imports; importlib doesn't support it and still passes as > __import__ > >> >> > (at > >> >> > least last time I ran the test suite that way). I thought we had > >> >> > agreed > >> >> > a > >> >> > while back that supporting None was not warranted in Python 3.0? > >> >> > Otherwise I > >> >> > will do whatever work is necessary for this to happen. > >> >> > >> >> I think it's still nice for the rare cases where you need to trick a > >> >> module into thinking another one doesn't exist. > >> > > >> > But None does not strictly mean "I don't exist". None is supposed to > >> > trigger > >> > an another import attempt for the module with a top-level name. It's > >> > that > >> > extra import trigger that has no real use in 3.0 and just complicates > >> > import > >> > semantics (IMO) needlessly. If you want a module to not exist then you > >> > either stick something else in (e.g. '42') or we remove the special > >> > semantics for None (which I thought we had). > >> > >> > >> I didn't realize None had other semantics attached to it. (Imagine > >> that dealing with import!) +1 for making it simply indicate an > >> ImportError. > > > > I'm +0 with having import raise ImportError if None is set in sys.modules > as > > long as we don't suddenly expect loaders to trigger the same thing if > they > > return None (actually, as of right now what loaders return count for > > nothing, but just want to be clear). > > -Brett > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From horos11 at gmail.com Fri Jul 24 07:31:31 2009 From: horos11 at gmail.com (Edward Peschko) Date: Thu, 23 Jul 2009 22:31:31 -0700 Subject: [Python-Dev] command line attachable debugger Message-ID: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com> all, I'I was wondering if there was a command line python debugger that was able to attach to an existing process. I'd very much like to be able to debug over a ssh session using screen. Ed (ps - and yes, I know about winpdb, etc... that is not exactly what I'm looking for..) From tjreedy at udel.edu Fri Jul 24 07:49:37 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 24 Jul 2009 01:49:37 -0400 Subject: [Python-Dev] Replacing PyWin32's PeekNamedPipe, ReadFile, and WriteFile In-Reply-To: References: <4A68565C.3020500@cheimes.de> <20090723125733.2543.1669789808.divmod.quotient.4684@henry.divmod.com> Message-ID: Lisandro Dalcin wrote: > On Thu, Jul 23, 2009 at 5:42 PM, Terry Reedy wrote: >> Lisandro Dalcin wrote: >>> On Thu, Jul 23, 2009 at 9:57 AM, Jean-Paul Calderone >>> wrote: >>> >>>> True, CPython has C infrastructure. What about the other Python >>>> runtimes, >>>> though? >>>> >>> Perhaps these other Python runtimes could implement the functionality >>> of PC/_subprocess.c and use ctypes for that ? >> Is it sensibly possible to augment a C file with #ifdefs so that it can be >> compiled either as a directly importable module or as a ctypes-accessible >> shared library? >> > > Of course it is posible... Moreover, you could likely reuse 100 % of > code intended for ctypes in implementing the extension module calls > intended for Python. Mac OS X could have some issues though (IIUC, .so > ext modules are not the same as .dylib dynamic libs, though perhaps > ctypes can still dlopen() .so files?) > > However, how that would help these other Python runtimes without C > infraestructure ?? I believe someone just posted that PyPy and IronPython have ctypes and Jython is working on it. That is what triggered this post. From kalman.gergely at duodecad.hu Fri Jul 24 10:32:36 2009 From: kalman.gergely at duodecad.hu (=?ISO-8859-1?Q?K=E1lm=E1n_Gergely?=) Date: Fri, 24 Jul 2009 10:32:36 +0200 Subject: [Python-Dev] python sendmsg()/recvmsg() implementation In-Reply-To: <20090723173739.GB565@panix.com> References: <20090609151026.22176.614294214.divmod.quotient.3795@henry.divmod.com> <4A3634EB.2020800@duodecad.hu> <4A365B93.7020501@gmail.com> <4A688317.9060600@duodecad.hu> <20090723173739.GB565@panix.com> Message-ID: <4A6971A4.6030705@duodecad.hu> Done, it's at: http://bugs.python.org/issue6560 Kalman Gergely Aahz wrote: > On Thu, Jul 23, 2009, K?lm?n Gergely wrote: > >> This is the rewritten-from-scratch implementation of the >> sendmsg()/recvmsg() methods. Any comments / suggestions / flames are >> very welcome. Currently it supports what I need and I'm only releasing >> it, because I don't have much time to develop it further in the >> forseeable future (1-2 months). It is rewritten from scratch, using >> the python c-api documents. I've tried my best, but I wouldn't bet >> that it works as it's supposed to. I'd be glad if someone could give >> me a review on what I've done wrong. >> > > Please post this to bugs.python.org > From bugtrack at roumenpetrov.info Fri Jul 24 14:16:06 2009 From: bugtrack at roumenpetrov.info (Roumen Petrov) Date: Fri, 24 Jul 2009 15:16:06 +0300 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A675C10.6060200@stackless.com> References: <4A675C10.6060200@stackless.com> Message-ID: <4A69A606.1040506@roumenpetrov.info> Christian Tismer wrote: > Hi all, > > I was hacking to get mingw32 builds of psyco to work > and found a pretty weird thing: > > I used mingw32 (stable distro) to build the psyco extension > on top of standard python2.6, built with Visual Studio, > and got weird crashes. > > The reason is in objimpl.h: > > typedef union _gc_head { > struct { > union _gc_head *gc_next; > union _gc_head *gc_prev; > Py_ssize_t gc_refs; > } gc; > long double dummy; /* force worst-case alignment */ > } PyGC_Head; [SNIP] Did the crash disappear is you add "__attribute__((aligned(8)))" after variable dummy ? Regards, Roumen From bugtrack at roumenpetrov.info Fri Jul 24 14:45:12 2009 From: bugtrack at roumenpetrov.info (Roumen Petrov) Date: Fri, 24 Jul 2009 15:45:12 +0300 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A68E0E0.7010508@stackless.com> Message-ID: <4A69ACD8.5040408@roumenpetrov.info> Antoine Pitrou wrote: > Christian Tismer stackless.com> writes: >> Well, I doubt that a 12 byte long double causes any other >> alignment but 4. > > In 32-bit mode, no. But under x86-64 Linux, a long double is 16 bytes (!!). And alignment is ? From solipsis at pitrou.net Fri Jul 24 14:55:29 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 24 Jul 2009 12:55:29 +0000 (UTC) Subject: [Python-Dev] mingw32 and gc-header weirdness References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A68E0E0.7010508@stackless.com> <4A69ACD8.5040408@roumenpetrov.info> Message-ID: Roumen Petrov roumenpetrov.info> writes: > > Antoine Pitrou wrote: > > > > In 32-bit mode, no. But under x86-64 Linux, a long double is 16 bytes (!!). > > And alignment is ? 16 bytes as well. From solipsis at pitrou.net Fri Jul 24 15:07:54 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 24 Jul 2009 13:07:54 +0000 (UTC) Subject: [Python-Dev] mingw32 and gc-header weirdness References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A68E0E0.7010508@stackless.com> <4A69ACD8.5040408@roumenpetrov.info> Message-ID: Antoine Pitrou pitrou.net> writes: > > Roumen Petrov roumenpetrov.info> writes: > > > > Antoine Pitrou wrote: > > > > > > In 32-bit mode, no. But under x86-64 Linux, a long double is 16 bytes (!!). > > > > And alignment is ? > > 16 bytes as well. This is probably why, by the way, a PyGC_Head takes only 12 bytes in 32-bit mode, but 32 bytes in 64-bit mode. From bugtrack at roumenpetrov.info Fri Jul 24 15:51:33 2009 From: bugtrack at roumenpetrov.info (Roumen Petrov) Date: Fri, 24 Jul 2009 16:51:33 +0300 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A68E0E0.7010508@stackless.com> <4A69ACD8.5040408@roumenpetrov.info> Message-ID: <4A69BC65.4000006@roumenpetrov.info> Antoine Pitrou wrote: > Antoine Pitrou pitrou.net> writes: >> Roumen Petrov roumenpetrov.info> writes: >>> Antoine Pitrou wrote: >>>> In 32-bit mode, no. But under x86-64 Linux, a long double is 16 bytes (!!). >>> And alignment is ? >> 16 bytes as well. > > This is probably why, by the way, a PyGC_Head takes only 12 bytes in 32-bit > mode, but 32 bytes in 64-bit mode. yes, 10x Roumen From aahz at pythoncraft.com Fri Jul 24 16:10:08 2009 From: aahz at pythoncraft.com (Aahz) Date: Fri, 24 Jul 2009 07:10:08 -0700 Subject: [Python-Dev] REVIEW: PyArg_ParseTuple with "s" format and NUL: Bogus TypeError detail string. In-Reply-To: <20090724005734.GA20019@tummy.com> References: <20090724005734.GA20019@tummy.com> Message-ID: <20090724141008.GA21866@panix.com> On Thu, Jul 23, 2009, Sean Reifschneider wrote: > > Thoughts? Please file a report at bugs.python.org to make sure it doesn't get lost. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "At Resolver we've found it useful to short-circuit any doubt and just refer to comments in code as 'lies'. :-)" --Michael Foord paraphrases Christian Muirhead on python-dev, 2009-03-22 From foom at fuhm.net Fri Jul 24 16:53:42 2009 From: foom at fuhm.net (James Y Knight) Date: Fri, 24 Jul 2009 10:53:42 -0400 Subject: [Python-Dev] command line attachable debugger In-Reply-To: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com> References: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com> Message-ID: <82FB8A40-F5AD-4980-83C6-558617B49532@fuhm.net> On Jul 24, 2009, at 1:31 AM, Edward Peschko wrote: > all, > > I'I was wondering if there was a command line python debugger that was > able to attach to an existing process. I'd very much like to be able > to debug over a ssh session using screen. > > Ed > > (ps - and yes, I know about winpdb, etc... that is not exactly what > I'm looking for..) Winpdb is *exactly* what you asked for, so if it's not what you're looking for you'll need to be more specific about what you want that it doesn't do... James From aahz at pythoncraft.com Fri Jul 24 17:13:56 2009 From: aahz at pythoncraft.com (Aahz) Date: Fri, 24 Jul 2009 08:13:56 -0700 Subject: [Python-Dev] command line attachable debugger In-Reply-To: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com> References: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com> Message-ID: <20090724151356.GA13491@panix.com> On Thu, Jul 23, 2009, Edward Peschko wrote: > > I'I was wondering if there was a command line python debugger that was > able to attach to an existing process. I'd very much like to be able > to debug over a ssh session using screen. python-dev is not the correct place to ask about this, please use comp.lang.python (python-dev is for questions about fixing bugs and adding features). -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "At Resolver we've found it useful to short-circuit any doubt and just refer to comments in code as 'lies'. :-)" --Michael Foord paraphrases Christian Muirhead on python-dev, 2009-03-22 From g.brandl at gmx.net Fri Jul 24 17:54:09 2009 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 24 Jul 2009 17:54:09 +0200 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <5ecc64238b6e429c72441815bf818f48@preisshare.net> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <5ecc64238b6e429c72441815bf818f48@preisshare.net> Message-ID: David Lyon schrieb: > On Fri, 24 Jul 2009 03:23:57 +0200, Christian Heimes > wrote: >> I'm sorry to inform you that a wxWindows based solution has zero change >> to get into the Python standard library ever. We are not going to add >> another GUI toolkit to the core distribution. > > In executable form, the Package Manager does not require wxWidgets > to be installed. > > There is no dependency for this to be installed. What does "in exectuable form" mean? Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From status at bugs.python.org Fri Jul 24 18:07:59 2009 From: status at bugs.python.org (Python tracker) Date: Fri, 24 Jul 2009 18:07:59 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20090724160759.C0C57785CD@psf.upfronthosting.co.za> ACTIVITY SUMMARY (07/17/09 - 07/24/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2278 open (+41) / 16106 closed (+16) / 18384 total (+57) Open issues with patches: 897 Average duration of open issues: 661 days. Median duration of open issues: 413 days. Open Issues Breakdown open 2245 (+41) pending 33 ( +0) Issues Created Or Reopened (58) _______________________________ warnings.warn segfaults on bad formatted string 07/18/09 CLOSED http://bugs.python.org/issue6415 reopened r.david.murray patch lib2to3 fails to convert 'thread' when not followed by a period 07/17/09 CLOSED http://bugs.python.org/issue6506 created ericp Enhance dis.dis to autocompile codestrings 07/17/09 http://bugs.python.org/issue6507 created tjreedy expose setresuid 07/17/09 http://bugs.python.org/issue6508 created solinym re.py - encounter unexpected 07/18/09 http://bugs.python.org/issue6509 created kaizhu zipfile: OSError File exists 07/18/09 CLOSED http://bugs.python.org/issue6510 created srid zipfile: Invalid argument when opening zero-sized files 07/18/09 http://bugs.python.org/issue6511 created srid UnicodeDecodeError: 'utf8' codec can't decode bytes in position 07/18/09 CLOSED http://bugs.python.org/issue6512 created wplappert Standard Library, Warnings, 28.5.4, example fails 07/18/09 CLOSED http://bugs.python.org/issue6513 created jasontiller "python -m unittest " does not run any tests 07/18/09 CLOSED http://bugs.python.org/issue6514 created exarkun http://docs.python.org/dev/library/unittest.html#load-tests-prot 07/18/09 http://bugs.python.org/issue6515 created exarkun reset owner/group to root for distutils tarballs 07/18/09 http://bugs.python.org/issue6516 created till configparser: add possibility to escape formatstrings 07/18/09 http://bugs.python.org/issue6517 created till patch Enable 'with' statement in ossaudiodev module 07/19/09 http://bugs.python.org/issue6518 created jjalocha Reorder 'with' statement for files in Python Tutorial 07/19/09 http://bugs.python.org/issue6519 created jjalocha urllib.urlopen does not have timeout parameter where as urllib2. 07/19/09 http://bugs.python.org/issue6520 created orsenthil easy Contradictory documentation for email.mime.text.MIMEText 07/19/09 http://bugs.python.org/issue6521 created pitrou docs for unittest.expectedFailure do not syntactically show it's 07/20/09 http://bugs.python.org/issue6522 created brett.cannon smtplib exception smtp.connect TypeError encode_plain 07/20/09 http://bugs.python.org/issue6523 created kmcguire tarfile module missing decompression support? 07/20/09 CLOSED http://bugs.python.org/issue6524 created hravnx Problem with string.lowercase in Windows XP 07/20/09 CLOSED http://bugs.python.org/issue6525 created PeterL importlib.import_module affects permissions of .pyc files subseq 07/20/09 http://bugs.python.org/issue6526 created r.david.murray patch test_ttk_guionly buildbot test crash: Tcl_FinalizeNotifier: noti 07/20/09 http://bugs.python.org/issue6527 created r.david.murray builtins colored as keyword at beginning of line 07/20/09 http://bugs.python.org/issue6528 created Freiddie get{pw,gr}{nam,[ug]id}() calls are not re-entrant 07/20/09 http://bugs.python.org/issue6529 created dhduvall Regression on "python -Wi" crash 07/21/09 CLOSED http://bugs.python.org/issue6530 created kbk needs review atexit_callfuncs() crashing within Py_Finalize() when using mult 07/21/09 http://bugs.python.org/issue6531 created grahamd thread.get_ident() should return unsigned value 07/21/09 http://bugs.python.org/issue6532 created sargo Make test_xmlrpc_net functional in the absence of time.xmlrpc.co 07/21/09 http://bugs.python.org/issue6533 created r.david.murray patch, patch, easy os.makedirs returns EACCES for "C:\" 07/21/09 CLOSED http://bugs.python.org/issue6534 created luke-jr optparse required field for Options 07/21/09 http://bugs.python.org/issue6535 created mindvirus patch, easy urllib2 howto contains typo 07/21/09 CLOSED http://bugs.python.org/issue6536 created kurtmckee string.split shouldn't split on non-breaking spaces 07/21/09 CLOSED http://bugs.python.org/issue6537 reopened ericp MatchObject is not a hyperlink in the 're' module documentation 07/21/09 http://bugs.python.org/issue6538 created r.david.murray easy unittest dir not created during install 07/21/09 CLOSED http://bugs.python.org/issue6539 created kbk patch, needs review bytearray.translate(): error in error handling 07/22/09 CLOSED http://bugs.python.org/issue6540 created haypo patch SpooledTemporaryFile breakages 07/22/09 http://bugs.python.org/issue6541 created leonov patch test_os TestInvalidFD.test_closerange causes test_pipes hang in 07/22/09 CLOSED http://bugs.python.org/issue6542 created r.david.murray patch traceback presented in wrong encoding 07/22/09 http://bugs.python.org/issue6543 created r_mosaic patch, needs review Fix refleak in kqueue implementation 07/22/09 http://bugs.python.org/issue6544 created georg.brandl patch test fails in distutils.tests.test_extensions if -O is used 07/22/09 CLOSED http://bugs.python.org/issue6545 created tarek [Distutils][PATCH] Add bdist_rpm option to select the name of th 07/22/09 http://bugs.python.org/issue6546 created OG7 patch shutil.copytree fails on dangling symlinks 07/22/09 http://bugs.python.org/issue6547 created TTimo cmath documentation misleading: suggests existence of real() an 07/22/09 http://bugs.python.org/issue6548 created marketdickinson patch ttk.Style not translating some Tcl options 07/22/09 http://bugs.python.org/issue6549 created markroseman asyncore incorrect failure when connection is refused and using 07/22/09 http://bugs.python.org/issue6550 created nirs patch test_codecs fails when ran after test_zipimport and test_mailbox 07/23/09 http://bugs.python.org/issue6551 created alexandre.vassalotti Build Applet.app 07/23/09 http://bugs.python.org/issue6552 created MartienF cPickle "binunicode" segmentation fault 07/23/09 CLOSED http://bugs.python.org/issue6553 created boogenhagn patch Do we have something like os.pid_exists()? 07/23/09 http://bugs.python.org/issue6554 created pmolina distutils config file should have the same name on both platform 07/23/09 http://bugs.python.org/issue6555 created zooko "HOME" is not a standard environment variable on Windows 07/23/09 http://bugs.python.org/issue6556 created zooko urllib.urlopen creates bad requests when location header of 301 07/23/09 http://bugs.python.org/issue6557 created till #ifdef linux is incorrect; should be #ifdef __linux__ (preferred 07/24/09 http://bugs.python.org/issue6558 created yaneurabeya [PATCH]add pass_fds paramter to subprocess.Popen() 07/24/09 http://bugs.python.org/issue6559 created zhigang patch socket sendmsg(), recvmsg() methods 07/24/09 http://bugs.python.org/issue6560 created synapse patch Regex '\d' should not match unicode category 'No'. 07/24/09 http://bugs.python.org/issue6561 created marketdickinson patch OverflowError in RLock.acquire() 07/24/09 http://bugs.python.org/issue6562 created davidar Issues Now Closed (37) ______________________ atexit module not safe in Python 3.0 with multiple interpreters 270 days http://bugs.python.org/issue4200 grahamd patch, needs review bugs in array.array with exports (buffer protocol) 230 days http://bugs.python.org/issue4509 alexandre.vassalotti patch, needs review urllib2.py timeouts do not propagate across redirects for 2.6.1 170 days http://bugs.python.org/issue5102 orsenthil patch, easy pydoc reports misleading failure if target module raises an Impo 156 days http://bugs.python.org/issue5230 r.david.murray patch, easy Python 2.6 makes .pyc/.pyo bytecode files executable 60 days http://bugs.python.org/issue6070 r.david.murray patch [IDLE] UnicodeDecodeError when invoking force-open-completions 50 days http://bugs.python.org/issue6144 ezio.melotti Make PyDescr_COMMON conform to standard C 52 days http://bugs.python.org/issue6151 alexandre.vassalotti patch, patch, needs review Make io.BytesIO and io.StringIO picklable. 46 days http://bugs.python.org/issue6218 alexandre.vassalotti patch, patch Better type checking for the arguments of io.StringIO 43 days http://bugs.python.org/issue6241 alexandre.vassalotti patch Fix reference leak in io.StringIO 43 days http://bugs.python.org/issue6242 alexandre.vassalotti patch Simplify string decoding in xmlrpc.client. 23 days http://bugs.python.org/issue6360 alexandre.vassalotti patch warnings.warn segfaults on bad formatted string 1 days http://bugs.python.org/issue6415 r.david.murray patch The cgi docs should advertize using "in" instead of "has_key" 17 days http://bugs.python.org/issue6423 ezio.melotti patch Fraction fails equality test with a user-defined type 11 days http://bugs.python.org/issue6431 marketdickinson patch Improve/update python.org/dev/ 8 days http://bugs.python.org/issue6449 r.david.murray easy Documentation of ElementTree.Element.getiterator implies element 3 days http://bugs.python.org/issue6489 georg.brandl 2to3 generates "from urllib.parse import pathname2url" 1 days http://bugs.python.org/issue6496 benjamin.peterson patch documentation error: missing comma between kwonlyargcount & nloc 1 days http://bugs.python.org/issue6502 georg.brandl Minor typos in tutorial (i/o chapter) 1 days http://bugs.python.org/issue6505 georg.brandl lib2to3 fails to convert 'thread' when not followed by a period 0 days http://bugs.python.org/issue6506 benjamin.peterson zipfile: OSError File exists 3 days http://bugs.python.org/issue6510 amaury.forgeotdarc UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0 days http://bugs.python.org/issue6512 ezio.melotti Standard Library, Warnings, 28.5.4, example fails 0 days http://bugs.python.org/issue6513 georg.brandl "python -m unittest " does not run any tests 2 days http://bugs.python.org/issue6514 michael.foord tarfile module missing decompression support? 0 days http://bugs.python.org/issue6524 hravnx Problem with string.lowercase in Windows XP 3 days http://bugs.python.org/issue6525 PeterL Regression on "python -Wi" crash 0 days http://bugs.python.org/issue6530 benjamin.peterson needs review os.makedirs returns EACCES for "C:\" 0 days http://bugs.python.org/issue6534 amaury.forgeotdarc urllib2 howto contains typo 0 days http://bugs.python.org/issue6536 ezio.melotti string.split shouldn't split on non-breaking spaces 0 days http://bugs.python.org/issue6537 benjamin.peterson unittest dir not created during install 0 days http://bugs.python.org/issue6539 benjamin.peterson patch, needs review bytearray.translate(): error in error handling 0 days http://bugs.python.org/issue6540 georg.brandl patch test_os TestInvalidFD.test_closerange causes test_pipes hang in 1 days http://bugs.python.org/issue6542 r.david.murray patch test fails in distutils.tests.test_extensions if -O is used 0 days http://bugs.python.org/issue6545 tarek cPickle "binunicode" segmentation fault 0 days http://bugs.python.org/issue6553 amaury.forgeotdarc patch Hang using files named prn.txt, etc 2810 days http://bugs.python.org/issue481171 amaury.forgeotdarc copy 2 725 days http://bugs.python.org/issue1764044 amaury.forgeotdarc Top Issues Most Discussed (10) ______________________________ 12 Can't import xmlrpclib, DocXMLRPCServer and SimpleXMLRPCServer 8 days open http://bugs.python.org/issue6499 10 Problem with string.lowercase in Windows XP 3 days closed http://bugs.python.org/issue6525 7 Do we have something like os.pid_exists()? 1 days open http://bugs.python.org/issue6554 7 UnicodeDecodeError: 'utf8' codec can't decode bytes in position 0 days closed http://bugs.python.org/issue6512 6 test_os TestInvalidFD.test_closerange causes test_pipes hang in 1 days closed http://bugs.python.org/issue6542 5 importlib.import_module affects permissions of .pyc files subse 4 days open http://bugs.python.org/issue6526 5 xml.sax.saxutils.XMLGenerator should write to io.RawIOBase. 186 days open http://bugs.python.org/issue4997 4 cmath documentation misleading: suggests existence of real() a 2 days open http://bugs.python.org/issue6548 4 optparse required field for Options 3 days open http://bugs.python.org/issue6535 4 configparser: add possibility to escape formatstrings 6 days open http://bugs.python.org/issue6517 From p.f.moore at gmail.com Fri Jul 24 18:08:32 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 24 Jul 2009 17:08:32 +0100 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <5ecc64238b6e429c72441815bf818f48@preisshare.net> Message-ID: <79990c6b0907240908k7e8cc9f9xf99c91ffd5256d5d@mail.gmail.com> 2009/7/24 Georg Brandl : > David Lyon schrieb: >> On Fri, 24 Jul 2009 03:23:57 +0200, Christian Heimes >> wrote: >>> I'm sorry to inform you that a wxWindows based solution has zero change >>> to get into the Python standard library ever. We are not going to add >>> another GUI toolkit to the core distribution. >> >> In executable form, the Package Manager does not require wxWidgets >> to be installed. >> >> There is no dependency for this to be installed. > > What does "in exectuable form" mean? I read this as meaning that David was proposing to ship a built application (on Windows, bundled up with something like py2exe, I guess) and any supporting DLLs such as the wxWindows ones would be bundled in, but the wxPython package would *not* be shipped as part of the standard library. If my assumption is right, that makes the package manager inappropriate for shipping with Python, whether or not there are other issues. In my view, for an "application" to be shipped with Python, whether it be Idle, or a package manager, or pydoc, or whatever, it must: - be written as a Python script - depend only on packages shipped with the standard library That is before any considerations of the value of the application itself. Given that David's package manager uses setuptools to do the installs (and now it appears it uses wxPython for its GUI) that immediately disqualifies it for inclusion in my view - regardless of its value as a package. Paul. From guido at python.org Fri Jul 24 19:39:53 2009 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Jul 2009 10:39:53 -0700 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: References: <66E5CD33-7DE4-481C-82E2-2F80E144EB4F@acm.org> <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> Message-ID: Still works, at least in some old 3.1 I had lying around: $ python3.1 Python 3.1a0 (py3k:70152, Mar 3 2009, 16:55:42) [GCC 4.0.1 (Apple Inc. build 5465)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.modules['string'] = None >>> import string Traceback (most recent call last): File "", line 1, in ImportError: No module named string >>> $ python3.1 Python 3.1a0 (py3k:70152, Mar 3 2009, 16:55:42) [GCC 4.0.1 (Apple Inc. build 5465)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import string >>> $ The experiment should be easily repeatable. :-) --Guido On Thu, Jul 23, 2009 at 8:35 PM, Brett Cannon wrote: > > > On Thu, Jul 23, 2009 at 20:18, Guido van Rossum wrote: >> >> So, I guess, we'll live with it for a while longer. Given that it >> managed to evade our attention for so long, I think that's fine. > > Can someone double-check me that the semantics can even be triggered in 3.1? > I just tried and couldn't come up with anything. Heck, I quick search for a > Py_None comparison in 3.1's import.c turned up nothing useful (other than > mark_miss() is the function used to set None in sys.modules). We might have > actually already removed it or made it so that the semantics can't be > triggered. > >> >> I agree that there's no reason for a None result from loaders to be >> interpreted the same way, assuming that's not how it works ATM. >> >> And we can live with import and importlib differing on this in 3.1 >> (though you could call it a bug in importlib and fix it for 3.1.1 -- >> not sure if you were planning on that). > > I can if people can trigger the semantics somehow so I have a test to go by. > -Brett > >> >> --Guido >> >> On Thu, Jul 23, 2009 at 7:50 PM, Brett Cannon wrote: >> > >> > >> > On Thu, Jul 23, 2009 at 19:48, Benjamin Peterson >> > wrote: >> >> >> >> 2009/7/23 Brett Cannon : >> >> > >> >> > >> >> > On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson >> >> > >> >> > wrote: >> >> >> >> >> >> 2009/7/23 Brett Cannon : >> >> >> > None in Python 3.1 is really useless in terms of its semantics in >> >> >> > relative >> >> >> > imports; importlib doesn't support it and still passes as >> >> >> > __import__ >> >> >> > (at >> >> >> > least last time I ran the test suite that way). I thought we had >> >> >> > agreed >> >> >> > a >> >> >> > while back that supporting None was not warranted in Python 3.0? >> >> >> > Otherwise I >> >> >> > will do whatever work is necessary for this to happen. >> >> >> >> >> >> I think it's still nice for the rare cases where you need to trick a >> >> >> module into thinking another one doesn't exist. >> >> > >> >> > But None does not strictly mean "I don't exist". None is supposed to >> >> > trigger >> >> > an another import attempt for the module with a top-level name. It's >> >> > that >> >> > extra import trigger that has no real use in 3.0 and just complicates >> >> > import >> >> > semantics (IMO) needlessly. If you want a module to not exist then >> >> > you >> >> > either stick something else in (e.g. '42') or we remove the special >> >> > semantics for None (which I thought we had). >> >> >> >> >> >> I didn't realize None had other semantics attached to it. (Imagine >> >> that dealing with import!) +1 for making it simply indicate an >> >> ImportError. >> > >> > I'm +0 with having import raise ImportError if None is set in >> > sys.modules as >> > long as we don't suddenly expect loaders to trigger the same thing if >> > they >> > return None (actually, as of right now what loaders return count for >> > nothing, but just want to be clear). >> > -Brett >> >> >> >> -- >> --Guido van Rossum (home page: http://www.python.org/~guido/) > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From brett at python.org Fri Jul 24 19:50:22 2009 From: brett at python.org (Brett Cannon) Date: Fri, 24 Jul 2009 10:50:22 -0700 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: References: <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> Message-ID: On Fri, Jul 24, 2009 at 10:39, Guido van Rossum wrote: > Still works, at least in some old 3.1 I had lying around: > > $ python3.1 > Python 3.1a0 (py3k:70152, Mar 3 2009, 16:55:42) > [GCC 4.0.1 (Apple Inc. build 5465)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import sys > >>> sys.modules['string'] = None > >>> import string > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named string > >>> > $ python3.1 > Python 3.1a0 (py3k:70152, Mar 3 2009, 16:55:42) > [GCC 4.0.1 (Apple Inc. build 5465)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import string > >>> > $ > > The experiment should be easily repeatable. :-) > =) Yes, the None raising ImportError semantics can be added easily and codified as official import semantics in 3.1. I was talking about the "None triggers another import as with relative imports" semantics and if there was some rather convoluted way to trigger that. I am guessing not as that would require a value of -1 for level which is no longer valid. I will fix importlib in both 3.2 and 3.1.1. And speaking of fixing, should I put the failing test in now and decorate it with unittest.expectedFailure until I get around to fixing it? -Brett > > --Guido > > On Thu, Jul 23, 2009 at 8:35 PM, Brett Cannon wrote: > > > > > > On Thu, Jul 23, 2009 at 20:18, Guido van Rossum > wrote: > >> > >> So, I guess, we'll live with it for a while longer. Given that it > >> managed to evade our attention for so long, I think that's fine. > > > > Can someone double-check me that the semantics can even be triggered in > 3.1? > > I just tried and couldn't come up with anything. Heck, I quick search for > a > > Py_None comparison in 3.1's import.c turned up nothing useful (other than > > mark_miss() is the function used to set None in sys.modules). We might > have > > actually already removed it or made it so that the semantics can't be > > triggered. > > > >> > >> I agree that there's no reason for a None result from loaders to be > >> interpreted the same way, assuming that's not how it works ATM. > >> > >> And we can live with import and importlib differing on this in 3.1 > >> (though you could call it a bug in importlib and fix it for 3.1.1 -- > >> not sure if you were planning on that). > > > > I can if people can trigger the semantics somehow so I have a test to go > by. > > -Brett > > > >> > >> --Guido > >> > >> On Thu, Jul 23, 2009 at 7:50 PM, Brett Cannon wrote: > >> > > >> > > >> > On Thu, Jul 23, 2009 at 19:48, Benjamin Peterson > > >> > wrote: > >> >> > >> >> 2009/7/23 Brett Cannon : > >> >> > > >> >> > > >> >> > On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson > >> >> > > >> >> > wrote: > >> >> >> > >> >> >> 2009/7/23 Brett Cannon : > >> >> >> > None in Python 3.1 is really useless in terms of its semantics > in > >> >> >> > relative > >> >> >> > imports; importlib doesn't support it and still passes as > >> >> >> > __import__ > >> >> >> > (at > >> >> >> > least last time I ran the test suite that way). I thought we had > >> >> >> > agreed > >> >> >> > a > >> >> >> > while back that supporting None was not warranted in Python 3.0? > >> >> >> > Otherwise I > >> >> >> > will do whatever work is necessary for this to happen. > >> >> >> > >> >> >> I think it's still nice for the rare cases where you need to trick > a > >> >> >> module into thinking another one doesn't exist. > >> >> > > >> >> > But None does not strictly mean "I don't exist". None is supposed > to > >> >> > trigger > >> >> > an another import attempt for the module with a top-level name. > It's > >> >> > that > >> >> > extra import trigger that has no real use in 3.0 and just > complicates > >> >> > import > >> >> > semantics (IMO) needlessly. If you want a module to not exist then > >> >> > you > >> >> > either stick something else in (e.g. '42') or we remove the special > >> >> > semantics for None (which I thought we had). > >> >> > >> >> > >> >> I didn't realize None had other semantics attached to it. (Imagine > >> >> that dealing with import!) +1 for making it simply indicate an > >> >> ImportError. > >> > > >> > I'm +0 with having import raise ImportError if None is set in > >> > sys.modules as > >> > long as we don't suddenly expect loaders to trigger the same thing if > >> > they > >> > return None (actually, as of right now what loaders return count for > >> > nothing, but just want to be clear). > >> > -Brett > >> > >> > >> > >> -- > >> --Guido van Rossum (home page: http://www.python.org/~guido/) > > > > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Jul 24 19:54:55 2009 From: brett at python.org (Brett Cannon) Date: Fri, 24 Jul 2009 10:54:55 -0700 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: References: <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> Message-ID: On Fri, Jul 24, 2009 at 10:39, Guido van Rossum wrote: > Still works, at least in some old 3.1 I had lying around: > Which reminds me, do we want to change the error message for ImportError in this case to say something like "import of {0} halted; None found in sys.modules"? Might be a tough exception to debug if you are not aware that some library shoved None into sys.modules. > > $ python3.1 > Python 3.1a0 (py3k:70152, Mar 3 2009, 16:55:42) > [GCC 4.0.1 (Apple Inc. build 5465)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import sys > >>> sys.modules['string'] = None > >>> import string > Traceback (most recent call last): > File "", line 1, in > ImportError: No module named string > >>> > $ python3.1 > Python 3.1a0 (py3k:70152, Mar 3 2009, 16:55:42) > [GCC 4.0.1 (Apple Inc. build 5465)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> import string > >>> > $ > > The experiment should be easily repeatable. :-) > > --Guido > > On Thu, Jul 23, 2009 at 8:35 PM, Brett Cannon wrote: > > > > > > On Thu, Jul 23, 2009 at 20:18, Guido van Rossum > wrote: > >> > >> So, I guess, we'll live with it for a while longer. Given that it > >> managed to evade our attention for so long, I think that's fine. > > > > Can someone double-check me that the semantics can even be triggered in > 3.1? > > I just tried and couldn't come up with anything. Heck, I quick search for > a > > Py_None comparison in 3.1's import.c turned up nothing useful (other than > > mark_miss() is the function used to set None in sys.modules). We might > have > > actually already removed it or made it so that the semantics can't be > > triggered. > > > >> > >> I agree that there's no reason for a None result from loaders to be > >> interpreted the same way, assuming that's not how it works ATM. > >> > >> And we can live with import and importlib differing on this in 3.1 > >> (though you could call it a bug in importlib and fix it for 3.1.1 -- > >> not sure if you were planning on that). > > > > I can if people can trigger the semantics somehow so I have a test to go > by. > > -Brett > > > >> > >> --Guido > >> > >> On Thu, Jul 23, 2009 at 7:50 PM, Brett Cannon wrote: > >> > > >> > > >> > On Thu, Jul 23, 2009 at 19:48, Benjamin Peterson > > >> > wrote: > >> >> > >> >> 2009/7/23 Brett Cannon : > >> >> > > >> >> > > >> >> > On Thu, Jul 23, 2009 at 19:38, Benjamin Peterson > >> >> > > >> >> > wrote: > >> >> >> > >> >> >> 2009/7/23 Brett Cannon : > >> >> >> > None in Python 3.1 is really useless in terms of its semantics > in > >> >> >> > relative > >> >> >> > imports; importlib doesn't support it and still passes as > >> >> >> > __import__ > >> >> >> > (at > >> >> >> > least last time I ran the test suite that way). I thought we had > >> >> >> > agreed > >> >> >> > a > >> >> >> > while back that supporting None was not warranted in Python 3.0? > >> >> >> > Otherwise I > >> >> >> > will do whatever work is necessary for this to happen. > >> >> >> > >> >> >> I think it's still nice for the rare cases where you need to trick > a > >> >> >> module into thinking another one doesn't exist. > >> >> > > >> >> > But None does not strictly mean "I don't exist". None is supposed > to > >> >> > trigger > >> >> > an another import attempt for the module with a top-level name. > It's > >> >> > that > >> >> > extra import trigger that has no real use in 3.0 and just > complicates > >> >> > import > >> >> > semantics (IMO) needlessly. If you want a module to not exist then > >> >> > you > >> >> > either stick something else in (e.g. '42') or we remove the special > >> >> > semantics for None (which I thought we had). > >> >> > >> >> > >> >> I didn't realize None had other semantics attached to it. (Imagine > >> >> that dealing with import!) +1 for making it simply indicate an > >> >> ImportError. > >> > > >> > I'm +0 with having import raise ImportError if None is set in > >> > sys.modules as > >> > long as we don't suddenly expect loaders to trigger the same thing if > >> > they > >> > return None (actually, as of right now what loaders return count for > >> > nothing, but just want to be clear). > >> > -Brett > >> > >> > >> > >> -- > >> --Guido van Rossum (home page: http://www.python.org/~guido/) > > > > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Jul 24 19:58:39 2009 From: guido at python.org (Guido van Rossum) Date: Fri, 24 Jul 2009 10:58:39 -0700 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: References: <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> Message-ID: On Fri, Jul 24, 2009 at 10:50 AM, Brett Cannon wrote: > > > On Fri, Jul 24, 2009 at 10:39, Guido van Rossum wrote: >> >> Still works, at least in some old 3.1 I had lying around: >> >> $ python3.1 >> Python 3.1a0 (py3k:70152, Mar ?3 2009, 16:55:42) >> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import sys >> >>> sys.modules['string'] = None >> >>> import string >> Traceback (most recent call last): >> ?File "", line 1, in >> ImportError: No module named string >> >>> >> $ python3.1 >> Python 3.1a0 (py3k:70152, Mar ?3 2009, 16:55:42) >> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin >> Type "help", "copyright", "credits" or "license" for more information. >> >>> import string >> >>> >> $ >> >> The experiment should be easily repeatable. :-) > > =) Yes, the None raising ImportError semantics can be added easily and > codified as official import semantics in 3.1. I thought it was too simple. :-) Maybe I should go back to bed and nurse my cough some more instead of playing hookie from being sick and indulging in email. > I was talking about the "None > triggers another import as with relative imports" semantics and if there was > some rather convoluted way to trigger that. I can't think of one and doubt you will find one -- it was introduced (in the earliest of early package import implementations) to optimize the problem we had with code living inside a package trying to stat() a toplevel module locally each time an import of it was processed (a problem if a package contains lots of modules each of which import os, sys, etc.). Since we no longer to relative import, I don't see how there could be another use -- if it's not there it's not there. > I am guessing not as that would > require a value of -1 for level which is no longer valid. Ah. Sure. > I will fix importlib in both 3.2 and 3.1.1. And speaking of fixing, should I > put the failing test in now and decorate it with unittest.expectedFailure > until I get around to fixing it? Oh, I don't know. All this new-fangled technology... Can't you just leave it in your workspace, unsubmitted, until you get to fixing it? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From brett at python.org Fri Jul 24 20:05:15 2009 From: brett at python.org (Brett Cannon) Date: Fri, 24 Jul 2009 11:05:15 -0700 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: References: <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> Message-ID: On Fri, Jul 24, 2009 at 10:58, Guido van Rossum wrote: > On Fri, Jul 24, 2009 at 10:50 AM, Brett Cannon wrote: > > > > > > On Fri, Jul 24, 2009 at 10:39, Guido van Rossum > wrote: > >> > >> Still works, at least in some old 3.1 I had lying around: > >> > >> $ python3.1 > >> Python 3.1a0 (py3k:70152, Mar 3 2009, 16:55:42) > >> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin > >> Type "help", "copyright", "credits" or "license" for more information. > >> >>> import sys > >> >>> sys.modules['string'] = None > >> >>> import string > >> Traceback (most recent call last): > >> File "", line 1, in > >> ImportError: No module named string > >> >>> > >> $ python3.1 > >> Python 3.1a0 (py3k:70152, Mar 3 2009, 16:55:42) > >> [GCC 4.0.1 (Apple Inc. build 5465)] on darwin > >> Type "help", "copyright", "credits" or "license" for more information. > >> >>> import string > >> >>> > >> $ > >> > >> The experiment should be easily repeatable. :-) > > > > =) Yes, the None raising ImportError semantics can be added easily and > > codified as official import semantics in 3.1. > > I thought it was too simple. :-) Maybe I should go back to bed and > nurse my cough some more instead of playing hookie from being sick and > indulging in email. > I'm glad I can help provide a distraction. And I hope you get over whatever you have soon. > > > I was talking about the "None > > triggers another import as with relative imports" semantics and if there > was > > some rather convoluted way to trigger that. > > I can't think of one and doubt you will find one -- it was introduced > (in the earliest of early package import implementations) to optimize > the problem we had with code living inside a package trying to stat() > a toplevel module locally each time an import of it was processed (a > problem if a package contains lots of modules each of which import os, > sys, etc.). Since we no longer to relative import, I don't see how > there could be another use -- if it's not there it's not there. > > > I am guessing not as that would > > require a value of -1 for level which is no longer valid. > > Ah. Sure. > > > I will fix importlib in both 3.2 and 3.1.1. And speaking of fixing, > should I > > put the failing test in now and decorate it with unittest.expectedFailure > > until I get around to fixing it? > > Oh, I don't know. All this new-fangled technology... Can't you just > leave it in your workspace, unsubmitted, until you get to fixing it? > Oh sure, I normally just leave it in my scratch svn checkout until I get around to it. But since we now have the decorator I was wondering if any discussion came up while I was gone of using it within our tests when we happen to get a working (failing) test case before we have a patch to fix it. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Jul 24 20:13:43 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 24 Jul 2009 18:13:43 +0000 (UTC) Subject: [Python-Dev] Document None values in sys.modules? References: <1afaf6160907231938u122a720ap7402eeeded0caff@mail.gmail.com> <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> Message-ID: Brett Cannon python.org> writes: > > Oh, I don't know. All this new-fangled technology... Can't you just > > leave it in your workspace, unsubmitted, until you get to fixing it? > > Oh sure, I normally just leave it in my scratch svn checkout until I get around to it. What? You're not even using a DVCS? From sridharr at activestate.com Fri Jul 24 20:47:32 2009 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Fri, 24 Jul 2009 11:47:32 -0700 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <4222a8490907230611tfbb8952m73345e4244eac9a2@mail.gmail.com> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <4A68264C.9060700@gmail.com> <36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com> <4222a8490907230611tfbb8952m73345e4244eac9a2@mail.gmail.com> Message-ID: On Thu, 23 Jul 2009 06:11:38 -0700, Jesse Noller wrote: > Then why not include pip, easy_install, and this bash script I use to > install packages into core? The more the merrier, right? > Answer: None of these are standards, and as nick points out, there's > issues with sysadmins, security, and other things. Not to mention > they're fundamentally not part of the language. > At the language summit, this was hashed out quite a bit. I think most > everyone agreed that tools like easy_install, pip, virtualenv, > zc.buildout, etc simply do not belong in core python. Once PEP 376 is implemented, my suggestion is to at least link to these tools (if not write a paragraph) as way of pointing users to alternative package managers that uses the 'uninstall' API. A mention of PyPI would also be helpful. -srid From brett at python.org Fri Jul 24 22:50:04 2009 From: brett at python.org (Brett Cannon) Date: Fri, 24 Jul 2009 13:50:04 -0700 Subject: [Python-Dev] Document None values in sys.modules? In-Reply-To: References: <1afaf6160907231948v27b64db6p64360db0f5795979@mail.gmail.com> Message-ID: On Fri, Jul 24, 2009 at 11:13, Antoine Pitrou wrote: > Brett Cannon python.org> writes: > > > Oh, I don't know. All this new-fangled technology... Can't you just > > > leave it in your workspace, unsubmitted, until you get to fixing it? > > > > Oh sure, I normally just leave it in my scratch svn checkout until I get > around to it. > > > What? You're not even using a DVCS? Not for Python; waiting for hg switch (all my other code uses hg). Never got a workflow involving both hg and svn up and going (and don't even mention gitsvn). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Jul 24 23:05:42 2009 From: brett at python.org (Brett Cannon) Date: Fri, 24 Jul 2009 14:05:42 -0700 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <4A68264C.9060700@gmail.com> <36724.203.171.195.115.1248342194.squirrel@syd-srv02.ezyreg.com> <4222a8490907230611tfbb8952m73345e4244eac9a2@mail.gmail.com> Message-ID: On Fri, Jul 24, 2009 at 11:47, Sridhar Ratnakumar wrote: > On Thu, 23 Jul 2009 06:11:38 -0700, Jesse Noller > wrote: > > Then why not include pip, easy_install, and this bash script I use to >> install packages into core? The more the merrier, right? >> Answer: None of these are standards, and as nick points out, there's >> issues with sysadmins, security, and other things. Not to mention >> they're fundamentally not part of the language. >> At the language summit, this was hashed out quite a bit. I think most >> everyone agreed that tools like easy_install, pip, virtualenv, >> zc.buildout, etc simply do not belong in core python. >> > > Once PEP 376 is implemented, my suggestion is to at least link to these > tools (if not write a paragraph) as way of pointing users to alternative > package managers that uses the 'uninstall' API. A mention of PyPI would also > be helpful. You can put that in the wiki, but it does not belong in the core documentation (except the PyPI mention as we do control that). That simply becomes a maintenance nightmare for use as we then have to keep up with the releases of external tools that we have no direct connection with. Plus it also give the illusion that Python has blessed these tools when in fact we have not. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lyon at preisshare.net Sat Jul 25 01:33:42 2009 From: david.lyon at preisshare.net (David Lyon) Date: Fri, 24 Jul 2009 19:33:42 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <5ecc64238b6e429c72441815bf818f48@preisshare.net> Message-ID: <74d8c1df41dac74a14d451324e0adfc7@preisshare.net> On Fri, 24 Jul 2009 17:54:09 +0200, Georg Brandl wrote: > David Lyon schrieb: >> In executable form, the Package Manager does not require wxWidgets >> to be installed. >> >> There is no dependency for this to be installed. > > What does "in exectuable form" mean? Compiled with py2exe. An executable ".exe" file is created under windows. David From david.lyon at preisshare.net Sat Jul 25 02:09:28 2009 From: david.lyon at preisshare.net (David Lyon) Date: Fri, 24 Jul 2009 20:09:28 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <79990c6b0907240908k7e8cc9f9xf99c91ffd5256d5d@mail.gmail.com> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <5ecc64238b6e429c72441815bf818f48@preisshare.net> <79990c6b0907240908k7e8cc9f9xf99c91ffd5256d5d@mail.gmail.com> Message-ID: On Fri, 24 Jul 2009 17:08:32 +0100, Paul Moore wrote: > I read this as meaning that David was proposing to ship a built > application (on Windows, bundled up with something like py2exe, I > guess) and any supporting DLLs such as the wxWindows ones would be > bundled in, but the wxPython package would *not* be shipped as part of > the standard library. That's correct. > In my view, for an "application" to be shipped with Python, whether it > be Idle, or a package manager, or pydoc, or whatever, it must: > - be written as a Python script > - depend only on packages shipped with the standard library This is true for the Package Manager in executable form. > Given that David's package manager uses setuptools to do the installs > (and now it appears it uses wxPython for its GUI) that immediately > disqualifies it for inclusion in my view - regardless of its value as > a package. Yes, It offers the choice to install via setuptools. As well as PIP. And it will be extended to Enstaller. Presently it used pkg_resources to read the list of packages installed which is part of setuptools. I was told it was the "right" and only way to read a list of packages. If using this package presents a problem, then it can be removed and I will replace the functionality with my own code. If the gui needs to be rewritten in TK this can be accomplished. I'm in no way pushing any GUI toolkit, and this was the one suggested to me by many. Redoing to meet the above criteria is achievable. If there's any other objections, can they be raised now before I start a rewrite? Thank you. David From david.lyon at preisshare.net Sat Jul 25 02:24:44 2009 From: david.lyon at preisshare.net (David Lyon) Date: Fri, 24 Jul 2009 20:24:44 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> Message-ID: <2e29b19bb5c64ce035f2c2a23f9f3534@preisshare.net> On Fri, 24 Jul 2009 03:23:57 +0200, Christian Heimes wrote: > I'm sorry to inform you that a wxWindows based solution has zero change > to get into the Python standard library ever. Is that a personal preference or is there a software engineering reason for this? I wasn't suggesting including it in the standard library as I like others know how troublesome it can be. > We are not going to add > another GUI toolkit to the core distribution. We might even remove TK > from the core and ship it as a separate package like bsddb3. So is this a bias against native window applications? Does the application need to be web based? or console based? or TK based... ? For it to be acceptable.. Thank you David From ben+python at benfinney.id.au Sat Jul 25 02:40:52 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 25 Jul 2009 10:40:52 +1000 Subject: [Python-Dev] Update to Python Documentation Website Request References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> Message-ID: <871vo561d7.fsf@benfinney.id.au> david.lyon at preisshare.net writes: > Distutils was once seperate and was then included in the standard > python. > > So i guess that I am working with the same goal in mind. I interpret this as expressing your intent to (eventually) have your application included in standard Python. David Lyon writes: > On Fri, 24 Jul 2009 03:23:57 +0200, Christian Heimes > wrote: > > I'm sorry to inform you that a wxWindows based solution has zero > > change to get into the Python standard library ever. > > I wasn't suggesting including it in the standard library as I like > others know how troublesome it can be. This appears to contradict what you said above. > > We are not going to add another GUI toolkit to the core > > distribution. We might even remove TK from the core and ship it as a > > separate package like bsddb3. > > So is this a bias against native window applications? No, it's a bias against adding things to Python that depend on things not already in Python. Are you, or are you not, intending for your application to be in standard Python? If that is your intent, I think Christian's answer is valid: applications, especially ones that depend on third-party widget libraries, shouldn't get added to Python. If that is not your intent, then your application shouldn't be mentioned in standard Python documentation. -- \ ?Stop ? Drive sideways.? ?detour sign, Kyushu, Japan | `\ | _o__) | Ben Finney From exarkun at divmod.com Sat Jul 25 03:06:10 2009 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Fri, 24 Jul 2009 21:06:10 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <871vo561d7.fsf@benfinney.id.au> Message-ID: <20090725010610.388.440797335.divmod.quotient.79@henry.divmod.com> On Sat, 25 Jul 2009 10:40:52 +1000, Ben Finney wrote: > [snip] > >If that is not your intent, then your application shouldn't be mentioned >in standard Python documentation. > Hm. But docutils isn't part of the standard library, and the documentation refers to it. And the docs link to ActivePython and Enthought's Python distribution. And they link to things like "Collin Winter?s functional module" and Django and PyOpenGL and so forth. I don't think anyone would seriously suggest removing all references to third-party software from the Python documentation. Perhaps the criterion that David is looking to have is explained is that people would probably prefer for his app to get a bit more popular before Python itself starts pointing people in its direction. Jean-Paul From david.lyon at preisshare.net Sat Jul 25 03:04:48 2009 From: david.lyon at preisshare.net (David Lyon) Date: Fri, 24 Jul 2009 21:04:48 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <871vo561d7.fsf@benfinney.id.au> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> Message-ID: <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> On Sat, 25 Jul 2009 10:40:52 +1000, Ben Finney wrote: > I interpret this as expressing your intent to (eventually) have your > application included in standard Python. It's my intention to get a Package Manager included in standard python - yes. >> I wasn't suggesting including it in the standard library as I like >> others know how troublesome it can be. > > This appears to contradict what you said above. Not at all. In source form the pythonpkgmgr requires wx package. In executable form it does not. > No, it's a bias against adding things to Python that depend on things > not already in Python. That implies that nothing new can be added then. > Are you, or are you not, intending for your application to be in > standard Python? > > If that is your intent, I think Christian's answer is valid: > applications, especially ones that depend on third-party widget > libraries, shouldn't get added to Python. > > If that is not your intent, then your application shouldn't be mentioned > in standard Python documentation. The Python Package Manager can be written to work in console mode. It can be redone to work with the TK that already ships. That doesn't present any great problem. If it was the interests of python users that were foremost, being able to install packages in a modern way - Then I think there would be more room to move than this. It's no different than any other windows application that ships with dll files. I fully appreciate that changes could be necessary and I'm prepared to accomodate them. Not much more I can do than that.. David From ben+python at benfinney.id.au Sat Jul 25 03:21:12 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 25 Jul 2009 11:21:12 +1000 Subject: [Python-Dev] Update to Python Documentation Website Request References: <871vo561d7.fsf@benfinney.id.au> <20090725010610.388.440797335.divmod.quotient.79@henry.divmod.com> Message-ID: <87ws5x4kxj.fsf@benfinney.id.au> Jean-Paul Calderone writes: > Hm. But docutils isn't part of the standard library, and the > documentation refers to it. Fair enough, because the documentation is generated using Docutils. > And the docs link to ActivePython and Enthought's Python distribution. I consider those to be exceptions that prove the rule: they are ?standard Python? in a different form. > And they link to things like "Collin Winter?s functional module" and > Django and PyOpenGL and so forth. I don't know enough about those to comment. > I don't think anyone would seriously suggest removing all references to > third-party software from the Python documentation. No, and nor did I. I'm suggesting that *adding* such a reference has a high barrier. > Perhaps the criterion that David is looking to have is explained is > that people would probably prefer for his app to get a bit more > popular before Python itself starts pointing people in its direction. Right. The process is usually in the following order: * get the code registered on PyPI * get a community of users and make the code meet their needs well * discuss adding some or all of the code to Python, by (in recent years) raising the topic on ?python-ideas? * defend the proposal against all comers to ensure it's actually a good idea * await further instructions -- \ ?It seems intuitively obvious to me, which means that it might | `\ be wrong.? ?Chris Torek | _o__) | Ben Finney From ben+python at benfinney.id.au Sat Jul 25 03:25:27 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 25 Jul 2009 11:25:27 +1000 Subject: [Python-Dev] Update to Python Documentation Website Request References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> Message-ID: <87skgl4kqg.fsf@benfinney.id.au> David Lyon writes: > Not at all. In source form the pythonpkgmgr requires wx package. In > executable form it does not. The only way it could be added is in source form; that's essential for free software like Python. So, if it's not suitable for adding to Python in source form, it's not suitable for adding to Python. > > No, it's a bias against adding things to Python that depend on things > > not already in Python. > > That implies that nothing new can be added then. You omit the important part: adding a new thing to Python *so long as it doesn't depend on anything outside Python*. > The Python Package Manager can be written to work in console mode. I think this would be best. The functionality you often discuss around this tool would be best implemented independently of any UI. It would, in fact, be best to work with the team performing ongoing active standardisation of distutils functionality. -- \ ?Consider the daffodil. And while you're doing that, I'll be | `\ over here, looking through your stuff.? ?Jack Handey | _o__) | Ben Finney From pje at telecommunity.com Sat Jul 25 03:37:35 2009 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 24 Jul 2009 21:37:35 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <5ecc64238b6e429c72441815bf818f48@preisshare.net> <79990c6b0907240908k7e8cc9f9xf99c91ffd5256d5d@mail.gmail.com> Message-ID: <20090725013432.793043A4099@sparrow.telecommunity.com> At 08:09 PM 7/24/2009 -0400, David Lyon wrote: >Presently it used pkg_resources to read the list of packages installed >which is part of setuptools. I was told it was the "right" and only way >to read a list of packages. "Right" is relative, but right now it is certainly the *only* way to read a list of installed packages, that's compatible with both distutils and setuptools-based ways of installing packages (including of course pip, enstaller, and buildout). PEP 376 proposes a stdlib replacement for a portion of this functionality, but the status of whether it's actually going to support anything other than distutils, pip, and setuptools' backward-compatibility mode is apparently still up for grabs. From ben+python at benfinney.id.au Sat Jul 25 03:38:29 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 25 Jul 2009 11:38:29 +1000 Subject: [Python-Dev] Update to Python Documentation Website Request References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> <87skgl4kqg.fsf@benfinney.id.au> Message-ID: <87ocr94k4q.fsf@benfinney.id.au> Ben Finney writes: > The functionality you often discuss around this tool would be best > implemented independently of any UI. It would, in fact, be best to > work with the team performing ongoing active standardisation of > distutils functionality. Sloppy use of ?in fact?. That should be ?in my opinion?. -- \ ?Our wines leave you nothing to hope for.? ?restaurant menu, | `\ Switzerland | _o__) | Ben Finney From horos11 at gmail.com Sat Jul 25 03:43:59 2009 From: horos11 at gmail.com (Edward Peschko) Date: Fri, 24 Jul 2009 18:43:59 -0700 Subject: [Python-Dev] command line attachable debugger In-Reply-To: <82FB8A40-F5AD-4980-83C6-558617B49532@fuhm.net> References: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com> <82FB8A40-F5AD-4980-83C6-558617B49532@fuhm.net> Message-ID: <5cfa99000907241843n3048e260o50a8d1c675e0d805@mail.gmail.com> > Winpdb is *exactly* what you asked for, so if it's not what you're looking > for you'll need to be more specific about what you want that it doesn't > do... > > James > Fair enough, I was assuming that winpdb didn't have a command line interface, but I was mistaken (rpdb2). In any case, I would argue that attach functionality is something that should be put into the core, along with pdb, as a standard. What if I want to use pudb? or pydb? or the interface which goes with perl? There should be a standard mechanism for debuggers to talk to a python process; not one-offs for each debugger, probably at the level of the python executable (the same way that gcc lets gdb attach with a pid.. Ed From david.lyon at preisshare.net Sat Jul 25 04:20:05 2009 From: david.lyon at preisshare.net (David Lyon) Date: Fri, 24 Jul 2009 22:20:05 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <87skgl4kqg.fsf@benfinney.id.au> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> <87skgl4kqg.fsf@benfinney.id.au> Message-ID: On Sat, 25 Jul 2009 11:25:27 +1000, Ben Finney > You omit the important part: adding a new thing to Python *so long as it > doesn't depend on anything outside Python*. I'm signing out on this silly discussion for now.... Any python program is dependant on things "outside" python. For example an operating system.. a computer... and a user perhaps for without such it wouldn't be possible to generate the all important 1Di0t errors that we so often have to deal with. >> The Python Package Manager can be written to work in console mode. > > I think this would be best. Haha - I'm glad somebody took this seriously... It was a sort of a joke comment but it's a serious possibility. > The functionality you often discuss around > this tool would be best implemented independently of any UI. It is anyway. That's why doing an entirely different UI isn't any major issue. I have a class and a possible command line interface behind the scenes. > It would, in fact, be best to work with the team performing ongoing > active standardisation of distutils functionality. I am already doing that. But there is a bias against windows development and a bias against native applications. That's fine because I know they are using python on different platforms. I'm working on a proposal to make setup.py object oriented and "modern". http://wiki.python.org/moin/Distutils/Proposals So I'm doing as much as I can - really. David From orsenthil at gmail.com Sat Jul 25 06:03:23 2009 From: orsenthil at gmail.com (Senthil Kumaran) Date: Sat, 25 Jul 2009 09:33:23 +0530 Subject: [Python-Dev] Backporting HTTPS via Proxy Support in urllib2 Message-ID: <20090725040323.GA6260@ubuntu.ubuntu-domain> Issue: http://bugs.python.org/issue1424152 mentions about HTTPS Support via proxy in urllib2. This is fixed in the trunk (Revision 72880), but there has been number of valid requests to backport it Python 2.6. While I agree and ready to backport to Python 2.6, I would like to ask here if there are any objections in this front. I am ready with the Python 3.x patch as well (with tests passed and tested under proxy setup). I shall apply that too, which is pending for a while now. :( -- Senthil Wad some power the giftie gie us To see oursels as others see us. -- R. Burns From stephen at xemacs.org Sat Jul 25 07:18:50 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 25 Jul 2009 14:18:50 +0900 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <5ecc64238b6e429c72441815bf818f48@preisshare.net> <79990c6b0907240908k7e8cc9f9xf99c91ffd5256d5d@mail.gmail.com> Message-ID: <873a8l49xh.fsf@uwakimon.sk.tsukuba.ac.jp> David Lyon writes: > On Fri, 24 Jul 2009 17:08:32 +0100, Paul Moore wrote: > > I read this as meaning that David was proposing to ship a built > > application (on Windows, bundled up with something like py2exe, I > > guess) and any supporting DLLs such as the wxWindows ones would be > > bundled in, but the wxPython package would *not* be shipped as part of > > the standard library. > > That's correct. -1, then. Python is open source. Anything shipped with the standard distribution should be buildable with the Pythonic tools in the standard distribution. Obviously the C compiler and operating system services are non-Pythonic, and the line is pretty fuzzy, but I would not want to extend similar consideration to wxPython. This kind of thing is what ActiveState and the OS distros are for. If you say this is discriminating against Windows, well, it probably is, but it should be addressed with a Windows solution, such as a script to generate MSIs. From glyph at twistedmatrix.com Sat Jul 25 07:50:09 2009 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Sat, 25 Jul 2009 01:50:09 -0400 Subject: [Python-Dev] command line attachable debugger In-Reply-To: <5cfa99000907241843n3048e260o50a8d1c675e0d805@mail.gmail.com> References: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com> <82FB8A40-F5AD-4980-83C6-558617B49532@fuhm.net> <5cfa99000907241843n3048e260o50a8d1c675e0d805@mail.gmail.com> Message-ID: On Fri, Jul 24, 2009 at 9:43 PM, Edward Peschko wrote: > There should be a standard mechanism for debuggers to talk to a python > process; not one-offs for each debugger, probably at the level of the > python executable (the same way that gcc lets gdb attach with a pid.. > Sounds like this is moving into hypothetical territory better-suited to python-ideas. (Although I'm sure that if you wanted to contribute polished, tested code for a standard remote debugger interface, few people would complain.) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Sat Jul 25 08:00:13 2009 From: ben+python at benfinney.id.au (Ben Finney) Date: Sat, 25 Jul 2009 16:00:13 +1000 Subject: [Python-Dev] Update to Python Documentation Website Request References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> <87skgl4kqg.fsf@benfinney.id.au> Message-ID: <87fxcl480i.fsf@benfinney.id.au> David Lyon writes: > On Sat, 25 Jul 2009 11:25:27 +1000, Ben Finney > > You omit the important part: adding a new thing to Python *so long > > as it doesn't depend on anything outside Python*. > > I'm signing out on this silly discussion for now.... > > Any python program is dependant on things "outside" python. I think you know quite well what ?depend on? means in this instance, so this is taking it to silly extremes. -- \ ?For your convenience we recommend courteous, efficient | `\ self-service.? ?supermarket, Hong Kong | _o__) | Ben Finney From nyamatongwe at gmail.com Sat Jul 25 08:24:03 2009 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Sat, 25 Jul 2009 16:24:03 +1000 Subject: [Python-Dev] command line attachable debugger In-Reply-To: References: <5cfa99000907232231h5f1c5bb4kc15f75a151d20d2@mail.gmail.com> <82FB8A40-F5AD-4980-83C6-558617B49532@fuhm.net> <5cfa99000907241843n3048e260o50a8d1c675e0d805@mail.gmail.com> Message-ID: <50862ebd0907242324j6d434d2ahd85f7a433af5e4db@mail.gmail.com> Glyph Lefkowitz: > Sounds like this is moving into hypothetical territory better-suited to > python-ideas.? (Although I'm sure that if you wanted to contribute polished, > tested code for a standard remote debugger interface, few people would > complain.) There is a remote debugger protocol called DBGP for different languages (including Python) and debuggers (such as Komodo) http://xdebug.org/docs-dbgp.php Neil From p.f.moore at gmail.com Sat Jul 25 11:28:51 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 25 Jul 2009 10:28:51 +0100 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> <87skgl4kqg.fsf@benfinney.id.au> Message-ID: <79990c6b0907250228y8ab057bi614e6e551466c69d@mail.gmail.com> 2009/7/25 David Lyon : >> It would, in fact, be best to work with the team performing ongoing >> active standardisation of distutils functionality. > > I am already doing that. > > But there is a bias against windows development and a bias > against native applications. That's fine because I know they > are using python on different platforms. ??? I see no bias as you describe in the distutils enhancement work. Everything so far has been carefully platform-neutral (ie, *not* biased against Windows, but equally not biased against other operating systems). Native applications are by definition not platform neutral. How does your proposal help Linux users? Mac OS? Solaris? I think you are seeing bias against Windows where non exists. What *does* exist is - A concern that tools work for everyone, regardless of OS - A limited number of people willing to explain Windows issues so that they get considered > I'm working on a proposal to make setup.py object oriented > and ?"modern". > > http://wiki.python.org/moin/Distutils/Proposals And that looks interesting, and potentially useful (although perhaps more radical than will end up being possible). > So I'm doing as much as I can - really. If your concern is to make things easier for Windows users, then your application is worthwhile, but it should probably remain an external project. If it gets overwhelming support, maybe providing a standardised version with a simplified UI in the core would be an option. How many users do you have currently? Paul. PS Does your application work with the large existing base of bdist_msi and bdist_wininst installers? Unless it will manage pywin32, cx_Oracle, PIL, wxPython, pyQT, pygame, numpy etc (all of which are available in binary form but not as eggs as far as I know, and have too complex a set of dependencies for me to build locally) it's useless to me. From martin at v.loewis.de Sat Jul 25 11:42:13 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 25 Jul 2009 11:42:13 +0200 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> Message-ID: <4A6AD375.8000805@v.loewis.de> > It's my intention to get a Package Manager included in standard > python - yes. In addition to the other constraints you'll have to meet for this to happen, you also have to wait a rather long time (several years) before inclusion becomes possible. This time is necessary for the community to accept your tool, and evaluate it. Ideally, the request for inclusion should not come from you, but from your users. > Not at all. In source form the pythonpkgmgr requires wx package. In > executable form it does not. If we include it, we *only* include it in source form. Inclusion in executable form is not acceptable. We also require that it works on all major systems, not just windows. >> No, it's a bias against adding things to Python that depend on things >> not already in Python. > > That implies that nothing new can be added then. No. It implies that any addition could only depend on Python (and OS API) - so things *can* be added. For example, setuptools could be added by this principle. OTOH, if your tool depends on setuptools, you'll have to wait for setuptools to get included before inclusion of your tool can be considered. Regards, Martin From martin at v.loewis.de Sat Jul 25 11:53:38 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 25 Jul 2009 11:53:38 +0200 Subject: [Python-Dev] Backporting HTTPS via Proxy Support in urllib2 In-Reply-To: <20090725040323.GA6260@ubuntu.ubuntu-domain> References: <20090725040323.GA6260@ubuntu.ubuntu-domain> Message-ID: <4A6AD622.2050400@v.loewis.de> > This is fixed in the trunk (Revision 72880), but there has been number > of valid requests to backport it Python 2.6. While I agree and ready > to backport to Python 2.6, I would like to ask here if there are any > objections in this front. In msg76855 Greg ruled that it is a new feature, and thus cannot be backported. I trust his judgement - so to convince me, you would need to convince Greg first :-) Regards, Martin From martin at v.loewis.de Sat Jul 25 13:05:20 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 25 Jul 2009 13:05:20 +0200 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A68D2BB.8060607@stackless.com> References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> Message-ID: <4A6AE6F0.2060303@v.loewis.de> > To Martin: So I disagree. The gc header is not responsible for > alignment in the first place, but to propagate it, correctly. I think we are in agreement in this respect. That's the whole point of the long double: to make the gc head's alignment the maximum alignment on that platform > And this fails miserably (in principle) since years. Here I agree. In principle, it works fine - it is just perhaps incomplete. > Proposal: We should use a simple construct that makes the > gc header size simply a multiple of 8 or 16, whatever needed. > Even a byte array would be ok. How would you implement that? In particular, how do you determine what is needed? > But please no long double :-) Well, the long double needs to stay, as for some platforms, long double is indeed the type with the maximum alignment. I propose to add another (regular) double into the union. Regards, Martin From martin at v.loewis.de Sat Jul 25 13:09:57 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 25 Jul 2009 13:09:57 +0200 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> Message-ID: <4A6AE805.3000005@v.loewis.de> >> This may not be recognized so far, because there is no builtin >> GC-enabled type that embeds a double. >> But if you create such a type, then the double will be correctly >> aligned in your object's structure, but then, when the object >> gets allocated, it is misaligned, because the header size is not a >> multiple of 8. > > I'm not sure a double aligned on a 4-byte boundary is "misaligned" on a x86 CPU. That's not really the point. The question is whether the double is misaligned wrt. the compiler's ABI. The compiler (or the operating system) specifies the alignment for all primitive types, and guarantees that everything is properly aligned. > Alignment is primarily important to avoid access violations on CPUs and > datatypes which don't support arbitrary alignment, although it can also be > useful for performance reasons. Alignment may also have propagated into other aspects. For example, if you have double foo; intptr_t bar = (intptr_t)&foo; then it may be guaranteed (by the ABI) that bar&7 == 0. So we really need to keep all promises wrt. alignment intact. > In any case, you seem to be right on this particular point: the PyGC_Head union > should probably contain a "double" alternative in addition to the "long double" > (and perhaps even a "long long" one). I agree. Regards, Martin From martin at v.loewis.de Sat Jul 25 13:13:03 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 25 Jul 2009 13:13:03 +0200 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> Message-ID: <4A6AE8BF.8040405@v.loewis.de> > Antoine Pitrou pitrou.net> writes: >> In any case, you seem to be right on this particular point: the PyGC_Head union >> should probably contain a "double" alternative in addition to the "long double" >> (and perhaps even a "long long" one). > > Sorry, I realize that this doesn't really address the point. I don't realize that. Why is your first proposal bad? > In addition to that union, we should also have a particular mechanism to compute > what the proper offset should be between the PyGC_Head and the PyObject. Why is that difficult? It's sizeof(PyGC_Head). Regards, Martin From martin at v.loewis.de Sat Jul 25 13:15:39 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 25 Jul 2009 13:15:39 +0200 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A68E0E0.7010508@stackless.com> References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A68E0E0.7010508@stackless.com> Message-ID: <4A6AE95B.20301@v.loewis.de> > For that reason, I don't like the addition of the opaque header > too much. If there were an option to make the header explicit, > we would not have to round up the object size to a multiple > of (8, 16), but could arrange embedded doubles as they fit the best. We could add the gc head *after* the object head. However, this will require manual changes to all types that support GC, so we can't do that before Python 4. >> (I disagree, however, that we should remove the "long double". After >> all, we >> also want alignment of PyObjects to allow inclusion of a "long double" >> in them) > > Well, I doubt that a 12 byte long double causes any other > alignment but 4. What you apparently fail to understand is this: On some platforms, sizeof(long double) == 16, and alignof(long double) == 16. It is these platforms why we need to keep the long double. Regards, Martin From solipsis at pitrou.net Sat Jul 25 13:30:13 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 25 Jul 2009 11:30:13 +0000 (UTC) Subject: [Python-Dev] mingw32 and gc-header weirdness References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A6AE8BF.8040405@v.loewis.de> Message-ID: Martin v. L?wis v.loewis.de> writes: > > > In addition to that union, we should also have a particular mechanism to compute > > what the proper offset should be between the PyGC_Head and the PyObject. > > Why is that difficult? It's sizeof(PyGC_Head). Is it enough to correctly propagate the alignment for, say, a long double in the following PyObject? I'm sorry, I'm not enough of a C lawyer. Regards Antoine. From david.lyon at preisshare.net Sat Jul 25 14:57:22 2009 From: david.lyon at preisshare.net (David Lyon) Date: Sat, 25 Jul 2009 08:57:22 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <87fxcl480i.fsf@benfinney.id.au> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> <87skgl4kqg.fsf@benfinney.id.au> <87fxcl480i.fsf@benfinney.id.au> Message-ID: On Sat, 25 Jul 2009 16:00:13 +1000, Ben Finney wrote: > I think you know quite well what ?depend on? means in this instance, > so this is taking it to silly extremes. haha - yes - no offence. It was just bad humour. Have a nice weekend David From david.lyon at preisshare.net Sat Jul 25 15:41:24 2009 From: david.lyon at preisshare.net (David Lyon) Date: Sat, 25 Jul 2009 09:41:24 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <79990c6b0907250228y8ab057bi614e6e551466c69d@mail.gmail.com> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> <87skgl4kqg.fsf@benfinney.id.au> <79990c6b0907250228y8ab057bi614e6e551466c69d@mail.gmail.com> Message-ID: <56b9b173c5deb6563cce6aa93c317b7c@preisshare.net> On Sat, 25 Jul 2009 10:28:51 +0100, Paul Moore wrote: > ??? I see no bias as you describe in the distutils enhancement work. ok > Native applications are by definition not platform neutral. How does > your proposal help Linux users? Mac OS? Solaris? I'm doing a Linux/Solaris version. But I find issues every day that must be addressed. > If your concern is to make things easier for Windows users, then your > application is worthwhile, but it should probably remain an external > project. If it gets overwhelming support, maybe providing a > standardised version with a simplified UI in the core would be an > option. That sounds ok. > How many users do you have currently? Approx 250 downloads this (first) month for the windows version. I need to extend coverage to Linux and the Mac. Given that I'm relatively new to all this there's a learning curve. But so far so good. David From nyamatongwe at gmail.com Sat Jul 25 16:11:30 2009 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Sun, 26 Jul 2009 00:11:30 +1000 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A6AE6F0.2060303@v.loewis.de> References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de> Message-ID: <50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com> Martin v. L?wis: > I propose to add another (regular) double into the union. Adding a regular double as a second dummy gives the same sizes and alignments with Mingw or MSVC as the original definition with MSVC: typedef union _gc_head { struct { union _gc_head *gc_next; union _gc_head *gc_prev; Py_ssize_t gc_refs; } gc; long double dummy; /* force worst-case alignment */ double dummy2; /* in case long double doesn't trigger worst-case */ } PyGC_Head; In regard to alignment penalties, a simple copy loop for doubles runs about 20% slower when misaligned on an my AMD processor. Other x86 processors can be much worse. As much as 2 to 3.25 times according to http://msdn.microsoft.com/en-us/library/aa290049%28VS.71%29.aspx Neil From solipsis at pitrou.net Sat Jul 25 16:19:36 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 25 Jul 2009 14:19:36 +0000 (UTC) Subject: [Python-Dev] mingw32 and gc-header weirdness References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de> <50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com> Message-ID: Neil Hodgson gmail.com> writes: > > typedef union _gc_head { > struct { > union _gc_head *gc_next; > union _gc_head *gc_prev; > Py_ssize_t gc_refs; > } gc; > long double dummy; /* force worst-case alignment */ > double dummy2; /* in case long double doesn't trigger worst-case */ > } PyGC_Head; We could add a PY_LONG_LONG to the mix just in case. By the way, will this kind of thing be frozen by the PEP ABI? Regards Antoine. From solipsis at pitrou.net Sat Jul 25 16:22:05 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 25 Jul 2009 14:22:05 +0000 (UTC) Subject: [Python-Dev] mingw32 and gc-header weirdness References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de> <50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com> Message-ID: Antoine Pitrou pitrou.net> writes: > > We could add a PY_LONG_LONG to the mix just in case. > By the way, will this kind of thing be frozen by the PEP ABI? (I meant the ABI PEP, sorry) From martin at v.loewis.de Sat Jul 25 17:22:06 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 25 Jul 2009 17:22:06 +0200 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A6AE8BF.8040405@v.loewis.de> Message-ID: <4A6B231E.5010502@v.loewis.de> >> Why is that difficult? It's sizeof(PyGC_Head). > > Is it enough to correctly propagate the alignment for, say, a long double in > the following PyObject? I'm sorry, I'm not enough of a C lawyer. Yes, sizeof must always be a multiple of the alignment. Otherwise, array elements would be misaligned. Regards, Martin From martin at v.loewis.de Sat Jul 25 17:25:01 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 25 Jul 2009 17:25:01 +0200 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de> <50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com> Message-ID: <4A6B23CD.1010306@v.loewis.de> > We could add a PY_LONG_LONG to the mix just in case. > By the way, will this kind of thing be frozen by the PEP ABI? Yes: alignof(PyGC_HEAD) would be specified as being the maximum alignment on a platform; sizeof(PyGC_HEAD) would be frozen. The actual content would not be frozen, i.e. it could be changed in principle as long as the change would be guaranteed not to affect the structure size. Regards, Martin From tjreedy at udel.edu Sat Jul 25 18:47:21 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 25 Jul 2009 12:47:21 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> <87skgl4kqg.fsf@benfinney.id.au> Message-ID: David Lyon wrote: >>> The Python Package Manager can be written to work in console mode. >> I think this would be best. > > Haha - I'm glad somebody took this seriously... It was a sort of a joke > comment but it's a serious possibility. I took it seriously too ;-). It seems to me that you project can have at least 3 components. 1) the core library module, which might or might not be targeted at eventual stdlib inclusion. 2) a small console driver script. 3) one or more GUI scripts. A TK version would be good from my viewpoint as it does not require download and installation of any other binaries. Removal of TK from the Windows distribution will be extremely contentious if seriously proposed, so it could not happen for a while. tjr From greg at krypto.org Sat Jul 25 19:05:09 2009 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 25 Jul 2009 10:05:09 -0700 Subject: [Python-Dev] Backporting HTTPS via Proxy Support in urllib2 In-Reply-To: <4A6AD622.2050400@v.loewis.de> References: <20090725040323.GA6260@ubuntu.ubuntu-domain> <4A6AD622.2050400@v.loewis.de> Message-ID: <52dc1c820907251005k2f52a3bco6c773a464d96e766@mail.gmail.com> On Sat, Jul 25, 2009 at 2:53 AM, "Martin v. L?wis" wrote: >> This is fixed in the trunk (Revision 72880), but there has been number >> of valid requests to backport it Python 2.6. While I agree and ready >> to backport to Python 2.6, I would like to ask here if there are any >> objections in this front. > > In msg76855 Greg ruled that it is a new feature, and thus cannot be > backported. I trust his judgement - so to convince me, you would need > to convince Greg first :-) > > Regards, > Martin So long as the new set_tunnel() method is hidden in the backport I'm fine with the patch. It fixes urllib2 do the right thing with https and a proxy without any public API change. -Greg From greg at krypto.org Sat Jul 25 19:25:35 2009 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 25 Jul 2009 10:25:35 -0700 Subject: [Python-Dev] pthreads, fork, import, and execvp In-Reply-To: <9e804ac0907231628vd9de823h95656a02ffc59bc4@mail.gmail.com> References: <446A4202.6070608@v.loewis.de> <446C2211.5060408@v.loewis.de> <446C6FC9.5060205@gmail.com> <446CB69E.4090609@v.loewis.de> <9e804ac0907161308t19dc5399q2484339b18fc6318@mail.gmail.com> <3d2ce8cb0907201126t2ce8866eq3580aed9e3e6bf2@mail.gmail.com> <9e804ac0907201716v584e3337t3be9a7bf3311ff3b@mail.gmail.com> <9e804ac0907231628vd9de823h95656a02ffc59bc4@mail.gmail.com> Message-ID: <52dc1c820907251025j4db82363yea166be14a012ada@mail.gmail.com> On Thu, Jul 23, 2009 at 4:28 PM, Thomas Wouters wrote: > > So attached (and at http://codereview.appspot.com/96125/show ) is a > preliminary fix, correcting the problem with os.fork(), os.forkpty() and > os.fork1(). This doesn't expose a general API for C code to use, for two > reasons: it's not easy, and I need this fix more than I need the API change > :-) (I actually need this fix myself for Python 2.4, but it applies fairly > cleanly.) This looks good to me. Your idea of making this an API called a 'fork lock' or something sounds good (though I think it needs a better name. PyBeforeFork & PyAfterFork?). The subprocess module, for example, disables garbage collection before forking and restores it afterwards to avoid http://bugs.python.org/issue1336. That type of thing could also be done in such a function. Related to the above subprocess fork + gc bug.. It'd be nice for CPython to have code that does the fork+misc twiddling+exec all in one C call without needing to execute Python code in the child process prior to the exec(). Most of the inner body of subprocess's _execute_child() method could be done that way. (with the obvious exception of the preexec_fn) > > To fix the mutex-across-fork problem correctly we should really acquire > three locks (the import lock, the GIL and the TLS lock, in that order.) The > import lock is re-entrant, and the TLS lock is tightly confined to the > thread-local-storage lookup functions, but the GIL is neither re-entrant nor > inspectable. That means we can't use pthread_atfork (we can't tell whether > we have the GIL already or not, right before the fork), nor can we provide a > convenient API for extension modules to use, really. The best we can do is > provide three functions, pthread_atfork-style: a 'prepare' function, an > 'after-fork-in-child' function, and an 'after-fork-in-parent' function. The > 'prepare' function would start by releasing the GIL, then acquire the import > lock, the GIL and the TLS lock in that order. It would also record > *somewhere* the thread_ident of the current thread. The 'in-parent' function > would simply release the TLS lock and the import lock, and the 'in-child' > would release those locks, call the threading._at_fork() function, and fix > up the TLS data, using the recorded thread ident to do lookups. The > 'in-child' function would replace the current PyOS_AfterFork() function > (which blindly reinitializes the GIL and the TLS lock, and calls > threading._at_fork().) > > Alternatively we could do what we do now, which is to ignore the fact that > thread idents may change by fork() and thus that thread-local data may > disappear, in which case the 'in-child' function could do a little less > work. I'm suitably scared and disgusted of the TLS implementation, the > threading implementations we support (including the pthread one) and the way > we blindly type-pun a pthread_t to a long, that I'm not sure I want to try > and build something correct and reliable on top of it. > > -- > Thomas Wouters > > Hi! I'm a .signature virus! copy me into your .signature file to help me > spread! > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > > From brett at python.org Sun Jul 26 00:28:39 2009 From: brett at python.org (Brett Cannon) Date: Sat, 25 Jul 2009 15:28:39 -0700 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <20090725010610.388.440797335.divmod.quotient.79@henry.divmod.com> References: <871vo561d7.fsf@benfinney.id.au> <20090725010610.388.440797335.divmod.quotient.79@henry.divmod.com> Message-ID: On Fri, Jul 24, 2009 at 18:06, Jean-Paul Calderone wrote: > On Sat, 25 Jul 2009 10:40:52 +1000, Ben Finney > > wrote: > >> [snip] >> >> If that is not your intent, then your application shouldn't be mentioned >> in standard Python documentation. >> >> > Hm. But docutils isn't part of the standard library, and the documentation > refers to it. Why do you think distutils is not part of the standard library? The official code location is Lib/distutils within the standard library. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Jul 26 00:35:22 2009 From: brett at python.org (Brett Cannon) Date: Sat, 25 Jul 2009 15:35:22 -0700 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <871vo561d7.fsf@benfinney.id.au> <20090725010610.388.440797335.divmod.quotient.79@henry.divmod.com> Message-ID: On Sat, Jul 25, 2009 at 15:28, Brett Cannon wrote: > > > On Fri, Jul 24, 2009 at 18:06, Jean-Paul Calderone wrote: > >> On Sat, 25 Jul 2009 10:40:52 +1000, Ben Finney < >> ben+python at benfinney.id.au > wrote: >> >>> [snip] >>> >>> If that is not your intent, then your application shouldn't be mentioned >>> in standard Python documentation. >>> >>> >> Hm. But docutils isn't part of the standard library, and the >> documentation >> refers to it. > > > Why do you think distutils is not part of the standard library? The > official code location is Lib/distutils within the standard library. > Oops, I misread the project JP mentioned. Ignore me. =) -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Sun Jul 26 00:36:44 2009 From: robert.kern at gmail.com (Robert Kern) Date: Sat, 25 Jul 2009 17:36:44 -0500 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <871vo561d7.fsf@benfinney.id.au> <20090725010610.388.440797335.divmod.quotient.79@henry.divmod.com> Message-ID: On 2009-07-25 17:28, Brett Cannon wrote: > > > On Fri, Jul 24, 2009 at 18:06, Jean-Paul Calderone > wrote: > > On Sat, 25 Jul 2009 10:40:52 +1000, Ben Finney > > > wrote: > > [snip] > > > If that is not your intent, then your application shouldn't be > mentioned > in standard Python documentation. > > > Hm. But docutils isn't part of the standard library, and the > documentation > refers to it. > > > Why do you think distutils is not part of the standard library? The > official code location is Lib/distutils within the standard library. He said "docutils", not "distutils". -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From nyamatongwe at gmail.com Sun Jul 26 00:44:07 2009 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Sun, 26 Jul 2009 08:44:07 +1000 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A6B23CD.1010306@v.loewis.de> References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de> <50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com> <4A6B23CD.1010306@v.loewis.de> Message-ID: <50862ebd0907251544o66a3e0e3qd9d946a80923959e@mail.gmail.com> Martin v. L?wis: > Yes: alignof(PyGC_HEAD) would be specified as being the maximum > alignment on a platform; sizeof(PyGC_HEAD) would be frozen. Maximum alignment currently on x86 is 16 bytes for SSE vector types. Next year AVX will add 32 byte types and while they are supposed to work OK with 16 byte alignment, performance will be better with 32 byte alignment. It is possible that some use could be found for vector instructions in core Python but it is more likely that they will only be used in specialized extensions that can take care of alignment issues for their own cases. http://en.wikipedia.org/wiki/Advanced_Vector_Extensions http://software.intel.com/en-us/forums/intel-avx-and-cpu-instructions/topic/61891/ Neil From martin at v.loewis.de Sun Jul 26 01:28:08 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 26 Jul 2009 01:28:08 +0200 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <50862ebd0907251544o66a3e0e3qd9d946a80923959e@mail.gmail.com> References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de> <50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com> <4A6B23CD.1010306@v.loewis.de> <50862ebd0907251544o66a3e0e3qd9d946a80923959e@mail.gmail.com> Message-ID: <4A6B9508.1030707@v.loewis.de> > Maximum alignment currently on x86 is 16 bytes for SSE vector > types. Next year AVX will add 32 byte types and while they are > supposed to work OK with 16 byte alignment, performance will be better > with 32 byte alignment. That doesn't really matter. What matters is the platform's ABI, i.e. what the maximum alignment is that the compiler gives, and what the alignment of a malloc() result is. That is unlikely to change even if x86 evolves. Regards, Martin From david.lyon at preisshare.net Sun Jul 26 03:38:55 2009 From: david.lyon at preisshare.net (David Lyon) Date: Sat, 25 Jul 2009 21:38:55 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> <87skgl4kqg.fsf@benfinney.id.au> Message-ID: <4a601dcc46d513bfd446cbf141654401@preisshare.net> On Sat, 25 Jul 2009 12:47:21 -0400, Terry Reedy wrote: >>>> The Python Package Manager can be written to work in console mode. >>> I think this would be best. >> >> Haha - I'm glad somebody took this seriously... It was a sort of a joke >> comment but it's a serious possibility. > > I took it seriously too ;-). It seems to me that you project can have at > least 3 components. It's logical and plausible. > 1) the core library module, which might or might not be targeted at eventual stdlib inclusion. I'm sure they'll fix distutils someday, and if they do, then I'll use that. > 2) a small console driver script. 3) one or more GUI scripts. It's possible. I'll give it serious consideration. > A TK version would be good from my > viewpoint as it does not require download and installation of any other > binaries. Sure. Who cares if it's not as slick as something else. I'll try when I have time. David From david.lyon at preisshare.net Sun Jul 26 06:32:55 2009 From: david.lyon at preisshare.net (David Lyon) Date: Sun, 26 Jul 2009 00:32:55 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <4A6AD375.8000805@v.loewis.de> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> <4A6AD375.8000805@v.loewis.de> Message-ID: On Sat, 25 Jul 2009 11:42:13 +0200, "Martin v. L?wis" wrote: >> It's my intention to get a Package Manager included in standard >> python - yes. > > In addition to the other constraints you'll have to meet for this > to happen, you also have to wait a rather long time (several years) > before inclusion becomes possible. This time is necessary for the > community to accept your tool, and evaluate it. Ideally, the request > for inclusion should not come from you, but from your users. Yes. But users have been asking for a tool and complaining loudly about the lack of such a tool. I know you're an extremely competent and those emails you perphaps floss over. But the pleas are many and when we compare python to other languages, python rates towards at the low end of the spectrum its third party-package management facilities. You can't seriously expect users to wait for years for an integrated package management tool. Especially on Windows - that's cruel :-) To date, there just hasn't been any movement on the building of such a tool due to whatever reasons. A Package Manager isn't a frivolous "nice-to-have" tool. It's essential for a new user. (if you want I can sift the last twelve months worth of ML archives and report on how many times easier package management has been asked for. And how difficult people are finding it) > No. It implies that any addition could only depend on Python (and > OS API) - so things *can* be added. For example, setuptools could > be added by this principle. OTOH, if your tool depends on setuptools, > you'll have to wait for setuptools to get included before inclusion > of your tool can be considered. Let's get precise. It doesn't "depend" on setuptools. But it will install setuptools if the user requests to use setuptools/easy_install. So we should only be back to the lack of a TK interface and the fact that the Package Manager Project is a new project, and needs to be working properly on more platforms. Thanks for your email. I appreciate the feedback, from everyone. Take care. David From martin at v.loewis.de Sun Jul 26 08:43:07 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 26 Jul 2009 08:43:07 +0200 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> <4A6AD375.8000805@v.loewis.de> Message-ID: <4A6BFAFB.1030807@v.loewis.de> > You can't seriously expect users to wait for years for an integrated > package management tool. Especially on Windows - that's cruel :-) Hmm. I'm -0 on providing a tool whose only purpose is to download files from a web server. I always use a web browser for that... > A Package Manager isn't a frivolous "nice-to-have" tool. It's > essential for a new user. Hmm. I would expect that a new user is faced with the challenge of finding out what packages to install more so than with actually installing them. If they read examples, they will see import statements, and then they have to find out how to make those work. Does your tool help with that? When the user is not so new anymore, I seriously doubt that they still ask for a package management tool - except perhaps for dependencies, and here they use easy_install. Regards, Martin From steve at pearwood.info Sun Jul 26 10:05:07 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 26 Jul 2009 18:05:07 +1000 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <4A6AD375.8000805@v.loewis.de> Message-ID: <200907261805.08277.steve@pearwood.info> On Sun, 26 Jul 2009 02:32:55 pm David Lyon wrote: > > In addition to the other constraints you'll have to meet for this > > to happen, you also have to wait a rather long time (several years) > > before inclusion becomes possible. This time is necessary for the > > community to accept your tool, and evaluate it. Ideally, the > > request for inclusion should not come from you, but from your > > users. > > Yes. But users have been asking for a tool and complaining loudly > about the lack of such a tool. I know you're an extremely competent > and those emails you perphaps floss over. But the pleas are many and > when we compare python to other languages, python rates towards at > the low end of the spectrum its third party-package management > facilities. > > You can't seriously expect users to wait for years for an integrated > package management tool. Especially on Windows - that's cruel :-) Life is cruel and hard. Anyone who tells you different is selling something. Nobody is saying that users have to wait for years for such a tool. They can download it from wherever you host it. But you shouldn't expect the Python dev team to accept an unproven tool into the official library before demonstrating both the need and the solution. (Just because users say they want something, doesn't mean they'll actually use the tool you provide -- perhaps they don't know what they really need, and perhaps your tool doesn't match their needs.) I'm a user, and personally I don't want Yet Another Integrated Package Management Tool. What I really want is the ability to install Python packages using the PM tool I already use, namely yum. (And I'd like a pony.) Failing that, I want the Python standard library to be stable and reliable. With the greatest of respect, fast-tracking unproven tools into the library based on the author's say-so is the first step to instability and unreliability. Putting your software on the Cheeseshop, and making regular announcements to the Python community (e.g. on comp.lang.python) will be a good way to publicise the tool, and if does meet a need, people will use it, and then, if it's good enough and popular enough and supported, it may be blessed by inclusion in the standard library. Although I'm not a Windows user, let me say thank you for your efforts on behalf of Python users. Good luck! -- Steven D'Aprano From ncoghlan at gmail.com Sun Jul 26 13:44:38 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Jul 2009 21:44:38 +1000 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <200907261805.08277.steve@pearwood.info> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> Message-ID: <4A6C41A6.9010500@gmail.com> Steven D'Aprano wrote: > I'm a user, and personally I don't want Yet Another Integrated Package > Management Tool. What I really want is the ability to install Python > packages using the PM tool I already use, namely yum. (And I'd like a > pony.) Picking up on this point... out of curiousity, I threw a couple of package names at "apt-get -s" on my main (Ubuntu 8.04) PC to see what worked. The first two names I tried (numpy, scipy) didn't work, but the next four (python-numpy, python-scipy, python-django, python-turbogears) all had hits. Because Turbo Gears incorporates so many other projects, the simulated run at installing it provided a nice overview of a variety of other Python packages in the Ubuntu repositories as well. No doubt a search in Adept would reveal a whole lot more "python-*" packages that are just a few clicks away for me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From david.lyon at preisshare.net Sun Jul 26 17:00:06 2009 From: david.lyon at preisshare.net (David Lyon) Date: Sun, 26 Jul 2009 11:00:06 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <4A6BFAFB.1030807@v.loewis.de> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> <4A6AD375.8000805@v.loewis.de> <4A6BFAFB.1030807@v.loewis.de> Message-ID: <4a78f7148f6d97adc0c04f325f7fcf2e@preisshare.net> On Sun, 26 Jul 2009 08:43:07 +0200, "Martin v. L?wis" wrote: > Hmm. I'm -0 on providing a tool whose only purpose is to download > files from a web server. I always use a web browser for that... It does a lot more than that. Firstly it shows what packages you already have installed and lets you manage them. Namely, deinstall, (manually) upgrade, view documentation and examples. In any case, even if it were only a tool to download packages it's still consistant with modern appliance design (mobile phones, programming languages, dedicated hardware) to have an internal application where a user can pretty easily download "checked" apps to their device/system. Nobody gives people just a browser to do that... and tell them "go browse". It isn't being done like that - except in python. > Hmm. I would expect that a new user is faced with the challenge > of finding out what packages to install more so than with actually > installing them. Yes. That's exactly the point. You're 100% right. That's why the pythonpkgmgr provides a useful search capability for pypi packages. It's faster and more natural to use a native app than do it all in the browser. That's true in Perl (cpan) as in Python and I'm honestly saying that I can't find any noteable faults in pypi the way I find it. > If they read examples, they will see import > statements, and then they have to find out how to make those work. > Does your tool help with that? Yes. It will open the website or homepage to the project/package in question. And find any documentation that might be stored in the package directory. If you only have "import interbasedb" it's a complicated process for a new user to located the home page or project documentation without pythonpkgmgr. Using pythonpkgmgr, they look on their list of installed packages, find the "interbasedb" or whatever module, and can get quick access to the pypi page or the package homepage. It's much easier. Much faster.. especially if the new user doesn't know the internals of package storage, pypi and the like. > When the user is not so new anymore, I seriously doubt that they > still ask for a package management tool - except perhaps for > dependencies, and here they use easy_install. Well, that might be one use-case. It's true that many python programmers will just want to stick to their own "known" packages. However, this tool aids people with more curiosity. Because users can so easily (and safely) install and deinstall anything. It's only a click to install a package, and another to deinstall. At the moment, pythonpkgmgr doesn't handle dependencies except that which is provided internally by easy_install/pip. I'm hoping to change this as my experience and understanding grows. David From david.lyon at preisshare.net Sun Jul 26 17:15:22 2009 From: david.lyon at preisshare.net (David Lyon) Date: Sun, 26 Jul 2009 11:15:22 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <200907261805.08277.steve@pearwood.info> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> Message-ID: On Sun, 26 Jul 2009 18:05:07 +1000, Steven D'Aprano wrote: > But you shouldn't expect the > Python dev team to accept an unproven tool into the official library > before demonstrating both the need and the solution. Of course... that's why I started off by asking what the process is. I am accepting that the current status of pythonpkgmgr is "unproven" so it will need some time for things to run their course. > (Just because > users say they want something, doesn't mean they'll actually use the > tool you provide -- perhaps they don't know what they really need, and > perhaps your tool doesn't match their needs.) Conversely, the process of satisfying needs is to discuss them. And people have been ever so helpful in that regard. > I'm a user, and personally I don't want Yet Another Integrated Package > Management Tool. What I really want is the ability to install Python > packages using the PM tool I already use, namely yum. ok - but no alternative to that is available on windows. pythonpkgmgr is not so different to that. And the idea behind it is to bring consistancy in package management across the different platforms. It does have configuration that are specific to python, and will continue to do so. Like being able to specify where one wants the package installed. > Putting your software on the Cheeseshop, and making regular > announcements to the Python community (e.g. on comp.lang.python) will > be a good way to publicise the tool, and if does meet a need, people > will use it, and then, if it's good enough and popular enough and > supported, it may be blessed by inclusion in the standard library. Thanks very much. Take care. David From fuzzyman at voidspace.org.uk Sun Jul 26 18:23:59 2009 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 26 Jul 2009 17:23:59 +0100 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> Message-ID: <4A6C831F.5090106@voidspace.org.uk> David Lyon wrote: > On Sun, 26 Jul 2009 18:05:07 +1000, Steven D'Aprano > wrote: > >> But you shouldn't expect the >> Python dev team to accept an unproven tool into the official library >> before demonstrating both the need and the solution. >> > > Of course... that's why I started off by asking what the process is. > > I am accepting that the current status of pythonpkgmgr is "unproven" so > it will need some time for things to run their course. > > It would be great to have a decent visual package manager for Python. It needs to be built on top of the work that Tarek is doing with distutils (and be compatible with his Distribute fork of setuptools) and it also needs to be established. The good news is that there is time for both... Aren't there other tools that have similar goals? PyPI browser, Yolk etc It is a shame that it needs to be built on Tk if it is to stand a chance of ever being included in the standard library - but with the right expertise it is *possible* to create decent looking UIs with Tk. Michael >> (Just because >> users say they want something, doesn't mean they'll actually use the >> tool you provide -- perhaps they don't know what they really need, and >> perhaps your tool doesn't match their needs.) >> > > Conversely, the process of satisfying needs is to discuss them. And > people have been ever so helpful in that regard. > > >> I'm a user, and personally I don't want Yet Another Integrated Package >> Management Tool. What I really want is the ability to install Python >> packages using the PM tool I already use, namely yum. >> > > ok - but no alternative to that is available on windows. > > pythonpkgmgr is not so different to that. And the idea behind it is > to bring consistancy in package management across the different > platforms. > > It does have configuration that are specific to python, and will > continue to do so. Like being able to specify where one wants the > package installed. > > >> Putting your software on the Cheeseshop, and making regular >> announcements to the Python community (e.g. on comp.lang.python) will >> be a good way to publicise the tool, and if does meet a need, people >> will use it, and then, if it's good enough and popular enough and >> supported, it may be blessed by inclusion in the standard library. >> > > Thanks very much. Take care. > > David > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > -- http://www.ironpythoninaction.com/ http://www.voidspace.org.uk/blog From martin at v.loewis.de Sun Jul 26 19:31:40 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 26 Jul 2009 19:31:40 +0200 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <4a78f7148f6d97adc0c04f325f7fcf2e@preisshare.net> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> <4A6AD375.8000805@v.loewis.de> <4A6BFAFB.1030807@v.loewis.de> <4a78f7148f6d97adc0c04f325f7fcf2e@preisshare.net> Message-ID: <4A6C92FC.4010307@v.loewis.de> >> If they read examples, they will see import >> statements, and then they have to find out how to make those work. >> Does your tool help with that? > > Yes. It will open the website or homepage to the project/package > in question. How does it know the project in question? > Using pythonpkgmgr, they look on their list of installed packages, > find the "interbasedb" or whatever module, and can get quick > access to the pypi page or the package homepage. It's much easier. How do they find interbasedb in the list of installed packages if they haven't installed interbasedb yet? Regards, Martin From martin at v.loewis.de Sun Jul 26 19:33:37 2009 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 26 Jul 2009 19:33:37 +0200 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> Message-ID: <4A6C9371.6000002@v.loewis.de> >> I'm a user, and personally I don't want Yet Another Integrated Package >> Management Tool. What I really want is the ability to install Python >> packages using the PM tool I already use, namely yum. > > ok - but no alternative to that is available on windows. For removal of packages, an alternative *is* available: APR. > pythonpkgmgr is not so different to that. And the idea behind it is > to bring consistancy in package management across the different > platforms. At the cost of being inconsistent within a platform. Regards, Martin From joshua at reverberate.org Sun Jul 26 20:01:51 2009 From: joshua at reverberate.org (Joshua Haberman) Date: Sun, 26 Jul 2009 18:01:51 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Py=5FTPFLAGS=5FHEAPTYPE_too_overloaded?= Message-ID: I'm writing a C Python extension that needs to generate PyTypeObjects dynamically. Unfortunately, the Py_TPFLAGS_HEAPTYPE flag is overloaded in a way that makes it difficult to achieve this goal. The documentation for Pt_TPFLAGS_HEAPTYPE says: Py_TPFLAGS_HEAPTYPE This bit is set when the type object itself is allocated on the heap. In this case, the ob_type field of its instances is considered a reference to the type, and the type object is INCREF?ed when a new instance is created, and DECREF?ed when an instance is destroyed (this does not apply to instances of subtypes; only the type referenced by the instance?s ob_type gets INCREF?ed or DECREF?ed). This sounds like exactly what I want. I want my type object INCREF'd and DECREF'd by its instances so it doesn't leak or get deleted prematurely. If this were all that Py_TPFLAGS_HEAPTYPE did, it would work great for me. Unfortunately, Py_TPFLAGS_HEAPTYPE is also overloaded to mean "user-defined type" (as opposed to a built-in type). It controls numerous subtle behaviors such as: - whether the type's name is module.type or just type. - whether you're allowed to set __name__, __module__, or __bases__ on the type. - whether you're allowed to set __class__ on instances of this type. - whether the module name comes from the type name or the __module__ attribute. - whether it will use type->tp_doc as the docstring - whether its repr() calls it a "class" or a "type". - whether you can set attributes of the type. - whether someone is attempting the Carlo Verre hack. So I'm stuck with an unenviable choice. I think the lesser of two evils is to *not* specify Py_TPFLAGS_HEAPTYPE, because the worst that will happen is that my types will leak. This is not as bad as having someone set __class__ on one of my instances, or set attributes on my type, etc. Ideally the interpreter would have a separate flag like Py_TPFLAGS_BUILTIN that would trigger all of the above behaviors, but still make it possible to have dynamically generated built-in types get garbage collected appropriately. At the very least, the documentation I cited above should make it clear that Py_TPFLAGS_HEAPTYPE controls more than just whether the type gets INCREF'd and DECREF'd. Based on the list of behaviors I discovered above, it is almost certainly not correct for a C exension type to be declared with Py_TPFLAGS_HEAPTYPE. Josh From greg.ewing at canterbury.ac.nz Mon Jul 27 03:05:55 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 27 Jul 2009 13:05:55 +1200 Subject: [Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded In-Reply-To: References: Message-ID: <4A6CFD73.8050201@canterbury.ac.nz> Joshua Haberman wrote: > This is not as bad as having someone > set __class__ on one of my instances, or set attributes on my type, etc. Is there any real need to prevent someone from doing those things? Note that even when you are allowed to change the __class__ of an instance, you're still prevented from changing it to something that has a different C layout, so you can't crash the intepreter that way. Similarly, built-in methods check that they're given an object of appropriate type at the C level. My suggestion is to just let it be a full heap type and accept whatever consequences follow. -- Greg From stephen at xemacs.org Mon Jul 27 04:18:25 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 27 Jul 2009 11:18:25 +0900 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <4A6C9371.6000002@v.loewis.de> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> Message-ID: <874osy6f7y.fsf@uwakimon.sk.tsukuba.ac.jp> "Martin v. L?wis" writes: > > pythonpkgmgr is not so different to that. And the idea behind it is > > to bring consistancy in package management across the different > > platforms. > > At the cost of being inconsistent within a platform. Indeed, and that seems to be one of the really big sticking points in all efforts to improve Python's package management: the conflict between user goals ("something that works for me without any effort[1] on my part") and sysadmin goals ("something that works and plays nicely with the rest of the system"). pythonpkgmgr seems entirely oblivious to the latter issue, and not particularly compatible with the way package management works in *nix OSS distros (including Linux distros, but also *BSD, MacPorts, and Fink). I think that will probably kill uptake on those platforms -- could be wrong, but caution is suggested. BTW, I think Tarek's efforts strike a good balance between these goals, and the whole PEP 376 process has demonstrated how difficult it is to serve these two masters at the same time. Footnotes: [1] I'm a professional economist: in my lexicon, effort minimization is a positive goal, not laziness. Zero is just an extreme case. From tismer at stackless.com Mon Jul 27 09:00:19 2009 From: tismer at stackless.com (Christian Tismer) Date: Mon, 27 Jul 2009 00:00:19 -0700 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com> References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de> <50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com> Message-ID: <4A6D5083.3050709@stackless.com> On 7/25/09 7:11 AM, Neil Hodgson wrote: > Martin v. L?wis: > >> I propose to add another (regular) double into the union. > > Adding a regular double as a second dummy gives the same sizes and > alignments with Mingw or MSVC as the original definition with MSVC: Great (checked that, too) This makes pretty much sense, also given that this waste happens under Windows, already. > In regard to alignment penalties, a simple copy loop for doubles > runs about 20% slower when misaligned on an my AMD processor. Other > x86 processors can be much worse. As much as 2 to 3.25 times according > to I am still unhappy with this waste of memory, just because the GC header has to be rounded up, regardlwss wether that is needed or not. We should keep Martin's hint in mind, that Python 4 could place the gc header at the end of structures, instead. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From lists at cheimes.de Mon Jul 27 09:48:43 2009 From: lists at cheimes.de (Christian Heimes) Date: Mon, 27 Jul 2009 09:48:43 +0200 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A6D5083.3050709@stackless.com> References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de> <50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com> <4A6D5083.3050709@stackless.com> Message-ID: <4A6D5BDB.7010809@cheimes.de> Christian Tismer wrote: > We should keep Martin's hint in mind, that Python 4 could place > the gc header at the end of structures, instead. Wow, 3.1 just came out and we already have the first PEP for Python 4k? :) Christian From joshua at reverberate.org Mon Jul 27 10:14:21 2009 From: joshua at reverberate.org (Joshua Haberman) Date: Mon, 27 Jul 2009 08:14:21 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Py=5FTPFLAGS=5FHEAPTYPE_too_overloaded?= References: <4A6CFD73.8050201@canterbury.ac.nz> Message-ID: Greg Ewing canterbury.ac.nz> writes: > Joshua Haberman wrote: > > This is not as bad as having someone > > set __class__ on one of my instances, or set attributes on my type, etc. > > Is there any real need to prevent someone from doing > those things? My ultimate goal is to make my types as much like regular built-in types as possible. Python as a language has chosen to "lock down" built-in objects, even going so far as to specifically check for the "Carlo Verre hack." I defer to those decisions to answer the question "is there any real need to prevent someone from doing these things?" If it's important for the built-in types, why should it be less important for mine? I don't want my type to be a second-class citizen just because I happen to be dynamically allocating it. If I were writing this extension for a language like Ruby, for which it is convention that built-in classes are "open," then I wouldn't mind allowing these things. I'm just trying to make my extension as idiomatic and "native" as possible. Josh From david.lyon at preisshare.net Mon Jul 27 11:42:31 2009 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 27 Jul 2009 05:42:31 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <874osy6f7y.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <874osy6f7y.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <2de05ec17eb7d0731bad0b6dd99d6c3b@preisshare.net> On Mon, 27 Jul 2009 11:18:25 +0900, "Stephen J. Turnbull" wrote: > [1] on > my part") and sysadmin goals ("something that works and plays nicely > with the rest of the system"). > > pythonpkgmgr seems entirely oblivious to the latter issue, and not > particularly compatible with the way package management works in *nix > OSS distros (including Linux distros, but also *BSD, MacPorts, and > Fink). Well I'm a sysadmin on linux in my day job. I'm entirely wondering what you mean by that ? I'm oblivious how? What can a developer mode project, like pythonpkgmgr possibly do except to make it easier to manage local package space? A developer shouldn't need to do sudo every time they want to put or try some python package from pypi. imho that's wrong and I'm helping to reduce that need. If the sysadmin puts a package in (/usr/lib/pythonX.Y/site-packages) via the O/S package manager (ie synaptics/yum/apt) then that should stick over what a developer is doing. But the developer should be able to stick their own packages in their own local space. Correct me if my assertion is wrong. David From david.lyon at preisshare.net Mon Jul 27 12:20:30 2009 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 27 Jul 2009 06:20:30 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <4A6C831F.5090106@voidspace.org.uk> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> <4A6C831F.5090106@voidspace.org.uk> Message-ID: On Sun, 26 Jul 2009 17:23:59 +0100, Michael Foord wrote: > It would be great to have a decent visual package manager for Python. Hopefully one day we'll have one - haha > It needs to be built on top of the work that Tarek is doing with > distutils (and be compatible with his Distribute fork of setuptools) and > it also needs to be established. I'm planning to support it - like I do for setuptools and pip > The good news is that there is time for both... Aren't there other tools > that have similar goals? PyPI browser, Yolk etc The fact is that there's way more glamorous things to work on than a package manager. Nobody has written a package manager for python imho simply because there's so many more fun things to do with python than that... > It is a shame that it needs to be built on Tk if it is to stand a chance > of ever being included in the standard library - but with the right > expertise it is *possible* to create decent looking UIs with Tk. I'm working on a TK version.. yes.. a TK version is better than nothing and the differences between a TK and a WX version is really only pretty minor. To me, wx is ancient... and tk is simply prehistoric. mr.new-to-python developer wouldn't really cares too much either I would suspect. All they want is to be up and running quickly. My only point is that Windows ain't no embedded system. It's not short on memory or disk space. If a package manager is 5 megabytes extra say, with it's libraries.. what's the extra download time on that ? compared to three days+ stuffing around trying to find out how to install packages for a new user. If the source needs to go in too.. that's a few kilobytes. That's like 57 extra bytes to an old-timer... I know.. it's such a sacrifice.. and we're all working on 300 baud... :-) David From cournape at gmail.com Mon Jul 27 12:29:14 2009 From: cournape at gmail.com (David Cournapeau) Date: Mon, 27 Jul 2009 19:29:14 +0900 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> <4A6C831F.5090106@voidspace.org.uk> Message-ID: <5b8d13220907270329s56664cbbhc8afe3ed73e371df@mail.gmail.com> On Mon, Jul 27, 2009 at 7:20 PM, David Lyon wrote: > My only point is that Windows ain't no embedded system. It's not > short on memory or disk space. If a package manager is 5 megabytes > extra say, with it's libraries.. what's the extra download time on > that ? compared to three days+ stuffing around trying to find out > how to install packages for a new user. The problem is not so much the size by itself that more code means more maintenance burden for python developers. Including new code means it has to work everywhere where python works currently, and that other people can understand/support the related code. Adding code to a project is far from free from python maintainers POV. cheers, David From david.lyon at preisshare.net Mon Jul 27 12:59:18 2009 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 27 Jul 2009 06:59:18 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <5b8d13220907270329s56664cbbhc8afe3ed73e371df@mail.gmail.com> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> <4A6C831F.5090106@voidspace.org.uk> <5b8d13220907270329s56664cbbhc8afe3ed73e371df@mail.gmail.com> Message-ID: On Mon, 27 Jul 2009 19:29:14 +0900, David Cournapeau wrote: >> My only point is that Windows ain't no embedded system. It's not >> short on memory or disk space. If a package manager is 5 megabytes >> extra say, with it's libraries.. what's the extra download time on >> that ? compared to three days+ stuffing around trying to find out >> how to install packages for a new user. > > The problem is not so much the size by itself that more code means > more maintenance burden for python developers. Including new code > means it has to work everywhere where python works currently, and that > other people can understand/support the related code. Adding code to a > project is far from free from python maintainers POV. Well I concede that you might have a point. And I know there have been some issues in the past with package management in python and people being able to understand other peoples code. In this specific case, I don't think you'll find the same problems. pythonpkgmgr is not an entirely self inclusive project. It leverages on setuptools and pip. I want to extend it to enstall. If there is any preference towards setuptools - it's only by test-time availability. In windows, it's built with py2exe. There are some supporting .dll files that need to go with it from wxpython. They're both really mainstream python(windows) toolkits. Everything in pythonpkgmgr is transperant - and was designed to be. Otherwise there's no point. If any developers want to join the project with 'commit' privaleges on the source, I'll make sure it will happen. If not.. I'm very happy just continuing on with my own merry way... :-) David From stephen at xemacs.org Mon Jul 27 13:12:54 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 27 Jul 2009 20:12:54 +0900 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <2de05ec17eb7d0731bad0b6dd99d6c3b@preisshare.net> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <874osy6f7y.fsf@uwakimon.sk.tsukuba.ac.jp> <2de05ec17eb7d0731bad0b6dd99d6c3b@preisshare.net> Message-ID: <87r5w2bcqx.fsf@uwakimon.sk.tsukuba.ac.jp> David Lyon writes: > On Mon, 27 Jul 2009 11:18:25 +0900, "Stephen J. Turnbull" > wrote: > > [1] on > > my part") and sysadmin goals ("something that works and plays nicely > > with the rest of the system"). > > > > pythonpkgmgr seems entirely oblivious to the latter issue, and not > > particularly compatible with the way package management works in *nix > > OSS distros (including Linux distros, but also *BSD, MacPorts, and > > Fink). > > Well I'm a sysadmin on linux in my day job. > > I'm entirely wondering what you mean by that ? I'm oblivious how? Not you; pythonpkgmgr. You've said nothing about how pythonpkgmgr is supposed to deal with multiple installed versions of Python or individual modules, or how it helps if you need a custom version of a module, etc., or how it can help the developer's "local" modules cooperate/not interfere with system versions, etc. > What can a developer mode project, like pythonpkgmgr possibly do > except to make it easier to manage local package space? Pave the road to DLL Hell with good intentions, of course. > A developer shouldn't need to do sudo every time they want > to put or try some python package from pypi. I don't remember *ever* sudo-ing to try a package from pypi. > If the sysadmin puts a package in (/usr/lib/pythonX.Y/site-packages) > via the O/S package manager (ie synaptics/yum/apt) then that > should stick over what a developer is doing. But I disagree about that, too. The last thing I want is to have my in-development environment hosed by system upgrades or installations. Eventually I want to deal with that for deployment, but in the early stages one can make faster progress on demos or initial target platforms by sticking to particular versions of prerequisite libraries. > But the developer should be able to stick their own packages > in their own local space. > > Correct me if my assertion is wrong. "Should", yes. But your assertion is incomplete IME: the developer *is* able to install packages in their own local space *without* sudo-ing. I just don't understand what problem you're claiming to solve. The problem of managing local package space is in some sense solved by virtualenv, but you haven't mentioned that at all. Is pythonpkgmgr an alternative? A complement? to virtualenv or zc.buildout? From eric.pruitt at gmail.com Mon Jul 27 18:01:30 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Mon, 27 Jul 2009 11:01:30 -0500 Subject: [Python-Dev] Implementing File Modes Message-ID: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> Hello, Since there was a bit of confusion last time, I'll start by saying I am working on the subprocess.Popen module for Google Summer of Code. One of the features I am implementing is a class so that a running process can stand in in place of a file. For examples, instead of open( "filelist", mode = 'r') one would call ProcessIOWrapper( "ls -l", mode = 'r'). I am trying to decide if I should fully implement the mode argument. Right now, it essentially ignores everything but a 'U' indicated universal newlines in the mode argument. Should I leave that as is or make it so that things like "r+", "w", "a" are handled the way they would be for an actual file? Eric -------------- next part -------------- An HTML attachment was scrubbed... URL: From tismer at stackless.com Mon Jul 27 18:11:45 2009 From: tismer at stackless.com (Christian Tismer) Date: Mon, 27 Jul 2009 09:11:45 -0700 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A6D5BDB.7010809@cheimes.de> References: <4A675C10.6060200@stackless.com> <5b8d13220907221717h4ba9ed00hbbdf1265d383ad25@mail.gmail.com> <4A67AEC3.6070000@stackless.com> <4A68D2BB.8060607@stackless.com> <4A6AE6F0.2060303@v.loewis.de> <50862ebd0907250711s1363d448n2b3a9dc940fa03f3@mail.gmail.com> <4A6D5083.3050709@stackless.com> <4A6D5BDB.7010809@cheimes.de> Message-ID: <4A6DD1C1.1020800@stackless.com> On 7/27/09 12:48 AM, Christian Heimes wrote: > Christian Tismer wrote: >> We should keep Martin's hint in mind, that Python 4 could place >> the gc header at the end of structures, instead. > > Wow, 3.1 just came out and we already have the first PEP for Python 4k? :) > > Christian Maybe it's even possible right now, with a couple of tweaks that allow to explicitly say where the header is, for a specific extension type. Might cost a few cycles, tho. Christian -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Mon Jul 27 18:13:51 2009 From: tismer at stackless.com (Christian Tismer) Date: Mon, 27 Jul 2009 09:13:51 -0700 Subject: [Python-Dev] mingw32 and gc-header weirdness In-Reply-To: <4A69A606.1040506@roumenpetrov.info> References: <4A675C10.6060200@stackless.com> <4A69A606.1040506@roumenpetrov.info> Message-ID: <4A6DD23F.1030006@stackless.com> On 7/24/09 5:16 AM, Roumen Petrov wrote: > Christian Tismer wrote: ... > Did the crash disappear is you add "__attribute__((aligned(8)))" after > variable dummy ? Did not try. But the proposed addition of a double does it, see the dev list. cheers - chris -- Christian Tismer :^) tismerysoft GmbH : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9A : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 802 86 56 mobile +49 173 24 18 776 fax +49 30 80 90 57 05 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From p.f.moore at gmail.com Mon Jul 27 18:46:15 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 27 Jul 2009 17:46:15 +0100 Subject: [Python-Dev] Implementing File Modes In-Reply-To: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> Message-ID: <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> 2009/7/27 Eric Pruitt : > Hello, > > Since there was a bit of confusion last time, I'll start by saying I am > working on the subprocess.Popen module for Google Summer of Code. One of the > features I am implementing is a class so that a running process can stand in > in place of a file. For examples, instead of open( "filelist", mode = 'r') > one would call ProcessIOWrapper( "ls -l", mode = 'r'). I am trying to decide > if I should fully implement the mode argument. Right now, it essentially > ignores everything but a 'U' indicated universal newlines in the mode > argument. Should I leave that as is or make it so that things like "r+", > "w", "a" are handled the way they would be for an actual file? I would expect "r" to produce a pipe that reads from stdout of the subprocess, and "w" to produce a pipe that writes to stdin of the subprocess. "a" would be the same as "w", and arguably "r+" would be a bidirectional pipe - read from the subprocess stdout and write to its stdin. I'd be OK with "r+" not being implemented (if it's too hard to avoid deadlocks) but "r" and "w" should be present. Paul. From python at mrabarnett.plus.com Mon Jul 27 19:21:30 2009 From: python at mrabarnett.plus.com (MRAB) Date: Mon, 27 Jul 2009 18:21:30 +0100 Subject: [Python-Dev] Implementing File Modes In-Reply-To: <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> Message-ID: <4A6DE21A.4040803@mrabarnett.plus.com> Paul Moore wrote: > 2009/7/27 Eric Pruitt : >> Hello, >> >> Since there was a bit of confusion last time, I'll start by saying I am >> working on the subprocess.Popen module for Google Summer of Code. One of the >> features I am implementing is a class so that a running process can stand in >> in place of a file. For examples, instead of open( "filelist", mode = 'r') >> one would call ProcessIOWrapper( "ls -l", mode = 'r'). I am trying to decide >> if I should fully implement the mode argument. Right now, it essentially >> ignores everything but a 'U' indicated universal newlines in the mode >> argument. Should I leave that as is or make it so that things like "r+", >> "w", "a" are handled the way they would be for an actual file? > > I would expect "r" to produce a pipe that reads from stdout of the > subprocess, and "w" to produce a pipe that writes to stdin of the > subprocess. "a" would be the same as "w", and arguably "r+" would be a > bidirectional pipe - read from the subprocess stdout and write to its > stdin. > > I'd be OK with "r+" not being implemented (if it's too hard to avoid > deadlocks) but "r" and "w" should be present. > What about stderr? You could add "e" if you want to read from it. From eric.pruitt at gmail.com Mon Jul 27 20:06:45 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Mon, 27 Jul 2009 12:06:45 -0600 Subject: [Python-Dev] Implementing File Modes In-Reply-To: <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> Message-ID: <171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com> I am implementing the file wrapper using changes to subprocess.Popen that also make it asynchronous and non-blocking so implementing "r+" should be trivial to do. How about handling stderr? I have the following ideas: leave out support for reading from stderr, make it so that there is an optional additional argument like "outputstderr = False", create another function that toggles / sets whether stderr or stdout is returned or mix the two outputs. Thanks for the input, Eric On Mon, Jul 27, 2009 at 10:46, Paul Moore wrote: > 2009/7/27 Eric Pruitt : > > Hello, > > > > Since there was a bit of confusion last time, I'll start by saying I am > > working on the subprocess.Popen module for Google Summer of Code. One of > the > > features I am implementing is a class so that a running process can stand > in > > in place of a file. For examples, instead of open( "filelist", mode = > 'r') > > one would call ProcessIOWrapper( "ls -l", mode = 'r'). I am trying to > decide > > if I should fully implement the mode argument. Right now, it essentially > > ignores everything but a 'U' indicated universal newlines in the mode > > argument. Should I leave that as is or make it so that things like "r+", > > "w", "a" are handled the way they would be for an actual file? > > I would expect "r" to produce a pipe that reads from stdout of the > subprocess, and "w" to produce a pipe that writes to stdin of the > subprocess. "a" would be the same as "w", and arguably "r+" would be a > bidirectional pipe - read from the subprocess stdout and write to its > stdin. > > I'd be OK with "r+" not being implemented (if it's too hard to avoid > deadlocks) but "r" and "w" should be present. > > Paul. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Jul 27 21:04:08 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 27 Jul 2009 20:04:08 +0100 Subject: [Python-Dev] Implementing File Modes In-Reply-To: <171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com> References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> <171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com> Message-ID: <79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com> 2009/7/27 Eric Pruitt : > I am implementing the file wrapper using changes to subprocess.Popen that > also make it asynchronous and non-blocking so implementing "r+" should be > trivial to do. How about handling stderr? I have the following ideas: leave > out support for reading from stderr, make it so that there is an optional > additional argument like "outputstderr = False", create another function > that toggles / sets whether stderr or stdout is returned or mix the two > outputs. I like MRAB's idea of using a (non-standard) "e" flag to include stderr. So "r" reads from stdout, "re" reads from stdout+stderr. Anything more complicated probably should just use "raw" Popen objects. Don't overcomplicate the interface. Paul. From steve at pearwood.info Tue Jul 28 01:26:23 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Jul 2009 09:26:23 +1000 Subject: [Python-Dev] Implementing File Modes In-Reply-To: <4A6DE21A.4040803@mrabarnett.plus.com> References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> <4A6DE21A.4040803@mrabarnett.plus.com> Message-ID: <200907280926.24081.steve@pearwood.info> On Tue, 28 Jul 2009 03:21:30 am MRAB wrote: > What about stderr? You could add "e" if you want to read from it. "Read from stderr" is just a read. "Write to stderr" is just a write. The difference between reading stdout and stderr is not that you have different modes, but that you are reading from different files. -- Steven D'Aprano From python at mrabarnett.plus.com Tue Jul 28 01:41:53 2009 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 28 Jul 2009 00:41:53 +0100 Subject: [Python-Dev] Implementing File Modes In-Reply-To: <200907280926.24081.steve@pearwood.info> References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> <4A6DE21A.4040803@mrabarnett.plus.com> <200907280926.24081.steve@pearwood.info> Message-ID: <4A6E3B41.60701@mrabarnett.plus.com> Steven D'Aprano wrote: > On Tue, 28 Jul 2009 03:21:30 am MRAB wrote: > >> What about stderr? You could add "e" if you want to read from it. > > "Read from stderr" is just a read. "Write to stderr" is just a write. > The difference between reading stdout and stderr is not that you have > different modes, but that you are reading from different files. > By the same argument, aren't stdin and stdout also different files? From greg.ewing at canterbury.ac.nz Tue Jul 28 01:52:43 2009 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 28 Jul 2009 11:52:43 +1200 Subject: [Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded In-Reply-To: References: <4A6CFD73.8050201@canterbury.ac.nz> Message-ID: <4A6E3DCB.7030601@canterbury.ac.nz> Joshua Haberman wrote: > Python as a language has chosen to "lock down" built-in > objects... If it's > important for the built-in types, why should it be less important for > mine? I'm not really sure why so much trouble is taken to lock down builtin types -- it seems to go against Python's general consenting-adults philosophy. I suppose it's felt that you should be able to rely on builtin types not changing their behaviour. This is probably more important for the core types than those in extension modules. Many of the standard library classes are written in Python, so this protection doesn't extend to them. > I don't want my type to be a second-class citizen just because I happen > to be dynamically allocating it. I don't think anyone will regard your types as second-class because they allow you to do *more* with them. The only real concern would be if it were somehow possible to crash the interpreter by modifying the type dict. I don't see how that could happen -- but maybe someone else on python-dev knows more about this? -- Greg From steve at pearwood.info Tue Jul 28 02:01:25 2009 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Jul 2009 10:01:25 +1000 Subject: [Python-Dev] Implementing File Modes In-Reply-To: <171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com> References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> <171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com> Message-ID: <200907281001.26339.steve@pearwood.info> On Tue, 28 Jul 2009 04:06:45 am Eric Pruitt wrote: > I am implementing the file wrapper using changes to subprocess.Popen > that also make it asynchronous and non-blocking so implementing "r+" > should be trivial to do. How about handling stderr? I have the > following ideas: leave out support for reading from stderr, make it > so that there is an optional additional argument like "outputstderr = > False", create another function that toggles / sets whether stderr or > stdout is returned or mix the two outputs. Leaving it out is always an option. As I see it, fundamentally you can either read from stdout and sterr as two different streams, or you can interleave (mix) them. To me, that suggests three functions: ProcessIOWrapper() # read from stdout (or write to stdin etc.) ProcessIOWrapperStdErr() # read/write from stderr ProcessIOWrapper2() # read from mixed stdout and stderr I don't like a function to toggle between one and the other: that smacks of relying on a global setting in a bad way. I suppose you could add an optional argument to ProcessIOWrapper() to select between stdout, stderr, or both together. -- Steven D'Aprano From glyph at twistedmatrix.com Tue Jul 28 02:32:51 2009 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 27 Jul 2009 20:32:51 -0400 Subject: [Python-Dev] Implementing File Modes In-Reply-To: <79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com> References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> <171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com> <79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com> Message-ID: On Mon, Jul 27, 2009 at 3:04 PM, Paul Moore wrote: > I like MRAB's idea of using a (non-standard) "e" flag to include > stderr. So "r" reads from stdout, "re" reads from stdout+stderr. > > Anything more complicated probably should just use "raw" Popen > objects. Don't overcomplicate the interface. > In my opinion, mangling stderr and stdout together is already an overcomplication. It shouldn't be implemented. It *seems* like a good idea, until you realize that subtle changes to your OS, environment, or buffering behavior may result in arbitrary, unparseable output. For example, let's say you've got a program whose output is a list of lines, each one containing a number. Sometimes it tries to import gtk, and fails to open its display. That's fine, and you can still deal with it, as long as the interleaved output looks like this: 100 200 Gtk-WARNING **: cannot open display: 300 400 but of course the output *might* (although unlikely with such small chunks of output) end up looking like this, instead: 100 2Gtk-WAR0NING0 **: can30not 0open display: 400 this is the sort of thing which is much more likely to happen once you start dealing with large volumes of data, where there are more page-boundaries for your buffers to get confused around, and you are playing with buffering options to improve performance. In other words, it's something that fails only at scale or under load, and is therefore extremely difficult to debug. This option *might* be okay if it were allowed only on subprocesses opened in a *text* mode, and if the buffering logic involved forced stderr and stdout to be line-delimited, and interleave only lines, rather than arbitrary chunks of bytes. Of course then if you use this flag with a program that outputs binary data with no newlines it will buffer forever and crash your program with a MemoryError, but at least that's easy to debug when it happens. -------------- next part -------------- An HTML attachment was scrubbed... URL: From david.lyon at preisshare.net Tue Jul 28 02:32:27 2009 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 27 Jul 2009 20:32:27 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <87r5w2bcqx.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <874osy6f7y.fsf@uwakimon.sk.tsukuba.ac.jp> <2de05ec17eb7d0731bad0b6dd99d6c3b@preisshare.net> <87r5w2bcqx.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <68eb1540f5da00887220865daef9daee@preisshare.net> On Mon, 27 Jul 2009 20:12:54 +0900, "Stephen J. Turnbull" wrote: > Not you; pythonpkgmgr. You've said nothing about how pythonpkgmgr is > supposed to deal with multiple installed versions of Python Under windows it can deal with multiple versions of python. You just go to options and select which version of python you wish to manage. It's as simple as that. > or how it helps if you need a custom version of a > module, etc., or how it can help the developer's "local" modules > cooperate/not interfere with system versions, etc. It manages local developer modules for python 2.6+. As for checks, it doesn't have so many at present. But it makes it a lot easier to see what you have installed. So if something needs to be removed and reinstalled - it can do that. > Pave the road to DLL Hell with good intentions, of course. Well - there's plenty of that > > > A developer shouldn't need to do sudo every time they want > > to put or try some python package from pypi. > > I don't remember *ever* sudo-ing to try a package from pypi. ok. But if you were using an O/S level package manager you would have to enter a su password to start the tool. Because that is needed to load packages into /usr/lib/pythonX.y > I just don't understand what problem you're claiming to solve. Making it simpler for new users to find and install packages from pypi. And manage the existing packages that are installed on their machine. > The > problem of managing local package space is in some sense solved by > virtualenv, but you haven't mentioned that at all. I never heard of it till a few months ago. To my knowledge it doesn't offer a gui. I'm told it's good for certain things and I believe it. My problem was installing packages on windows boxes, not in the development. > Is pythonpkgmgr an > alternative? A complement? to virtualenv or zc.buildout? pythonpkgmgr is aimed at featherweight users. It could be highly complementory to both virtualenv or zc.buildout at some stage if it were customised to work with those. For example, it could work with virtualenv by having a a drop down list of the different 'environments'. I don't know how to do it right now - but I'm hoping I can figure it out sometime. With zc.buildout, there's no reason why it couldn't generate a package list in that format, or do an import from a manifest. pythonpkgmgr already can generate a list of local packages as a manifest - but not in the zc.buildout format. I'm sure in the future - I'll figure that one out as well. David From david.lyon at preisshare.net Tue Jul 28 02:44:58 2009 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 27 Jul 2009 20:44:58 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <4A6C92FC.4010307@v.loewis.de> References: <87bpnb7ufv.fsf@benfinney.id.au> <51673.203.171.195.227.1248333757.squirrel@syd-srv02.ezyreg.com> <871vo561d7.fsf@benfinney.id.au> <245a19a7ce30a6656c9af4ff3a4a96a6@preisshare.net> <4A6AD375.8000805@v.loewis.de> <4A6BFAFB.1030807@v.loewis.de> <4a78f7148f6d97adc0c04f325f7fcf2e@preisshare.net> <4A6C92FC.4010307@v.loewis.de> Message-ID: On Sun, 26 Jul 2009 19:31:40 +0200, "Martin v. L?wis" wrote: >>> If they read examples, they will see import >>> statements, and then they have to find out how to make those work. >>> Does your tool help with that? >> >> Yes. It will open the website or homepage to the project/package >> in question. > > How does it know the project in question? Because it is either in the pypi search results or list of installed packages. > How do they find interbasedb in the list of installed packages if > they haven't installed interbasedb yet? They'd already installed it. But perphaps through a .exe installer. David From david.lyon at preisshare.net Tue Jul 28 02:50:57 2009 From: david.lyon at preisshare.net (David Lyon) Date: Mon, 27 Jul 2009 20:50:57 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <4A6C9371.6000002@v.loewis.de> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> Message-ID: <1b9d4c7557e19faed93300f157d07b79@preisshare.net> On Sun, 26 Jul 2009 19:33:37 +0200, "Martin v. L?wis" wrote: >> pythonpkgmgr is not so different to that. And the idea behind it is >> to bring consistancy in package management across the different >> platforms. > > At the cost of being inconsistent within a platform. It has the most generic of user interfaces. On Windows, a command line interface is the most out-of-place thing anybody could ask for. 19/20 kids these days wouldn't even know what a command line interface is until you show them. The first thing that they would look for is a GUI tool in the Python X.Y programs menu... So I respectfully say that there couldn't be anything less true than the assertion that a GUI package manager is inconsistent within a modern GUI desktop environment. David From stephen at xemacs.org Tue Jul 28 04:26:44 2009 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 28 Jul 2009 11:26:44 +0900 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <68eb1540f5da00887220865daef9daee@preisshare.net> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <874osy6f7y.fsf@uwakimon.sk.tsukuba.ac.jp> <2de05ec17eb7d0731bad0b6dd99d6c3b@preisshare.net> <87r5w2bcqx.fsf@uwakimon.sk.tsukuba.ac.jp> <68eb1540f5da00887220865daef9daee@preisshare.net> Message-ID: <87iqhdbl0b.fsf@uwakimon.sk.tsukuba.ac.jp> David Lyon writes: > It manages local developer modules for python 2.6+. > pythonpkgmgr is aimed at featherweight users. You were talking about "developers", but now they're "featherweight users"? I'm sorry, but the more you post, the less I like the idea of including it with Python. Please do develop it, do post it to pypi, and do discuss it on python-list. I'm sure there is an audience for it, not only for Windows users but for the more GUI-oriented POSIX- based platforms as well. But I really don't think that python-dev is the appropriate place any more. From aahz at pythoncraft.com Tue Jul 28 04:48:07 2009 From: aahz at pythoncraft.com (Aahz) Date: Mon, 27 Jul 2009 19:48:07 -0700 Subject: [Python-Dev] Going nomail Message-ID: <20090728024807.GA28303@panix.com> I just started a new job today, and I've got a bunch of other stuff going on in my life, so I'm setting python-dev and python-ideas to nomail for a while. Please feel free to ping me directly if you want. I'll be back. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Many customs in this life persist because they ease friction and promote productivity as a result of universal agreement, and whether they are precisely the optimal choices is much less important." --Henry Spencer From martin at v.loewis.de Tue Jul 28 07:12:25 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 28 Jul 2009 07:12:25 +0200 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <1b9d4c7557e19faed93300f157d07b79@preisshare.net> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <1b9d4c7557e19faed93300f157d07b79@preisshare.net> Message-ID: <4A6E88B9.5010504@v.loewis.de> >>> pythonpkgmgr is not so different to that. And the idea behind it is >>> to bring consistancy in package management across the different >>> platforms. >> At the cost of being inconsistent within a platform. > > It has the most generic of user interfaces. > [...] > > So I respectfully say that there couldn't be anything less true than > the assertion that a GUI package manager is inconsistent within a > modern GUI desktop environment. That's not my concern - I'm not worried about how it looks, but what it does. If it "manages packages", I would expect it to manage them the same way as the native package manager does, but alas, it doesn't (IIUC). So there are now two incompatible ways to install a package: either with the native manager, or with pythonpkgmgr. If you install them one way, and try to remove them the other way, you lose. Regards, Martin From david.lyon at preisshare.net Tue Jul 28 07:20:59 2009 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 28 Jul 2009 01:20:59 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <4A6E88B9.5010504@v.loewis.de> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <1b9d4c7557e19faed93300f157d07b79@preisshare.net> <4A6E88B9.5010504@v.loewis.de> Message-ID: <14cf53f0cf73387c177f045fd21151b7@preisshare.net> On Tue, 28 Jul 2009 07:12:25 +0200, "Martin v. L?wis" wrote: > So there are now two incompatible ways to install a package: > either with the native manager, or with pythonpkgmgr. If you install > them one way, and try to remove them the other way, you lose. pythonpkgmgr is only a thin wrapper for easy_install/pip. If there is a problem, then it is already a pre-existing problem that is not of my doing. It's certainly true that if a python package is installed with a windows installer or as a .deb/.rpm, then that package should be uninstalled using the appropriate and corresponding tool. The same is true if the package was installed with either pip or easy_install. All I can do is put it in my tracker as a bug and deal with it later when I have time. Good point David From martin at v.loewis.de Tue Jul 28 07:55:11 2009 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 28 Jul 2009 07:55:11 +0200 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <14cf53f0cf73387c177f045fd21151b7@preisshare.net> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <1b9d4c7557e19faed93300f157d07b79@preisshare.net> <4A6E88B9.5010504@v.loewis.de> <14cf53f0cf73387c177f045fd21151b7@preisshare.net> Message-ID: <4A6E92BF.7080901@v.loewis.de> >> So there are now two incompatible ways to install a package: >> either with the native manager, or with pythonpkgmgr. If you install >> them one way, and try to remove them the other way, you lose. > > pythonpkgmgr is only a thin wrapper for easy_install/pip. > > If there is a problem, then it is already a pre-existing problem > that is not of my doing. Yes, eggs have the same problem. That's one of the reasons they don't get integrated into Python. Regards, Martin From david.lyon at preisshare.net Tue Jul 28 08:57:22 2009 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 28 Jul 2009 02:57:22 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <4A6E92BF.7080901@v.loewis.de> References: <4A6AD375.8000805@v.loewis.de> <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <1b9d4c7557e19faed93300f157d07b79@preisshare.net> <4A6E88B9.5010504@v.loewis.de> <14cf53f0cf73387c177f045fd21151b7@preisshare.net> <4A6E92BF.7080901@v.loewis.de> Message-ID: <3fcf944249977ccd966d62a73ac0db5d@preisshare.net> On Tue, 28 Jul 2009 07:55:11 +0200, "Martin v. L?wis" wrote: > Yes, eggs have the same problem. That's one of the reasons they > don't get integrated into Python. Yes but egg_info is included in python... and the egg is not.... Hence, what goes in and what doesn't isn't always that rational. I'm just accepting that for the moment. David From greg at krypto.org Tue Jul 28 09:53:48 2009 From: greg at krypto.org (Gregory P. Smith) Date: Tue, 28 Jul 2009 00:53:48 -0700 Subject: [Python-Dev] Implementing File Modes In-Reply-To: References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> <171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com> <79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com> Message-ID: <52dc1c820907280053v6ab7cb4fo31b7d2e49c53021@mail.gmail.com> On Mon, Jul 27, 2009 at 5:32 PM, Glyph Lefkowitz wrote: > On Mon, Jul 27, 2009 at 3:04 PM, Paul Moore wrote: >> >> I like MRAB's idea of using a (non-standard) "e" flag to include >> stderr. So "r" reads from stdout, "re" reads from stdout+stderr. >> >> Anything more complicated probably should just use "raw" Popen >> objects. Don't overcomplicate the interface. > > In my opinion, mangling stderr and stdout together is already an > overcomplication.? It shouldn't be implemented. > > It seems like a good idea, until you realize that subtle changes to your OS, > environment, or buffering behavior may result in arbitrary, unparseable > output. Agreed. Leave stderr support out of this. People who need stderr should use the full subprocess.Popen interface. Quick hack unixy types will just run their process using a shell (which already seems to be the default based on the "ls -l" example) and add 2>&1. This functionality is basically the equivalent of adding the | symbol on either or both ends of a filename given to open() in perl. (but without being so gross). I do wonder why you're trying to make it behave exactly like open() including the mode= syntax. Why not just define several names based on the behavior? ProcessReadWrapper() ProcessWriteWrapper() ProcessReadWriteWrapper() -gps > > For example, let's say you've got a program whose output is a list of lines, > each one containing a number.? Sometimes it tries to import gtk, and fails > to open its display. > > That's fine, and you can still deal with it, as long as the interleaved > output looks like this: > > 100 > 200 > Gtk-WARNING **: cannot open display: > 300 > 400 > > but of course the output might (although unlikely with such small chunks of > output) end up looking like this, instead: > > 100 > 2Gtk-WAR0NING0 **: > can30not 0open display: > > 400 > > this is the sort of thing which is much more likely to happen once you start > dealing with large volumes of data, where there are more page-boundaries for > your buffers to get confused around, and you are playing with buffering > options to improve performance.? In other words, it's something that fails > only at scale or under load, and is therefore extremely difficult to debug. > > This option might be okay if it were allowed only on subprocesses opened in > a text mode, and if the buffering logic involved forced stderr and stdout to > be line-delimited, and interleave only lines, rather than arbitrary chunks > of bytes.? Of course then if you use this flag with a program that outputs > binary data with no newlines it will buffer forever and crash your program > with a MemoryError, but at least that's easy to debug when it happens. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > > From p.f.moore at gmail.com Tue Jul 28 12:50:00 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Jul 2009 11:50:00 +0100 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <3fcf944249977ccd966d62a73ac0db5d@preisshare.net> References: <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <1b9d4c7557e19faed93300f157d07b79@preisshare.net> <4A6E88B9.5010504@v.loewis.de> <14cf53f0cf73387c177f045fd21151b7@preisshare.net> <4A6E92BF.7080901@v.loewis.de> <3fcf944249977ccd966d62a73ac0db5d@preisshare.net> Message-ID: <79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com> 2009/7/28 David Lyon : > On Tue, 28 Jul 2009 07:55:11 +0200, "Martin v. L?wis" > wrote: >> Yes, eggs have the same problem. That's one of the reasons they >> don't get integrated into Python. > > Yes but egg_info is included in python... > > and the egg is not.... > > Hence, what goes in and what doesn't isn't always that rational. I'm > just accepting that for the moment. egg_info data is in to allow "standard" (setup.py install and hence OS package manager managed) packages to provide metadata in a discoverable way. Using a format that is (reasonably) compatible with setuptools is simply a matter of co-operating with existing de facto standards. Eggs themselves (as a distribution format) are just zip files with a funny extension, and as such are supported by Python. The infrastructure and philosophy around eggs (easy_install, the various .pth file manipulations, multi-version installs, etc) are supported by Python (in the trivial sense that they are possible) but are not "blessed" by standard library inclusion, precisely because of the issues being mentioned here. Your package manager has the same issues as the egg infrastructure (lack of integration with system package managers being the biggest one) and so is not suitable for the standard library in precisely the same way. Paul. From ncoghlan at gmail.com Tue Jul 28 13:26:58 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jul 2009 21:26:58 +1000 Subject: [Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded In-Reply-To: <4A6E3DCB.7030601@canterbury.ac.nz> References: <4A6CFD73.8050201@canterbury.ac.nz> <4A6E3DCB.7030601@canterbury.ac.nz> Message-ID: <4A6EE082.3010509@gmail.com> Greg Ewing wrote: > The only real concern would be if it were somehow possible to crash > the interpreter by modifying the type dict. I don't see how that > could happen -- but maybe someone else on python-dev knows more about > this? I believe a major part of the issue is that attempting to answer the question "Will allowing mutating operation *X* on object/type/float/int/etc create any interpreter stability or security problems?" is awfully close to trying to prove a negative. Certainly, at least some of the "generic" operations involving types include additional sanity checks that are bypassed for the builtin types. One specific example I can think of is that object.__hash__ is special cased in a few places due to the way its definition interacts with the definition of comparison operations. Allowing changes to the contents of object's tp_hash slot could lead to much weirdness when it came to __hash__ inheritance. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From ncoghlan at gmail.com Tue Jul 28 13:29:54 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jul 2009 21:29:54 +1000 Subject: [Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded In-Reply-To: <4A6EE082.3010509@gmail.com> References: <4A6CFD73.8050201@canterbury.ac.nz> <4A6E3DCB.7030601@canterbury.ac.nz> <4A6EE082.3010509@gmail.com> Message-ID: <4A6EE132.8020507@gmail.com> Nick Coghlan wrote: > One specific example I can think of is that object.__hash__ is special > cased in a few places due to the way its definition interacts with the > definition of comparison operations. Allowing changes to the contents of > object's tp_hash slot could lead to much weirdness when it came to > __hash__ inheritance. Just thought of a much better example as I clicked send: the basic numeric types (especially int) are locked down because they are special-cased all over the place (including in the main interpreter eval loop) in order to speed up simple arithmetic. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From rocky at gnu.org Tue Jul 28 14:10:45 2009 From: rocky at gnu.org (Rocky Bernstein) Date: Tue, 28 Jul 2009 08:10:45 -0400 Subject: [Python-Dev] License for code extracted from a Python library? Message-ID: <6cd6de210907280510m19261f82x5ab8e592a17ced48@mail.gmail.com> Hi - I'm not sure if this is the right place to ask this, but I don't know of a more appropriate place. Sometime around 2007 I first extracted a function that was in Python's Cmd class and used that. Since then, I have modified and generalized it a bit more and turned it into a Python egg called columnize. I also ported it to Ruby and made Ruby package or gem. A couple of people/companies have asked about redistributing this and have asked for a more MIT-like and less GPL-like license. I guess they will bundle this with their proprietary code. For the Python package, I used PSL Version 2, but is this legitimate? code.google.com doesn't seem to recognize in their list of licenses so that suggests what I did might not be the right thing. (So instead, code.google.com I selected the MIT License.) Can a MIT License be used for code extracted from Python's standard library? Other comments or suggestions? Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Jul 28 14:43:55 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jul 2009 22:43:55 +1000 Subject: [Python-Dev] License for code extracted from a Python library? In-Reply-To: <6cd6de210907280510m19261f82x5ab8e592a17ced48@mail.gmail.com> References: <6cd6de210907280510m19261f82x5ab8e592a17ced48@mail.gmail.com> Message-ID: <4A6EF28B.4050304@gmail.com> Rocky Bernstein wrote: > Can a MIT License be used for code extracted from Python's standard > library? Other comments or suggestions? The extracted code itself would stay under the PSF license since you don't have the rights to change the license on that. However, as the PSL itself is a very permissive license, you can bundle it with code under whatever other license you or users of your package choose to use (even proprietary ones). That said, while I know a fair bit about software licensing I'm not a lawyer, and even if I was one, I still wouldn't be *your* lawyer, so take what I have to say with a fairly large grain of salt! :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From david.lyon at preisshare.net Tue Jul 28 16:14:52 2009 From: david.lyon at preisshare.net (David Lyon) Date: Tue, 28 Jul 2009 10:14:52 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com> References: <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <1b9d4c7557e19faed93300f157d07b79@preisshare.net> <4A6E88B9.5010504@v.loewis.de> <14cf53f0cf73387c177f045fd21151b7@preisshare.net> <4A6E92BF.7080901@v.loewis.de> <3fcf944249977ccd966d62a73ac0db5d@preisshare.net> <79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com> Message-ID: <2e8c8bc947207c89e75c74cbbcba8920@preisshare.net> On Tue, 28 Jul 2009 11:50:00 +0100, Paul Moore wrote: > egg_info data is in to allow "standard" (setup.py install and hence OS > package manager managed) packages to provide metadata in a > discoverable way. Using a format that is (reasonably) compatible with > setuptools is simply a matter of co-operating with existing de facto > standards. > > Eggs themselves (as a distribution format) are just zip files with a > funny extension, and as such are supported by Python. ok - I get it..... It's all one big monty python comedy thing where things are quirky and inconsistent - there's little continuety from one scene to the next but you have to stop sometimes and have a laugh... because to word quote from you - "funny" Your whole email whilst perphaps technically correct is terribly difficult for a software engineering person to follow. Monty python was only meant to be a funny film for entertainment, not a philosophy for building software from.... Why we can have the egg_info without allowing the egg is kind of like having a chicken and egg story without allowing the chicken.. or is it the egg.. oh I lost track... I think I need to watch the movie again to understand what's happening here... Let me go away confused... don't ask me any more questions or elaborate with more answers. Thanks. :-) David From p.f.moore at gmail.com Tue Jul 28 20:23:01 2009 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Jul 2009 19:23:01 +0100 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <2e8c8bc947207c89e75c74cbbcba8920@preisshare.net> References: <4A6C9371.6000002@v.loewis.de> <1b9d4c7557e19faed93300f157d07b79@preisshare.net> <4A6E88B9.5010504@v.loewis.de> <14cf53f0cf73387c177f045fd21151b7@preisshare.net> <4A6E92BF.7080901@v.loewis.de> <3fcf944249977ccd966d62a73ac0db5d@preisshare.net> <79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com> <2e8c8bc947207c89e75c74cbbcba8920@preisshare.net> Message-ID: <79990c6b0907281123j586f698ek896e7512c53a6845@mail.gmail.com> 2009/7/28 David Lyon : > ok - I get it..... [...] > Your whole email whilst perphaps technically correct is terribly > difficult for a software engineering person to follow. OK, I'm sorry if my attempts to help you didn't do so. > Let me go away confused... don't ask me any more questions or > elaborate with more answers. Thanks. :-) Don't worry, I'll do my best not to from now on :-) Paul. From ncoghlan at gmail.com Wed Jul 29 02:56:20 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jul 2009 10:56:20 +1000 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <2e8c8bc947207c89e75c74cbbcba8920@preisshare.net> References: <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <1b9d4c7557e19faed93300f157d07b79@preisshare.net> <4A6E88B9.5010504@v.loewis.de> <14cf53f0cf73387c177f045fd21151b7@preisshare.net> <4A6E92BF.7080901@v.loewis.de> <3fcf944249977ccd966d62a73ac0db5d@preisshare.net> <79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com> <2e8c8bc947207c89e75c74cbbcba8920@preisshare.net> Message-ID: <4A6F9E34.5010708@gmail.com> David Lyon wrote: > Your whole email whilst perphaps technically correct is terribly > difficult for a software engineering person to follow. It made perfect sense to me. The words "eggs" brings with it a whole lot more baggage than just the sum of the technical parts in the language core that support them (primarily distutils and zipimport). I find it unfortunate that the name for the distutils metadata format contains the word "egg" because it implies much greater consensus around the philosophy behind eggs than actually exists. A lot of the baggage associated with the "eggs" concept is related to the inherent conflict between different approaches to dependency management: 1. Use the system package management system for everything (preferred by many, perhaps even most, *nix sysadmins, but not an option on Windows) 2. Create a Python specific package management system independent of the system package manager (an area dominated by setuptools, including both eggs and non-zipped package distributions) 3. Bundle everything into a monolithic application or framework (the typical approach on Windows with py2exe, but also the philosophy behind tools like virtualenv) All 3 of those philosophies have good arguments in their favour and they all have good arguments against them as well. Your comments about your package management system suggest that it is just yet another entrant in category 2 and you have said nothing to allay the concerns of those that despise that philosophy with a passion because of the problems it causes them when trying to manage their systems using the first philosophy. Since the Python constituency includes developers and system administrators that favour all 3 philosophies (and probably other variants that I haven't thought to describe), anything that makes it into the standard library will need to adequately balance the interests of all parties (e.g. as has occurred in the PEP 376 metadata enhancement discussions). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From tjreedy at udel.edu Wed Jul 29 07:42:43 2009 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 29 Jul 2009 01:42:43 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <4A6F9E34.5010708@gmail.com> References: <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <1b9d4c7557e19faed93300f157d07b79@preisshare.net> <4A6E88B9.5010504@v.loewis.de> <14cf53f0cf73387c177f045fd21151b7@preisshare.net> <4A6E92BF.7080901@v.loewis.de> <3fcf944249977ccd966d62a73ac0db5d@preisshare.net> <79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com> <2e8c8bc947207c89e75c74cbbcba8920@preisshare.net> <4A6F9E34.5010708@gmail.com> Message-ID: Nick Coghlan wrote: > David Lyon wrote: >> Your whole email whilst perphaps technically correct is terribly >> difficult for a software engineering person to follow. > > It made perfect sense to me. Like David, I found it a bit disjointed too. > > The words "eggs" brings with it a whole lot more baggage than just the > sum of the technical parts in the language core that support them > (primarily distutils and zipimport). I find it unfortunate that the name > for the distutils metadata format contains the word "egg" because it > implies much greater consensus around the philosophy behind eggs than > actually exists. > > A lot of the baggage associated with the "eggs" concept is related to > the inherent conflict between different approaches to dependency management: > 1. Use the system package management system for everything (preferred by > many, perhaps even most, *nix sysadmins, but not an option on Windows) > 2. Create a Python specific package management system independent of the > system package manager (an area dominated by setuptools, including both > eggs and non-zipped package distributions) > 3. Bundle everything into a monolithic application or framework (the > typical approach on Windows with py2exe, but also the philosophy behind > tools like virtualenv) > > All 3 of those philosophies have good arguments in their favour and they > all have good arguments against them as well. Your comments about your > package management system suggest that it is just yet another entrant in > category 2 and you have said nothing to allay the concerns of those that > despise that philosophy with a passion because of the problems it causes > them when trying to manage their systems using the first philosophy. > > Since the Python constituency includes developers and system > administrators that favour all 3 philosophies (and probably other > variants that I haven't thought to describe), anything that makes it > into the standard library will need to adequately balance the interests > of all parties (e.g. as has occurred in the PEP 376 metadata enhancement > discussions). However, the above clarifies for me what the alternatives and issues are. Thank you for posting. Terry Jan Reedy From david.lyon at preisshare.net Wed Jul 29 08:36:30 2009 From: david.lyon at preisshare.net (David Lyon) Date: Wed, 29 Jul 2009 02:36:30 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: <4A6F9E34.5010708@gmail.com> References: <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <1b9d4c7557e19faed93300f157d07b79@preisshare.net> <4A6E88B9.5010504@v.loewis.de> <14cf53f0cf73387c177f045fd21151b7@preisshare.net> <4A6E92BF.7080901@v.loewis.de> <3fcf944249977ccd966d62a73ac0db5d@preisshare.net> <79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com> <2e8c8bc947207c89e75c74cbbcba8920@preisshare.net> <4A6F9E34.5010708@gmail.com> Message-ID: On Wed, 29 Jul 2009 10:56:20 +1000, Nick Coghlan wrote: > The words "eggs" brings with it a whole lot more baggage than just the > sum of the technical parts in the language core that support them > (primarily distutils and zipimport). Well, in this case, (talking metaphorically) we have the ability to roll the eggs, crack a whole in them and suck out the contents (deal with a package in a zipped egg) So metaphorically we could say that python can work with egg shells.. As for what's in the egg... well lets just say that it's a bit floaty.. And perphaps that is the best way for things to be for a while. > I find it unfortunate that the name > for the distutils metadata format contains the word "egg" because it > implies much greater consensus around the philosophy behind eggs than > actually exists. I see it a different way. I thinks eggs are simple - as in like a java bean. A simple way to move a package from one place to another. What seems to have gone wrong is that there is a lot of pioneering and interconnected and interdependent technology within them. They're an egg.. but in the past they've had little monsters that bite your fingers when you try to put them in the pan... :-) I think "Egg" term is very good. But maybe we are best not trying to over-specify their contents. Maybe we should say it has certain things (EGG_INFO, PACKAGE DATA) as in (YOLK, WHITE) and pretty much leave it at that. If more detail is required - rtm. > A lot of the baggage associated with the "eggs" concept is related to > the inherent conflict between different approaches to dependency > management: That's not an egg problem. That's a packaging/metadata problem. It's the package dependency issue that's the problem. That's a distutils problem. > 1. Use the system package management system for everything (preferred by > many, perhaps even most, *nix sysadmins, but not an option on Windows) Yes, because the package dependency issues are just so great. > 2. Create a Python specific package management system independent of the > system package manager (an area dominated by setuptools, including both > eggs and non-zipped package distributions) More work needs to go into distutils. Not taking away from any existing work - but setuptools has relied on the deficiencies of distutils to gain a foothold. Fix distutils (give me time to think..) and the need for setuptools and any further fork is probably negated. > 3. Bundle everything into a monolithic application or framework (the > typical approach on Windows with py2exe, but also the philosophy behind > tools like virtualenv) Windows users are using py2exe and other products over distutils. To a normal windows developer, distutils makes imho no sense in the way that it goes about things now. For example, very simple concepts like "program directories", (where you stick the program) isn't an available option in distutils. But it is the first thing that you need to know in a windows program. So it's very hard to get to step 1... > Your comments about your > package management system suggest that it is just yet another entrant in > category 2 and you have said nothing to allay the concerns of those that > despise that philosophy with a passion because of the problems it causes > them when trying to manage their systems using the first philosophy. I'm a windows user.. I can't be in category #2.. I'm in category #1, have nothing... (except now my own tool) > Since the Python constituency includes developers and system > administrators that favour all 3 philosophies (and probably other > variants that I haven't thought to describe), anything that makes it > into the standard library will need to adequately balance the interests > of all parties (e.g. as has occurred in the PEP 376 metadata enhancement > discussions). Well at least you are saying that there is some way of reconciling all the different options... There's an awful lot to take in, and there must be 20,000 lines of emails for every 1 line of python code that is required to fix this thing. Take care David From ncoghlan at gmail.com Wed Jul 29 10:10:26 2009 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jul 2009 18:10:26 +1000 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <200907261805.08277.steve@pearwood.info> <4A6C9371.6000002@v.loewis.de> <1b9d4c7557e19faed93300f157d07b79@preisshare.net> <4A6E88B9.5010504@v.loewis.de> <14cf53f0cf73387c177f045fd21151b7@preisshare.net> <4A6E92BF.7080901@v.loewis.de> <3fcf944249977ccd966d62a73ac0db5d@preisshare.net> <79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com> <2e8c8bc947207c89e75c74cbbcba8920@preisshare.net> <4A6F9E34.5010708@gmail.com> Message-ID: <4A7003F2.4080308@gmail.com> David Lyon wrote: > There's an awful lot to take in, and there must be 20,000 lines of > emails for every 1 line of python code that is required to fix this > thing. Yep, which goes way back to one of my first emails in this thread: compared to the social aspects, the technical aspects of packaging tools are relatively straightforward :) It's great to see some energy directed towards solving these issues, but I think it's important for those trying to contribute to realise what they're dealing with. I just hope I don't kill their enthusiasm in the process :P Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- From jnoller at gmail.com Wed Jul 29 15:11:24 2009 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 29 Jul 2009 09:11:24 -0400 Subject: [Python-Dev] Update to Python Documentation Website Request In-Reply-To: References: <1b9d4c7557e19faed93300f157d07b79@preisshare.net> <4A6E88B9.5010504@v.loewis.de> <14cf53f0cf73387c177f045fd21151b7@preisshare.net> <4A6E92BF.7080901@v.loewis.de> <3fcf944249977ccd966d62a73ac0db5d@preisshare.net> <79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com> <2e8c8bc947207c89e75c74cbbcba8920@preisshare.net> <4A6F9E34.5010708@gmail.com> Message-ID: <4222a8490907290611l1b31c3acr71c2baae3913af3e@mail.gmail.com> On Wed, Jul 29, 2009 at 2:36 AM, David Lyon wrote: > On Wed, 29 Jul 2009 10:56:20 +1000, Nick Coghlan > wrote: >> The words "eggs" brings with it a whole lot more baggage than just the >> sum of the technical parts in the language core that support them >> (primarily distutils and zipimport). > > Well, in this case, (talking metaphorically) we have the ability > to roll the eggs, crack a whole in them and suck out the contents > (deal with a package in a zipped egg) > > So metaphorically we could say that python can work with egg shells.. > > As for what's in the egg... well lets just say that it's a bit floaty.. > > And perphaps that is the best way for things to be for a while. > >> I find it unfortunate that the name >> for the distutils metadata format contains the word "egg" because it >> implies much greater consensus around the philosophy behind eggs than >> actually exists. > > I see it a different way. I thinks eggs are simple - as in like a java > bean. A simple way to move a package from one place to another. > > What seems to have gone wrong is that there is a lot of pioneering > and interconnected and interdependent technology within them. They're > an egg.. but in the past they've had little monsters that bite your > fingers when you try to put them in the pan... :-) > > I think "Egg" term is very good. But maybe we are best not trying > to over-specify their contents. > > Maybe we should say it has certain things (EGG_INFO, PACKAGE DATA) > as in (YOLK, WHITE) and pretty much leave it at that. If more detail > is required - rtm. > >> A lot of the baggage associated with the "eggs" concept is related to >> the inherent conflict between different approaches to dependency >> management: > > That's not an egg problem. That's a packaging/metadata problem. > > It's the package dependency issue that's the problem. That's a distutils > problem. > >> 1. Use the system package management system for everything (preferred by >> many, perhaps even most, *nix sysadmins, but not an option on Windows) > > Yes, because the package dependency issues are just so great. > >> 2. Create a Python specific package management system independent of the >> system package manager (an area dominated by setuptools, including both >> eggs and non-zipped package distributions) > > More work needs to go into distutils. Not taking away from any existing > work - but setuptools has relied on the deficiencies of distutils to > gain a foothold. > > Fix distutils (give me time to think..) and the need for setuptools and > any further fork is probably negated. > >> 3. Bundle everything into a monolithic application or framework (the >> typical approach on Windows with py2exe, but also the philosophy behind >> tools like virtualenv) > > Windows users are using py2exe and other products over distutils. To a > normal windows developer, distutils makes imho no sense in the way that > it goes about things now. > > For example, very simple concepts like "program directories", (where > you stick the program) isn't an available option in distutils. But > it is the first thing that you need to know in a windows program. > > So it's very hard to get to step 1... > >> Your comments about your >> package management system suggest that it is just yet another entrant in >> category 2 and you have said nothing to allay the concerns of those that >> despise that philosophy with a passion because of the problems it causes >> them when trying to manage their systems using the first philosophy. > > I'm a windows user.. I can't be in category #2.. > > I'm in category #1, have nothing... (except now my own tool) > >> Since the Python constituency includes developers and system >> administrators that favour all 3 philosophies (and probably other >> variants that I haven't thought to describe), anything that makes it >> into the standard library will need to adequately balance the interests >> of all parties (e.g. as has occurred in the PEP 376 metadata enhancement >> discussions). > > Well at least you are saying that there is some way of reconciling > all the different options... > > There's an awful lot to take in, and there must be 20,000 lines of > emails for every 1 line of python code that is required to fix this > thing. > > Take care > > David > I really do think this mail thread needs to move to disutils-sig or python-ideas. jesse From solipsis at pitrou.net Wed Jul 29 15:45:36 2009 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Jul 2009 13:45:36 +0000 (UTC) Subject: [Python-Dev] Update to Python Documentation Website Request References: <1b9d4c7557e19faed93300f157d07b79@preisshare.net> <4A6E88B9.5010504@v.loewis.de> <14cf53f0cf73387c177f045fd21151b7@preisshare.net> <4A6E92BF.7080901@v.loewis.de> <3fcf944249977ccd966d62a73ac0db5d@preisshare.net> <79990c6b0907280350s25a9a2dna7c40a4f512fdadf@mail.gmail.com> <2e8c8bc947207c89e75c74cbbcba8920@preisshare.net> <4A6F9E34.5010708@gmail.com> <4222a8490907290611l1b31c3acr71c2baae3913af3e@mail.gmail.com> Message-ID: Jesse Noller gmail.com> writes: > > I really do think this mail thread needs to move to disutils-sig or > python-ideas. +1 From ideasman42 at gmail.com Wed Jul 29 23:41:49 2009 From: ideasman42 at gmail.com (Campbell Barton) Date: Wed, 29 Jul 2009 14:41:49 -0700 Subject: [Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded In-Reply-To: References: Message-ID: <7c1ab96d0907291441s3428cd99j2cd6ed607f79c369@mail.gmail.com> On Sun, Jul 26, 2009 at 11:01 AM, Joshua Haberman wrote: > I'm writing a C Python extension that needs to generate PyTypeObjects > dynamically. ?Unfortunately, the Py_TPFLAGS_HEAPTYPE flag is overloaded > in a way that makes it difficult to achieve this goal. > > The documentation for Pt_TPFLAGS_HEAPTYPE says: > > Py_TPFLAGS_HEAPTYPE > > ?This bit is set when the type object itself is allocated > ?on the heap. In this case, the ob_type field of its > ?instances is considered a reference to the type, and the > ?type object is INCREF?ed when a new instance is created, > ?and DECREF?ed when an instance is destroyed (this does not > ?apply to instances of subtypes; only the type referenced > ?by the instance?s ob_type gets INCREF?ed or DECREF?ed). > > This sounds like exactly what I want. ?I want my type object INCREF'd > and DECREF'd by its instances so it doesn't leak or get deleted > prematurely. ?If this were all that Py_TPFLAGS_HEAPTYPE did, it would > work great for me. > > Unfortunately, Py_TPFLAGS_HEAPTYPE is also overloaded to mean > "user-defined type" (as opposed to a built-in type). ?It controls > numerous subtle behaviors such as: > > - whether the type's name is module.type or just type. > - whether you're allowed to set __name__, __module__, or __bases__ on the type. > - whether you're allowed to set __class__ on instances of this type. > - whether the module name comes from the type name or the __module__ attribute. > - whether it will use type->tp_doc as the docstring > - whether its repr() calls it a "class" or a "type". > - whether you can set attributes of the type. > - whether someone is attempting the Carlo Verre hack. > > So I'm stuck with an unenviable choice. ?I think the lesser of two evils > is to *not* specify Py_TPFLAGS_HEAPTYPE, because the worst that will > happen is that my types will leak. ?This is not as bad as having someone > set __class__ on one of my instances, or set attributes on my type, etc. > > Ideally the interpreter would have a separate flag like > Py_TPFLAGS_BUILTIN that would trigger all of the above behaviors, but > still make it possible to have dynamically generated built-in types get > garbage collected appropriately. > > At the very least, the documentation I cited above should make it clear > that Py_TPFLAGS_HEAPTYPE controls more than just whether the type gets > INCREF'd and DECREF'd. ?Based on the list of behaviors I discovered > above, it is almost certainly not correct for a C exension type to be > declared with Py_TPFLAGS_HEAPTYPE. > > Josh Hi Joshua, recently I also needed to dynamically make subtypes from C, I tried 2 ways of doing this, one is to do the C equivalent of calling type("name",(bases,...), dict) and the other is to malloc() PyTypeObject's, fill in the slots and run PyType_Ready on them to initialize them. It seems the first is the expected way to make your own types so I assume thats what your doing?, Just wondering because if you do it the second way I think youll have more control and the types will be more limited (like internal types). I'm not expert enough in this area to know if malloc'ing PyTypeObject and initializing has some other problems. - Campbell From eric.pruitt at gmail.com Thu Jul 30 01:41:05 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Wed, 29 Jul 2009 18:41:05 -0500 Subject: [Python-Dev] Implementing File Modes In-Reply-To: <52dc1c820907280053v6ab7cb4fo31b7d2e49c53021@mail.gmail.com> References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> <171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com> <79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com> <52dc1c820907280053v6ab7cb4fo31b7d2e49c53021@mail.gmail.com> Message-ID: <171e8a410907291641k14c68eafi1dd0ef375f2a6906@mail.gmail.com> My motivation came from an instance when I was using subprocess.Popen for a Linux / Windows cross platform program. In part of the program, I was writing and reading to a cron like object. On Windows, it was a text file and on Linux it would be the crontab executable. Had I been able to substitute the "open()" function with my wrapper, it would have been the only change I had to make for cross platform compatibility; instead of having to change numerous lines because Linux would need Popen and Windows would need a regular file open(), I could simply make it so that if the platform was Linux, my wrapper is used in place of that. Just another example would be having an external program decrypt a file that can be in plain text or encrypted that might go something like this: if encryptionEnabled: fileobj = subprocess.ProcessIOWrapper("gpg --decrypt supersecret.html.gpg") else: fileobj = open("notsosecret.html") >From there, the functions would not have to be modified despite using a running process versus a file object. On Tue, Jul 28, 2009 at 01:53, Gregory P. Smith wrote: > On Mon, Jul 27, 2009 at 5:32 PM, Glyph Lefkowitz > wrote: > > On Mon, Jul 27, 2009 at 3:04 PM, Paul Moore wrote: > >> > >> I like MRAB's idea of using a (non-standard) "e" flag to include > >> stderr. So "r" reads from stdout, "re" reads from stdout+stderr. > >> > >> Anything more complicated probably should just use "raw" Popen > >> objects. Don't overcomplicate the interface. > > > > In my opinion, mangling stderr and stdout together is already an > > overcomplication. It shouldn't be implemented. > > > > It seems like a good idea, until you realize that subtle changes to your > OS, > > environment, or buffering behavior may result in arbitrary, unparseable > > output. > > Agreed. Leave stderr support out of this. People who need stderr > should use the full subprocess.Popen interface. Quick hack unixy > types will just run their process using a shell (which already seems > to be the default based on the "ls -l" example) and add 2>&1. This > functionality is basically the equivalent of adding the | symbol on > either or both ends of a filename given to open() in perl. (but > without being so gross). > > I do wonder why you're trying to make it behave exactly like open() > including the mode= syntax. > > Why not just define several names based on the behavior? > > ProcessReadWrapper() > ProcessWriteWrapper() > ProcessReadWriteWrapper() > > -gps > > > > > For example, let's say you've got a program whose output is a list of > lines, > > each one containing a number. Sometimes it tries to import gtk, and > fails > > to open its display. > > > > That's fine, and you can still deal with it, as long as the interleaved > > output looks like this: > > > > 100 > > 200 > > Gtk-WARNING **: cannot open display: > > 300 > > 400 > > > > but of course the output might (although unlikely with such small chunks > of > > output) end up looking like this, instead: > > > > 100 > > 2Gtk-WAR0NING0 **: > > can30not 0open display: > > > > 400 > > > > this is the sort of thing which is much more likely to happen once you > start > > dealing with large volumes of data, where there are more page-boundaries > for > > your buffers to get confused around, and you are playing with buffering > > options to improve performance. In other words, it's something that > fails > > only at scale or under load, and is therefore extremely difficult to > debug. > > > > This option might be okay if it were allowed only on subprocesses opened > in > > a text mode, and if the buffering logic involved forced stderr and stdout > to > > be line-delimited, and interleave only lines, rather than arbitrary > chunks > > of bytes. Of course then if you use this flag with a program that > outputs > > binary data with no newlines it will buffer forever and crash your > program > > with a MemoryError, but at least that's easy to debug when it happens. > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From devin.c.cook at gmail.com Thu Jul 30 03:20:36 2009 From: devin.c.cook at gmail.com (Devin Cook) Date: Wed, 29 Jul 2009 21:20:36 -0400 Subject: [Python-Dev] Implementing File Modes In-Reply-To: <171e8a410907291641k14c68eafi1dd0ef375f2a6906@mail.gmail.com> References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> <171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com> <79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com> <52dc1c820907280053v6ab7cb4fo31b7d2e49c53021@mail.gmail.com> <171e8a410907291641k14c68eafi1dd0ef375f2a6906@mail.gmail.com> Message-ID: <8847f980907291820n44f3347fu577cd3032ccb924@mail.gmail.com> Hmm... can't you do this? if encryptionEnabled: p = subprocess.Popen(["gpg", "--decrypt", "supersecret.html.gpg"], stdin = subprocess.PIPE) fileobj = p.stdin else: fileobj = open("notsosecret.html") I think that works. Is there something this way won't work for? You can also do the same thing to get stdout and stderr file objects. I guess a wrapper would simplify this process. -Devin On Wed, Jul 29, 2009 at 7:41 PM, Eric Pruitt wrote: > My motivation came from an instance when I was using subprocess.Popen for a > Linux / Windows cross platform program. In part of the program, I was > writing and reading to a cron like object. On Windows, it was a text file > and on Linux it would be the crontab executable. Had I been able to > substitute the "open()" function with my wrapper, it would have been the > only change I had to make for cross platform compatibility; instead of > having to change numerous lines because Linux would need Popen and Windows > would need a regular file open(), I could simply make it so that if the > platform was Linux, my wrapper is used in place of that. Just another > example would be having an external program decrypt a file that can be in > plain text or encrypted that might go something like this: > > if encryptionEnabled: > ??? fileobj = subprocess.ProcessIOWrapper("gpg --decrypt > supersecret.html.gpg") > else: > ??? fileobj = open("notsosecret.html") > From eric.pruitt at gmail.com Thu Jul 30 05:02:22 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Wed, 29 Jul 2009 22:02:22 -0500 Subject: [Python-Dev] Implementing File Modes In-Reply-To: <8847f980907291820n44f3347fu577cd3032ccb924@mail.gmail.com> References: <171e8a410907270901s444e814y67d7573115c18395@mail.gmail.com> <79990c6b0907270946k55ed3992n1853ad8dd91b7f13@mail.gmail.com> <171e8a410907271106n520a975eu2e003404cb1b372e@mail.gmail.com> <79990c6b0907271204n45c0ffc4gc254869e79633da2@mail.gmail.com> <52dc1c820907280053v6ab7cb4fo31b7d2e49c53021@mail.gmail.com> <171e8a410907291641k14c68eafi1dd0ef375f2a6906@mail.gmail.com> <8847f980907291820n44f3347fu577cd3032ccb924@mail.gmail.com> Message-ID: <171e8a410907292002w66d26d91h5879d92b9fcb1735@mail.gmail.com> Well, with a few changes to your code, that would indeed work (you are using stdin as your pipe. Correct me if I'm wrong but if you intend to read from it, you need to change it to "stdout = subprocess.PIPE" and the other lines as well to reflect this change). My Google Summer of Code modifications to subprocess.Popen provide non-blocking, asynchronous I/O support and my file wrapper is built upon that augmented functionality. If I remember correctly, when I was working on the program where I first thought a file wrapper for subprocess.Popen would be rather handy, I also ran into blocking I/O as well. On Wed, Jul 29, 2009 at 20:20, Devin Cook wrote: > Hmm... can't you do this? > > if encryptionEnabled: > p = subprocess.Popen(["gpg", "--decrypt", "supersecret.html.gpg"], > stdin = subprocess.PIPE) > fileobj = p.stdin > else: > fileobj = open("notsosecret.html") > > I think that works. Is there something this way won't work for? You > can also do the same thing to get stdout and stderr file objects. I > guess a wrapper would simplify this process. > > -Devin > > On Wed, Jul 29, 2009 at 7:41 PM, Eric Pruitt wrote: > > My motivation came from an instance when I was using subprocess.Popen for > a > > Linux / Windows cross platform program. In part of the program, I was > > writing and reading to a cron like object. On Windows, it was a text file > > and on Linux it would be the crontab executable. Had I been able to > > substitute the "open()" function with my wrapper, it would have been the > > only change I had to make for cross platform compatibility; instead of > > having to change numerous lines because Linux would need Popen and > Windows > > would need a regular file open(), I could simply make it so that if the > > platform was Linux, my wrapper is used in place of that. Just another > > example would be having an external program decrypt a file that can be in > > plain text or encrypted that might go something like this: > > > > if encryptionEnabled: > > fileobj = subprocess.ProcessIOWrapper("gpg --decrypt > > supersecret.html.gpg") > > else: > > fileobj = open("notsosecret.html") > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From hrvoje.niksic at avl.com Thu Jul 30 10:13:38 2009 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Thu, 30 Jul 2009 10:13:38 +0200 Subject: [Python-Dev] Py_TPFLAGS_HEAPTYPE too overloaded In-Reply-To: <7c1ab96d0907291441s3428cd99j2cd6ed607f79c369@mail.gmail.com> References: <7c1ab96d0907291441s3428cd99j2cd6ed607f79c369@mail.gmail.com> Message-ID: <4A715632.6010604@avl.com> Campbell Barton wrote: > I'm not expert enough in this area to know if malloc'ing PyTypeObject > and initializing has some other problems. The only problem is that such types will be expected to be around forever - they are not reference-counted like heap types, so there is no mechanism to free them once they are no longer needed. From eric.pruitt at gmail.com Thu Jul 30 18:37:29 2009 From: eric.pruitt at gmail.com (Eric Pruitt) Date: Thu, 30 Jul 2009 11:37:29 -0500 Subject: [Python-Dev] Implementing File Modes Message-ID: <171e8a410907300937x655e77c1h188d743d5a267cbf@mail.gmail.com> > On Tue, 28 Jul 2009 04:06:45 am Eric Pruitt wrote: > > I am implementing the file wrapper using changes to subprocess.Popen > > that also make it asynchronous and non-blocking so implementing "r+" > > should be trivial to do. How about handling stderr? I have the > > following ideas: leave out support for reading from stderr, make it > > so that there is an optional additional argument like "outputstderr = > > False", create another function that toggles / sets whether stderr or > > stdout is returned or mix the two outputs. > > Leaving it out is always an option. > > As I see it, fundamentally you can either read from stdout and sterr as > two different streams, or you can interleave (mix) them. To me, that > suggests three functions: > > ProcessIOWrapper() ?# read from stdout (or write to stdin etc.) > ProcessIOWrapperStdErr() ?# read/write from stderr > ProcessIOWrapper2() ?# read from mixed stdout and stderr > > I don't like a function to toggle between one and the other: that smacks > of relying on a global setting in a bad way. I suppose you could add an > optional argument to ProcessIOWrapper() to select between stdout, > stderr, or both together. > > > > -- > Steven D'Aprano How would having a toggle function rely on a global setting? Each class would simply have its own member variable like "self.readsstderr." It's a moot point though as I've decided to use separate functions as you suggested. With separate functions, the user doesn't have to worry about modifying the mode keyword if stderr is needed and as an added bonus, it also makes the code a lot more readable. Eric From status at bugs.python.org Fri Jul 31 18:07:24 2009 From: status at bugs.python.org (Python tracker) Date: Fri, 31 Jul 2009 18:07:24 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20090731160724.190F8785FE@psf.upfronthosting.co.za> ACTIVITY SUMMARY (07/24/09 - 07/31/09) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 2292 open (+33) / 16143 closed (+18) / 18435 total (+51) Open issues with patches: 899 Average duration of open issues: 660 days. Median duration of open issues: 414 days. Open Issues Breakdown open 2261 (+32) pending 30 ( +0) Issues Created Or Reopened (52) _______________________________ string.lowercase/uppercase/letters not affected by locale change 07/25/09 CLOSED http://bugs.python.org/issue6525 reopened ezio.melotti inserting None into sys.modules does not raise ImportError with 07/24/09 http://bugs.python.org/issue6563 created brett.cannon Error in Sec. 8.4 of Tutorial 07/24/09 CLOSED http://bugs.python.org/issue6564 created gmarsden improper use of __setitem__ in ElementTree for Python 3.1 07/24/09 http://bugs.python.org/issue6565 created aroberge json.dumps converts None to "null" (not null) 07/24/09 http://bugs.python.org/issue6566 created srid Make inf be almost equal to inf 07/25/09 http://bugs.python.org/issue6567 reopened lucaspmelo unittest test discovery improvements 07/24/09 http://bugs.python.org/issue6568 created michael.foord unittest document bug (random.shuffle sequence) 07/25/09 CLOSED http://bugs.python.org/issue6569 created hsmtkk patch Documentation Clarity 07/25/09 http://bugs.python.org/issue6570 created StMark Help index 07/25/09 CLOSED http://bugs.python.org/issue6571 created cjw Manas Thapliyal wants to chat 07/25/09 CLOSED http://bugs.python.org/issue6572 created gravitywarrior1 set union method ignores arguments appearing after the original 07/25/09 CLOSED http://bugs.python.org/issue6573 created ssmout patch List the __future__ features in a table 07/26/09 http://bugs.python.org/issue6574 created ezio.melotti patch, easy Can't download docs 07/26/09 CLOSED http://bugs.python.org/issue6575 created zuo re docs: wrong link targets 07/26/09 CLOSED http://bugs.python.org/issue6576 created zuo Links wrongly targeting to builtin functions' instead of module 07/26/09 CLOSED http://bugs.python.org/issue6577 created zuo 2 problems with 'Docs for other versions' section on left HTML d 07/26/09 CLOSED http://bugs.python.org/issue6578 created zuo No update about automatic numbering of fields in format strings 07/26/09 http://bugs.python.org/issue6579 reopened georg.brandl No deprecation warning for list comprehension leak conflict 07/26/09 http://bugs.python.org/issue6580 created mrbax inspect.py sys._getframe patch for Python 2.6 07/26/09 CLOSED http://bugs.python.org/issue6581 created michael.foord patch, patch, needs review test_telnetlib doesn't test Telnet.write 07/26/09 http://bugs.python.org/issue6582 created jackdied easy 2to3 fails to fix test.test_support 07/27/09 http://bugs.python.org/issue6583 created joe.amenta gzip module has no custom exception 07/27/09 http://bugs.python.org/issue6584 created srid configure.in forces specific autoconf version 07/27/09 CLOSED http://bugs.python.org/issue6585 created cavallo71 Documentation of os.write and os.read are inaccurate. 07/27/09 CLOSED http://bugs.python.org/issue6586 created cliechti interrupts during long writes cause connection corruption with S 07/28/09 http://bugs.python.org/issue6587 created janssen insert cookies into cookie jar - cookielib.py 07/28/09 http://bugs.python.org/issue6588 created jondebonis patch smtpd.SMTPServer can cause asyncore.loop to enter infinite event 07/28/09 http://bugs.python.org/issue6589 created cmcginty Py3K can't set the browser's character encoding! (CGI) 07/28/09 CLOSED http://bugs.python.org/issue6590 created [PhysiC] add reference to fcntl.ioctl in the socket module 07/28/09 CLOSED http://bugs.python.org/issue6591 created gagenellina patch otparse: processing variable number of short option arguments 07/28/09 http://bugs.python.org/issue6592 created tanvalley Documentation: gettext install link 07/28/09 CLOSED http://bugs.python.org/issue6593 created efrerich json C serializer performance tied to structure depth on some sy 07/28/09 http://bugs.python.org/issue6594 created swalker Make Decimal constructor accept all unicode decimal digits in in 07/29/09 http://bugs.python.org/issue6595 created marketdickinson patch urllib2 bug on CentOS 07/29/09 http://bugs.python.org/issue6596 created rk3dov Deprecate iterable.next in Python > 2.6.x when called with -3 op 07/29/09 CLOSED http://bugs.python.org/issue6597 created mattrussell calling email.utils.make_msgid frequently has a non-trivial prob 07/30/09 http://bugs.python.org/issue6598 created mwh 2to3 test_print_function_option fails on Windows 07/30/09 http://bugs.python.org/issue6599 created abbeyj patch MemoryError in AiX 64-bit build - PyMem_MALLOC failed 07/30/09 http://bugs.python.org/issue6600 created srid Fractions do not support other Fractions as numerators or denomi 07/30/09 CLOSED http://bugs.python.org/issue6601 created honeyman BaseHTTPServer log_message should log to sys.stdout 07/30/09 http://bugs.python.org/issue6602 created tomkm Compilation error if configuref --with-computed-gotos 07/30/09 http://bugs.python.org/issue6603 created wiget test_distutils subtest test_get_exe_bytes fails depending on exe 07/30/09 http://bugs.python.org/issue6604 created matejcik patch smtplib.SMTP.sendmail() rejected after quit(),connect() sequence 07/30/09 http://bugs.python.org/issue6605 created cmcginty csv.Sniffer.sniff on data with doublequotes doesn't set up the d 07/30/09 http://bugs.python.org/issue6606 created jtate patch asctime causing python to crash 07/30/09 CLOSED http://bugs.python.org/issue6607 created AmirHabibi asctime causing python to crash 07/30/09 http://bugs.python.org/issue6608 created AmirHabibi zipfile: WindowsError [267] The directory name is invalid 07/31/09 http://bugs.python.org/issue6609 created srid Subprocess descriptor debacle 07/31/09 http://bugs.python.org/issue6610 created christian.heimes easy, needs review HTMLParser cannot deal with mixture of arbitrary data and charac 07/31/09 http://bugs.python.org/issue6611 created liudongmiao at gmail.com 'import site' fails when called from an unlinked directory 07/31/09 http://bugs.python.org/issue6612 created labrat ctypes.PyThreadState_SetAsyncExc does not work on linux 64bit ma 07/31/09 http://bugs.python.org/issue6613 created coutinhotiago Issues Now Closed (40) ______________________ The ValueError raised by failing to unpack sequence should have 412 days http://bugs.python.org/issue3071 jackdied patch telnetlib process_rawq buffer handling is confused 168 days http://bugs.python.org/issue5188 jackdied patch Subprocess.getstatusoutput Fails Executing 'dir' Command on Wind 98 days http://bugs.python.org/issue5808 sorin spurious space after opening parenthesis when auto-completing 78 days http://bugs.python.org/issue5992 pitrou Document and slightly simplify lnotab tracing 75 days http://bugs.python.org/issue6042 alexandre.vassalotti patch, needs review read_until 63 days http://bugs.python.org/issue6106 jackdied Make logging configuration files easier to use 63 days http://bugs.python.org/issue6136 vsajip logging.basicConfig(level='DEBUG', ... and setLevel("DEBUG") res 24 days http://bugs.python.org/issue6314 vsajip patch trunk does not build with --enable-unicode=ucs4 37 days http://bugs.python.org/issue6330 eric.smith easy nb_divide missing in docs 35 days http://bugs.python.org/issue6336 georg.brandl TimedRotatingFileHandler permission denied rename failure on Win 35 days http://bugs.python.org/issue6343 vsajip zipfile: Invalid argument when opening zero-sized files 11 days http://bugs.python.org/issue6511 amaury.forgeotdarc string.lowercase/uppercase/letters not affected by locale change 0 days http://bugs.python.org/issue6525 georg.brandl cmath documentation misleading: suggests existence of real() an 6 days http://bugs.python.org/issue6548 marketdickinson patch Regex '\d' should not match unicode category 'No'. 4 days http://bugs.python.org/issue6561 marketdickinson patch, needs review Error in Sec. 8.4 of Tutorial 0 days http://bugs.python.org/issue6564 georg.brandl unittest document bug (random.shuffle sequence) 0 days http://bugs.python.org/issue6569 benjamin.peterson patch Help index 0 days http://bugs.python.org/issue6571 georg.brandl Manas Thapliyal wants to chat 0 days http://bugs.python.org/issue6572 benjamin.peterson set union method ignores arguments appearing after the original 2 days http://bugs.python.org/issue6573 rhettinger patch Can't download docs 0 days http://bugs.python.org/issue6575 georg.brandl re docs: wrong link targets 0 days http://bugs.python.org/issue6576 georg.brandl Links wrongly targeting to builtin functions' instead of module 0 days http://bugs.python.org/issue6577 georg.brandl 2 problems with 'Docs for other versions' section on left HTML d 0 days http://bugs.python.org/issue6578 georg.brandl inspect.py sys._getframe patch for Python 2.6 0 days http://bugs.python.org/issue6581 michael.foord patch, patch, needs review configure.in forces specific autoconf version 1 days http://bugs.python.org/issue6585 r.david.murray Documentation of os.write and os.read are inaccurate. 2 days http://bugs.python.org/issue6586 georg.brandl Py3K can't set the browser's character encoding! (CGI) 0 days http://bugs.python.org/issue6590 [PhysiC] add reference to fcntl.ioctl in the socket module 1 days http://bugs.python.org/issue6591 georg.brandl patch Documentation: gettext install link 1 days http://bugs.python.org/issue6593 georg.brandl Deprecate iterable.next in Python > 2.6.x when called with -3 op 1 days http://bugs.python.org/issue6597 brett.cannon Fractions do not support other Fractions as numerators or denomi 0 days http://bugs.python.org/issue6601 marketdickinson asctime causing python to crash 0 days http://bugs.python.org/issue6607 alexandre.vassalotti Hang using files named prn.txt, etc 2812 days http://bugs.python.org/issue481171 ezio.melotti Allow set swig include dirs in setup.py 2108 days http://bugs.python.org/issue828336 wiget patch sendmsg() and recvmsg() for C socket module 1546 days http://bugs.python.org/issue1194378 ezio.melotti patch telnetlib expect() and read_until() do not time out properly 1346 days http://bugs.python.org/issue1360221 jackdied patch urllib/urllib2: HTTPS over (Squid) Proxy fails 1268 days http://bugs.python.org/issue1424152 orsenthil patch urllib2+https+proxy not working 1231 days http://bugs.python.org/issue1448934 orsenthil Telnetlib dosn't accept u'only ascii' 714 days http://bugs.python.org/issue1772794 jackdied Top Issues Most Discussed (10) ______________________________ 28 base64 module ignores non-alphabet characters 1212 days open http://bugs.python.org/issue1466065 8 asyncore incorrect failure when connection is refused and using 9 days open http://bugs.python.org/issue6550 8 Regexp 2.7 (modifications to current re 2.2.2) 472 days open http://bugs.python.org/issue2636 7 MemoryError in AiX 64-bit build - PyMem_MALLOC failed 2 days open http://bugs.python.org/issue6600 7 Make inf be almost equal to inf 6 days open http://bugs.python.org/issue6567 7 OverflowError in RLock.acquire() 7 days open http://bugs.python.org/issue6562 6 csv.Sniffer.sniff on data with doublequotes doesn't set up the 1 days open http://bugs.python.org/issue6606 6 No update about automatic numbering of fields in format strings 5 days open http://bugs.python.org/issue6579 6 Regex '\d' should not match unicode category 'No'. 4 days closed http://bugs.python.org/issue6561 6 email.FeedParser.BufferedSubFile improperly handles "\r\n" 804 days open http://bugs.python.org/issue1721862 From jacobolus at gmail.com Fri Jul 31 23:16:02 2009 From: jacobolus at gmail.com (Jacob Rus) Date: Fri, 31 Jul 2009 21:16:02 +0000 (UTC) Subject: [Python-Dev] standard library mimetypes module pathologically broken? Message-ID: Hi all, In an attempt to figure out some twisted.web code, I was reading through the Python Standard Library?s mimetypes module today, and was shocked at the poor quality of the code. I wonder how the mimetypes code made it into the standard library, and whether anyone has ever bothered to read it or update it: it is an embarrassment. Much of the code is redundant, portions fail to execute, control flow is routed through a horribly confusing mess of spaghetti, and most of the complexity has no clear benefit as far as I can tell. I probably should drop the subject and get back to work, but as a good citizen, it?s hard to just ignore this sort of thing. mimetypes.py stores its types in a pair of dictionaries, one for "strict" use, and the other for "non-standard types". It creates the strict dictionary by default out of apache's mime.types file, and then overrides the entries it finds with a set of exceptions. Then it creates the non-standard dictionary, which is set to match if the strict parameter is set to False when guessing types. Just in this basic design, and in the list of types in the file, there are several problems: * Various apache mime types files are read, if found, but the ordering of the files is such that older versions of apache are sometimes read after newer ones, overriding updated mime types with out-of-date versions if multiple versions of apache are installed on the system. * The vast majority of types declared in mimetypes.py are duplicates of types already declared by Apache. In a few cases this is to change the apache default (make an exception, that is), but in most cases the mime type and extension are completely identical. This huge number of redundant types makes the file substantially harder to follow. No comments are provided to explain why various sets of exceptions are made to Apache's default mime types, and in several cases mimetypes.py seems to just be out of date as compared to recent versions of Apache, for instance not knowing about the 'text/troff' type which was registered in January 2006 in RFC 4263. * The 'non-standard' type dictionary is nearly useless, because all of the types it declares are already in apache's mime.types file, meaning that types are, as far as I can tell trying to follow ugly program flow, *never* drawn from the non-strict dictionary, except in the improbable situation where the mimetypes module is initialized with a custom set of apache-mime.types?like files, which does not include those 'non-standard' types. I personally cannot see a use case for initializing the module with a custom set of mime types, but then leaving the very few types included as non-strict to the defaults: this seems like a fragile and pathological use case. Given this, I don?t see any benefit to dragging the 'strict' parameter along all the way through the code, and would advise getting rid of it altogether. Does anyone know of any code that uses the mimetypes module with strict set to False, where the non-strict code path ever *actually* is executed? But though these problems, which affect actual use of the code and are therefore probably most important, are significant, they really pale in comparison to the awful quality of implementation. I'll try to briefly outline my understanding of how code flows in mimetypes.py, and what the problems are. I haven't stepped through the code in a debugger, this is just from reading it, so I apologize in advance if I get something wrong. This is, however, some of the worst code I?ve seen in the standard library or anywhere else. * It defines __all__: I didn?t even realize __all__ could be used for single-file modules (w/o submodules), but it definitely shouldn?t be here. This specific __all__ oddly does not include all of the documented variables and functions in the mimetypes class. It?s not clear why someone calling import * here wouldn?t want the bits not included. * It creates a _default_mime_types() function which declares a bunch of global variables, and then immediately calls _default_mime_types() below the definition. There is literally no difference in result between this and just putting those variables at the top level of the file, so I have no idea why this function exists, except to make the code more confusing. * It allows command line usage: I don?t think this is necessary for a part of the standard library like this. There are better tools for finding mime types from the command line which ship with most operating systems. * Its API is pretty poorly designed. It offers 6 functions when about 3 are needed, and it takes a couple reads-through of the code to figure out exactly what any of them are supposed to do. * The operation is crazy: It defines a MimeTypes class which actually stores the type mappings, but this class is designed to be a singleton. The way that such a design is enforced is through the use of the module-global 'init' function, which makes an instance of the class, and then maps all of the functions in the module global namespace to instance methods. But confusingly, all such functions are also defined independently of the init function, with definitions such as: def guess_type(url, strict=True): if not inited: init() return guess_type(url, strict) I?d be amazed if anyone could guess what that code was trying to do. I did a double-take when I saw it. Of course, that return call is only ever reached the first time this function is called, if init() has not happened yet. This was all presumably done for lazy initialization, so that the type information would only be loaded when needed. Needless to say, there are more pythonic ways to accomplish such a goal. Oh, also, the other good one here is that it means that someone who writes `from mimetypes import guess_types` gets something different than someone who writes: `import mimetypes; guess_types = mimetypes.guess_types`. In the former case, this wrapper function is saved as guess_type, which each time just calls the (changed after init()) mimetypes.guess_types function. This caused a performance nightmare before March of this year, when there was no check for `if not inited` before running init() (amazing!?). * Because the type datastore is set up to be a singleton, any time init() is called in one section of code, it resets any types which have been added manually: this means that if init() is called by different pieces of code in the same python program, they will interfere with each-others? type databases, and break each-other. This is extremely fragile and, in my opinion, crazy. It is hard for me to imagine any use case that would benefit from this ability to clobber custom type mappings, and I very much doubt that any code calling the mimetypes module realizes that the contract of the API is so flimsy by definition. In practice, I would not advise consumers of this API to ever call init() manually, or to ever add custom mime type mappings, because they are setting themselves up for hard-to-track bugs down the line. * The 'inited' flag is a documented part of the interface, in the standard library documentation. I cannot imagine any reason to set this flag manually: setting it to false when it was true will have no effect, because the top-level functions have already been replaced by instance methods of the 'db' MimeTypes instance. Setting it to true when it was false will make the code just break outright. * In python 3, this has been changed a bit. There?s still an inited flag, and it still in the docs, but now awful code from above has been changed slightly, to: def guess_type(url, strict=True): if _db is None: init() return _db.guess_type(url, strict) Which is still embarrassingly confusing. On the upside, the inited flag now does literally nothing, but remains defined, and in the docs. * The 'types_map' and 'common_types' (for 'strict' and 'common' types, respectively) dictionaries are also a documented part of the interface. When init() is called, a new MimeTypes instance makes a (different) types_map which is a tuple of two dictionaries, for 'strict' and 'common' types. Then this instance reads the apache mime.types files and adds the types to its pair of self.types_map dictionaries, and then after that looks at the global types_map and common_types dictionaries and adds *those* types to its self.types_map. Then at the end it replaces the global types_map with self.types_map[True] and replaces common_types with self.types_map[False]. Unfortunately, while changing these dictionaries will have an effect on the operation of the library, it will not update the types_map_inv mapping, so inverse lookups will not behave as the changer expects. If these dictionaries are going to remain documented, the documentation should be clear to describe them as read only to avoid very confusing bugs. * Speaking of these dictionaries, .copy() is called on those two and a few other inside MimeTypes.__init__(), which happens every time the global init() function is called, but then init() puts the copies back in the global namespace, meaning that the original is discarded. Basically the only reason for the .copy() is to make sure that the correct updates are applied to the apache mimetype defaults, but the code will gladly re-read all of the apache files even after its mapped types are already in these dictionaries, essentially making re-initializing a (very expensive) no-op. All we?re doing is a lot of unnecessary extra disk reads and memory allocations and deallocations. The only time this has any effect is when a non-singleton MimeTypes instance is created, as in the read_mime_types function. * And that read_mime_types function is a doozy. It tries to open a filename, spits back None if there?s an IOError (instead of raising the exception as it should), and then creates a new MimeTypes instance (remember, this is identical to the singleton MimeTypes instance because it starts itself from that one?s mappings), adds any new types it finds in the file with that name, and then returns the 'strict' types_map from it. I?m not sure whether any sane user of this API would expect it to return the existing type mappings *plus* the extra ones in the provided filename, but I really can?t imagine this function ever being particularly useful: it requires you are reading mime types in apache format, but not the apache mime type files you already looked at, and then the only way to find out what new mappings were defined is to take the difference of the default mappings with the result of the function. * The code itself, on a line-by-line basis, is unpythonic and unnecessarily verbose, confusing, and slow. The code should be rewritten to use python 2.3?2.6 features: even leaving its functionality identical it could be cut to about half the number of lines, and made clearer. In case the above doesn?t make this clear: this code is extremely confusing. Trying to read it has caused all the people around me to look up as I shout "what the fuck??!" at the screen every few minutes, as each new revelation gives another surprise. I?m not convinced that I completely understand what the code does, because it has been quite effectively obfuscated, but I understand enough to want to throw the whole thing out, and start essentially from scratch. So the question is, what should be done about this? I?d like to hear how people use the mimetypes module, and how they expect it to work, to figure out the sanest possible mostly-backwards-compatible replacement which could be dropped in (ideally this would just allow the use of default mimetypes and rip out the ability to alter the default datastore: or is there some easy way to change this away from a singleton without breaking code which calls these methods?), and then extend that replacement to support a somewhat saner model for anyone who actually wants to extend the set of mappings. My guess is that replacement code could actually fix subtle bugs in existing uses of this module, by people who had a sane expectation of how it was supposed to work. At the very least, the parts about figuring out exactly which exceptions to Apache?s set of default types are useful would be a good idea, and I?d maybe even recommend including an up-to-date copy of Apache?s mime.types file in the Python Standard Library, and then only overriding its definitions for future versions of Apache (and then overriding the combination of both of those with further exceptions deemed useful for python, with comments explaining why each exception), so that we?re not bothering to look up horribly out-of-date types in multiple locations from Apache 1, 1.2, 1.3, etc. I?d also recommend making the API for overriding definitions be the same as the code used to declare the default overrides, because as it is there are three ways do define types: a) in a mime.types formatted file, b) in a python dictionary that gets initialized with a confusing bit of code, and c) through the add_type function. Does anyone else have thoughts about this, or maybe some good (it had better be *really* good) explanations why this code is the way it is? I'd be happy to try to rewrite it, but I think I?d need a bit of help figuring out how to make the rewrite backwards-compatible. Note: someone else has had fun with this module: Cheers, Jacob Rus