From sandro.tosi at gmail.com Wed Jun 1 00:59:08 2011 From: sandro.tosi at gmail.com (Sandro Tosi) Date: Wed, 1 Jun 2011 00:59:08 +0200 Subject: [Python-Dev] Some additions to .hgignore Message-ID: Hi all, following http://docs.python.org/devguide/coverage.html doc you'll end up with several "new" files/dirs in your checkout: - .coverage, used by coveragepy to save its info - coverage/ , the symlink to coveragepy/coverage - htmlcov/ , the dir where the coverage HTML pages are written I think they should be added to .hgignore so that hg st won't show them. I'm writing here since I don't think an issue is needed for such matter, if that's not the case, I apologize. Regards, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From martin at v.loewis.de Wed Jun 1 07:37:09 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 01 Jun 2011 07:37:09 +0200 Subject: [Python-Dev] Sniffing passwords from PyPI using insecure connection In-Reply-To: References: Message-ID: <4DE5D005.5000001@v.loewis.de> > The requested one character change is > - DEFAULT_REPOSITORY = 'http://pypi.python.org/pypi' > + DEFAULT_REPOSITORY = 'https://pypi.python.org/pypi' > > If Tarek (or perhaps Eric) agree that it is appropriate and otherwise > innocuous, then Martin and Barry can decide whether to include in 2.5/2.6. I don't plan any further 2.5 releases, so unless a critical security issue pops up, 2.5.6 will have been the last release. Regards, Martin From tjreedy at udel.edu Wed Jun 1 08:33:53 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 01 Jun 2011 02:33:53 -0400 Subject: [Python-Dev] Sniffing passwords from PyPI using insecure connection In-Reply-To: <4DE5D005.5000001@v.loewis.de> References: <4DE5D005.5000001@v.loewis.de> Message-ID: On 6/1/2011 1:37 AM, "Martin v. L?wis" wrote: >> The requested one character change is >> - DEFAULT_REPOSITORY = 'http://pypi.python.org/pypi' >> + DEFAULT_REPOSITORY = 'https://pypi.python.org/pypi' >> >> If Tarek (or perhaps Eric) agree that it is appropriate and otherwise >> innocuous, then Martin and Barry can decide whether to include in 2.5/2.6. > > I don't plan any further 2.5 releases, so unless a critical security > issue pops up, 2.5.6 will have been the last release. OK. I removed 2.5 from all open issues, closing a few. You could remove 2.5 from the displayed version list so that people cannot add it back or to new issues. -- Terry Jan Reedy From martin at v.loewis.de Wed Jun 1 08:48:04 2011 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 01 Jun 2011 08:48:04 +0200 Subject: [Python-Dev] Extending os.chown() to accept user/group names In-Reply-To: <1306343861.20117.4.camel@marge> References: <1306343861.20117.4.camel@marge> Message-ID: <4DE5E0A4.8050608@v.loewis.de> > Le mercredi 25 mai 2011 ? 18:46 +0200, Charles-Fran?ois Natali a ?crit : >> While we're at it, adding a "recursive" argument to this shutil.chown >> could also be useful. > > I don't like the idea of a recursive flag. I think shutil.chown should aim to mimic chown(1). At least GNU chown has a -R flag (not sure about POSIX chown), and it's useful in practice. Regards, Martin From ezio.melotti at gmail.com Wed Jun 1 12:37:09 2011 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Wed, 01 Jun 2011 13:37:09 +0300 Subject: [Python-Dev] [Python-checkins] cpython: test.support: add requires_mac_ver() function In-Reply-To: References: Message-ID: <4DE61655.5000401@gmail.com> Hi, On 01/06/2011 13.28, victor.stinner wrote: > http://hg.python.org/cpython/rev/35212b113730 > changeset: 70574:35212b113730 > user: Victor Stinner > date: Wed Jun 01 12:28:04 2011 +0200 > summary: > test.support: add requires_mac_ver() function I would expect this to be a decorator, similar to requires_IEEE_754 and requires_zlib. > Add also linux_version() to __all__. For consistency, this should be a decorator too, possibly named requires_linux_ver(). A requires_windows(_ver) sounds like a useful addition too, given the number of tests that are specific to Windows. > files: > Lib/test/support.py | 22 +++++++++++++++++++++- > Lib/test/test_math.py | 7 ++++--- > 2 files changed, 25 insertions(+), 4 deletions(-) > > > diff --git a/Lib/test/support.py b/Lib/test/support.py > --- a/Lib/test/support.py > +++ b/Lib/test/support.py > @@ -37,7 +37,8 @@ > "Error", "TestFailed", "ResourceDenied", "import_module", > "verbose", "use_resources", "max_memuse", "record_original_stdout", > "get_original_stdout", "unload", "unlink", "rmtree", "forget", > - "is_resource_enabled", "requires", "find_unused_port", "bind_port", > + "is_resource_enabled", "requires", "linux_version", "requires_mac_ver", > + "find_unused_port", "bind_port", > "IPV6_ENABLED", "is_jython", "TESTFN", "HOST", "SAVEDCWD", "temp_cwd", > "findfile", "sortdict", "check_syntax_error", "open_urlresource", > "check_warnings", "CleanImport", "EnvironmentVarGuard", "TransientResource", > @@ -299,6 +300,25 @@ > except ValueError: > return 0, 0, 0 > > +def requires_mac_ver(*min_version): > + """Raise SkipTest if the OS is Mac OS X and the OS X version if less than > + min_version. > + > + For example, support.requires_linux_version(10, 5) raises SkipTest if the This should be requires_mac_ver > + version is less than 10.5. > + """ > + if sys.platform != 'darwin': > + return > + version_txt = platform.mac_ver()[0] > + try: > + version = tuple(map(int, version_txt.split('.'))) > + except ValueError: > + return > + if version< min_version: > + min_version_txt = '.'.join(map(str, min_version)) > + raise unittest.SkipTest("Mac OS X %s or higher required, not %s" > + % (min_version_txt, version_txt)) > + > HOST = 'localhost' > > def find_unused_port(family=socket.AF_INET, socktype=socket.SOCK_STREAM): > diff --git a/Lib/test/test_math.py b/Lib/test/test_math.py > --- a/Lib/test/test_math.py > +++ b/Lib/test/test_math.py > @@ -2,6 +2,7 @@ > # XXXX Should not do tests around zero only > > from test.support import run_unittest, verbose, requires_IEEE_754 > +from test import support > import unittest > import math > import os > @@ -669,10 +670,10 @@ > self.assertTrue(math.isnan(math.log2(NAN))) > > @requires_IEEE_754 > - @unittest.skipIf(sys.platform == 'darwin' > - and platform.mac_ver()[0].startswith('10.4.'), > - 'Mac OS X Tiger log2() is not accurate enough') > def testLog2Exact(self): > + # log2() is not accurate enough on Mac OS X Tiger (10.4) > + support.requires_mac_ver(10, 5) > + > # Check that we get exact equality for log2 of powers of 2. > actual = [math.log2(math.ldexp(1.0, n)) for n in range(-1074, 1024)] > expected = [float(n) for n in range(-1074, 1024)] > Best Regards, Ezio Melotti From fuzzyman at voidspace.org.uk Wed Jun 1 12:49:32 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 01 Jun 2011 11:49:32 +0100 Subject: [Python-Dev] Some additions to .hgignore In-Reply-To: References: Message-ID: <4DE6193C.3000708@voidspace.org.uk> On 31/05/2011 23:59, Sandro Tosi wrote: > Hi all, > following http://docs.python.org/devguide/coverage.html doc you'll end > up with several "new" files/dirs in your checkout: > > - .coverage, used by coveragepy to save its info > - coverage/ , the symlink to coveragepy/coverage > - htmlcov/ , the dir where the coverage HTML pages are written > > I think they should be added to .hgignore so that hg st won't show them. > > I'm writing here since I don't think an issue is needed for such > matter, if that's not the case, I apologize. That sounds good to me. An issue certainly wouldn't hurt. All the best, Michael > Regards, -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From barry at python.org Wed Jun 1 13:08:18 2011 From: barry at python.org (Barry Warsaw) Date: Wed, 1 Jun 2011 07:08:18 -0400 Subject: [Python-Dev] Sniffing passwords from PyPI using insecure connection In-Reply-To: References: <4DE5D005.5000001@v.loewis.de> Message-ID: <20110601070818.244edc03@neurotica.wooz.org> On Jun 01, 2011, at 02:33 AM, Terry Reedy wrote: >On 6/1/2011 1:37 AM, "Martin v. L?wis" wrote: >>> The requested one character change is >>> - DEFAULT_REPOSITORY = 'http://pypi.python.org/pypi' >>> + DEFAULT_REPOSITORY = 'https://pypi.python.org/pypi' >>> >>> If Tarek (or perhaps Eric) agree that it is appropriate and otherwise >>> innocuous, then Martin and Barry can decide whether to include in 2.5/2. >6. >> >> I don't plan any further 2.5 releases, so unless a critical security >> issue pops up, 2.5.6 will have been the last release. > >OK. I removed 2.5 from all open issues, closing a few. You could remove 2.5 >from the displayed version list so that people cannot add it back or to new >issues. I followed up on the tracker. I'm +0 on adding this to 2.6, but not until after the 2.6.7 release on Friday. How well has this change been tested? Are there people for whom this could break things? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From lukasz at langa.pl Wed Jun 1 16:35:16 2011 From: lukasz at langa.pl (=?iso-8859-2?Q?=A3ukasz_Langa?=) Date: Wed, 1 Jun 2011 16:35:16 +0200 Subject: [Python-Dev] Extending os.chown() to accept user/group names In-Reply-To: <4DE5E0A4.8050608@v.loewis.de> References: <1306343861.20117.4.camel@marge> <4DE5E0A4.8050608@v.loewis.de> Message-ID: <3DB8721D-E2F6-407E-B141-8C432AB83DA9@langa.pl> Wiadomo?? napisana przez Martin v. L?wis w dniu 2011-06-01, o godz. 08:48: >> Le mercredi 25 mai 2011 ? 18:46 +0200, Charles-Fran?ois Natali a ?crit : >>> While we're at it, adding a "recursive" argument to this shutil.chown >>> could also be useful. >> >> I don't like the idea of a recursive flag. > > I think shutil.chown should aim to mimic chown(1). At least GNU chown > has a -R flag (not sure about POSIX chown), and it's useful in practice. cp(1) and rm(1) also have "-r", still there are `shutil.copytree` and `shutil.rmtree`. I would like to keep that consistency, e.g. to have `shutil.chown` and `shutil.chowntree` as previously discussed by Charles-Fran?ois, Nick and Petri. -- Best regards, ?ukasz Langa Senior Systems Architecture Engineer IT Infrastructure Department Grupa Allegro Sp. z o.o. From merwok at netwok.org Wed Jun 1 19:15:54 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Wed, 01 Jun 2011 19:15:54 +0200 Subject: [Python-Dev] Docs for the packaging module In-Reply-To: References: Message-ID: <4DE673CA.6070503@netwok.org> Hi, Looks like nobody cares enough about the packaging docs :) If there is no feedback, here?s what I propose to do: - Add new Doc/library/packaging* files (library reference for developers of packaging tools) - Add new packaging-based ?Installing Python Projects? to Doc/install, replacing old distutils docs - Add new packaging-based ?Distributing Python Projects? Doc/packaging, replacing old distutils docs (+ link to it from the main HTML page instead of linking to Doc/distutils) For users needing the legacy distutils docs in 3.3, I would move the older distutils Doc/install/index.rst to Doc/distutils/install.rst, and add a link to Doc/distutils in Doc/library/distutils (because the main page would no longer link to Doc/distutils). Regards From benjamin at python.org Wed Jun 1 19:37:35 2011 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 1 Jun 2011 12:37:35 -0500 Subject: [Python-Dev] Docs for the packaging module In-Reply-To: <4DE673CA.6070503@netwok.org> References: <4DE673CA.6070503@netwok.org> Message-ID: 2011/6/1 ?ric Araujo : > Hi, > > Looks like nobody cares enough about the packaging docs :) Perhaps your solutions are perfect already. :) >?If there is > no feedback, here?s what I propose to do: > > - Add new Doc/library/packaging* files (library reference for developers > of packaging tools) > > - Add new packaging-based ?Installing Python Projects? to Doc/install, > replacing old distutils docs > > - Add new packaging-based ?Distributing Python Projects? Doc/packaging, > replacing old distutils docs (+ link to it from the main HTML page > instead of linking to Doc/distutils) > > For users needing the legacy distutils docs in 3.3, I would move the > older distutils Doc/install/index.rst to Doc/distutils/install.rst, and > add a link to Doc/distutils in Doc/library/distutils (because the main > page would no longer link to Doc/distutils). Or we could just delete them and have people look at docs for old versions of Python. -- Regards, Benjamin From merwok at netwok.org Wed Jun 1 19:46:43 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Wed, 01 Jun 2011 19:46:43 +0200 Subject: [Python-Dev] Docs for the packaging module In-Reply-To: References: <4DE673CA.6070503@netwok.org> Message-ID: <4DE67B03.5000008@netwok.org> Hi Benjamin, >> For users needing the legacy distutils docs in 3.3, I would move the >> older distutils Doc/install/index.rst to Doc/distutils/install.rst, and >> add a link to Doc/distutils in Doc/library/distutils (because the main >> page would no longer link to Doc/distutils). > > Or we could just delete them and have people look at docs for old > versions of Python. Given that we haven?t removed the code, I would prefer to keep the doc too. Tarek has given me the greenlight, so I will commit the new docs momentarily. Cheers From ziade.tarek at gmail.com Wed Jun 1 19:58:58 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Wed, 1 Jun 2011 19:58:58 +0200 Subject: [Python-Dev] Docs for the packaging module In-Reply-To: <4DE673CA.6070503@netwok.org> References: <4DE673CA.6070503@netwok.org> Message-ID: I do care :) Looks fine Please push, as we can always change and fix things afterwards in the tip before 3.3 is out. Le 1 juin 2011 19:38, "?ric Araujo" a ?crit : > Hi, > > Looks like nobody cares enough about the packaging docs :) If there is > no feedback, here?s what I propose to do: > > - Add new Doc/library/packaging* files (library reference for developers > of packaging tools) > > - Add new packaging-based ?Installing Python Projects? to Doc/install, > replacing old distutils docs > > - Add new packaging-based ?Distributing Python Projects? Doc/packaging, > replacing old distutils docs (+ link to it from the main HTML page > instead of linking to Doc/distutils) > > For users needing the legacy distutils docs in 3.3, I would move the > older distutils Doc/install/index.rst to Doc/distutils/install.rst, and > add a link to Doc/distutils in Doc/library/distutils (because the main > page would no longer link to Doc/distutils). > > Regards > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Wed Jun 1 19:58:59 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 01 Jun 2011 13:58:59 -0400 Subject: [Python-Dev] Docs for the packaging module In-Reply-To: <4DE673CA.6070503@netwok.org> References: <4DE673CA.6070503@netwok.org> Message-ID: <20110601175859.D425A25059F@webabinitio.net> On Wed, 01 Jun 2011 19:15:54 +0200, wrote: > Looks like nobody cares enough about the packaging docs :) If there is > no feedback, here???s what I propose to do: I think you should go ahead and make your changes, and then we'll be able to see what it really looks like and decide if anything ought to be tweaked. -- R. David Murray http://www.bitdance.com From tdkadich at gmail.com Wed Jun 1 19:54:13 2011 From: tdkadich at gmail.com (Timothy Kadich) Date: Wed, 1 Jun 2011 10:54:13 -0700 Subject: [Python-Dev] Question about Module Loading Message-ID: If a Python program imported a module, say numpy for example, where in the source is this line interpreted and then handled by import.c ? Thank you in advance! -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Wed Jun 1 21:03:54 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 01 Jun 2011 15:03:54 -0400 Subject: [Python-Dev] Question about Module Loading In-Reply-To: References: Message-ID: <20110601190355.7D21225059F@webabinitio.net> On Wed, 01 Jun 2011 10:54:13 -0700, Timothy Kadich wrote: > If a Python program imported a module, say numpy for example, where in the > source is this line interpreted and then handled by import.c ? Your question is not as simple as you think (I think), but I'm guessing you will want to look at Python/ceval.c. -- R. David Murray http://www.bitdance.com From benjamin at python.org Wed Jun 1 21:04:49 2011 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 1 Jun 2011 14:04:49 -0500 Subject: [Python-Dev] Question about Module Loading In-Reply-To: References: Message-ID: 2011/6/1 Timothy Kadich : > If a Python program imported a module, say numpy for example, where in the > source is this line interpreted and then handled by import.c ? Many different files. Start from TARGET(IMPORT_NAME) in ceval.c. -- Regards, Benjamin From tdkadich at gmail.com Thu Jun 2 00:43:08 2011 From: tdkadich at gmail.com (Timothy Kadich) Date: Wed, 1 Jun 2011 15:43:08 -0700 Subject: [Python-Dev] Question about Module Loading In-Reply-To: References: Message-ID: I don't understand what you mean by "TARGET(IMPORT_NAME)". I can't find that string in ceval.c. On 1 June 2011 12:04, Benjamin Peterson wrote: > 2011/6/1 Timothy Kadich : > > If a Python program imported a module, say numpy for example, where in > the > > source is this line interpreted and then handled by import.c ? > > Many different files. Start from TARGET(IMPORT_NAME) in ceval.c. > > > -- > Regards, > Benjamin > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tdkadich at gmail.com Thu Jun 2 00:51:10 2011 From: tdkadich at gmail.com (Timothy Kadich) Date: Wed, 1 Jun 2011 15:51:10 -0700 Subject: [Python-Dev] Question about Module Loading In-Reply-To: References: Message-ID: Nevermind. In 2.6 (what I'm working with) it's "case IMPORT_NAME:" Thank you for letting me know where to start. On 1 June 2011 15:43, Timothy Kadich wrote: > I don't understand what you mean by "TARGET(IMPORT_NAME)". I can't find > that string in ceval.c. > > > On 1 June 2011 12:04, Benjamin Peterson wrote: > >> 2011/6/1 Timothy Kadich : >> > If a Python program imported a module, say numpy for example, where in >> the >> > source is this line interpreted and then handled by import.c ? >> >> Many different files. Start from TARGET(IMPORT_NAME) in ceval.c. >> >> >> -- >> Regards, >> Benjamin >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Jun 2 02:37:52 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 01 Jun 2011 20:37:52 -0400 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Multiple clean-ups to the docs for builtin functions. In-Reply-To: References: Message-ID: <4DE6DB60.3030906@udel.edu> On 6/1/2011 6:50 PM, raymond.hettinger wrote: > - with open("mydata.txt") as fp: > - for line in iter(fp.readline, "STOP"): > process_line(line) As noted on the tracker, this will always loop forever. Even if "STOP" is corrected to "STOP\n", it will still loop forever if the file does not have the magic value. > + with open('mydata.txt') as fp: > + for line in iter(fp.readline, ''): > process_line(line) While I have a big problem with infinite loops, I have a (smaller) problem with useless code. The new loop line line is a roundabout way to spell "for line in fp". While this would have been useful before files were turned into iterables (if there was such a time after iter() was introduced), it is not now. What might be useful and what the example could show is how to stop early if a sentinel is present, while still stopping when the iterable runs out. The following alternate fix to the original does just that. with open("mydata.txt") as fp: for line in iter(fp.__next__, "STOP\n"): process_line(line) A tested a runnable variation with and without the exact sentinal. It generalizes to *any* iteration than one might want to stop early. This still has the objection that the loop could be written as for line in fp: if line == "STOP\n": break process_line(line) which is easily generalized to any stopping conditions. It is hard to think of useful examples of iter(func, sentinal). To be sure of someday stopping, func must someday (either certainly or with probability 1) either raise StopIteration or produce the sentinal (which should therefore be hard to misspell). To be useful, func should not be a method of an iterable (or at least not produce the same objects as the iterable would when iterated.) It also must produce different values, at least sometimes, when called, which means either maintaining internal state or pulling values from another source. Here is a completely different example which meets these criteria. It can actually be run (though not doctested ;-). It uses random.randint to produce 25 random waiting times for a binary process to hit one of the two values. from random import randint for i in range(25): print(sum(iter(lambda:randint(0,1), 0)), end=',') The outer loop could be removed, but it hints at how this could be used for empirical probability studies. --- Terry Jan Reedy From fuzzyman at voidspace.org.uk Thu Jun 2 11:42:56 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 02 Jun 2011 10:42:56 +0100 Subject: [Python-Dev] Windows download links not working for 2.7.2rc1 and 3.1.4rc1 Message-ID: <4DE75B20.8000600@voidspace.org.uk> Hey all, The links to the Windows downloads for 2.7.2rc1 and 3.1.4rc1 are 404. (From the release pages.) http://python.org/ftp/python/3.1.3/python-3.1.4rc1.msi http://python.org/ftp/python/2.7.1/python-2.7.2rc1.msi All the best, Michael Foord -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From ncoghlan at gmail.com Thu Jun 2 15:56:07 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Jun 2011 23:56:07 +1000 Subject: [Python-Dev] [Python-checkins] cpython: The compiler class for EMX was removed In-Reply-To: References: Message-ID: On Thu, Jun 2, 2011 at 3:47 AM, eric.araujo wrote: > http://hg.python.org/cpython/rev/c3f8991cd74d > changeset: ? 70587:c3f8991cd74d > user: ? ? ? ??ric Araujo > date: ? ? ? ?Wed Jun 01 15:20:44 2011 +0200 > summary: > ?The compiler class for EMX was removed This is the kind of checkin comment where the phrasing is a little confusing. It could mean either "was removed by mistake and has now been restored" or else "is no longer supported and has been removed". (Checking the diff resolves the ambiguity in favour of the latter interpretation, but not all diffs are as simple as this one). It's worth trying to make the final state of the source tree explicit in the checkin message (no need to edit the history, just a note for future reference). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From merwok at netwok.org Thu Jun 2 16:47:22 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 02 Jun 2011 16:47:22 +0200 Subject: [Python-Dev] [Python-checkins] cpython: The compiler class for EMX was removed In-Reply-To: References: Message-ID: <4DE7A27A.1010803@netwok.org> Le 02/06/2011 15:56, Nick Coghlan a ?crit : > On Thu, Jun 2, 2011 at 3:47 AM, eric.araujo wrote: >> The compiler class for EMX was removed > This is the kind of checkin comment where the phrasing is a little > confusing. Yep. Next time I?ll be longer and use something like ?Remove obsolete mentions of a compiler that was removed?. From barry at python.org Thu Jun 2 20:19:00 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 2 Jun 2011 14:19:00 -0400 Subject: [Python-Dev] [Python-checkins] peps: Add rules for indenting continuation lines. In-Reply-To: References: Message-ID: <20110602141900.7cd024c0@neurotica.wooz.org> On Jun 02, 2011, at 08:09 PM, guido.van.rossum wrote: >+ Continuation lines should align wrapped elements either vertically using >+ Python's implicit line joining inside parentheses, brackets and braces, >+ or using a hanging indent of double your code indention, in which case >+ there should be no argument on the first line. For example: >+ >+ Yes: # Aligned with opening delimiter >+ foo = long_function_name(var_one, var_two, >+ var_three, var_four) >+ >+ # Double code indention for hanging indent; nothing on first line >+ foo = long_function_name( >+ var_one, var_two, var_three, >+ var_four) >+ >+ No: # Stuff on first line forbidden >+ foo = long_function_name(var_one, var_two, >+ var_three, var_four) >+ >+ # 2-space hanging indent forbidden >+ foo = long_function_name( >+ var_one, var_two, var_three, >+ var_four) As I mentioned to Guido, I'm not sure where the double-indent recommendation comes from, but it's entirely possible I missed that discussion. I agree with the recommendations, but think a single-indentation level looks fine. E.g. return dict( fqdn_listname=member.mailing_list, address=member.address.email, role=role, user=path_to('users/{0}'.format(member.user.user_id)), self_link=path_to('members/{0}'.format(member.member_id)), ) or return b'{0}://{1}:{2}/{3}/{4}'.format( ('https' if as_boolean(config.webservice.use_https) else 'http'), config.webservice.hostname, config.webservice.port, config.webservice.api_version, (resource[1:] if resource.startswith('/') else resource), ) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From v+python at g.nevcal.com Thu Jun 2 20:50:05 2011 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 02 Jun 2011 11:50:05 -0700 Subject: [Python-Dev] [Python-checkins] peps: Add rules for indenting continuation lines. In-Reply-To: <20110602141900.7cd024c0@neurotica.wooz.org> References: <20110602141900.7cd024c0@neurotica.wooz.org> Message-ID: <4DE7DB5D.9080908@g.nevcal.com> On 6/2/2011 11:19 AM, Barry Warsaw wrote: > On Jun 02, 2011, at 08:09 PM, guido.van.rossum wrote: > >> + Continuation lines should align wrapped elements either vertically using >> + Python's implicit line joining inside parentheses, brackets and braces, >> + or using a hanging indent of double your code indention, in which case >> + there should be no argument on the first line. For example: >> + >> + Yes: # Aligned with opening delimiter >> + foo = long_function_name(var_one, var_two, >> + var_three, var_four) >> + >> + # Double code indention for hanging indent; nothing on first line >> + foo = long_function_name( >> + var_one, var_two, var_three, >> + var_four) >> + >> + No: # Stuff on first line forbidden >> + foo = long_function_name(var_one, var_two, >> + var_three, var_four) >> + >> + # 2-space hanging indent forbidden >> + foo = long_function_name( >> + var_one, var_two, var_three, >> + var_four) > > As I mentioned to Guido, I'm not sure where the double-indent recommendation > comes from, but it's entirely possible I missed that discussion. I agree with > the recommendations, but think a single-indentation level looks fine. E.g. > > return dict( > fqdn_listname=member.mailing_list, > address=member.address.email, > role=role, > user=path_to('users/{0}'.format(member.user.user_id)), > self_link=path_to('members/{0}'.format(member.member_id)), > ) > > or > > return b'{0}://{1}:{2}/{3}/{4}'.format( > ('https' if as_boolean(config.webservice.use_https) else 'http'), > config.webservice.hostname, > config.webservice.port, > config.webservice.api_version, > (resource[1:] if resource.startswith('/') else resource), > ) > > -Barry One place a double indent is extremely nice is for lines that initiate a new indentation, but are themselves continued: if some_function( Some, Parameters, To, Pass, ) If_True_Operations() is much more readable than: if some_function( Some, Parameters, To, Pass, ) If_True_Operations() -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jun 2 21:01:04 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 2 Jun 2011 12:01:04 -0700 Subject: [Python-Dev] [Python-checkins] peps: Add rules for indenting continuation lines. In-Reply-To: <4DE7DB5D.9080908@g.nevcal.com> References: <20110602141900.7cd024c0@neurotica.wooz.org> <4DE7DB5D.9080908@g.nevcal.com> Message-ID: Bingo. That's why. (Though you are missing some colons in your examples. :-) --Guido On Thu, Jun 2, 2011 at 11:50 AM, Glenn Linderman wrote: > On 6/2/2011 11:19 AM, Barry Warsaw wrote: > > On Jun 02, 2011, at 08:09 PM, guido.van.rossum wrote: > > + Continuation lines should align wrapped elements either vertically > using > + Python's implicit line joining inside parentheses, brackets and braces, > + or using a hanging indent of double your code indention, in which case > + there should be no argument on the first line. For example: > + > + Yes: # Aligned with opening delimiter > + foo = long_function_name(var_one, var_two, > + var_three, var_four) > + > + # Double code indention for hanging indent; nothing on first line > + foo = long_function_name( > + var_one, var_two, var_three, > + var_four) > + > + No: # Stuff on first line forbidden > + foo = long_function_name(var_one, var_two, > + var_three, var_four) > + > + # 2-space hanging indent forbidden > + foo = long_function_name( > + var_one, var_two, var_three, > + var_four) > > As I mentioned to Guido, I'm not sure where the double-indent recommendation > comes from, but it's entirely possible I missed that discussion. I agree > with > the recommendations, but think a single-indentation level looks fine. E.g. > > return dict( > fqdn_listname=member.mailing_list, > address=member.address.email, > role=role, > user=path_to('users/{0}'.format(member.user.user_id)), > self_link=path_to('members/{0}'.format(member.member_id)), > ) > > or > > return b'{0}://{1}:{2}/{3}/{4}'.format( > ('https' if as_boolean(config.webservice.use_https) else 'http'), > config.webservice.hostname, > config.webservice.port, > config.webservice.api_version, > (resource[1:] if resource.startswith('/') else resource), > ) > > -Barry > > One place a double indent is extremely nice is for lines that initiate a new > indentation, but are themselves continued: > > if some_function( > ??????? Some, > ??????? Parameters, > ??????? To, > ??????? Pass, > ??????? ) > ??? If_True_Operations() > > is much more readable than: > > if some_function( > ??? Some, > ??? Parameters, > ??? To, > ??? Pass, > ??? ) > ??? If_True_Operations() > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (python.org/~guido) From rdmurray at bitdance.com Thu Jun 2 21:07:06 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 02 Jun 2011 15:07:06 -0400 Subject: [Python-Dev] [Python-checkins] peps: Add rules for indenting continuation lines. In-Reply-To: <20110602141900.7cd024c0@neurotica.wooz.org> References: <20110602141900.7cd024c0@neurotica.wooz.org> Message-ID: <20110602190707.39C6425059F@webabinitio.net> On Thu, 02 Jun 2011 14:19:00 -0400, Barry Warsaw wrote: > On Jun 02, 2011, at 08:09 PM, guido.van.rossum wrote: > >+ Yes: # Aligned with opening delimiter > >+ foo = long_function_name(var_one, var_two, > >+ var_three, var_four) > >+ > >+ # Double code indention for hanging indent; nothing on first li= > ne > >+ foo = long_function_name( > >+ var_one, var_two, var_three, > >+ var_four) > >+ > >+ No: # Stuff on first line forbidden > >+ foo = long_function_name(var_one, var_two, > >+ var_three, var_four) > >+ > >+ # 2-space hanging indent forbidden > >+ foo = long_function_name( > >+ var_one, var_two, var_three, > >+ var_four) > > As I mentioned to Guido, I'm not sure where the double-indent recommendation > comes from, but it's entirely possible I missed that discussion. I agree with > the recommendations, but think a single-indentation level looks fine. E.g. Personally, I use "enough" indentation. Sometimes that is a single indentation level, but sometimes it is more. Two spaces is definitely right out, though :) The place where a single indentation level is *not* enough is when the line being indented is the statement starting a suite: for x in long_function_name( var_one, var_two, var_three): print x vs for x in long_function_name( var_one, var_two, var_three): print x That's a case where I'd be likely to use even more than two indentation levels. Usually, though, I try to refactor the statement so it fits on one line. -- R. David Murray http://www.bitdance.com From martin at v.loewis.de Thu Jun 2 21:39:21 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 02 Jun 2011 21:39:21 +0200 Subject: [Python-Dev] Windows download links not working for 2.7.2rc1 and 3.1.4rc1 In-Reply-To: <4DE75B20.8000600@voidspace.org.uk> References: <4DE75B20.8000600@voidspace.org.uk> Message-ID: <4DE7E6E9.2020300@v.loewis.de> > The links to the Windows downloads for 2.7.2rc1 and 3.1.4rc1 are 404. > (From the release pages.) Thanks, fixed. Martin From barry at python.org Thu Jun 2 21:42:30 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 2 Jun 2011 15:42:30 -0400 Subject: [Python-Dev] [Python-checkins] peps: Add rules for indenting continuation lines. In-Reply-To: <20110602190707.39C6425059F@webabinitio.net> References: <20110602141900.7cd024c0@neurotica.wooz.org> <20110602190707.39C6425059F@webabinitio.net> Message-ID: <20110602154230.04087942@limelight.wooz.org> On Jun 02, 2011, at 03:07 PM, R. David Murray wrote: >Personally, I use "enough" indentation. Sometimes that is a single >indentation level, but sometimes it is more. Two spaces is definitely >right out, though :) > >The place where a single indentation level is *not* enough is when the >line being indented is the statement starting a suite: > > for x in long_function_name( > var_one, var_two, var_three): > print x > > vs > > for x in long_function_name( > var_one, var_two, var_three): > print x > >That's a case where I'd be likely to use even more than two indentation >levels. Usually, though, I try to refactor the statement so it fits >on one line. That's a recommendation I can definitely get on board with! I do exactly the same thing, although sometimes I use a comment between the compound statement and the indented stanza. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From v+python at g.nevcal.com Thu Jun 2 21:57:07 2011 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 02 Jun 2011 12:57:07 -0700 Subject: [Python-Dev] [Python-checkins] peps: Add rules for indenting continuation lines. In-Reply-To: References: <20110602141900.7cd024c0@neurotica.wooz.org> <4DE7DB5D.9080908@g.nevcal.com> Message-ID: <4DE7EB13.30600@g.nevcal.com> On 6/2/2011 12:01 PM, Guido van Rossum wrote: > Bingo. That's why. (Though you are missing some colons in your examples.:-) > > --Guido You operate as a good Python compiler :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Jun 3 00:15:58 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 2 Jun 2011 18:15:58 -0400 Subject: [Python-Dev] [Python-checkins] peps: Add rules for indenting continuation lines. In-Reply-To: <4DE7EB13.30600@g.nevcal.com> References: <20110602141900.7cd024c0@neurotica.wooz.org> <4DE7DB5D.9080908@g.nevcal.com> <4DE7EB13.30600@g.nevcal.com> Message-ID: <20110602181558.53cc8616@neurotica.wooz.org> On Jun 02, 2011, at 12:57 PM, Glenn Linderman wrote: >On 6/2/2011 12:01 PM, Guido van Rossum wrote: >> Bingo. That's why. (Though you are missing some colons in your examples.:-) >> >> --Guido > >You operate as a good Python compiler :) Actually, this is a key insight, which I just mentioned in a private response to Steve. How about this for the rule: If the hanging line ends in a colon, then double-indent (or "more-ly indented") seems appropriate. If not, then a single indent is sufficient. That handles cases like this, where double indent makes sense: def some_really_long_function_name( an_argument, another_argument, and_a_third_argument): foo() --- if some_really_long_function_name( an_argument, another_argument, and_a_third_argument): foo() --- and these cases where a single indent is fine: x = some_really_long_function_name( an_argument, another_argument, and_a_third_argument) foo(x) --- d = dict( a=1, b=2, c=3, d=4) return d Does that cover it? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From guido at python.org Fri Jun 3 00:49:08 2011 From: guido at python.org (Guido van Rossum) Date: Thu, 2 Jun 2011 15:49:08 -0700 Subject: [Python-Dev] [Python-checkins] peps: Add rules for indenting continuation lines. In-Reply-To: <20110602181558.53cc8616@neurotica.wooz.org> References: <20110602141900.7cd024c0@neurotica.wooz.org> <4DE7DB5D.9080908@g.nevcal.com> <4DE7EB13.30600@g.nevcal.com> <20110602181558.53cc8616@neurotica.wooz.org> Message-ID: On Thu, Jun 2, 2011 at 3:15 PM, Barry Warsaw wrote: > On Jun 02, 2011, at 12:57 PM, Glenn Linderman wrote: > >>On 6/2/2011 12:01 PM, Guido van Rossum wrote: >>> Bingo. That's why. (Though you are missing some colons in your examples.:-) >>> >>> --Guido >> >>You operate as a good Python compiler :) > > Actually, this is a key insight, which I just mentioned in a private response > to Steve. ?How about this for the rule: > > If the hanging line ends in a colon, then double-indent (or "more-ly > indented") seems appropriate. ?If not, then a single indent is sufficient. > > That handles cases like this, where double indent makes sense: > > def some_really_long_function_name( > ? ? ? ?an_argument, > ? ? ? ?another_argument, > ? ? ? ?and_a_third_argument): > ? ?foo() > > --- > > ? ?if some_really_long_function_name( > ? ? ? ? ? ?an_argument, > ? ? ? ? ? ?another_argument, > ? ? ? ? ? ?and_a_third_argument): > ? ? ? ?foo() > > --- > > and these cases where a single indent is fine: > > ? ?x = some_really_long_function_name( > ? ? ? ?an_argument, > ? ? ? ?another_argument, > ? ? ? ?and_a_third_argument) > ? ?foo(x) > > --- > > ? ?d = dict( > ? ? ? ?a=1, > ? ? ? ?b=2, > ? ? ? ?c=3, > ? ? ? ?d=4) > ? ?return d > > Does that cover it? Except that the rule gets more complicated. I don't think that always using the double indent is going to mean a lot more line breaks, so I don't think there's much benefit to the added complication. -- --Guido van Rossum (python.org/~guido) From v+python at g.nevcal.com Fri Jun 3 01:16:30 2011 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 02 Jun 2011 16:16:30 -0700 Subject: [Python-Dev] [Python-checkins] peps: Add rules for indenting continuation lines. In-Reply-To: References: <20110602141900.7cd024c0@neurotica.wooz.org> <4DE7DB5D.9080908@g.nevcal.com> <4DE7EB13.30600@g.nevcal.com> <20110602181558.53cc8616@neurotica.wooz.org> Message-ID: <4DE819CE.700@g.nevcal.com> On 6/2/2011 3:49 PM, Guido van Rossum wrote: > Except that the rule gets more complicated. I don't think that always > using the double indent is going to mean a lot more line breaks, so I > don't think there's much benefit to the added complication. Further, tools like python-mode would have to go back and fix the indent to be double indented when there are multiple lines, and the : is not typed until the last line... not impossible, but more complicated, and some of the intervening lines might be too long for the double indent, and then manual adjustments would have to happen. Ick. Double indent from the beginning, if there is nothing after the (. Or key the double indent off the leading keywords that end with : Here's a language question, though: if there are keywords that imply the need for a :, and it isn't on the same line, why is \ needed to continue to the next line (or parentheses, etc.)? If the : is truly omitted, like I did in my example, there'll be other syntax errors to report. If it is just on a later line, why complain about it missing? With complex conditions, I wind up adding extra () rather than trailing \, and that is about the only time I have ever found the need to use \ (but my workaround is to add the ()). -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri Jun 3 00:18:53 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 03 Jun 2011 10:18:53 +1200 Subject: [Python-Dev] [Python-checkins] peps: Add rules for indenting continuation lines. In-Reply-To: References: <20110602141900.7cd024c0@neurotica.wooz.org> <4DE7DB5D.9080908@g.nevcal.com> Message-ID: <4DE80C4D.4070808@canterbury.ac.nz> Guido van Rossum wrote: > Bingo. That's why. (Though you are missing some colons in your examples. :-) > > --Guido > > On Thu, Jun 2, 2011 at 11:50 AM, Glenn Linderman wrote: > >>One place a double indent is extremely nice is for lines that initiate a new >>indentation, but are themselves continued: >> >>if some_function( >> Some, >> Parameters, >> To, >> Pass, >> ) >> If_True_Operations() Another way to approach that is if some_function( Some, Parameters, To, Pass, ): If_True_Operations() i.e. indent the *body* one more place. This avoids the jarriness of seeing an outdent that doesn't correspond to the closing of a suite. -- Greg From v+python at g.nevcal.com Fri Jun 3 03:35:42 2011 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 02 Jun 2011 18:35:42 -0700 Subject: [Python-Dev] [Python-checkins] peps: Add rules for indenting continuation lines. In-Reply-To: <4DE80C4D.4070808@canterbury.ac.nz> References: <20110602141900.7cd024c0@neurotica.wooz.org> <4DE7DB5D.9080908@g.nevcal.com> <4DE80C4D.4070808@canterbury.ac.nz> Message-ID: <4DE83A6E.6020709@g.nevcal.com> On 6/2/2011 3:18 PM, Greg Ewing wrote: > i.e. indent the *body* one more place. This avoids the > jarriness of seeing an outdent that doesn't correspond > to the closing of a suite. -1. There are likely many more lines in the suite than in the conditional, that, by being double indented, would now need continuations themselves. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ben+python at benfinney.id.au Fri Jun 3 04:16:37 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Fri, 03 Jun 2011 12:16:37 +1000 Subject: [Python-Dev] [Python-checkins] peps: Add rules for indenting continuation lines. References: <20110602141900.7cd024c0@neurotica.wooz.org> <4DE7DB5D.9080908@g.nevcal.com> <4DE80C4D.4070808@canterbury.ac.nz> Message-ID: <8739jr64my.fsf@benfinney.id.au> Greg Ewing writes: > Another way to approach that is > > if some_function( > Some, > Parameters, > To, > Pass, > ): > If_True_Operations() > > i.e. indent the *body* one more place. This avoids the jarriness of > seeing an outdent that doesn't correspond to the closing of a suite. ?1. Continuation lines for a single statement *should* look different (I disagree with ?jarring?) from a suite of statements. -- \ ?Simplicity and elegance are unpopular because they require | `\ hard work and discipline to achieve and education to be | _o__) appreciated.? ?Edsger W. Dijkstra | Ben Finney From regebro at gmail.com Fri Jun 3 12:09:34 2011 From: regebro at gmail.com (Lennart Regebro) Date: Fri, 3 Jun 2011 12:09:34 +0200 Subject: [Python-Dev] [Python-checkins] peps: Add rules for indenting continuation lines. In-Reply-To: <20110602181558.53cc8616@neurotica.wooz.org> References: <20110602141900.7cd024c0@neurotica.wooz.org> <4DE7DB5D.9080908@g.nevcal.com> <4DE7EB13.30600@g.nevcal.com> <20110602181558.53cc8616@neurotica.wooz.org> Message-ID: On Fri, Jun 3, 2011 at 00:15, Barry Warsaw wrote: > On Jun 02, 2011, at 12:57 PM, Glenn Linderman wrote: > >>On 6/2/2011 12:01 PM, Guido van Rossum wrote: >>> Bingo. That's why. (Though you are missing some colons in your examples.:-) >>> >>> --Guido >> >>You operate as a good Python compiler :) > > Actually, this is a key insight, which I just mentioned in a private response > to Steve. ?How about this for the rule: > > If the hanging line ends in a colon, then double-indent (or "more-ly > indented") seems appropriate. ?If not, then a single indent is sufficient. > > That handles cases like this, where double indent makes sense: > > def some_really_long_function_name( > ? ? ? ?an_argument, > ? ? ? ?another_argument, > ? ? ? ?and_a_third_argument): > ? ?foo() > > --- > > ? ?if some_really_long_function_name( > ? ? ? ? ? ?an_argument, > ? ? ? ? ? ?another_argument, > ? ? ? ? ? ?and_a_third_argument): > ? ? ? ?foo() > > --- > > and these cases where a single indent is fine: > > ? ?x = some_really_long_function_name( > ? ? ? ?an_argument, > ? ? ? ?another_argument, > ? ? ? ?and_a_third_argument) > ? ?foo(x) > > --- > > ? ?d = dict( > ? ? ? ?a=1, > ? ? ? ?b=2, > ? ? ? ?c=3, > ? ? ? ?d=4) > ? ?return d > > Does that cover it? I like it. From status at bugs.python.org Fri Jun 3 18:07:23 2011 From: status at bugs.python.org (Python tracker) Date: Fri, 3 Jun 2011 18:07:23 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20110603160723.721B91CB61@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2011-05-27 - 2011-06-03) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 2815 ( +2) closed 21221 (+56) total 24036 (+58) Open issues with patches: 1224 Issues opened (38) ================== #12197: non-blocking SSL write in Windows sends large data but raises http://bugs.python.org/issue12197 opened by dsiroky #12198: zipfile.py:1047: DeprecationWarning: 'H' format requires 0 <= http://bugs.python.org/issue12198 opened by gnezdo #12200: bdist_wininst install_script not run on uninstall http://bugs.python.org/issue12200 opened by mhammond #12201: Returning FILETIME is unsupported in msilib.SummaryInformation http://bugs.python.org/issue12201 opened by markm #12202: Check status returns in msilib.SummaryInformation.GetProperty( http://bugs.python.org/issue12202 opened by markm #12204: str.upper converts to title http://bugs.python.org/issue12204 opened by py.user #12206: Documentation Std. Library 15.7.5 "LogRecord objects": Paramet http://bugs.python.org/issue12206 opened by mponweiser #12207: Document ast.PyCF_ONLY_AST http://bugs.python.org/issue12207 opened by eric.araujo #12208: Glitches in email.policy docs http://bugs.python.org/issue12208 opened by eric.araujo #12209: Minor edits to faulthandler doc http://bugs.python.org/issue12209 opened by eric.araujo #12210: test_smtplib: intermittent failures on FreeBSD http://bugs.python.org/issue12210 opened by skrah #12211: math.copysign must keep object type. http://bugs.python.org/issue12211 opened by umedoblock #12213: BufferedRandom, BufferedRWPair: issues with interlaced read-wr http://bugs.python.org/issue12213 opened by haypo #12214: platform module can't detect archlinux distribution http://bugs.python.org/issue12214 opened by lilaboc #12215: TextIOWrapper: issues with interlaced read-write http://bugs.python.org/issue12215 opened by haypo #12217: Cross-link docs for faulthandler, traceback and pdb http://bugs.python.org/issue12217 opened by eric.araujo #12219: tkinter.filedialog.askopenfilename XT dialog on Windows 7 http://bugs.python.org/issue12219 opened by hfischer #12220: minidom xmlns not handling spaces in xmlns attribute value fie http://bugs.python.org/issue12220 opened by hfischer #12222: All pysetup commands should respect exit codes http://bugs.python.org/issue12222 opened by tarek #12223: Datamodel documentation page: 'operator' where 'operand' shoul http://bugs.python.org/issue12223 opened by Vladimir.M. #12224: problem with siginterrupt http://bugs.python.org/issue12224 opened by Zhiping.Deng #12226: use secured channel for uploading packages to pypi http://bugs.python.org/issue12226 opened by techtonik #12228: Stat doc - fix swap of UF_OPAQUE and UF_NOUNLINK description http://bugs.python.org/issue12228 opened by sandro.tosi #12231: regrtest: add -k and -K options to filter tests by function/fi http://bugs.python.org/issue12231 opened by haypo #12232: embedded python import cmath http://bugs.python.org/issue12232 opened by Thanh Ly #12234: unittest2 could enable regex debugging for more information http://bugs.python.org/issue12234 opened by iElectric #12237: Document how to open non-default webbrowser http://bugs.python.org/issue12237 opened by hirsh #12238: Readline module loading in interactive mode http://bugs.python.org/issue12238 opened by Niels.Heinen #12239: msilib VT_EMPTY SummaryInformation properties raise an error ( http://bugs.python.org/issue12239 opened by markm #12240: Allow multiple setup_hooks http://bugs.python.org/issue12240 opened by erik.bray #12242: distutils2 environment marker for current compiler http://bugs.python.org/issue12242 opened by eli.collins #12243: getpass.getuser works on OSX http://bugs.python.org/issue12243 opened by louiscipher #12245: Document the meaning of FLT_ROUNDS constants for sys.float_inf http://bugs.python.org/issue12245 opened by rhettinger #12246: create installation path if it's non-existent http://bugs.python.org/issue12246 opened by tshepang #12248: __dir__ semantics changed in Python 2.7.2 http://bugs.python.org/issue12248 opened by barry #12249: add missing command http://bugs.python.org/issue12249 opened by tshepang #12253: Document packaging.manifest and packaging.errors http://bugs.python.org/issue12253 opened by eric.araujo #12254: PEP-3107 has a wrong attribute name for function annotations http://bugs.python.org/issue12254 opened by Roman.Alexeev Most recent 15 issues with no replies (15) ========================================== #12254: PEP-3107 has a wrong attribute name for function annotations http://bugs.python.org/issue12254 #12253: Document packaging.manifest and packaging.errors http://bugs.python.org/issue12253 #12245: Document the meaning of FLT_ROUNDS constants for sys.float_inf http://bugs.python.org/issue12245 #12242: distutils2 environment marker for current compiler http://bugs.python.org/issue12242 #12239: msilib VT_EMPTY SummaryInformation properties raise an error ( http://bugs.python.org/issue12239 #12232: embedded python import cmath http://bugs.python.org/issue12232 #12228: Stat doc - fix swap of UF_OPAQUE and UF_NOUNLINK description http://bugs.python.org/issue12228 #12223: Datamodel documentation page: 'operator' where 'operand' shoul http://bugs.python.org/issue12223 #12220: minidom xmlns not handling spaces in xmlns attribute value fie http://bugs.python.org/issue12220 #12217: Cross-link docs for faulthandler, traceback and pdb http://bugs.python.org/issue12217 #12214: platform module can't detect archlinux distribution http://bugs.python.org/issue12214 #12208: Glitches in email.policy docs http://bugs.python.org/issue12208 #12201: Returning FILETIME is unsupported in msilib.SummaryInformation http://bugs.python.org/issue12201 #12197: non-blocking SSL write in Windows sends large data but raises http://bugs.python.org/issue12197 #12179: Race condition using PyGILState_Ensure on a new thread http://bugs.python.org/issue12179 Most recent 15 issues waiting for review (15) ============================================= #12249: add missing command http://bugs.python.org/issue12249 #12246: create installation path if it's non-existent http://bugs.python.org/issue12246 #12243: getpass.getuser works on OSX http://bugs.python.org/issue12243 #12231: regrtest: add -k and -K options to filter tests by function/fi http://bugs.python.org/issue12231 #12228: Stat doc - fix swap of UF_OPAQUE and UF_NOUNLINK description http://bugs.python.org/issue12228 #12226: use secured channel for uploading packages to pypi http://bugs.python.org/issue12226 #12215: TextIOWrapper: issues with interlaced read-write http://bugs.python.org/issue12215 #12214: platform module can't detect archlinux distribution http://bugs.python.org/issue12214 #12213: BufferedRandom, BufferedRWPair: issues with interlaced read-wr http://bugs.python.org/issue12213 #12210: test_smtplib: intermittent failures on FreeBSD http://bugs.python.org/issue12210 #12209: Minor edits to faulthandler doc http://bugs.python.org/issue12209 #12208: Glitches in email.policy docs http://bugs.python.org/issue12208 #12207: Document ast.PyCF_ONLY_AST http://bugs.python.org/issue12207 #12202: Check status returns in msilib.SummaryInformation.GetProperty( http://bugs.python.org/issue12202 #12201: Returning FILETIME is unsupported in msilib.SummaryInformation http://bugs.python.org/issue12201 Top 10 most discussed issues (10) ================================= #11975: Fix referencing of built-in types (list, int, ...) http://bugs.python.org/issue11975 19 msgs #11416: netrc module does not handle multiple entries for a single hos http://bugs.python.org/issue11416 13 msgs #12226: use secured channel for uploading packages to pypi http://bugs.python.org/issue12226 12 msgs #12185: Decimal documentation lists "first" and "second" arguments, sh http://bugs.python.org/issue12185 10 msgs #11623: Distutils is reporting OSX 10.6 w/ XCode 4 as "universal" http://bugs.python.org/issue11623 9 msgs #11906: test_argparse failure in interactive mode http://bugs.python.org/issue11906 8 msgs #12082: Python/import.c still references fstat even with DONT_HAVE_FST http://bugs.python.org/issue12082 7 msgs #6490: os.popen documentation in 2.6 is probably wrong http://bugs.python.org/issue6490 5 msgs #12191: Add shutil.chown to allow to use user and group name (and not http://bugs.python.org/issue12191 5 msgs #12246: create installation path if it's non-existent http://bugs.python.org/issue12246 5 msgs Issues closed (47) ================== #3265: Python-2.5.2/Modules/_ctypes/malloc_closure.c:70: error: `MAP_ http://bugs.python.org/issue3265 closed by terry.reedy #8578: PyWeakref_GetObject http://bugs.python.org/issue8578 closed by python-dev #8678: crashers in rgbimg http://bugs.python.org/issue8678 closed by terry.reedy #9382: os.popen referenced but not documented in Python 3.x http://bugs.python.org/issue9382 closed by eric.araujo #9738: Document the encoding of functions bytes arguments of the C AP http://bugs.python.org/issue9738 closed by haypo #9812: cPickle segfault with nested dicts in threaded env http://bugs.python.org/issue9812 closed by terry.reedy #10449: ???os.environ was modified by test_httpservers??? http://bugs.python.org/issue10449 closed by eric.araujo #10616: Change PyObject_AsCharBuffer() error message http://bugs.python.org/issue10616 closed by python-dev #11217: python-32 not linked in /usr/local/bin in framework builds http://bugs.python.org/issue11217 closed by ned.deily #11449: tarfile tries to file_.tell() even when creating a new archive http://bugs.python.org/issue11449 closed by strombrg #11612: xml.dom.minidom fail to parse SVG file. http://bugs.python.org/issue11612 closed by ned.deily #11864: sporadic failure in test_concurrent_futures http://bugs.python.org/issue11864 closed by bquinlan #12028: threading._get_ident(): remove it in the doc or make it public http://bugs.python.org/issue12028 closed by python-dev #12057: HZ codec has no test http://bugs.python.org/issue12057 closed by haypo #12085: subprocess.Popen.__del__ raises AttributeError if __init__ was http://bugs.python.org/issue12085 closed by python-dev #12089: regrtest.py doesn't check for unexpected output anymore? http://bugs.python.org/issue12089 closed by python-dev #12101: PEPs should have consecutive revision numbers http://bugs.python.org/issue12101 closed by eric.araujo #12106: reflect syntatic sugar in with ast http://bugs.python.org/issue12106 closed by python-dev #12125: test_sysconfig fails on OpenIndiana because of test_packaging http://bugs.python.org/issue12125 closed by tarek #12151: test_logging fails sometimes http://bugs.python.org/issue12151 closed by vinay.sajip #12171: Reset method of the incremental encoders of CJK codecs calls t http://bugs.python.org/issue12171 closed by python-dev #12175: FileIO.readall() read the file position and size at each read http://bugs.python.org/issue12175 closed by haypo #12190: intern filenames in bytecode http://bugs.python.org/issue12190 closed by benjamin.peterson #12196: add pipe2() to the os module http://bugs.python.org/issue12196 closed by charles-francois.natali #12199: Unify TryExcept and TryFinally http://bugs.python.org/issue12199 closed by python-dev #12203: isinstance not functioning as documented http://bugs.python.org/issue12203 closed by benjamin.peterson #12205: test_subprocess fails due to uninstalled test subdirectory http://bugs.python.org/issue12205 closed by ned.deily #12212: Minor proofreading typo in index.rst http://bugs.python.org/issue12212 closed by benjamin.peterson #12216: future imports change the reporting of syntaxerrors http://bugs.python.org/issue12216 closed by python-dev #12218: Removing wsgiref.egg-info http://bugs.python.org/issue12218 closed by tarek #12221: segfaults with unexpanded $Revision$ id's in release candidate http://bugs.python.org/issue12221 closed by python-dev #12225: current tip doesn't build without mercurial installed http://bugs.python.org/issue12225 closed by r.david.murray #12227: multiprocessing.Pool initialization seems to call methods it s http://bugs.python.org/issue12227 closed by Graham.Cummins #12229: Remove unused variable in bufferedio.c http://bugs.python.org/issue12229 closed by python-dev #12230: test_subprocess.test_pass_fds() failed on x86 Tiger 3.x http://bugs.python.org/issue12230 closed by python-dev #12233: Lists in class objects not independent http://bugs.python.org/issue12233 closed by rhettinger #12235: subprocess throws wrong exception if script can't be executed http://bugs.python.org/issue12235 closed by r.david.murray #12236: Tkinter __version__ uses subversion substitution http://bugs.python.org/issue12236 closed by benjamin.peterson #12241: email.header.Header encoding fails on empty header http://bugs.python.org/issue12241 closed by r.david.murray #12244: cStringIO inconsistencies http://bugs.python.org/issue12244 closed by georg.brandl #12247: Finding subprocess.py depends on how IDLE is opened http://bugs.python.org/issue12247 closed by ned.deily #12250: regrtest: make --timeout explicit http://bugs.python.org/issue12250 closed by python-dev #12251: subprocess(..., stdout=sys.stderr) closes stderr for child http://bugs.python.org/issue12251 closed by charles-francois.natali #12252: '\u' in unicode raw string raise an syntax error http://bugs.python.org/issue12252 closed by amaury.forgeotdarc #1495802: cygwin: popen3 lock up http://bugs.python.org/issue1495802 closed by charles-francois.natali #1189811: pydoc may hide non-private doc strings. http://bugs.python.org/issue1189811 closed by ncoghlan #985064: plistlib crashes too easily on bad files http://bugs.python.org/issue985064 closed by ned.deily From rosslagerwall at gmail.com Fri Jun 3 18:34:01 2011 From: rosslagerwall at gmail.com (Ross Lagerwall) Date: Fri, 03 Jun 2011 18:34:01 +0200 Subject: [Python-Dev] os.stat and nanosecond precision In-Reply-To: <20110526060903.GB7580@colossus> References: <1306343861.20117.4.camel@marge> <20110526060903.GB7580@colossus> Message-ID: <1307118841.1447.6.camel@hobo> With regards to http://bugs.python.org/issue11457 What should the name of the (seconds, nanoseconds) tuple be? st_atim, st_ctim and st_mtim has bee suggested and is what the POSIX specification uses. This is confusingly similar to the existing st_atime, st_ctime and st_mtime. Also, should it be that these attributes are always defined (and just have 0 for nanoseconds if the OS doesn't support it) or should it be that they are only defined if the OS supports nanosecond precision. If they are always defined, it would make usage simpler. Cheers Ross From alexander.belopolsky at gmail.com Fri Jun 3 18:52:03 2011 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 3 Jun 2011 12:52:03 -0400 Subject: [Python-Dev] os.stat and nanosecond precision In-Reply-To: <1307118841.1447.6.camel@hobo> References: <1306343861.20117.4.camel@marge> <20110526060903.GB7580@colossus> <1307118841.1447.6.camel@hobo> Message-ID: On Fri, Jun 3, 2011 at 12:34 PM, Ross Lagerwall wrote: > .. > What should the name of the (seconds, nanoseconds) tuple be? > st_atim, st_ctim and st_mtim has bee suggested and is what the POSIX > specification uses. This is confusingly similar to the existing > st_atime, st_ctime and st_mtime. > Still, inventing new names would make it even more confusing. +1 for POSIX names. > Also, should it be that these attributes are always defined (and just > have 0 for nanoseconds if the OS doesn't support it) or should > it be that they are only defined if the OS supports nanosecond > precision. > If they are always defined, it would make usage simpler. +1 to have them always defined. From raymond.hettinger at gmail.com Fri Jun 3 19:43:14 2011 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 3 Jun 2011 10:43:14 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Fix reST label for collections ABCs. In-Reply-To: References: Message-ID: <1AA56E88-906F-455D-AE11-510E07CCE04A@gmail.com> On Jun 3, 2011, at 10:27 AM, eric.araujo wrote: > > Fix reST label for collections ABCs. > > The previous markup hijacked the abstract-base-classes glossary entry, > which resulted in the HTML linking to collections.abc when defining the > generic ABC concept. Now the glossary links to the abc module. I think the users are better served by links to collections.abc, io.abc, etc. For the most part, glossary readers will be interested in actual abstract classes than in the underlying implementation. IOW, I believe this edit makes the docs worse rather than better. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From merwok at netwok.org Fri Jun 3 19:47:32 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 03 Jun 2011 19:47:32 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Fix reST label for collections ABCs. In-Reply-To: <1AA56E88-906F-455D-AE11-510E07CCE04A@gmail.com> References: <1AA56E88-906F-455D-AE11-510E07CCE04A@gmail.com> Message-ID: <4DE91E34.2070602@netwok.org> Le 03/06/2011 19:43, Raymond Hettinger a ?crit : > I think the users are better served by links to collections.abc, io.abc, etc. The specific problem I addressed was that :ref:`abstract-base-classes` was replaced by ?Collections Abstract Base Classes?, which was wrong: the glossary entry talks about ?Abstract Base Classes?. I will add links to abc submodules as you suggested tomorrow. Thanks for the review. From martin at v.loewis.de Fri Jun 3 22:08:47 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 03 Jun 2011 22:08:47 +0200 Subject: [Python-Dev] os.stat and nanosecond precision In-Reply-To: <1307118841.1447.6.camel@hobo> References: <1306343861.20117.4.camel@marge> <20110526060903.GB7580@colossus> <1307118841.1447.6.camel@hobo> Message-ID: <4DE93F4F.7000004@v.loewis.de> > What should the name of the (seconds, nanoseconds) tuple be? See my comment: -1 on having such a tuple in the first place. We have the decimal type to represent arbitrary-precision time stamps. > st_atim, st_ctim and st_mtim has bee suggested and is what the POSIX > specification uses. This is confusingly similar to the existing > st_atime, st_ctime and st_mtime. That the POSIX spec uses it is IMO a strong argument. > Also, should it be that these attributes are always defined (and just > have 0 for nanoseconds if the OS doesn't support it) Definitely. There may be a day when ps, fs, or as resolution comes along. It would be good if a) the fields are always available, and b) have "unspecified"/"best-effort" precision, to accomodate future changes. I wish the decimal type would have been available in 2.3, when I changed the fields to be doubles instead of longs... Regards, Martin From martin at v.loewis.de Fri Jun 3 23:40:21 2011 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 03 Jun 2011 23:40:21 +0200 Subject: [Python-Dev] Sniffing passwords from PyPI using insecure connection In-Reply-To: <20110601070818.244edc03@neurotica.wooz.org> References: <4DE5D005.5000001@v.loewis.de> <20110601070818.244edc03@neurotica.wooz.org> Message-ID: <4DE954C5.7080609@v.loewis.de> > I followed up on the tracker. I'm +0 on adding this to 2.6, but not until > after the 2.6.7 release on Friday. > > How well has this change been tested? Are there people for whom this could > break things? As others have pointed out: it would break systems that don't have the _ssl module built. Regards, Martin From victor.stinner at haypocalc.com Sat Jun 4 00:06:11 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 4 Jun 2011 00:06:11 +0200 Subject: [Python-Dev] [Python-checkins] cpython: Add NEWS and whatsnew entries for the packaging module In-Reply-To: References: Message-ID: <201106040006.11119.victor.stinner@haypocalc.com> Le vendredi 03 juin 2011 17:28:48, eric.araujo a ?crit : > +packaging > +--------- > + > +:mod:`distutils` has undergone additions and refactoring under a new name, > +:mod:`packaging`, to allow developers to break backward compatibility. > +:mod:`distutils` is still provided in the standard library, but users are > +encouraged to transition to :mod:`packaging`. For older versions of > Python, a +backport compatible with 2.4+ and 3.1+ will be made available > on PyPI under the +name :mod:`distutils2`. Even for Python 2.4, really? Do you really need to support this old Python release? Victor From ziade.tarek at gmail.com Sat Jun 4 00:12:34 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 4 Jun 2011 00:12:34 +0200 Subject: [Python-Dev] Sniffing passwords from PyPI using insecure connection In-Reply-To: <4DE954C5.7080609@v.loewis.de> References: <4DE5D005.5000001@v.loewis.de> <20110601070818.244edc03@neurotica.wooz.org> <4DE954C5.7080609@v.loewis.de> Message-ID: On Fri, Jun 3, 2011 at 11:40 PM, "Martin v. L?wis" wrote: >> I followed up on the tracker. ?I'm +0 on adding this to 2.6, but not until >> after the 2.6.7 release on Friday. >> >> How well has this change been tested? ?Are there people for whom this could >> break things? > > As others have pointed out: it would break systems that don't have the > _ssl module built. yeah, we would need to fallback to http in that case. while using https by default is a nice addition, maybe we should also look at adding a scp-like upload/register command, since the server has now this ability. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com > -- Tarek Ziad? | http://ziade.org From tseaver at palladion.com Sat Jun 4 00:29:02 2011 From: tseaver at palladion.com (Tres Seaver) Date: Fri, 03 Jun 2011 18:29:02 -0400 Subject: [Python-Dev] [Python-checkins] cpython: Add NEWS and whatsnew entries for the packaging module In-Reply-To: <201106040006.11119.victor.stinner@haypocalc.com> References: <201106040006.11119.victor.stinner@haypocalc.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/03/2011 06:06 PM, Victor Stinner wrote: > Le vendredi 03 juin 2011 17:28:48, eric.araujo a ?crit : >> +packaging >> +--------- >> + >> +:mod:`distutils` has undergone additions and refactoring under a new name, >> +:mod:`packaging`, to allow developers to break backward compatibility. >> +:mod:`distutils` is still provided in the standard library, but users are >> +encouraged to transition to :mod:`packaging`. For older versions of >> Python, a +backport compatible with 2.4+ and 3.1+ will be made available >> on PyPI under the +name :mod:`distutils2`. > > Even for Python 2.4, really? Do you really need to support this old Python Yes. Many projects distribute packages to folks still using 2.4. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk3pYC0ACgkQ+gerLs4ltQ638ACghJiU7Ts7D5PrcfA2r1ZoJklW 7zkAn2vJwI1W1N7GVTrpR6/8Lt48ltBz =xKNv -----END PGP SIGNATURE----- From barry at python.org Sat Jun 4 02:18:31 2011 From: barry at python.org (Barry Warsaw) Date: Fri, 3 Jun 2011 20:18:31 -0400 Subject: [Python-Dev] Released: Python 2.6.7 Message-ID: <20110603201831.6f488702@neurotica.wooz.org> Hello Pythoneers and Pythonistas, I'm happy to announce the final release of Python 2.6.7. Python 2.6 is in security-fix only mode. This means that general bug maintenance has ended, and only critical security issues are being fixed. We will support Python 2.6 in security-fix only mode until October 2013. Also, this is a source-only release; no installers for Windows or Mac OS X will be provided. Please download the source from: http://www.python.org/download/releases/2.6.7/ The NEWS file contains the list of changes since 2.6.6: http://www.python.org/download/releases/2.6.7/NEWS.txt Many thanks go out to the entire Python community for their contributions and help in making Python 2.6.7 available. Enjoy, -Barry (on behalf of the Python development community) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From drsalists at gmail.com Sat Jun 4 02:29:49 2011 From: drsalists at gmail.com (Dan Stromberg) Date: Fri, 3 Jun 2011 17:29:49 -0700 Subject: [Python-Dev] [Python-checkins] cpython: Add NEWS and whatsnew entries for the packaging module In-Reply-To: References: <201106040006.11119.victor.stinner@haypocalc.com> Message-ID: On Fri, Jun 3, 2011 at 3:29 PM, Tres Seaver wrote: > > Even for Python 2.4, really? Do you really need to support this old > Python > > > Yes. Many projects distribute packages to folks still using 2.4. > Supporting detail: I recently installed the latest CentOS, 5.6, and found that it still Ships with CPython 2.4. I installed a CPython 3.2 in /usr/local almost immediately, but I can see how some might want 2.4 support yet. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Jun 4 07:54:17 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 4 Jun 2011 15:54:17 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Add NEWS and whatsnew entries for the packaging module In-Reply-To: References: <201106040006.11119.victor.stinner@haypocalc.com> Message-ID: On Sat, Jun 4, 2011 at 10:29 AM, Dan Stromberg wrote: > Supporting detail: I recently installed the latest CentOS, 5.6, and found > that it still Ships with CPython 2.4. > > I installed a CPython 3.2 in /usr/local almost immediately, but I can see > how some might want 2.4 support yet. Yeah, this came up at Brispy last week. Until CentOS 6 comes out and hosting providers based on CentOS make that version available to their customers, quite a few people are going to want 2.4 support. That's likely to take a while, since the last note from the CentOS folks was a month ago [1], and essentially just said "soon". Once it happens, CentOS 6 users will be able to join RHEL6 users on 2.6.2. [1] https://www.centos.org/modules/newbb/viewtopic.php?topic_id=25878&forum=53#12 Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Sat Jun 4 20:50:26 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 4 Jun 2011 20:50:26 +0200 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <4DE2643D.3070906@v.loewis.de> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> Message-ID: <20110604205026.374c0019@pitrou.net> Hello, On Sun, 29 May 2011 17:20:29 +0200 "Martin v. L?wis" wrote: > > I would like to suggest that we remove the socket HOWTO (currently at > > http://docs.python.org/dev/howto/sockets.html) > > -1. I think there should be a Python-oriented introduction to sockets. > You may have complaints about the specific wording of the text, but > please understand that these are probably irrelevant to most > first-time readers of this text. My observation is that people actually > don't read the text that much, but instead try to imitate the examples. So what you're saying is that the text is mostly useless (or at least quite dispensable), but you think it's fine that people waste their time trying to read it? Some of the people reading our docs are not fluent English readers, and it can be quite an effort for them to read some big chunk of text which will be ultimately pointless. > So if the examples are good (and I think they are, mostly), it's of > minor relevance whether the text makes all sense the first time. I'm not sure why the examples are good (for example, modern client code should probably use create_connection() with a host name, not connect()). Also, really, to socket beginners, I think the primary advice should be: first try to find some high-level library that does the dirty work for you (for example some protocol-specific lib on the client side, or something like Twisted or asyncore on the server side). Not "hey, here's how you write a threaded server in 4 lines of code, and wow, look, you can also write non-blocking code using select() too!". > It's not important > to first-time readers to actually understand that, and the wording > explicitly tells them that they don't need to understand. It says > "there is more stuff, and you won't need it, and the stuff you need > is called INET and STREAM". Well... in a couple of months, someone will tell them their code doesn't support IPv6 and they'll be lost. > > what's "select"?) > > It's well explained in the section Non-blocking Sockets, isn't it? I don't think it explains well how a non-blocking socket works. It's very opinionated and has little useful technical content. EAGAIN and EWOULDBLOCK are not even mentioned! > It's a HOWTO - of course it has advise without justification. Well, I think that's bad. When we give advice to users, we should explain the motivation of the advice given. Otherwise we convey the impression that there's some magic that people shouldn't try to understand. > > what is a "nasty death" and how is that supposed to > > happen? couldn't the author have put a 3-line example to demonstrate > > this supposed drawback and how it manifests?). > > It may well be that the author didn't fully understand the problem > when writing the text, so I wouldn't mind removing this specific > paragraph. +1. When reading it I get the idea that the OS might kill sockets in my back, while in reality the only way a EBADF can happen is if I explicitly close the socket - i.e. a programming error on my part. > I'd drop the entire "Performance" section - there is much more > to be said about socket performance than a few paragraphs of text, > and for the target audience, performance is probably no concern. +1 :) Thank you Antoine. From nyamatongwe at gmail.com Sun Jun 5 00:58:00 2011 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Sun, 5 Jun 2011 08:58:00 +1000 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <20110604205026.374c0019@pitrou.net> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> Message-ID: Antoine Pitrou: > So what you're saying is that the text is mostly useless (or at least > quite dispensable), but you think it's fine that people waste their > time trying to read it? I found it useful when starting to write socket code. Later on I learnt more but, as an introduction, this document was great. It is written in an approachable manner and doesn't spend time on details unimportant to initial understanding. Neil From benjamin at python.org Sun Jun 5 03:45:31 2011 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 4 Jun 2011 20:45:31 -0500 Subject: [Python-Dev] keyword-only arguments and varags Message-ID: Currently, def f(*, kw, **kwargs): pass is valid syntax, but def f(*args, *, kw): pass is not. I don't see any mention of it in the PEP but perhaps there is a good reason this isn't allowed. It seems to be perfectly well-defined to me. -- Regards, Benjamin From ncoghlan at gmail.com Sun Jun 5 07:58:09 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 5 Jun 2011 15:58:09 +1000 Subject: [Python-Dev] keyword-only arguments and varags In-Reply-To: References: Message-ID: On Sun, Jun 5, 2011 at 11:45 AM, Benjamin Peterson wrote: > Currently, > > def f(*, kw, **kwargs): pass > > is valid syntax, but > > def f(*args, *, kw): pass > > is not. > > I don't see any mention of it in the PEP but perhaps there is a good > reason this isn't allowed. It seems to be perfectly well-defined to > me. Really? There's two single-stars there. One says "accept arbitrary positional arguments and place them in a tuple named args", the second says "don't accept any more positional args". You can't have it both ways, so the compiler complains. The following works fine to mix keyword-only arguments with arbitrary positional arguments, so I don't see a problem: def f(*args, kw): pass Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From martin at v.loewis.de Sun Jun 5 08:32:38 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 05 Jun 2011 08:32:38 +0200 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <20110604205026.374c0019@pitrou.net> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> Message-ID: <4DEB2306.5080206@v.loewis.de> >> -1. I think there should be a Python-oriented introduction to sockets. >> You may have complaints about the specific wording of the text, but >> please understand that these are probably irrelevant to most >> first-time readers of this text. My observation is that people actually >> don't read the text that much, but instead try to imitate the examples. > > So what you're saying is that the text is mostly useless (or at least > quite dispensable), but you think it's fine that people waste their > time trying to read it? No, that's not what I said. I said the people actually *don't* read the text, so they won't waste time with it. They only glance at the text, enough to understand the examples. > Some of the people reading our docs are not fluent English readers, and > it can be quite an effort for them to read some big chunk of text which > will be ultimately pointless. You completely misunderstood. I didn't say that the reading the text is pointless. I said that people don't read this text, nor any software documentation, in particular when they are not fluent in English. >> So if the examples are good (and I think they are, mostly), it's of >> minor relevance whether the text makes all sense the first time. > > I'm not sure why the examples are good (for example, modern client > code should probably use create_connection() with a host name, not > connect()). I disagree. create_connection is an advanced function - you shouldn't be using it unless you know what it is doing. As a socket tutorial, people do have to know about connect. > Also, really, to socket beginners, I think the primary advice should > be: first try to find some high-level library that does the dirty work > for you (for example some protocol-specific lib on the client side, or > something like Twisted or asyncore on the server side). No no no no no. Absolutely not. a) telling people who want to learn sockets "don't learn sockets, learn some higher-level library" is besides the point. What do you tell them when they did that, and now come back to learn sockets? b) telling people to use Twisted or asyncore on the server side if they are new to sockets is bad advice. People *first* have to understand sockets, and *then* can use these libraries and frameworks. Those libraries aren't made to be black boxes that work even if you don't know how - you *have* to know how they work inside, or else you can't productively use them. > Not "hey, > here's how you write a threaded server in 4 lines of code, and wow, > look, you can also write non-blocking code using select() too!". I'd happily kill the entire non-blocking discussion from the tutorial if it hurts you as much as it hurts me. I wish this non-blocking stuff never went into Python. > Well... in a couple of months, someone will tell them their code doesn't > support IPv6 and they'll be lost. No. In a couple of months, they'll understand sockets much better, so they'll be able to support IPv6 easily. >> It's a HOWTO - of course it has advise without justification. > > Well, I think that's bad. When we give advice to users, we should > explain the motivation of the advice given. Otherwise we convey the > impression that there's some magic that people shouldn't try to > understand. It's not that bad. Please reconsider. People do get a lot of advise in their lives that isn't motivated down to the root cause, and accept advise from authority. Only if they understand what it does, they ask why. Regards, Martin From victor.stinner at haypocalc.com Sun Jun 5 11:38:00 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 5 Jun 2011 11:38:00 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.1): Do not add txt files twice. In-Reply-To: References: Message-ID: <201106051138.00689.victor.stinner@haypocalc.com> I added the "if dir=='cjkencodings':" to msi.py, based on tests for other subdirectories in Lib/tests/. Can you explain me why cjkencodings should not have a special case? The fix should maybe be ported to 3.2, 3.3 and 2.7. Victor Le dimanche 05 juin 2011 11:00:30, martin.v.loewis a ?crit : > http://hg.python.org/cpython/rev/df7b9c5d6ba1 > changeset: 70643:df7b9c5d6ba1 > branch: 3.1 > parent: 70571:0639e630426c > user: Martin v. L?wis > date: Sun Jun 05 10:55:57 2011 +0200 > summary: > Do not add txt files twice. > > files: > Tools/msi/msi.py | 2 -- > 1 files changed, 0 insertions(+), 2 deletions(-) > > > diff --git a/Tools/msi/msi.py b/Tools/msi/msi.py > --- a/Tools/msi/msi.py > +++ b/Tools/msi/msi.py > @@ -1021,8 +1021,6 @@ > lib.add_file("zipdir.zip") > if dir=='tests' and parent.physical=='distutils': > lib.add_file("Setup.sample") > - if dir=='cjkencodings': > - lib.glob("*.txt") > if dir=='decimaltestdata': > lib.glob("*.decTest") > if dir=='output': From solipsis at pitrou.net Sun Jun 5 12:33:13 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 5 Jun 2011 12:33:13 +0200 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <4DEB2306.5080206@v.loewis.de> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> Message-ID: <20110605123313.64a4a720@pitrou.net> On Sun, 05 Jun 2011 08:32:38 +0200 "Martin v. L?wis" wrote: > >> -1. I think there should be a Python-oriented introduction to sockets. > >> You may have complaints about the specific wording of the text, but > >> please understand that these are probably irrelevant to most > >> first-time readers of this text. My observation is that people actually > >> don't read the text that much, but instead try to imitate the examples. > > > > So what you're saying is that the text is mostly useless (or at least > > quite dispensable), but you think it's fine that people waste their > > time trying to read it? > > No, that's not what I said. I said the people actually *don't* read > the text, so they won't waste time with it. They only glance at the > text, enough to understand the examples. I'm sorry, that sounds like a very outlandish argument to make. Did you run a user survey? If people only "glance at the text", then what is the text for? Why not kill the text and rename the page "socket examples" so that there is no misunderstanding and so that we don't waste time trying to maintain (and argue about) it? > > I'm not sure why the examples are good (for example, modern client > > code should probably use create_connection() with a host name, not > > connect()). > > I disagree. create_connection is an advanced function - you shouldn't > be using it unless you know what it is doing. Can you explain? I would certainly use it myself, and I don't understand how it's "advanced". It's simply higher-level. Actually, we've been actually replacing uses of connect() with create_connection() in various parts of the stdlib, so that our client modules get IPv6-compatible. > No no no no no. Absolutely not. > a) telling people who want to learn sockets "don't learn sockets, > learn some higher-level library" is besides the point. What do > you tell them when they did that, and now come back to learn > sockets? You said yourself that the HOWTO doesn't claim to explain sockets, so how can you make such a point now? If people want to learn sockets for real, the HOWTO is hopeless for them. > I'd happily kill the entire non-blocking discussion from the tutorial > if it hurts you as much as it hurts me. +1. Regards Antoine. From aahz at pythoncraft.com Sun Jun 5 17:20:25 2011 From: aahz at pythoncraft.com (Aahz) Date: Sun, 5 Jun 2011 08:20:25 -0700 Subject: [Python-Dev] FWD: gpg keys have problems Message-ID: <20110605152025.GA25276@panix.com> I'm not currently reading python-dev, dunno if this has come up before: ----- Forwarded message from "Michael J. Dinneen" ----- > Date: Sun, 05 Jun 2011 19:33:04 +1200 > From: "Michael J. Dinneen" > To: webmaster at python.org > Subject: gpg keys have problems > Organization: University of Auckland > > >From your python download page you need to update the public keys to not use the faulty MD5 digest algorithm. > (see the link listed below) > > $ gpg --import pubkeys.txt > gpg: key 6A45C816: public key "Anthony Baxter " imported > gpg: WARNING: digest algorithm MD5 is deprecated > gpg: please see http://www.gnupg.org/faq/weak-digest-algos.html for more information > gpg: key ED9D77D5: public key "Barry A. Warsaw " imported > gpg: Total number processed: 2 > gpg: imported: 2 (RSA: 1) > gpg: no ultimately trusted keys found > ----- End forwarded message ----- -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ Looking back over the years, after I learned Python I realized that I never really had enjoyed programming before. From benjamin at python.org Sun Jun 5 18:31:44 2011 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 5 Jun 2011 11:31:44 -0500 Subject: [Python-Dev] keyword-only arguments and varags In-Reply-To: References: Message-ID: 2011/6/5 Nick Coghlan : > On Sun, Jun 5, 2011 at 11:45 AM, Benjamin Peterson wrote: >> Currently, >> >> def f(*, kw, **kwargs): pass >> >> is valid syntax, but >> >> def f(*args, *, kw): pass >> >> is not. >> >> I don't see any mention of it in the PEP but perhaps there is a good >> reason this isn't allowed. It seems to be perfectly well-defined to >> me. > > Really? There's two single-stars there. One says "accept arbitrary > positional arguments and place them in a tuple named args", the second > says "don't accept any more positional args". You can't have it both > ways, so the compiler complains. Thank you. More proof I shouldn't write emails after 22:00 local time. > > The following works fine to mix keyword-only arguments with arbitrary > positional arguments, so I don't see a problem: > > def f(*args, kw): pass Move along, nothing to see here... -- Regards, Benjamin From glyph at twistedmatrix.com Sun Jun 5 22:28:10 2011 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Sun, 5 Jun 2011 13:28:10 -0700 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <4DEB2306.5080206@v.loewis.de> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> Message-ID: <1F2B102E-29F8-4F12-801C-C3EE9170F08B@twistedmatrix.com> On Jun 4, 2011, at 11:32 PM, Martin v. L?wis wrote: > b) telling people to use Twisted or asyncore on the server side > if they are new to sockets is bad advice. People *first* have > to understand sockets, and *then* can use these libraries > and frameworks. Those libraries aren't made to be black boxes > that work even if you don't know how - you *have* to know how > they work inside, or else you can't productively use them. First, Twisted doesn't always use the BSD sockets API; the Windows IOCP reactor, especially, starts off with the socket() function, but things go off in a different direction pretty quickly from there. So it's perfectly fine to introduce yourself to networking via Twisted, and many users have done just that. If you're using it idiomatically, you should never encounter a socket object or file descriptor poking through the API anywhere. Asyncore is different: you do need to know how sockets work in order to use it, because you're expected to call .send() and .recv() yourself. (And, in my opinion, this is a serious design flaw, for reasons which will hopefully be elucidated in the PEP that Laurens is now writing.) Second, it makes me a little sad that it appears to be folk wisdom that Twisted is only for servers. A lot of work has gone into making it equally appropriate for clients. This is especially true if your client has a GUI, where Twisted is often better than a protocol-specific library, which may either be blocking or have its own ad-hoc event loop. I don't have an opinion on the socket HOWTO per se, only on the possibility of linking to Twisted as an alternate implementation mechanism. It really would be better to say "go use Twisted rather than reading any of the following" than "read the following, which will help you understand Twisted". -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Sun Jun 5 23:13:07 2011 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 5 Jun 2011 14:13:07 -0700 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <1F2B102E-29F8-4F12-801C-C3EE9170F08B@twistedmatrix.com> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <1F2B102E-29F8-4F12-801C-C3EE9170F08B@twistedmatrix.com> Message-ID: <857C8A06-E922-428A-BBE3-09FB63640C05@gmail.com> > On Jun 4, 2011, at 11:32 PM, Martin v. L?wis wrote: > >> b) telling people to use Twisted or asyncore on the server side >> if they are new to sockets is bad advice. People *first* have >> to understand sockets, and *then* can use these libraries >> and frameworks. Those libraries aren't made to be black boxes >> that work even if you don't know how - you *have* to know how >> they work inside, or else you can't productively use them. It would be fine to have a separate networking howto guide that covers libraries, frameworks, and high level APIs, but I agree with Martin that the Socket HOWTO needs to cover sockets. That's what a person expects to learn when they click on the Socket HOWTO link. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Mon Jun 6 00:22:11 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 06 Jun 2011 00:22:11 +0200 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <20110605123313.64a4a720@pitrou.net> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <20110605123313.64a4a720@pitrou.net> Message-ID: <4DEC0193.5080208@v.loewis.de> >>> I'm not sure why the examples are good (for example, modern client >>> code should probably use create_connection() with a host name, not >>> connect()). >> >> I disagree. create_connection is an advanced function - you shouldn't >> be using it unless you know what it is doing. > > Can you explain? I would certainly use it myself, and I don't > understand how it's "advanced". It's simply higher-level. It uses getaddrinfo, which might return multiple addresses, which are then tried in sequence. So even though it's called "create_connection", it may actually attempt to create multiple connections. As a consequence, it may wait some time for one connection to complete, and then succeed on a different address. These phenomena can only be understood when you know what it is actually doing. > Actually, we've been actually replacing uses of connect() with > create_connection() in various parts of the stdlib, so that our > client modules get IPv6-compatible. And that's fine - the people making this changes most certainly where capable of using advanced API. >> No no no no no. Absolutely not. >> a) telling people who want to learn sockets "don't learn sockets, >> learn some higher-level library" is besides the point. What do >> you tell them when they did that, and now come back to learn >> sockets? > > You said yourself that the HOWTO doesn't claim to explain sockets Did I say that? If so, I didn't mean to. It explains how to use the socket API. Regards, Martin From martin at v.loewis.de Mon Jun 6 00:35:54 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 06 Jun 2011 00:35:54 +0200 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <1F2B102E-29F8-4F12-801C-C3EE9170F08B@twistedmatrix.com> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <1F2B102E-29F8-4F12-801C-C3EE9170F08B@twistedmatrix.com> Message-ID: <4DEC04CA.1060608@v.loewis.de> > First, Twisted doesn't always use the BSD sockets API; the Windows IOCP > reactor, especially, starts off with the socket() function, but things > go off in a different direction pretty quickly from there. Hmm. Are you saying it doesn't use listen, connect, bind, send, recv? To me, that's the core of BSD sockets. I can understand it doesn't use select(2). > So it's > perfectly fine to introduce yourself to networking via Twisted, and many > users have done just that. If you're using it idiomatically, you should > never encounter a socket object or file descriptor poking through the > API anywhere. And that's all fine. I still claim that you have to *understand* sockets in order to use it properly. By this, I mean stuff like "what is a TCP connection? how is it established?", "how is UDP different from TCP?", "when data arrives, what layers of software does it go through?", "what is a port number?", etc. > Second, it makes me a little sad that it appears to be folk wisdom that > Twisted is only for servers. A lot of work has gone into making it > equally appropriate for clients. This is especially true if your client > has a GUI, where Twisted is often better than a protocol-specific > library, which may either be blocking or have its own ad-hoc event loop. I think that's because many of the problems that Twisted solves don't exist in many of the client applications - in particular, you often don't have many simultaneous connections. GUI apps may be the interesting special case, but I expect that people dealing with these rather use separate threads. > I don't have an opinion on the socket HOWTO per se, only on the > possibility of linking to Twisted as an alternate implementation > mechanism. It really would be better to say "go use Twisted rather than > reading any of the following" than "read the following, which will help > you understand Twisted". Wouldn't you agree that Twisted is very difficult to learn, and thus much heavier than sockets? And I don't blame the Twisted API for that, but rather the mental model of overlapping activities that people have severe problems with. Regards, Martin From exarkun at twistedmatrix.com Mon Jun 6 02:40:35 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 06 Jun 2011 00:40:35 -0000 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <4DEC04CA.1060608@v.loewis.de> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <1F2B102E-29F8-4F12-801C-C3EE9170F08B@twistedmatrix.com> <4DEC04CA.1060608@v.loewis.de> Message-ID: <20110606004035.3983.961114140.divmod.xquotient.546@localhost.localdomain> On 5 Jun, 10:35 pm, martin at v.loewis.de wrote: >>First, Twisted doesn't always use the BSD sockets API; the Windows >>IOCP >>reactor, especially, starts off with the socket() function, but things >>go off in a different direction pretty quickly from there. > >Hmm. Are you saying it doesn't use listen, connect, bind, send, recv? >To me, that's the core of BSD sockets. I can understand it doesn't >use select(2). Yes, that's correct. Those aren't the best APIs to use on Windows, so they aren't necessarily used on Windows. >>So it's >>perfectly fine to introduce yourself to networking via Twisted, and >>many >>users have done just that. If you're using it idiomatically, you >>should >>never encounter a socket object or file descriptor poking through the >>API anywhere. > >And that's all fine. I still claim that you have to *understand* >sockets in order to use it properly. By this, I mean stuff like >"what is a TCP connection? how is it established?", "how is UDP >different from TCP?", "when data arrives, what layers of software >does it go through?", "what is a port number?", etc. These may be good things to understand. The current socket howto doesn't explain them, though. >>Second, it makes me a little sad that it appears to be folk wisdom >>that >>Twisted is only for servers. A lot of work has gone into making it >>equally appropriate for clients. This is especially true if your >>client >>has a GUI, where Twisted is often better than a protocol-specific >>library, which may either be blocking or have its own ad-hoc event >>loop. > >I think that's because many of the problems that Twisted solves don't >exist in many of the client applications - in particular, you often >don't have many simultaneous connections. GUI apps may be the >interesting special case, but I expect that people dealing with these >rather use separate threads. On the contrary, many of the problems do exist in client applications (every time I have to use virt-manager I want to throw it out a window). Some people certainly would rather use threading, but that doesn't say anything about whether Twisted solves problems relevant to clients, only about the fact that a lot of people like to use threads. >>I don't have an opinion on the socket HOWTO per se, only on the >>possibility of linking to Twisted as an alternate implementation >>mechanism. It really would be better to say "go use Twisted rather >>than >>reading any of the following" than "read the following, which will >>help >>you understand Twisted". > >Wouldn't you agree that Twisted is very difficult to learn, and thus >much heavier than sockets? And I don't blame the Twisted API for that, >but rather the mental model of overlapping activities that people have >severe problems with. This discussion has a significant problem, in taking "Twisted" as a monolithic all-or-nothing entity. Restricting the scope to merely the lowest-level socket replacement APIs - ie, the bare TCP, UDP, etc functionality - no, Twisted is not very difficult to learn. Expanding the scope to include the higher level functionality, it is much easier to learn than reimplementing line parsing and concurrency and so forth. However, does that really have anything to do with improving the socket howto? The Python documentation can include a clear explanation of what functionality the socket module provides - without forcing you to read Stevens _or_ use Twisted, but it can still refer you to both, since it is very likely that you'll need at least one of them in addition to the socket module. Jean-Paul From nadeem.vawda at gmail.com Mon Jun 6 08:06:36 2011 From: nadeem.vawda at gmail.com (Nadeem Vawda) Date: Mon, 6 Jun 2011 08:06:36 +0200 Subject: [Python-Dev] Add LRUCache class to stdlib Message-ID: I would like to propose adding a class to the stdlib to provide a more flexible LRU caching mechanism. As things stand, the functools.lru_cache() decorator is fine for memoization of pure functions, but does not accommodate situations where cache entries become stale and must be invalidated/updated (e.g. filecmp.cmp(); cf. issue #11802). I was thinking it would be a good idea to factor out the the replacement code into a separate class that could then be reused by code for which the lru_cache() decorator is not applicable. The implementation of lru_cache() itself would also become considerably simpler. Implementation: class LRUCache: """Least-recently-used cache class. See: http://en.wikipedia.org/wiki/Cache_algorithms#Least_Recently_Used """ def __init__(self, maxsize): """Initialize an LRUCache. If *maxsize* is None, the LRU replacement mechanism is disabled, and the cache can grow without bound. """ if maxsize is None: self.cache = dict() else: self.cache = OrderedDict() self.lock = Lock() self.hits = self.misses = 0 self.maxsize = maxsize def __getitem__(self, key): with self.lock: try: value = self.cache[key] except KeyError: self.misses += 1 raise else: self.hits += 1 if self.maxsize is not None: self.cache.move_to_end(key) return value def __setitem__(self, key, value): with self.lock: self.cache[key] = value if self.maxsize is not None and len(self.cache) > self.maxsize: self.cache.popitem(0) def info(self): """Report cache statistics""" with self.lock: return _CacheInfo(self.hits, self.misses, self.maxsize, len(self.cache)) I'm not sure where this class should go; it would be convenient to just stick it in functools along with lru_cache(), but perhaps the collections package would be more appropriate? Cheers, Nadeem From solipsis at pitrou.net Mon Jun 6 09:17:20 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 6 Jun 2011 09:17:20 +0200 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <4DEC0193.5080208@v.loewis.de> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <20110605123313.64a4a720@pitrou.net> <4DEC0193.5080208@v.loewis.de> Message-ID: <20110606091720.0ddffff1@pitrou.net> On Mon, 06 Jun 2011 00:22:11 +0200 "Martin v. L?wis" wrote: > >>> I'm not sure why the examples are good (for example, modern client > >>> code should probably use create_connection() with a host name, not > >>> connect()). > >> > >> I disagree. create_connection is an advanced function - you shouldn't > >> be using it unless you know what it is doing. > > > > Can you explain? I would certainly use it myself, and I don't > > understand how it's "advanced". It's simply higher-level. > > It uses getaddrinfo, which might return multiple addresses, which > are then tried in sequence. So even though it's called > "create_connection", it may actually attempt to create multiple > connections. As a consequence, it may wait some time for one connection > to complete, and then succeed on a different address. > > These phenomena can only be understood when you know what it is > actually doing. So what? You can say the exact same thing about every API under the sun. Yet the sockets HOWTO *doesn't* explain what the socket APIs are actually doing. From g.brandl at gmx.net Mon Jun 6 09:50:54 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 06 Jun 2011 09:50:54 +0200 Subject: [Python-Dev] cpython: always clear parser error In-Reply-To: References: Message-ID: On 06/06/11 05:13, benjamin.peterson wrote: > http://hg.python.org/cpython/rev/9faa9270457e > changeset: 70654:9faa9270457e > user: Benjamin Peterson > date: Sun Jun 05 22:14:05 2011 -0500 > summary: > always clear parser error > > files: > Modules/parsermodule.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) This looks like a fixup of rev 3ffd8dea77bf; it would be nice to say so in the commit message: the current message is not obvious at all. Georg From stephen at xemacs.org Mon Jun 6 09:52:31 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 6 Jun 2011 16:52:31 +0900 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <20110606004035.3983.961114140.divmod.xquotient.546@localhost.localdomain> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <1F2B102E-29F8-4F12-801C-C3EE9170F08B@twistedmatrix.com> <4DEC04CA.1060608@v.loewis.de> <20110606004035.3983.961114140.divmod.xquotient.546@localhost.localdomain> Message-ID: <19948.34623.477105.907414@uwakimon.sk.tsukuba.ac.jp> exarkun at twistedmatrix.com writes: > However, does that really have anything to do with improving the socket > howto? Thank you! > The Python documentation can include a clear explanation of what > functionality the socket module provides - without forcing you to read > Stevens _or_ use Twisted, but it can still refer you to both, since it > is very likely that you'll need at least one of them in addition to the > socket module. +1 I suggest s/very likely/likely/, though. From stephen at xemacs.org Mon Jun 6 10:01:15 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 6 Jun 2011 17:01:15 +0900 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <20110605123313.64a4a720@pitrou.net> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <20110605123313.64a4a720@pitrou.net> Message-ID: <19948.35147.349129.663487@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > Did you run a user survey? Martin undoubtedly has a lot of experience with users, and it's quite reasonable for him to express his opinions based on that informal sample, yes. The issue here is the difference between existential and universal quantifiers. Martin's arguments are not inconsistent. They simply acknowledge the existence of subsamples of users of the same document with different needs and/or approaches to reading the document. He does not and has never claimed that all of his arguments apply to all of the potential readers. You might question whether the same document should serve both the "cargo cult the examples" group and the "read the fine print" group. That's a valid question, but here my feeling is that the answer is "yes". I very often "cargo cult" my first program, then go back to the fine print and experiment by gradually changing that program to test my understanding of the detailed explanations. It is often easiest to use the same document for both purposes because I already know where it is and the quality of the writing. From solipsis at pitrou.net Mon Jun 6 10:11:21 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 06 Jun 2011 10:11:21 +0200 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <19948.35147.349129.663487@uwakimon.sk.tsukuba.ac.jp> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <20110605123313.64a4a720@pitrou.net> <19948.35147.349129.663487@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1307347881.3506.1.camel@localhost.localdomain> Le lundi 06 juin 2011 ? 17:01 +0900, Stephen J. Turnbull a ?crit : > You might question whether the same document should serve both the > "cargo cult the examples" group and the "read the fine print" group. > That's a valid question, but here my feeling is that the answer is > "yes". So did you read the discussion before posting? The sockets HOWTO *doesn't* serve both groups. Actually, I would argue that it serves neither of them. From martin at v.loewis.de Mon Jun 6 10:23:18 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 06 Jun 2011 10:23:18 +0200 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <19948.35147.349129.663487@uwakimon.sk.tsukuba.ac.jp> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <20110605123313.64a4a720@pitrou.net> <19948.35147.349129.663487@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4DEC8E76.5070109@v.loewis.de> > > Did you run a user survey? > > Martin undoubtedly has a lot of experience with users, and it's quite > reasonable for him to express his opinions based on that informal > sample, yes. In particular, this is collected experience from interaction with students learning Python, or other languages. When they try to solve a problem, they don't read specification-style documentation. Instead they look for examples that they can imitate. [I notice that you (Stephen) also confirmed this from your own experience] > The issue here is the difference between existential and universal > quantifiers. Martin's arguments are not inconsistent. They simply > acknowledge the existence of subsamples of users of the same document > with different needs and/or approaches to reading the document. He > does not and has never claimed that all of his arguments apply to all > of the potential readers. Exactly so. I'd like to settle this discussion based on the anecdotal report of several users on this list that they considered the tutorial useful. > You might question whether the same document should serve both the > "cargo cult the examples" group and the "read the fine print" group. > That's a valid question, but here my feeling is that the answer is > "yes". I very often "cargo cult" my first program, then go back to > the fine print and experiment by gradually changing that program to > test my understanding of the detailed explanations. It is often > easiest to use the same document for both purposes because I already > know where it is and the quality of the writing. In that spirit, I'd be in favor of removing outright errors from the document, and overly subjective and argumentative passages. Other than that, I still think its fine as it stands. Regards, Martin From martin at v.loewis.de Mon Jun 6 10:33:14 2011 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 06 Jun 2011 10:33:14 +0200 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <1307347881.3506.1.camel@localhost.localdomain> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <20110605123313.64a4a720@pitrou.net> <19948.35147.349129.663487@uwakimon.sk.tsukuba.ac.jp> <1307347881.3506.1.camel@localhost.localdomain> Message-ID: <4DEC90CA.8070604@v.loewis.de> Am 06.06.2011 10:11, schrieb Antoine Pitrou: > Le lundi 06 juin 2011 ? 17:01 +0900, Stephen J. Turnbull a ?crit : >> You might question whether the same document should serve both the >> "cargo cult the examples" group and the "read the fine print" group. >> That's a valid question, but here my feeling is that the answer is >> "yes". > > So did you read the discussion before posting? The sockets HOWTO > *doesn't* serve both groups. > Actually, I would argue that it serves neither of them. How can you make such claims when several people have indicated that the howto *actually* helped them? Please let this rest. Regards, Martin From solipsis at pitrou.net Mon Jun 6 12:13:40 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 6 Jun 2011 12:13:40 +0200 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <4DEC90CA.8070604@v.loewis.de> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <20110605123313.64a4a720@pitrou.net> <19948.35147.349129.663487@uwakimon.sk.tsukuba.ac.jp> <1307347881.3506.1.camel@localhost.localdomain> <4DEC90CA.8070604@v.loewis.de> Message-ID: <20110606121340.0f7a3464@pitrou.net> On Mon, 06 Jun 2011 10:33:14 +0200 "Martin v. L?wis" wrote: > Am 06.06.2011 10:11, schrieb Antoine Pitrou: > > Le lundi 06 juin 2011 ? 17:01 +0900, Stephen J. Turnbull a ?crit : > >> You might question whether the same document should serve both the > >> "cargo cult the examples" group and the "read the fine print" group. > >> That's a valid question, but here my feeling is that the answer is > >> "yes". > > > > So did you read the discussion before posting? The sockets HOWTO > > *doesn't* serve both groups. > > Actually, I would argue that it serves neither of them. > > How can you make such claims when several people have indicated that > the howto *actually* helped them? The point here is that the examples in that document are very poor (the only substantial example actually duplicates existing functionality - in a sub-optimal manner - without even mentioning the existence of said functionality), and the technical explanations are nearly non-existent. So I'll happy stand by my claims. The Python documentation isn't meant to host any potentially helpful document, however flawed. We have the Internet for that. Regards Antoine. From victor.stinner at haypocalc.com Mon Jun 6 12:29:46 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 6 Jun 2011 12:29:46 +0200 Subject: [Python-Dev] Buildbots and regrtest timeout Message-ID: <201106061229.46577.victor.stinner@haypocalc.com> Hi, Stephan Krah asked me to change how the default timeout is defined for regrtest (issue #12250): "The implicit timeout in regrtest.py makes it harder to write automated test scripts for 3rd party modules. First, you have to remember to set --timeout=0 for long running tests. Then, you have to remember not to use the --timeout option when compiling --without-threads. I'd much prefer that there's no timeout unless explicitly specified. For the buildbots, I think this could be done in the Makefile." First I replaced the hardcoded constant in regrtest.py by a command line argument (--timeout=3600) in the TESTOPTS variable of the Makefile, so if you call regrtest directly (without make), there is no more default timeout. But today I saw a a buildbot timeout without any traceback: a possible hang in test_io on "x86 FreeBSD 7.2 3.x" buildbot, "command timed out: 3900 seconds without output". I realized that some buildbots (all buildbots?) override the TESTOPTS variable ("make buildbottest TESTOPTS= TESTPYTHONOPTS=" for "x86 FreeBSD 7.2 3.x"). I moved the timeout option from TESTOPTS to a new variable: TESTTIMEOUT, with a default value of 1 hour (3600 seconds). The timeout is now only used for "make buildbottest". Use TESTTIMEOUT=0 to disable completly regrtest timeout. The timeout argument if ignored (with a warning) if faulthandler.dump_tracebacks_later() function is missing (e.g. if Python is compiled without threads). Please, don't disable the TESTTIMEOUT option for your buildbot. You may want to use a shorter or longer timeout, it just have to be smaller than the buildbot timeout (3900 seconds by default, 1 hour 6 minutes, just a little bit more than regrtest timeout: 1 hour). If you want a timeout longer than 1 hour, you can to change TESTTIMEOUT and the buildbot timeout (I don't know how to configure the buildbot timeout). I didn't touch test, testall, testuniversal, quicktest make rules, which don't use the regrtest timeout anymore. I would prefer to use the same timeout options for all test rules in TESTOPTS, but it doesn't work because some buildbots do override TESTOPTS variable. Victor From steve at pearwood.info Mon Jun 6 13:37:53 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 06 Jun 2011 21:37:53 +1000 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <20110606121340.0f7a3464@pitrou.net> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <20110605123313.64a4a720@pitrou.net> <19948.35147.349129.663487@uwakimon.sk.tsukuba.ac.jp> <1307347881.3506.1.camel@localhost.localdomain> <4DEC90CA.8070604@v.loewis.de> <20110606121340.0f7a3464@pitrou.net> Message-ID: <4DECBC11.8060407@pearwood.info> Antoine Pitrou wrote: > On Mon, 06 Jun 2011 10:33:14 +0200 > "Martin v. L?wis" wrote: [...] >> How can you make such claims when several people have indicated that >> the howto *actually* helped them? > > The point here is that the examples in that document are very poor > (the only substantial example actually duplicates existing > functionality - in a sub-optimal manner - without even mentioning the > existence of said functionality), and the technical explanations are > nearly non-existent. So I'll happy stand by my claims. You know, for the amount of discussion about whether or not the doc is worth keeping, we probably could have fixed all the problems with it :) I believe that "status quo wins" is worth applying here. In the absence of evidence that the HOWTO is actively harmful, we should keep it. I'm of two minds whether it should go into the wiki. I would hate for the wiki to become the place where bad docs go to die, but on the other hand putting it in the wiki may encourage lightweight incremental fixes. I think the Socket HOWTO is important enough to fix, not throw out. I also dislike link-rot, and throwing it out causes link-rot. I'd rather see a bunch of concrete bug reports for the HOWTO than just a dismissive "throw it out and start again". > The Python documentation isn't meant to host any potentially helpful > document, however flawed. We have the Internet for that. I think it is unfair to dismiss the document as "potentially" helpful when a number of people have said that it *actually* did help them. -- Steven From p.f.moore at gmail.com Mon Jun 6 13:48:52 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 6 Jun 2011 12:48:52 +0100 Subject: [Python-Dev] Buildbots and regrtest timeout In-Reply-To: <201106061229.46577.victor.stinner@haypocalc.com> References: <201106061229.46577.victor.stinner@haypocalc.com> Message-ID: On 6 June 2011 11:29, Victor Stinner wrote: > Stephan Krah asked me to change how the default timeout is defined for regrtest > (issue #12250): > > "The implicit timeout in regrtest.py makes it harder to write automated > test scripts for 3rd party modules. First, you have to remember to > set --timeout=0 for long running tests. Then, you have to remember > not to use the --timeout option when compiling --without-threads. > > I'd much prefer that there's no timeout unless explicitly specified. > For the buildbots, I think this could be done in the Makefile." > > First I replaced the hardcoded constant in regrtest.py by a command line > argument (--timeout=3600) in the TESTOPTS variable of the Makefile, so if you > call regrtest directly (without make), there is no more default timeout. > > But today I saw a a buildbot timeout without any traceback: a possible hang in > test_io on "x86 FreeBSD 7.2 3.x" buildbot, "command timed out: 3900 seconds > without output". I realized that some buildbots (all buildbots?) override the > TESTOPTS variable ("make buildbottest TESTOPTS= TESTPYTHONOPTS=" for "x86 > FreeBSD 7.2 3.x"). > > I moved the timeout option from TESTOPTS to a new variable: TESTTIMEOUT, with > a default value of 1 hour (3600 seconds). The timeout is now only used for > "make buildbottest". Use TESTTIMEOUT=0 to disable completly regrtest timeout. > The timeout argument if ignored (with a warning) if > faulthandler.dump_tracebacks_later() function is missing (e.g. if Python is > compiled without threads). > > Please, don't disable the TESTTIMEOUT option for your buildbot. You may want > to use a shorter or longer timeout, it just have to be smaller than the > buildbot timeout (3900 seconds by default, 1 hour 6 minutes, just a little bit > more than regrtest timeout: 1 hour). If you want a timeout longer than 1 hour, > you can to change TESTTIMEOUT and the buildbot timeout (I don't know how to > configure the buildbot timeout). > > I didn't touch test, testall, testuniversal, quicktest make rules, which don't > use the regrtest timeout anymore. I would prefer to use the same timeout > options for all test rules in TESTOPTS, but it doesn't work because some > buildbots do override TESTOPTS variable. How does this impact Windows buildbots? As they don't use the makefile, did you add an override to the Windows scripts as well, or will WIndows now use the default of no timeout? Paul. From stephen at xemacs.org Mon Jun 6 15:59:04 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 6 Jun 2011 22:59:04 +0900 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <1307347881.3506.1.camel@localhost.localdomain> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <20110605123313.64a4a720@pitrou.net> <19948.35147.349129.663487@uwakimon.sk.tsukuba.ac.jp> <1307347881.3506.1.camel@localhost.localdomain> Message-ID: <19948.56616.844528.700142@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > So did you read the discussion before posting? Yes. It's rude to assume that those who disagree with you are irresponsible and uninformed. Would you please stop it? > The sockets HOWTO *doesn't* serve both groups. > Actually, I would argue that it serves neither of them. I know that is your opinion, because I've read your posts. I disagree. The document is imperfect, but for me it served a certain purpose. Therefore I stand with the camp that says improving the document is the way to go. From solipsis at pitrou.net Mon Jun 6 16:06:12 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 06 Jun 2011 16:06:12 +0200 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <19948.56616.844528.700142@uwakimon.sk.tsukuba.ac.jp> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <20110605123313.64a4a720@pitrou.net> <19948.35147.349129.663487@uwakimon.sk.tsukuba.ac.jp> <1307347881.3506.1.camel@localhost.localdomain> <19948.56616.844528.700142@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1307369172.3506.2.camel@localhost.localdomain> Le lundi 06 juin 2011 ? 22:59 +0900, Stephen J. Turnbull a ?crit : > Antoine Pitrou writes: > > > So did you read the discussion before posting? > > Yes. It's rude to assume that those who disagree with you are > irresponsible and uninformed. Would you please stop it? > > > The sockets HOWTO *doesn't* serve both groups. > > Actually, I would argue that it serves neither of them. > > I know that is your opinion, because I've read your posts. I > disagree. The document is imperfect, but for me it served a certain > purpose. Therefore I stand with the camp that says improving the > document is the way to go. Don't hesitate to contribute. Regards Antoine. From merwok at netwok.org Mon Jun 6 18:01:37 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Mon, 06 Jun 2011 18:01:37 +0200 Subject: [Python-Dev] packaging landed in stdlib In-Reply-To: References: Message-ID: <4DECF9E1.6080303@netwok.org> Hi, My GSoC student made me realize that it could be quite difficult to work with the packaging module codebase when you?re familiar with the distutils code. I wrote a blog post about file-level changes from distutils to packaging, to help people find their way: https://wokslog.wordpress.com/2011/06/04/distutils-diff/ Regards From kevinjcoyne at hotmail.com Mon Jun 6 19:43:07 2011 From: kevinjcoyne at hotmail.com (kevin coyne) Date: Mon, 6 Jun 2011 13:43:07 -0400 Subject: [Python-Dev] Help requested for Python ISO Standard Message-ID: To whom it may concern: I am working on an ISO Annex of Vulnerabilities for the Python language and am asking for help getting a list of language features that exhibit: unspecified behavior, undefined behavior, or implementation defined behavior. I am also searching for a list of deprecated features. Guido (copied) had suggested I contact you folks for help as we are coming up on an ISO deadline and I have not been successful finding this information. Thanks very much in advance for your help! Regards, Kevin Coyne Cell: 703.901.6774 -------------- next part -------------- An HTML attachment was scrubbed... URL: From db3l.net at gmail.com Mon Jun 6 20:55:28 2011 From: db3l.net at gmail.com (David Bolen) Date: Mon, 06 Jun 2011 14:55:28 -0400 Subject: [Python-Dev] Buildbots and regrtest timeout References: <201106061229.46577.victor.stinner@haypocalc.com> Message-ID: Victor Stinner writes: > But today I saw a a buildbot timeout without any traceback: a possible hang in > test_io on "x86 FreeBSD 7.2 3.x" buildbot, "command timed out: 3900 seconds > without output". I realized that some buildbots (all buildbots?) override the > TESTOPTS variable ("make buildbottest TESTOPTS= TESTPYTHONOPTS=" for "x86 > FreeBSD 7.2 3.x"). (...) > Please, don't disable the TESTTIMEOUT option for your buildbot. You may want > to use a shorter or longer timeout, (...) Just a note, given the phrasing above. None of this is anything that I for example, as a buildbot operator, am actively controlling. That command, including the environment variable overrides, is exactly as provided by the master for a given test run. So I'd think you could adjust it if needed through changes in the master build configuration and probably without having to add an environment variable or Per Paul's follow-up on Windows, buildbot under Windows seems to impose a 1200s idle timeout (just for no output), but I'm not positive how it's calculated. The test process itself has never, I'm pretty sure, specified a timeout to regrtest (via the test.bat, rt.bat, regrtest.py path). (Oh, I guess the --timeout option itself in regrtest is fairly new) So if there's a change in defaults for regrtest that will change Windows behavior implicitly, and I believe at this point the buildbots will be inconsistent, since you're only overriding the regrtest default in a subset of buildbot types. -- David From barry at python.org Mon Jun 6 21:23:14 2011 From: barry at python.org (Barry Warsaw) Date: Mon, 6 Jun 2011 15:23:14 -0400 Subject: [Python-Dev] FWD: gpg keys have problems In-Reply-To: <20110605152025.GA25276@panix.com> References: <20110605152025.GA25276@panix.com> Message-ID: <20110606152314.3130cd6f@neurotica.wooz.org> On Jun 05, 2011, at 08:20 AM, Aahz wrote: >> >From your python download page you need to update the public keys to not >> use the faulty MD5 digest algorithm. (see the link listed below) >> >> $ gpg --import pubkeys.txt >> gpg: key 6A45C816: public key "Anthony Baxter " imported >> gpg: WARNING: digest algorithm MD5 is deprecated >> gpg: please see http://www.gnupg.org/faq/weak-digest-algos.html for more information >> gpg: key ED9D77D5: public key "Barry A. Warsaw " imported >> gpg: Total number processed: 2 >> gpg: imported: 2 (RSA: 1) >> gpg: no ultimately trusted keys found This only looks like a problem with Anthony's key. He could update his key, but OTOH probably has little incentive to just for Python release management. Anthony was release manager for 2.5, but Martin took that over, and also, Python 2.5 is very near EOL even for security releases. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From skip at pobox.com Mon Jun 6 22:36:11 2011 From: skip at pobox.com (skip at pobox.com) Date: Mon, 6 Jun 2011 15:36:11 -0500 Subject: [Python-Dev] Help requested for Python ISO Standard In-Reply-To: References: Message-ID: <19949.14907.982155.4509@montanaro.dyndns.org> kevin> I am working on an ISO Annex of Vulnerabilities for the Python kevin> language and am asking for help getting a list of language kevin> features that exhibit: kevin> unspecified behavior, undefined behavior, or implementation kevin> defined behavior. I am also searching for a list of deprecated kevin> features. One place to search: http://svn.python.org/projects/python/branches/py3k/Lib/test/crashers/ -- Skip Montanaro - skip at pobox.com - http://www.smontanaro.net/ From anurag.chourasia at gmail.com Mon Jun 6 23:23:25 2011 From: anurag.chourasia at gmail.com (Anurag Chourasia) Date: Mon, 6 Jun 2011 17:23:25 -0400 Subject: [Python-Dev] AIX 5.3 - Build of Python 2.7.1 Message-ID: Hi All, Python2.7.1 build on AIX 5.3 ML 12 is failing at the make step with the error indicated below de -I./Include -I /opt/freeware/include -I /opt/freeware/include/readline -I /opt/freeware/include/ncurses -DPy_BUILD_CORE -o Parser/tokenizer_pgen.o Parser/tokenizer_pgen.c gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -I /opt/freeware/include -I /opt/freeware/include/readline -I /opt/freeware/include/ncurses -DPy_BUILD_CORE -o Parser/printgrammar.o Parser/printgrammar.c gcc -pthread -c -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -I /opt/freeware/include -I /opt/freeware/include/readline -I /opt/freeware/include/ncurses -DPy_BUILD_CORE -o Parser/pgenmain.o Parser/pgenmain.c gcc -pthread -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes Parser/acceler.o Parser/grammar1.o Parser/listnode.o Parser/node.o Parser/parser.o Parser/parsetok.o Parser/bitset.o Parser/metagrammar.o Parser/firstsets.o Parser/grammar.o Parser/pgen.o Objects/obmalloc.o Python/mysnprintf.o Python/pyctype.o Parser/tokenizer_pgen.o Parser/printgrammar.o Parser/pgenmain.o -ldl -o Parser/pgen ld: 0711-593 SEVERE ERROR: Symbol C_BSTAT (entry 559) in object Parser/grammar1.o: The symbol refers to a csect with symbol number 0, which was not found. The new symbol cannot be associated with a csect and is being ignored. ld: 0711-593 SEVERE ERROR: Symbol C_BSTAT (entry 579) in object Parser/firstsets.o: The symbol refers to a csect with symbol number 0, which was not found. The new symbol cannot be associated with a csect and is being ignored. collect2: ld returned 12 exit status make: 1254-004 The error code from the last command is 1. Is there anything I could try to get past this? Here is the configuration and make step that i utilized ./configure ?enable-shared --disable-ipv6 --with-gcc=gcc CPPFLAGS="-I /opt/freeware/include -I /opt/freeware/include/readline -I /opt/freeware/include/ncurses" make Regards, Anurag -------------- next part -------------- An HTML attachment was scrubbed... URL: From mr_mcl at live.co.uk Tue Jun 7 00:15:51 2011 From: mr_mcl at live.co.uk (C McL) Date: Mon, 6 Jun 2011 23:15:51 +0100 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <19948.56616.844528.700142@uwakimon.sk.tsukuba.ac.jp> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de>,<20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de>, <20110605123313.64a4a720@pitrou.net>, <19948.35147.349129.663487@uwakimon.sk.tsukuba.ac.jp>, <1307347881.3506.1.camel@localhost.localdomain>, <19948.56616.844528.700142@uwakimon.sk.tsukuba.ac.jp> Message-ID: > Date: Mon, 6 Jun 2011 22:59:04 +0900 > From: stephen at xemacs.org > To: solipsis at pitrou.net > CC: python-dev at python.org > Subject: Re: [Python-Dev] The socket HOWTO > > I know that is your opinion, because I've read your posts. I > disagree. The document is imperfect, but for me it served a certain > purpose. Therefore I stand with the camp that says improving the > document is the way to go. +1. I've been reading the postings on this discussion intently, as I have had experience with the socket HOWTO when I was first learning about sockets. I agree with the view that Martin v. L?wis expressed, that as a beginner I didn't read too much into the text at first because I was more concerned with trying out the examples and getting used to writing the code and such. I would also say that, I wasn't too bothered if the guide never went into too much detail about all the terms it was mentioning, it isn't after all a definitive guide on sockets, but the terms can always be googled later if one so wished. I wholeheartedly disagree with removing it, that would be a real shame and I dislike the idea of moving it to the wiki (I cannot even remember ever visiting the wiki). I may not be a Python Guru but I think my "n00bishness" helps in this particular discussion and I would say I would have to agree to an improvement over the suggested alternatives. Craig -------------- next part -------------- An HTML attachment was scrubbed... URL: From kevinjcoyne at hotmail.com Tue Jun 7 00:14:49 2011 From: kevinjcoyne at hotmail.com (kevin coyne) Date: Mon, 6 Jun 2011 18:14:49 -0400 Subject: [Python-Dev] Help requested for Python ISO Standard In-Reply-To: <19949.14907.982155.4509@montanaro.dyndns.org> References: , <19949.14907.982155.4509@montanaro.dyndns.org> Message-ID: Skip: Thanks, appreciate the link I've checked them all out and some may be useful to my task. Regards, Kevin Coyne Cell: 703.901.6774 > Date: Mon, 6 Jun 2011 15:36:11 -0500 > To: kevinjcoyne at hotmail.com > CC: python-dev at python.org; l.wagone at radium.ncsc.mil; guido at python.org > Subject: Re: [Python-Dev] Help requested for Python ISO Standard > From: skip at pobox.com > > > kevin> I am working on an ISO Annex of Vulnerabilities for the Python > kevin> language and am asking for help getting a list of language > kevin> features that exhibit: > > kevin> unspecified behavior, undefined behavior, or implementation > kevin> defined behavior. I am also searching for a list of deprecated > kevin> features. > > One place to search: > > http://svn.python.org/projects/python/branches/py3k/Lib/test/crashers/ > > -- > Skip Montanaro - skip at pobox.com - http://www.smontanaro.net/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Jun 7 03:50:22 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Jun 2011 11:50:22 +1000 Subject: [Python-Dev] Help requested for Python ISO Standard In-Reply-To: References: <19949.14907.982155.4509@montanaro.dyndns.org> Message-ID: On Tue, Jun 7, 2011 at 8:14 AM, kevin coyne wrote: > Skip: > Thanks, appreciate the link ?I've checked them all out and some may be > useful to my task. Another two scans to try would be to look for "cpython" in the test suite and "impl-detail" in the documentation. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Tue Jun 7 04:47:37 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Jun 2011 12:47:37 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #12040: Expose a new attribute `sentinel` on instances of In-Reply-To: References: Message-ID: On Tue, Jun 7, 2011 at 3:36 AM, antoine.pitrou wrote: > + ? ?@property > + ? ?def sentinel(self): > + ? ? ? ?''' > + ? ? ? ?Return a file descriptor (Unix) or handle (Windows) suitable for > + ? ? ? ?waiting for process termination. > + ? ? ? ?''' > + ? ? ? ?try: > + ? ? ? ? ? ?return self._sentinel > + ? ? ? ?except AttributeError: > + ? ? ? ? ? ?raise ValueError("process not started") > + We should probably follow this model for threading.Thread.ident as well (i.e. throwing an exception rather than returning None if the thread hasn't been started yet). Also, for runtime state errors, we tend to use RuntimeError rather than ValueError (e.g. see the errors thrown by contextlib._GeneratorContextManager) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From g.brandl at gmx.net Tue Jun 7 08:57:10 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 07 Jun 2011 08:57:10 +0200 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: References: Message-ID: On 06/07/11 05:20, brett.cannon wrote: > http://hg.python.org/cpython/rev/fc282e375703 > changeset: 70695:fc282e375703 > user: Brett Cannon > date: Mon Jun 06 20:20:36 2011 -0700 > summary: > Remove some extraneous parentheses and swap the comparison order to > prevent accidental assignment. > > Silences a warning from LLVM/clang 2.9. Swapping the comparison order here seems a bit inconsistent to me. There are lots of others around (e.g. "len == 0" in the patch context below). Why is this one so special? I think that another developer even got told off once for these kinds of comparisons. I hope the Clang warning is only about the parentheses. Georg > files: > Modules/arraymodule.c | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c > --- a/Modules/arraymodule.c > +++ b/Modules/arraymodule.c > @@ -2091,7 +2091,7 @@ > if (len == 0) { > return PyUnicode_FromFormat("array('%c')", (int)typecode); > } > - if ((typecode == 'u')) > + if ('u' == typecode) > v = array_tounicode(a, NULL); > else > v = array_tolist(a, NULL); From solipsis at pitrou.net Tue Jun 7 10:25:01 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Jun 2011 10:25:01 +0200 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to References: Message-ID: <20110607102501.6f112546@pitrou.net> On Tue, 07 Jun 2011 08:57:10 +0200 Georg Brandl wrote: > On 06/07/11 05:20, brett.cannon wrote: > > http://hg.python.org/cpython/rev/fc282e375703 > > changeset: 70695:fc282e375703 > > user: Brett Cannon > > date: Mon Jun 06 20:20:36 2011 -0700 > > summary: > > Remove some extraneous parentheses and swap the comparison order to > > prevent accidental assignment. > > > > Silences a warning from LLVM/clang 2.9. > > Swapping the comparison order here seems a bit inconsistent to me. There are > lots of others around (e.g. "len == 0" in the patch context below). Why is > this one so special? Agreed. Either we do it wholesale (I find these "reversed" comparisons a bit ugly myself) or there's no point doing it on a single occurrence. Regards Antoine. From lukasz at langa.pl Tue Jun 7 10:47:45 2011 From: lukasz at langa.pl (=?iso-8859-2?Q?=A3ukasz_Langa?=) Date: Tue, 7 Jun 2011 10:47:45 +0200 Subject: [Python-Dev] The socket HOWTO In-Reply-To: References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de>, <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de>, <20110605123313.64a4a720@pitrou.net>, <19948.35147.349129.663487@uwakimon.sk.tsukuba.ac.jp>, <1307347881.3506.1.camel@localhost.localdomain>, <19948.56616.844528.700142@uwakimon.sk.tsukuba.ac.jp> Message-ID: Wiadomo?? napisana przez C McL w dniu 2011-06-07, o godz. 00:15: > I cannot even remember ever visiting the wiki. FWIW neither can I. The Wiki link on the front page is below Jobs and Merchandise so it's easy to miss it altogether ;-) -- Best regards, ?ukasz Langa Senior Systems Architecture Engineer IT Infrastructure Department Grupa Allegro Sp. z o.o. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Tue Jun 7 11:03:22 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 07 Jun 2011 11:03:22 +0200 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: References: Message-ID: <4DEDE95A.2010603@egenix.com> Georg Brandl wrote: > On 06/07/11 05:20, brett.cannon wrote: >> http://hg.python.org/cpython/rev/fc282e375703 >> changeset: 70695:fc282e375703 >> user: Brett Cannon >> date: Mon Jun 06 20:20:36 2011 -0700 >> summary: >> Remove some extraneous parentheses and swap the comparison order to >> prevent accidental assignment. >> >> Silences a warning from LLVM/clang 2.9. > > Swapping the comparison order here seems a bit inconsistent to me. There are > lots of others around (e.g. "len == 0" in the patch context below). Why is > this one so special? > > I think that another developer even got told off once for these kinds of > comparisons. > > I hope the Clang warning is only about the parentheses. I agree with Georg: "if ('u' == typecode)" is not well readable, since you usually put the variable part on the left and the constant part on the right of an equal comparison. If clang warns about this, clang needs to be fixed, not our C code :-) > Georg > >> files: >> Modules/arraymodule.c | 2 +- >> 1 files changed, 1 insertions(+), 1 deletions(-) >> >> >> diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c >> --- a/Modules/arraymodule.c >> +++ b/Modules/arraymodule.c >> @@ -2091,7 +2091,7 @@ >> if (len == 0) { >> return PyUnicode_FromFormat("array('%c')", (int)typecode); >> } >> - if ((typecode == 'u')) >> + if ('u' == typecode) >> v = array_tounicode(a, NULL); >> else >> v = array_tolist(a, NULL); > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/mal%40egenix.com -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 07 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-05-23: Released eGenix mx Base 3.2.0 http://python.egenix.com/ 2011-05-25: Released mxODBC 3.1.1 http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 13 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From eliben at gmail.com Tue Jun 7 11:47:13 2011 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 7 Jun 2011 12:47:13 +0300 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: <4DEDE95A.2010603@egenix.com> References: <4DEDE95A.2010603@egenix.com> Message-ID: >> Swapping the comparison order here seems a bit inconsistent to me. There are >> lots of others around (e.g. "len == 0" in the patch context below). Why is >> this one so special? >> >> I think that another developer even got told off once for these kinds of >> comparisons. >> >> I hope the Clang warning is only about the parentheses. > > I agree with Georg: "if ('u' == typecode)" is not well readable, > since you usually put the variable part on the left and the constant > part on the right of an equal comparison. > > If clang warns about this, clang needs to be fixed, not our > C code :-) > +1 Placing the constant first in a comparison is a fundamental style issue. Personally I also don't like doing that, but whatever way is chosen must be consistent. It's definitely wrong to change this in a single place. We have PEP-7 for these things! AFAIK, Clang doesn't produce a warning for this, at least without special static-analysis warning levels. Eli From ncoghlan at gmail.com Tue Jun 7 12:10:10 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Jun 2011 20:10:10 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #12040: Expose a new attribute `sentinel` on instances of In-Reply-To: <20110607102256.1dabc92d@pitrou.net> References: <20110607102256.1dabc92d@pitrou.net> Message-ID: On Tue, Jun 7, 2011 at 6:22 PM, Antoine Pitrou wrote: > On Tue, 7 Jun 2011 12:47:37 +1000 > Nick Coghlan wrote: >> >> We should probably follow this model for threading.Thread.ident as >> well (i.e. throwing an exception rather than returning None if the >> thread hasn't been started yet). > > Well, Process.ident returns None as well ;) Hmm, I guess in that case my preference for keeping the Thread and Process APIs reasonably consistent just edges out my dislike of returning None when a "you shouldn't do that" exception would be more appropriate. >> Also, for runtime state errors, we tend to use RuntimeError rather >> than ValueError (e.g. see the errors thrown by >> contextlib._GeneratorContextManager) > > Well, it depends. For example, closed files raise a ValueError when you > try to do operations on them. I tend to think of RuntimeError as > something that happens without any clear responsibility from the user, > although that's debatable (recursion errors can occur because of a > programming bug, but also simply because a structure is nested much too > deeply). Yeah, I only thought of the closed file counterexample after I had already posted. I guess I just have a slight personal preference for RuntimeError to flag state problems as they're less likely to be caught up in an overly broad try statement in user code (as catching RuntimeError seems to be less common than catching ValueError). > Of course, current coding practice in other multiprocessing.Process > methods and properties doesn't help, since it uses asserts to guard > against misuse! I have a vague recollection of an existing tracker issue complaining about that... or am I just remembering the one that addressed similar problems in threading? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From glyph at twistedmatrix.com Tue Jun 7 05:54:12 2011 From: glyph at twistedmatrix.com (Glyph Lefkowitz) Date: Mon, 6 Jun 2011 20:54:12 -0700 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <4DEC04CA.1060608@v.loewis.de> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <1F2B102E-29F8-4F12-801C-C3EE9170F08B@twistedmatrix.com> <4DEC04CA.1060608@v.loewis.de> Message-ID: <1E59F13B-852E-4DB7-8FB2-D7B067C90920@twistedmatrix.com> On Jun 5, 2011, at 3:35 PM, Martin v. L?wis wrote: > And that's all fine. I still claim that you have to *understand* > sockets in order to use it properly. By this, I mean stuff like > "what is a TCP connection? how is it established?", "how is UDP > different from TCP?", "when data arrives, what layers of software > does it go through?", "what is a port number?", etc. Yes, these are all excellent concepts to be familiar with. But the word "socket" (and the socket HOWTO) refers to a specific way to interface with those concepts, the Berkeley socket API: . Which you don't have to know anything about if you're going to use Twisted. You should know about IPC in general, and TCP/UDP specifically if you're going to use Twisted, but sockets are completely optional. Also, I feel that I should point out that the sockets HOWTO does not cover even a single one of these concepts in any useful depth. If you think that these are what it should be explaining, it needs some heavy editing. Here's what it has to say about each one: > what is a TCP connection? The only place that the characters "TCP" appear in the entire document is in the phrase "... which is completely different from TCP_NODELAY ...". Nowhere is a TCP connection explained at a conceptual level, except to say that it's something a web browser does. > how is UDP different from TCP? The phrase "UDP" never appears in the HOWTO. DGRAM sockets get a brief mention as "anything else" in the sentence: "... you?ll get better behavior and performance from a STREAM socket than anything else ...". (To be fair, I do endorse teaching that "the difference between TCP and UDP is that you should not use UDP" to anyone not sufficiently advanced to read the relevant reference documentation themselves.) > when data arrives, what layers of software does it go through? There's no discussion of this that I can find at all. > what is a port number? Aside from a few comments in the code examples, the only discussion of port numbers is "low number ports are usually reserved for ?well known? services (HTTP, SNMP etc)." It would be very good to have a "Python networking overview" somewhere that explained this stuff at a very high level, and described how data might get into or out of your program, with links to things like the socket HOWTO that describe more specific techniques. This would be useful because most commonly, I think that data will get into Python network programs via WSGI, not direct sockets or anything like Twisted. To be clear, having read it now: I do _not_ agree with Antoine that this document should be deleted. I dimly recall that it helped me understand some things in the very early days of Twisted. While it's far from perfect, it might help someone in a similar situation understand those things as well today. I just found it interesting that the main concepts one would associate with such a HOWTO are nowhere to be found :). -glyph -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Tue Jun 7 19:37:44 2011 From: eliben at gmail.com (Eli Bendersky) Date: Tue, 7 Jun 2011 20:37:44 +0300 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <1E59F13B-852E-4DB7-8FB2-D7B067C90920@twistedmatrix.com> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <1F2B102E-29F8-4F12-801C-C3EE9170F08B@twistedmatrix.com> <4DEC04CA.1060608@v.loewis.de> <1E59F13B-852E-4DB7-8FB2-D7B067C90920@twistedmatrix.com> Message-ID: > Yes, these are all excellent concepts to be familiar with. ?But the word > "socket" (and the socket HOWTO) refers to a specific way to interface with > those concepts, the Berkeley socket API: > . ?Which you don't have to > know anything about if you're going to use Twisted. ?You should know about > IPC in general, and TCP/UDP specifically if you're going to use Twisted, but > sockets are completely optional. > Also, I feel that I should point out that the sockets HOWTO does not cover > even a single one of these concepts in any useful depth. ?If you think that > these are what it should be explaining, it needs some heavy editing. ?Here's > what it has to say about each one: > > what is a TCP connection? > > The only place that the characters "TCP" appear in the entire document is in > the phrase "...?which is completely different from?TCP_NODELAY ...". > ?Nowhere is a TCP connection explained at a conceptual level, except to say > that it's something a web browser does. > > how is UDP?different from TCP? > > The phrase "UDP" never appears in the HOWTO. ?DGRAM sockets get a brief > mention as "anything else" in the sentence: "... you?ll get better behavior > and performance from a STREAM socket than anything else ...". ?(To be fair, > I do endorse teaching that "the difference between TCP and UDP is that you > should not use UDP" to anyone not sufficiently advanced to read the relevant > reference documentation themselves.) > > when data arrives, what layers of software?does it go through? > > There's no discussion of this that I can find at all. > > what is a port number? > > Aside from a few comments in the code examples, the only discussion of port > numbers is "low number ports are usually reserved for ?well known? services > (HTTP,?SNMP etc)." > It would be very good to have a "Python networking overview" somewhere that > explained this stuff at a very high level, and described how data might get > into or out of your program, with links to things like the socket HOWTO that > describe more specific techniques. ?This would be useful because most Just be careful not to reproduce http://www.apress.com/9781590593714 :-) These things tend to get out of hand very quickly. Eli From amauryfa at gmail.com Tue Jun 7 21:43:57 2011 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 7 Jun 2011 21:43:57 +0200 Subject: [Python-Dev] AIX 5.3 - Build of Python 2.7.1 In-Reply-To: References: Message-ID: Hi, 2011/6/6 Anurag Chourasia : > ld: 0711-593 SEVERE ERROR: Symbol C_BSTAT (entry 559) in object > Parser/grammar1.o: A quick Google search reveals a probable issue with gcc on recent AIX systems. http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46072 A lot of users report this issue on various programs. Nothing Python can do about. You could try to disable the -g option. -- Amaury Forgeot d'Arc From guido at python.org Tue Jun 7 23:11:41 2011 From: guido at python.org (Guido van Rossum) Date: Tue, 7 Jun 2011 14:11:41 -0700 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: References: <4DEDE95A.2010603@egenix.com> Message-ID: On Tue, Jun 7, 2011 at 2:47 AM, Eli Bendersky wrote: >>> Swapping the comparison order here seems a bit inconsistent to me. There are >>> lots of others around (e.g. "len == 0" in the patch context below). Why is >>> this one so special? >>> >>> I think that another developer even got told off once for these kinds of >>> comparisons. >>> >>> I hope the Clang warning is only about the parentheses. >> >> I agree with Georg: "if ('u' == typecode)" is not well readable, >> since you usually put the variable part on the left and the constant >> part on the right of an equal comparison. >> >> If clang warns about this, clang needs to be fixed, not our >> C code :-) >> > > +1 > > Placing the constant first in a comparison is a fundamental style > issue. Personally I also don't like doing that, but whatever way is > chosen must be consistent. It's definitely wrong to change this in a > single place. We have PEP-7 for these things! Right. I personally really despise putting the constant first. > AFAIK, Clang doesn't produce a warning for this, at least without > special static-analysis warning levels. CLang shouldn't force our hand here. -- --Guido van Rossum (python.org/~guido) From alexander.belopolsky at gmail.com Tue Jun 7 23:42:17 2011 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 7 Jun 2011 17:42:17 -0400 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: References: <4DEDE95A.2010603@egenix.com> Message-ID: .. >>> I agree with Georg: "if ('u' == typecode)" is not well readable, >>> since you usually put the variable part on the left and the constant >>> part on the right of an equal comparison. I appear to be in the minority here, but this particular example does not strike me as egregiously unreadable. To the contrary, by bringing the constant to the front, this form saves me from having to read to the end of the line. The same mental economy appears when constants are brought to the front of chained if-elif cases in Python: if 'a' == typecode: .. elif 'b' == typecode: .. is slightly more readable than the more traditional alternative. Probably because I can mentally ignore the "== typecode" part and see the switch structure more clearly. Either way, I don't see a big issue here and I would keep "len == 0" intact even if I reordered typecode == 'u' as Brett did. The only consistency that I would enforce is to use the same order in the chained if-elif cases, but otherwise this should be left to the discretion of the author. From victor.stinner at haypocalc.com Wed Jun 8 00:23:20 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 8 Jun 2011 00:23:20 +0200 Subject: [Python-Dev] Byte filenames in the posix module on Windows Message-ID: <201106080023.20795.victor.stinner@haypocalc.com> Hi, Last november, we "decided" (right?) to deprecate bytes filenames in the posix module on Windows in Python 3.2 and drop the support in 3.3: see "Removal of Win32 ANSI API" thread on python-dev. Python 3.2 has been released, so we should shift the versions numbers. I would like to take care of this. I propose three steps: 1) Remove the bytes implementation of each function (when the code is not shared with other OSes), decode bytes from MBCS and reuse the Unicode code. 2) Deprecate the bytes code in Python 3.3 3) Drop bytes support in Python 3.4 (I'm only talking about the posix module on Windows) The first step should not change anything for the user, but it will remove a lot of duplicated code. I expect something like removing the half of the code specific to Windows in the posix module. If we decide to keep bytes filenames on Windows, we can stop before the second step. -- One important point is the choice of the error handler: I would like to mimic the ANSI API and so I will use MultiByteToWideChar() with flags=0 (e.g. MBCS codec with ignore error handler, but see also the issue #12281 !). The MBCS codec uses the ANSI code page which can be a multibyte encoding, like ShiftJIS (cp932 with a japanese locale). os.fsdecode(), PyUnicode_DecodeFSDefault() and PyUnicode_FSDecoder() use the strict error handler to decode filenames on Windows. We may also use strict in the posix module. I'm +0 for this because it warns the developer (and user?) that he/she is doing something really bad. -- I would like to simplify posixmodule.c because I saw that it is difficult to patch it to fix bugs (you have to patch two functions, or more, for each fix). A recent example is the os.stat() symlink issue on Windows: http://bugs.python.org/issue12084 Since Windows 2000, filenames are stored as Unicode internally (e.g. VFAT and NTFS use UTF-16), the ANSI API was kept for backward compatibility (for lazy developers!). If you use bytes for filenames on Windows, you may get encode errors because the ANSI code page is a small subset of Unicode. For your information, I have a last -huge- pending patch to only use Unicode in the import machinery, issue #11619 ;-) Using this patch, you can use characters not encodable to the ANSI code page in your module name/path, yeah! But I am not completly convinced that we need this patch... Victor From dmalcolm at redhat.com Tue Jun 7 23:35:29 2011 From: dmalcolm at redhat.com (David Malcolm) Date: Tue, 07 Jun 2011 17:35:29 -0400 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: <4DEDE95A.2010603@egenix.com> References: <4DEDE95A.2010603@egenix.com> Message-ID: <1307482529.16924.331.camel@surprise> On Tue, 2011-06-07 at 11:03 +0200, M.-A. Lemburg wrote: > Georg Brandl wrote: > > On 06/07/11 05:20, brett.cannon wrote: > >> http://hg.python.org/cpython/rev/fc282e375703 > >> changeset: 70695:fc282e375703 > >> user: Brett Cannon > >> date: Mon Jun 06 20:20:36 2011 -0700 > >> summary: > >> Remove some extraneous parentheses and swap the comparison order to > >> prevent accidental assignment. > >> > >> Silences a warning from LLVM/clang 2.9. > > > > Swapping the comparison order here seems a bit inconsistent to me. There are > > lots of others around (e.g. "len == 0" in the patch context below). Why is > > this one so special? > > > > I think that another developer even got told off once for these kinds of > > comparisons. > > > > I hope the Clang warning is only about the parentheses. > > I agree with Georg: "if ('u' == typecode)" is not well readable, > since you usually put the variable part on the left and the constant > part on the right of an equal comparison. [FWIW, I'm one of the reprobates that likes to put the constant on the LHS when I'm coding in C, but I see I'm in the minority here] I know that this style is unpopular, but if it helps, try mentally pronouncing "==" in C as "is the value of". In this example, when I read that line, my mind is thinking: "if 'u' is the value of typecode" After ~12 years of doing this, it comes naturally. I appreciate that this may come across as weird though :) [snip] Hope this is helpful Dave From rdmurray at bitdance.com Wed Jun 8 03:16:32 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 07 Jun 2011 21:16:32 -0400 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: References: <4DEDE95A.2010603@egenix.com> Message-ID: <20110608011632.DB76E25012E@webabinitio.net> On Tue, 07 Jun 2011 17:42:17 -0400, Alexander Belopolsky wrote: > .. > >>> I agree with Georg: "if ('u' == typecode)" is not well readable, > >>> since you usually put the variable part on the left and the constant > >>> part on the right of an equal comparison. > > I appear to be in the minority here, but this particular example does > not strike me as egregiously unreadable. To the contrary, by bringing > the constant to the front, this form saves me from having to read to > the end of the line. The same mental economy appears when constants > are brought to the front of chained if-elif cases in Python: > > if 'a' == typecode: > .. > elif 'b' == typecode: > .. > > is slightly more readable than the more traditional alternative. > Probably because I can mentally ignore the "== typecode" part and see > the switch structure more clearly. I don't do much C coding, so I don't have the right to an opinion on that (FWIW, I find constant-first jarring). But I'd hate to see the above in python code. The fact that you like it because it makes it easier to read as a switch-like statement should instead tell you that it is time to refactor the code. -- R. David Murray http://www.bitdance.com From tjreedy at udel.edu Wed Jun 8 04:43:32 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 07 Jun 2011 22:43:32 -0400 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: <1307482529.16924.331.camel@surprise> References: <4DEDE95A.2010603@egenix.com> <1307482529.16924.331.camel@surprise> Message-ID: On 6/7/2011 5:35 PM, David Malcolm wrote: > I know that this style is unpopular, but if it helps, try mentally > pronouncing "==" in C as "is the value of". > > In this example, when I read that line, my mind is thinking: > > "if 'u' is the value of typecode" > > After ~12 years of doing this, it comes naturally. I appreciate that > this may come across as weird though :) Whereas I read it as 'has the value' (or just 'is' ;=). Not being tempted to reverse in Python is one advantage of not having assignment expressions. -- Terry Jan Reedy From brett at python.org Wed Jun 8 05:07:49 2011 From: brett at python.org (Brett Cannon) Date: Tue, 7 Jun 2011 20:07:49 -0700 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: References: Message-ID: On Mon, Jun 6, 2011 at 23:57, Georg Brandl wrote: > On 06/07/11 05:20, brett.cannon wrote: > > http://hg.python.org/cpython/rev/fc282e375703 > > changeset: 70695:fc282e375703 > > user: Brett Cannon > > date: Mon Jun 06 20:20:36 2011 -0700 > > summary: > > Remove some extraneous parentheses and swap the comparison order to > > prevent accidental assignment. > > > > Silences a warning from LLVM/clang 2.9. > > Swapping the comparison order here seems a bit inconsistent to me. There > are > lots of others around (e.g. "len == 0" in the patch context below). Why is > this one so special? > > Old habit on how to do comparisons in C. Because C treats assignment as an expression it means comparisons can accidentally become an assignment if you accidentally leave out an = sign. By reversing this order it is simply not possible to have that silent bug and instead you would get a compiler error about trying to assign to a constant. I'll revert that part of the change. > I think that another developer even got told off once for these kinds of > comparisons. > > I hope the Clang warning is only about the parentheses. > Yes, Clang only warned about the parentheses. -Brett > > Georg > > > files: > > Modules/arraymodule.c | 2 +- > > 1 files changed, 1 insertions(+), 1 deletions(-) > > > > > > diff --git a/Modules/arraymodule.c b/Modules/arraymodule.c > > --- a/Modules/arraymodule.c > > +++ b/Modules/arraymodule.c > > @@ -2091,7 +2091,7 @@ > > if (len == 0) { > > return PyUnicode_FromFormat("array('%c')", (int)typecode); > > } > > - if ((typecode == 'u')) > > + if ('u' == typecode) > > v = array_tounicode(a, NULL); > > else > > v = array_tolist(a, NULL); > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Jun 8 13:04:48 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 08 Jun 2011 21:04:48 +1000 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: References: <4DEDE95A.2010603@egenix.com> <1307482529.16924.331.camel@surprise> Message-ID: <4DEF5750.3000005@pearwood.info> Terry Reedy wrote: > On 6/7/2011 5:35 PM, David Malcolm wrote: > >> I know that this style is unpopular, but if it helps, try mentally >> pronouncing "==" in C as "is the value of". >> >> In this example, when I read that line, my mind is thinking: >> >> "if 'u' is the value of typecode" >> >> After ~12 years of doing this, it comes naturally. I appreciate that >> this may come across as weird though :) > > Whereas I read it as 'has the value' (or just 'is' ;=). Am I the only one who reads == as "equals"? -- Steven From dsdale24 at gmail.com Wed Jun 8 17:01:22 2011 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 8 Jun 2011 11:01:22 -0400 Subject: [Python-Dev] Can we improve support for abstract base classes with desciptors Message-ID: I would like to try to address some shortfalls with the way python deals with abstract base classes containing descriptors. I originally was just concerned with improving support for defining abstract properties with the decorator syntax and converting between abstract and concrete properties, but recently realized that the problem extends to descriptors in general. ABCs ---- First, a bit of background may be in order. An abstract base class is defined by specifying its metaclass as ABCMeta (or a subclass thereof):: class MyABC(metaclass=ABCMeta): @abstractmethod def foo(self): pass When trying to instantiate MyABC or any of its subclasses, ABCMeta inspects the current class namespace for items tagged with __isabstractmethod__=True:: class ABCMeta(type): #[...] def __new__(mcls, name, bases, namespace): cls = super().__new__(mcls, name, bases, namespace) # Compute set of abstract method names abstracts = {name for name, value in namespace.items() if getattr(value, "__isabstractmethod__", False)} ABCMeta then checks if any of the base classes define any items tagged with __isabstractmethod__ and whether they remain abstract in the current class namespace:: for base in bases: for name in getattr(base, "__abstractmethods__", set()): value = getattr(cls, name, None) if getattr(value, "__isabstractmethod__", False): abstracts.add(name) cls.__abstractmethods__ = frozenset(abstracts) In Objects/typeobject.c, __abstractmethods__ is actually a descriptor, and setting it gives the type a chance to set an internal flag specifying if it has any abstract methods defined. When object_new is called in typeobject.c, the flag is checked and an error is raised if any abstract methods were identified. Issues with ABCs and descriptors -------------------------------- In order for this scheme to work, ABCMeta needs to identify all of the abstract methods, but there are some limitations when we consider descriptors. For example, Python's property is a composite object, whose behavior is defined by the getter, setter, and deleter methods with which it is composed. Since there is already an @abstractmethod decorator, I would have suspected that defining abstract properties would be intuitive:: class MyABC(metaclass=ABCMeta): @abstractmethod def _get_foo(self): pass @abstractmethod def _set_foo(self, val): pass foo = property(_get_foo, _set_foo) @property @abstractmethod def bar(self): pass @bar.setter @abstractmethod def bar(self, val): pass Ideally, one would want the flexibility of defining a concrete getter and an abstract setter, for example. However, ABCMeta does not inspect the descriptors of a class to see if they contain any abstract methods. It only inspects the descriptor itself for a True __isabstractmethod__ attribute. This places the burdon on every descriptor implementation to provide its own support for ABC compatibility. For example, support for abstract properties was attempted by adding abstractproperty to the abc module. abstractproperty subclasses the property builtin (as opposed to the relationship between every other abstract and concrete class in the python language). Here is the definition of abstractproperty, in its entirety (modulo docstrings):: class abstractproperty(property): __isabstractmethod__ = True A number of problems manifest with this approach, and I think they all can be traced to the fact that the abstractedness of a descriptor is currently not dependent upon the abstractedness of the methods with which it is composed. The documentation for abstractproperty doesn't suggest using @abstractmethod:: class C(metaclass=ABCMeta): def getx(self): ... def setx(self, value): ... x = abstractproperty(getx, setx) which leads to Issue #1: What is abstract about C.x? How does a subclass of C know whether it needs to override the getter or setter? Issue #2: The decorator syntax cannot be used to convert an abstract property into a concrete one. (This relates to Issue #1: how would a descriptor even know when such a conversion would be appropriate?) Running the following code:: from abc import ABCMeta, abstractmethod, abstractproperty class AbstractFoo(metaclass=ABCMeta): @abstractproperty def bar(self): return 1 @bar.setter def bar(self, val): pass class ConcreteFoo(AbstractFoo): @AbstractFoo.bar.getter def bar(self): return 1 @bar.setter def bar(self, val): pass foo = ConcreteFoo() yields:: TypeError: Can't instantiate abstract class ConcreteFoo with abstract methods bar Issue #3: The following class *is* instantiable, even though AbstractFoo declared that a setter for bar is required:: class ConcreteFoo(AbstractFoo): @property def bar(self): pass Previous attempt to improve abc.abstractproperty ------------------------------------------------ It seems to me that the strategy used by abc.abstractproperty is fundamentally ill-advised. I explored the possibility of extending abstractproperty, redefining its getter, setter, and deleter methods such that they would work in conjunction with the @abstractmethod decorator and yield an instance of the builtin property once all abstract methods were replaced with concrete ones (http://bugs.python.org/issue11610). Issues #1 and #2 were addressed, but there were still problems with that approach. It did not address Issue #3, and it also introduced a new issue, #4:: class AbstractFoo(metaclass=ABCMeta): @abstractproperty # bar would be an abstractproperty, even though the getter is concrete def bar(self): return 1 @bar.setter # bar.setter inspected the getter and the new setter, did not identify # any abstract methods, and thus returned an instance of the built-in # property def bar(self, val): pass @bar.deleter # bar is a concrete property, its deleter decorator does not know it # is supposed to check for abstract methods, so it will return an # instance of the built-in property: @abstractmethod def bar(self): pass By the time the deleter was specified, bar was a concrete property, which does not know it should return an instance of abstractproperty (in part because the inheritance diagram for property/abstractproperty is inverted). Thus, AbstractFoo was instantiable, even though it shouldn't be. Finally, issue #5: the current approach taken by ABCMeta and abstractproperty places the burdon on descriptors to identify themselves to ABCMeta as abstract. Considering the issues encountered with abstractproperty, this may be an onerous requirement. There has been a fair amount of discussion at http://bugs.python.org/issue11610 , which can be summarized as a) concerns about maintaining backward compatibility, and b) objections to requiring @abstractmethod to specify that a method being passed to abstractproperty is abstract. Extending ABCMeta: A Promising Way Forward ------------------------------------------ I think the key is to focus on Issue #3. ABCMeta needs to be improved to recognize descriptor objects, both in the current namespace as well as the base classes, and to identify any abstract methods associated with the descriptors. I suggest the following approach in ABCMeta:: def __new__(mcls, name, bases, namespace): cls = super().__new__(mcls, name, bases, namespace) # Compute set of abstract method names def isdescriptor(val): return hasattr(val, '__get__') or hasattr(val, '__set__') \ or hasattr(val, '__delete__') def getabstracts(ns): return [name for name, value in ns.items() if getattr(value, "__isabstractmethod__", False)] abstracts = getabstracts(namespace) for item, val in namespace.items(): # further inspect descriptors for abstract methods: if isdescriptor(val): ## unfortunately, can't import inspect: #from inspect import getmembers #d = dict(getmembers(val)) ## so instead, use the following: d = dict((k, getattr(val, k, None)) for k in dir(val)) for name in getabstracts(d): # add the abstract descriptor methods to the list: abstracts.append('%s.%s'%(item, name)) for base in bases: for name in getattr(base, "__abstractmethods__", set()): if '.' in name: # base class identified a descriptor abstract method: k, v = name.split('.') desc = getattr(cls, k, None) val = getattr(desc, v, None) else: val = getattr(cls, name, None) if val is None or getattr(val, "__isabstractmethod__", False): abstracts.append(name) cls.__abstractmethods__ = frozenset(abstracts) I rolled everything into __new__ just to keep it as simple as possible for the sake of discussion. Python already provides the rest of the framework needed for descriptors to work properly with ABCs. This implementation actually works; I've tested it with an existing python-3.2 install:: from abc import ABCMeta, abstractmethod class AbstractFoo(metaclass=ABCMeta): @property @abstractmethod def bar(self): return 1 @bar.setter @abstractmethod def bar(self, val): pass >>> abstractfoo = AbstractFoo() Traceback (most recent call last): File "temp.py", line 17, in abstractfoo = AbstractFoo() TypeError: Can't instantiate abstract class AbstractFoo with abstract methods bar.fget, bar.fset as expected. Note the more informative error message indicating what about the bar property is abstract. Also:: class ConcreteFoo(AbstractFoo): @AbstractFoo.bar.getter def bar(self): return 1 >>> foo = ConcreteFoo() Traceback (most recent call last): File "temp.py", line 24, in foo = ConcreteFoo() TypeError: Can't instantiate abstract class ConcreteFoo with abstract methods bar.fset So issue #1 is addressed, since we are explicitly specifying which descriptor methods are abstract. Issue #2 has been addressed, since the following class is instantiable:: class ConcreteFoo(AbstractFoo): @AbstractFoo.bar.getter def bar(self): return 1 @bar.setter def bar(self, val): pass Issue #3 is also addressed. In the following example, even though I redefine the bar property as a readonly property, ABCMeta recognizes that a setter is needed:: class ConcreteFoo(AbstractFoo): @property def bar(self): return 1 >>> foo = ConcreteFoo() Traceback (most recent call last): File "temp.py", line 24, in foo = ConcreteFoo() TypeError: Can't instantiate abstract class ConcreteFoo with abstract methods bar.fset Issue #4 (introduced in a previous attempt to solve the problem using abstractproperty) is also addressed:: class AbstractFoo(metaclass=ABCMeta): @property def bar(self): return 1 @bar.setter def bar(self, val): pass @bar.deleter @abstractmethod def bar(self): pass >>> abstractfoo = AbstractFoo() Traceback (most recent call last): File "temp.py", line 15, in abstractfoo = AbstractFoo() TypeError: Can't instantiate abstract class AbstractFoo with abstract methods bar.fdel Finally, this approach addresses Issue #5 by holding ABCMeta responsible for identifying the abstractedness of descriptor methods, rather than placing that burdon on the desciptor objects to identify themselves as abstract. If ABCMeta were extended as above to identify abstract methods associated with descriptors, third parties would simply decorate methods used to compose the descriptors with @abstractmethod. This change to ABCMeta would not effect the behavior of abstractproperty, so backward compatibility would be maintained in that respect. But I think abstractproperty should be deprecated, or at the very least removed from the documentation. The documentation for @abstractmethod in >=python-3.3 should be extended to provide examples with properties/descriptors. The syntax would be backward compatible with older python versions, but with =python3.3. In my opinion, this is a feature: python-3.3 has identified a bug in ConcreteFoo. The developer would not have tagged that method as abstract unless they had intended for consumers of AbstractFoo to provide a concrete implementation. Darren From ncoghlan at gmail.com Wed Jun 8 17:01:48 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Jun 2011 01:01:48 +1000 Subject: [Python-Dev] The socket HOWTO In-Reply-To: References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <1F2B102E-29F8-4F12-801C-C3EE9170F08B@twistedmatrix.com> <4DEC04CA.1060608@v.loewis.de> <1E59F13B-852E-4DB7-8FB2-D7B067C90920@twistedmatrix.com> Message-ID: On Wed, Jun 8, 2011 at 3:37 AM, Eli Bendersky wrote: > Just be careful not to reproduce http://www.apress.com/9781590593714 :-) > These things tend to get out of hand very quickly. At the level Glyph and Martin are talking about, you're more likely to end up with http://authors.phptr.com/tanenbaumcn4/ :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Jun 8 17:06:55 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Jun 2011 01:06:55 +1000 Subject: [Python-Dev] The socket HOWTO In-Reply-To: <1E59F13B-852E-4DB7-8FB2-D7B067C90920@twistedmatrix.com> References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <1F2B102E-29F8-4F12-801C-C3EE9170F08B@twistedmatrix.com> <4DEC04CA.1060608@v.loewis.de> <1E59F13B-852E-4DB7-8FB2-D7B067C90920@twistedmatrix.com> Message-ID: On Tue, Jun 7, 2011 at 1:54 PM, Glyph Lefkowitz wrote: > how is UDP?different from TCP? > > The phrase "UDP" never appears in the HOWTO. ?DGRAM sockets get a brief > mention as "anything else" in the sentence: "... you?ll get better behavior > and performance from a STREAM socket than anything else ...". ?(To be fair, > I do endorse teaching that "the difference between TCP and UDP is that you > should not use UDP" to anyone not sufficiently advanced to read the relevant > reference documentation themselves.) And if UDP starts sounding tempting due to excessively high latency, these days it's worth looking up the specs for the interplanetary internet instead. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Jun 8 17:12:18 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Jun 2011 01:12:18 +1000 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: <1307482529.16924.331.camel@surprise> References: <4DEDE95A.2010603@egenix.com> <1307482529.16924.331.camel@surprise> Message-ID: On Wed, Jun 8, 2011 at 7:35 AM, David Malcolm wrote: > After ~12 years of doing this, it comes naturally. ?I appreciate that > this may come across as weird though :) I actually thought Brett's rationale in the checkin comment was reasonable (if you get in the habit of putting constants on the left, then the classic "'=' instead of '=='" typo is a compiler error instead of a reassignment). Call it a +0 in favour of letting people put constants on the left in C code if they prefer it that way, so long as any given if/elif chain is consistent in the style it uses. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From alexander.belopolsky at gmail.com Wed Jun 8 17:14:04 2011 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 8 Jun 2011 11:14:04 -0400 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: <4DEF5750.3000005@pearwood.info> References: <4DEDE95A.2010603@egenix.com> <1307482529.16924.331.camel@surprise> <4DEF5750.3000005@pearwood.info> Message-ID: On Wed, Jun 8, 2011 at 7:04 AM, Steven D'Aprano wrote: .. >> Whereas I read it as 'has the value' (or just 'is' ;=). > > Am I the only one who reads == as "equals"? If you are, you are the only one who reads it correctly. Consider >>> a = 2 >>> a == 2.0 True From rdmurray at bitdance.com Wed Jun 8 17:19:02 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 08 Jun 2011 11:19:02 -0400 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: <4DEF5750.3000005@pearwood.info> References: <4DEDE95A.2010603@egenix.com> <1307482529.16924.331.camel@surprise> <4DEF5750.3000005@pearwood.info> Message-ID: <20110608151903.7F4D425012E@webabinitio.net> On Wed, 08 Jun 2011 21:04:48 +1000, Steven D'Aprano wrote: > Terry Reedy wrote: > > On 6/7/2011 5:35 PM, David Malcolm wrote: > > > >> I know that this style is unpopular, but if it helps, try mentally > >> pronouncing "==" in C as "is the value of". > >> > >> In this example, when I read that line, my mind is thinking: > >> > >> "if 'u' is the value of typecode" > >> > >> After ~12 years of doing this, it comes naturally. I appreciate that > >> this may come across as weird though :) > > > > Whereas I read it as 'has the value' (or just 'is' ;=). > > > Am I the only one who reads == as "equals"? No :) Especially considering that Python actually has an 'is' operator.... -- R. David Murray http://www.bitdance.com From ncoghlan at gmail.com Wed Jun 8 17:55:28 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Jun 2011 01:55:28 +1000 Subject: [Python-Dev] Can we improve support for abstract base classes with desciptors In-Reply-To: References: Message-ID: On Thu, Jun 9, 2011 at 1:01 AM, Darren Dale wrote: [snip excellent analysis of the problem] I have some suggestions regarding a few details of your current code, but your basic proposal looks sound to me. I would tweak __new__ along the following lines though: def __new__(mcls, name, bases, namespace): cls = super().__new__(mcls, name, bases, namespace) # Compute set of abstract method names # CHANGE 1: refactor descriptor and abstract method scan to happen in a single pass def is_descriptor(value): return (hasattr(value, '__get__') or hasattr(value, '__set__') or hasattr(value, '__delete__')) def is_abstract(value): return getattr(value, "__isabstractmethod__", False) def get_abstract_names_for_item(item): name, value = item if is_abstract(value): return [name] elif is_descriptor(value): # CHANGE 2: Deliberately ignore descriptors on the descriptor objects # CHANGE 3: Use new-style string formatting return ['{}.{}'.format(name, attr) for attr in dir(value) if is_abstract(getattr(value, attr))] return [] def get_abstract_names(ns): names = [] for item in ns.items(): names.extend(get_abstract_names_for_item(item)) return names abstract_names = get_abstract_names(namespace.items()) for base in bases: for name in getattr(base, "__abstractmethods__", ()): # CHANGE 4: Using rpartition better tolerates weird naming in the metaclass # (weird naming in descriptors will still blow up in the earlier search for abstract names) descr_name, is_descr, attr = name.rpartition('.') if is_descr: # base class identified a descriptor abstract method: descr = getattr(cls, descr_name, None) val = getattr(descr, attr, None) else: val = getattr(cls, name, None) if val is None or is_abstract(val): abstract_names.append(name) cls.__abstractmethods__ = frozenset(abstract_names) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From debatem1 at gmail.com Wed Jun 8 20:07:15 2011 From: debatem1 at gmail.com (geremy condra) Date: Wed, 8 Jun 2011 11:07:15 -0700 Subject: [Python-Dev] The socket HOWTO In-Reply-To: References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <1F2B102E-29F8-4F12-801C-C3EE9170F08B@twistedmatrix.com> <4DEC04CA.1060608@v.loewis.de> <1E59F13B-852E-4DB7-8FB2-D7B067C90920@twistedmatrix.com> Message-ID: On Tue, Jun 7, 2011 at 10:37 AM, Eli Bendersky wrote: > > Just be careful not to reproduce http://www.apress.com/9781590593714 :-) > These things tend to get out of hand very quickly. You say that like it's a bad thing. The first few chapters of that would make a great replacement for the howto. Geremy Condra From tom.whittock at gmail.com Wed Jun 8 21:30:36 2011 From: tom.whittock at gmail.com (Tom Whittock) Date: Wed, 8 Jun 2011 20:30:36 +0100 Subject: [Python-Dev] fatal error callback issue Message-ID: I'm writing in regards to http://bugs.python.org/issue1195571 I'm embedding Python in my application and ran into a need for this functionality. I wrote a similar patch myself, and was about to submit it. When I searched for similar issues I found that this one has been available since 2005. I'd really like to help get this patch approved and integrated into the python sources. I'm sure many other python embedders have run into this in the past - it's a legitimate concern for any process which is not 100% hosted in the Python world. Thanks. Tom. From dsdale24 at gmail.com Thu Jun 9 00:51:52 2011 From: dsdale24 at gmail.com (Darren Dale) Date: Wed, 8 Jun 2011 18:51:52 -0400 Subject: [Python-Dev] Can we improve support for abstract base classes with desciptors In-Reply-To: References: Message-ID: On Wed, Jun 8, 2011 at 11:55 AM, Nick Coghlan wrote: > On Thu, Jun 9, 2011 at 1:01 AM, Darren Dale wrote: > [snip excellent analysis of the problem] > > I have some suggestions regarding a few details of your current code, > but your basic proposal looks sound to me. > > I would tweak __new__ along the following lines though: [snip] Thank you, I agree. Concerning the following block: > ? ? ? def get_abstract_names(ns): > ? ? ? ? ? names = [] > ? ? ? ? ? for item in ns.items(): > ? ? ? ? ? ? ? names.extend(get_abstract_names_for_item(item)) > ? ? ? ? ? return names > > ? ? ? abstract_names = get_abstract_names(namespace.items()) That should be "get_abstract_names(namespace)", since ns.items() gets called again in the for loop. I think the get_abstract_names function isn't needed though, since it is only ever called that one time. Any reason not replace the above block with:: abstract_names = [] for item in namespace.items(): abstract_names.extend(get_abstract_names_for_item(item)) > ? ? ? for base in bases: > ? ? ? ? ? for name in getattr(base, "__abstractmethods__", ()): > ? ? ? ? ? ? ? # CHANGE 4: Using rpartition better tolerates weird > naming in the metaclass > ? ? ? ? ? ? ? # (weird naming in descriptors will still blow up in > the earlier search for abstract names) Could you provide an example of weird naming? Darren From victor.stinner at haypocalc.com Thu Jun 9 02:46:19 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 09 Jun 2011 02:46:19 +0200 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Merge In-Reply-To: References: Message-ID: <1307580379.14102.2.camel@marge> Le jeudi 09 juin 2011 ? 02:30 +0200, brian.curtin a ?crit : > http://hg.python.org/cpython/rev/f1509fc75435 > changeset: 70715:f1509fc75435 > branch: 2.7 > parent: 70661:6e7a98cfcfab > user: Brian Curtin > date: Wed Jun 08 19:29:53 2011 -0500 > summary: > Merge FYI this commit has only one parent, it's not a merge. It should have a changelog describing the patch. ... whereas the following commit has two parents, it's a merge: > http://hg.python.org/cpython/rev/567f30527913 > changeset: 70714:567f30527913 > parent: 70712:964d0d65a2a9 > parent: 70713:88e318166eaf > user: Brian Curtin > date: Wed Jun 08 18:43:57 2011 -0500 > summary: > Fix #11583. Changed os.path.isdir to use GetFileAttributes (...) You can use "merge", "Merge 3.2", or other similir changelog. I prefer to use "(Merge 3.2)Fix #11583. Changed os.path._isdir ..." : copy/paste the changelog of the original commit with "(Merge 3.2) " prefix. Victor From tjreedy at udel.edu Thu Jun 9 02:47:17 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 08 Jun 2011 20:47:17 -0400 Subject: [Python-Dev] fatal error callback issue In-Reply-To: References: Message-ID: On 6/8/2011 3:30 PM, Tom Whittock wrote: > I'm writing in regards to http://bugs.python.org/issue1195571 > > I'm embedding Python in my application and ran into a need for this > functionality. I wrote a similar patch myself, and was about to submit > it. When I searched for similar issues I found that this one has been > available since 2005. > > I'd really like to help get this patch approved and integrated into > the python sources. I'm sure many other python embedders have run into > this in the past - it's a legitimate concern for any process which is > not 100% hosted in the Python world. Add a comment like this to the issue itself. Also review the most recent patch, considering Victor's comments. Is it better than yours? Can you improve it? Can you test it? -- Terry Jan Reedy From ncoghlan at gmail.com Thu Jun 9 04:01:01 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Jun 2011 12:01:01 +1000 Subject: [Python-Dev] Can we improve support for abstract base classes with desciptors In-Reply-To: References: Message-ID: On Thu, Jun 9, 2011 at 8:51 AM, Darren Dale wrote: > That should be "get_abstract_names(namespace)", since ns.items() gets > called again in the for loop. I think the get_abstract_names function > isn't needed though, since it is only ever called that one time. Any > reason not replace the above block with:: > > ? ? ? ?abstract_names = [] > ? ? ? ?for item in namespace.items(): > ? ? ? ? ? ?abstract_names.extend(get_abstract_names_for_item(item)) Nope, inlining that part makes sense. >> ? ? ? for base in bases: >> ? ? ? ? ? for name in getattr(base, "__abstractmethods__", ()): >> ? ? ? ? ? ? ? # CHANGE 4: Using rpartition better tolerates weird >> naming in the metaclass >> ? ? ? ? ? ? ? # (weird naming in descriptors will still blow up in >> the earlier search for abstract names) > > Could you provide an example of weird naming? >>> class C(object): ... pass ... >>> setattr(C, 'weird.name', staticmethod(int)) >>> c = C() >>> c.weird.name Traceback (most recent call last): File "", line 1, in AttributeError: 'C' object has no attribute 'weird' >>> c.weird.name Traceback (most recent call last): File "", line 1, in AttributeError: 'C' object has no attribute 'weird' >>> getattr(c, 'weird.name')() 0 This is definitely something that could legitimately be dismissed as "well, don't do that then" (particularly since similarly weird names on the descriptors will still break). However, I also prefer the way partition based code reads over split-based code, so I still like the modified version. Full tolerance for weird naming would require storing 2-tuples in __abstractmethods__ which would cause a whole new set of problems and isn't worth the hassle. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From eliben at gmail.com Thu Jun 9 06:30:57 2011 From: eliben at gmail.com (Eli Bendersky) Date: Thu, 9 Jun 2011 07:30:57 +0300 Subject: [Python-Dev] The socket HOWTO In-Reply-To: References: <20110521170725.51eab5f9@pitrou.net> <4DE2643D.3070906@v.loewis.de> <20110604205026.374c0019@pitrou.net> <4DEB2306.5080206@v.loewis.de> <1F2B102E-29F8-4F12-801C-C3EE9170F08B@twistedmatrix.com> <4DEC04CA.1060608@v.loewis.de> <1E59F13B-852E-4DB7-8FB2-D7B067C90920@twistedmatrix.com> Message-ID: On Wed, Jun 8, 2011 at 21:07, geremy condra wrote: > On Tue, Jun 7, 2011 at 10:37 AM, Eli Bendersky wrote: > > > > Just be careful not to reproduce http://www.apress.com/9781590593714 :-) > > These things tend to get out of hand very quickly. > > You say that like it's a bad thing. The first few chapters of that > would make a great replacement for the howto. > > Not a bad thing at all, and I'm sorry if I made it sound that way. I just meant that it may turn into a *whole book* if too many details are added. I had no intention to criticize this specific book. Frankly I didn't even read it, I just remembered that a book with this title came out recently. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Thu Jun 9 08:16:05 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 09 Jun 2011 08:16:05 +0200 Subject: [Python-Dev] cpython (3.2): Fix #11583. Changed os.path.isdir to use GetFileAttributes instead of os.stat. In-Reply-To: References: Message-ID: On 06/09/11 02:00, brian.curtin wrote: > http://hg.python.org/cpython/rev/88e318166eaf > changeset: 70713:88e318166eaf > branch: 3.2 > parent: 70700:0aa3064d1cef > user: Brian Curtin > date: Wed Jun 08 18:17:18 2011 -0500 > summary: > Fix #11583. Changed os.path.isdir to use GetFileAttributes instead of os.stat. > > By changing to the Windows GetFileAttributes API in nt._isdir we can figure > out if the path is a directory without opening the file via os.stat. This has > the minor benefit of speeding up os.path.isdir by at least 2x for regular > files and 10-15x improvements were seen on symbolic links (which opened the > file multiple times during os.stat). Since os.path.isdir is used in > several places on interpreter startup, we get a minor speedup in startup time. > > files: > Lib/ntpath.py | 13 ++++++++++ > Misc/NEWS | 3 ++ > Modules/posixmodule.c | 37 +++++++++++++++++++++++++++++++ > 3 files changed, 53 insertions(+), 0 deletions(-) > > > diff --git a/Lib/ntpath.py b/Lib/ntpath.py > --- a/Lib/ntpath.py > +++ b/Lib/ntpath.py > @@ -672,3 +672,16 @@ > def sameopenfile(f1, f2): > """Test whether two file objects reference the same file""" > return _getfileinformation(f1) == _getfileinformation(f2) > + > + > +try: > + # The genericpath.isdir implementation uses os.stat and checks the mode > + # attribute to tell whether or not the path is a directory. > + # This is overkill on Windows - just pass the path to GetFileAttributes > + # and check the attribute from there. > + from nt import _isdir > +except ImportError: > + from genericpath import isdir as _isdir > + > +def isdir(path): > + return _isdir(path) Not that it matters, but ISTM that this would be faster as try: from nt import _isdir as isdir except ImportError: pass Georg From victor.stinner at haypocalc.com Thu Jun 9 11:05:17 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 09 Jun 2011 11:05:17 +0200 Subject: [Python-Dev] cpython (3.2): Fix #11583. Changed os.path.isdir to use GetFileAttributes instead of os.stat. In-Reply-To: References: Message-ID: <1307610317.15909.12.camel@marge> Le jeudi 09 juin 2011 ? 08:16 +0200, Georg Brandl a ?crit : > On 06/09/11 02:00, brian.curtin wrote: > > http://hg.python.org/cpython/rev/88e318166eaf > > changeset: 70713:88e318166eaf > > branch: 3.2 > > parent: 70700:0aa3064d1cef > > user: Brian Curtin > > date: Wed Jun 08 18:17:18 2011 -0500 > > summary: > > Fix #11583. Changed os.path.isdir to use GetFileAttributes instead of os.stat. > > > > By changing to the Windows GetFileAttributes API in nt._isdir we can figure > > out if the path is a directory without opening the file via os.stat. This has > > the minor benefit of speeding up os.path.isdir by at least 2x for regular > > files and 10-15x improvements were seen on symbolic links (which opened the > > file multiple times during os.stat). Since os.path.isdir is used in > > several places on interpreter startup, we get a minor speedup in startup time. > > > > files: > > Lib/ntpath.py | 13 ++++++++++ > > Misc/NEWS | 3 ++ > > Modules/posixmodule.c | 37 +++++++++++++++++++++++++++++++ > > 3 files changed, 53 insertions(+), 0 deletions(-) > > > > > > diff --git a/Lib/ntpath.py b/Lib/ntpath.py > > --- a/Lib/ntpath.py > > +++ b/Lib/ntpath.py > > @@ -672,3 +672,16 @@ > > def sameopenfile(f1, f2): > > """Test whether two file objects reference the same file""" > > return _getfileinformation(f1) == _getfileinformation(f2) > > + > > + > > +try: > > + # The genericpath.isdir implementation uses os.stat and checks the mode > > + # attribute to tell whether or not the path is a directory. > > + # This is overkill on Windows - just pass the path to GetFileAttributes > > + # and check the attribute from there. > > + from nt import _isdir > > +except ImportError: > > + from genericpath import isdir as _isdir > > + > > +def isdir(path): > > + return _isdir(path) > > Not that it matters, but ISTM that this would be faster as > > try: > from nt import _isdir as isdir > except ImportError: > pass I would matter if _isdir() had a docstring, but it doesn't :-) genericpath.isdir() has the following doc: def isdir(s): """Return true if the pathname refers to an existing directory.""" Victor From vinay_sajip at yahoo.co.uk Thu Jun 9 12:51:31 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 9 Jun 2011 10:51:31 +0000 (UTC) Subject: [Python-Dev] 3.2.1 and Issue 12291 Message-ID: I just filed an issue which shows a serious breakage in the marshal module: "file written using marshal in 3.2 can be read by 2.7, but not 3.2 or 3.3" http://bugs.python.org/issue12291 Perhaps this issue should be marked as a release blocker for 3.2.1 - what do others (particularly Georg, of course) think? While it does appear to have been broken for some time, it seems a bit too serious to leave until a 3.2.2 release. Regards, Vinay Sajip From dsdale24 at gmail.com Thu Jun 9 13:45:57 2011 From: dsdale24 at gmail.com (Darren Dale) Date: Thu, 9 Jun 2011 07:45:57 -0400 Subject: [Python-Dev] Can we improve support for abstract base classes with desciptors In-Reply-To: References: Message-ID: On Wed, Jun 8, 2011 at 10:01 PM, Nick Coghlan wrote: > On Thu, Jun 9, 2011 at 8:51 AM, Darren Dale wrote: >>> ? ? ? for base in bases: >>> ? ? ? ? ? for name in getattr(base, "__abstractmethods__", ()): >>> ? ? ? ? ? ? ? # CHANGE 4: Using rpartition better tolerates weird >>> naming in the metaclass >>> ? ? ? ? ? ? ? # (weird naming in descriptors will still blow up in >>> the earlier search for abstract names) >> >> Could you provide an example of weird naming? > >>>> class C(object): > ... ? pass > ... >>>> setattr(C, 'weird.name', staticmethod(int)) [...] > This is definitely something that could legitimately be dismissed as > "well, don't do that then" (particularly since similarly weird names > on the descriptors will still break). However, I also prefer the way > partition based code reads over split-based code, so I still like the > modified version. Yes, I like your modified version as well. I just wanted to understand your concern, since it had never occurred to me to try something like "setattr(C, 'pathological.name', ...)". > Full tolerance for weird naming would require storing 2-tuples in > __abstractmethods__ which would cause a whole new set of problems and > isn't worth the hassle. I'm glad you feel that way. I'll work on a patch that includes docs and unit tests and post it at http://bugs.python.org/issue11610. What do you think about deprecating abstractproperty, or removing it from the documentation? Darren From ncoghlan at gmail.com Thu Jun 9 14:27:05 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Jun 2011 22:27:05 +1000 Subject: [Python-Dev] Can we improve support for abstract base classes with desciptors In-Reply-To: References: Message-ID: On Thu, Jun 9, 2011 at 9:45 PM, Darren Dale wrote: > What do you think about deprecating > abstractproperty, or removing it from the documentation? Unless anyone specifically howls at the idea, +1 to both (since abstractproperty doesn't actually achieve the intended purpose and will become redundant with the proper fix in 3.3). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From barry at python.org Thu Jun 9 17:01:42 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 9 Jun 2011 11:01:42 -0400 Subject: [Python-Dev] 3.2.1 and Issue 12291 In-Reply-To: References: Message-ID: <20110609110142.62cdd2eb@neurotica.wooz.org> On Jun 09, 2011, at 10:51 AM, Vinay Sajip wrote: >I just filed an issue which shows a serious breakage in the marshal module: > >"file written using marshal in 3.2 can be read by 2.7, but not 3.2 or 3.3" > >http://bugs.python.org/issue12291 > >Perhaps this issue should be marked as a release blocker for 3.2.1 - what do >others (particularly Georg, of course) think? While it does appear to have >been broken for some time, it seems a bit too serious to leave until a 3.2.2 >release. Sounds bad ;). I marked it a release blocker, but of course Georg will have the final say. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From brian.curtin at gmail.com Thu Jun 9 17:04:58 2011 From: brian.curtin at gmail.com (Brian Curtin) Date: Thu, 9 Jun 2011 10:04:58 -0500 Subject: [Python-Dev] cpython (3.2): Fix #11583. Changed os.path.isdir to use GetFileAttributes instead of os.stat. In-Reply-To: <1307610317.15909.12.camel@marge> References: <1307610317.15909.12.camel@marge> Message-ID: On Thu, Jun 9, 2011 at 04:05, Victor Stinner wrote: > Le jeudi 09 juin 2011 ? 08:16 +0200, Georg Brandl a ?crit : > > On 06/09/11 02:00, brian.curtin wrote: > > > http://hg.python.org/cpython/rev/88e318166eaf > > > changeset: 70713:88e318166eaf > > > branch: 3.2 > > > parent: 70700:0aa3064d1cef > > > user: Brian Curtin > > > date: Wed Jun 08 18:17:18 2011 -0500 > > > summary: > > > Fix #11583. Changed os.path.isdir to use GetFileAttributes instead of > os.stat. > > > > > > By changing to the Windows GetFileAttributes API in nt._isdir we can > figure > > > out if the path is a directory without opening the file via os.stat. > This has > > > the minor benefit of speeding up os.path.isdir by at least 2x for > regular > > > files and 10-15x improvements were seen on symbolic links (which opened > the > > > file multiple times during os.stat). Since os.path.isdir is used in > > > several places on interpreter startup, we get a minor speedup in > startup time. > > > > > > files: > > > Lib/ntpath.py | 13 ++++++++++ > > > Misc/NEWS | 3 ++ > > > Modules/posixmodule.c | 37 +++++++++++++++++++++++++++++++ > > > 3 files changed, 53 insertions(+), 0 deletions(-) > > > > > > > > > diff --git a/Lib/ntpath.py b/Lib/ntpath.py > > > --- a/Lib/ntpath.py > > > +++ b/Lib/ntpath.py > > > @@ -672,3 +672,16 @@ > > > def sameopenfile(f1, f2): > > > """Test whether two file objects reference the same file""" > > > return _getfileinformation(f1) == _getfileinformation(f2) > > > + > > > + > > > +try: > > > + # The genericpath.isdir implementation uses os.stat and checks the > mode > > > + # attribute to tell whether or not the path is a directory. > > > + # This is overkill on Windows - just pass the path to > GetFileAttributes > > > + # and check the attribute from there. > > > + from nt import _isdir > > > +except ImportError: > > > + from genericpath import isdir as _isdir > > > + > > > +def isdir(path): > > > + return _isdir(path) > > > > Not that it matters, but ISTM that this would be faster as > > > > try: > > from nt import _isdir as isdir > > except ImportError: > > pass > > I would matter if _isdir() had a docstring, but it doesn't :-) > genericpath.isdir() has the following doc: > > def isdir(s): > """Return true if the pathname refers to an existing directory.""" http://hg.python.org/lookup/d40609dd01e0 adds the docstring back in and redoes the imports as Georg mentioned, which is better. Thanks for having a look. -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Jun 10 18:07:22 2011 From: status at bugs.python.org (Python tracker) Date: Fri, 10 Jun 2011 18:07:22 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20110610160722.E6A101CDA4@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2011-06-03 - 2011-06-10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 2826 (+11) closed 21268 (+47) total 24094 (+58) Open issues with patches: 1236 Issues opened (45) ================== #5906: Risk of confusion in multiprocessing module - daemonic process http://bugs.python.org/issue5906 reopened by pakal #9516: sysconfig: $MACOSX_DEPLOYMENT_TARGET mismatch: now "10.3" but http://bugs.python.org/issue9516 reopened by eric.araujo #12255: A few changes to .*ignore http://bugs.python.org/issue12255 opened by eric.araujo #12256: Link isinstance/issubclass doc to abc module http://bugs.python.org/issue12256 opened by eric.araujo #12257: Rework/replace use of DISTUTILS_USE_SDK in packaging http://bugs.python.org/issue12257 opened by eric.araujo #12258: Clean up bytes I/O in get_compiler_versions http://bugs.python.org/issue12258 opened by eric.araujo #12259: Test and document which compilers can be created on which plat http://bugs.python.org/issue12259 opened by eric.araujo #12260: Make install default to user site-packages http://bugs.python.org/issue12260 opened by eric.araujo #12261: urllib.parse docs still refer to urlparse http://bugs.python.org/issue12261 opened by tlesher #12263: punycode codec ignores the error handler argument http://bugs.python.org/issue12263 opened by haypo #12268: file readline, readlines & readall methods can lose data on EI http://bugs.python.org/issue12268 opened by gregory.p.smith #12271: Python 2.7.x on IA64 running SLES 11 SP1 http://bugs.python.org/issue12271 opened by v.claudic #12272: Python 2.7.1 version conflict for package "Tcl" on Windows 7 http://bugs.python.org/issue12272 opened by jackie3 #12273: Change ast.__version__ calculation to provide consistent order http://bugs.python.org/issue12273 opened by ncoghlan #12274: "Print window" menu on IDLE aborts whole application http://bugs.python.org/issue12274 opened by gagenellina #12275: urllib.request.HTTPRedirectHandler won't redirect to a URL wit http://bugs.python.org/issue12275 opened by lilydjwg #12276: 3.x ignores sys.tracebacklimit=0 http://bugs.python.org/issue12276 opened by gagenellina #12277: Missing comma in os.walk docs http://bugs.python.org/issue12277 opened by Retro #12278: Core mentorship mention in the devguide http://bugs.python.org/issue12278 opened by adam.woodbeck #12279: Add build_distinfo command to packaging http://bugs.python.org/issue12279 opened by eric.araujo #12281: bytes.decode('mbcs', 'ignore') does replace undecodable bytes http://bugs.python.org/issue12281 opened by haypo #12282: build process adds CWD (null entry) to PYTHONPATH if PYTHONPAT http://bugs.python.org/issue12282 opened by r.david.murray #12284: argparse.ArgumentParser: usage example option http://bugs.python.org/issue12284 opened by jonash #12285: Unexpected behavior for 0 or negative processes in multiproces http://bugs.python.org/issue12285 opened by jorgsk #12287: ossaudiodev: stack corruption with FD >= FD_SETSIZE http://bugs.python.org/issue12287 opened by neologix #12288: Python 2.7.1 tkSimpleDialog initialvalue http://bugs.python.org/issue12288 opened by busfault #12289: http.server.CGIHTTPRequestHandler doesn't check if a Python sc http://bugs.python.org/issue12289 opened by haypo #12290: __setstate__ is called for false values http://bugs.python.org/issue12290 opened by eltoder #12291: file written using marshal in 3.2 can be read by 2.7, but not http://bugs.python.org/issue12291 opened by vinay.sajip #12294: multiprocessing.Pool: Need a way to find out if work are finis http://bugs.python.org/issue12294 opened by mozbugbox #12295: Fix ResourceWarning in turtledemo help window http://bugs.python.org/issue12295 opened by eric.araujo #12296: Minor clarification in devguide http://bugs.python.org/issue12296 opened by eric.araujo #12297: Clarifications to atexit.register and unregister doc http://bugs.python.org/issue12297 opened by eric.araujo #12299: Stop documenting functions added by site as builtins http://bugs.python.org/issue12299 opened by eric.araujo #12300: Document pydoc.help http://bugs.python.org/issue12300 opened by eric.araujo #12301: Use :data:`sys.thing` instead of ``sys.thing`` throughout http://bugs.python.org/issue12301 opened by eric.araujo #12302: packaging test command needs access to .dist-info http://bugs.python.org/issue12302 opened by michael.mulich #12303: expose sigwaitinfo() and sigtimedwait() in the signal module http://bugs.python.org/issue12303 opened by haypo #12304: expose signalfd(2) in the signal module http://bugs.python.org/issue12304 opened by haypo #12305: Building PEPs doesn't work on Python 3 http://bugs.python.org/issue12305 opened by ericsnow #12306: zlib: Expose zlibVersion to query runtime version of zlib http://bugs.python.org/issue12306 opened by torsten #12307: Inconsistent formatting of section titles in PEP 0 http://bugs.python.org/issue12307 opened by ericsnow #12308: Add link to PEP 0 for topical index in wiki http://bugs.python.org/issue12308 opened by ericsnow #12309: os.environ was modified by test_packaging http://bugs.python.org/issue12309 opened by haypo #12310: Segfault in test_multiprocessing http://bugs.python.org/issue12310 opened by haypo Most recent 15 issues with no replies (15) ========================================== #12309: os.environ was modified by test_packaging http://bugs.python.org/issue12309 #12308: Add link to PEP 0 for topical index in wiki http://bugs.python.org/issue12308 #12307: Inconsistent formatting of section titles in PEP 0 http://bugs.python.org/issue12307 #12306: zlib: Expose zlibVersion to query runtime version of zlib http://bugs.python.org/issue12306 #12305: Building PEPs doesn't work on Python 3 http://bugs.python.org/issue12305 #12303: expose sigwaitinfo() and sigtimedwait() in the signal module http://bugs.python.org/issue12303 #12301: Use :data:`sys.thing` instead of ``sys.thing`` throughout http://bugs.python.org/issue12301 #12300: Document pydoc.help http://bugs.python.org/issue12300 #12299: Stop documenting functions added by site as builtins http://bugs.python.org/issue12299 #12297: Clarifications to atexit.register and unregister doc http://bugs.python.org/issue12297 #12296: Minor clarification in devguide http://bugs.python.org/issue12296 #12295: Fix ResourceWarning in turtledemo help window http://bugs.python.org/issue12295 #12294: multiprocessing.Pool: Need a way to find out if work are finis http://bugs.python.org/issue12294 #12288: Python 2.7.1 tkSimpleDialog initialvalue http://bugs.python.org/issue12288 #12272: Python 2.7.1 version conflict for package "Tcl" on Windows 7 http://bugs.python.org/issue12272 Most recent 15 issues waiting for review (15) ============================================= #12308: Add link to PEP 0 for topical index in wiki http://bugs.python.org/issue12308 #12306: zlib: Expose zlibVersion to query runtime version of zlib http://bugs.python.org/issue12306 #12304: expose signalfd(2) in the signal module http://bugs.python.org/issue12304 #12297: Clarifications to atexit.register and unregister doc http://bugs.python.org/issue12297 #12296: Minor clarification in devguide http://bugs.python.org/issue12296 #12295: Fix ResourceWarning in turtledemo help window http://bugs.python.org/issue12295 #12291: file written using marshal in 3.2 can be read by 2.7, but not http://bugs.python.org/issue12291 #12289: http.server.CGIHTTPRequestHandler doesn't check if a Python sc http://bugs.python.org/issue12289 #12287: ossaudiodev: stack corruption with FD >= FD_SETSIZE http://bugs.python.org/issue12287 #12285: Unexpected behavior for 0 or negative processes in multiproces http://bugs.python.org/issue12285 #12281: bytes.decode('mbcs', 'ignore') does replace undecodable bytes http://bugs.python.org/issue12281 #12278: Core mentorship mention in the devguide http://bugs.python.org/issue12278 #12277: Missing comma in os.walk docs http://bugs.python.org/issue12277 #12268: file readline, readlines & readall methods can lose data on EI http://bugs.python.org/issue12268 #12261: urllib.parse docs still refer to urlparse http://bugs.python.org/issue12261 Top 10 most discussed issues (10) ================================= #12248: __dir__ semantics changed in Python 2.7.2 http://bugs.python.org/issue12248 18 msgs #12014: str.format parses replacement field incorrectly http://bugs.python.org/issue12014 12 msgs #12246: Warn when trying to install third-party module from an uninsta http://bugs.python.org/issue12246 12 msgs #8927: Handle version incompatibilities in dependencies http://bugs.python.org/issue8927 11 msgs #12211: Better document math.copysign behavior. http://bugs.python.org/issue12211 11 msgs #12226: use secured channel for uploading packages to pypi http://bugs.python.org/issue12226 11 msgs #11457: Expose nanosecond precision from system calls http://bugs.python.org/issue11457 10 msgs #7511: msvc9compiler.py: ValueError when trying to compile with VC Ex http://bugs.python.org/issue7511 8 msgs #12084: os.stat() on windows doesn't consider relative symlink http://bugs.python.org/issue12084 8 msgs #12291: file written using marshal in 3.2 can be read by 2.7, but not http://bugs.python.org/issue12291 8 msgs Issues closed (47) ================== #3771: test_httpservers intermittent failure, test_post and EINTR http://bugs.python.org/issue3771 closed by gregory.p.smith #3992: remove custom log module from distutils2 http://bugs.python.org/issue3992 closed by eric.araujo #4949: Constness in PyErr_NewException http://bugs.python.org/issue4949 closed by benjamin.peterson #7015: Getting call trace while executing "modules spam" at help prom http://bugs.python.org/issue7015 closed by eric.araujo #8407: expose pthread_sigmask(), pthread_kill(), sigpending() and sig http://bugs.python.org/issue8407 closed by haypo #9205: Parent process hanging in multiprocessing if children terminat http://bugs.python.org/issue9205 closed by pitrou #9344: please add posix.getgrouplist() http://bugs.python.org/issue9344 closed by rosslagerwall #10027: os.lstat/os.stat don't set st_nlink on Windows http://bugs.python.org/issue10027 closed by brian.curtin #10424: better error message from argparse when positionals missing http://bugs.python.org/issue10424 closed by r.david.murray #10645: Remove egg-info files in stdlib http://bugs.python.org/issue10645 closed by eric.araujo #10694: zipfile.py end of central directory detection not robust http://bugs.python.org/issue10694 closed by r.david.murray #10884: pkgutil EggInfoDistribution requirements for .egg-info metadat http://bugs.python.org/issue10884 closed by michael.mulich #11028: Implement the setup.py -> setup.cfg in mkcfg http://bugs.python.org/issue11028 closed by eric.araujo #11041: On the distutils2 documentation, 'requires-python' shouldn't b http://bugs.python.org/issue11041 closed by eric.araujo #11092: setup.cfg isn't packaged when running sdist http://bugs.python.org/issue11092 closed by eric.araujo #11203: gzip doc is behind http://bugs.python.org/issue11203 closed by teamnoir #11583: os.path.isdir() is slow on windows http://bugs.python.org/issue11583 closed by brian.curtin #11893: Obsolete SSLFakeFile in smtplib? http://bugs.python.org/issue11893 closed by pitrou #12019: Dead or buggy code in importlib.test.__main__ http://bugs.python.org/issue12019 closed by eric.araujo #12021: mmap.read requires an argument http://bugs.python.org/issue12021 closed by neologix #12040: Expose a Process.sentinel property (and fix polling loop in Pr http://bugs.python.org/issue12040 closed by pitrou #12080: decimal.py: performance in _power_exact http://bugs.python.org/issue12080 closed by mark.dickinson #12127: Inconsistent leading zero treatment http://bugs.python.org/issue12127 closed by mark.dickinson #12168: SysLogHandler incorrectly appends \000 to messages http://bugs.python.org/issue12168 closed by vinay.sajip #12214: platform module can't detect archlinux distribution http://bugs.python.org/issue12214 closed by eric.araujo #12224: problem with siginterrupt http://bugs.python.org/issue12224 closed by haypo #12234: unittest2 could enable regex debugging for more information http://bugs.python.org/issue12234 closed by r.david.murray #12237: Document how to open non-default webbrowser http://bugs.python.org/issue12237 closed by terry.reedy #12249: add missing command http://bugs.python.org/issue12249 closed by eric.araujo #12254: PEP-3107 has a wrong attribute name for function annotations http://bugs.python.org/issue12254 closed by eric.araujo #12262: Not Inheriting File Descriptors on Windows? http://bugs.python.org/issue12262 closed by 3vi1 #12264: parser.expr(): reference count error if more than one argument http://bugs.python.org/issue12264 closed by python-dev #12265: revamp argument errors http://bugs.python.org/issue12265 closed by python-dev #12266: str.capitalize contradicts oneself http://bugs.python.org/issue12266 closed by r.david.murray #12267: Difficult to compile python in Visual Studio 2010 express http://bugs.python.org/issue12267 closed by loewis #12269: IDLE 2.7 hangs on Mac OS X 10.6 http://bugs.python.org/issue12269 closed by ned.deily #12270: Please update copyright notice on bugs.python.org to 2011? http://bugs.python.org/issue12270 closed by ezio.melotti #12280: If statement http://bugs.python.org/issue12280 closed by brian.curtin #12283: python3.2 smtplib _quote_periods http://bugs.python.org/issue12283 closed by r.david.murray #12286: float('nan') breaks sort() method on a list of floats http://bugs.python.org/issue12286 closed by ezio.melotti #12292: bpython http://bugs.python.org/issue12292 closed by amaury.forgeotdarc #12293: wrong arguments passed to SMTP.sendmail in example http://bugs.python.org/issue12293 closed by r.david.murray #12298: Sphinx glitch in library/functions http://bugs.python.org/issue12298 closed by georg.brandl #12311: memory leak with self-referencing dict http://bugs.python.org/issue12311 closed by Albert.Zeyer #12312: is ok http://bugs.python.org/issue12312 closed by brian.curtin #1083299: Distutils doesn't pick up all the files it should. http://bugs.python.org/issue1083299 closed by eric.araujo #1699259: replacing char* with const char* in sysmodule.c/.h http://bugs.python.org/issue1699259 closed by sebastinas From guido at python.org Fri Jun 10 21:49:26 2011 From: guido at python.org (Guido van Rossum) Date: Fri, 10 Jun 2011 12:49:26 -0700 Subject: [Python-Dev] cpython: Remove some extraneous parentheses and swap the comparison order to In-Reply-To: References: <4DEDE95A.2010603@egenix.com> <1307482529.16924.331.camel@surprise> Message-ID: On Wed, Jun 8, 2011 at 8:12 AM, Nick Coghlan wrote: > On Wed, Jun 8, 2011 at 7:35 AM, David Malcolm wrote: >> After ~12 years of doing this, it comes naturally. ?I appreciate that >> this may come across as weird though :) > > I actually thought Brett's rationale in the checkin comment was > reasonable (if you get in the habit of putting constants on the left, > then the classic "'=' instead of '=='" typo is a compiler error > instead of a reassignment). I really like consistency across the code base. I really don't like constant-on-the-left, and it's basically not used in the current codebase. Please be consistent and don't start using it. > Call it a +0 in favour of letting people put constants on the left in > C code if they prefer it that way, so long as any given if/elif chain > is consistent in the style it uses. Sorry, I give it a -1. (I'd like to be able to read the codebase still... :-) -- --Guido van Rossum (python.org/~guido) From wolfson at gmail.com Fri Jun 10 23:15:54 2011 From: wolfson at gmail.com (Ben Wolfson) Date: Fri, 10 Jun 2011 14:15:54 -0700 Subject: [Python-Dev] PEP 3101 implementation vs. documentation Message-ID: Hello, I'm writing because discussion in a bug report I submitted () has suggested that, insofar as at least part of the issue revolves around the interpretation of PEP 3101, that aspect belonged on python-dev. In particular, I was told that the PEP, not the documentation, is authoritative. Since I'm the one who thinks something is wrong, it seems appropriate for me to be the one to bring it up. Basically, the issue is that the current behavior of str.format is at variance at least with the documentation , is almost certainly at variance with PEP3101 in one respect, and is in my opinion at variance with PEP3101 in another respect as well, regarding what characters can be present in what the grammar given in the documentation calls an element_index, that is, the bit between the square brackets in "{0.attr[idx]}". Both discovering the current behavior and interpreting the documentation are pretty straightforward; interpreting what the PEP actually calls for is more vexed. I'll do the first two things first. TOC for the remainder: 1. What does the current implementation do? 2. What does the documentation say? 3. What does the PEP say? [this part is long, but the PEP is not clear, and I wanted to be thorough] 4. Who cares? 1. What does the current implementation do? Suppose you have this dictionary: d = {"@": 0, "!": 1, ":": 2, "^": 3, "}": 4, "{": {"}": 5}, } Then the following expressions have the following results: (a) "{0[@]}".format(d) --> '0' (b) "{0[!]}".format(d) --> ValueError: Missing ']' in format string (c) "{0[:]}".format(d) --> ValueError: Missing ']' in format string (d) "{0[^]}".format(d) --> '3' (e) "{0[}]}".format(d) --> ValueError: Missing ']' in format string (f) "{0[{]}".format(d) --> ValueError: unmatched '{' in format (g) "{0[{][}]}".format(d) --> '5' Given (e) and (f), I think (g) should be a little surprising, though you can probably guess what's going on and it's not hard to see why it happens if you look at the source: (e) and (f) fail because MarkupIterator_next (in Objects/stringlib/string_format.h) scans through the string looking for curly braces, because it treats them as semantically significant no matter what context they occur in. So, according to MarkupIterator_next, the *first* right curly brace in (e) indicates the end of the replacement field, giving "{0[}". In (f), the second left curly brace indicates (to MarkupIterator_next) the start of a *new* replacement field, and since there's only one right curly brace, it complains. In (g), MarkupIterator_next treats the second left curly brace as starting a new replacement field and the first right curly brace as closing it. However, actually, those braces don't define new replacement fields, as indicated by the fact that the whole expression treats the element_index fields as just plain old strings. (So the current implementation is somewhat schizophrenic, acting at one point as if the braces have to be balanced because '{[]}' is a replacement field and at another point treating the braces as insignificant.) The explanation for (b) and (c) is that parse_field (same source file) treats ':' and '!' as indicating the end of the field_name section of the replacement field, regardless of whether those characters occur within square brackets or not. So, that's what the current implementation does, in those cases. 2. What does the documentation say? The documentation gives a grammar for replacement fields: """ replacement_field ::= "{" [field_name] ["!" conversion] [":" format_spec] "}" field_name ::= arg_name ("." attribute_name | "[" element_index "]")* arg_name ::= [identifier | integer] attribute_name ::= identifier element_index ::= integer | index_string index_string ::= + conversion ::= "r" | "s" format_spec ::= """ Given this definition of index_string, all of (a)--(g) should be legal, and the results should be the strings '0', '1', '2', '3', "{'}': 5}", and '5'. There is no need to exclude ':', '!', '}', or '{' from the index_string field; allowing them creates no ambiguity, because the field is delimited by square brackets. Tangent: the definition of attribute_name here also does not describe the current behavior ("{0. ;}".format(x) works fine and will call getattr(x, " ;")) and the PEP does not require the attribute_name to be an identifier; in fact it explicitly states that the attribute_name doesn't need to be a valid Python identifier. attribute_name should read (to reflect something like actual behavior, anyway) " +". The same goes for arg_name (with the integer alternation). Observe: >>> x = lambda: None >>> setattr(x, ']]', 3) >>> "{].]]}".format(**{"]":x}) # (h) '3' One can also presently do this (hence "something like actual behavior"): >>> setattr(x, 'f}', 4) >>> "{a{s.f}}".format(**{"a{s":x}) '4' But not this: >>> "{a{s.func_name}".format(**{"a{s":x}) as it raises a ValueError, for the same reason as explains (g) above. 3. What does the PEP say? Well... It's actually hard to tell! Summary: The PEP does not contain a grammar for replacement fields, and is surprisingly nonspecific about what can appear where, at least when talking about the part of the replacement field before the format specifier. The most reasonable interpretation of the parts of the PEP that seem to be relevant favors the documentation, rather than the implementation. This can be separated into two sub-questions. A. What does the PEP say about ':' and '!'? It says two things that pertain to element_index fields. The first is this: """ The rules for parsing an item key are very simple. If it starts with a digit, then it is treated as a number, otherwise it is used as a string. Because keys are not quote-delimited, it is not possible to specify arbitrary dictionary keys (e.g., the strings "10" or ":-]") from within a format string. """ So it notes that some things can't be used: - Because anything composed entirely of digits is treated as a number, you can't get a string composed entirely of digits. Clear enough. - What's the explanation for the second example, though? Well, you can't have a right square bracket in the index_string, so that would already mean that you can't do this: "{0[:-]]}".format(...) regardless of the whether colons are legal or not. So, although the PEP gives an example of a string that can't in the element_index part of a replacement field, and that string contains a colon, that string would have been disallowed for other reasons anyway. The second is this: """ The str.format() function will have a minimalist parser which only attempts to figure out when it is "done" with an identifier (by finding a '.' or a ']', or '}', etc.). """ This requires some interpretation. For one thing, the contents of an element_index aren't identifiers. For another, it's not true that you're done with an identifier (or whatever) whenever you see *any* of those; it depends on context. When parsing this: "{0[a.b]}" the parser should not stop at the '.'; it should keep going until it reaches the ']', and that will give the element_index. And when parsing this: "{0.f]oo[bar].baz}", it's done with the identifier "foo" when it reaches the '[', not when it reaches the second '.', and not when it reaches the ']', either (recall (h)). The "minimalist parser" is, I take it, one that works like this: when parsing an arg_name you're done when you reach a '[', a ':', a '!', a '.', '{', or a '}'; the same rules apply when parsing a attribute_name; when parsing an element_index you're done when you see a ']'. Now as regards the curly braces there are some other parts of the PEP that perhaps should be taken into account (see below), but I can find no justification for excluding ':' and '!' from the element_index field; the bit quoted above about having a minimalist parser isn't a good justification for that, and it's the only part of the entire PEP that comes close to addressing the question. B. What does it say about '}' and '{'? It still doesn't say much explicitly. There's the quotation I just gave, and then these passages: """ Brace characters ('curly braces') are used to indicate a replacement field within the string: [...] Braces can be escaped by doubling: """ Taken by itself, this would suggest that wherever there's an unescaped brace, there's a replacement field. That would mean that the current implementation's behavior is correct in (e) and (f) but incorrect in (g). However, I think this is a bad interpretation; unescaped braces can indicate the presence of a replacement field without that meaning that *within* a replacement field braces have that meaning, no matter where within the replacement field they occur. Later in the PEP, talking about this example: "{0:{1}}".format(a, b) We have this: """ These 'internal' replacement fields can only occur in the format specifier part of the replacement field. Internal replacement fields cannot themselves have format specifiers. This implies also that replacement fields cannot be nested to arbitrary levels. Note that the doubled '}' at the end, which would normally be escaped, is not escaped in this case. The reason is because the '{{' and '}}' syntax for escapes is only applied when used *outside* of a format field. Within a format field, the brace characters always have their normal meaning. """ The claim "within a format field, the brace characters always have their normal meaning" might be taken to mean that within a *replacement* field, brace characters always indicate the start (or end) of a replacement field. But the PEP at this point is clearly talking about the formatting section of a replacement field---the part that follows the ':', if present. ("Format field" is nowhere defined in the PEP, but it seems reasonable to take it to mean "the format specifier of a replacement field".) However, it seems most reasonable to me to take "normal meaning" to mean "just a brace character". Note that the present implementation only kinda sorta conforms to the PEP in this respect: >>> import datetime >>> format(datetime.datetime.now(), "{{%Y") '{{2011' >>> "{0:{{%{1}}".format(datetime.datetime.now(), 'Y') # (i) Traceback (most recent call last): File "", line 1, in ValueError: unmatched '{' in format >>> "{0:{{%{1}}}}".format(datetime.datetime.now(), 'Y') # (j) '{2011}' Here the brace characters in (i) and (j) are treated, again in MarkupIterator_next, as indicating the start of a replacement field. In (i), this leads the function to throw an exception; since they're balanced in (j), processing proceeds further, and the doubled braces aren't treated as indicating the start or end of a replacement field---because they're escaped. Given that the format spec part of a replacement field can contain further replacement fields, this is, I think, correct behavior, but it's not easy to see how it comports with the PEP, whose language is not very exact. The call to the built-in format() bypasses the mechanism that leads to these results. The PEP is very, very nonspecific about the parts of the replacement field that precede the format specifier. I don't know what kind of discussion surrounded the drawing up of the grammar that appears in the documentation, but I think that it, and not the implementation, should be followed. The implementation only works the way it does because of parsing shortcuts: it raises ValueErrors for (b) and (c) because it generalizes something true of the attribute_name field (encountering a ':' or '!' means one has moved on to the format_spec or conversion part of the replacement field) to the element_index field. And it raises an error for (e) and (f), but lets (g) through, for the reasons already mentioned. It is, in that respect, inconsistent; it treats the curly brace as having one semantic significance at one point and an entirely different significance at another point, so that it does the right thing in the case of (g) entirely by accident. There is, I think, no way to construe the PEP so that it is reasonable to do what the present implementation does in all three cases (if "{" indicates the start of a replacement field in (f), it should do so in (g) as well); I think it's actually pretty difficult to construe the PEP in any way that makes what it does in the case of (e) and (f) correct. 4. Who cares? Well, I do. (Obviously.) I even have a use case: I want to be able to put arbitrary (or as close to arbitrary as possible) strings in the element_index field, because I've got some objects that (should!) enable me to do this: p.say("I'm warning you, {e.red.underline[don't touch that!]}") and have this written ("e" for "effects") to p.out: I'm warning you, \x1b[31m\x1b[4mdon't touch that!\x1b[0m I have a way around the square bracket problem, but it would be quite burdensome to have to deal with all of !:{} as well; enough that I would fall back on something like this: "I'm warning you, {0}".format(e.red.underline("don't touch that!")) or some similar interpolation-based strategy, which I think would be a shame, because of the way it chops up the string. But I also think the present behavior is extremely counterintuitive, unnecessarily complex, and illogical (even unpythonic!). Isn't it bizarre that (g) should work, given what (e) and (f) do? Isn't it strange that (b) and (c) should fail, given that there's no real reason for them to do so---no ambiguity that has to be avoided? And something's gotta give; the documentation and the implementation do not correspond. Beyond the counterintuitiveness of the present implementation, it is also, I think, self-inconsistent. (e) and (f) fail because the interior brace is treated as starting or ending a replacement field, even though interior replacement fields aren't allowed in that part of a replacement field. (g) succeeds because the braces are balanced: they are treated at one point as if they were part of a replacement field, and at another (correctly) as if they are not. But this makes the failure of (e) and (f) unaccountable. It would not have been impossible for the PEP to allow interior replacement fields anywhere, and not just in the format spec, in which case we might have had this: (g') "{0[{][}]}".format(range(10), **{'][':4}) --> '3' or this: (g'') "{0[{][}]}".format({'4':3}, **{'][':4}) --> '3' or something with that general flavor. As far as I can tell, the only way to consistently maintain that (e) and (f) should fail requires that one take (g') or (g'') to be correct: either the interior braces signal replacement fields (hence must be balanced) or they don't (or they're escaped). Regarding the documentation, it could of course be rendered correct by changing *it*, rather than the implementation. But wouldn't it be tremendously weird to have to explain that, in the part of the replacement field preceding the conversion, you can't employ any curly braces, unless they're balanced, and you can't employ ':' or '!' at all, even though they have no semantic significance? So these are out: {0[{].foo} {0[}{}]} {0[a:b]} But these are in: {0[{}{}]} {0[{{].foo.}}} (k) ((k) does work, if you give it an object with the right structure, though it probably should not.) And, moreover, someone would then have to actually change the documentation, whereas there's a patch already, attached to the bug report linked way up at the top of this email, that makes (a)--(g) all work, leaves (i) and (j) as they are, and has the welcome side-effect of making (k) *not* work (if any code anywhere relies on (k) or things like it working, I will be very surprised: anyway the fact that (k) works is, technically, undocumented). It is also quite simple. It doesn't effect the built-in format(), because the built-in format() is concerned only with the format-specifier part of the replacement field and not all the stuff that comes before that, telling str.format *what* object to format. Thanks for reading, -- Ben Wolfson "Human kind has used its intelligence to vary the flavour of drinks, which may be sweet, aromatic, fermented or spirit-based. ... Family and social life also offer numerous other occasions to consume drinks for pleasure." [Larousse, "Drink" entry] From sam.edwards at Colorado.EDU Sat Jun 11 02:23:47 2011 From: sam.edwards at Colorado.EDU (Sam Edwards) Date: Fri, 10 Jun 2011 18:23:47 -0600 Subject: [Python-Dev] Python jails Message-ID: <4DF2B593.8040302@colorado.edu> Hello! This is my first posting to the python-dev list, so please forgive me if I violate any unspoken etiquette here. :) I was looking at Python 2.x's f_restricted frame flag (or, rather, the numerous ways around it) and noticed that most (all?) of the attacks to escape restricted execution involved the attacker grabbing something he wasn't supposed to have. IMO, Python's extensive introspection features make that a losing battle, since it's simply too easy to forget to blacklist something and the attacker finding it. Not only that, even with a perfect vacuum-sealed jail, an attacker can still bring down the interpreter by exhausting memory or consuming excess CPU. I think I might have a way of securely sealing-in untrusted code. It's a fairly nascent idea, though, and I haven't worked out all of the details yet, so I'm posting what I have so far for feedback and for others to try to poke holes in it. Absolutely nothing here is final. I'm just framing out what I generally had in mind. Obviously, it will need to be adjusted to be consistent with "the Python way" - my hope is that this can become a PEP. :) >>> # It all starts with the introduction of a new type, called a jail. (I haven't yet worked out whether it should be a builtin type, ... # or a module.) Unjailed code can create jails, which will run the untrusted code and keep strict limits on it. ... >>> j = jail() >>> dir(j) ['__class__', '__delattr__', '__doc__', '__format__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'acquire', 'getcpulimit', 'getcpuusage', 'getmemorylimit', 'getmemoryusage', 'gettimelimit', 'gettimeusage', 'release', 'setcpulimit', 'setmemorylimit', 'settimelimit'] >>> # The jail monitors three things: Memory (in bytes), real time (in seconds), and CPU time (also in seconds) ... # and it also allows you to impose limits on them. If any limit is non-zero, code in that jail may not exceed its limit. ... # Exceeding a memory limit will result in a MemoryError. I haven't decided what CPU/real time limits should raise. ... # The other two calls are "acquire" and "release," which allow you to seal (any) objects inside the jail, or bust them # out. Objects inside the jail (i.e. created by code in that jail) contribute their __sizeof__() to the j.getmemoryusage() ... >>> def stealPasswd(): ... return open('/etc/passwd','r').read() ... >>> j.acquire(stealPasswd) >>> j.getmemoryusage() # The stealPasswd function, its code, etc. are now locked away within the jail. 375 >>> stealPasswd() Traceback (most recent call last): File "", line 1, in JailError: tried to access an object outside of the jail The object in question is, of course, 'open'. Unlike the f_restricted model, the jail was freely able to grab the open() function, but was absolutely unable to touch it: It can't call it, set/get/delete attributes/items, or pass it as an argument to any functions. There are three criteria that determine whether an object can be accessed: a. The code accessing the object is not within a jail; or b. The object belongs to the same jail as the code accessing the object; or c. The object has an __access__ function, and theObject.__access__(theJail) returns True. For the jail to be able to access 'open', it needs to be given access explicitly. I haven't quite decided how this should work, but I had in mind the creation of a "guard" (essentially a proxy) that allows the jail to access the object. It belongs to the same jail as the guarded object (and is therefore impossible to create within a jail unless the guarded object belongs to the same jail), has a list of jails (or None for 'any') that the guard will allow to __access__ it (the guard is immutable, so jails can't mess with it even though they can access it), and what the guard will allow though it (read-write, read-only, call-within-jail, call-outside-jail). I have a couple remaining issues that I haven't quite sussed out: * How exactly do guards work? I had in mind a system of proxies (memory usage is a concern, especially in memory-limited jails - maybe allow __access__ to return specific modes of access rather than all-or-nothing?) that recursively return more guards after operations. (e.g., if I have a guard allowing read+call on sys, sys.stdout would return another guard allowing read+call on sys.stdout, likewise for sys.stdout.write) * How are objects sealed in the jail? j.acquire can lead to serious problems with lots of references getting recursively sealed in. Maybe disallow sealing in anything but code objects, or allow explicitly running code within a jail like j.execute(code, globals(), locals()), which works fine since any objects created by jailed code are also jailed. * How do imports work? Should __import__ be modified so that when a jail invokes it, the import runs normally (unjailed), and then returns the module with a special guard that allows read-only+call-within, but not on builtins? This has a nice advantage, since jailed code can import e.g. socket, and maybe even create a socket, but won't be able to do sock.connect(...), since socket.connect (which is running with jailed permissions) can't touch the builtin _socket module. * Is obj.__access__(j) the best way to decide access? It doesn't allow programmers much freedom to customize the jail policy since they can't modify __access__ for builtins. Maybe the jail should have the first chance (such as j.__access__(obj)), which allows programmers to subclass the jail, and the jail can fallback to obj.__access__(j) * How does Python keep track of what jail each frame is in? Maybe each frame can have a frame.f_jail, which references the jail object restricting that frame (or None for unjailed code) - frames' jails default to the jail holding the code object, or can be explicitly overridden (as in j.execute(code, globals(), locals())) * When are jails switched? Obviously, jailed code called from unjailed code (or even from other unjailed code) should be executed in the callee jail... But if a jailed caller is calling unjailed code, does the jail follow, or does the unjailed code run in an unjailed frame? How do programmers specify that? ...that's pretty much my two (erm, twenty) cents on the matter. Again, any feedback/adversarial reasoning you guys can offer is much appreciated. From rdmurray at bitdance.com Sat Jun 11 02:41:32 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 10 Jun 2011 20:41:32 -0400 Subject: [Python-Dev] Python jails In-Reply-To: <4DF2B593.8040302@colorado.edu> References: <4DF2B593.8040302@colorado.edu> Message-ID: <20110611004134.0ABB62505A0@webabinitio.net> On Fri, 10 Jun 2011 18:23:47 -0600, Sam Edwards wrote: > Hello! This is my first posting to the python-dev list, so please > forgive me if I violate any unspoken etiquette here. :) Well, hopefully we won't bite, though of course I can't promise anything for anyone else :) I haven't read through your post, but if you don't know about it I suspect that you will be interested in the following: http://code.activestate.com/pypm/pysandbox/ I'm pretty sure Victor will be happy to have someone else interested in this topic. -- R. David Murray http://www.bitdance.com From guido at python.org Sat Jun 11 02:44:26 2011 From: guido at python.org (Guido van Rossum) Date: Fri, 10 Jun 2011 17:44:26 -0700 Subject: [Python-Dev] Python jails In-Reply-To: <4DF2B593.8040302@colorado.edu> References: <4DF2B593.8040302@colorado.edu> Message-ID: Hi Sam, Have you seen this? http://tav.espians.com/paving-the-way-to-securing-the-python-interpreter.html It might relate a similar idea. There were a few iterations of Tav's approach. --Guido On Fri, Jun 10, 2011 at 5:23 PM, Sam Edwards wrote: > Hello! This is my first posting to the python-dev list, so please > forgive me if I violate any unspoken etiquette here. :) > > I was looking at Python 2.x's f_restricted frame flag (or, rather, the > numerous ways around it) and noticed that most (all?) > of the attacks to escape restricted execution involved the attacker > grabbing something he wasn't supposed to have. > IMO, Python's extensive introspection features make that a losing > battle, since it's simply too easy to forget to blacklist > something and the attacker finding it. Not only that, even with a > perfect vacuum-sealed jail, an attacker can still bring down > the interpreter by exhausting memory or consuming excess CPU. > > I think I might have a way of securely sealing-in untrusted code. It's a > fairly nascent idea, though, and I haven't worked out > all of the details yet, so I'm posting what I have so far for feedback > and for others to try to poke holes in it. > > Absolutely nothing here is final. I'm just framing out what I generally > had in mind. Obviously, it will need to be adjusted to > be consistent with "the Python way" - my hope is that this can become a > PEP. :) > > >>>> # It all starts with the introduction of a new type, called a jail. > (I haven't yet worked out whether it should be a builtin type, > ... # or a module.) Unjailed code can create jails, which will run the > untrusted code and keep strict limits on it. > ... >>>> j = jail() >>>> dir(j) > ['__class__', '__delattr__', '__doc__', '__format__', > '__getattribute__', '__hash__', > '__init__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', > '__setattr__', > '__sizeof__', '__str__', '__subclasshook__', 'acquire', 'getcpulimit', > 'getcpuusage', > 'getmemorylimit', 'getmemoryusage', 'gettimelimit', 'gettimeusage', > 'release', > 'setcpulimit', 'setmemorylimit', 'settimelimit'] >>>> # The jail monitors three things: Memory (in bytes), real time (in > seconds), and CPU time (also in seconds) > ... # and it also allows you to impose limits on them. If any limit is > non-zero, code in that jail may not exceed its limit. > ... # Exceeding a memory limit will result in a MemoryError. I haven't > decided what CPU/real time limits should raise. > ... # The other two calls are "acquire" and "release," which allow you > to seal (any) objects inside the jail, or bust them > ? ?# out. Objects inside the jail (i.e. created by code in that jail) > contribute their __sizeof__() to the j.getmemoryusage() > ... >>>> def stealPasswd(): > ... ? ? ? ? return open('/etc/passwd','r').read() > ... >>>> j.acquire(stealPasswd) >>>> j.getmemoryusage() # The stealPasswd function, its code, etc. are > now locked away within the jail. > 375 >>>> stealPasswd() > Traceback (most recent call last): > ?File "", line 1, in > JailError: tried to access an object outside of the jail > > The object in question is, of course, 'open'. Unlike the f_restricted > model, the jail was freely able to grab > the open() function, but was absolutely unable to touch it: It can't > call it, set/get/delete attributes/items, > or pass it as an argument to any functions. There are three criteria > that determine whether an object can > be accessed: > a. The code accessing the object is not within a jail; or > b. The object belongs to the same jail as the code accessing the object; or > c. The object has an __access__ function, and > theObject.__access__(theJail) returns True. > > For the jail to be able to access 'open', it needs to be given access > explicitly. I haven't quite decided > how this should work, but I had in mind the creation of a "guard" > (essentially a proxy) that allows the jail > to access the object. It belongs to the same jail as the guarded object > (and is therefore impossible to create > within a jail unless the guarded object belongs to the same jail), has a > list of jails (or None for 'any') that the > guard will allow to __access__ it (the guard is immutable, so jails > can't mess with it even though they can > access it), and what the guard will allow though it (read-write, > read-only, call-within-jail, call-outside-jail). > > I have a couple remaining issues that I haven't quite sussed out: > * How exactly do guards work? I had in mind a system of proxies (memory > usage is a concern, especially > ? ?in memory-limited jails - maybe allow __access__ to return specific > modes of access rather than > ? ?all-or-nothing?) that recursively return more guards after > operations. (e.g., if I have a guard allowing > ? ?read+call on sys, sys.stdout would return another guard allowing > read+call on sys.stdout, likewise for > ? ?sys.stdout.write) > * How are objects sealed in the jail? j.acquire can lead to serious > problems with lots of references > ? ?getting recursively sealed in. Maybe disallow sealing in anything > but code objects, or allow explicitly > ? ?running code within a jail like j.execute(code, globals(), > locals()), which works fine since any objects > ? ?created by jailed code are also jailed. > * How do imports work? Should __import__ be modified so that when a jail > invokes it, the import runs > ? ?normally (unjailed), and then returns the module with a special > guard that allows read-only+call-within, > ? ?but not on builtins? This has a nice advantage, since jailed code > can import e.g. socket, and maybe even > ? ?create a socket, but won't be able to do sock.connect(...), since > socket.connect (which is running with > ? ?jailed permissions) can't touch the builtin _socket module. > * Is obj.__access__(j) the best way to decide access? It doesn't allow > programmers much freedom to > ? ?customize the jail policy since they can't modify __access__ for > builtins. Maybe the jail should have > ? ?the first chance (such as j.__access__(obj)), which allows > programmers to subclass the jail, and the jail > ? ?can fallback to obj.__access__(j) > * How does Python keep track of what jail each frame is in? Maybe each > frame can have a frame.f_jail, > ? ?which references the jail object restricting that frame (or None for > unjailed code) - frames' jails default > ? ?to the jail holding the code object, or can be explicitly overridden > (as in j.execute(code, globals(), locals())) > * When are jails switched? Obviously, jailed code called from unjailed > code (or even from other unjailed > ? ?code) should be executed in the callee jail... But if a jailed > caller is calling unjailed code, does the jail > ? ?follow, or does the unjailed code run in an unjailed frame? How do > programmers specify that? > > ...that's pretty much my two (erm, twenty) cents on the matter. Again, > any feedback/adversarial reasoning > you guys can offer is much appreciated. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From pje at telecommunity.com Sat Jun 11 02:52:28 2011 From: pje at telecommunity.com (P.J. Eby) Date: Fri, 10 Jun 2011 20:52:28 -0400 Subject: [Python-Dev] Python jails In-Reply-To: <4DF2B593.8040302@colorado.edu> References: <4DF2B593.8040302@colorado.edu> Message-ID: <20110611005243.737573A4060@sparrow.telecommunity.com> At 06:23 PM 6/10/2011 -0600, Sam Edwards wrote: >I have a couple remaining issues that I haven't quite sussed out: >[long list of questions deleted] You might be able to answer some of them by looking at this project: http://pypi.python.org/pypi/RestrictedPython Which implements the necessary ground machinery for doing that sort of thing, in the form of a specialized Python compiler (implemented in Python, for 2.3 through 2.7) that allows you to implement whatever sorts of guards and security policies you want on top of it. Even if it doesn't answer all your questions in and of itself, it may prove a fruitful environment in which you can experiment with various approaches and see which ones you actually like, without first having to write a bunch of code yourself. Discussing an official implementation of this sort of thing as a language feature is probably best left to python-ideas, though, until and unless you actually have a PEP to propose. From sam.edwards at Colorado.EDU Sat Jun 11 06:08:55 2011 From: sam.edwards at Colorado.EDU (Sam Edwards) Date: Fri, 10 Jun 2011 22:08:55 -0600 Subject: [Python-Dev] Python jails In-Reply-To: <4DF2B593.8040302@colorado.edu> References: <4DF2B593.8040302@colorado.edu> Message-ID: <4DF2EA57.3030506@colorado.edu> All, Thanks for the quick responses! I've skimmed the pysandbox code yesterday. I think Victor has the right idea with relying on a whitelist, as well as limiting execution time. The fact that untrusted code can still execute memory exhaustion attacks is the only thing that still worries me: It's hard to write a server that will run hundreds of scripts from untrusted users, since one of them can bring down the entire server by writing an infinite loop that allocates tons of objects. Python needs a way to hook the object-allocation process in order to (effectively) limit how much memory untrusted code can consume. Tav's blog post makes some interesting points... The object-capability model definitely has the benefit of efficiency; simply getting the reference to an object means the untrusted code is trusted with full capability to that object (which saves having to query the jail every time the object is touched) - it's just as fast as unrestricted Python, which I like. Perhaps my jails idea should then be refactored into some mechanism for monitoring and limiting memory and CPU usage -- it's the perfect thing to ship as an extension, the only shame is that it requires interpreter support. Anyway, in light of Tav's post which seems to suggest that f_restricted frames are impossible to escape (if used correctly), why was f_restricted removed in Python 3? Is it simply that it's too easy to make a mistake and accidentally give an attacker an unsafe object, or is there some fundamental flaw with it? Could you see something like f_restricted (or f_jail) getting put back in Python 3, if it were a good deal more bulletproof? And, yeah, I've been playing with RestrictedPython. It's pretty good, but it lacks memory- and CPU-limiting, which is my main focus right now. And yes, I should probably have posted this to python-ideas, thanks. :) This is a very long way away from a PEP. PyPy's sandboxing feature is probably closest to what I'd like, but I'm looking for something that can coexist in the same process (since running hundreds of interpreter processes continuously has a lot of system memory overhead, it's better if the many untrusted, but independent, jails could share a single interpreter) From victor.stinner at haypocalc.com Sat Jun 11 10:34:30 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 11 Jun 2011 10:34:30 +0200 Subject: [Python-Dev] Python jails In-Reply-To: <20110611004134.0ABB62505A0@webabinitio.net> References: <4DF2B593.8040302@colorado.edu> <20110611004134.0ABB62505A0@webabinitio.net> Message-ID: <4DF32896.8020504@haypocalc.com> Le 11/06/2011 02:41, R. David Murray a ?crit : > I haven't read through your post, but if you don't know about it I > suspect that you will be interested in the following: > > http://code.activestate.com/pypm/pysandbox/ > > I'm pretty sure Victor will be happy to have someone else interested in > this topic. > Yes, I am happy :-) The project URL is https://github.com/haypo/pysandbox/ Activestate page is wrong: pysanbox does support Python 3 (Python 2.5 - 3.3). pysandbox uses different policy depending on the problem. For example, whitelist for builtins, blacklist for object attributes. pysandbox is based on Tav's ideas. The main idea of pysandbox is to execute untrusted in a new empty namespace, the untrusted namespace. Objects imported into this namespace are imported as proxies to get a read-only view of the Python namespace. Importing modules is protected by a whitelist (modules and symbols names). To protect the namespace, some introspection attributes are hidden like __subclasses__ or __self__. Performances are supposed to be close to a classic Python interpreter (I didn't run a benchmark, I don't really care). An empty namespace is not enough to protect Python: pysandbox denies the execution of arbitrary bytecode, write files, write to stdout/stderr, exit Python, etc. Tav's sandbox is good to deny everything, whereas you can configure pysandbox to enable some features (e.g. exit Python, useful for an interpreter). About restricted mode: you can also configure pysandbox to use it, but the restricted mode is too much restrictive: you cannot open files, whereas pysandbox allows to read files in a whitelist (e.g. useful to display a backtrace). If you would like to implement your own sandbox: great! You should try pysandbox test suite, I'm proud of it :-) I am still not sure that pysandbox approach is the good one: if you find a vulnerability to escape pysandbox "jail" (see pysandbox Changelog, it would not be the first time), you can do anything. PyPy sandbox and "Seccomp nurse" (specific to Linux?) use two processes: the Python process cannot do anything, it relies completly in a trusted process which control all operations. I don't understand exactly how it is different: a vulnerability in the trusted process gives also full control, but it's maybe a safer approach. Or the difference is maybe that the implementation is simpler (less code?) and so safer (less code usually means less bugs). "Seccomp nurse": http://chdir.org/~nico/seccomp-nurse/ I tested recently AppEngine sandbox (testable online via http://shell.appspot.com/): it is secure *and* powerful, quite all modules are allowed (except not ctypes, as expected). AppEngine is not a Python sandbox: it's a sandbox between the Python process and Linux kernel, so it protects also modules written in C (pysandbox is unable to protect these modules). AppEngine modifies the Python standard library to cooperate with the low-level sandbox, e.g. raise nice error messages with open(filename, "w"): invalid file mode (instead of an ugly OSError with a cryptic message). Get more information about pysandbox and other sandboxes in pysandbox README file. Victor From ncoghlan at gmail.com Sat Jun 11 11:16:37 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Jun 2011 19:16:37 +1000 Subject: [Python-Dev] PEP 3101 implementation vs. documentation In-Reply-To: References: Message-ID: On Sat, Jun 11, 2011 at 7:15 AM, Ben Wolfson wrote: [snip very thorough analysis] To summarise (after both the above post and the discussion on the tracker) The current str.format implementation differs from the documentation in two ways: 1. It ignores the presence of an unclosed index field when processing a replacement field (placing additional restrictions on allowable characters in index strings). 2. Replacement fields that appear in name specifiers are processed by the parser for brace-matching purposes, but not substituted More accurate documentation would state that: 1. Numeric name fields start with a digit and are terminated by any non-numeric character. 2. An identifier name field is terminated by any one of: '}' (terminates the replacement field, unless preceded by a matching '{' character, in which case it is ignored and included in the string) '!' (terminates name field, starts conversion specifier) ':' (terminates name field, starts format specifier) '.' (terminates current name field, starts new name field for subattribute) '[' (terminates name field, starts index field) 3. An index field is terminated by one of: '}' (terminates the replacement field, unless preceded by a matching '{' character, in which case it is ignored and included in the string) '!' (terminates index field, starts conversion specifier) ':' (terminates index field, starts format specifier) ']' (terminates index field, subsequent character will determine next field) This existing behaviour can certainly be documented as such, but is rather unintuitive and (given that '}', '!' and ']' will always error out if appearing in an index field) somewhat silly. So, the two changes that I believe Ben is proposing would be as follows: 1. When processing a name field, brace-matching is suspended. Between the opening '{' character and the closing '}', '!' or ':' character, additional '{' characters are ignored for matching purposes. 2. When processing an index field, all special processing is suspended until the terminating ']' is reached The rules for name fields would then become: 1. Numeric fields start with a digit and are terminated by any non-numeric character. 2. An identifier name field is terminated by any one of: '}' (terminates the replacement field) '!' (terminates identifier field, starts conversion specifier) ':' (terminates identifier field, starts format specifier) '.' (terminates identifier field, starts new identifier field for subattribute) '[' (terminates identifier field, starts index field) 3. An index field is terminated by ']' (subsequent character will determine next field) That second set of rules is *far* more in line with the behaviour of the rest of the language than the status quo, so unless the difficulty of making the str.format mini-language parser work that way is truly prohibitive, it certainly seems worthwhile to tidy up the semantics. The index field behaviour should definitely be fixed, as it poses no backwards compatibility concerns. The brace matching behaviour should probably be left alone, as changing it would potentially break currently valid format strings (e.g. "{a{0}}".format(**{'a{0}':1}) produces '1' now, but would raise an exception if the brace matching rules were changed). So +1 on making the str.format parser accept anything other than ']' inside an index field and turn the whole thing into an ordinary string, -1 on making any other changes to the brace-matching behaviour. That would leave us with the following set of rules for name fields: 1. Numeric fields start with a digit and are terminated by any non-numeric character. 2. An identifier name field is terminated by any one of: '}' (terminates the replacement field, unless preceded by a matching '{' character, in which case it is ignored and included in the string) '!' (terminates identifier field, starts conversion specifier) ':' (terminates identifier field, starts format specifier) '.' (terminates identifier field, starts new identifier field for subattribute) '[' (terminates identifier field, starts index field) 3. An index field is terminated by ']' (subsequent character will determine next field) Note that brace-escaping currently doesn't work inside name fields, so that should also be fixed: >>> "{0[{{]}".format({'{':1}) Traceback (most recent call last): File "", line 1, in ValueError: unmatched '{' in format >>> "{a{{}".format(**{'a{':1}) Traceback (most recent call last): File "", line 1, in ValueError: unmatched '{' in format As far as I can recall, the details of this question didn't come up when PEP 3101 was developed, so the PEP isn't a particularly good source to justify anything in relation to this - it is best to consider the current behaviour to just be the way it happened to be implemented rather than a deliberate design choice. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From petri at digip.org Sat Jun 11 12:32:13 2011 From: petri at digip.org (Petri Lehtinen) Date: Sat, 11 Jun 2011 13:32:13 +0300 Subject: [Python-Dev] PEP 3101 implementation vs. documentation In-Reply-To: References: Message-ID: <20110611103133.GA27053@toronto> Nick Coghlan wrote: [snip] > The rules for name fields would then become: > > 1. Numeric fields start with a digit and are terminated by any > non-numeric character. > > 2. An identifier name field is terminated by any one of: > '}' (terminates the replacement field) > '!' (terminates identifier field, starts conversion specifier) > ':' (terminates identifier field, starts format specifier) > '.' (terminates identifier field, starts new identifier field for > subattribute) > '[' (terminates identifier field, starts index field) > > 3. An index field is terminated by ']' (subsequent character will > determine next field) +1 > That second set of rules is *far* more in line with the behaviour of > the rest of the language than the status quo, so unless the difficulty > of making the str.format mini-language parser work that way is truly > prohibitive, it certainly seems worthwhile to tidy up the semantics. > > The index field behaviour should definitely be fixed, as it poses no > backwards compatibility concerns. The brace matching behaviour should > probably be left alone, as changing it would potentially break > currently valid format strings (e.g. "{a{0}}".format(**{'a{0}':1}) > produces '1' now, but would raise an exception if the brace matching > rules were changed). -1 for leaving the brace matching behavior alone, as it's very unintuitive for *the user*. For the implementor it may make sense to count matching braces, but definitely not for the user. I don't believe that "{a{0}}" is a real use case that someone might already use, as it's a hard violation of what the documentation currently says. I'd rather disallow braces in the replacement field before the format specifier altogether. Or closing braces at the minimum. Furthermore, the double-escaping sounds reasonable in the format specifier, but not elsewhere. My motivation is that the user should be able to have a quick glance on the format string and see where the replacement fields are. This is probably what the PEP intends to say when disallowing braces inside the replacement field. In my opinion, it's easy to write the parser in a way that braces are parsed in any imaginable manner. Or maybe not easy, but not harder than any other way of handling braces. > So +1 on making the str.format parser accept anything other than ']' > inside an index field and turn the whole thing into an ordinary > string, -1 on making any other changes to the brace-matching > behaviour. > > That would leave us with the following set of rules for name fields: > > 1. Numeric fields start with a digit and are terminated by any > non-numeric character. > > 2. An identifier name field is terminated by any one of: > '}' (terminates the replacement field, unless preceded by a > matching '{' character, in which case it is ignored and included in > the string) > '!' (terminates identifier field, starts conversion specifier) > ':' (terminates identifier field, starts format specifier) > '.' (terminates identifier field, starts new identifier field for > subattribute) > '[' (terminates identifier field, starts index field) > > 3. An index field is terminated by ']' (subsequent character will > determine next field) > > Note that brace-escaping currently doesn't work inside name fields, so > that should also be fixed: > > >>> "{0[{{]}".format({'{':1}) > Traceback (most recent call last): > File "", line 1, in > ValueError: unmatched '{' in format > >>> "{a{{}".format(**{'a{':1}) > Traceback (most recent call last): > File "", line 1, in > ValueError: unmatched '{' in format -1. Why do we need braces inside replacement fields at all (except for inner replacements in the format specier)? I strongly believe that the PEP's use case is the simple one: '{foo}'.format(foo=10) In my opinoin, these '{!#%}'.format(**{'!#%': 10}) cases are not real. The current documentation requires field_name to be a valid identifier, an this is a sane requirement. The only problem is that parsing identifiers correctly is very hard, so it can be made simpler by allowing some non-identifiers. But we still don't have to accept braces. --- As a somewhat another issue, I'm confused about this: >>> '{a[1][2]}'.format(a={1:{2:3}}) '3' and even more about this: >>> '{a[1].foo[2]}'.format(a={1:namedtuple('x', 'foo')({2:3})}) '3' Why does this work? It's against the current documentation. The documented syntax only allows zero or one attribute names and zero or one element index, in this order. Is it intentional that we allow arbitrary chains of getattr and __getitem__? If we do, this should be documented, too. Petri From wolfson at gmail.com Sat Jun 11 18:19:50 2011 From: wolfson at gmail.com (Ben Wolfson) Date: Sat, 11 Jun 2011 09:19:50 -0700 Subject: [Python-Dev] PEP 3101 implementation vs. documentation In-Reply-To: References: Message-ID: On Sat, Jun 11, 2011 at 2:16 AM, Nick Coghlan wrote: > On Sat, Jun 11, 2011 at 7:15 AM, Ben Wolfson wrote: > To summarise (after both the above post and the discussion on the tracker) Thanks for the summary! > > That would leave us with the following set of rules for name fields: > > 1. Numeric fields start with a digit and are terminated by any > non-numeric character. > > 2. An identifier name field is terminated by any one of: > ? ?'}' (terminates the replacement field, unless preceded by a > matching '{' character, in which case it is ignored and included in > the string) > ? ?'!' (terminates identifier field, starts conversion specifier) > ? ?':' (terminates identifier field, starts format specifier) > ? ?'.' (terminates identifier field, starts new identifier field for > subattribute) > ? ?'[' (terminates identifier field, starts index field) > > 3. An index field is terminated by ']' (subsequent character will > determine next field) A minor clarification since I mentioned a patch: the patch as it exists implements *these*---Nick's---semantics. That is, it will allow these: "{0.{a}}".format(x) "{0.{[{].}}".format(x) But not this, because it keeps current brace-matching in this context: "{0.{a}".format(x) And it treats this: "{0.a}}".format(x) as the markup "{0.a}" followed by the character data "}". The patch would have to be changed to turn off brace balancing in name fields as well. In either case there would be potential breakage, since this: "{0[{}.}}".format(...) currently works, but would not work anymore, under either set of rules. (The likelihood that this potential breakage would anywhere be actual breakage is pretty slim, though.) > Note that brace-escaping currently doesn't work inside name fields, so > that should also be fixed: > >>>> "{0[{{]}".format({'{':1}) > Traceback (most recent call last): > ?File "", line 1, in > ValueError: unmatched '{' in format >>>> "{a{{}".format(**{'a{':1}) > Traceback (most recent call last): > ?File "", line 1, in > ValueError: unmatched '{' in format This is a slightly different issue, though, isn't it? As far as I can tell, if the brace-matching rules are kept in place, there would never be any *need* for escaping. You can't have an internal replacement field in this part of the replacement field, so '{' can always safely be assumed to be Just a Brace and not the start of a replacement field, regardless of whether it's doubled, and '}' will either be in an index field (where it can't have the significance of ending the replacement field) or it will be (a) the end of the replacement field or (b) not the end of the replacement field because matched by an earlier '{'. So there would never be any role for escaping to play. There would be a role for escaping if the rules for name fields are that '}' terminates them, no matching done; then, you could double them to get a '}' in the name field. But, to be honest, that strikes me as introducing a lot of heavy machinery for very little gain; opening and closing braces would have to be escaped to accomodate this one thing. And it's not as if you can escape ']' in index fields---which would be a parallel case. It seems significantly simpler to me to leave the escaping behavior as it is in this part of the replacement field. -- Ben Wolfson "Human kind has used its intelligence to vary the flavour of drinks, which may be sweet, aromatic, fermented or spirit-based. ... Family and social life also offer numerous other occasions to consume drinks for pleasure." [Larousse, "Drink" entry] From benjamin at python.org Sat Jun 11 20:28:16 2011 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 11 Jun 2011 13:28:16 -0500 Subject: [Python-Dev] [Python-checkins] cpython (3.1): onto 3.1.5 In-Reply-To: <4DF3B324.1010708@udel.edu> References: <4DF3B324.1010708@udel.edu> Message-ID: 2011/6/11 Terry Reedy : > >> +What's New in Python 3.1.5? >> +=========================== >> + >> +*Release date: XXXX-XX-XX* >> + >> +Core and Builtins >> +----------------- >> + >> +Library >> +------- >> + >> + > > I presume that only security patches should be added. Indeed. -- Regards, Benjamin From tjreedy at udel.edu Sat Jun 11 21:17:49 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 11 Jun 2011 15:17:49 -0400 Subject: [Python-Dev] PEP 3101 implementation vs. documentation In-Reply-To: <20110611103133.GA27053@toronto> References: <20110611103133.GA27053@toronto> Message-ID: On 6/11/2011 6:32 AM, Petri Lehtinen wrote: > Nick Coghlan wrote: > [snip] It seems to me that the intent of the pep and the current doc is that field_names should match what one would write in code except that quotes are left off of literal string keys. Which is to say, the brackets [] serve as quote marks. So once '[' is found, the scanner must shift to 'in index' mode and accept everything until a matching ']' is found, ending 'in index' mode. The arg_name is documented as int or identifier and attribute_name as identifier, period. Anything more than that is an implementation accident which people should not count on in either future versions or alternate implementations. I can imagine uses for nested replacement fields in the field_name or conversion spec. Ie, '{ {0}:{1}d'.format(2,5,333,444) == ' 333', whereas changing the first arg to 3 would produce ' 444'. If braces are allowed in either of the first two segments (outside of the 'quoted' within braces context), I think it should only be for the purpose of a feature addition that makes them significant. It strikes me that the underlying problem is that the replacement_field scanner is, apparently, hand-crafted rather than being generated from the corresponding grammar, as is the overall Python lexer-parser. So it has no necessary connection with the grammar. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Sun Jun 12 01:29:23 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 12 Jun 2011 11:29:23 +1200 Subject: [Python-Dev] PEP 3101 implementation vs. documentation In-Reply-To: References: Message-ID: <4DF3FA53.7000906@canterbury.ac.nz> Ben Wolfson wrote: > You can't have an internal replacement > field in this part of the replacement field, so '{' can always safely > be assumed to be Just a Brace and not the start of a replacement > field, regardless of whether it's doubled, I'm worried that the rules in this area are getting too complicated for a human to follow. If braces are allowed as plain data between square brackets and/or vice versa, it's going to be a confusing mess to read, and there will always be some doubt in the programmer's mind as to whether they have to be escaped somehow or not. I'm inclined to think that any such difficult cases should simply be disallowed. If the docs say an identifier is required someplace, the implementation should adhere strictly to that. It's not *that* hard to parse an indentifier properly, and IMO any use case that requires putting arbitrary characters into an item selector is abusing the format mechanism and should be redesigned to work some other way. -- Greg From ncoghlan at gmail.com Sun Jun 12 15:18:37 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Jun 2011 23:18:37 +1000 Subject: [Python-Dev] [Python-checkins] cpython (2.7): allow "fake" filenames in findsource (closes #9284) In-Reply-To: References: Message-ID: On Sun, Jun 12, 2011 at 6:56 AM, benjamin.peterson wrote: > summary: > ?allow "fake" filenames in findsource (closes #9284) > > This allows findsource() to work in doctests. > > A patch from Dirkjan Ochtman. Either this exception should be mentioned in the inspect.getsource() documentation or else doctest should be monkeypatching inspect as well as linecache. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From benjamin at python.org Sun Jun 12 17:31:11 2011 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 12 Jun 2011 10:31:11 -0500 Subject: [Python-Dev] [Python-checkins] cpython (2.7): allow "fake" filenames in findsource (closes #9284) In-Reply-To: References: Message-ID: 2011/6/12 Nick Coghlan : > On Sun, Jun 12, 2011 at 6:56 AM, benjamin.peterson > wrote: >> summary: >> ?allow "fake" filenames in findsource (closes #9284) >> >> This allows findsource() to work in doctests. >> >> A patch from Dirkjan Ochtman. > > Either this exception should be mentioned in the inspect.getsource() > documentation or else doctest should be monkeypatching inspect as well > as linecache. I should have made more clear in the message that this is actually a regression from 2.6. -- Regards, Benjamin From ncoghlan at gmail.com Sun Jun 12 17:55:13 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Jun 2011 01:55:13 +1000 Subject: [Python-Dev] [Python-checkins] cpython (2.7): allow "fake" filenames in findsource (closes #9284) In-Reply-To: References: Message-ID: On Mon, Jun 13, 2011 at 1:31 AM, Benjamin Peterson wrote: > I should have made more clear in the message that this is actually a > regression from 2.6. Actually looking at the inspect docs, I'm not sure where such a note would fit anyway. I'll think about it a bit more - I have a suspicion there may be a flawed assumption in that inspect code and it should be passing more queries through to linecache rather than trying to second-guess it regarding what source code is available. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From jiawei.h.li at gmail.com Sat Jun 11 10:29:16 2011 From: jiawei.h.li at gmail.com (Jiawei Li) Date: Sat, 11 Jun 2011 16:29:16 +0800 Subject: [Python-Dev] Implement Aspect-oriented programming Message-ID: For example, the logging module is not very useful right now, as it requires sprinkling small one-liners all over my code - not exactly ideal. Why not take a page from aspect-oriented programming and allow for injection of code with point cuts? -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas.lueg at googlemail.com Sun Jun 12 17:29:13 2011 From: lukas.lueg at googlemail.com (Lukas Lueg) Date: Sun, 12 Jun 2011 17:29:13 +0200 Subject: [Python-Dev] Lazy unpacking for struct module Message-ID: Hi. We extensively use the struct module to crunch large amounts of binary data. There are basically two operations for us that only seem to the naked eye as one: Filtering (see if certain fields have certain values, throw everything away if not) and inspection (care about all the fields' values). The filtering-part is very important as most of the binary data can actually be thrown away and never have to be inspected any further. When thinking about how to increase performance, one thought was that a lot of objects are generated by the struct module that we never need: Unpacking six fields in order to look at one and then throwing everything away is inefficient concerning the five other fields. It also makes filtering and inspecting basically the same operation regarding the (slow) unpacking of values so we don't really benefit from filtering. This is a huge problem when crunching gigabytes of data and creating millions of fields. One solution to this is using two format-strings instead of only one (e.g. '4s4s i 4s2s2s'): One that unpacks just the filtered fields (e.g. '8x i 8x') and one that unpacks all the fields except the one already created by the filter (e.g. '4s4s 4x 4s2s2s'). This solution works very well and increases throughput by far. It however also creates complexity in the code as we have to keep track and combine field-values that came from the filtering-part with the ones unpacked during inspection-part (we don't want to simply unpack twice). I'd like to propose an enhancement to the struct module that should solve this dilemma and ask for your comments. The function s_unpack_internal() inside _struct.c currently unpacks all values from the buffer-object passed to it and returns a tuple holding these values. Instead, the function could create a tuple-like object that holds a reference to it's own Struct-object (which holds the format) and a copy of the memory it is supposed to unpack. This object allows access to the unpacked values through the sequence protocol, basically unpacking the fields if - and only if - accessed through sq_item (e.g. foo = struct.unpack('2s2s', 'abcd'); foo[0] == 'ab'). The object can also unpack all fields only once (as all unpacked objects are immutable, we can hold references to them and return these instead once known). This approach is possible because there are no further error conditions inside the unpacking-functions that we would *have* to deal with at the time .unpack() is called; in other words: Unpacking can't fail if the format-string's syntax had been correct and can therefor be deferred (while packing can't). I understand that this may seem like a single-case-optimization. We can however assume that most people will benefit from the new behavior unknowingly while everyone else takes now harm: The object mimicking the otherwise returned tuple is immutable (therefor it's not suddenly part of GC) and the memory overhead caused by holding references to the original memory a little longer (reclaimed after the result becomes unreachable) should be comparable to the memory used by unneeded fields (reclaimed directly after creation). I'd like to hear your thoughts and am perfectly willing to provide a patch if it has a chance of inclusion. Best regards Lukas From regebro at gmail.com Sun Jun 12 18:52:53 2011 From: regebro at gmail.com (Lennart Regebro) Date: Sun, 12 Jun 2011 18:52:53 +0200 Subject: [Python-Dev] Implement Aspect-oriented programming In-Reply-To: References: Message-ID: On Sat, Jun 11, 2011 at 10:29, Jiawei Li wrote: > For example, the logging module is not very useful right now, as it requires > sprinkling small one-liners all over my code - not exactly ideal. > Why not take a page from aspect-oriented programming and allow for injection > of code with point cuts? I'm not sure why you would say this isn't allowed already... -- Lennart Regebro: http://regebro.wordpress.com/ Porting to Python 3: http://python3porting.com/ From phd at phdru.name Sun Jun 12 18:54:23 2011 From: phd at phdru.name (Oleg Broytman) Date: Sun, 12 Jun 2011 20:54:23 +0400 Subject: [Python-Dev] Implement Aspect-oriented programming In-Reply-To: References: Message-ID: <20110612165423.GA22939@iskra.aviel.ru> Hi! This mailing list is to work on developing Python (discussing bugs and patches). There is python-ideas mailing list to discuss possible future improvements. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From ethan at stoneleaf.us Sun Jun 12 19:18:24 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 12 Jun 2011 10:18:24 -0700 Subject: [Python-Dev] Python 3.x and bytes In-Reply-To: References: <4DD2C2A5.3080403@stoneleaf.us> <4DD2D89D.4000303@stoneleaf.us> <4DD2F661.2050005@stoneleaf.us> Message-ID: <4DF4F4E0.5060003@stoneleaf.us> Guido van Rossum wrote: > On Thu, May 19, 2011 at 1:43 AM, Nick Coghlan wrote: >> Proposals to address this include: >> - introduce a "character" literal to allow c'a' as an alternative to ord('a') > > -1; the result is not a *character* but an integer. I'm personally > favoring using b'a'[0] and possibly hiding this in a constant > definition. Using this method, my code now looks like: # constants EOH = b'\r'[0] CHAR = b'C'[0] DATE = b'D'[0] FLOAT = b'F'[0] INT = b'I'[0] LOGICAL = b'L'[0] MEMO = b'M'[0] NUMBER = b'N'[0] This is not beautiful code. ~Ethan~ From raymond.hettinger at gmail.com Sun Jun 12 19:27:06 2011 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 12 Jun 2011 10:27:06 -0700 Subject: [Python-Dev] Lazy unpacking for struct module In-Reply-To: References: Message-ID: <6656372E-7E82-477E-8C42-6CB686A1ADD1@gmail.com> On Jun 12, 2011, at 8:29 AM, Lukas Lueg wrote: > Hi. > > We extensively use the struct module to crunch large amounts of binary > data. There are basically two operations for us that only seem to the > naked eye as one: Filtering (see if certain fields have certain > values, throw everything away if not) and inspection (care about all > the fields' values). The filtering-part is very important as most of > the binary data can actually be thrown away and never have to be > inspected any further. When thinking about how to increase > performance, one thought was that a lot of objects are generated by > the struct module that we never need: Unpacking six fields in order to > look at one and then throwing everything away is inefficient > concerning the five other fields. It also makes filtering and > inspecting basically the same operation regarding the (slow) unpacking > of values so we don't really benefit from filtering. This is a huge > problem when crunching gigabytes of data and creating millions of > fields. > > One solution to this is using two format-strings instead of only one > (e.g. '4s4s i 4s2s2s'): One that unpacks just the filtered fields > (e.g. '8x i 8x') and one that unpacks all the fields except the one > already created by the filter (e.g. '4s4s 4x 4s2s2s'). This solution > works very well and increases throughput by far. I This is what people normally do (unpack just the values they need, when they need them). > t however also > creates complexity in the code as we have to keep track and combine > field-values that came from the filtering-part with the ones unpacked > during inspection-part (we don't want to simply unpack twice). > > I'd like to propose an enhancement to the struct module that should > solve this dilemma and ask for your comments. > > The function s_unpack_internal() inside _struct.c currently unpacks > all values from the buffer-object passed to it and returns a tuple > holding these values. Instead, the function could create a tuple-like > object that holds a reference to it's own Struct-object (which holds > the format) and a copy of the memory it is supposed to unpack. This > object allows access to the unpacked values through the sequence > protocol, basically unpacking the fields if - and only if - accessed > through sq_item (e.g. foo = struct.unpack('2s2s', 'abcd'); foo[0] == > 'ab'). The object can also unpack all fields only once (as all > unpacked objects are immutable, we can hold references to them and > return these instead once known). This approach is possible because > there are no further error conditions inside the unpacking-functions > that we would *have* to deal with at the time .unpack() is called; in > other words: Unpacking can't fail if the format-string's syntax had > been correct and can therefor be deferred (while packing can't). > > I understand that this may seem like a single-case-optimization. Yes, it does. > We > can however assume that most people will benefit from the new behavior > unknowingly while everyone else takes now harm: The object mimicking > the otherwise returned tuple is immutable (therefor it's not suddenly > part of GC) and the memory overhead caused by holding references to > the original memory a little longer (reclaimed after the result > becomes unreachable) should be comparable to the memory used by > unneeded fields (reclaimed directly after creation). > > I'd like to hear your thoughts and am perfectly willing to provide a > patch if it has a chance of inclusion. The problem you're trying to solve isn't unique to structs. That's why we get periodic requests for ropes-like behaviors for strings for example. Someone could also request lazy extraction of fields in regular expressions or lazy parses of json objects. I don't think there is a net win from adding complexity to the struct module. Introducing lazy behaviors creates its own overhead that would compete with code optimized using the traditional approach (unpack what you need, when you need it). Also, the new behaviors add to the cognitive load when learning and remembering how to use this module. In general, Python has opted for the most straight-forward, least magical implementations of object (why we use array based lists instead of blist for example). The are exceptions such as Python 3's version of super() but this isn't the norm. I do suggest that you publish your code as a third-party module to make the optional available and to validate whether there is any real interest in this. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Sun Jun 12 19:34:42 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Jun 2011 19:34:42 +0200 Subject: [Python-Dev] Python 3.x and bytes In-Reply-To: <4DF4F4E0.5060003@stoneleaf.us> References: <4DD2C2A5.3080403@stoneleaf.us> <4DD2D89D.4000303@stoneleaf.us> <4DD2F661.2050005@stoneleaf.us> <4DF4F4E0.5060003@stoneleaf.us> Message-ID: <4DF4F8B2.5050802@v.loewis.de> > # constants > > EOH = b'\r'[0] > CHAR = b'C'[0] > DATE = b'D'[0] > FLOAT = b'F'[0] > INT = b'I'[0] > LOGICAL = b'L'[0] > MEMO = b'M'[0] > NUMBER = b'N'[0] > > This is not beautiful code. In this case, I think the intent would be better captured with def ASCII(c): return c.encode('ascii') EOH = ASCII('\r') # 0D CHAR = ASCII('C') # 43 DATE = ASCII('D') # 44 FLOAT = ASCII('F') # 46 INT = ASCII('I') # 49 LOGICAL = ASCII('L') # 4C MEMO = ASCII('M') # 4D NUMBER = ASCII('N') # 4E This expresses the intent that a) these are really byte values, not characters, and b) the specific choice of byte values was motivated by ASCII. Regards, Martin From hagen at zhuliguan.net Sun Jun 12 19:46:22 2011 From: hagen at zhuliguan.net (=?ISO-8859-1?Q?Hagen_F=FCrstenau?=) Date: Sun, 12 Jun 2011 19:46:22 +0200 Subject: [Python-Dev] Python 3.x and bytes In-Reply-To: <4DF4F4E0.5060003@stoneleaf.us> References: <4DD2C2A5.3080403@stoneleaf.us> <4DD2D89D.4000303@stoneleaf.us> <4DD2F661.2050005@stoneleaf.us> <4DF4F4E0.5060003@stoneleaf.us> Message-ID: <4DF4FB6E.2030009@zhuliguan.net> > EOH = b'\r'[0] > CHAR = b'C'[0] > DATE = b'D'[0] > FLOAT = b'F'[0] > INT = b'I'[0] > LOGICAL = b'L'[0] > MEMO = b'M'[0] > NUMBER = b'N'[0] > > This is not beautiful code. You still have the alternative EOH = ord('\r') CHAR = ord('C') ... which looks fine to me. Cheers, Hagen From benjamin at python.org Sun Jun 12 19:57:20 2011 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 12 Jun 2011 12:57:20 -0500 Subject: [Python-Dev] [RELEASE] Python 2.7.2 Message-ID: On behalf of the Python development team, I'm rosy to announce the immediate availability of Python 2.7.2. Since the release candidate 2 weeks ago, there have been 2 changes: 1. pyexpat.__version__ has be changed to be the Python version. 2. A regression from 3.1.3 in the handling of comments in the netrc module has been resolved. (see issue #12009). 2.7.2 is the second in bugfix release for the Python 2.7 series. 2.7 is the last major verison of the 2.x line and will be receiving only bug fixes while new feature development focuses on 3.x. The 2.7 series includes many features that were first released in Python 3.1. The faster io module, the new nested with statement syntax, improved float repr, set literals, dictionary views, and the memoryview object have been backported from 3.1. Other features include an ordered dictionary implementation, unittests improvements, a new sysconfig module, auto-numbering of fields in the str/unicode format method, and support for ttk Tile in Tkinter. For a more extensive list of changes in 2.7, see http://doc.python.org/dev/whatsnew/2.7.html or Misc/NEWS in the Python distribution. To download Python 2.7.2 visit: http://www.python.org/download/releases/2.7.1/ The 2.7.2 changelog is at: http://hg.python.org/cpython/raw-file/eb3c9b74884c/Misc/NEWS 2.7 documentation can be found at: http://docs.python.org/2.7/ This is a production release, please report any bugs to http://bugs.python.org/ Enjoy and for those in the northern hemisphere, have a nice summer! -- Benjamin Peterson Release Manager benjamin at python.org (on behalf of the entire python-dev team and 2.7.2's contributors) From benjamin at python.org Sun Jun 12 19:58:47 2011 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 12 Jun 2011 12:58:47 -0500 Subject: [Python-Dev] [RELEASED] Python 3.1.4 Message-ID: On behalf of the Python development team, I'm sanguine to announce a release candidate for the fourth bugfix release for the Python 3.1 series, Python 3.1.4. Since the 3.1.4 release candidate 2 weeks ago, there have been three changes: 1. test_zipfile has been fixed on systems with an ASCII filesystem encoding. 2. pyexpat.__version__ has be changed to be the Python version. 3. A regression from 2.7.1 in the handling of comments in the netrc module has been resolved. (see issue #12009). 3.1.4 will the last bug fix release in the 3.1 series before 3.1. After 3.1.4, 3.1 will be in security-only fix mode. The Python 3.1 version series focuses on the stabilization and optimization of the features and changes that Python 3.0 introduced. For example, the new I/O system has been rewritten in C for speed. File system APIs that use unicode strings now handle paths with undecodable bytes in them. Other features include an ordered dictionary implementation, a condensed syntax for nested with statements, and support for ttk Tile in Tkinter. For a more extensive list of changes in 3.1, see http://doc.python.org/3.1/whatsnew/3.1.html or Misc/NEWS in the Python distribution. This is a production release. To download Python 3.1.4 visit: http://www.python.org/download/releases/3.1.4/ A list of changes in 3.1.4 can be found here: http://hg.python.org/cpython/raw-file/feae9f9e9f30/Misc/NEWS The 3.1 documentation can be found at: http://docs.python.org/3.1 Bugs can always be reported to: http://bugs.python.org Enjoy and be merry! -- Benjamin Peterson Release Manager benjamin at python.org (on behalf of the entire python-dev team and 3.1.4's contributors) From lukas.lueg at googlemail.com Sun Jun 12 19:43:39 2011 From: lukas.lueg at googlemail.com (Lukas Lueg) Date: Sun, 12 Jun 2011 19:43:39 +0200 Subject: [Python-Dev] Lazy unpacking for struct module In-Reply-To: <6656372E-7E82-477E-8C42-6CB686A1ADD1@gmail.com> References: <6656372E-7E82-477E-8C42-6CB686A1ADD1@gmail.com> Message-ID: > This is what people normally do (unpack just the values they need, > when they need them). Due to the fact that there hundreds of format-strings which dynamically compiled from a more verbose language at runtime, we will have significant complexity in the code in order to generate format strings that parse just the fields that are needed for filtering. It's not just put-a-string-here-and-there. > I don't think there is a net win from adding complexity to the struct > module. ?Introducing lazy behaviors creates its own overhead > that would compete with code optimized using the traditional > approach (unpack what you need, when you need it). ?Also, > the new behaviors add to the cognitive load when learning > and remembering how to use this module. The complexity is very well handled. Remember that the interface to the module does not change at all and the documentation would be exactly the same. There is no special case introduced here the user has to know about. I also think this case has very little black magic in it since we are dealing only with immutable objects and do not have delayed error conditions (both usually being the primary source of headaches when introducing lazy behavior). From tjreedy at udel.edu Sun Jun 12 20:28:19 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 12 Jun 2011 14:28:19 -0400 Subject: [Python-Dev] [RELEASE] Python 2.7.2 In-Reply-To: References: Message-ID: On 6/12/2011 1:57 PM, Benjamin Peterson wrote: > > To download Python 2.7.2 visit: > > http://www.python.org/download/releases/2.7.1/ That should be http://www.python.org/download/releases/2.7.2/ -- Terry Jan Reedy From p.f.moore at gmail.com Sun Jun 12 21:31:59 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 12 Jun 2011 20:31:59 +0100 Subject: [Python-Dev] [RELEASED] Python 3.1.4 In-Reply-To: References: Message-ID: On 12 June 2011 18:58, Benjamin Peterson wrote: > On behalf of the Python development team, I'm sanguine to announce a release > candidate for the fourth bugfix release for the Python 3.1 series, Python 3.1.4. Is this actually a RC, or is that a typo? Paul. From benjamin at python.org Sun Jun 12 21:52:42 2011 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 12 Jun 2011 14:52:42 -0500 Subject: [Python-Dev] [RELEASED] Python 3.1.4 In-Reply-To: References: Message-ID: 2011/6/12 Paul Moore : > On 12 June 2011 18:58, Benjamin Peterson wrote: >> On behalf of the Python development team, I'm sanguine to announce a release >> candidate for the fourth bugfix release for the Python 3.1 series, Python 3.1.4. > > Is this actually a RC, or is that a typo? That is a typo. This is a final release! > Paul. > -- Regards, Benjamin From tjreedy at udel.edu Sun Jun 12 22:31:47 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 12 Jun 2011 16:31:47 -0400 Subject: [Python-Dev] Lazy unpacking for struct module In-Reply-To: References: Message-ID: On 6/12/2011 11:29 AM, Lukas Lueg wrote: This sort of speculative idea might fit the python-ideas list better. [Summary: we often need to extract a field or two from a binary record in order to decide whether to toss it or unpack it all and process.] > One solution to this is using two format-strings instead of only one > (e.g. '4s4s i 4s2s2s'): One that unpacks just the filtered fields > (e.g. '8x i 8x') and one that unpacks all the fields except the one > already created by the filter (e.g. '4s4s 4x 4s2s2s'). This solution > works very well and increases throughput by far. It however also > creates complexity in the code as we have to keep track and combine > field-values that came from the filtering-part with the ones unpacked > during inspection-part (we don't want to simply unpack twice). With just 1 or 2 filter fields, and very many other fields, I would just unpack everything, including the filter field. I expect the extra time to do that would be comparalbe to the extra time to combine. It certainly would make your code easier. I suspect you could write a function to create the filter field only format by field number from the everything format. > I'd like to propose an enhancement to the struct module that should > solve this dilemma and ask for your comments. > > The function s_unpack_internal() inside _struct.c currently unpacks > all values from the buffer-object passed to it and returns a tuple > holding these values. Instead, the function could create a tuple-like > object that holds a reference to it's own Struct-object (which holds > the format) and a copy of the memory it is supposed to unpack. This > object allows access to the unpacked values through the sequence > protocol, basically unpacking the fields if - and only if - accessed > through sq_item (e.g. foo = struct.unpack('2s2s', 'abcd'); foo[0] == > 'ab'). The object can also unpack all fields only once (as all > unpacked objects are immutable, we can hold references to them and > return these instead once known). This approach is possible because > there are no further error conditions inside the unpacking-functions > that we would *have* to deal with at the time .unpack() is called; in > other words: Unpacking can't fail if the format-string's syntax had > been correct and can therefor be deferred (while packing can't). > > I understand that this may seem like a single-case-optimization. Yep. > We > can however assume that most people will benefit from the new behavior > unknowingly while everyone else takes now harm: I will not assume that without code and timings. I would expect that unpacking one field at a time would take longer than all at once. To me, this is the sort of thing that should be written, listed on PyPI, and tested by multiple users on multiple systems first. -- Terry Jan Reedy From benjamin at python.org Sun Jun 12 22:37:15 2011 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 12 Jun 2011 15:37:15 -0500 Subject: [Python-Dev] is anyone using Misc/RPM? Message-ID: If no one is using it, I'd like to delete it. I also don't think we should be in business of distributing distribution specific files. -- Regards, Benjamin From martin at v.loewis.de Sun Jun 12 23:39:39 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 12 Jun 2011 23:39:39 +0200 Subject: [Python-Dev] is anyone using Misc/RPM? In-Reply-To: References: Message-ID: <4DF5321B.4000603@v.loewis.de> Am 12.06.2011 22:37, schrieb Benjamin Peterson: > I also don't think we > should be in business of distributing distribution specific files. I disagree. We certainly include PCbuild/*.vcproj, and Tools/msi, which are also "distribution-specific". Likewise, we have plenty of OSX-specific files (including special-casing for specific releases thereof). So having an RPM spec file in the source doesn't sound bad to me. Of course, if it's unmaintained (in the sense that it doesn't actually work), I could agree to have it deleted. Make sure Sean Reifschneider doesn't object. People are apparently using it - at least, they report bugs against it: http://bugs.python.org/issue5776 http://bugs.python.org/issue5063 Regards, Martin From benjamin at python.org Sun Jun 12 23:44:55 2011 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 12 Jun 2011 16:44:55 -0500 Subject: [Python-Dev] is anyone using Misc/RPM? In-Reply-To: <4DF5321B.4000603@v.loewis.de> References: <4DF5321B.4000603@v.loewis.de> Message-ID: 2011/6/12 "Martin v. L?wis" : > Am 12.06.2011 22:37, schrieb Benjamin Peterson: >> I also don't think we >> should be in business of distributing distribution specific files. > > I disagree. We certainly include PCbuild/*.vcproj, and Tools/msi, > which are also "distribution-specific". Likewise, we have plenty > of OSX-specific files (including special-casing for specific releases > thereof). I should qualify: We shouldn't distribute distribution-specific files for which we don't provide binaries. -- Regards, Benjamin From greg.ewing at canterbury.ac.nz Mon Jun 13 00:12:11 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 13 Jun 2011 10:12:11 +1200 Subject: [Python-Dev] Python 3.x and bytes In-Reply-To: <4DF4F4E0.5060003@stoneleaf.us> References: <4DD2C2A5.3080403@stoneleaf.us> <4DD2D89D.4000303@stoneleaf.us> <4DD2F661.2050005@stoneleaf.us> <4DF4F4E0.5060003@stoneleaf.us> Message-ID: <4DF539BB.4070409@canterbury.ac.nz> Guido van Rossum wrote: >> On Thu, May 19, 2011 at 1:43 AM, Nick Coghlan wrote: >> >>> Proposals to address this include: >>> - introduce a "character" literal to allow c'a' as an alternative to >>> ord('a') > > -1; the result is not a *character* but an integer. Would you be happier if it were spelled i'a' instead? -- Greg From greg.ewing at canterbury.ac.nz Mon Jun 13 00:18:53 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 13 Jun 2011 10:18:53 +1200 Subject: [Python-Dev] Lazy unpacking for struct module In-Reply-To: <6656372E-7E82-477E-8C42-6CB686A1ADD1@gmail.com> References: <6656372E-7E82-477E-8C42-6CB686A1ADD1@gmail.com> Message-ID: <4DF53B4D.1000607@canterbury.ac.nz> Raymond Hettinger wrote: > The problem you're trying to solve isn't unique to structs. > That's why we get periodic requests for ropes-like behaviors I don't think he's asking for rope-like behaviour here. Rather, he's asking for iterator-like or view-like behaviour -- for the same reasons we have both zip() and izip(), for example, or that dict.items() became a view in Py3. It seems like a reasonable request to me. However, I'm wondering whether the ctypes module might already provide what he wants. -- Greg From raymond.hettinger at gmail.com Mon Jun 13 00:56:10 2011 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 12 Jun 2011 15:56:10 -0700 Subject: [Python-Dev] Lazy unpacking for struct module In-Reply-To: <4DF53B4D.1000607@canterbury.ac.nz> References: <6656372E-7E82-477E-8C42-6CB686A1ADD1@gmail.com> <4DF53B4D.1000607@canterbury.ac.nz> Message-ID: On Jun 12, 2011, at 3:18 PM, Greg Ewing wrote: > Raymond Hettinger wrote: > >> The problem you're trying to solve isn't unique to structs. >> That's why we get periodic requests for ropes-like behaviors > > I don't think he's asking for rope-like behaviour here. How would you describe the creation of a lazy result that keeps a reference to the underlying buffer? That is my understanding of how ropes work. Raymond From stephen at xemacs.org Mon Jun 13 03:42:43 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 13 Jun 2011 10:42:43 +0900 Subject: [Python-Dev] is anyone using Misc/RPM? In-Reply-To: References: <4DF5321B.4000603@v.loewis.de> Message-ID: <877h8qmrqk.fsf@uwakimon.sk.tsukuba.ac.jp> Benjamin Peterson writes: > I should qualify: We shouldn't distribute distribution-specific files > for which we don't provide binaries. Probably it belongs in a "contrib" area of the tree, but one of the things I find really annoying about distros is the way they refuse to use my perfectly good locally built Python (XEmacs, Mailman, Django, Zope, ...). Having the magic incantation to persuade them that the locally built software satisfies dependencies in the source itself is very convenient. In fact, even if you *do* provide binaries it may be useful to have both the "provided" installer configuration (which may require things like DBMSes, perhaps several of them) and a bare-bones config for DIYers to use. (Violates TOOWTDI, I know, but PBP sometimes.) From stephen at xemacs.org Mon Jun 13 03:52:14 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 13 Jun 2011 10:52:14 +0900 Subject: [Python-Dev] Python 3.x and bytes In-Reply-To: <4DF4F4E0.5060003@stoneleaf.us> References: <4DD2C2A5.3080403@stoneleaf.us> <4DD2D89D.4000303@stoneleaf.us> <4DD2F661.2050005@stoneleaf.us> <4DF4F4E0.5060003@stoneleaf.us> Message-ID: <8762oamrap.fsf@uwakimon.sk.tsukuba.ac.jp> Ethan Furman writes: > Using this method, my code now looks like: > > # constants [...] > This is not beautiful code. Put mascara on a pig, and you have a pig with mascara on, not Bette Davis. I don't necessarily think you're doing anybody a service by making the hack of using ASCII bytes as mnemonics more beautiful. I think Martin's version is as beautiful as this code should get. From ncoghlan at gmail.com Mon Jun 13 06:42:33 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Jun 2011 14:42:33 +1000 Subject: [Python-Dev] Python 3.x and bytes In-Reply-To: <4DF4F4E0.5060003@stoneleaf.us> References: <4DD2C2A5.3080403@stoneleaf.us> <4DD2D89D.4000303@stoneleaf.us> <4DD2F661.2050005@stoneleaf.us> <4DF4F4E0.5060003@stoneleaf.us> Message-ID: On Mon, Jun 13, 2011 at 3:18 AM, Ethan Furman wrote: > > This is not beautiful code. Agreed, but: EOH, CHAR, DATE, FLOAT, INT, LOGICAL, MEMO, NUMBER = b'\rCDFILMN' is a shorter way to write the same thing. Going two per line makes it easier to mentally map the characters: EOH, CHAR = b'\rC' DATE, FLOAT = b'DF' INT, LOGICAL = b'IL' MEMO, NUMBER = b'MN' Or, as a variant on Martin's solution: FORMAT_CHARS = dict( EOH = '\r', CHAR= 'C', DATE = 'D', FLOAT = 'F', INT = 'I', LOGICAL = 'L', MEMO = 'M', NUMBER = 'N' ) FORMAT_CODES = {name : char.encode('ascii') for name, char in FORMAT_CHARS} globals().update(FORMAT_CODES) Sure, there's no "one obvious way" at this stage, but that's because we don't know yet if there even *should* be an obvious way to do this (as conflating text and binary data is a bad idea in principle). By not blessing any one way of handling the situation, we give alternative solutions time to evolve naturally. If one turns out to be clearly superior to the decode/process/encode cycle then hopefully that will become clear at some point in the future. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Mon Jun 13 07:34:33 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Jun 2011 15:34:33 +1000 Subject: [Python-Dev] Lazy unpacking for struct module In-Reply-To: References: Message-ID: On Mon, Jun 13, 2011 at 6:31 AM, Terry Reedy wrote: > With just 1 or 2 filter fields, and very many other fields, I would just > unpack everything, including the filter field. I expect the extra time to do > that would be comparalbe to the extra time to combine. It certainly would > make your code easier. I suspect you could write a function to create the > filter field only format by field number from the everything format. Indeed, while the "filter format" part makes sense to me, the decision to go with field combination rather than just extracting the filter fields a second time isn't such an obvious improvement. OTOH, it also seems like this is something that could be published as a cookbook recipe that generated the appropriate filtered formats on the fly from an existing struct definition. So given format "a b c d e", it could either extract each field individually, or else be asked to generate specific formats and their complements (e.g, asking for the 2nd and 3rd field could return a 2-tuple of formats "x b c x x" and "a x x c d e"). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ocean-city at m2.ccsnet.ne.jp Mon Jun 13 07:36:48 2011 From: ocean-city at m2.ccsnet.ne.jp (Hirokazu Yamamoto) Date: Mon, 13 Jun 2011 14:36:48 +0900 Subject: [Python-Dev] Fwd: [Python-committers] Pulling from contributors repositories Message-ID: <4DF5A1F0.2030207@m2.ccsnet.ne.jp> I've read the Python-committers thread "Pulling from contributors repositories", which is about version control system. It seems there are two main issues, linear (cleaner) history on pushing, and NEWS merging. I'm newby of bazaar, but it seems to have a solution for first issue. $ bzr checkout /repo/trunk $ bzr merge /repo/feature-a $ bzr revert --forget-merges $ bzr push See http://doc.bazaar.canonical.com/latest/en/user-guide/adv_merging.html#merging-without-parents From victor.stinner at haypocalc.com Mon Jun 13 10:44:22 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 13 Jun 2011 10:44:22 +0200 Subject: [Python-Dev] Lazy unpacking for struct module In-Reply-To: <6656372E-7E82-477E-8C42-6CB686A1ADD1@gmail.com> References: <6656372E-7E82-477E-8C42-6CB686A1ADD1@gmail.com> Message-ID: <1307954662.22317.6.camel@marge> Le dimanche 12 juin 2011 ? 10:27 -0700, Raymond Hettinger a ?crit : > I do suggest that you publish your code as a third-party module > to make the optional available and to validate whether there > is any real interest in this. See the Hachoir project: it is a lazy parser supporting sub-structures (whereas struct only supports flat structures). It is difficult to implement a lazy parser: I chose to use Python generators to implement them. Hachoir should not enter Python standard library: it evoles too fast and it is too big (60K lines of Python). See also: bdec: http://www.protocollogic.com/ Construct: http://construct.wikispaces.com/ FileAlyzer: http://www.safer-networking.org/fr/filealyzer/index.html DataWorkshop: http://www.dataworkshop.de/ DataWorkshop project is dead since 2005. I don't remember if FileAlyzer is a free software or not. I agree with Raymond: struct module should be kept simple, and if you want a lazy parser: it should be a third party project. Victor From vinay_sajip at yahoo.co.uk Mon Jun 13 13:47:33 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Jun 2011 11:47:33 +0000 (UTC) Subject: [Python-Dev] In-Python virtualisation and packaging Message-ID: Back in March this year, Carl Meyer did some work to see if it was feasible to bring virtualenv functionality into Python [1] (code at [2]). Carl's changes were to Python code only, which was almost but not quite enough. I built on his changes with updates to C code in getpath.c/getpathp.c, and my code is at [3]. I've kept it synchronised with the core cpython repo, including the recently committed packaging modules. While there are issues to work through such as dealing with source builds (and no doubt plenty of others), it now seems possible to create virtual environments and install stuff into them using just the stdlib (modulo currently needing Distribute for the packages which don't yet support setup.cfg-based packaging, but it's all done automatically for a user). So you can do e.g. $ python3.3 -m virtualize /tmp/venv $ source /tmp/venv/bin/activate.sh $ pysetup3 install Mako and so on. A log of early experiments, which seems reasonably promising, is at [4]. Do people agree that it may be fitting, proper and timely to bring virtualisation into Python, and are there any fundamental flaws anyone can see with the approach used? If people want to experiment with this code without cloning and building, I created a Debian package using checkinstall, which can be installed using sudo dpkg -i pythonv_3.3-1_i386.deb and removed using sudo dpkg -r pythonv I can make this Debian package available for download, if anyone wants it. Regards, Vinay Sajip [1] http://mail.python.org/pipermail/distutils-sig/2011-March/017519.html [2] https://bitbucket.org/carljm/cpythonv [3] https://bitbucket.org/vinay.sajip/pythonv [4] https://gist.github.com/1022601 From fuzzyman at voidspace.org.uk Mon Jun 13 13:55:52 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 13 Jun 2011 12:55:52 +0100 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: References: Message-ID: <4DF5FAC8.4090606@voidspace.org.uk> On 13/06/2011 12:47, Vinay Sajip wrote: > Back in March this year, Carl Meyer did some work to see if it was feasible > to bring virtualenv functionality into Python [1] (code at [2]). > > Carl's changes were to Python code only, which was almost but not quite enough. > I built on his changes with updates to C code in getpath.c/getpathp.c, and my > code is at [3]. I've kept it synchronised with the core cpython repo, including > the recently committed packaging modules. > > While there are issues to work through such as dealing with source builds (and > no doubt plenty of others), it now seems possible to create virtual > environments and install stuff into them using just the stdlib (modulo > currently needing Distribute for the packages which don't yet support > setup.cfg-based packaging, but it's all done automatically for a user). So you > can do e.g. > > $ python3.3 -m virtualize /tmp/venv > $ source /tmp/venv/bin/activate.sh > $ pysetup3 install Mako > > and so on. A log of early experiments, which seems reasonably promising, is at > [4]. > > Do people agree that it may be fitting, proper and timely to bring > virtualisation into Python, and are there any fundamental flaws anyone can see > with the approach used? It would certainly need a PEP. There are two options: Bring the full functionality into the standard library so that Python supports virtual environments out of the box. As is the case with adding anything to the standard library it will then be impossible to add features to the virtualization support in Python 3.3 once 3.3 is released - new features will go into 3.4. Add only the minimal changes required to support a third-party virtual environment tool. Virtual environments are phenomenally useful, so I would support having the full tool in the standard library, but it does raise maintenance and development issues. Don't forget windows support! ;-) All the best, Michael Foord > If people want to experiment with this code without cloning and building, I > created a Debian package using checkinstall, which can be installed using > > sudo dpkg -i pythonv_3.3-1_i386.deb > > and removed using > > sudo dpkg -r pythonv > > I can make this Debian package available for download, if anyone wants it. > > Regards, > > Vinay Sajip > > [1] http://mail.python.org/pipermail/distutils-sig/2011-March/017519.html > [2] https://bitbucket.org/carljm/cpythonv > [3] https://bitbucket.org/vinay.sajip/pythonv > [4] https://gist.github.com/1022601 > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From ncoghlan at gmail.com Mon Jun 13 14:09:14 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Jun 2011 22:09:14 +1000 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: <4DF5FAC8.4090606@voidspace.org.uk> References: <4DF5FAC8.4090606@voidspace.org.uk> Message-ID: On Mon, Jun 13, 2011 at 9:55 PM, Michael Foord wrote: > Virtual environments are phenomenally useful, so I would support having the > full tool in the standard library, but it does raise maintenance and > development issues. > > Don't forget windows support! ;-) Given that it is desirable for tools like virtualenv to support *old* versions of Python on *new* versions of operating systems, it seems to me that there is an inherent element of their feature set that makes including the whole tool questionable. OTOH, it may make sense to have a baseline tool provided innately, but provide the appropriate third party hooks to allow alternative tools to evolve independently of the stdlib. How well does the regression test suite cope when run inside such a virtualised environment? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Mon Jun 13 14:22:32 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Jun 2011 12:22:32 +0000 (UTC) Subject: [Python-Dev] In-Python virtualisation and packaging References: <4DF5FAC8.4090606@voidspace.org.uk> Message-ID: Michael Foord voidspace.org.uk> writes: > It would certainly need a PEP. Of course. > There are two options: > > Bring the full functionality into the standard library so that Python > supports virtual environments out of the box. As is the case with adding > anything to the standard library it will then be impossible to add > features to the virtualization support in Python 3.3 once 3.3 is > released - new features will go into 3.4. > > Add only the minimal changes required to support a third-party virtual > environment tool. > Agreed. As I see it, the "minimal changes required" are everything in my fork except for "virtualize.py", which was actually written as an external module "pmv.py" - Poor Man's Virtualenv ;-) Having it as a minimal implementation in the stdlib accords with "batteries included", but even as it stands, virtualize.py does try to cater for customisation. Firstly, there's a virtualizer_factory callable which can be overridden for customisation. That's called to produce a virtualizer, whose virtualize method is called with the target directory. The virtualize() function in virtualize.py just does this set of steps. I can't claim to have thought of everything, but it's a simple API which could have any number of implementations, not just the default one in the Virtualizer class in virtualize.py. > Don't forget windows support! I haven't. Though I haven't tested the most recent changes on Windows yet, I have tested the basic approach under Windows (the code doesn't rely on symlinks, but rather, copies of executables/DLLs). (All Windows testing so far has admittedly been using source builds rather than via a binary installer.) Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Mon Jun 13 14:50:01 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Jun 2011 12:50:01 +0000 (UTC) Subject: [Python-Dev] In-Python virtualisation and packaging References: <4DF5FAC8.4090606@voidspace.org.uk> Message-ID: Nick Coghlan gmail.com> writes: > Given that it is desirable for tools like virtualenv to support *old* > versions of Python on *new* versions of operating systems, it seems to > me that there is an inherent element of their feature set that makes > including the whole tool questionable. You're right in terms of the current Python ecosystem and 3.x adoption, because of course this approach requires support from Python itself in terms of its site.py code. However, virtual environments have a utility beyond supporting older Pythons on newer OSes, since another common use case is having different library environments sandboxed from each other on different projects, even if all those projects are using Python 3.3+. The virtualenv module does an intricate bootstrapping dance which needs to accommodate each successive Python version, so there's maintenance overhead that way, too. Carl Meyer, being a pip and virtualenv maintainer, will probably have useful views on this. > OTOH, it may make sense to have a baseline tool provided innately, but > provide the appropriate third party hooks to allow alternative tools > to evolve independently of the stdlib. Yes - I'm thinking that what I've proposed is the baseline tool, and the question is about what the virtualisation API needs to look like to allow third-party tools to progress independently of the stdlib but in an interoperable way (a bit like packaging, I suppose). > How well does the regression test suite cope when run inside such a > virtualised environment? https://gist.github.com/1022705 325 tests OK. 5 tests failed: test_email test_importlib test_lib2to3 test_packaging test_sysconfig test_importlib might be broken because I accidentally committed some changes to marshal.c while working on #12291. test_packaging fails because of #12313. test_email fails for a similar reason - Makefile.pre.in is missing test_email in LIBSUBDIRS. test_sysconfig is probably failing because of changes I made, and I'm not sure of test_lib2to3. I will investigate! Regards, Vinay Sajip From solipsis at pitrou.net Mon Jun 13 14:57:36 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 13 Jun 2011 14:57:36 +0200 Subject: [Python-Dev] In-Python virtualisation and packaging References: Message-ID: <20110613145736.66bd8f79@pitrou.net> On Mon, 13 Jun 2011 11:47:33 +0000 (UTC) Vinay Sajip wrote: > > $ python3.3 -m virtualize /tmp/venv > $ source /tmp/venv/bin/activate.sh > $ pysetup3 install Mako > > and so on. A log of early experiments, which seems reasonably promising, is at > [4]. > > Do people agree that it may be fitting, proper and timely to bring > virtualisation into Python, and are there any fundamental flaws anyone can see > with the approach used? This sounds really great, and definitely needs a PEP so that we can ask many questions :) As a side-note, I think calling it "virtualization" is a recipe for confusion. Regards Antoine. From ncoghlan at gmail.com Mon Jun 13 15:04:52 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Jun 2011 23:04:52 +1000 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: <20110613145736.66bd8f79@pitrou.net> References: <20110613145736.66bd8f79@pitrou.net> Message-ID: On Mon, Jun 13, 2011 at 10:57 PM, Antoine Pitrou wrote: > As a side-note, I think calling it "virtualization" is a recipe for > confusion. Indeed, OS level virtualisation pretty much has a lock on that term. "virtual environments" skates close to it but manages to avoid it well enough to avoid confusion. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Mon Jun 13 15:07:11 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Jun 2011 23:07:11 +1000 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: References: <4DF5FAC8.4090606@voidspace.org.uk> Message-ID: On Mon, Jun 13, 2011 at 10:50 PM, Vinay Sajip wrote: > You're right in terms of the current Python ecosystem and 3.x adoption, because > of course this approach requires support from Python itself in terms of its > site.py code. However, virtual environments have a utility beyond supporting > older Pythons on newer OSes, since another common use case is having different > library environments sandboxed from each other on different projects, even if > all those projects are using Python 3.3+. Yeah, even if the innate one struggles on later OS releases that changed things in a backwards incompatible way, it will still be valuable on the OS versions that are around at the time that version of Python gets released. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Mon Jun 13 15:09:00 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Jun 2011 23:09:00 +1000 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: References: <4DF5FAC8.4090606@voidspace.org.uk> Message-ID: On Mon, Jun 13, 2011 at 10:22 PM, Vinay Sajip wrote: > Michael Foord voidspace.org.uk> writes: >> Don't forget windows support! > > I haven't. Though I haven't tested the most recent changes on Windows yet, I > have tested the basic approach under Windows (the code doesn't rely on > symlinks, but rather, copies of executables/DLLs). (All Windows testing so far > has admittedly been using source builds rather than via a binary installer.) You should be able to use symlinks even on Windows these days (although granted they won't on portable media that uses a non-symlink friendly filesystem, regardless of OS). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Mon Jun 13 15:42:06 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Jun 2011 13:42:06 +0000 (UTC) Subject: [Python-Dev] In-Python virtualisation and packaging References: <4DF5FAC8.4090606@voidspace.org.uk> Message-ID: Nick Coghlan gmail.com> writes: > You should be able to use symlinks even on Windows these days > (although granted they won't on portable media that uses a non-symlink > friendly filesystem, regardless of OS). Plus I'm not sure Windows XP supports true symlinks - I think you have to make do with "junctions" a.k.a. "reparse points" which are more shambolic than symbolic ;-) I know symlinks are available on Vista, Windows Server 2008 and later, but XP is still very common. Regards, Vinay Sajip From brian.curtin at gmail.com Mon Jun 13 15:47:57 2011 From: brian.curtin at gmail.com (Brian Curtin) Date: Mon, 13 Jun 2011 08:47:57 -0500 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: References: <4DF5FAC8.4090606@voidspace.org.uk> Message-ID: On Mon, Jun 13, 2011 at 08:42, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > > > You should be able to use symlinks even on Windows these days > > (although granted they won't on portable media that uses a non-symlink > > friendly filesystem, regardless of OS). > > Plus I'm not sure Windows XP supports true symlinks - I think you have to > make > do with "junctions" a.k.a. "reparse points" which are more shambolic than > symbolic ;-) I know symlinks are available on Vista, Windows Server 2008 > and > later, but XP is still very common. I don't think we have any stdlib support for junctions, although we could certainly add it. In 3.2 we added symlink support for files and directories, which as you say is a Vista and beyond feature. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Mon Jun 13 15:47:06 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Jun 2011 13:47:06 +0000 (UTC) Subject: [Python-Dev] In-Python virtualisation and packaging References: <20110613145736.66bd8f79@pitrou.net> Message-ID: Nick Coghlan gmail.com> writes: > On Mon, Jun 13, 2011 at 10:57 PM, Antoine Pitrou pitrou.net> > wrote: > > As a side-note, I think calling it "virtualization" is a recipe for > > confusion. > > Indeed, OS level virtualisation pretty much has a lock on that term. > "virtual environments" skates close to it but manages to avoid it well > enough to avoid confusion. Or as they involving encapsulating paths and libaries, perhaps we could call them "capsules" ;-) though I think the term virtualenv is pretty entrenched now in the Python community. Regards, Vinay Sajip From barry at python.org Mon Jun 13 17:02:24 2011 From: barry at python.org (Barry Warsaw) Date: Mon, 13 Jun 2011 11:02:24 -0400 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: References: Message-ID: <20110613110224.05d6962e@neurotica.wooz.org> On Jun 13, 2011, at 11:47 AM, Vinay Sajip wrote: >Do people agree that it may be fitting, proper and timely to bring >virtualisation into Python, and are there any fundamental flaws anyone can see >with the approach used? Yes, absolutely. We'll hash out the details when the PEP is published, and bikeshed on all the terminology, but I really think this would be a very powerful addition to the standard library, so +1. Hopefully, the maintenance issues can be sorted out. Question: how hard would it be to backport the work you've done to Python 3.2? Obviously I'm not saying it should be ported to the official 3.2 branch, but if *someone* were interested in doing so, would it be possible? Sounds like you can almost get there with stdlib changes, but would require a few C changes too (I haven't looked at the diff yet). I'm just wondering if the same API could be made available to Python 3.2 as a third party module. It sounds like "almost, but not quite". >If people want to experiment with this code without cloning and building, I >created a Debian package using checkinstall, which can be installed using > >sudo dpkg -i pythonv_3.3-1_i386.deb > >and removed using > >sudo dpkg -r pythonv > >I can make this Debian package available for download, if anyone wants it. Is the Debian packaging branch available too? I'd be happy to throw that in my PPA for Ubuntu users to play with. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From vinay_sajip at yahoo.co.uk Mon Jun 13 18:00:30 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Jun 2011 16:00:30 +0000 (UTC) Subject: [Python-Dev] In-Python virtualisation and packaging References: <20110613110224.05d6962e@neurotica.wooz.org> Message-ID: Barry Warsaw python.org> writes: > Question: how hard would it be to backport the work you've done to Python 3.2? > Obviously I'm not saying it should be ported to the official 3.2 branch, but > if *someone* were interested in doing so, would it be possible? Sounds like > you can almost get there with stdlib changes, but would require a few C > changes too (I haven't looked at the diff yet). I'm just wondering if the > same API could be made available to Python 3.2 as a third party module. It > sounds like "almost, but not quite". I think it's feasible - as far as I know, there's nothing 3.3 specific about the changes that were made, other than just happening to be against the default branch. AFAIK the getpath.c/getpathp.c changes will also work on 3.2, as all they do is look for a config file in a specific place and read a path from it if it's there. If it's not there, no biggie. If it's there, it sets up the sys.prefix/sys.exec_prefix values from that path. Possibly Carl's original Python changes would be easier to work from, since the sysconfig stuff has now changed quite a bit because of packaging coming in to cpython. For one thing, the _INSTALL_SCHEMES dict is replaced by reading that data from a config file. > Is the Debian packaging branch available too? I'd be happy to throw that in > my PPA for Ubuntu users to play with. My Debian-packaging-fu is not that good, I'm afraid, so there's no branch for the .deb, as such. I made the package by running make and then sudo checkinstall -D --fstrans=no which takes forever (God knows why - it's many many minutes at 100% CPU) but eventually comes up with the .deb. Regards, Vinay Sajip From dmalcolm at redhat.com Mon Jun 13 18:21:19 2011 From: dmalcolm at redhat.com (David Malcolm) Date: Mon, 13 Jun 2011 12:21:19 -0400 Subject: [Python-Dev] is anyone using Misc/RPM? In-Reply-To: References: Message-ID: <1307982079.16924.5309.camel@surprise> On Sun, 2011-06-12 at 15:37 -0500, Benjamin Peterson wrote: > If no one is using it, I'd like to delete it. I also don't think we > should be in business of distributing distribution specific files. FWIW, Fedora and RHEL don't use this particular .spec file; we roll our own. I can't speak for all of the other RPM-using distributions, of course. From vinay_sajip at yahoo.co.uk Mon Jun 13 18:25:11 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Jun 2011 16:25:11 +0000 (UTC) Subject: [Python-Dev] Implement Aspect-oriented programming References: Message-ID: Jiawei Li gmail.com> writes: > For example, the logging module is not very useful right now, as it requires sprinkling small one-liners all over my code - not exactly ideal. > Why not take a page from aspect-oriented programming and allow for injection of code with point cuts? If you're only interested in logging method entry and exit - in which case, you're not really using logging to its full potential - then an AOP style approach may work for you. But the point of logging is to send messages to yourself (and others) from your code, and an AOP approach will not lend itself to intelligent, context-sensitive messages. Regards, Vinay Sajip From barry at python.org Mon Jun 13 18:41:43 2011 From: barry at python.org (Barry Warsaw) Date: Mon, 13 Jun 2011 12:41:43 -0400 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: References: <20110613110224.05d6962e@neurotica.wooz.org> Message-ID: <20110613124143.1ca85583@neurotica.wooz.org> On Jun 13, 2011, at 04:00 PM, Vinay Sajip wrote: >My Debian-packaging-fu is not that good, I'm afraid, so there's no branch for >the .deb, as such. I made the package by running make and then > >sudo checkinstall -D --fstrans=no > >which takes forever (God knows why - it's many many minutes at 100% CPU) but >eventually comes up with the .deb. Ah, no I don't think that'll be helpful. I can probably reuse the python3.3 packaging stuff to do a PPA. (It takes that long because it basically does a `make install`.) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From opensourcesurvey at cs.ua.edu Mon Jun 13 18:45:55 2011 From: opensourcesurvey at cs.ua.edu (Jeffrey Carver) Date: Mon, 13 Jun 2011 11:45:55 -0500 Subject: [Python-Dev] Participation Requested: Survey about Open-Source Software Development Message-ID: <00c601cc29e9$5d706eb0$18514c10$@cs.ua.edu> Hi, Drs. Jeffrey Carver, Rosanna Guadagno, Debra McCallum, and Mr. Amiangshu Bosu, University of Alabama, and Dr. Lorin Hochstein, University of Southern California, are conducting a survey of open-source software developers. This survey seeks to understand how developers on distributed, virtual teams, like open-source projects, interact with each other to accomplish their tasks. You must be at least 19 years of age to complete the survey. The survey should take approximately 15 minutes to complete. If you are actively participating as a developer, please consider completing our survey. Here is the link to the survey: http://goo.gl/HQnux We apologize for inconvenience and if you receive multiple copies of this email. This survey has been approved by The University of Alabama IRB board. Thanks, Dr. Jeffrey Carver Assistant Professor University of Alabama (v) 205-348-9829 (f) 205-348-0219 http://www.cs.ua.edu/~carver From vinay_sajip at yahoo.co.uk Mon Jun 13 19:31:27 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Jun 2011 17:31:27 +0000 (UTC) Subject: [Python-Dev] In-Python virtualisation and packaging References: <20110613110224.05d6962e@neurotica.wooz.org> <20110613124143.1ca85583@neurotica.wooz.org> Message-ID: Barry Warsaw python.org> writes: > Ah, no I don't think that'll be helpful. I can probably reuse the python3.3 > packaging stuff to do a PPA. Okay, go for it! Is there a specific tutorial somewhere about making a PPA for Python? (As opposed to more generalised tutorials - or would they be sufficient?) > (It takes that long because it basically does a > `make install`.) I realise that, as well as recording what it's doing, but that part seems to happen fairly quickly. Then it says "Copying files to the temporary directory..." and that part seems to take forever. The whole deb is under 25MB, what could be taking many minutes? Regards, Vinay From martin at v.loewis.de Mon Jun 13 21:03:18 2011 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 13 Jun 2011 21:03:18 +0200 Subject: [Python-Dev] is anyone using Misc/RPM? In-Reply-To: References: <4DF5321B.4000603@v.loewis.de> Message-ID: <4DF65EF6.6020507@v.loewis.de> >> I disagree. We certainly include PCbuild/*.vcproj, and Tools/msi, >> which are also "distribution-specific". Likewise, we have plenty >> of OSX-specific files (including special-casing for specific releases >> thereof). > > I should qualify: We shouldn't distribute distribution-specific files > for which we don't provide binaries. Hmm. We have VS6, VS7, and VS8 project files, OS/2 makefiles, and configure.in has specifics for Solaris, even though we don't provide binaries for any of these. I don't think that's a useful principle to follow. Regards, Martin From martin at v.loewis.de Mon Jun 13 21:36:24 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 13 Jun 2011 21:36:24 +0200 Subject: [Python-Dev] is anyone using Misc/RPM? In-Reply-To: <1307982079.16924.5309.camel@surprise> References: <1307982079.16924.5309.camel@surprise> Message-ID: <4DF666B8.7060205@v.loewis.de> Am 13.06.2011 18:21, schrieb David Malcolm: > On Sun, 2011-06-12 at 15:37 -0500, Benjamin Peterson wrote: >> If no one is using it, I'd like to delete it. I also don't think we >> should be in business of distributing distribution specific files. > > FWIW, Fedora and RHEL don't use this particular .spec file; we roll our > own. > > I can't speak for all of the other RPM-using distributions, of course. I doubt any of the distributions uses it, but that's not it's purpose, either. Instead, it is meant for people who want to roll their own RPM. Regards, Martin From wolfson at gmail.com Mon Jun 13 22:00:36 2011 From: wolfson at gmail.com (Ben Wolfson) Date: Mon, 13 Jun 2011 13:00:36 -0700 Subject: [Python-Dev] PEP 3101 implementation vs. documentation In-Reply-To: <4DF3FA53.7000906@canterbury.ac.nz> References: <4DF3FA53.7000906@canterbury.ac.nz> Message-ID: On Sat, Jun 11, 2011 at 4:29 PM, Greg Ewing wrote: > Ben Wolfson wrote: >> >> You can't have an internal replacement >> field in this part of the replacement field, so '{' can always safely >> be assumed to be Just a Brace and not the start of a replacement >> field, regardless of whether it's doubled, > > I'm worried that the rules in this area are getting too > complicated for a human to follow. If braces are allowed > as plain data between square brackets and/or vice versa, > it's going to be a confusing mess to read, and there will > always be some doubt in the programmer's mind as to whether > they have to be escaped somehow or not. > > I'm inclined to think that any such difficult cases should > simply be disallowed. If the docs say an identifier is required > someplace, the implementation should adhere strictly to that. There are two cases with the braces: attribute selection and item selection. The docs say that attributes should be identifiers, and that the argument name should be an integer or an identifier, but that the item selector can essentially be an arbitrary string as long as it doesn't contain ']', which indicates its end. The docs as they stand suggest that braces in item selectors should be treated as plain data: """ Format strings contain ?replacement fields? surrounded by curly braces {}. Anything that is not contained in braces is considered literal text, which is copied unchanged to the output. If you need to include a brace character in the literal text, it can be escaped by doubling: {{ and }}. """ Since it mentions escaping only in the context of how to get a brace in literal text rather than in a replacement field. Current behavior is to perform escapes in the format spec part of a replacement field, and that, I think, makes sense, since there can be an internal replacement field there. However, it's still pretty simple to tell whether braces need to be escaped or not: a brace in the format spec does need to be escaped, a brace before the format spec doesn't. > It's not *that* hard to parse an indentifier properly, and > IMO any use case that requires putting arbitrary characters > into an item selector is abusing the format mechanism and > should be redesigned to work some other way. If by "item selector" you mean (using the names from the grammar in the docs) the element_index, I don't see why this should be the case; dictionaries can contain non-identified keys, after all. If you mean the attribute_name (and arg_name) parts, then requiring an identifier (or an integer for arg_name) makes a lot more sense. I assume that Talin had some reason for stating otherwise in the PEP (one of the few things that does get explicitly said about the field_name part), but I'm kind of at a loss for why; you would need to have a custom __getattribute__ to exploit it, and it would be a lot less confusing just to use __getitem__. -- Ben Wolfson "Human kind has used its intelligence to vary the flavour of drinks, which may be sweet, aromatic, fermented or spirit-based. ... Family and social life also offer numerous other occasions to consume drinks for pleasure." [Larousse, "Drink" entry] From solipsis at pitrou.net Mon Jun 13 22:32:00 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 13 Jun 2011 22:32:00 +0200 Subject: [Python-Dev] is anyone using Misc/RPM? References: <4DF5321B.4000603@v.loewis.de> <4DF65EF6.6020507@v.loewis.de> Message-ID: <20110613223200.3039a8d1@pitrou.net> On Mon, 13 Jun 2011 21:03:18 +0200 "Martin v. L?wis" wrote: > >> I disagree. We certainly include PCbuild/*.vcproj, and Tools/msi, > >> which are also "distribution-specific". Likewise, we have plenty > >> of OSX-specific files (including special-casing for specific releases > >> thereof). > > > > I should qualify: We shouldn't distribute distribution-specific files > > for which we don't provide binaries. > > Hmm. We have VS6, VS7, and VS8 project files, OS/2 makefiles, and > configure.in has specifics for Solaris, even though we don't provide > binaries for any of these. I don't think that's a useful principle > to follow. Well, if we want to nitpick, all these files are for compilation, no for distribution ;) Regards Antoine. From greg.ewing at canterbury.ac.nz Mon Jun 13 23:54:17 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 14 Jun 2011 09:54:17 +1200 Subject: [Python-Dev] Lazy unpacking for struct module In-Reply-To: References: <6656372E-7E82-477E-8C42-6CB686A1ADD1@gmail.com> <4DF53B4D.1000607@canterbury.ac.nz> Message-ID: <4DF68709.8070206@canterbury.ac.nz> Raymond Hettinger wrote: > How would you describe the creation of a lazy result > that keeps a reference to the underlying buffer? I'd call it a view. There is plenty of precedence for this kind of object in Python -- I gave a few examples before. The distinguishing feature of ropes, as I understand the term, is that you get a lazy object when you *combine* other objects, e.g. concatenating strings. That's not being asked for here -- there is only taking apart going on, not putting together. It's also different from the oft-rejected idea of lazy string slicing, because extracting a field would give you a separate object, e.g. an int or string, not a reference to the original data object being unpacked. So I really can't see what harm it could do, except for maybe a tiny performance reduction in the case where you extract all the fields, or refer to extracted fields repeatedly. Even if that turned out to be the case, and was considered objectionable, it could still be useful to provide view-oriented unpacking as an option. -- Greg From ethan at stoneleaf.us Tue Jun 14 00:11:15 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 13 Jun 2011 15:11:15 -0700 Subject: [Python-Dev] Python 3.x and bytes In-Reply-To: <4DF4F4E0.5060003@stoneleaf.us> References: <4DD2C2A5.3080403@stoneleaf.us> <4DD2D89D.4000303@stoneleaf.us> <4DD2F661.2050005@stoneleaf.us> <4DF4F4E0.5060003@stoneleaf.us> Message-ID: <4DF68B03.2010603@stoneleaf.us> Thank you all for the responses. Rather than reply to each, I just made one big summary. :) ---------------------------------------------------------------- Martin v. L?wis wrote: > Ethan Furman wrote: >> # constants >> >> EOH = b'\r'[0] >> CHAR = b'C'[0] >> DATE = b'D'[0] >> FLOAT = b'F'[0] >> INT = b'I'[0] >> LOGICAL = b'L'[0] >> MEMO = b'M'[0] >> NUMBER = b'N'[0] >> >> This is not beautiful code. > > In this case, I think the intent would be better captured with > > def ASCII(c): > return c.encode('ascii') > > EOH = ASCII('\r') # 0D > CHAR = ASCII('C') # 43 > DATE = ASCII('D') # 44 > FLOAT = ASCII('F') # 46 > INT = ASCII('I') # 49 > LOGICAL = ASCII('L') # 4C > MEMO = ASCII('M') # 4D > NUMBER = ASCII('N') # 4E > > This expresses the intent that a) these are really byte values, > not characters, and b) the specific choice of byte values was > motivated by ASCII. Definitely easier to read. If I go this route I'll probably use ord(), though, since ascii and unicode are the same for the first 127 chars, and there will be plenty of places to error out with a more appropriate message if I get garbage. Since I really don't care what the actual integer values are, I'll skip those comments, too. ---------------------------------------------------------------- Hagen F?rstenau wrote: > You still have the alternative > > EOH = ord('\r') > CHAR = ord('C') > ... > > which looks fine to me. Yes it does. I just dislike the (to me unnecessary) extra function call. For those tuning in late to this thread, these are workarounds for this not working: field_type = header[11] # field_type is now an int, not a 1-byte bstr if field_type == r'C': # r'C' is a 1-byte bstr, so this always fails ---------------------------------------------------------------- Greg Ewing wrote: > Guido van Rossum wrote: >>> On Thu, May 19, 2011 at 1:43 AM, Nick Coghlan wrote: >>>> Proposals to address this include: >>>> - introduce a "character" literal to allow c'a' as an alternative >>>> to ord('a') >> >> -1; the result is not a *character* but an integer. > > Would you be happier if it were spelled i'a' instead? That would work for me, although I would prefer a'a' (for ASCII). :) ---------------------------------------------------------------- Stephen J. Turnbull wrote: > Put mascara on a pig, and you have a pig with mascara on, not Bette > Davis. I don't necessarily think you're doing anybody a service by > making the hack of using ASCII bytes as mnemonics more beautiful. I > think Martin's version is as beautiful as this code should get. I'll either use Martin's or Nick's. The point of beauty here is the ease of readability. I think less readable is worse, and we shouldn't have to have ugly, hard to read code nor inefficient code just because we have to deal with byte streams that aren't unicode. ---------------------------------------------------------------- Nick Coghlan wrote: > Agreed, but: > > EOH, CHAR, DATE, FLOAT, INT, LOGICAL, MEMO, NUMBER = b'\rCDFILMN' > > is a shorter way to write the same thing. > > Going two per line makes it easier to mentally map the characters: > > EOH, CHAR = b'\rC' > DATE, FLOAT = b'DF' > INT, LOGICAL = b'IL' > MEMO, NUMBER = b'MN' Wow. I didn't realize that could be done. That very nearly makes up for not being able to do it one char at a time. Thanks, Nick! ---------------------------------------------------------------- ~Ethan~ From carl at oddbird.net Tue Jun 14 01:27:05 2011 From: carl at oddbird.net (Carl Meyer) Date: Mon, 13 Jun 2011 18:27:05 -0500 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: <4DF5FAC8.4090606@voidspace.org.uk> References: <4DF5FAC8.4090606@voidspace.org.uk> Message-ID: <4DF69CC9.9000704@oddbird.net> On 06/13/2011 06:55 AM, Michael Foord wrote: > There are two options: > > Bring the full functionality into the standard library so that Python > supports virtual environments out of the box. As is the case with adding > anything to the standard library it will then be impossible to add > features to the virtualization support in Python 3.3 once 3.3 is > released - new features will go into 3.4. I think it's not hard to provide enough hooks to allow third-party tools to extend the virtualenv-creation process, while still having enough code in the stdlib to allow actual creation of virtualenvs. Virtualenv already has very few features, and doesn't get very much by way of new feature requests -- all the UI sugar and significant shell integration goes into Doug Hellmann's virtualenvwrapper, and I wouldn't foresee that changing. IOW, I don't think the maintenance concerns outweigh the benefits of being able to create virtualenvs with an out-of-the-box Python. > Add only the minimal changes required to support a third-party virtual > environment tool. > > Virtual environments are phenomenally useful, so I would support having > the full tool in the standard library, but it does raise maintenance and > development issues. > > Don't forget windows support! ;-) > > All the best, > > Michael Foord From carl at oddbird.net Tue Jun 14 01:46:33 2011 From: carl at oddbird.net (Carl Meyer) Date: Mon, 13 Jun 2011 18:46:33 -0500 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: References: <4DF5FAC8.4090606@voidspace.org.uk> Message-ID: <4DF6A159.1010105@oddbird.net> On 06/13/2011 08:07 AM, Nick Coghlan wrote: > On Mon, Jun 13, 2011 at 10:50 PM, Vinay Sajip wrote: >> You're right in terms of the current Python ecosystem and 3.x adoption, because >> of course this approach requires support from Python itself in terms of its >> site.py code. However, virtual environments have a utility beyond supporting >> older Pythons on newer OSes, since another common use case is having different >> library environments sandboxed from each other on different projects, even if >> all those projects are using Python 3.3+. > > Yeah, even if the innate one struggles on later OS releases that > changed things in a backwards incompatible way, it will still be > valuable on the OS versions that are around at the time that version > of Python gets released. FWIW, historically pretty much every new Python version has broken virtualenv, and new OS versions almost never have. Virtualenv isn't especially OS-dependent (not nearly as much as some other stdlib modules): the most OS-dependent piece is "shell activation", and that's a feature I would prefer to entirely leave out of the stdlib virtualenv (it's a convenience, not a necessity for virtualenv use, and the need to maintain it for a variety of OS shells is a maintenance burden I don't think Python should adopt). In fact, the only new-OS-version adjustment I can recall virtualenv needing to make is when Debian introduced dist-packages -- but even that doesn't really apply, as that was distro-packager change to Python itself. With a built-in virtualenv it would be the distro packagers responsibility to make sure their patch to Python doesn't break the virtualenv module. So I don't think a virtualenv stdlib module would be at all likely to break on a new OS release, if Python itself is not broken by that OS release. (It certainly wouldn't be the stdlib module most likely to be broken by OS changes, in comparison to e.g. shutil, threading...) Carl From pje at telecommunity.com Tue Jun 14 01:40:29 2011 From: pje at telecommunity.com (P.J. Eby) Date: Mon, 13 Jun 2011 19:40:29 -0400 Subject: [Python-Dev] Python 3.x and bytes In-Reply-To: <4DF68B03.2010603@stoneleaf.us> References: <4DD2C2A5.3080403@stoneleaf.us> <4DD2D89D.4000303@stoneleaf.us> <4DD2F661.2050005@stoneleaf.us> <4DF4F4E0.5060003@stoneleaf.us> <4DF68B03.2010603@stoneleaf.us> Message-ID: <20110613234035.8D9A33A40D9@sparrow.telecommunity.com> At 03:11 PM 6/13/2011 -0700, Ethan Furman wrote: >Nick Coghlan wrote: > > Agreed, but: > > > > EOH, CHAR, DATE, FLOAT, INT, LOGICAL, MEMO, NUMBER = b'\rCDFILMN' > > > > is a shorter way to write the same thing. > > > > Going two per line makes it easier to mentally map the characters: > > > > EOH, CHAR = b'\rC' > > DATE, FLOAT = b'DF' > > INT, LOGICAL = b'IL' > > MEMO, NUMBER = b'MN' > >Wow. I didn't realize that could be done. That very nearly makes >up for not being able to do it one char at a time. You can still do it one at a time: CHAR, = b'C' INT, = b'I' ... etc. I just tried it with Python 3.1 and it works there. From carl at oddbird.net Tue Jun 14 01:50:44 2011 From: carl at oddbird.net (Carl Meyer) Date: Mon, 13 Jun 2011 18:50:44 -0500 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: <4DF6A159.1010105@oddbird.net> References: <4DF5FAC8.4090606@voidspace.org.uk> <4DF6A159.1010105@oddbird.net> Message-ID: <4DF6A254.8030405@oddbird.net> On 06/13/2011 06:46 PM, Carl Meyer wrote: > FWIW, historically pretty much every new Python version has broken > virtualenv I should clarify that this is because of the delicate stdlib bootstrapping virtualenv currently has to do; the built-in virtualenv eliminates this entirely and will require much less maintenance for new Python versions. Carl From fuzzyman at voidspace.org.uk Tue Jun 14 02:00:45 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 14 Jun 2011 01:00:45 +0100 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: <4DF6A159.1010105@oddbird.net> References: <4DF5FAC8.4090606@voidspace.org.uk> <4DF6A159.1010105@oddbird.net> Message-ID: <4DF6A4AD.1060706@voidspace.org.uk> On 14/06/2011 00:46, Carl Meyer wrote: > [snip...] > So I don't think a virtualenv stdlib module would be at all likely to > break on a new OS release, if Python itself is not broken by that OS > release. (It certainly wouldn't be the stdlib module most likely to be > broken by OS changes, in comparison to e.g. shutil, threading...) > And if we gain Carl as a Python committer to help maintain it, then I'd say it is worth doing for that reason alone... Michael > Carl > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From ncoghlan at gmail.com Tue Jun 14 02:35:41 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Jun 2011 10:35:41 +1000 Subject: [Python-Dev] Python 3.x and bytes In-Reply-To: <20110613234035.8D9A33A40D9@sparrow.telecommunity.com> References: <4DD2C2A5.3080403@stoneleaf.us> <4DD2D89D.4000303@stoneleaf.us> <4DD2F661.2050005@stoneleaf.us> <4DF4F4E0.5060003@stoneleaf.us> <4DF68B03.2010603@stoneleaf.us> <20110613234035.8D9A33A40D9@sparrow.telecommunity.com> Message-ID: On Tue, Jun 14, 2011 at 9:40 AM, P.J. Eby wrote: > You can still do it one at a time: > > CHAR, = b'C' > INT, ?= b'I' > ... > > etc. ?I just tried it with Python 3.1 and it works there. I almost mentioned that, although it does violate one of the "unwritten rules of the Zen" (in this case, "syntax shall not look like grit on Tim's monitor") Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From greg.ewing at canterbury.ac.nz Tue Jun 14 02:36:16 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 14 Jun 2011 12:36:16 +1200 Subject: [Python-Dev] PEP 3101 implementation vs. documentation In-Reply-To: References: <4DF3FA53.7000906@canterbury.ac.nz> Message-ID: <4DF6AD00.3000207@canterbury.ac.nz> Ben Wolfson wrote: > If by "item selector" you mean (using the names from the grammar in > the docs) the element_index, I don't see why this should be the case; > dictionaries can contain non-identified keys, after all. Of course they can, but that's not the point. The point is that putting arbitrary strings between [...] in a format spec without any form of quoting or requirement for bracket matching leads to something that's too confusing for humans to read. IMO the spec should be designed so that the format string can be parsed using the same lexical analysis rules as Python code. That means anything that is meant to "hang together" as a single unit, such as an item selector, needs to look like a single Python token, e.g. an integer or identifier. I realise this is probably more restrictive than the PEP suggests, but I think it would be better that way all round. -- Greg From cs at zip.com.au Tue Jun 14 03:35:03 2011 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 14 Jun 2011 11:35:03 +1000 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: References: Message-ID: <20110614013503.GA29499@cskk.homeip.net> On 13Jun2011 13:47, Vinay Sajip wrote: | Nick Coghlan gmail.com> writes: | | > On Mon, Jun 13, 2011 at 10:57 PM, Antoine Pitrou pitrou.net> | > wrote: | > > As a side-note, I think calling it "virtualization" is a recipe for | > > confusion. | > | > Indeed, OS level virtualisation pretty much has a lock on that term. | > "virtual environments" skates close to it but manages to avoid it well | > enough to avoid confusion. | | Or as they involving encapsulating paths and libaries, perhaps we could call | them "capsules" ;-) though I think the term virtualenv is pretty entrenched now | in the Python community. "virtualenv" by all means - we all know what is meant. But "virtualisation" - I also am -1 on that. Indeed, when I started reading this thread my expectation was wrong for that very reason. Same issue with "capsules" (yes I know you weren't serious) - too generic a term, too vague. Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ It looked good-natured, she thought; Still it had very long claws and a great many teeth, so she felt it ought to be treated with respect. From cs at zip.com.au Tue Jun 14 03:43:58 2011 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 14 Jun 2011 11:43:58 +1000 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: References: Message-ID: <20110614014358.GA30397@cskk.homeip.net> On 13Jun2011 17:31, Vinay Sajip wrote: | Barry Warsaw python.org> writes: | > Ah, no I don't think that'll be helpful. I can probably reuse the python3.3 | > packaging stuff to do a PPA. | | Okay, go for it! Is there a specific tutorial somewhere about making a PPA for | Python? (As opposed to more generalised tutorials - or would they be sufficient?) | | > (It takes that long because it basically does a | > `make install`.) | | I realise that, as well as recording what it's doing, but that part seems to | happen fairly quickly. | | Then it says "Copying files to the temporary directory..." and that part seems | to take forever. The whole deb is under 25MB, what could be taking many minutes? [ wild speculation ... ] How does it decide what to copy? If it is a "blind" make-me-a-package tool it may be scanning the whole OS install or something (expensive but linear) and maybe then doing some ghastly O(n^2) changed file comparison. Inefficient comparison stuff leaks into the real world all the time; the Linux kernel installs have a "hardlinks" program which is one of my pet hates for this very reason - it runs over the modules trees looking for identical module files to hard link and if you've got several kernels lying around it is unforgivably slow. Or maybe it loads the package install db into memory and does something expensive to see what's not accounted for. [ end speculation, but nothing useful now follows ... ] Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ "He deserves death!" "Deserves it! I daresay he does. And many die that deserve life. Is it in your power to give it to them? Then do not be so quick to deal out death in judgement, for even the very wise may not see all ends." - Gandalf, _The Lord of the Rings_ From wolfson at gmail.com Tue Jun 14 03:46:51 2011 From: wolfson at gmail.com (Ben Wolfson) Date: Mon, 13 Jun 2011 18:46:51 -0700 Subject: [Python-Dev] PEP 3101 implementation vs. documentation In-Reply-To: <4DF6AD00.3000207@canterbury.ac.nz> References: <4DF3FA53.7000906@canterbury.ac.nz> <4DF6AD00.3000207@canterbury.ac.nz> Message-ID: On Mon, Jun 13, 2011 at 5:36 PM, Greg Ewing wrote: > Ben Wolfson wrote: > >> If by "item selector" you mean (using the names from the grammar in >> the docs) the element_index, I don't see why this should be the case; >> dictionaries can contain non-identified keys, after all. > > Of course they can, but that's not the point. The point is > that putting arbitrary strings between [...] in a format > spec without any form of quoting or requirement for bracket > matching leads to something that's too confusing for humans > to read. But there is a requirement for bracket matching: the "[" that opens the element_index is matched by the next "]". Arguably (as Terry Reedy said) this is also a form of quoting, in which the square brackets are the quotation operators. It seems no more confusing to me than allowing arbitrary strings between in '"..."'; those quotation marks aren't even oriented. (Admittedly, syntax highlighting helps there.) Compared to this: "{0: ^+#10o}", a string like this: "this is normal text, but {e.underline[this text is is udnerlined {sic}!]}---and we're back to normal now" is pretty damn readable to this human, nor do I see what about the rule "when you see a [, keep going until you see a ]" is supposed to be insuperably confusing. (Compare---not that it helps my case in regard to readability---grouping in regular expressions, where you don't usually have the aid of special syntax highlighting inside the string; you see a '(', you know that you've encountered a group which continues until the next (unescaped!) ')'. The stuff that comes inside the parentheses might look like line noise---and the whole thing might look like line noise---but *that* rule about the structure of a regexp is pretty straightforward.) > IMO the spec should be designed so that the format string > can be parsed using the same lexical analysis rules as > Python code. That means anything that is meant to "hang > together" as a single unit, such as an item selector, > needs to look like a single Python token, e.g. an integer > or identifier. If that's the rationale, why not change the spec so that instead of this: "{0[spam]}" You do this: "{0['spam']}" ? Hangs together; single Python token. Bonus: it would make it possible for this to work: (a) "{0['12']}".format({'12': 4}) whereas currently this: "{0[12]}".format(...) passes the integer 12 to __getitem__, and (a) passes the string "'12'". (Discovery: the "abuse" of the format mechanism I want to perpetrate via element_index can also be perpetrated with a custom __format__ method: >>> class foo: ... def __format__(self, a): return a * 2 ... >>> "hello {0::![} world".format(foo()) 'hello :![:![ world' . So any reform to make it impossible to use str.format creatively will have to be fairly radical. I actually think that my intended abuse is actually a perfectly reasonable use, but it would be disallowed if only integers and identifiers can be in the element_index field.) -- Ben Wolfson "Human kind has used its intelligence to vary the flavour of drinks, which may be sweet, aromatic, fermented or spirit-based. ... Family and social life also offer numerous other occasions to consume drinks for pleasure." [Larousse, "Drink" entry] From barry at python.org Tue Jun 14 04:01:26 2011 From: barry at python.org (Barry Warsaw) Date: Mon, 13 Jun 2011 22:01:26 -0400 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: <4DF6A4AD.1060706@voidspace.org.uk> References: <4DF5FAC8.4090606@voidspace.org.uk> <4DF6A159.1010105@oddbird.net> <4DF6A4AD.1060706@voidspace.org.uk> Message-ID: <20110613220126.73a36b39@neurotica.wooz.org> On Jun 14, 2011, at 01:00 AM, Michael Foord wrote: >On 14/06/2011 00:46, Carl Meyer wrote: >> [snip...] >> So I don't think a virtualenv stdlib module would be at all likely to >> break on a new OS release, if Python itself is not broken by that OS >> release. (It certainly wouldn't be the stdlib module most likely to be >> broken by OS changes, in comparison to e.g. shutil, threading...) >> > >And if we gain Carl as a Python committer to help maintain it, then I'd say >it is worth doing for that reason alone... +1 -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From exarkun at twistedmatrix.com Tue Jun 14 03:56:06 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Tue, 14 Jun 2011 01:56:06 -0000 Subject: [Python-Dev] Python 3.x and bytes In-Reply-To: References: <4DD2C2A5.3080403@stoneleaf.us> <4DD2D89D.4000303@stoneleaf.us> <4DD2F661.2050005@stoneleaf.us> <4DF4F4E0.5060003@stoneleaf.us> <4DF68B03.2010603@stoneleaf.us> <20110613234035.8D9A33A40D9@sparrow.telecommunity.com> Message-ID: <20110614015606.1894.1336593159.divmod.xquotient.147@localhost.localdomain> On 12:35 am, ncoghlan at gmail.com wrote: >On Tue, Jun 14, 2011 at 9:40 AM, P.J. Eby >wrote: >>You can still do it one at a time: >> >>CHAR, = b'C' >>INT, ?= b'I' >>... >> >>etc. ?I just tried it with Python 3.1 and it works there. > >I almost mentioned that, although it does violate one of the >"unwritten rules of the Zen" (in this case, "syntax shall not look >like grit on Tim's monitor") [CHAR] = b'C' [INT] = b'I' ... Jean-Paul From larry at hastings.org Tue Jun 14 04:36:05 2011 From: larry at hastings.org (Larry Hastings) Date: Mon, 13 Jun 2011 19:36:05 -0700 Subject: [Python-Dev] Call For Topics for Python Language Summit at EuroPython 2011 Message-ID: <4DF6C915.50803@hastings.org> Howdy howdy. Michael Foord can't make it to EuroPython this year--and congratulations to him on his new baby!--so he's asked me to chair the Python Language Summit in his stead. Although I have some suggested topics from Michael, I don't have any burning issues of my own to add to the docket. That's where *you* come in. If you have a topic you'd like to suggest, please email it privately to me (larry at hastings dot org) along with an estimate of how long you think it'll take to discuss. Please have all suggestions to me no later than Friday June 17th. Also, please include the phrase "Python Language Summit" in the subject line, in case it gets caught in my spam filters. You don't have to be attending the summit to suggest a topic--I'll consider suggestions from anybody. Hope to see you at EuroPython, /larry/ From jannis at leidel.info Tue Jun 14 11:15:20 2011 From: jannis at leidel.info (Jannis Leidel) Date: Tue, 14 Jun 2011 11:15:20 +0200 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: <4DF6A159.1010105@oddbird.net> References: <4DF5FAC8.4090606@voidspace.org.uk> <4DF6A159.1010105@oddbird.net> Message-ID: On 14.06.2011, at 01:46, Carl Meyer wrote: > On 06/13/2011 08:07 AM, Nick Coghlan wrote: >> On Mon, Jun 13, 2011 at 10:50 PM, Vinay Sajip wrote: >>> You're right in terms of the current Python ecosystem and 3.x adoption, because >>> of course this approach requires support from Python itself in terms of its >>> site.py code. However, virtual environments have a utility beyond supporting >>> older Pythons on newer OSes, since another common use case is having different >>> library environments sandboxed from each other on different projects, even if >>> all those projects are using Python 3.3+. >> >> Yeah, even if the innate one struggles on later OS releases that >> changed things in a backwards incompatible way, it will still be >> valuable on the OS versions that are around at the time that version >> of Python gets released. > > FWIW, historically pretty much every new Python version has broken > virtualenv, and new OS versions almost never have. Virtualenv isn't > especially OS-dependent (not nearly as much as some other stdlib > modules): the most OS-dependent piece is "shell activation", and that's > a feature I would prefer to entirely leave out of the stdlib virtualenv > (it's a convenience, not a necessity for virtualenv use, and the need to > maintain it for a variety of OS shells is a maintenance burden I don't > think Python should adopt). > > In fact, the only new-OS-version adjustment I can recall virtualenv > needing to make is when Debian introduced dist-packages -- but even that > doesn't really apply, as that was distro-packager change to Python > itself. With a built-in virtualenv it would be the distro packagers > responsibility to make sure their patch to Python doesn't break the > virtualenv module. FTR, there is some special casing for Mac OS framework installs included, too. Not sure if that should be considered a stability threatening issue though since Apple hasn't changed much on that layout, AFAIK. > So I don't think a virtualenv stdlib module would be at all likely to > break on a new OS release, if Python itself is not broken by that OS > release. (It certainly wouldn't be the stdlib module most likely to be > broken by OS changes, in comparison to e.g. shutil, threading...) Jannis From ronaldoussoren at mac.com Tue Jun 14 11:44:45 2011 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 14 Jun 2011 11:44:45 +0200 Subject: [Python-Dev] In-Python virtualisation and packaging In-Reply-To: References: <4DF5FAC8.4090606@voidspace.org.uk> <4DF6A159.1010105@oddbird.net> Message-ID: On 14 Jun, 2011, at 11:15, Jannis Leidel wrote: > > On 14.06.2011, at 01:46, Carl Meyer wrote: > >> >> In fact, the only new-OS-version adjustment I can recall virtualenv >> needing to make is when Debian introduced dist-packages -- but even that >> doesn't really apply, as that was distro-packager change to Python >> itself. With a built-in virtualenv it would be the distro packagers >> responsibility to make sure their patch to Python doesn't break the >> virtualenv module. > > FTR, there is some special casing for Mac OS framework installs included, too. > Not sure if that should be considered a stability threatening issue though > since Apple hasn't changed much on that layout, AFAIK. Apple hasn't changed anything w.r.t. the basic layout of frameworks for a long time, but does mess with the structure of site-packages in their releases of Python. That shouldn't affect this feature though. For the most part a Python.framework is just a unix install stuffed inside framework. The special-case code in virtualenv for frameworks is needed because a framework uses another mechanism to look for sys.prefix than a classical unix install: sys.prefix is the directory that contains the python shared library. There is another feature of a framework install that would be nice to have in a virtualenv: the python and pythonw commands for a framework build are small programs that use execv to start the real interpreter that's stored in a Python.app inside the framework. This is needed to be able to access GUI functionality from the command-line as Apple's GUI frameworks assume they are used by code in an application bundle. Ronald From skip at montanaro.dyndns.org Tue Jun 14 15:10:15 2011 From: skip at montanaro.dyndns.org (Skip Montanaro) Date: Tue, 14 Jun 2011 08:10:15 -0500 (CDT) Subject: [Python-Dev] Pathscale compilers open source Message-ID: <20110614131015.91A3015F3A6A@montanaro.dyndns.org> One of my colleagues with a background in the high performance computing realm sent me this press release: http://www.pathscale.com/ekopath4-open-source-announcement I'm not personally familiar with the Pathscale compilers, but thought some folks here might be and might want to experiment with them. Skip From lukas.lueg at googlemail.com Tue Jun 14 18:28:30 2011 From: lukas.lueg at googlemail.com (Lukas Lueg) Date: Tue, 14 Jun 2011 18:28:30 +0200 Subject: [Python-Dev] Lazy unpacking for struct module In-Reply-To: <4DF68709.8070206@canterbury.ac.nz> References: <6656372E-7E82-477E-8C42-6CB686A1ADD1@gmail.com> <4DF53B4D.1000607@canterbury.ac.nz> <4DF68709.8070206@canterbury.ac.nz> Message-ID: > So I really can't see what harm it could do, except for > maybe a tiny performance reduction in the case where you > extract all the fields, or refer to extracted fields > repeatedly. Referring to the view-object multiple times should not be a problem since the object can create and hold references to the unpacked values it created; remember that they are all immutable. From pje at telecommunity.com Tue Jun 14 19:23:13 2011 From: pje at telecommunity.com (P.J. Eby) Date: Tue, 14 Jun 2011 13:23:13 -0400 Subject: [Python-Dev] Python 3.x and bytes In-Reply-To: <20110614015606.1894.1336593159.divmod.xquotient.147@localh ost.localdomain> References: <4DD2C2A5.3080403@stoneleaf.us> <4DD2D89D.4000303@stoneleaf.us> <4DD2F661.2050005@stoneleaf.us> <4DF4F4E0.5060003@stoneleaf.us> <4DF68B03.2010603@stoneleaf.us> <20110613234035.8D9A33A40D9@sparrow.telecommunity.com> <20110614015606.1894.1336593159.divmod.xquotient.147@localhost.localdomain> Message-ID: <20110614172318.BE9203A4100@sparrow.telecommunity.com> At 01:56 AM 6/14/2011 +0000, exarkun at twistedmatrix.com wrote: >On 12:35 am, ncoghlan at gmail.com wrote: >>On Tue, Jun 14, 2011 at 9:40 AM, P.J. Eby wrote: >>>You can still do it one at a time: >>> >>>CHAR, = b'C' >>>INT, = b'I' >>>... >>> >>>etc. I just tried it with Python 3.1 and it works there. >> >>I almost mentioned that, although it does violate one of the >>"unwritten rules of the Zen" (in this case, "syntax shall not look >>like grit on Tim's monitor") > > [CHAR] = b'C' > [INT] = b'I' > ... Holy carpal tunnel time machine... That works in 2.3. (Without the 'b' of course.) Didn't know you could just use list syntax like that. It's an extra character to type, and two more shift keyings, but brevity isn't always the soul of clarity. From ethan at stoneleaf.us Tue Jun 14 19:46:42 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 14 Jun 2011 10:46:42 -0700 Subject: [Python-Dev] Python 3.x and bytes In-Reply-To: <20110614172318.BE9203A4100@sparrow.telecommunity.com> References: <4DD2C2A5.3080403@stoneleaf.us> <4DD2D89D.4000303@stoneleaf.us> <4DD2F661.2050005@stoneleaf.us> <4DF4F4E0.5060003@stoneleaf.us> <4DF68B03.2010603@stoneleaf.us> <20110613234035.8D9A33A40D9@sparrow.telecommunity.com> <20110614015606.1894.1336593159.divmod.xquotient.147@localhost.localdomain> <20110614172318.BE9203A4100@sparrow.telecommunity.com> Message-ID: <4DF79E82.7090208@stoneleaf.us> P.J. Eby wrote: > At 01:56 AM 6/14/2011 +0000, exarkun at twistedmatrix.com wrote: >> On 12:35 am, ncoghlan at gmail.com wrote: >>> On Tue, Jun 14, 2011 at 9:40 AM, P.J. Eby wrote: >>>> You can still do it one at a time: >>>> >>>> CHAR, = b'C' >>>> INT, = b'I' >>>> ... >>>> >>>> etc. I just tried it with Python 3.1 and it works there. >>> >>> I almost mentioned that, although it does violate one of the >>> "unwritten rules of the Zen" (in this case, "syntax shall not look >>> like grit on Tim's monitor") >> >> [CHAR] = b'C' >> [INT] = b'I' >> ... > > Holy carpal tunnel time machine... That works in 2.3. (Without the 'b' > of course.) Didn't know you could just use list syntax like that. It's > an extra character to type, and two more shift keyings, but brevity > isn't always the soul of clarity. I'm thinking I like to the 'new' tuple-assignment character... ,= ! CHAR ,= b'C' DATE ,= b'D' LOGICAL ,= b'L' ;) ~Ethan~ From lukasz at langa.pl Tue Jun 14 19:57:49 2011 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Tue, 14 Jun 2011 19:57:49 +0200 Subject: [Python-Dev] Python 3.x and bytes In-Reply-To: <4DF79E82.7090208@stoneleaf.us> References: <4DD2C2A5.3080403@stoneleaf.us> <4DD2D89D.4000303@stoneleaf.us> <4DD2F661.2050005@stoneleaf.us> <4DF4F4E0.5060003@stoneleaf.us> <4DF68B03.2010603@stoneleaf.us> <20110613234035.8D9A33A40D9@sparrow.telecommunity.com> <20110614015606.1894.1336593159.divmod.xquotient.147@localhost.localdomain> <20110614172318.BE9203A4100@sparrow.telecommunity.com> <4DF79E82.7090208@stoneleaf.us> Message-ID: <6E9112BF-2100-46C7-871E-6EE0E4AE2049@langa.pl> Wiadomo?? napisana przez Ethan Furman w dniu 2011-06-14, o godz. 19:46: >>> [CHAR] = b'C' >>> [INT] = b'I' > CHAR ,= b'C' > DATE ,= b'D' > LOGICAL ,= b'L' Perl Jam! -- Best regards, ?ukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Tue Jun 14 20:26:58 2011 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 14 Jun 2011 14:26:58 -0400 Subject: [Python-Dev] Pathscale compilers open source References: <20110614131015.91A3015F3A6A@montanaro.dyndns.org> Message-ID: Skip Montanaro wrote: > One of my colleagues with a background in the high performance computing > realm sent me this press release: > > http://www.pathscale.com/ekopath4-open-source-announcement > > I'm not personally familiar with the Pathscale compilers, but thought some > folks here might be and might want to experiment with them. > > Skip I just rebuilt all my c++ (boost::python) modules using pathscale, and I notice many crash with double-free on exit. According to valgrind, this comes from the pathscale stl: Just a heads-up. ==1927== Invalid free() / delete / delete[] ==1927== at 0x4A0556E: free (vg_replace_malloc.c:366) ==1927== by 0xDA77622: operator delete(void*) (in /home/nbecker/ekopath-4.0.10/lib/4.0.10/x8664/64/libcxxrt.so) ==1927== by 0xD7BB91A: std::allocator::deallocate(char*, unsigned long) (in /home/nbecker/ekopath-4.0.10/lib/4.0.10/x8664/64/libstl.so) ==1927== by 0xD7BB99B: std::string::_C_unlink(char*) (in /home/nbecker/ekopath-4.0.10/lib/4.0.10/x8664/64/libstl.so) ==1927== by 0xD7C4309: std::basic_string, std::allocator >::~basic_string() (in /home/nbecker/ekopath-4.0.10/lib/4.0.10/x8664/64/libstl.so) ==1927== by 0x3D64438940: __run_exit_handlers (in /lib64/libc-2.14.so) From barry at python.org Tue Jun 14 22:54:14 2011 From: barry at python.org (Barry Warsaw) Date: Tue, 14 Jun 2011 16:54:14 -0400 Subject: [Python-Dev] packaging was not getting installed Message-ID: <20110614165414.340f2138@neurotica.wooz.org> I just fixed Makefile.pre.in to install the packaging directory and all its subdirs. Without this `python3.3 -c 'import packaging'` fails from the installation target directory. I'm not sure the Fellowship intends for every subdir to get installed, so please double check. I just added everything that came from `find Lib/packaging -type d`. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From nad at acm.org Tue Jun 14 23:09:16 2011 From: nad at acm.org (Ned Deily) Date: Tue, 14 Jun 2011 14:09:16 -0700 Subject: [Python-Dev] packaging was not getting installed References: <20110614165414.340f2138@neurotica.wooz.org> Message-ID: In article <20110614165414.340f2138 at neurotica.wooz.org>, Barry Warsaw wrote: > I just fixed Makefile.pre.in to install the packaging directory and all its > subdirs. Without this `python3.3 -c 'import packaging'` fails from the > installation target directory. I'm not sure the Fellowship intends for every > subdir to get installed, so please double check. I just added everything that > came from `find Lib/packaging -type d`. http://bugs.python.org/issue12313 -- Ned Deily, nad at acm.org From merwok at netwok.org Wed Jun 15 18:03:42 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Wed, 15 Jun 2011 18:03:42 +0200 Subject: [Python-Dev] Fwd: [Python-committers] Pulling from contributors repositories In-Reply-To: <4DF5A1F0.2030207@m2.ccsnet.ne.jp> References: <4DF5A1F0.2030207@m2.ccsnet.ne.jp> Message-ID: <4DF8D7DE.4070002@netwok.org> Le 13/06/2011 07:36, Hirokazu Yamamoto a ?crit : > I've read the Python-committers thread "Pulling from contributors > repositories", which is about version control system. It seems there are > two main issues, linear (cleaner) history on pushing, and NEWS merging. > I'm newby of bazaar, but it seems to have a solution for first issue. We are using Mercurial. Regards From merwok at netwok.org Wed Jun 15 18:07:58 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Wed, 15 Jun 2011 18:07:58 +0200 Subject: [Python-Dev] pysetup as a top script In-Reply-To: References: Message-ID: <4DF8D8DE.9050204@netwok.org> Le 31/05/2011 08:45, Tarek Ziad? a ?crit : > On Tue, May 31, 2011 at 7:13 AM, Nick Coghlan wrote: >> On Tue, May 31, 2011 at 2:44 AM, Tarek Ziad? wrote: >>> If no one objects, I'll promote Tools/scripts/pysetup3 to a top level >>> script that gets installed in scripts/ like 2to3, pydoc etc.. >>> >>> That way, people will be able to use it directly when installing, >>> removing projects, or studying what's installed >> Cool. >> >> Now I'm trying to remember if it was a list discussion or the language >> summit where you got the initial consensus on that approach... > The thread starts here: > http://mail.python.org/pipermail/python-dev/2010-October/104535.html > > The pysetup top-level script was mentioned here: > http://mail.python.org/pipermail/python-dev/2010-October/104581.html A few other reasons that were not mentioned previously: - In 2.4, we can?t run ?-m distutils2.run?, but a pysetup2.4 script works - It?s nice for users to have something shorter than ?python3.3 -m packaging.run run sdist? (I like to take ?make? as the ideal goal) - It sends a message that we care about packaging (personal opinion) Regards From sandro.tosi at gmail.com Wed Jun 15 18:42:35 2011 From: sandro.tosi at gmail.com (Sandro Tosi) Date: Wed, 15 Jun 2011 18:42:35 +0200 Subject: [Python-Dev] Some additions to .hgignore In-Reply-To: <4DE6193C.3000708@voidspace.org.uk> References: <4DE6193C.3000708@voidspace.org.uk> Message-ID: On 2011-06-01, Michael Foord wrote: > That sounds good to me. An issue certainly wouldn't hurt. So be it: http://bugs.python.org/issue12341 :) Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From solipsis at pitrou.net Thu Jun 16 00:21:40 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 Jun 2011 00:21:40 +0200 Subject: [Python-Dev] Parser/intrcheck.c Message-ID: <20110616002140.77861fdd@pitrou.net> Hello, I may be missing something, but I'm wondering whether Parser/intrcheck.c is still used anywhere. It's only mentioned in some comments: $ grep -r intrcheck.c * Modules/signalmodule.c:1197:/* Replacements for intrcheck.c functionality PC/os2vacpp/makefile.omk:217: # intrcheck.c -- Not Referenced by Anyone (?) Python/sigcheck.c:3: interrupt occurs. It can't be in the intrcheck.c file since that And if I remove it and "make clean", I can still rebuild successfully. Regards Antoine. From mal at egenix.com Thu Jun 16 19:18:45 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 16 Jun 2011 19:18:45 +0200 Subject: [Python-Dev] Python language summit on ustream.tv Message-ID: <4DFA3AF5.3050306@egenix.com> Dear Python Developers, for the upcoming language summit at EuroPython, I'd like to try out whether streaming such meetings would work. I'll setup a webcam and stream the event live to a private channel on ustream.tv. These are the details in case you want to watch: URL: http://www.ustream.tv/channel/python-language-summit PWD: fpmUtuL4 Date: Sunday, 2011-06-19 Time: 10:00 - 16:00 CEST with breaks I'm not sure whether I can stream the whole summit, but at least the morning session should be possible, provided the network works on that day. Interaction will likely be a bit difficult in case we have heated discussions :-), but we'll keep the IRC channel #python-language-summit on freenode open as well. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 16 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ 2011-05-23: Released eGenix mx Base 3.2.0 http://python.egenix.com/ 2011-05-25: Released mxODBC 3.1.1 http://python.egenix.com/ 2011-06-20: EuroPython 2011, Florence, Italy 4 days to go ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From db3l.net at gmail.com Thu Jun 16 21:08:18 2011 From: db3l.net at gmail.com (David Bolen) Date: Thu, 16 Jun 2011 15:08:18 -0400 Subject: [Python-Dev] Buildbot status web pages Message-ID: I've been receiving 503 errors from the buildbot web status pages beneath www.python.org/dev/buildbot for a day or two now - is there perhaps something that needs a bit of a kick-start? Thanks. -- David From ncoghlan at gmail.com Fri Jun 17 07:20:44 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 17 Jun 2011 15:20:44 +1000 Subject: [Python-Dev] [Python-checkins] cpython (3.2): Fix typo. In-Reply-To: References: Message-ID: On Fri, Jun 17, 2011 at 7:33 AM, raymond.hettinger wrote: > http://hg.python.org/cpython/rev/0fe3b81c7c89 > changeset: ? 70822:0fe3b81c7c89 > branch: ? ? ?3.2 > parent: ? ? ?70819:8a9d25804c3d > user: ? ? ? ?Raymond Hettinger > date: ? ? ? ?Thu Jun 16 22:32:10 2011 +0100 > summary: > ?Fix typo. > > files: > ?Doc/whatsnew/3.2.rst | ?2 +- > ?1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Doc/whatsnew/3.2.rst b/Doc/whatsnew/3.2.rst > --- a/Doc/whatsnew/3.2.rst > +++ b/Doc/whatsnew/3.2.rst > @@ -385,7 +385,7 @@ > ?===================================================== > > ?This informational PEP clarifies how bytes/text issues are to be handled by the > -WGSI protocol. ?The challenge is that string handling in Python 3 is most > +WSGI protocol. ?The challenge is that string handling in Python 3 is most > ?conveniently handled with the :class:`str` type even though the HTTP protocol > ?is itself bytes oriented. Ah, the cognitive scientist in me *loves* diffs like this one. Even *knowing* there's a typo on the line, it's still hard to spot :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From lukas.lueg at googlemail.com Fri Jun 17 16:44:12 2011 From: lukas.lueg at googlemail.com (Lukas Lueg) Date: Fri, 17 Jun 2011 16:44:12 +0200 Subject: [Python-Dev] Lazy unpacking for struct module In-Reply-To: References: <6656372E-7E82-477E-8C42-6CB686A1ADD1@gmail.com> <4DF53B4D.1000607@canterbury.ac.nz> <4DF68709.8070206@canterbury.ac.nz> Message-ID: The followup: I've implemented a new StructView-object for 3.3a-trunk. The object takes and existing Struct-object and provides on-access unpacking. The breaking point where this object is faster than calling Struct.unpack seems to be somewhere around 12 fields in the format-string. Format strings with less fields expose too much overhead of entering the C-code and staying there a little longer to unpack all fields is actually faster. Having fifteen or more fields in a format-string seems unlikely and I'll therefor abandon the idea of providing this mechanism. 2011/6/14 Lukas Lueg : >> So I really can't see what harm it could do, except for >> maybe a tiny performance reduction in the case where you >> extract all the fields, or refer to extracted fields >> repeatedly. > > Referring to the view-object multiple times should not be a problem > since the object can create and hold references to the unpacked values > it created; remember that they are all immutable. > From status at bugs.python.org Fri Jun 17 18:07:24 2011 From: status at bugs.python.org (Python tracker) Date: Fri, 17 Jun 2011 18:07:24 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20110617160724.855FE1CCB6@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2011-06-10 - 2011-06-17) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 2843 (+17) closed 21292 (+24) total 24135 (+41) Open issues with patches: 1244 Issues opened (29) ================== #10884: pkgutil EggInfoDistribution requirements for .egg-info metadat http://bugs.python.org/issue10884 reopened by eric.araujo #12298: Sphinx glitch in library/functions http://bugs.python.org/issue12298 reopened by eric.araujo #12313: make install misses test dirs for packaging and email modules http://bugs.python.org/issue12313 opened by eric.araujo #12314: regrtest checks (os.environ, sys.path, etc.) are hard to use http://bugs.python.org/issue12314 opened by eric.araujo #12315: Improve http.client.HTTPResponse.read documentation http://bugs.python.org/issue12315 opened by harobed #12317: inspect.getabsfile() is not documented http://bugs.python.org/issue12317 opened by giampaolo.rodola #12319: [http.client] HTTPConnection.putrequest not support "chunked" http://bugs.python.org/issue12319 opened by harobed #12321: documentation of ElementTree.find http://bugs.python.org/issue12321 opened by patrick.vrijlandt #12322: ElementPath 1.3 expressions documentation http://bugs.python.org/issue12322 opened by patrick.vrijlandt #12323: ElementPath 1.3 expressions http://bugs.python.org/issue12323 opened by patrick.vrijlandt #12326: Linux 3: tests should avoid using sys.platform == 'linux2' http://bugs.python.org/issue12326 opened by neologix #12327: in HTTPConnection the are len(body) and TypeError catch except http://bugs.python.org/issue12327 opened by harobed #12328: multiprocessing's overlapped PipeConnection on Windows http://bugs.python.org/issue12328 opened by sbt #12331: lib2to3 and packaging tests fail because they write into prote http://bugs.python.org/issue12331 opened by vinay.sajip #12335: pysetup create: ask before moving an existing setup.cfg http://bugs.python.org/issue12335 opened by barry #12336: tkinter.test.test_ttk.test_widgets.test_select() failure on Fr http://bugs.python.org/issue12336 opened by haypo #12337: Need real TextIOWrapper for stdin/stdout http://bugs.python.org/issue12337 opened by r_mosaic #12338: multiprocessing.util._eintr_retry doen't recalculate timeouts http://bugs.python.org/issue12338 opened by sbt #12340: Access violation when using the C version of the io module http://bugs.python.org/issue12340 opened by OscarL #12341: Some additions to .hgignore http://bugs.python.org/issue12341 opened by sandro.tosi #12342: characters with ord above 65535 fail to display in IDLE http://bugs.python.org/issue12342 opened by wujek.srujek #12343: ssl documentation needs comments about non-blocking behaviour http://bugs.python.org/issue12343 opened by jcea #12344: Add **kwargs to get_reinitialized_command http://bugs.python.org/issue12344 opened by higery #12346: Python 2.7.2 source code build (release) depends on mercurial http://bugs.python.org/issue12346 opened by Graeme.Winter #12347: add an "extras" in packaging.pypi.simple.Crawler http://bugs.python.org/issue12347 opened by tarek #12348: case sensitivness in packaging.pypi.simple.Crawler http://bugs.python.org/issue12348 opened by tarek #12350: Improve stat_result.st_blocks and st_blksize documentation http://bugs.python.org/issue12350 opened by liw #12352: multiprocessing.Value() hangs http://bugs.python.org/issue12352 opened by greg.ath #12353: argparse cannot handle empty arguments http://bugs.python.org/issue12353 opened by bjacobs Most recent 15 issues with no replies (15) ========================================== #12353: argparse cannot handle empty arguments http://bugs.python.org/issue12353 #12352: multiprocessing.Value() hangs http://bugs.python.org/issue12352 #12347: add an "extras" in packaging.pypi.simple.Crawler http://bugs.python.org/issue12347 #12338: multiprocessing.util._eintr_retry doen't recalculate timeouts http://bugs.python.org/issue12338 #12336: tkinter.test.test_ttk.test_widgets.test_select() failure on Fr http://bugs.python.org/issue12336 #12323: ElementPath 1.3 expressions http://bugs.python.org/issue12323 #12322: ElementPath 1.3 expressions documentation http://bugs.python.org/issue12322 #12321: documentation of ElementTree.find http://bugs.python.org/issue12321 #12317: inspect.getabsfile() is not documented http://bugs.python.org/issue12317 #12315: Improve http.client.HTTPResponse.read documentation http://bugs.python.org/issue12315 #12303: expose sigwaitinfo() and sigtimedwait() in the signal module http://bugs.python.org/issue12303 #12297: Clarifications to atexit.register and unregister doc http://bugs.python.org/issue12297 #12296: Minor clarification in devguide http://bugs.python.org/issue12296 #12295: Fix ResourceWarning in turtledemo help window http://bugs.python.org/issue12295 #12294: multiprocessing.Pool: Need a way to find out if work are finis http://bugs.python.org/issue12294 Most recent 15 issues waiting for review (15) ============================================= #12350: Improve stat_result.st_blocks and st_blksize documentation http://bugs.python.org/issue12350 #12344: Add **kwargs to get_reinitialized_command http://bugs.python.org/issue12344 #12341: Some additions to .hgignore http://bugs.python.org/issue12341 #12328: multiprocessing's overlapped PipeConnection on Windows http://bugs.python.org/issue12328 #12315: Improve http.client.HTTPResponse.read documentation http://bugs.python.org/issue12315 #12313: make install misses test dirs for packaging and email modules http://bugs.python.org/issue12313 #12308: Add link to PEP 0 for topical index in wiki http://bugs.python.org/issue12308 #12306: zlib: Expose zlibVersion to query runtime version of zlib http://bugs.python.org/issue12306 #12304: expose signalfd(2) in the signal module http://bugs.python.org/issue12304 #12297: Clarifications to atexit.register and unregister doc http://bugs.python.org/issue12297 #12296: Minor clarification in devguide http://bugs.python.org/issue12296 #12295: Fix ResourceWarning in turtledemo help window http://bugs.python.org/issue12295 #12291: file written using marshal in 3.2 can be read by 2.7, but not http://bugs.python.org/issue12291 #12290: __setstate__ is called for false values http://bugs.python.org/issue12290 #12289: http.server.CGIHTTPRequestHandler doesn't check if a Python sc http://bugs.python.org/issue12289 Top 10 most discussed issues (10) ================================= #11610: Improved support for abstract base classes with descriptors http://bugs.python.org/issue11610 15 msgs #12313: make install misses test dirs for packaging and email modules http://bugs.python.org/issue12313 13 msgs #12326: Linux 3: tests should avoid using sys.platform == 'linux2' http://bugs.python.org/issue12326 12 msgs #12343: ssl documentation needs comments about non-blocking behaviour http://bugs.python.org/issue12343 12 msgs #12281: bytes.decode('mbcs', 'ignore') does replace undecodable bytes http://bugs.python.org/issue12281 9 msgs #11553: Docs for: import, packages, site.py, .pth files http://bugs.python.org/issue11553 7 msgs #12167: test_packaging reference leak http://bugs.python.org/issue12167 7 msgs #12287: ossaudiodev: stack corruption with FD >= FD_SETSIZE http://bugs.python.org/issue12287 7 msgs #12328: multiprocessing's overlapped PipeConnection on Windows http://bugs.python.org/issue12328 7 msgs #9302: distutils API Reference: setup() and Extension parameters' des http://bugs.python.org/issue9302 6 msgs Issues closed (26) ================== #9284: inspect.findsource() cannot find source for doctest code http://bugs.python.org/issue9284 closed by python-dev #10457: "Related help topics" shown outside pager http://bugs.python.org/issue10457 closed by eric.araujo #11595: Miscellaneous bugs in cfg_to_args() utility function http://bugs.python.org/issue11595 closed by eric.araujo #12009: netrc module crashes if netrc file has comment lines http://bugs.python.org/issue12009 closed by python-dev #12084: os.stat() on windows doesn't consider relative symlink http://bugs.python.org/issue12084 closed by brian.curtin #12133: ResourceWarning in urllib.request http://bugs.python.org/issue12133 closed by haypo #12206: Documentation Std. Library 15.7.5 "LogRecord objects": Paramet http://bugs.python.org/issue12206 closed by python-dev #12240: Allow multiple setup_hooks http://bugs.python.org/issue12240 closed by eric.araujo #12246: Warn when trying to install third-party module from an uninsta http://bugs.python.org/issue12246 closed by eric.araujo #12248: __dir__ semantics changed in Python 2.7.2 http://bugs.python.org/issue12248 closed by benjamin.peterson #12310: Segfault in test_multiprocessing http://bugs.python.org/issue12310 closed by haypo #12316: test_signal: test_sigwait_thread failure on FreeBSD 6.4 buildb http://bugs.python.org/issue12316 closed by haypo #12318: list + tuple inconsistency http://bugs.python.org/issue12318 closed by rhettinger #12320: test_packaging failures http://bugs.python.org/issue12320 closed by haypo #12324: [3.2] sorted(big dict) http://bugs.python.org/issue12324 closed by mark.dickinson #12325: regex matches incorrectly on literal dot (99.9% confirmed) http://bugs.python.org/issue12325 closed by r.david.murray #12329: XHTML entity apos missing in htmlentitydefs http://bugs.python.org/issue12329 closed by eric.araujo #12330: Named set methods on collections.Set and dict view objects http://bugs.python.org/issue12330 closed by Julian #12332: Float division http://bugs.python.org/issue12332 closed by benjamin.peterson #12333: test_packaging failures under Solaris http://bugs.python.org/issue12333 closed by haypo #12334: Strange sort error http://bugs.python.org/issue12334 closed by rengel #12339: logging.Formatter.format() assumes exception to support str() http://bugs.python.org/issue12339 closed by vinay.sajip #12345: Add math.tau http://bugs.python.org/issue12345 closed by pitrou #12349: Typo in 3.2 "What's New": WGSI / WSGI http://bugs.python.org/issue12349 closed by sandro.tosi #12351: Update URL for pycrypto project http://bugs.python.org/issue12351 closed by python-dev #1711800: SequenceMatcher bug with insert/delete block after "replace" http://bugs.python.org/issue1711800 closed by terry.reedy From martin at v.loewis.de Sun Jun 19 13:07:14 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 19 Jun 2011 13:07:14 +0200 Subject: [Python-Dev] [Python-checkins] cpython: #11781: update windows build script to account for move of email tests In-Reply-To: <4DFD07EF.6060405@udel.edu> References: <4DFD07EF.6060405@udel.edu> Message-ID: <4DFDD862.9030709@v.loewis.de> >> # This should contain all non-.svn files listed in >> subversion > > Should this be mercurial Perhaps. > >> for f in os.listdir(lib.absolute): >> if f.endswith(".txt") or f==".svn":continue > > and this .hg? No. Martin From victor.stinner at haypocalc.com Sun Jun 19 16:17:04 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 19 Jun 2011 16:17:04 +0200 Subject: [Python-Dev] [Python-checkins] cpython: edit and rewrite In-Reply-To: References: Message-ID: <1308493024.30115.6.camel@marge> Le samedi 18 juin 2011 ? 02:51 +0200, benjamin.peterson a ?crit : > .. function:: dump_traceback(file=sys.stderr, all_threads=True) > > - Dump the traceback of all threads, or of the current thread if *all_threads* > - is ``False``, into *file*. > + Dump the traceback of all threads into *file*. If *all_threads* is ``True``, > + produce tracebacks for every running thread. Otherwise, dump only the current > + thread. The first sentence is confusing. I suggest: Dump the traceback of all threads into *file*. If *all_threads* is ``False``, dump the traceback of only the current thread. or Dump the traceback into *file*. If *all_threads* is ``True``, produce tracebacks for every running thread. Otherwise, dump only the current thread. > @@ -69,15 +75,14 @@ > .. function:: dump_tracebacks_later(timeout, repeat=False, file=sys.stderr, exit=False) > > Dump the tracebacks of all threads, after a timeout of *timeout* seconds, or > - each *timeout* seconds if *repeat* is ``True``. If *exit* is True, call > - :c:func:`_exit` with status=1 after dumping the tracebacks to terminate > - immediatly the process, which is not safe. For example, :c:func:`_exit` > - doesn't flush file buffers. If the function is called twice, the new call > - replaces previous parameters (resets the timeout). The timer has a > - sub-second resolution. > + every *timeout* seconds if *repeat* is ``True``. If *exit* is ``True``, call > + :c:func:`_exit` with status=1 after dumping the tracebacks. (Note > + :c:func:`_exit` doesn't flush file buffers.) If the function is called twice, > + the new call replaces previous parameters and resets the timeout. The timer > + has a sub-second resolution. You removed "to terminate immediatly the process, which is not safe" sentence which is very important. It doesn't exit like sys.exit(): it exits immediatly. Not flushing file buffers is just an example. Anyway, thank you for rephrasing the doc. Victor From ncoghlan at gmail.com Sun Jun 19 17:18:03 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 20 Jun 2011 01:18:03 +1000 Subject: [Python-Dev] [Python-checkins] devguide: Fix copy-paste glitch :) In-Reply-To: References: Message-ID: On Mon, Jun 20, 2011 at 12:57 AM, eric.araujo wrote: > http://hg.python.org/devguide/rev/f55ad3dc4526 > changeset: ? 434:f55ad3dc4526 > user: ? ? ? ??ric Araujo > date: ? ? ? ?Sun Jun 19 16:56:58 2011 +0200 > summary: > ?Fix copy-paste glitch :) > > files: > ?committing.rst | ?2 +- > ?1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/committing.rst b/committing.rst > --- a/committing.rst > +++ b/committing.rst > @@ -16,7 +16,7 @@ > ? (using Tools/scripts/reindent-rst.py) > ?* Has the documentation been updated? > ?* Has the test suite been updated? > -* Has ``Misc/ACKS`` been updated? > +* Has ``Misc/NEWS`` been updated? > ?* Has ``Misc/ACKS`` been updated? > ?* Has the test suite been run? D'oh! I had it right at one point, too, but then I rearranged it and must have clobbered the wrong line in the process. Thanks for fixing it :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From victor.stinner at haypocalc.com Sun Jun 19 19:18:15 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 19 Jun 2011 19:18:15 +0200 Subject: [Python-Dev] [Python-checkins] devguide: Add a communications section to the devguide FAQ (closes #11690) In-Reply-To: References: Message-ID: <1308503895.30320.2.camel@marge> Le dimanche 19 juin 2011 ? 16:51 +0200, nick.coghlan a ?crit : > +Where should I ask general Python questions? > +-------------------------------------------- > + > +General Python questions should still go to `python-list`_ or `python-tutor`_ > +or similar resources, such as StackOverflow_ or ``#python`` on IRC. You should mention the IRC server, I suppose that you are talking about the Freenode server. There are other channels in other languages, like #python-fr (on Freenode). You may also add a reference to the #python-dev chanel. I am connected most of the time, but not always available. Victor From jimjjewett at gmail.com Sun Jun 19 21:40:01 2011 From: jimjjewett at gmail.com (Jim Jewett) Date: Sun, 19 Jun 2011 15:40:01 -0400 Subject: [Python-Dev] [Python-checkins] cpython: #6771: Move wrapper function into __init__ and eliminate wrapper module In-Reply-To: References: Message-ID: Does this really need to be a bare except? On Sat, Jun 18, 2011 at 8:21 PM, r.david.murray wrote: > http://hg.python.org/cpython/rev/9c96c3adbcd1 > changeset: ? 70867:9c96c3adbcd1 > user: ? ? ? ?R David Murray > date: ? ? ? ?Sat Jun 18 20:21:09 2011 -0400 > summary: > ?#6771: Move wrapper function into __init__ and eliminate wrapper module > > Andrew agreed in the issue that eliminating the module file made sense. > Wrapper has only been exposed as a function, and so there is no (easy) > way to access the wrapper module, which in any case only had the one > function in it. ?Since __init__ already contains a couple wrapper > functions, it seems to make sense to just move wrapper there instead of > importing it from a single function module. > > files: > ?Lib/curses/__init__.py | ?46 +++++++++++++++++++++++++++- > ?Lib/curses/wrapper.py ?| ?50 ------------------------------ > ?Misc/NEWS ? ? ? ? ? ? ?| ? 4 ++ > ?3 files changed, 49 insertions(+), 51 deletions(-) > > > diff --git a/Lib/curses/__init__.py b/Lib/curses/__init__.py > --- a/Lib/curses/__init__.py > +++ b/Lib/curses/__init__.py > @@ -13,7 +13,6 @@ > ?__revision__ = "$Id$" > > ?from _curses import * > -from curses.wrapper import wrapper > ?import os as _os > ?import sys as _sys > > @@ -57,3 +56,48 @@ > ? ? has_key > ?except NameError: > ? ? from has_key import has_key > + > +# Wrapper for the entire curses-based application. ?Runs a function which > +# should be the rest of your curses-based application. ?If the application > +# raises an exception, wrapper() will restore the terminal to a sane state so > +# you can read the resulting traceback. > + > +def wrapper(func, *args, **kwds): > + ? ?"""Wrapper function that initializes curses and calls another function, > + ? ?restoring normal keyboard/screen behavior on error. > + ? ?The callable object 'func' is then passed the main window 'stdscr' > + ? ?as its first argument, followed by any other arguments passed to > + ? ?wrapper(). > + ? ?""" > + > + ? ?try: > + ? ? ? ?# Initialize curses > + ? ? ? ?stdscr = initscr() > + > + ? ? ? ?# Turn off echoing of keys, and enter cbreak mode, > + ? ? ? ?# where no buffering is performed on keyboard input > + ? ? ? ?noecho() > + ? ? ? ?cbreak() > + > + ? ? ? ?# In keypad mode, escape sequences for special keys > + ? ? ? ?# (like the cursor keys) will be interpreted and > + ? ? ? ?# a special value like curses.KEY_LEFT will be returned > + ? ? ? ?stdscr.keypad(1) > + > + ? ? ? ?# Start color, too. ?Harmless if the terminal doesn't have > + ? ? ? ?# color; user can test with has_color() later on. ?The try/catch > + ? ? ? ?# works around a minor bit of over-conscientiousness in the curses > + ? ? ? ?# module -- the error return from C start_color() is ignorable. > + ? ? ? ?try: > + ? ? ? ? ? ?start_color() > + ? ? ? ?except: > + ? ? ? ? ? ?pass > + > + ? ? ? ?return func(stdscr, *args, **kwds) > + ? ?finally: > + ? ? ? ?# Set everything back to normal > + ? ? ? ?if 'stdscr' in locals(): > + ? ? ? ? ? ?stdscr.keypad(0) > + ? ? ? ? ? ?echo() > + ? ? ? ? ? ?nocbreak() > + ? ? ? ? ? ?endwin() > diff --git a/Lib/curses/wrapper.py b/Lib/curses/wrapper.py > deleted file mode 100644 > --- a/Lib/curses/wrapper.py > +++ /dev/null > @@ -1,50 +0,0 @@ > -"""curses.wrapper > - > -Contains one function, wrapper(), which runs another function which > -should be the rest of your curses-based application. ?If the > -application raises an exception, wrapper() will restore the terminal > -to a sane state so you can read the resulting traceback. > - > -""" > - > -import curses > - > -def wrapper(func, *args, **kwds): > - ? ?"""Wrapper function that initializes curses and calls another function, > - ? ?restoring normal keyboard/screen behavior on error. > - ? ?The callable object 'func' is then passed the main window 'stdscr' > - ? ?as its first argument, followed by any other arguments passed to > - ? ?wrapper(). > - ? ?""" > - > - ? ?try: > - ? ? ? ?# Initialize curses > - ? ? ? ?stdscr = curses.initscr() > - > - ? ? ? ?# Turn off echoing of keys, and enter cbreak mode, > - ? ? ? ?# where no buffering is performed on keyboard input > - ? ? ? ?curses.noecho() > - ? ? ? ?curses.cbreak() > - > - ? ? ? ?# In keypad mode, escape sequences for special keys > - ? ? ? ?# (like the cursor keys) will be interpreted and > - ? ? ? ?# a special value like curses.KEY_LEFT will be returned > - ? ? ? ?stdscr.keypad(1) > - > - ? ? ? ?# Start color, too. ?Harmless if the terminal doesn't have > - ? ? ? ?# color; user can test with has_color() later on. ?The try/catch > - ? ? ? ?# works around a minor bit of over-conscientiousness in the curses > - ? ? ? ?# module -- the error return from C start_color() is ignorable. > - ? ? ? ?try: > - ? ? ? ? ? ?curses.start_color() > - ? ? ? ?except: > - ? ? ? ? ? ?pass > - > - ? ? ? ?return func(stdscr, *args, **kwds) > - ? ?finally: > - ? ? ? ?# Set everything back to normal > - ? ? ? ?if 'stdscr' in locals(): > - ? ? ? ? ? ?stdscr.keypad(0) > - ? ? ? ? ? ?curses.echo() > - ? ? ? ? ? ?curses.nocbreak() > - ? ? ? ? ? ?curses.endwin() > diff --git a/Misc/NEWS b/Misc/NEWS > --- a/Misc/NEWS > +++ b/Misc/NEWS > @@ -193,6 +193,10 @@ > ?Library > ?------- > > +- Issue #6771: moved the curses.wrapper function from the single-function > + ?wrapper module into __init__, eliminating the module. ?Since __init__ was > + ?already importing the function to curses.wrapper, there is no API change. > + > ?- Issue #11584: email.header.decode_header no longer fails if the header > ? passed to it is a Header object, and Header/make_header no longer fail > ? if given binary unknown-8bit input. > > -- > Repository URL: http://hg.python.org/cpython > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > > From merwok at netwok.org Sun Jun 19 22:59:26 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sun, 19 Jun 2011 22:59:26 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.1): Fix closes issue12261 - Minor documention changes in the urllib.parse.rst In-Reply-To: References: Message-ID: <4DFE632E.2010308@netwok.org> Hi, Remember that 3.1 is in security mode, and as such will not get new documentation releases. See the previous threads about 2.6 docs or security releases for more info. Regards From senthil at uthcode.com Sun Jun 19 23:41:22 2011 From: senthil at uthcode.com (Senthil Kumaran) Date: Sun, 19 Jun 2011 14:41:22 -0700 Subject: [Python-Dev] [Python-checkins] cpython (3.1): Fix closes issue12261 - Minor documention changes in the urllib.parse.rst In-Reply-To: <4DFE632E.2010308@netwok.org> References: <4DFE632E.2010308@netwok.org> Message-ID: <20110619214122.GA3964@mathmagic> On Sun, Jun 19, 2011 at 10:59:26PM +0200, ?ric Araujo wrote: > > Remember that 3.1 is in security mode, and as such will not get new > documentation releases. See the previous threads about 2.6 docs or > security releases for more info. Thanks for the information. I missed that somehow. Noted and that check that thread. -- Senthil From rdmurray at bitdance.com Mon Jun 20 04:39:32 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 19 Jun 2011 22:39:32 -0400 Subject: [Python-Dev] [Python-checkins] cpython: #6771: Move wrapper function into __init__ and eliminate wrapper module In-Reply-To: References: Message-ID: <20110620023933.0ABC1250D3A@webabinitio.net> On Sun, 19 Jun 2011 15:40:01 -0400, Jim Jewett wrote: > Does this really need to be a bare except? No, but that's a separate bug report, which you are welcome to file. The issue I closed was about moving the existing code. -- R. David Murray http://www.bitdance.com From g.brandl at gmx.net Mon Jun 20 09:11:20 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 20 Jun 2011 09:11:20 +0200 Subject: [Python-Dev] cpython (3.2): Fix closes Issue12315 - Updates to http.client documentation. In-Reply-To: References: Message-ID: On 20.06.2011 02:00, senthil.kumaran wrote: > http://hg.python.org/cpython/rev/d801b570b1dd > changeset: 70882:d801b570b1dd > branch: 3.2 > parent: 70880:9e58a638f028 > user: Senthil Kumaran > date: Sun Jun 19 16:56:49 2011 -0700 > summary: > Fix closes Issue12315 - Updates to http.client documentation. > > files: > Doc/library/http.client.rst | 13 ++++++++++++- > 1 files changed, 12 insertions(+), 1 deletions(-) > > > diff --git a/Doc/library/http.client.rst b/Doc/library/http.client.rst > --- a/Doc/library/http.client.rst > +++ b/Doc/library/http.client.rst > @@ -543,6 +543,9 @@ > A debugging hook. If :attr:`debuglevel` is greater than zero, messages > will be printed to stdout as the response is read and parsed. > > +.. attribute:: HTTPResponse.closed > + > + Is True if the stream is closed. > > Examples > -------- This is not a big deal, and I'm not picking specially on you here, Senthil, it's just something that I've noticed several times: Newlines are a valuable tool for structuring reST files (just like in Python files). I tried to set up a convention to separate large blocks (such as sections) by two newlines, to make it easier to scroll and find what you're looking for. Please try to keep this intact. Thanks, Georg From g.brandl at gmx.net Mon Jun 20 09:13:07 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 20 Jun 2011 09:13:07 +0200 Subject: [Python-Dev] cpython (3.2): Fix closes Issue12359 - Minor update to module import description. In-Reply-To: References: Message-ID: On 20.06.2011 02:42, senthil.kumaran wrote: > http://hg.python.org/cpython/rev/bf8b4c43fb94 > changeset: 70886:bf8b4c43fb94 > branch: 3.2 > parent: 70884:4444afcfb22e > user: Senthil Kumaran > date: Sun Jun 19 17:37:06 2011 -0700 > summary: > Fix closes Issue12359 - Minor update to module import description. > > files: > Doc/tutorial/modules.rst | 15 ++++++++------- > 1 files changed, 8 insertions(+), 7 deletions(-) > > > diff --git a/Doc/tutorial/modules.rst b/Doc/tutorial/modules.rst > --- a/Doc/tutorial/modules.rst > +++ b/Doc/tutorial/modules.rst > @@ -159,13 +159,14 @@ > > .. index:: triple: module; search; path > > -When a module named :mod:`spam` is imported, the interpreter searches for a file > -named :file:`spam.py` in the current directory, and then in the list of > -directories specified by the environment variable :envvar:`PYTHONPATH`. This > -has the same syntax as the shell variable :envvar:`PATH`, that is, a list of > -directory names. When :envvar:`PYTHONPATH` is not set, or when the file is not > -found there, the search continues in an installation-dependent default path; on > -Unix, this is usually :file:`.:/usr/local/lib/python`. > +When a module named :mod:`spam` is imported, the interpreter searches for a > +file named :file:`spam.py` in the directory containing the input script (or > +thecurrent directory), and then in the list of directories specified by the By just adding "or the current directory", you've actually made this more confusing: now the reader will wonder when it's the script directory and when it's the current directory. Georg From senthil at uthcode.com Mon Jun 20 09:34:07 2011 From: senthil at uthcode.com (Senthil Kumaran) Date: Mon, 20 Jun 2011 00:34:07 -0700 Subject: [Python-Dev] cpython (3.2): Fix closes Issue12359 - Minor update to module import description. In-Reply-To: References: Message-ID: <20110620073407.GA9220@mathmagic> On Mon, Jun 20, 2011 at 09:13:07AM +0200, Georg Brandl wrote: > On 20.06.2011 02:42, senthil.kumaran wrote: > > summary: > > Fix closes Issue12359 - Minor update to module import description. > > ... > > +When a module named :mod:`spam` is imported, the interpreter searches for a > > +file named :file:`spam.py` in the directory containing the input script (or > > +thecurrent directory), and then in the list of directories specified by the > > By just adding "or the current directory", you've actually made this more > confusing: now the reader will wonder when it's the script directory and when > it's the current directory. I added that statement in the bracket, after looking at another instance in the next para which had this mention. I think, the point here is that the reader would understand, where the import is looking for the module based on the context. Fine with removing this sentence ("or the current directory"), if statement explains the idea better without it. -- Senthil From senthil at uthcode.com Mon Jun 20 09:36:02 2011 From: senthil at uthcode.com (Senthil Kumaran) Date: Mon, 20 Jun 2011 00:36:02 -0700 Subject: [Python-Dev] cpython (3.2): Fix closes Issue12315 - Updates to http.client documentation. In-Reply-To: References: Message-ID: <20110620073602.GB9220@mathmagic> On Mon, Jun 20, 2011 at 09:11:20AM +0200, Georg Brandl wrote: > Newlines are a valuable tool for structuring reST files (just like in Python > files). I tried to set up a convention to separate large blocks (such as > sections) by two newlines, to make it easier to scroll and find what you're > looking for. Please try to keep this intact. > Noted. In the next checkin to this file, I shall correct this one and add extra line before the Example section. Thanks, Senthil From fijall at gmail.com Mon Jun 20 10:08:04 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Mon, 20 Jun 2011 10:08:04 +0200 Subject: [Python-Dev] Language summit writeup anyone? Message-ID: Hi. Unfortunately I'm missing Europython (and language summit) this year. Did anyone do a writeup on what was discussed? Cheers, fijal From doug.hellmann at gmail.com Mon Jun 20 13:20:44 2011 From: doug.hellmann at gmail.com (Doug Hellmann) Date: Mon, 20 Jun 2011 07:20:44 -0400 Subject: [Python-Dev] Language summit writeup anyone? In-Reply-To: References: Message-ID: On Jun 20, 2011, at 4:08 AM, Maciej Fijalkowski wrote: > Hi. > > Unfortunately I'm missing Europython (and language summit) this year. > Did anyone do a writeup on what was discussed? Brian Curtin or I can help get the writeup posted to the Python Insider blog. I'm sure there are a lot of people who don't follow this list who would be interested in hearing about the outcome. Doug From solipsis at pitrou.net Mon Jun 20 13:31:31 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 20 Jun 2011 13:31:31 +0200 Subject: [Python-Dev] Language summit writeup anyone? In-Reply-To: References: Message-ID: <20110620133131.553e4c87@msiwind> Hi, Le Mon, 20 Jun 2011 10:08:04 +0200, Maciej Fijalkowski a ?crit : > > Unfortunately I'm missing Europython (and language summit) this year. > Did anyone do a writeup on what was discussed? Mark Dickinson has been taking notes, but since there only a few of us (roughly 10 attendants), it was mostly casual and friendly chatting :) Regards Antoine. From g.brandl at gmx.net Mon Jun 20 17:37:55 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 20 Jun 2011 17:37:55 +0200 Subject: [Python-Dev] cpython (3.2): Fix closes Issue12359 - Minor update to module import description. In-Reply-To: <20110620073407.GA9220@mathmagic> References: <20110620073407.GA9220@mathmagic> Message-ID: On 20.06.2011 09:34, Senthil Kumaran wrote: > On Mon, Jun 20, 2011 at 09:13:07AM +0200, Georg Brandl wrote: >> On 20.06.2011 02:42, senthil.kumaran wrote: >> > summary: >> > Fix closes Issue12359 - Minor update to module import description. >> > ... >> > +When a module named :mod:`spam` is imported, the interpreter searches for a >> > +file named :file:`spam.py` in the directory containing the input script (or >> > +thecurrent directory), and then in the list of directories specified by the >> >> By just adding "or the current directory", you've actually made this more >> confusing: now the reader will wonder when it's the script directory and when >> it's the current directory. > > I added that statement in the bracket, after looking at another > instance in the next para which had this mention. I think, the point > here is that the reader would understand, where the import is looking > for the module based on the context. Fine with removing this sentence > ("or the current directory"), if statement explains the idea better > without it. Thanks! Georg From opensourcesurvey at cs.ua.edu Mon Jun 20 21:45:27 2011 From: opensourcesurvey at cs.ua.edu (Jeffrey Carver) Date: Mon, 20 Jun 2011 14:45:27 -0500 Subject: [Python-Dev] REMINDER: Participation Requested: Survey about Open-Source Software Development Message-ID: <00f501cc2f82$9ada3f30$d08ebd90$@cs.ua.edu> Hi, Apologies for any inconvenience and thank you to those who have already completed the survey. We will keep the survey open for another couple of weeks. But, we do hope you will consider responding to the email request below (sent 2 weeks ago). Thanks, Dr. Jeffrey Carver Assistant Professor University of Alabama (v) 205-348-9829 (f) 205-348-0219 http://www.cs.ua.edu/~carver -----Original Message----- From: Jeffrey Carver [mailto:opensourcesurvey at cs.ua.edu] Sent: Monday, June 13, 2011 11:46 AM To: 'python-dev at python.org' Subject: Participation Requested: Survey about Open-Source Software Development Hi, Drs. Jeffrey Carver, Rosanna Guadagno, Debra McCallum, and Mr. Amiangshu Bosu, University of Alabama, and Dr. Lorin Hochstein, University of Southern California, are conducting a survey of open-source software developers. This survey seeks to understand how developers on distributed, virtual teams, like open-source projects, interact with each other to accomplish their tasks. You must be at least 19 years of age to complete the survey. The survey should take approximately 15 minutes to complete. If you are actively participating as a developer, please consider completing our survey. Here is the link to the survey: http://goo.gl/HQnux We apologize for inconvenience and if you receive multiple copies of this email. This survey has been approved by The University of Alabama IRB board. Thanks, Dr. Jeffrey Carver Assistant Professor University of Alabama (v) 205-348-9829 (f) 205-348-0219 http://www.cs.ua.edu/~carver From guido at python.org Tue Jun 21 01:40:02 2011 From: guido at python.org (Guido van Rossum) Date: Mon, 20 Jun 2011 16:40:02 -0700 Subject: [Python-Dev] Parser/intrcheck.c In-Reply-To: <20110616002140.77861fdd@pitrou.net> References: <20110616002140.77861fdd@pitrou.net> Message-ID: I think it's safe to remove it. The last reference to it I found was in the 2.0 release, where there is a Parser/Makefile (generated from Parser/Makefile.in) which contains the following gem: # This target is used by the master Makefile to add the objects to the library add2lib: $(OBJS) $(AR) cr $(LIBRARY) $(AROBJS) if test ! -f ../Modules/hassignal; \ then echo adding intrcheck.o; $(AR) r $(LIBRARY) intrcheck.o; \ else echo leaving intrcheck.o out; fi touch add2lib This Makefile.in was deleted in http://svn.python.org/view?view=revision So I think you're fine killing that file. --Guido On Wed, Jun 15, 2011 at 3:21 PM, Antoine Pitrou wrote: > > Hello, > > I may be missing something, but I'm wondering whether > Parser/intrcheck.c is still used anywhere. > > It's only mentioned in some comments: > > $ grep -r intrcheck.c * > Modules/signalmodule.c:1197:/* Replacements for intrcheck.c functionality > PC/os2vacpp/makefile.omk:217: ?# intrcheck.c ? ? -- Not Referenced by Anyone (?) > Python/sigcheck.c:3: ? interrupt occurs. ?It can't be in the intrcheck.c file since that > > And if I remove it and "make clean", I can still rebuild successfully. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From guido at python.org Tue Jun 21 01:40:27 2011 From: guido at python.org (Guido van Rossum) Date: Mon, 20 Jun 2011 16:40:27 -0700 Subject: [Python-Dev] Parser/intrcheck.c In-Reply-To: References: <20110616002140.77861fdd@pitrou.net> Message-ID: On Mon, Jun 20, 2011 at 4:40 PM, Guido van Rossum wrote: > I think it's safe to remove it. The last reference to it I found was > in the 2.0 release, where there is a Parser/Makefile (generated from > Parser/Makefile.in) which contains the following gem: > > # This target is used by the master Makefile to add the objects to the library > add2lib: ? ? ? ?$(OBJS) > ? ? ? ? ? ? ? ?$(AR) cr $(LIBRARY) $(AROBJS) > ? ? ? ? ? ? ? ?if test ! -f ../Modules/hassignal; \ > ? ? ? ? ? ? ? ?then echo adding intrcheck.o; $(AR) r $(LIBRARY) intrcheck.o; \ > ? ? ? ? ? ? ? ?else echo leaving intrcheck.o out; fi > ? ? ? ? ? ? ? ?touch add2lib > > This Makefile.in was deleted in http://svn.python.org/view?view=revision http://svn.python.org/view?view=revision&revision=19308 > So I think you're fine killing that file. > > --Guido > > On Wed, Jun 15, 2011 at 3:21 PM, Antoine Pitrou wrote: >> >> Hello, >> >> I may be missing something, but I'm wondering whether >> Parser/intrcheck.c is still used anywhere. >> >> It's only mentioned in some comments: >> >> $ grep -r intrcheck.c * >> Modules/signalmodule.c:1197:/* Replacements for intrcheck.c functionality >> PC/os2vacpp/makefile.omk:217: ?# intrcheck.c ? ? -- Not Referenced by Anyone (?) >> Python/sigcheck.c:3: ? interrupt occurs. ?It can't be in the intrcheck.c file since that >> >> And if I remove it and "make clean", I can still rebuild successfully. >> >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > -- > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) From dickinsm at gmail.com Tue Jun 21 10:53:46 2011 From: dickinsm at gmail.com (Mark Dickinson) Date: Tue, 21 Jun 2011 10:53:46 +0200 Subject: [Python-Dev] Language summit writeup anyone? In-Reply-To: <20110620133131.553e4c87@msiwind> References: <20110620133131.553e4c87@msiwind> Message-ID: On Mon, Jun 20, 2011 at 1:31 PM, Antoine Pitrou wrote: > Maciej Fijalkowski a ?crit : >> >> Unfortunately I'm missing Europython (and language summit) this year. >> Did anyone do a writeup on what was discussed? > > Mark Dickinson has been taking notes, but since there only a few of us > (roughly 10 attendants), it was mostly casual and friendly chatting :) Hi guys, I'm working on it. I'm soliciting feedback on a draft from those who were present at the meeting (if you *were* present at the meeting and didn't receive the draft writeup, please shout!). Once I've established that I haven't grossly misrepresented anyone, I'll send the writeup to python-dev. Mark From ziade.tarek at gmail.com Tue Jun 21 13:42:05 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 21 Jun 2011 13:42:05 +0200 Subject: [Python-Dev] packaging backport Message-ID: Hello I am starting the backport of Packaging into a standalone version named Distutils2 -- this is very important to speed up the development of packaging itself since it'll give of more feedback from 2.x projects To do this I need to create a script that does a mass renaming of 'packaging' into 'distutils2', then I can start to play with 3to2 and ...3.xto3.x :) . I tried to script rope but the py3k version seem not compatible with our current AST -- and it's a bit overkill for this task I guess. Before I start to write my own refactoring tool, I was wondering if anyone here had some experience in this, and could give me some hints. Thanks Tarek -- Tarek Ziad? | http://ziade.org From ncoghlan at gmail.com Tue Jun 21 14:44:43 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 21 Jun 2011 22:44:43 +1000 Subject: [Python-Dev] packaging backport In-Reply-To: References: Message-ID: 2to3 or Brett's mnfy are likely to be reasonable starting points. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Tue Jun 21 15:12:38 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 21 Jun 2011 09:12:38 -0400 Subject: [Python-Dev] packaging backport In-Reply-To: References: Message-ID: <20110621131238.B9DA72506D8@webabinitio.net> On Tue, 21 Jun 2011 13:42:05 +0200, =?ISO-8859-1?Q?Tarek_Ziad=E9?= wrote: > Before I start to write my own refactoring tool, I was wondering if > anyone here had some experience in this, and could give me some hints. Coul you could just write a 3to2 fixer? I don't know how hard it is to run just a selected set of fixers (so that you could use it to generate python3 code), but it seems to me that renaming modules is something that 3to2 (and 2to3, of course) should be good at. -- R. David Murray http://www.bitdance.com From ziade.tarek at gmail.com Tue Jun 21 15:24:26 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 21 Jun 2011 15:24:26 +0200 Subject: [Python-Dev] packaging backport In-Reply-To: References: Message-ID: On Tue, Jun 21, 2011 at 2:44 PM, Nick Coghlan wrote: > 2to3 or Brett's mnfy are likely to be reasonable starting points. Will look at mnfy, thanks -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Tue Jun 21 15:27:53 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 21 Jun 2011 15:27:53 +0200 Subject: [Python-Dev] packaging backport In-Reply-To: <20110621131238.B9DA72506D8@webabinitio.net> References: <20110621131238.B9DA72506D8@webabinitio.net> Message-ID: On Tue, Jun 21, 2011 at 3:12 PM, R. David Murray wrote: > On Tue, 21 Jun 2011 13:42:05 +0200, =?ISO-8859-1?Q?Tarek_Ziad=E9?= wrote: >> Before I start to write my own refactoring tool, I was wondering if >> anyone here had some experience in this, and could give me some hints. > > Coul you could just write a 3to2 fixer? ?I don't know how hard it is to > run just a selected set of fixers (so that you could use it to generate > python3 code), but it seems to me that renaming modules is something > that 3to2 (and 2to3, of course) should be good at. The one thing rope is good at is to find where a given variable name is used, and rename all occurrences recursively. So basically, when you rename an import, it renames all the code that uses it. I don't really know how 2to3/3to2 work but I assumed that it does not do this, but simply give you a hook for every visited node. IOW that looking for dependencies is to be done Cheers Tarek > > -- > R. David Murray ? ? ? ? ? http://www.bitdance.com > -- Tarek Ziad? | http://ziade.org From ziade.tarek at gmail.com Tue Jun 21 15:43:09 2011 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 21 Jun 2011 15:43:09 +0200 Subject: [Python-Dev] packaging backport In-Reply-To: References: <20110621131238.B9DA72506D8@webabinitio.net> Message-ID: On Tue, Jun 21, 2011 at 3:37 PM, Joe Amenta wrote: ... > > Yes, 2to3/3to2 does not do anything fancy like that. ?The tools are purely > concerned with syntax, whereas renaming imports is semantic. ?The good news > is that this line: > import packaging as _myPackaging > can be replaced by: > import distutils2 as _myPackaging > and code that uses _myPackaging will work. ?I've attached a fixer that can > go into your lib3to2/fixes folder. ?You should also edit fix_imports.py and > add the line: > "packaging" : "distutils2", > to the MAPPING dictionary near the top, then you can run 3to2 like this: > 3to2 -fpackaging -fimports files_to_fix.py > (-w option to write changes to the files modified) > Hope this helps, It does, thanks a lot ! Cheers Tarek -- Tarek Ziad? | http://ziade.org From amentajo at msu.edu Tue Jun 21 15:37:05 2011 From: amentajo at msu.edu (Joe Amenta) Date: Tue, 21 Jun 2011 09:37:05 -0400 Subject: [Python-Dev] packaging backport In-Reply-To: References: <20110621131238.B9DA72506D8@webabinitio.net> Message-ID: On Tue, Jun 21, 2011 at 9:27 AM, Tarek Ziad? wrote: > On Tue, Jun 21, 2011 at 3:12 PM, R. David Murray > wrote: > > On Tue, 21 Jun 2011 13:42:05 +0200, =?ISO-8859-1?Q?Tarek_Ziad=E9?= < > ziade.tarek at gmail.com> wrote: > >> Before I start to write my own refactoring tool, I was wondering if > >> anyone here had some experience in this, and could give me some hints. > > > > Coul you could just write a 3to2 fixer? I don't know how hard it is to > > run just a selected set of fixers (so that you could use it to generate > > python3 code), but it seems to me that renaming modules is something > > that 3to2 (and 2to3, of course) should be good at. > > The one thing rope is good at is to find where a given variable name > is used, and rename all occurrences recursively. So basically, when > you rename an import, it renames all the code that uses it. > > I don't really know how 2to3/3to2 work but I assumed that it does not > do this, but simply give you a hook for every visited node. IOW that > looking for dependencies is to be done > > Cheers > Tarek > > > > > -- > > R. David Murray http://www.bitdance.com > > > > > > -- > Tarek Ziad? | http://ziade.org > Yes, 2to3/3to2 does not do anything fancy like that. The tools are purely concerned with syntax, whereas renaming imports is semantic. The good news is that this line: import packaging as _myPackaging can be replaced by: import distutils2 as _myPackaging and code that uses _myPackaging will work. I've attached a fixer that can go into your lib3to2/fixes folder. You should also edit fix_imports.py and add the line: "packaging" : "distutils2", to the MAPPING dictionary near the top, then you can run 3to2 like this: 3to2 -fpackaging -fimports files_to_fix.py (-w option to write changes to the files modified) Hope this helps, --Joe Amenta -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: fix_packaging.py Type: application/octet-stream Size: 358 bytes Desc: not available URL: From vinay_sajip at yahoo.co.uk Wed Jun 22 02:06:39 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 22 Jun 2011 00:06:39 +0000 (UTC) Subject: [Python-Dev] In-Python virtualisation and packaging References: <4DF5FAC8.4090606@voidspace.org.uk> Message-ID: Nick Coghlan gmail.com> writes: > How well does the regression test suite cope when run inside such a > virtualised environment? I followed this up, and three tests fail: test_lib2to3, test_packaging and test_sysconfig. These are errors which show up on the default branch too [1][2]; full results are at [3]. I've been keeping the pythonv branch synchronised with default - these results appear to be quite stable/repeatable (old versions of the results are available in the gist at [3]). I did another test: in a pythonv-created environment, I tested pythonv/pysetup3 by trying to install all PyPI packages with a Python 3 trove classifier, where a source distribution can be found. This smoke test shows a total of 398 such packages, 310 of which were installed in the environment without errors. That's 78% - not too bad for this early stage in the game. The details of the failing 88 packages are at [4], and some of these are pysetup3 issues but a fair few are bugs in the packages themselves (e.g. SyntaxErrors in setup.py, or missing readme files that setup.py expects to be there) or missing dependencies like boost.python or similar C-level dependencies. These tests were done with a patched version of Distribute which uses sys.site_prefix is available, falling back to sys.prefix when not (so the Distribute change is backward compatible). Regards, Vinay Sajip [1] http://bugs.python.org/issue12331 [2] http://bugs.python.org/issue9100 [3] https://gist.github.com/1022705 [4] http://gist.github.com/1037662 From techtonik at gmail.com Wed Jun 22 14:47:01 2011 From: techtonik at gmail.com (anatoly techtonik) Date: Wed, 22 Jun 2011 15:47:01 +0300 Subject: [Python-Dev] Is there any fun with benchmarks Message-ID: I run across a snippet in SCons.Util (don't worry, I've double-checked To: field) that claims it is faster than os.path.splitext() while basically doing the same thing. def splitext(path): "Same as os.path.splitext() but faster." sep = rightmost_separator(path, os.sep) dot = path.rfind('.') # An ext is only real if it has at least one non-digit char if dot > sep and not containsOnly(path[dot:], "0123456789."): return path[:dot],path[dot:] else: return path,"" I wonder if upcoming speed.python.org has any means to validate these claims for different Python releases? Is there any place where I can upload my two to compare performance? Are there any instructions how to create such snippets and add/enhance dataset for them? Any plans or opinions if that will be useful or not? -- anatoly t. From p.f.moore at gmail.com Wed Jun 22 15:08:15 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 22 Jun 2011 14:08:15 +0100 Subject: [Python-Dev] Is there any fun with benchmarks In-Reply-To: References: Message-ID: On 22 June 2011 13:47, anatoly techtonik wrote: > I run across a snippet in SCons.Util (don't worry, I've double-checked > To: field) that claims it is faster than os.path.splitext() while > basically doing the same thing. Actually, it doesn't do the same thing. Doesn't handle files like .profile properly. Also, this one seems to treat numerics differently. So I'm not sure what you're trying to prove in a comparison...? Paul. From ncoghlan at gmail.com Wed Jun 22 15:24:38 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 Jun 2011 23:24:38 +1000 Subject: [Python-Dev] Is there any fun with benchmarks In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 10:47 PM, anatoly techtonik wrote: > I wonder if upcoming speed.python.org has any means to validate these > claims for different Python releases? > Is there any place where I can upload my two to compare performance? > Are there any instructions how to create such snippets and add/enhance > dataset for them? > Any plans or opinions if that will be useful or not? The timeit module handles microbenchmarks on short snippets without any real problems. speed.python.org is about *macro* benchmarks - getting a feel for the overall interpreter performance under a variety of real world workflows. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From fijall at gmail.com Wed Jun 22 17:20:53 2011 From: fijall at gmail.com (Maciej Fijalkowski) Date: Wed, 22 Jun 2011 17:20:53 +0200 Subject: [Python-Dev] Is there any fun with benchmarks In-Reply-To: References: Message-ID: On Wed, Jun 22, 2011 at 3:24 PM, Nick Coghlan wrote: > On Wed, Jun 22, 2011 at 10:47 PM, anatoly techtonik wrote: >> I wonder if upcoming speed.python.org has any means to validate these >> claims for different Python releases? >> Is there any place where I can upload my two to compare performance? >> Are there any instructions how to create such snippets and add/enhance >> dataset for them? >> Any plans or opinions if that will be useful or not? > > The timeit module handles microbenchmarks on short snippets without > any real problems. speed.python.org is about *macro* benchmarks - > getting a feel for the overall interpreter performance under a variety > of real world workflows. > > Cheers, > Nick. > I think the question that timeit doesn't answer and speed potentially can (I don't know if it should, but that's a matter of opinion) is how those numbers differ among various interpreters/OSes/versions. This is something for what you need a special offloaded server support From barry at python.org Wed Jun 22 18:39:08 2011 From: barry at python.org (Barry Warsaw) Date: Wed, 22 Jun 2011 12:39:08 -0400 Subject: [Python-Dev] PEP 382 sprint Message-ID: <20110622123908.4b65c1d5@neurotica.colubris.lan> Hi folks, Yesterday, 6 Washington DC area Pythonistas met to work on PEP 382. I wrote up a summary based on my notes and blogged about it here: http://www.wefearchange.org/2011/06/pep-382-sprint-summary.html Hopefully, the other participants will correct my mistakes and fill in the holes. A few other things to mention: * I resurrected the import-sig in order to shepherd PEP 382 to landing in Python 3.3. If you're at all interested in helping out, please join the mailing list: http://mail.python.org/mailman/admin/import-sig * We created a wiki page to track our results, questions, and plan of action: http://wiki.python.org/moin/Pep382Sprint I want to thank my fellow sprint participants for coming out and really doing a great job working on this. And I especially want to thank the PSF for sponsoring our sprint. What a great way to encourage Python developers to meet and work on improving Python! Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From g.brandl at gmx.net Wed Jun 22 21:40:57 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 22 Jun 2011 21:40:57 +0200 Subject: [Python-Dev] cpython: #1874: detect invalid multipart CTE and report it as a defect. In-Reply-To: References: Message-ID: On 22.06.2011 19:48, r.david.murray wrote: > http://hg.python.org/cpython/rev/5a2602939d5d > changeset: 70925:5a2602939d5d > user: R David Murray > date: Wed Jun 22 13:47:53 2011 -0400 > summary: > #1874: detect invalid multipart CTE and report it as a defect. > > files: > Lib/email/errors.py | 3 + > Lib/email/feedparser.py | 5 ++ > Lib/test/test_email/test_email.py | 45 +++++++++++++++++++ > Misc/NEWS | 3 + > 4 files changed, 56 insertions(+), 0 deletions(-) > > > diff --git a/Lib/email/errors.py b/Lib/email/errors.py > --- a/Lib/email/errors.py > +++ b/Lib/email/errors.py > @@ -55,3 +55,6 @@ > > class MultipartInvariantViolationDefect(MessageDefect): > """A message claimed to be a multipart but no subparts were found.""" > + > +class InvalidMultipartContentTransferEncodingDefect(MessageDefect): > + """An invalid content transfer encoding was set on the multipart itself.""" Dear Mr. Murray, thank you very much for competing in the PSU's Longest Class Name in the Standard Library competition. Unfortunately, your class name of 45 characters has been surpassed by four other contestants. However, it is my pleasure to inform you that your entry wins the consolation prize for the Longest Exception Name! A framed certificate and a PSU-branded wooden keyboard will be delivered to you shortly. Yours sincerely, the PSU ministry of silly stats From rdmurray at bitdance.com Wed Jun 22 23:40:12 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 22 Jun 2011 17:40:12 -0400 Subject: [Python-Dev] cpython: #1874: detect invalid multipart CTE and report it as a defect. In-Reply-To: References: Message-ID: <20110622214013.157E62506D2@webabinitio.net> On Wed, 22 Jun 2011 21:40:57 +0200, Georg Brandl wrote: > On 22.06.2011 19:48, r.david.murray wrote: > > diff --git a/Lib/email/errors.py b/Lib/email/errors.py > > --- a/Lib/email/errors.py > > +++ b/Lib/email/errors.py > > @@ -55,3 +55,6 @@ > > > > class MultipartInvariantViolationDefect(MessageDefect): > > """A message claimed to be a multipart but no subparts were found.""" > > + > > +class InvalidMultipartContentTransferEncodingDefect(MessageDefect): > > + """An invalid content transfer encoding was set on the multipart itself.""" > > Dear Mr. Murray, > > thank you very much for competing in the PSU's Longest Class Name in the > Standard Library competition. Unfortunately, your class name of 45 characters > has been surpassed by four other contestants. > > However, it is my pleasure to inform you that your entry wins the consolation > prize for the Longest Exception Name! A framed certificate and a PSU-branded > wooden keyboard will be delivered to you shortly. See, there are hidden benefits to following the existing coding conventions of stdlib modules... (I initially called it InvalidMultipartCTEDefect, but all of the other names were spelled out, so....) -- R. David Murray http://www.bitdance.com From dickinsm at gmail.com Fri Jun 24 10:52:40 2011 From: dickinsm at gmail.com (Mark Dickinson) Date: Fri, 24 Jun 2011 10:52:40 +0200 Subject: [Python-Dev] EuroPython Language Summit report Message-ID: EuroPython 2011 Language Summit =============================== Here's a brief report on the EuroPython 2011 Language Summit, held on Sunday 19 June, 2011 in Florence. It was a fairly small meeting, with a lot of informal and loosely-focused discussion and few conclusions reached. I've outlined the topics that we discussed below. Present: Antonio Cuni Mark Dickinson Larry Hastings (chair) Marc-Andr? Lemburg Ezio Melotti Antoine Pitrou Armin Ronacher Armin Rigo Mark Ramm Topics covered ============== Python 3 adoption ----------------- This was a long and wide-ranging discussion on the general state of Python 3 adoption. Some highlights: - Even amongst those present at the meeting, most are still using Python 2 for everyday work. - pypy: no definitive plans yet for PyPy supporting Python 3. - The web community is still very much focused on Python 2. - There's ongoing work (or perhaps just discussion?) on being able to generate Python 3 documentation using Sphinx running on Python 3. This would be particularly useful when using extensions like 'autodoc' for Python 3 code. - [Armin Ronacher] Python 3's Unicode support still has some dark areas. One example: when opening a text file for reading and writing, the default encoding used depends on the platform and on various environment variables. - Regular expression support in Python 3 needs improvement (support for Unicode character classes is a particular need). [Subtopic: what needs to be done to get the new regex module into Python? Should it replace the existing module? What about backwards compatibility issues?] - 2to3: It's still painful to upgrade a codebase from Python 2 to Python 3, and there doesn't seem to be a consensus on best practices yet. There was a long discussion that I can't hope to do justice to on whether 2to3 should be part of the standard library or not. (With side discussions on distutils and packaging in Python 3.) Armin Ronacher was one of the main participants in this discussion (and IIUC, the main opponent of having 2to3 in the standard library); Armin, do you want to summarize this? Another major topic of discussion was on how to manage Python 2 and Python 3 in plugin environments (e.g., Blender): is there some way that both Python 2 and Python 3 plugins could be used within the same system? This got a bit technical; perhaps someone else at the meeting would like to elaborate on this? State of Python 3 on the web ---------------------------- Brief discussion. Summary: it's improving; better than before. We have CherryPy and WSGI. There are issues in WSGI that are going to become more apparent as more people move to Python 3 frameworks. Discussion of open PEPS ----------------------- PEP 3151: (Reworking the OS and IO exception hierarchy.) Antoine summarized this; no real discussion here. PEP 3118: (Not open, but ...) Revising the buffer protocol. The buffer protocol implementation is still buggy and incomplete. (Mostly okay for 1d buffers, not even close for higher-dimensional work.) PEP 397: Python launcher for Windows. (This was actually discussed later, after lunch; apologies for messing with the timeline here.) There was some confusion over the purpose of this PEP. Armin Ronacher had some objections to the PEP in its current form, but it's not clear to me what those objects are, or whether they still exist. Armin, care to elaborate? PEP 0380: Syntax for delegating to a subgenerator. PEP 3150: Statement local namespaces. PEP 3152: Cofunctions. For all three of the above PEPs, there was some feeling that additional syntax changes to the language are unnecessary and make it harder to learn; where possible, we should prefer using 3rd partly libraries. Issue 12326 (Linux 3) --------------------- >From the issue report by Charles-Fran?ois Natali:: Linus recently decided that the next Linux kernel version would 3.0 [1]. As a consequence, sys.platform (which is actually MACHDEP) will be 'linux3' on machines running Linux 3 kernels, and tests checking explicitely against 'linux2' will either break and won't run. There was significant discussion on how this should be solved. While there's a lot of discussion already on the tracker, the choice of solution may have significant ramifications, so it seems a good idea that this be discussed more widely on python-dev. Major questions:: - what should be done with Python 2.7? As it stands, as I understand it, sys.platform *will* become 'linux3' on Linux 3.x (it's a computed value somewhere) in Python 2.7.2, and since Python 2.7.2 is already out there we can't change that. Some of the proposed solutions to the issue would involve sys.platform changing between Python 2.7.2 and Python 2.7.3; is such a change acceptable for a bugfix release? - it's easy to fix up all uses of "== 'linux2' " in the standard library; is it okay to just let 3rd party code break here? - What should be done about Lib/plat-linux2? The rough consensus at the summit was that for Python 2.7 at least, the only realistic solution seems to be to do nothing except document the problem, and point people to the platform module. I'm not sure if this answers the Lib/plat-linux2 question. Python 2.7 ---------- How long will this be maintained? Original decision was 5 years. PyPI on EC2 ----------- There was some discussion several months ago about the possibility of hosting PyPI on EC2. Is this still something that should be considered? PyPI seems to have been much more problem free in recent times. virtualenv in Python 3.3? ------------------------- Apparently there was some discussion at the last PyCon about the possibility of virtualenv going into Python 3.3. As far as I know there's currently no open tracker item or PEP for this. Larry Hastings knows more here. -- Update from Larry: """I'm supportive of putting it in in 3.3, and I'm sitting with Raymond Hettinger right now and he supports it too. So I think if we get a working implementation it'll go in. It's under heavy discussion in c.l.p-d so I gather it's moving forward. Last I knew it was Carl Meyer pushing it forward, but Vinay Sanjip is the current standard-bearer. I understand Windows support will be a bit tricky; I don't know if they have someone who's going to handle it for them.""" Python website -------------- The Python website is painful to edit right now. From victor.stinner at haypocalc.com Fri Jun 24 13:18:44 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 24 Jun 2011 13:18:44 +0200 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: References: Message-ID: <1308914324.2342.7.camel@marge> Le vendredi 24 juin 2011 ? 10:52 +0200, Mark Dickinson a ?crit : > - [Armin Ronacher] Python 3's Unicode support still has some dark areas. What? Unicode support is perfect in Python 3! > One example: when opening a text file for reading and writing, the default > encoding used depends on the platform and on various environment variables. ... oh, I agree. This choice is a big portability issue. Mac OS X, most Linux distro, BSD systems use UTF-8 local encoding, whereas Windows use legacy code pages like cp1252 (something like ISO-8859-1) or cp952 (shift jis). But sometimes, the locale is "C" (e.g. on our buildbots) and programs start to fail with Unicode errors... I see two options to improve the situation. (1) hard way: change open() API to make encoding a mandatory argument. Problem: it breaks compatibility with Python 3.0, 3.1 and 3.2 (ooops!); the encoding argument is the 4th argument, you have to use a keyword or choose a value for the buffering argument. I proposed to change open() API in Python 3.1 to make all arguments -except the filename and the mode- keyword-only arguments, but Guido rejected my idea: "Remember, for 3.0 we're trying to get a release out of the door, not cram in new features, no matter how small." http://bugs.python.org/issue4121 (2) soft way: add a warning if the encoding is implicit (locale encoding). I don't know what is the best warning type, and if it should be always displayed, only once, or not by default. Even if it is hidden by default, a careful developer will be able to use -Werror to fix bugs... I suspect that most tests fail if open() raises an exception if the encoding is not specified (e.g. see distutils/packaging issues about the file encoding). Victor From ncoghlan at gmail.com Fri Jun 24 14:50:05 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Jun 2011 22:50:05 +1000 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: References: Message-ID: On Fri, Jun 24, 2011 at 6:52 PM, Mark Dickinson wrote: > PEP 3118: (Not open, but ...) ?Revising the buffer protocol. ?The buffer > ? ?protocol implementation is still buggy and incomplete. ?(Mostly okay for 1d > ? ?buffers, not even close for higher-dimensional work.) Issue 10181 is the place to start for anyone that wants to help with this one! :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Fri Jun 24 15:08:09 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 24 Jun 2011 13:08:09 +0000 (UTC) Subject: [Python-Dev] EuroPython Language Summit report References: Message-ID: Mark Dickinson gmail.com> writes: > virtualenv in Python 3.3? > ------------------------- > > Apparently there was some discussion at the last PyCon about the possibility > of virtualenv going into Python 3.3. As far as I know there's currently no > open tracker item or PEP for this. Larry Hastings knows more here. > > Update from Larry: """I'm supportive of putting it in in 3.3, and I'm > sitting with Raymond Hettinger right now and he supports it too. So I > think if we get a working implementation it'll go in. It's under heavy > discussion in c.l.p-d so I gather it's moving forward. Last I knew it was > Carl Meyer pushing it forward, but Vinay Sajip is the current standard- > bearer. I understand Windows support will be a bit tricky; I don't know if > they have someone who's going to handle it for them.""" Mark, thanks for the summary. Re. a PEP for virtual environments in Python, Carl is working on the PEP. The first draft by Carl with my initial comments is at [1]. There's still some work to do before we can actually present it as a PEP we're happy with. I'm pleased to report good progress with the proof of concept implementation. There are some issues still with packaging which I'm working through with ?ric Araujo, but I've gone ahead and committed changes on my branch to get things working. It's a good exercising of the packaging functionality :-) What I have at the moment is pythonv [2]: A modified Python which supports virtual environments via changes in Modules/getpath.c, PC/getpathp.c, Lib/site.py, Lib/sysconfig.py, Lib/sysconfig.cfg and Lib/distutils/sysconfig.py. These changes basically detect if you're running in a virtual environment by looking for an env.cfg, and if found, fixing it up so sys.path is set as it should be. In addition, sys.site_prefix and sys.site_exec_prefix are created - if not in a virtual env, these have the same values as sys.prefix and sys.exec_prefix. With just these changes, the basics of a virtual environment are provided (in terms of isolation from other environments and the system Python). However, in order to be useful, the packaging tools have to work with sys.site_prefix and sys.site_exec_prefix, so changes have been made to sysconfig, distutils and packaging to do this. I'm presently working on a demonstration application which integrates the above work with Doug Hellmann's virtualenvwrapper and Guillermo L?pez' port of it to Powershell to get a nice basic tool that'll support virtual environments with packaging as an easy-to-use CLI, as well as Distribute support while people migrate over to packaging, on Windows as well as Linux. I'm expecting to cover both the Linux and Windows aspects (though I won't say no to help ;-) and working through packaging issues relating to improved Windows support. The basic functionality is there now, both in Windows and Linux - the focus of work is in the ease-of-use CLI stuff which is not envisaged to be part of core Python, but nevertheless needs to be done to make virtual envs as painless as possible. BTW Of the 398 packages on PyPI having a Py3K trove classifier, I've tested installing all of them into a pythonv virtual env with Distribute, and 310 install without errors. Of the remaining 88, some are due to missing dependencies, others due to broken packages on PyPI. BTW of the Python regression test suite passes all tests in a pythonv virtualenv, other than those for which there already are tracker issues for the default branch in hg.python.org/cpython (test_lib2to3, test_packaging, test_sysconfig). Full test results are at [3]. All in all, it's looking reasonably good, and we hope to report more progress on the PEP and the proof of concept soon! Regards, Vinay Sajip [1] http://goo.gl/6u0S0 [2] https://bitbucket.org/vinay.sajip/pythonv [3] https://gist.github.com/1022705 From rdmurray at bitdance.com Fri Jun 24 16:55:15 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 24 Jun 2011 10:55:15 -0400 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: References: Message-ID: <20110624145516.84DA32506EA@webabinitio.net> On Fri, 24 Jun 2011 10:52:40 +0200, Mark Dickinson wrote: > EuroPython 2011 Language Summit > =============================== [...] > Unicode character classes is a particular need). [Subtopic: what needs > to be done to get the new regex module into Python? Should it replace > the existing module? What about backwards compatibility issues?] I'm pretty sure regex has backward compatibility as a goal for just this reason (so it can replace the current module). > PEP 0380: Syntax for delegating to a subgenerator. > > PEP 3150: Statement local namespaces. > > PEP 3152: Cofunctions. > > For all three of the above PEPs, there was some feeling that additional syntax > changes to the language are unnecessary and make it harder to learn; where > possible, we should prefer using 3rd partly libraries. I disagree with this with respect to 380. Haven't looked at the others. > Python 2.7 > ---------- > > How long will this be maintained? > Original decision was 5 years. As I recall, the actual decision was "at *least* 5 years". It's only been one so far...was the discussion that five years was too short or too long? As the code bases continue to drift apart, and we fix the fixable bugs, the number of patches going in to 2.7 will decrease over time, so I don't think the burden of continuing to support it will be too heavy. Currently about half of the non-feature-request issues (ie: bugs) in the tracker are tagged 2.7. I expect to see that percentage continue to decrease over time. -- R. David Murray http://www.bitdance.com From status at bugs.python.org Fri Jun 24 18:07:25 2011 From: status at bugs.python.org (Python tracker) Date: Fri, 24 Jun 2011 18:07:25 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20110624160725.00E5B1D41C@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2011-06-17 - 2011-06-24) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 2845 ( +2) closed 21335 (+43) total 24180 (+45) Open issues with patches: 1235 Issues opened (30) ================== #3067: setlocale error message is confusing http://bugs.python.org/issue3067 reopened by terry.reedy #12354: packaging.pypi.simple docs use both client and crawler variabl http://bugs.python.org/issue12354 opened by gruszczy #12355: Crawler doesn't follow redirection http://bugs.python.org/issue12355 opened by gruszczy #12361: Memory Leak in File Logging http://bugs.python.org/issue12361 opened by japerk #12364: Timeout (1 hour) in test_concurrent_futures.tearDown() on spar http://bugs.python.org/issue12364 opened by haypo #12365: URLopener should support context manager protocol http://bugs.python.org/issue12365 opened by mcjeff #12366: packaging.pypi.dist should abstract download errors. http://bugs.python.org/issue12366 opened by michael.mulich #12368: packaging.pypi.simple.Crawler assumes external download links http://bugs.python.org/issue12368 opened by michael.mulich #12372: semaphore errors on AIX 7.1 http://bugs.python.org/issue12372 opened by reshmi.george #12374: Execution model should explain compile vs definition vs execut http://bugs.python.org/issue12374 opened by ncoghlan #12375: Add packages_root to sys.path for hooks http://bugs.python.org/issue12375 opened by erik.bray #12376: unittest.TextTestResult.__init__ breaks under complex __mro__ http://bugs.python.org/issue12376 opened by branker #12377: Clean up use of packages_root/package_dir http://bugs.python.org/issue12377 opened by erik.bray #12378: smtplib.SMTP_SSL leaks socket connections on SSL error http://bugs.python.org/issue12378 opened by joeshaw #12379: build outside source fail in head http://bugs.python.org/issue12379 opened by rpetrov #12380: bytearray methods center, ljust, rjust don't accept a bytearra http://bugs.python.org/issue12380 opened by py.user #12381: bytearray methods count, find, index don't support None as in http://bugs.python.org/issue12381 opened by py.user #12382: [msilib] Obscure exception message when trying to open a non-e http://bugs.python.org/issue12382 opened by Robin.Jarry #12384: difflib.SequenceMatcher and Match: code and doc bugs http://bugs.python.org/issue12384 opened by terry.reedy #12385: the help for bytearray.maketrans describes bytes.maketrans http://bugs.python.org/issue12385 opened by py.user #12386: packaging fails in install_distinfo when writing RESOURCES http://bugs.python.org/issue12386 opened by vinay.sajip #12387: IDLE save keyboard shortcut problem http://bugs.python.org/issue12387 opened by Jacob.VB #12388: cannot specify recursive extra_files in packaging setup.cfg http://bugs.python.org/issue12388 opened by vinay.sajip #12391: packaging install fails to clean up temp files http://bugs.python.org/issue12391 opened by vinay.sajip #12392: pthread_kill() doesn't work on the main thread on FreeBSD6 http://bugs.python.org/issue12392 opened by haypo #12393: Packaging should provide support for extensible categories http://bugs.python.org/issue12393 opened by vinay.sajip #12394: packaging: generate scripts from callable (dotted paths) http://bugs.python.org/issue12394 opened by vinay.sajip #12395: packaging remove fails under Windows http://bugs.python.org/issue12395 opened by vinay.sajip #12397: re match object methods have no docstrings http://bugs.python.org/issue12397 opened by nedbat #12398: Sending binary data with a POST request in httplib can cause U http://bugs.python.org/issue12398 opened by sorin Most recent 15 issues with no replies (15) ========================================== #12398: Sending binary data with a POST request in httplib can cause U http://bugs.python.org/issue12398 #12397: re match object methods have no docstrings http://bugs.python.org/issue12397 #12391: packaging install fails to clean up temp files http://bugs.python.org/issue12391 #12376: unittest.TextTestResult.__init__ breaks under complex __mro__ http://bugs.python.org/issue12376 #12347: add an "extras" in packaging.pypi.simple.Crawler http://bugs.python.org/issue12347 #12336: tkinter.test.test_ttk.test_widgets.test_select() failure on Fr http://bugs.python.org/issue12336 #12323: ElementPath 1.3 expressions http://bugs.python.org/issue12323 #12322: ElementPath 1.3 expressions documentation http://bugs.python.org/issue12322 #12317: inspect.getabsfile() is not documented http://bugs.python.org/issue12317 #12297: Clarifications to atexit.register and unregister doc http://bugs.python.org/issue12297 #12295: Fix ResourceWarning in turtledemo help window http://bugs.python.org/issue12295 #12294: multiprocessing.Pool: Need a way to find out if work are finis http://bugs.python.org/issue12294 #12288: Python 2.7.1 tkSimpleDialog initialvalue http://bugs.python.org/issue12288 #12259: Test and document which compilers can be created on which plat http://bugs.python.org/issue12259 #12258: Clean up bytes I/O in get_compiler_versions http://bugs.python.org/issue12258 Most recent 15 issues waiting for review (15) ============================================= #12392: pthread_kill() doesn't work on the main thread on FreeBSD6 http://bugs.python.org/issue12392 #12372: semaphore errors on AIX 7.1 http://bugs.python.org/issue12372 #12354: packaging.pypi.simple docs use both client and crawler variabl http://bugs.python.org/issue12354 #12353: argparse cannot handle empty arguments http://bugs.python.org/issue12353 #12350: Improve stat_result.st_blocks and st_blksize documentation http://bugs.python.org/issue12350 #12344: Add **kwargs to get_reinitialized_command http://bugs.python.org/issue12344 #12341: Some additions to .hgignore http://bugs.python.org/issue12341 #12328: multiprocessing's overlapped PipeConnection on Windows http://bugs.python.org/issue12328 #12308: Add link to PEP 0 for topical index in wiki http://bugs.python.org/issue12308 #12306: zlib: Expose zlibVersion to query runtime version of zlib http://bugs.python.org/issue12306 #12304: expose signalfd(2) in the signal module http://bugs.python.org/issue12304 #12303: expose sigwaitinfo() and sigtimedwait() in the signal module http://bugs.python.org/issue12303 #12297: Clarifications to atexit.register and unregister doc http://bugs.python.org/issue12297 #12296: Minor clarification in devguide http://bugs.python.org/issue12296 #12295: Fix ResourceWarning in turtledemo help window http://bugs.python.org/issue12295 Top 10 most discussed issues (10) ================================= #12291: file written using marshal in 3.2 can be read by 2.7, but not http://bugs.python.org/issue12291 14 msgs #12394: packaging: generate scripts from callable (dotted paths) http://bugs.python.org/issue12394 14 msgs #12326: Linux 3: tests should avoid using sys.platform == 'linux2' http://bugs.python.org/issue12326 11 msgs #4470: smtplib SMTP_SSL not working. http://bugs.python.org/issue4470 8 msgs #12255: A few changes to .*ignore http://bugs.python.org/issue12255 8 msgs #2202: urllib2 fails against IIS 6.0 (No support for MD5-sess auth) http://bugs.python.org/issue2202 7 msgs #3067: setlocale error message is confusing http://bugs.python.org/issue3067 7 msgs #12353: argparse cannot handle empty arguments http://bugs.python.org/issue12353 7 msgs #10206: python program starting with unmatched quote spews spaces to s http://bugs.python.org/issue10206 6 msgs #11812: transient socket failure to connect to 'localhost' http://bugs.python.org/issue11812 6 msgs Issues closed (41) ================== #1874: email parser does not register a defect for invalid Content-Tr http://bugs.python.org/issue1874 closed by r.david.murray #4015: Make installed scripts executable on windows http://bugs.python.org/issue4015 closed by eric.araujo #5905: strptime fails in non-UTF locale http://bugs.python.org/issue5905 closed by haypo #6697: Check that _PyUnicode_AsString() result is not NULL http://bugs.python.org/issue6697 closed by haypo #6734: Imap lib implicit conversion from bytes to string http://bugs.python.org/issue6734 closed by r.david.murray #6771: Curses.wrapper: documentation/implementation error http://bugs.python.org/issue6771 closed by r.david.murray #9921: os.path.join('x','') behavior http://bugs.python.org/issue9921 closed by r.david.murray #10354: tempfile.template is broken http://bugs.python.org/issue10354 closed by r.david.murray #10636: subprocess module has race condition with SIGCHLD handlers http://bugs.python.org/issue10636 closed by rosslagerwall #11637: Add cwd to sys.path for hooks http://bugs.python.org/issue11637 closed by eric.araujo #11690: Devguide: Add "communication" FAQ http://bugs.python.org/issue11690 closed by ncoghlan #11700: mailbox.py proxy close method cannot be called multiple times http://bugs.python.org/issue11700 closed by r.david.murray #11781: test/test_email directory does not get installed by 'make inst http://bugs.python.org/issue11781 closed by r.david.murray #11795: Better core dev guidelines for committing submitted patches http://bugs.python.org/issue11795 closed by ncoghlan #12090: 3.2: build --without-threads fails http://bugs.python.org/issue12090 closed by georg.brandl #12223: Datamodel documentation page: 'operator' where 'operand' shoul http://bugs.python.org/issue12223 closed by eli.bendersky #12261: urllib.parse docs still refer to urlparse http://bugs.python.org/issue12261 closed by python-dev #12278: Core mentorship mention in the devguide http://bugs.python.org/issue12278 closed by python-dev #12285: Unexpected behavior for 0 or negative processes in multiproces http://bugs.python.org/issue12285 closed by haypo #12289: http.server.CGIHTTPRequestHandler doesn't check if a Python sc http://bugs.python.org/issue12289 closed by python-dev #12313: make install misses test dirs for packaging and email modules http://bugs.python.org/issue12313 closed by vinay.sajip #12315: Improve http.client.HTTPResponse.read documentation http://bugs.python.org/issue12315 closed by python-dev #12337: Need real TextIOWrapper for stdin/stdout http://bugs.python.org/issue12337 closed by haypo #12356: more argument error improving http://bugs.python.org/issue12356 closed by python-dev #12357: Python dist modifications for secure PyPI uploads http://bugs.python.org/issue12357 closed by eric.araujo #12358: validate server certificate when uploading packages to PyPI http://bugs.python.org/issue12358 closed by eric.araujo #12359: tutorial: Module search path description is incorrect http://bugs.python.org/issue12359 closed by python-dev #12360: Doc Typo http://bugs.python.org/issue12360 closed by python-dev #12362: General Windows stdout redirection not working http://bugs.python.org/issue12362 closed by r.david.murray #12363: test_signal.test_without_siginterrupt() sporadic failures on F http://bugs.python.org/issue12363 closed by python-dev #12367: select.error has no errno attribute http://bugs.python.org/issue12367 closed by pitrou #12369: Revised core mentorship section of help.rst http://bugs.python.org/issue12369 closed by ncoghlan #12370: Use of super overwrites use of __class__ in class namespace http://bugs.python.org/issue12370 closed by python-dev #12371: datetime.now() bug http://bugs.python.org/issue12371 closed by haypo #12373: Duplicate packets in Multicast Receiver http://bugs.python.org/issue12373 closed by neologix #12383: subprocess.Popen(..., env={}) fails to pass empty env. http://bugs.python.org/issue12383 closed by python-dev #12389: typo in urllib: missing space after dot at sentence end http://bugs.python.org/issue12389 closed by r.david.murray #12390: urllib.parse.urlencode encoding lists as strings http://bugs.python.org/issue12390 closed by orsenthil #12396: Delivery failed http://bugs.python.org/issue12396 closed by eric.araujo #976869: Stripping script extensions with distutils http://bugs.python.org/issue976869 closed by eric.araujo #870479: Scripts need platform-dependent handling http://bugs.python.org/issue870479 closed by eric.araujo From tjreedy at udel.edu Fri Jun 24 22:30:06 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 24 Jun 2011 16:30:06 -0400 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: <1308914324.2342.7.camel@marge> References: <1308914324.2342.7.camel@marge> Message-ID: On 6/24/2011 7:18 AM, Victor Stinner wrote: > Le vendredi 24 juin 2011 ? 10:52 +0200, Mark Dickinson a ?crit : >> One example: when opening a text file for reading and writing, the default >> encoding used depends on the platform and on various environment variables. > > ... oh, I agree. This choice is a big portability issue. Mac OS X, most > Linux distro, BSD systems use UTF-8 local encoding, whereas Windows use > legacy code pages like cp1252 (something like ISO-8859-1) or cp952 > (shift jis). But sometimes, the locale is "C" (e.g. on our buildbots) > and programs start to fail with Unicode errors... > > I see two options to improve the situation. The third is to make utf-8 the default. I believe this *is* the proper long term solution and both options are contrary to this. I believe that this is what I want for myself even on Windows. I believe utf-8 is the default or only storage by cross-platform international programs (certainly the ones I use). > (1) hard way: change open() API to make encoding a mandatory argument. > (2) soft way: add a warning if the encoding is implicit (locale > encoding). (3) In 3.3, if the default is used and it is not utf-8, add a warning that the default will become utf-8 always in 3.4. Actually, I would like a PendingDeprecationWarning in 3.2.1 if possible. -- Terry Jan Reedy From v+python at g.nevcal.com Fri Jun 24 22:45:53 2011 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 24 Jun 2011 13:45:53 -0700 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: References: <1308914324.2342.7.camel@marge> Message-ID: <4E04F781.60001@g.nevcal.com> On 6/24/2011 1:30 PM, Terry Reedy wrote: > > The third is to make utf-8 the default. I believe this *is* the proper > long term solution and both options are contrary to this. +1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at haypocalc.com Fri Jun 24 23:08:18 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 24 Jun 2011 23:08:18 +0200 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: References: <1308914324.2342.7.camel@marge> Message-ID: <1308949698.3352.24.camel@marge> Le vendredi 24 juin 2011 ? 16:30 -0400, Terry Reedy a ?crit : > > I see two options to improve the situation. > > The third is to make utf-8 the default. I believe this *is* the proper > long term solution and both options are contrary to this. Oh yes, I also prefer this option, but I suspect that some people prefer to not break backward compatibility. Or should we consider this bad design choice as a bug? The UTF-8 encoder (of Python 3) raises an error if the text contains a surrogate character. The surrogatepass error handler should be used to encode surrogages. ... The surrogateescape can be used to encode back undecodable bytes (e.g. filename decoded by Python using the surrogateescape), but it is not a good idea to write illegal byte sequences (most programs will fail to open the file). > I believe that this is what I want for myself even on Windows. Can you open a UTF-8 files in all Windows program (and the text is displayed correctly)? I remember that notepad.exe writes an evil UTF-8 BOM, but I don't know if it requires this BOM to detect the UTF-8 encoding. Or do some program expect text files encoded to the ANSI code page? If you want to write files in the ANSI code page, you can use encoding="mbcs" (or use an explicit code page, like encoding="cp1252"). > (3) In 3.3, if the default is used and it is not utf-8, add a warning > that the default will become utf-8 always in 3.4. Actually, I would like > a PendingDeprecationWarning in 3.2.1 if possible. I'm not sure that the "and it is not utf-8" condition is a good idea. If you develop on Linux but your users are on Windows, you will not get the warning (even with -Werror) nor your users (users don't want to see warnings)... Or maybe an user using Windows and Linux will notice the bug without the warning :-) It doesn't mean that it is not possible to check your program: you can change your locale encoding (e.g. use LANG=C). At least, it will be possible to check test_distutils and test_packaging using LANG=C and -Werror :-) -- A fourth option is to use ASCII by default! Your program will work and be portable until you write the first non-ASCII character... Oh wait, it remembers me the Unicode nightmare of Python 2! Victor From ncoghlan at gmail.com Sat Jun 25 02:46:39 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Jun 2011 10:46:39 +1000 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: <20110624145516.84DA32506EA@webabinitio.net> References: <20110624145516.84DA32506EA@webabinitio.net> Message-ID: On Sat, Jun 25, 2011 at 12:55 AM, R. David Murray wrote: [quoting VM summit notes] >> PEP 0380: Syntax for delegating to a subgenerator. >> >> PEP 3150: Statement local namespaces. >> >> PEP 3152: Cofunctions. >> >> For all three of the above PEPs, there was some feeling that additional syntax >> changes to the language are unnecessary and make it harder to learn; where >> possible, we should prefer using 3rd partly libraries. > > I disagree with this with respect to 380. ?Haven't looked at the others. Indeed, PEP 380 is *really* hard to do properly without language support. The language moratorium and lack of a Python 3 compatible patch are the only reasons it didn't go into 3.2 (and there's a patch porting the implementation to 3.3 up on bitbucket, although we've been tinkering with the compiler so it is likely out of date at this point). I'm the author PEP 3150 and *I* think it's a highly questionable and most likely terrible idea (hence the Deferred status). However, it's a good place to gather the related discussions, since proposals in that vein recur often on python-ideas. PEP 3152 definitely fits into the "let third parties experiment" bucket, though - once PEP 380 makes generators first class peers to functions in their support for modularisation, then we need to let the wider community play with the concept for a while before we embed anything more into the core language or standard library. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From greg.ewing at canterbury.ac.nz Sat Jun 25 13:27:25 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 25 Jun 2011 23:27:25 +1200 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: References: <20110624145516.84DA32506EA@webabinitio.net> Message-ID: <4E05C61D.9090402@canterbury.ac.nz> Nick Coghlan wrote: > Indeed, PEP 380 is *really* hard to do properly without language > support. The language moratorium and lack of a Python 3 compatible > patch Pardon? My original patch was for 3.1.2. -- Greg From ncoghlan at gmail.com Sat Jun 25 14:55:48 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Jun 2011 22:55:48 +1000 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: <4E05C61D.9090402@canterbury.ac.nz> References: <20110624145516.84DA32506EA@webabinitio.net> <4E05C61D.9090402@canterbury.ac.nz> Message-ID: On Sat, Jun 25, 2011 at 9:27 PM, Greg Ewing wrote: > Nick Coghlan wrote: > >> Indeed, PEP 380 is *really* hard to do properly without language >> support. The language moratorium and lack of a Python 3 compatible >> patch > > Pardon? My original patch was for 3.1.2. My mistake. We must have changed something else somewhere along the line that broke your patch... Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From pje at telecommunity.com Sat Jun 25 17:31:42 2011 From: pje at telecommunity.com (P.J. Eby) Date: Sat, 25 Jun 2011 11:31:42 -0400 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: References: <20110624145516.84DA32506EA@webabinitio.net> Message-ID: <20110625153157.4B1063A4093@sparrow.telecommunity.com> At 10:46 AM 6/25/2011 +1000, Nick Coghlan wrote: >Indeed, PEP 380 is *really* hard to do properly without language >support. No, it isn't. You add a decorator, a 'from_' class, and a 'return_' function, and there you go. (See my previous code sketches here in early PEP 380 discussions.) Python frameworks have been doing variations of the same thing (with varying features and APIs) for at least 7 years now -- even on Python versions that lack decorators or the ability to return values from yield statements. So the main benefit of a PEP for this functionality would be providing a common implementation/API - and that could be initially done in the stdlib, without any added syntax support. From rdmurray at bitdance.com Sat Jun 25 18:32:41 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Sat, 25 Jun 2011 12:32:41 -0400 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: <20110625153157.4B1063A4093@sparrow.telecommunity.com> References: <20110624145516.84DA32506EA@webabinitio.net> <20110625153157.4B1063A4093@sparrow.telecommunity.com> Message-ID: <20110625163242.349EC2506ED@webabinitio.net> On Sat, 25 Jun 2011 11:31:42 -0400, "P.J. Eby" wrote: > At 10:46 AM 6/25/2011 +1000, Nick Coghlan wrote: > >Indeed, PEP 380 is *really* hard to do properly without language > >support. > > No, it isn't. You add a decorator, a 'from_' class, and a 'return_' > function, and there you go. (See my previous code sketches here in > early PEP 380 discussions.) > > Python frameworks have been doing variations of the same thing (with > varying features and APIs) for at least 7 years now -- even on Python > versions that lack decorators or the ability to return values from > yield statements. > > So the main benefit of a PEP for this functionality would be > providing a common implementation/API - and that could be initially > done in the stdlib, without any added syntax support. So your proposed code would allow me, when writing a generator in my code, do something that would allow me to yield up all the values from an arbitrary generator I'm calling, over which I have no control (ie: I can't modify its code)? -- R. David Murray http://www.bitdance.com From riscutiavlad at gmail.com Sat Jun 25 18:33:04 2011 From: riscutiavlad at gmail.com (Vlad Riscutia) Date: Sat, 25 Jun 2011 09:33:04 -0700 Subject: [Python-Dev] ctypes: Configurable bitfield allocation strategy Message-ID: I recently started looking at some ctypes issues. I dug a bit into http://bugs.python.org/issue6069 and then I found http://bugs.python.org/issue11920. They both boil down to the fact that bitfield allocation is up to the compiler, which is different in GCC and MSVC. Currently we have hard-coded allocation strategy based on paltform in cfields.c: > if (bitsize /* this is a bitfield request */ > && *pfield_size /* we have a bitfield open */ > #ifdef MS_WIN32 > /* MSVC, GCC with -mms-bitfields */ > && dict->size * 8 == *pfield_size > #else > /* GCC */ > && dict->size * 8 <= *pfield_size > #endif > && (*pbitofs + bitsize) <= *pfield_size) { > /* continue bit field */ > fieldtype = CONT_BITFIELD; > #ifndef MS_WIN32 > } else if (bitsize /* this is a bitfield request */ > && *pfield_size /* we have a bitfield open */ > && dict->size * 8 >= *pfield_size > && (*pbitofs + bitsize) <= dict->size * 8) { > /* expand bit field */ > fieldtype = EXPAND_BITFIELD; > #endif So when creating a bitfield structure, it's size can be different on Linux vs Windows. class MyStructure(ctypes.BigEndianStructure): _pack_ = 1 # aligned to 8 bits, not ctypes default of 32 _fields_ = [ ("Data0", ctypes.c_uint32, 32), ("Data1", ctypes.c_uint8, 3), ("Data2", ctypes.c_uint16, 12), ] sizeof for above structure is 6 on GCC build and 7 on MSVC build. This leads to some confusion and issues, because we can't always interop correctly with code compiled with different compiler than the one Python is compiled with on the platform. Short term solution is to add a warning in the documentation about this. Longer term though, I think it would be better to add a property on the Structure class for configurable allocation strategy, for example Native (default), GCC, MSVC and when allocating the bitfield, use given strategy. Native would behave similar to what happens now, but we would also allow GCC-style allocation on Windows for example and the ability to extend this if we ever run into similar issues with other compilers. This shouldn't be too difficult to implement, will be backwards compatible and it would improve interop. I would like to hear some opinions on this. Thank you, Vlad -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Jun 25 18:44:36 2011 From: guido at python.org (Guido van Rossum) Date: Sat, 25 Jun 2011 09:44:36 -0700 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: <20110625163242.349EC2506ED@webabinitio.net> References: <20110624145516.84DA32506EA@webabinitio.net> <20110625153157.4B1063A4093@sparrow.telecommunity.com> <20110625163242.349EC2506ED@webabinitio.net> Message-ID: On Sat, Jun 25, 2011 at 9:32 AM, R. David Murray wrote: > On Sat, 25 Jun 2011 11:31:42 -0400, "P.J. Eby" wrote: >> At 10:46 AM 6/25/2011 +1000, Nick Coghlan wrote: >> >Indeed, PEP 380 is *really* hard to do properly without language >> >support. >> >> No, it isn't. ?You add a decorator, a 'from_' class, and a 'return_' >> function, and there you go. ?(See my previous code sketches here in >> early PEP 380 discussions.) >> >> Python frameworks have been doing variations of the same thing (with >> varying features and APIs) for at least 7 years now -- even on Python >> versions that lack decorators or the ability to return values from >> yield statements. >> >> So the main benefit of a PEP for this functionality would be >> providing a common implementation/API - and that could be initially >> done in the stdlib, without any added syntax support. > > So your proposed code would allow me, when writing a generator in > my code, do something that would allow me to yield up all the > values from an arbitrary generator I'm calling, over which I have > no control (ie: I can't modify its code)? Let me cut this short. PEP 380 is pretty much approved. I know there are a few details worth quibbling over, but they are not going to jeopardize acceptance of the PEP. We are waiting for an implementation in Python 3.3. In fact, I wouldn't mind at this point if someone took their best effort at an implementation and checked it into the 3.3 branch, and we can do the quibbling over the details while we have a working implementation to experiment with. -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Sat Jun 25 21:09:24 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 25 Jun 2011 15:09:24 -0400 Subject: [Python-Dev] ctypes: Configurable bitfield allocation strategy In-Reply-To: References: Message-ID: On 6/25/2011 12:33 PM, Vlad Riscutia wrote: > I recently started looking at some ctypes issues. I dug a bit into > http://bugs.python.org/issue6069 and then I found > http://bugs.python.org/issue11920. They both boil down to the fact that > bitfield allocation is up to the compiler, which is different in GCC and > MSVC. Currently we have hard-coded allocation strategy based on paltform > in cfields.c: > >> if (bitsize /* this is a bitfield request */ > >> && *pfield_size /* we have a bitfield open */ >> #ifdef MS_WIN32 >> /* MSVC, GCC with -mms-bitfields */ >> && dict->size * 8 == *pfield_size >> #else >> /* GCC */ >> && dict->size * 8<= *pfield_size >> #endif >> && (*pbitofs + bitsize)<= *pfield_size) { >> /* continue bit field */ >> fieldtype = CONT_BITFIELD; >> #ifndef MS_WIN32 >> } else if (bitsize /* this is a bitfield request */ >> && *pfield_size /* we have a bitfield open */ >> && dict->size * 8>= *pfield_size >> && (*pbitofs + bitsize)<= dict->size * 8) { >> /* expand bit field */ >> fieldtype = EXPAND_BITFIELD; >> #endif > > So when creating a bitfield structure, it's size can be different on > Linux vs Windows. > > class MyStructure(ctypes.BigEndianStructure): > _pack_ = 1 # aligned to 8 bits, not ctypes default of 32 > _fields_ = [ > ("Data0", ctypes.c_uint32, 32), > ("Data1", ctypes.c_uint8, 3), > ("Data2", ctypes.c_uint16, 12), > ] > > sizeof for above structure is 6 on GCC build and 7 on MSVC build. This > leads to some confusion and issues, because we can't always interop > correctly with code compiled with different compiler than the one Python > is compiled with on the platform. Just curious, are you saying that this is the 'cause' of the two bug reports, or 'just' something you discovered while investigating them? > Short term solution is to add a warning in the documentation about this. For 2.7/3.2, yes. > Longer term though, I think it > would be better to add a property on the Structure class for > configurable allocation strategy, for example Native (default), GCC, > MSVC and when allocating the bitfield, use given strategy. Native would > behave similar to what happens now, but we would also allow GCC-style > allocation on Windows for example and the ability to extend this if we > ever run into similar issues with other compilers. This shouldn't be too > difficult to implement, will be backwards compatible and it would > improve interop. I would like to hear some opinions on this. If this would allow the MSVC-compilied Python to better access dlls compiled with gcc (cygwin) on Windows, definitely -- in 3.3. If the new feature is (currently) only useful on Windows, doc should say so. -- Terry Jan Reedy From riscutiavlad at gmail.com Sat Jun 25 21:15:12 2011 From: riscutiavlad at gmail.com (Vlad Riscutia) Date: Sat, 25 Jun 2011 12:15:12 -0700 Subject: [Python-Dev] ctypes: Configurable bitfield allocation strategy In-Reply-To: References: Message-ID: This is the cause of both bug reports and yes, it should improve interop with GCC-compiled code on Windows. My point is that this is not a platform thing, it's more of a compiler thing. Currently issues appear on Windows for interop between MSVC Python and other GCC code but since bitfield allocation is up to the compiler, it could appear on any other platform due to different compilers being used. On Sat, Jun 25, 2011 at 12:09 PM, Terry Reedy wrote: > On 6/25/2011 12:33 PM, Vlad Riscutia wrote: > >> I recently started looking at some ctypes issues. I dug a bit into >> http://bugs.python.org/**issue6069 and >> then I found >> http://bugs.python.org/**issue11920 . >> They both boil down to the fact that >> bitfield allocation is up to the compiler, which is different in GCC and >> MSVC. Currently we have hard-coded allocation strategy based on paltform >> in cfields.c: >> >> if (bitsize /* this is a bitfield request */ >>> >> >> && *pfield_size /* we have a bitfield open */ >>> #ifdef MS_WIN32 >>> /* MSVC, GCC with -mms-bitfields */ >>> && dict->size * 8 == *pfield_size >>> #else >>> /* GCC */ >>> && dict->size * 8<= *pfield_size >>> #endif >>> && (*pbitofs + bitsize)<= *pfield_size) { >>> /* continue bit field */ >>> fieldtype = CONT_BITFIELD; >>> #ifndef MS_WIN32 >>> } else if (bitsize /* this is a bitfield request */ >>> && *pfield_size /* we have a bitfield open */ >>> && dict->size * 8>= *pfield_size >>> && (*pbitofs + bitsize)<= dict->size * 8) { >>> /* expand bit field */ >>> fieldtype = EXPAND_BITFIELD; >>> #endif >>> >> >> So when creating a bitfield structure, it's size can be different on >> Linux vs Windows. >> >> class MyStructure(ctypes.**BigEndianStructure): >> _pack_ = 1 # aligned to 8 bits, not ctypes default of 32 >> _fields_ = [ >> ("Data0", ctypes.c_uint32, 32), >> ("Data1", ctypes.c_uint8, 3), >> ("Data2", ctypes.c_uint16, 12), >> ] >> >> sizeof for above structure is 6 on GCC build and 7 on MSVC build. This >> leads to some confusion and issues, because we can't always interop >> correctly with code compiled with different compiler than the one Python >> is compiled with on the platform. >> > > Just curious, are you saying that this is the 'cause' of the two bug > reports, or 'just' something you discovered while investigating them? > > > > Short term solution is to add a warning in the documentation about this. > > For 2.7/3.2, yes. > > > > Longer term though, I think it > >> would be better to add a property on the Structure class for >> configurable allocation strategy, for example Native (default), GCC, >> MSVC and when allocating the bitfield, use given strategy. Native would >> behave similar to what happens now, but we would also allow GCC-style >> allocation on Windows for example and the ability to extend this if we >> ever run into similar issues with other compilers. This shouldn't be too >> difficult to implement, will be backwards compatible and it would >> improve interop. I would like to hear some opinions on this. >> > > If this would allow the MSVC-compilied Python to better access dlls > compiled with gcc (cygwin) on Windows, definitely -- in 3.3. > If the new feature is (currently) only useful on Windows, doc should say > so. > > -- > Terry Jan Reedy > > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > riscutiavlad%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun Jun 26 01:33:13 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 26 Jun 2011 11:33:13 +1200 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: <20110625153157.4B1063A4093@sparrow.telecommunity.com> References: <20110624145516.84DA32506EA@webabinitio.net> <20110625153157.4B1063A4093@sparrow.telecommunity.com> Message-ID: <4E067039.3060101@canterbury.ac.nz> P.J. Eby wrote: > At 10:46 AM 6/25/2011 +1000, Nick Coghlan wrote: > >> Indeed, PEP 380 is *really* hard to do properly without language >> support. > > No, it isn't. You add a decorator, a 'from_' class, and a 'return_' > function, and there you go. (See my previous code sketches here in > early PEP 380 discussions.) Will it handle *all* of the generator protocol correctly, including send(), exception handling, and generator closing? Also, how efficient would it be? A major benefit of a built-in implementation is that it can be almost as fast as using the sub-generator directly. -- Greg From greg.ewing at canterbury.ac.nz Sun Jun 26 01:37:05 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 26 Jun 2011 11:37:05 +1200 Subject: [Python-Dev] ctypes: Configurable bitfield allocation strategy In-Reply-To: References: Message-ID: <4E067121.9070802@canterbury.ac.nz> Vlad Riscutia wrote: > Longer term though, I think it > would be better to add a property on the Structure class for > configurable allocation strategy, for example Native (default), GCC, > MSVC It could also be good to have a mode which lets you specify *exactly* how the bits are laid out, independent of any particular compiler. -- Greg From ncoghlan at gmail.com Sun Jun 26 05:16:35 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Jun 2011 13:16:35 +1000 Subject: [Python-Dev] PEP 380 acceptance (was Re: EuroPython Language Summit report) Message-ID: On Sun, Jun 26, 2011 at 2:44 AM, Guido van Rossum wrote: > Let me cut this short. PEP 380 is pretty much approved. I know there > are a few details worth quibbling over, but they are not going to > jeopardize acceptance of the PEP. We are waiting for an implementation > in Python 3.3. In fact, I wouldn't mind at this point if someone took > their best effort at an implementation and checked it into the 3.3 > branch, and we can do the quibbling over the details while we have a > working implementation to experiment with. Based on this message, I've bumped PEP 380 to Accepted, and I'm now working on committing Renaud Blanch's forward port of Greg's original patch (see http://bugs.python.org/issue11682). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sun Jun 26 06:29:57 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Jun 2011 14:29:57 +1000 Subject: [Python-Dev] PEP 380 acceptance (was Re: EuroPython Language Summit report) In-Reply-To: References: Message-ID: On Sun, Jun 26, 2011 at 1:16 PM, Nick Coghlan wrote: > On Sun, Jun 26, 2011 at 2:44 AM, Guido van Rossum wrote: >> Let me cut this short. PEP 380 is pretty much approved. I know there >> are a few details worth quibbling over, but they are not going to >> jeopardize acceptance of the PEP. We are waiting for an implementation >> in Python 3.3. In fact, I wouldn't mind at this point if someone took >> their best effort at an implementation and checked it into the 3.3 >> branch, and we can do the quibbling over the details while we have a >> working implementation to experiment with. > > Based on this message, I've bumped PEP 380 to Accepted, and I'm now > working on committing Renaud Blanch's forward port of Greg's original > patch (see http://bugs.python.org/issue11682). I hit a snag with this. The real tests of the PEP 380 functionality aren't currently part of the patch - they're a big set of "golden output" tests in the zipfile hosted on Greg's site. Those need to be refactored into proper unittest or doctest based additions to the test suite and incorporated into the patch before I could commit this with a clear conscience. That's not going to be as quick as I first thought. Renaud's patch mostly applies cleanly at the moment - the only change is that the "#endif" for the Py_LIMITED_API check needs to be moved in pyerrors.h so it also covers the new StopIteration struct definition. Regards, Nick. [1] http://www.cosc.canterbury.ac.nz/greg.ewing/python/yield-from/yield_from.html -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From barry at python.org Sun Jun 26 13:30:31 2011 From: barry at python.org (Barry Warsaw) Date: Sun, 26 Jun 2011 12:30:31 +0100 Subject: [Python-Dev] [Python-checkins] peps: Record Guido's acceptance of PEP 380 In-Reply-To: References: Message-ID: <20110626123031.5091420a@snowdog> On Jun 26, 2011, at 05:02 AM, nick.coghlan wrote: >http://hg.python.org/peps/rev/9f7a0b4e38a7 >changeset: 3889:9f7a0b4e38a7 >user: Nick Coghlan >date: Sun Jun 26 13:02:17 2011 +1000 >summary: > Record Guido's acceptance of PEP 380 > >files: > pep-0380.txt | 10 ++++++++-- > 1 files changed, 8 insertions(+), 2 deletions(-) > > >diff --git a/pep-0380.txt b/pep-0380.txt >--- a/pep-0380.txt >+++ b/pep-0380.txt >@@ -3,11 +3,11 @@ > Version: $Revision$ > Last-Modified: $Date$ > Author: Gregory Ewing >-Status: Draft >+Status: Accepted > Type: Standards Track > Content-Type: text/x-rst > Created: 13-Feb-2009 >-Python-Version: 3.x >+Python-Version: 3.3 > Post-History: Please add a Resolution header as per PEP 1. Thanks! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Sun Jun 26 15:55:02 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Jun 2011 23:55:02 +1000 Subject: [Python-Dev] [Python-checkins] cpython (3.2): #11669: rephrase footnote in the Compound Statements page. In-Reply-To: References: Message-ID: On Sun, Jun 26, 2011 at 6:38 PM, ezio.melotti wrote: > -.. [#] The exception is propagated to the invocation stack only if there is no > - ? :keyword:`finally` clause that negates the exception. > +.. [#] The exception is propagated to the invocation stack unless > + ? there is a :keyword:`finally` clause which happens to raise another > + ? exception. That new exception causes the old one to be lost. I believe the footnote was talking about this case: >>> def f(): ... try: ... raise Exception() ... finally: ... return "What exception?" ... >>> f() 'What exception?' The new wording doesn't accurately reflect that. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From sandro.tosi at gmail.com Sun Jun 26 16:02:52 2011 From: sandro.tosi at gmail.com (Sandro Tosi) Date: Sun, 26 Jun 2011 16:02:52 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.2): #11669: rephrase footnote in the Compound Statements page. In-Reply-To: References: Message-ID: Hi Nick, given I'm "guilty" for this patch, I'd reply :) On Sun, Jun 26, 2011 at 15:55, Nick Coghlan wrote: > On Sun, Jun 26, 2011 at 6:38 PM, ezio.melotti > wrote: >> -.. [#] The exception is propagated to the invocation stack only if there is no >> - ? :keyword:`finally` clause that negates the exception. >> +.. [#] The exception is propagated to the invocation stack unless >> + ? there is a :keyword:`finally` clause which happens to raise another >> + ? exception. That new exception causes the old one to be lost. > > I believe the footnote was talking about this case: > >>>> def f(): > ... ? try: > ... ? ? raise Exception() > ... ? finally: > ... ? ? return "What exception?" > ... >>>> f() > 'What exception?' > > The new wording doesn't accurately reflect that. I gave my interpretation of the footnote at: http://bugs.python.org/issue11669#msg139092 . Does this clarify it? Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From senthil at uthcode.com Sun Jun 26 20:52:42 2011 From: senthil at uthcode.com (Senthil Kumaran) Date: Sun, 26 Jun 2011 11:52:42 -0700 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation Message-ID: <20110626185242.GB2710@mathmagic> Hello! http://bugs.python.org/issue10403 is a documentation bug which talks about using the term 'attribute' instead of the term 'member' when it denotes the class attributes. Agreed. But the discussion goes on to mention that, "Members and methods" should just be "attributes". I find this bit odd. If the two terms are used together, then replacing it with attributes is fine. But the term 'methods' cannot be replaced with 'attributes' as it changes the meaning. Take this case, :class:`BZ2File` provides all of the methods specified by the :class:`io.BufferedIOBase`, except for :meth:`detach` and :meth:`truncate`. Iteration and the :keyword:`with` statement are supported. is correct, whereas replacing "methods with attributes" would make it as :class:`BZ2File` provides all of the attributes specified by the :class:`io.BufferedIOBase`, except for :meth:`detach` and :meth:`truncate`. Iteration and the :keyword:`with` statement are supported. It does not seem correct. My stance is, "It is attributes instead of term members and the term method when it denotes methods can be left as such." Can I still hold on to that and modify the patch which Adam has submitted or the 'attribute' substitution everywhere makes sense? Thanks, Senthil From tjreedy at udel.edu Sun Jun 26 23:52:36 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 26 Jun 2011 17:52:36 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <20110626185242.GB2710@mathmagic> References: <20110626185242.GB2710@mathmagic> Message-ID: On 6/26/2011 2:52 PM, Senthil Kumaran wrote: > http://bugs.python.org/issue10403 is a documentation bug which talks > about using the term 'attribute' instead of the term 'member' when it > denotes the class attributes. Agreed. > > But the discussion goes on to mention that, > > "Members and methods" should just be "attributes". The terms 'member' ('data attribute' in modern terms) and 'method' go back to original Python when builtin types (or instances thereof) had members, methods, neither, or possibly both (but I do not remember anything with both). I believe there were separate builtin functions for retrieving them. "Member' is obsolete; 'method' definitely is not. > I find this bit odd. If the two terms are used together, then > replacing it with attributes is fine. Agreed. > But the term 'methods' cannot be > replaced with 'attributes' as it changes the meaning. Also agreed. > Take this case, > > :class:`BZ2File` provides all of the methods specified by the > :class:`io.BufferedIOBase`, except for :meth:`detach` and :meth:`truncate`. > Iteration and the :keyword:`with` statement are supported. > > is correct, whereas replacing "methods with attributes" would make it as > > :class:`BZ2File` provides all of the attributes specified by the > :class:`io.BufferedIOBase`, except for :meth:`detach` and :meth:`truncate`. > Iteration and the :keyword:`with` statement are supported. > > It does not seem correct. It may not even be correct. I would leave it as is unless there are inherited data attributes so that the correction makes more sense than the original. A blind change of 'method' to 'attribute' is wrong. > My stance is, "It is attributes instead of term members and the term method > when it denotes methods can be left as such." Can I still hold on to that and > modify the patch which Adam has submitted Yes. > or the 'attribute' substitution everywhere makes sense? No. My strong history-based opinions ;-). -- Terry Jan Reedy From riscutiavlad at gmail.com Mon Jun 27 02:48:23 2011 From: riscutiavlad at gmail.com (Vlad Riscutia) Date: Sun, 26 Jun 2011 17:48:23 -0700 Subject: [Python-Dev] ctypes: Configurable bitfield allocation strategy In-Reply-To: <4E067121.9070802@canterbury.ac.nz> References: <4E067121.9070802@canterbury.ac.nz> Message-ID: Well it's not really layout, because alignment is handled by pack option. It is how the field gets allocated. At this point I believe it will be more complex to come up with custom allocation option, precisely because it's up to each compiler to allocate the structure. Such flexibility will add a lot of complexity and it might not payoff. This is debatable, but I would go with a 3 option strategy at this point - native, GCC, MSVC and if we actually find interop issues with other compilers we can look into custom allocation. I will try to refactor the C code to more easily accommodate a mode option (while leaving behavior the same for now), then we can decide how the library interface should look like. Thank you, Vlad On Sat, Jun 25, 2011 at 4:37 PM, Greg Ewing wrote: > Vlad Riscutia wrote: > >> Longer term though, I think it would be better to add a property on the >> Structure class for configurable allocation strategy, for example Native >> (default), GCC, MSVC >> > > It could also be good to have a mode which lets you specify > *exactly* how the bits are laid out, independent of any > particular compiler. > > -- > Greg > > ______________________________**_________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/**mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** > riscutiavlad%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Jun 27 03:27:36 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 Jun 2011 11:27:36 +1000 Subject: [Python-Dev] [Python-checkins] cpython (3.2): #11669: rephrase footnote in the Compound Statements page. In-Reply-To: References: Message-ID: On Mon, Jun 27, 2011 at 12:02 AM, Sandro Tosi wrote: > I gave my interpretation of the footnote at: > http://bugs.python.org/issue11669#msg139092 . Does this clarify it? No, because while there *are* ways a finally clause can kill an exception completely, reraising another exception is not really one of them (as we set __context__ appropriately in that case, even if it means forcing instantiation of an as yet unrealised exception). >From PEP 3134: "This PEP handles exceptions that occur during 'except' blocks and 'finally' blocks in the same way. Reading the traceback makes it clear where the exceptions occurred, so additional mechanisms for distinguishing the two cases would only add unnecessary complexity." And from the interactive prompt: >>> try: ... raise TypeError ... except TypeError: ... raise ValueError ... Traceback (most recent call last): File "", line 2, in TypeError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 4, in ValueError >>> try: ... raise TypeError ... finally: ... raise ValueError ... Traceback (most recent call last): File "", line 2, in TypeError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "", line 4, in ValueError Note that PEP 3134 exception chaining means the original exception is not lost in either case - it is available via the __context__ attribute of the subsequent exception. However, *other* control flow statements in a finally clause *can* mean that an exception is lost completely, just as if it had been caught and not reraised: - the "return" example from my first message ('yield' will allow the exception to propagate when the generator is resumed) - 'break' in a loop ('continue' is not permitted inside a 'finally' clause) The old wording was at least vaguely right ("there are ways that a finally clause can kill an exception, but we aren't going to tell you what they are"). The new wording is precisely wrong, as it excludes the return and break cases (in both 2.x and 3.x) and, in 3.x, fails to note that the old exception isn't lost completely due to exception chaining (even if it is indeed hidden from any exception handlers in the current call stack). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Mon Jun 27 03:32:32 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 Jun 2011 11:32:32 +1000 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> Message-ID: On Mon, Jun 27, 2011 at 7:52 AM, Terry Reedy wrote: >> or the 'attribute' substitution everywhere makes sense? > > No. > > My strong history-based opinions ;-). +1 to what Terry said. "Members" is a historical relic that is best replaced by "attributes" or "data attributes" if we want to explicitly exclude methods for some reason. "Methods" is a subset of attributes that explicitly excludes data attributes. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From pje at telecommunity.com Mon Jun 27 03:43:49 2011 From: pje at telecommunity.com (P.J. Eby) Date: Sun, 26 Jun 2011 21:43:49 -0400 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: <20110625163242.349EC2506ED@webabinitio.net> References: <20110624145516.84DA32506EA@webabinitio.net> <20110625153157.4B1063A4093@sparrow.telecommunity.com> <20110625163242.349EC2506ED@webabinitio.net> Message-ID: <20110627014358.375273A403A@sparrow.telecommunity.com> At 12:32 PM 6/25/2011 -0400, R. David Murray wrote: >So your proposed code would allow me, when writing a generator in >my code, do something that would allow me to yield up all the >values from an arbitrary generator I'm calling, over which I have >no control (ie: I can't modify its code)? With a decorator on your own function, yes. See: http://mail.python.org/pipermail/python-dev/2010-July/102320.html for details. Mostly, though, that proposal was a suggestion for how the "optimized" implementation would work - i.e., a suggestion that PEP 380 be implemented that way under the hood, by implicitly turning 'yield from' into 'yield From()' and wrapping the generator itself with another From() instance. (IOW, that was a proposal for how to avoid the extra overhead of recursive yielding in a series of nested yield-from's.) From solipsis at pitrou.net Mon Jun 27 10:24:28 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 27 Jun 2011 10:24:28 +0200 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation References: <20110626185242.GB2710@mathmagic> Message-ID: <20110627102428.6fc1955b@msiwind> Le Mon, 27 Jun 2011 11:32:32 +1000, Nick Coghlan a ?crit : > On Mon, Jun 27, 2011 at 7:52 AM, Terry Reedy wrote: > >> or the 'attribute' substitution everywhere makes sense? > > > > No. > > > > My strong history-based opinions ;-). > > +1 to what Terry said. > > "Members" is a historical relic that is best replaced by "attributes" > or "data attributes" if we want to explicitly exclude methods for some > reason. "Methods" is a subset of attributes that explicitly excludes > data attributes. While I know it is technically right, I find it a bit strange to refer to methods as "attributes". We're describing an API, not the inner working of the object model. Also, people just discovering Python will probably be a bit surprised if we start refer to methods as "attributes". FWIW, I tend to understand "members" as "methods + attributes", which makes it a nice term to use for that purpose. Regards Antoine. From p.f.moore at gmail.com Mon Jun 27 10:47:05 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 27 Jun 2011 09:47:05 +0100 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <20110627102428.6fc1955b@msiwind> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> Message-ID: On 27 June 2011 09:24, Antoine Pitrou wrote: > While I know it is technically right, I find it a bit strange to refer to > methods as "attributes". We're describing an API, not the inner working of > the object model. Also, people just discovering Python will probably be a > bit surprised if we start refer to methods as "attributes". +1 > FWIW, I tend to understand "members" as "methods + attributes", which makes > it a nice term to use for that purpose. +1 Paul. From fuzzyman at voidspace.org.uk Mon Jun 27 11:42:54 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 27 Jun 2011 10:42:54 +0100 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <20110627102428.6fc1955b@msiwind> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> Message-ID: <4E08509E.1030401@voidspace.org.uk> On 27/06/2011 09:24, Antoine Pitrou wrote: > Le Mon, 27 Jun 2011 11:32:32 +1000, > Nick Coghlan a ?crit : > >> On Mon, Jun 27, 2011 at 7:52 AM, Terry Reedy wrote: >>>> or the 'attribute' substitution everywhere makes sense? >>> No. >>> >>> My strong history-based opinions ;-). >> +1 to what Terry said. >> >> "Members" is a historical relic that is best replaced by "attributes" >> or "data attributes" if we want to explicitly exclude methods for some >> reason. "Methods" is a subset of attributes that explicitly excludes >> data attributes. > While I know it is technically right, I find it a bit strange to refer to > methods as "attributes". We're describing an API, not the inner working of > the object model. Also, people just discovering Python will probably be a > bit surprised if we start refer to methods as "attributes". > > FWIW, I tend to understand "members" as "methods + attributes", which makes > it a nice term to use for that purpose. That is my understanding / use of the terms as well. Michael > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From phd at phdru.name Mon Jun 27 11:45:26 2011 From: phd at phdru.name (Oleg Broytman) Date: Mon, 27 Jun 2011 13:45:26 +0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <20110627102428.6fc1955b@msiwind> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> Message-ID: <20110627094526.GA7950@iskra.aviel.ru> On Mon, Jun 27, 2011 at 10:24:28AM +0200, Antoine Pitrou wrote: > FWIW, I tend to understand "members" as "methods + attributes", which makes > it a nice term to use for that purpose. That's my feeling too. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From tom.whittock at gmail.com Mon Jun 27 12:48:35 2011 From: tom.whittock at gmail.com (Tom Whittock) Date: Mon, 27 Jun 2011 11:48:35 +0100 Subject: [Python-Dev] Patching builtin_id to allow for C proxy objects? Message-ID: Hi all. I'm writing a module to proxy C++ objects into Python for a large C++ application. There are hundreds of thousands of C++ objects, some of which are temporary while others are very long lived. Currently every time one of these objects is accessed from Python, a new "myproxy" instance is created. So if I were to access the same field of an object twice, I would receive two python objects proxying the same underlying C++ object. This messes up "id" and "is", and is causing me issues when, for example, I run into circular references when enoding json or otherwise attempt to determine whether two proxy objects refer to the same C++ object. I can't see how to cache the "myproxy" objects instead of returning new instances - due to the architecture of the C++ application, there's no weak reference support at all, and the number of objects is very large. My current plan would be for me to override the id builtin to return the underlying C++ object instance pointer instead of the PyObject instance pointer in the case of the "myproxy" object type, probably using a new type method slot tp_id or similar. The old behaviour would be unchanged for all other types, naturally. I'd also need to alter ceval.c to use builtin_id instead of the raw pointer for comparison when using PyCmp_IS and PyCmp_IS_NOT. I can see that there could very well be many other sites throughout the C source where the pointer was directly compared, and would cause interesting issues for me down the line. I'm just not sure what else to try. I'd like to know if I'm being laughably naive or not before I went about this plan, and whether it'd be worthwhile contributing the patch back, considering the number of potentially overridden-id-unaware areas throught the rest of the python source base. Thanks. Tom. From stefan_ml at behnel.de Mon Jun 27 13:27:30 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 27 Jun 2011 13:27:30 +0200 Subject: [Python-Dev] Patching builtin_id to allow for C proxy objects? In-Reply-To: References: Message-ID: Tom Whittock, 27.06.2011 12:48: > I'm writing a module to proxy C++ objects into Python for a large C++ > application. There are hundreds of thousands of C++ objects, some of > which are temporary while others are very long lived. > > Currently every time one of these objects is accessed from Python, a > new "myproxy" instance is created. So if I were to access the same > field of an object twice, I would receive two python objects proxying > the same underlying C++ object. This messes up "id" and "is" Note that "is" actually compares the addresses, not the id(). > and is > causing me issues when, for example, I run into circular references > when enoding json or otherwise attempt to determine whether two proxy > objects refer to the same C++ object. > > I can't see how to cache the "myproxy" objects instead of returning > new instances - due to the architecture of the C++ application, > there's no weak reference support at all, and the number of objects is > very large. > > My current plan would be for me to override the id builtin to return > the underlying C++ object instance pointer instead of the PyObject > instance pointer in the case of the "myproxy" object type Where would you get the proxy object from anyway? IMHO, there are two obvious way get what you want: map the C++ object address (integer!) to a proxy object using a dict, or use a backpointer from the C++ object to its proxy. The second is substantially faster, but may require changes to the C++ class struct. I don't see how changes to CPython's core can help you here. Stefan From greg.ewing at canterbury.ac.nz Mon Jun 27 14:02:33 2011 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 28 Jun 2011 00:02:33 +1200 Subject: [Python-Dev] Patching builtin_id to allow for C proxy objects? In-Reply-To: References: Message-ID: <4E087159.7000009@canterbury.ac.nz> Tom Whittock wrote: > Currently every time one of these objects is accessed from Python, a > new "myproxy" instance is created. So if I were to access the same > field of an object twice, I would receive two python objects proxying > the same underlying C++ object. Perhaps you could use a WeakValueDictionary to keep a mapping from a C++ object address to its Python proxy. Then as long as a proxy object is alive, accessing the same C++ object again will get you the same proxy object. When there are no longer any references to the proxy object from Python, it will go away. The next time you access that C++ object you'll get a new proxy, but that won't matter, because the original proxy is no longer around to compare it with. This depends on there being some way for the proxy object to ensure that the C++ object remains alive as long as it does. It also won't solve the problem of keeping the id of the proxy for longer than the proxy exists, but that's probably not something you should be relying on anyway. The id of *any* Python object is only valid while the object lives, and if it's still alive you have a real reference somewhere that you can use instead of the id for identity testing. -- Greg From tom.whittock at gmail.com Mon Jun 27 14:31:06 2011 From: tom.whittock at gmail.com (Tom Whittock) Date: Mon, 27 Jun 2011 13:31:06 +0100 Subject: [Python-Dev] Patching builtin_id to allow for C proxy objects? In-Reply-To: <4E087159.7000009@canterbury.ac.nz> References: <4E087159.7000009@canterbury.ac.nz> Message-ID: Hi Greg thanks for your quick reply. > Perhaps you could use a WeakValueDictionary to keep a mapping > from a C++ object address to its Python proxy. Thank you, I'll implement this and see whether it works out. I'll certainly be better off if it does. I was avoiding holding weak references due to perhaps unfounded concerns about increasing the lifetime and speed/memory impact of certain temporary objects which are created at very high frequency. I'll test it and see before diving into messing with id. But now I'm thinking about it again, I can see a plan for not needing to affect that pathway at all. Seems I fell into the trap of making things too complicated for myself. > It also won't solve the problem of keeping the id of the > proxy for longer than the proxy exists, but that's probably > not something you should be relying on anyway. The id of > *any* Python object is only valid while the object lives, > and if it's still alive you have a real reference somewhere > that you can use instead of the id for identity testing. Thanks, yes. I'm actually kind of concerned about the usage of id in the markers set which the json library uses for circular referencing checks for exactly this reason. It seems to assume that the objects lifetime lasts for the duration of the encoding operation. I have no idea if that's actually the case in my situation, where data members are property getters producing probably very short lived proxies generated from C++. I guess I'll find out :) Thanks again, Tom. From tom.whittock at gmail.com Mon Jun 27 16:05:37 2011 From: tom.whittock at gmail.com (Tom Whittock) Date: Mon, 27 Jun 2011 15:05:37 +0100 Subject: [Python-Dev] Patching builtin_id to allow for C proxy objects? In-Reply-To: References: <4E087159.7000009@canterbury.ac.nz> Message-ID: Hi again. Just to let you know that Greg's suggestion worked beautifully - I guess my id idea was just me trying to make life hard for myself. My concerns over the json modules usage of id seem unjustified, as circular references are detected now that the weak reference dictionary is in place. Thanks for your help, and sorry for bothering dev with something which was a regular python programming issue after all. Tom. On 27 June 2011 13:31, Tom Whittock wrote: > Hi Greg thanks for your quick reply. > >> Perhaps you could use a WeakValueDictionary to keep a mapping >> from a C++ object address to its Python proxy. > > Thank you, I'll implement this and see whether it works out. I'll > certainly be better off if it does. I was avoiding holding weak > references due to perhaps unfounded concerns about increasing the > lifetime and speed/memory impact of certain temporary objects which > are created at very high frequency. I'll test it and see before diving > into messing with id. But now I'm thinking about it again, I can see a > plan for not needing to affect that pathway at all. > > Seems I fell into the trap of making things too complicated for myself. > >> It also won't solve the problem of keeping the id of the >> proxy for longer than the proxy exists, but that's probably >> not something you should be relying on anyway. The id of >> *any* Python object is only valid while the object lives, >> and if it's still alive you have a real reference somewhere >> that you can use instead of the id for identity testing. > > Thanks, yes. I'm actually kind of concerned about the usage of id in > the markers set which the json library uses for circular referencing > checks for exactly this reason. It seems to assume that the objects > lifetime lasts for the duration of the encoding operation. I have no > idea if that's actually the case in my situation, where data members > are property getters producing probably very short lived proxies > generated from C++. I guess I'll find out :) > > Thanks again, > Tom. > From rdmurray at bitdance.com Mon Jun 27 16:08:21 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 27 Jun 2011 10:08:21 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> Message-ID: <20110627140826.877032505A3@webabinitio.net> On Mon, 27 Jun 2011 09:47:05 +0100, Paul Moore wrote: > On 27 June 2011 09:24, Antoine Pitrou wrote: > > While I know it is technically right, I find it a bit strange to refer to > > methods as "attributes". We're describing an API, not the inner working of > > the object model. Also, people just discovering Python will probably be a > > bit surprised if we start refer to methods as "attributes". > > +1 > > > FWIW, I tend to understand "members" as "methods + attributes", which makes > > it a nice term to use for that purpose. > > +1 Wow, all these people who like 'members', and I can't think of ever using that term in a Python context. While I agree that using 'attribute' when only methods are being discussed would most likely be confusing, and that it can be tricky to clearly word things when both are being discussed, the existence in the language of getattr, setattr, and related methods argues against using the term 'members'. 'data attributes' can so easily become something else in Python...it seems to me that the only real difference between 'data attributes' and 'method attributes' in Python is that the latter can be called and the former can't. But even that is not an accurate distinction, since a 'data attribute' could, in fact, return a callable. I guess what I'm saying is that I am more comfortable calling them all attributes than calling them all members. The term 'members' isn't used anywhere in the language itself, as far as I can recall, whereas getattr and setattr are evidence that the language considers them all attributes. I think we do the documentation readers a disservice by obscuring that fact by using other terminology. -- R. David Murray http://www.bitdance.com From rob.cliffe at btinternet.com Mon Jun 27 16:19:42 2011 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 27 Jun 2011 15:19:42 +0100 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <20110627140826.877032505A3@webabinitio.net> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> Message-ID: <4E08917E.3020509@btinternet.com> On 27/06/2011 15:08, R. David Murray wrote: > Wow, all these people who like 'members', and I can't think of ever > using that term in a Python context. > > While I agree that using 'attribute' when only methods are being discussed > would most likely be confusing, and that it can be tricky to clearly > word things when both are being discussed, the existence in the language > of getattr, setattr, and related methods argues against using the term > 'members'. > > 'data attributes' can so easily become something else in Python...it > seems to me that the only real difference between 'data attributes' and > 'method attributes' in Python is that the latter can be called and the > former can't. But even that is not an accurate distinction, since a > 'data attribute' could, in fact, return a callable. > > I guess what I'm saying is that I am more comfortable calling them > all attributes than calling them all members. The term 'members' > isn't used anywhere in the language itself, as far as I can recall, > whereas getattr and setattr are evidence that the language considers > them all attributes. I think we do the documentation readers a > disservice by obscuring that fact by using other terminology. > +1. 'function attributes' ? 'def attributes' ? Or just stick with 'method attributes' ? From fuzzyman at voidspace.org.uk Mon Jun 27 16:27:12 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 27 Jun 2011 15:27:12 +0100 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <20110627140826.877032505A3@webabinitio.net> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> Message-ID: <4E089340.7060004@voidspace.org.uk> On 27/06/2011 15:08, R. David Murray wrote: > On Mon, 27 Jun 2011 09:47:05 +0100, Paul Moore wrote: >> On 27 June 2011 09:24, Antoine Pitrou wrote: >>> While I know it is technically right, I find it a bit strange to refer to >>> methods as "attributes". We're describing an API, not the inner working of >>> the object model. Also, people just discovering Python will probably be a >>> bit surprised if we start refer to methods as "attributes". >> +1 >> >>> FWIW, I tend to understand "members" as "methods + attributes", which makes >>> it a nice term to use for that purpose. >> +1 > Wow, all these people who like 'members', and I can't think of ever > using that term in a Python context. > > While I agree that using 'attribute' when only methods are being discussed > would most likely be confusing, and that it can be tricky to clearly > word things when both are being discussed, the existence in the language > of getattr, setattr, and related methods argues against using the term > 'members'. > > 'data attributes' can so easily become something else in Python...it > seems to me that the only real difference between 'data attributes' and > 'method attributes' in Python is that the latter can be called and the > former can't. But even that is not an accurate distinction, since a > 'data attribute' could, in fact, return a callable. > > I guess what I'm saying is that I am more comfortable calling them > all attributes than calling them all members. The term 'members' > isn't used anywhere in the language itself, as far as I can recall, > whereas getattr and setattr are evidence that the language considers > them all attributes. I think we do the documentation readers a > disservice by obscuring that fact by using other terminology. > Well perhaps, but where does the language draw the distinction between attributes and "data attributes" as you all them (a term entirely new to me)? Only in the descriptor protocol and that term isn't used there (data-descriptors and non data-descriptors is terminology used in the documentation there). If you're saying that data attributes isn't clear either (I couldn't quite tell from your email) then how *do* we draw a distinction. We could talk about instance attributes, non-descriptor class attributes and descriptors, but that terminology requires a reasonably advanced understanding of the Python data model. I don't think that "all members, made up of attributes plus methods" is hard to understand. That's a great benefit. The fact that you can technically treat methods as attributes too is a minor detail. All the best, Michael Foord > -- > R. David Murray http://www.bitdance.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From ncoghlan at gmail.com Mon Jun 27 16:32:59 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jun 2011 00:32:59 +1000 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <20110627140826.877032505A3@webabinitio.net> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> Message-ID: On Tue, Jun 28, 2011 at 12:08 AM, R. David Murray wrote: > While I agree that using 'attribute' when only methods are being discussed > would most likely be confusing, and that it can be tricky to clearly > word things when both are being discussed, the existence in the language > of getattr, setattr, and related methods argues against using the term > 'members'. Yep, to me "attribute" just means "something that can be accessed using attribute notation". What it actually *is* is completely up for grabs at that point. > 'data attributes' can so easily become something else in Python...it > seems to me that the only real difference between 'data attributes' and > 'method attributes' in Python is that the latter can be called and the > former can't. ?But even that is not an accurate distinction, since a > 'data attribute' could, in fact, return a callable. > > I guess what I'm saying is that I am more comfortable calling them > all attributes than calling them all members. ?The term 'members' > isn't used anywhere in the language itself, as far as I can recall, > whereas getattr and setattr are evidence that the language considers > them all attributes. ?I think we do the documentation readers a > disservice by obscuring that fact by using other terminology. It's worse than that - the specific meaning of "members" in the context of Python's history specifically *excludes* methods. The superset is "attributes" - as noted, the names of the builtins and magic methods make that terminology quite explicit. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Mon Jun 27 16:36:20 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jun 2011 00:36:20 +1000 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E089340.7060004@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E089340.7060004@voidspace.org.uk> Message-ID: On Tue, Jun 28, 2011 at 12:27 AM, Michael Foord wrote: > Well perhaps, but where does the language draw the distinction between > attributes and "data attributes" as you all them (a term entirely new to > me)? Only in the descriptor protocol and that term isn't used there > (data-descriptors and non data-descriptors is terminology used in the > documentation there). > > If you're saying that data attributes isn't clear either (I couldn't quite > tell from your email) then how *do* we draw a distinction. We could talk > about instance attributes, non-descriptor class attributes and descriptors, > but that terminology requires a reasonably advanced understanding of the > Python data model. > > I don't think that "all members, made up of attributes plus methods" is hard > to understand. That's a great benefit. The fact that you can technically > treat methods as attributes too is a minor detail. It has almost no precedent in the Python context and what precedent it does have is wrong (since it excluded methods). And no, the fact that methods can be treated as attributes is not a minor detail. It is *fundamental* to Python's object model that *methods are not a special case of attribute access*. All attributes work the same way, it is just the way functions implement the descriptor protocol that makes instance methods behave the way they do. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From benjamin at python.org Mon Jun 27 16:38:10 2011 From: benjamin at python.org (Benjamin Peterson) Date: Mon, 27 Jun 2011 09:38:10 -0500 Subject: [Python-Dev] [Python-checkins] peps: Happy Tau Day folks! :) In-Reply-To: References: Message-ID: 2011/6/27 nick.coghlan : > http://hg.python.org/peps/rev/1e3d663c67ee > changeset: ? 3892:1e3d663c67ee > user: ? ? ? ?Nick Coghlan > date: ? ? ? ?Tue Jun 28 00:23:57 2011 +1000 > summary: > ?Happy Tau Day folks! :) > > files: > ?pep-0628.html | ?149 ++++++++++++++++++++++++++++++++++++++ You're not going to give us the source? -- Regards, Benjamin From ncoghlan at gmail.com Mon Jun 27 16:45:08 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jun 2011 00:45:08 +1000 Subject: [Python-Dev] [Python-checkins] peps: Happy Tau Day folks! :) In-Reply-To: References: Message-ID: On Tue, Jun 28, 2011 at 12:38 AM, Benjamin Peterson wrote: > You're not going to give us the source? Just a teensy error with hg add, tab completion and not checking the diff before committing. Fixed now, though :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From rndblnch at gmail.com Mon Jun 27 17:09:29 2011 From: rndblnch at gmail.com (renaud) Date: Mon, 27 Jun 2011 15:09:29 +0000 (UTC) Subject: [Python-Dev] PEP 380 acceptance (was Re: EuroPython Language Summit report) References: Message-ID: Nick Coghlan gmail.com> writes: > I hit a snag with this. The real tests of the PEP 380 functionality > aren't currently part of the patch - they're a big set of "golden > output" tests in the zipfile hosted on Greg's site. Those need to be > refactored into proper unittest or doctest based additions to the test > suite and incorporated into the patch before I could commit this with > a clear conscience. let me know if i can help. > Renaud's patch mostly applies cleanly at the moment - the only change > is that the "#endif" for the Py_LIMITED_API check needs to be moved in > pyerrors.h so it also covers the new StopIteration struct definition. if this helps, i've updated the patch to fix this. https://bitbucket.org/rndblnch/cpython-pep380/changeset/6014d1720625 renaud From stephen at xemacs.org Mon Jun 27 17:32:07 2011 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Tue, 28 Jun 2011 00:32:07 +0900 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E089340.7060004@voidspace.org.uk> Message-ID: <87vcvr1ea0.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > And no, the fact that methods can be treated as attributes is not a > minor detail. It is *fundamental* to Python's object model that > *methods are not a special case of attribute access*. That's ambiguous. I assume you mean "just a case of attribute access, and not special in any way"? From janssen at parc.com Mon Jun 27 17:51:48 2011 From: janssen at parc.com (Bill Janssen) Date: Mon, 27 Jun 2011 08:51:48 PDT Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E089340.7060004@voidspace.org.uk> Message-ID: <25745.1309189908@parc.com> Nick Coghlan wrote: > And no, the fact that methods can be treated as attributes is not a > minor detail. It is *fundamental* to Python's object model that > *methods are not a special case of attribute access*. All attributes > work the same way, it is just the way functions implement the > descriptor protocol that makes instance methods behave the way they > do. Well put, Nick. This paragraph is a good thing to read a couple of times. Bill From rdmurray at bitdance.com Mon Jun 27 18:56:32 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 27 Jun 2011 12:56:32 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E089340.7060004@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E089340.7060004@voidspace.org.uk> Message-ID: <20110627165633.0A8652505A3@webabinitio.net> On Mon, 27 Jun 2011 15:27:12 +0100, Michael Foord wrote: > On 27/06/2011 15:08, R. David Murray wrote: > > 'data attributes' can so easily become something else in Python...it > > seems to me that the only real difference between 'data attributes' and > > 'method attributes' in Python is that the latter can be called and the > > former can't. But even that is not an accurate distinction, since a > > 'data attribute' could, in fact, return a callable. > > > > I guess what I'm saying is that I am more comfortable calling them > > all attributes than calling them all members. The term 'members' > > isn't used anywhere in the language itself, as far as I can recall, > > whereas getattr and setattr are evidence that the language considers > > them all attributes. I think we do the documentation readers a > > disservice by obscuring that fact by using other terminology. > > Well perhaps, but where does the language draw the distinction between > attributes and "data attributes" as you all them (a term entirely new to It doesn't, that's the point. You'll note I put "data attributes" in quotes :) > me)? Only in the descriptor protocol and that term isn't used there > (data-descriptors and non data-descriptors is terminology used in the > documentation there). > > If you're saying that data attributes isn't clear either (I couldn't > quite tell from your email) then how *do* we draw a distinction. We > could talk about instance attributes, non-descriptor class attributes > and descriptors, but that terminology requires a reasonably advanced > understanding of the Python data model. That's why I said it could be difficult to find good wording when discussing both methods and "other things". Most people have a pretty clear idea of what methods are, but the non-method stuff in Python does not have any simple description that is also accurate. Maybe 'non-method attribute' is as close as we can get? > I don't think that "all members, made up of attributes plus methods" is > hard to understand. That's a great benefit. The fact that you can > technically treat methods as attributes too is a minor detail. Well, I would find that very hard to understand, since methods are attributes, and as Nick said that is *fundamental* to the language, not a minor detail. -- R. David Murray http://www.bitdance.com From janssen at parc.com Mon Jun 27 19:01:13 2011 From: janssen at parc.com (Bill Janssen) Date: Mon, 27 Jun 2011 10:01:13 PDT Subject: [Python-Dev] Snow Leopard buildbot failing again... Message-ID: <26766.1309194073@parc.com> I see that parc-snowleopard-1 went down again. I've done a software update, rebooted, and installed the latest buildslave, 0.8.4. I can ping dinsdale.python.org successfully from the machine. However, when I start the buildslave, I get this: $ buildslave start ~/buildarea/ /Library/Python/2.6/site-packages/Twisted-8.2.0-py2.6-macosx-10.6-universal.egg/twisted/internet/_sslverify.py:5: DeprecationWarning: the md5 module is deprecated; use hashlib instead import itertools, md5 Following twistd.log until startup finished.. /Library/Python/2.6/site-packages/buildbot_slave-0.8.4-py2.6.egg/buildslave/scripts/logwatcher.py:67: PotentialZombieWarning: spawnProcess called, but the SIGCHLD handler is not installed. This probably means you have not yet called reactor.run, or called reactor.run(installSignalHandler=0). You will probably never see this process finish, and it may become a zombie process. env=os.environ, /Library/Python/2.6/site-packages/Twisted-8.2.0-py2.6-macosx-10.6-universal.egg/twisted/persisted/sob.py:12: DeprecationWarning: the md5 module is deprecated; use hashlib instead import os, md5, sys /Library/Python/2.6/site-packages/Twisted-8.2.0-py2.6-macosx-10.6-universal.egg/twisted/python/filepath.py:12: DeprecationWarning: the sha module is deprecated; use the hashlib module instead import sha 2011-06-27 09:45:31-0700 [-] Log opened. 2011-06-27 09:45:31-0700 [-] twistd 8.2.0 (/usr/bin/python 2.6.1) starting up. 2011-06-27 09:45:31-0700 [-] reactor class: twisted.internet.selectreactor.SelectReactor. 2011-06-27 09:45:31-0700 [-] Starting BuildSlave -- version: 0.8.4 2011-06-27 09:45:31-0700 [-] recording hostname in twistd.hostname The buildslave took more than 10 seconds to start, so we were unable to confirm that it started correctly. Please 'tail twistd.log' and look for a line that says 'configuration update complete' to verify correct startup. $ Now, I thought of updating Twisted, but the Twisted downloads page says, ``Twisted is no longer making a .dmg for OS X because OS X versions 10.5 and greater ship with Twisted installed and the default Python path configuration makes it impossible for a user to upgrade Twisted without possibly breaking applications which depend on the system Twisted installation.'' Anyone know just what those applications are? All this machine is doing is running the Python buildbot. Bill From fdrake at acm.org Mon Jun 27 19:02:22 2011 From: fdrake at acm.org (Fred Drake) Date: Mon, 27 Jun 2011 13:02:22 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <25745.1309189908@parc.com> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E089340.7060004@voidspace.org.uk> <25745.1309189908@parc.com> Message-ID: Egads. Back when I wrote "Members and methods" should just be "attributes". I used quotes to specifically indicate that this applied to the phrase "members and methods", not their separate use. I guess I wasn't obvious enough. The general & Python-historical uses of "members" is unfortunate. My current position on this is that we should avoid the use of "members", because either use will confuse a large set of readers. As Nick points out, these are all attributes, regardless of their implementation or type of the value. "Methods" is a convenient and widely understood term to refer to a general class of attributes, when that actually matches the meaning. For non-method or not-necessarily-a-method attributes, I'm inclined to just stick with calling them attributes at this point. Even more important, we need to decide what to call them, and add appropriate words to the glossary. And then make the documentation match that. -Fred -- Fred L. Drake, Jr.? ? "Give me the luxuries of life and I will willingly do without the necessities." ?? --Frank Lloyd Wright From fuzzyman at voidspace.org.uk Mon Jun 27 19:10:52 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 27 Jun 2011 18:10:52 +0100 Subject: [Python-Dev] Snow Leopard buildbot failing again... In-Reply-To: <26766.1309194073@parc.com> References: <26766.1309194073@parc.com> Message-ID: <4E08B99C.5040807@voidspace.org.uk> On 27/06/2011 18:01, Bill Janssen wrote: > I see that parc-snowleopard-1 went down again. I've done a software > update, rebooted, and installed the latest buildslave, 0.8.4. I can > ping dinsdale.python.org successfully from the machine. However, when I > start the buildslave, I get this: > > $ buildslave start ~/buildarea/ > /Library/Python/2.6/site-packages/Twisted-8.2.0-py2.6-macosx-10.6-universal.egg/twisted/internet/_sslverify.py:5: DeprecationWarning: the md5 module is deprecated; use hashlib instead > import itertools, md5 > Following twistd.log until startup finished.. > /Library/Python/2.6/site-packages/buildbot_slave-0.8.4-py2.6.egg/buildslave/scripts/logwatcher.py:67: PotentialZombieWarning: spawnProcess called, but the SIGCHLD handler is not installed. This probably means you have not yet called reactor.run, or called reactor.run(installSignalHandler=0). You will probably never see this process finish, and it may become a zombie process. > env=os.environ, > /Library/Python/2.6/site-packages/Twisted-8.2.0-py2.6-macosx-10.6-universal.egg/twisted/persisted/sob.py:12: DeprecationWarning: the md5 module is deprecated; use hashlib instead > import os, md5, sys > /Library/Python/2.6/site-packages/Twisted-8.2.0-py2.6-macosx-10.6-universal.egg/twisted/python/filepath.py:12: DeprecationWarning: the sha module is deprecated; use the hashlib module instead > import sha > 2011-06-27 09:45:31-0700 [-] Log opened. > 2011-06-27 09:45:31-0700 [-] twistd 8.2.0 (/usr/bin/python 2.6.1) starting up. > 2011-06-27 09:45:31-0700 [-] reactor class: twisted.internet.selectreactor.SelectReactor. > 2011-06-27 09:45:31-0700 [-] Starting BuildSlave -- version: 0.8.4 > 2011-06-27 09:45:31-0700 [-] recording hostname in twistd.hostname > > The buildslave took more than 10 seconds to start, so we were unable to > confirm that it started correctly. Please 'tail twistd.log' and look for a > line that says 'configuration update complete' to verify correct startup. > > $ > > Now, I thought of updating Twisted, but the Twisted downloads page says, > > ``Twisted is no longer making a .dmg for OS X because OS X versions 10.5 > and greater ship with Twisted installed and the default Python path > configuration makes it impossible for a user to upgrade Twisted without > possibly breaking applications which depend on the system Twisted > installation.'' > > Anyone know just what those applications are? All this machine is doing > is running the Python buildbot. Well the calendar server (at least) uses the system twisted. It is best to leave the system python alone and install / use a python.org python install. Are the buildbot scripts hardcoded to use the system python on Macs? If so you will have to stick with the system version of twisted, which seems to be problematic. All the best, Michael > Bill > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From solipsis at pitrou.net Mon Jun 27 19:27:26 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 27 Jun 2011 19:27:26 +0200 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E089340.7060004@voidspace.org.uk> Message-ID: <20110627192726.51455b5d@msiwind> Le Tue, 28 Jun 2011 00:36:20 +1000, Nick Coghlan a ?crit : > And no, the fact that methods can be treated as attributes is not a > minor detail. It is *fundamental* to Python's object model that > *methods are not a special case of attribute access*. Uh, and so what? Again, the stdlib docs are describing APIs from a high-level viewpoint, not the Python object model. Just because a method is treated like other attributes, or a function is treated like other callable objects, doesn't mean we should replace "method" with "attribute", or "function" with "callable object", or "class" with "instance of the type type". I'm -1 on such a change anyway. Regards Antoine. From exarkun at twistedmatrix.com Mon Jun 27 19:36:41 2011 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Mon, 27 Jun 2011 17:36:41 -0000 Subject: [Python-Dev] Snow Leopard buildbot failing again... In-Reply-To: <26766.1309194073@parc.com> References: <26766.1309194073@parc.com> Message-ID: <20110627173641.1894.764887772.divmod.xquotient.459@localhost.localdomain> On 05:01 pm, janssen at parc.com wrote: >I see that parc-snowleopard-1 went down again. I've done a software >update, rebooted, and installed the latest buildslave, 0.8.4. I can >ping dinsdale.python.org successfully from the machine. However, when >I >start the buildslave, I get this: > >[snip] > >The buildslave took more than 10 seconds to start, so we were unable to >confirm that it started correctly. Please 'tail twistd.log' and look >for a >line that says 'configuration update complete' to verify correct >startup. Did you look at twistd.log? Jean-Paul From tjreedy at udel.edu Mon Jun 27 20:33:38 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 27 Jun 2011 14:33:38 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <20110627102428.6fc1955b@msiwind> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> Message-ID: On 6/27/2011 4:24 AM, Antoine Pitrou wrote: > Le Mon, 27 Jun 2011 11:32:32 +1000, > Nick Coghlan a ?crit : > >> "Members" is a historical relic that is best replaced by "attributes" >> or "data attributes" if we want to explicitly exclude methods for some >> reason. "Methods" is a subset of attributes that explicitly excludes >> data attributes. > > While I know it is technically right, I find it a bit strange to refer to > methods as "attributes". We're describing an API, not the inner working of > the object model. Also, people just discovering Python will probably be a > bit surprised if we start refer to methods as "attributes". > > FWIW, I tend to understand "members" as "methods + attributes", which makes > it a nice term to use for that purpose. Let me repeat that that is historically wrong for Python, and illustrate why the term 'members' should not be used. From the 1.5 Language Reference, 3.2 Standard type hierarchy: "There are also some 'generic' special attributes, not listed with the individual objects: __methods__ is a list of the method names of a built-in object, if it has any; __members__ is a list of the data attribute names of a built-in object, if it has any." -- Terry Jan Reedy From janssen at parc.com Mon Jun 27 21:00:09 2011 From: janssen at parc.com (Bill Janssen) Date: Mon, 27 Jun 2011 12:00:09 PDT Subject: [Python-Dev] Snow Leopard buildbot failing again... In-Reply-To: <20110627173641.1894.764887772.divmod.xquotient.459@localhost.localdomain> References: <26766.1309194073@parc.com> <20110627173641.1894.764887772.divmod.xquotient.459@localhost.localdomain> Message-ID: <29834.1309201209@parc.com> exarkun at twistedmatrix.com wrote: > On 05:01 pm, janssen at parc.com wrote: > >I see that parc-snowleopard-1 went down again. I've done a software > >update, rebooted, and installed the latest buildslave, 0.8.4. I can > > ping dinsdale.python.org successfully from the machine. However, > > when I > >start the buildslave, I get this: > > > >[snip] > > > > >The buildslave took more than 10 seconds to start, so we were unable to > > confirm that it started correctly. Please 'tail twistd.log' and look > > for a > > line that says 'configuration update complete' to verify correct > > startup. > > Did you look at twistd.log? > > Jean-Paul Um... not to look for "configuration update complete", I'm afraid. Isn't that a buildmaster-only thing? Here's what was there. 2011-06-27 09:45:31-0700 [-] Log opened. 2011-06-27 09:45:31-0700 [-] twistd 8.2.0 (/usr/bin/python 2.6.1) starting up. 2011-06-27 09:45:31-0700 [-] reactor class: twisted.internet.selectreactor.SelectReactor. 2011-06-27 09:45:31-0700 [-] Starting BuildSlave -- version: 0.8.4 2011-06-27 09:45:31-0700 [-] recording hostname in twistd.hostname 2011-06-27 09:46:01-0700 [-] Starting factory 2011-06-27 09:46:01-0700 [-] Connecting to dinsdale.python.org:9020 2011-06-27 09:46:02-0700 [Broker,client] message from master: attached 2011-06-27 09:46:02-0700 [Broker,client] I have a leftover directory '3.1.parc-snowleopard-1' that is not being used by the buildmaster: you can delete it now 2011-06-27 09:46:03-0700 [Broker,client] SlaveBuilder.remote_print(AMD64 Snow Leopard 2 custom): message from master: attached 2011-06-27 09:46:03-0700 [Broker,client] SlaveBuilder.remote_print(AMD64 Snow Leopard 2 2.7): message from master: attached 2011-06-27 09:46:03-0700 [Broker,client] SlaveBuilder.remote_print(AMD64 Snow Leopard 2 3.x): message from master: attached 2011-06-27 09:46:03-0700 [Broker,client] SlaveBuilder.remote_print(AMD64 Snow Leopard 2 3.2): message from master: attached 2011-06-27 09:46:05-0700 [Broker,client] Connected to dinsdale.python.org:9020; slave is ready 2011-06-27 09:46:05-0700 [Broker,client] sending application-level keepalives every 600 seconds There is that 30-second lag between "recording hostname in twistd.hostname" and "Starting factory", which is presumably what triggers the warning from buildslave. See and . Bill From janssen at parc.com Mon Jun 27 21:05:38 2011 From: janssen at parc.com (Bill Janssen) Date: Mon, 27 Jun 2011 12:05:38 PDT Subject: [Python-Dev] Snow Leopard buildbot failing again... In-Reply-To: <20110627173641.1894.764887772.divmod.xquotient.459@localhost.localdomain> References: <26766.1309194073@parc.com> <20110627173641.1894.764887772.divmod.xquotient.459@localhost.localdomain> Message-ID: <30385.1309201538@parc.com> I also find interesting, because it seems a good explanation of why the build slave might go into the zombie state of attempting to reconnect to the master. Bill From rdmurray at bitdance.com Mon Jun 27 21:22:16 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 27 Jun 2011 15:22:16 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <20110627192726.51455b5d@msiwind> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E089340.7060004@voidspace.org.uk> <20110627192726.51455b5d@msiwind> Message-ID: <20110627192217.66A3F2505A3@webabinitio.net> On Mon, 27 Jun 2011 19:27:26 +0200, Antoine Pitrou wrote: > Le Tue, 28 Jun 2011 00:36:20 +1000, > Nick Coghlan a ??crit : > > And no, the fact that methods can be treated as attributes is not a > > minor detail. It is *fundamental* to Python's object model that > > *methods are not a special case of attribute access*. > > Uh, and so what? > > Again, the stdlib docs are describing APIs from a high-level viewpoint, not > the Python object model. Just because a method is treated like other > attributes, or a function is treated like other callable objects, doesn't > mean we should replace "method" with "attribute", or "function" with > "callable object", or "class" with "instance of the type type". I don't think there is any disagreement with regards to replacing 'methods' with 'attributes'. This should not be done, in general, and I don't think anybody involved in this particular discussion thinks it should be. The question is what to call non-method attributes, or how we refer to a mix of method and non-method attributes. In some cases just saying 'attributes' may make sense for the latter, but in most cases it probably doesn't, because many people coming from other languages do not expect methods to be first class objects. So I'd like to be able to say, in those cases, "attributes (XXXs and methods)". The subject of this thread is the removal of the use of the term 'members' for XXX, for the historical reasons cited. IMO the goal should be to help the reader's mental model match what Python actually does. In that sense Python's object model *does* matter[*], in that we shouldn't make it harder for the reader to figure out what is "really" going on. In the cases you cited, you are arguing that the more specific term (method, function, class) should not be replaced in the docs by the more generic term (attribute, callable object, instance of type type). I don't think anyone would argue with you about that in general. (In specific there might in fact be some places in the docs where such a change would improve clarity!) So, the correct generic term for something that can be accessed via attribute notation is attribute. The more specific term for an attribute that is a method is method. We don't currently have a more specific collective term for attributes that aren't methods. *That* is the problem. -- R. David Murray http://www.bitdance.com [*] it also seems to me that the object model is, in many ways, *part* of the API, in the sense that the API is how you access the object model, and so understanding the object model is, if not essentially, then at least very helpful. From fuzzyman at voidspace.org.uk Mon Jun 27 21:30:12 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 27 Jun 2011 20:30:12 +0100 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <20110627192217.66A3F2505A3@webabinitio.net> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E089340.7060004@voidspace.org.uk> <20110627192726.51455b5d@msiwind> <20110627192217.66A3F2505A3@webabinitio.net> Message-ID: <4E08DA44.5010507@voidspace.org.uk> On 27/06/2011 20:22, R. David Murray wrote: > [snip...] > So, the correct generic term for something that can be accessed > via attribute notation is attribute. The more specific term for an > attribute that is a method is method. We don't currently have a more > specific collective term for attributes that aren't methods. *That* > is the problem. I think part of the problem is also that *some* people's usage of the term doesn't match the precise terminology. It sounds like if I say "an object's attributes" some of us will expect that to include the methods and some of us not. I'd say anecdotally that when people talk about object attributes collectively they are *not* including methods. When they talk about attribute lookup that rightly includes everything. We do have other terms: instance attributes (which do not include methods) and class attributes (which technically do - but is also used to indicate attributes set on the class rather than the instance but not including methods [1]). A precise term to describe "attributes that are not methods" would still be helpful. I guess the closest we have is "non-descriptors", but that requires a knowledge of the descriptor protocol for it to be useful. Making-things-worse-and-not-better'ly yours, Michael [1] I'm talking about *usage* of the term here... I guess usage is inconsistent anyway, sometimes people will mean to include methods and sometimes not. > -- > R. David Murray http://www.bitdance.com > > [*] it also seems to me that the object model is, in many ways, *part* > of the API, in the sense that the API is how you access the object model, > and so understanding the object model is, if not essentially, then at > least very helpful. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From rdmurray at bitdance.com Mon Jun 27 22:37:21 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 27 Jun 2011 16:37:21 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E08DA44.5010507@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E089340.7060004@voidspace.org.uk> <20110627192726.51455b5d@msiwind> <20110627192217.66A3F2505A3@webabinitio.net> <4E08DA44.5010507@voidspace.org.uk> Message-ID: <20110627203721.C6F7F2505A3@webabinitio.net> On Mon, 27 Jun 2011 20:30:12 +0100, Michael Foord wrote: > On 27/06/2011 20:22, R. David Murray wrote: > > [snip...] > > So, the correct generic term for something that can be accessed > > via attribute notation is attribute. The more specific term for an > > attribute that is a method is method. We don't currently have a more > > specific collective term for attributes that aren't methods. *That* > > is the problem. > > I think part of the problem is also that *some* people's usage of the > term doesn't match the precise terminology. > > It sounds like if I say "an object's attributes" some of us will expect > that to include the methods and some of us not. I'd say anecdotally that > when people talk about object attributes collectively they are *not* > including methods. When they talk about attribute lookup that rightly > includes everything. Yes, I think that it is common to use 'attributes' to mean 'non-method attributes', and often the context can make clear which usage is intended. What we should do in our own docs is a different question, but we do have other cases where "it is clear from context" is considered a valid argument for technically imprecise terminology. And I think that's reasonable. > A precise term to describe "attributes that are not methods" would still > be helpful. I guess the closest we have is "non-descriptors", but that > requires a knowledge of the descriptor protocol for it to be useful. I'm not really as comfortable as I think I should be with the descriptor stuff, but aren't properties descriptors in the sense you are using it here? Yet I don't think a property should be called a method, even though it is implemented via a function(s) that looks a lot like a method. This is why talking about non-method attributes collectively is so fraught.... -- R. David Murray http://www.bitdance.com From tjreedy at udel.edu Tue Jun 28 00:05:33 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 27 Jun 2011 18:05:33 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E08509E.1030401@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <4E08509E.1030401@voidspace.org.uk> Message-ID: On 6/27/2011 5:42 AM, Michael Foord wrote: > On 27/06/2011 09:24, Antoine Pitrou wrote: >> FWIW, I tend to understand "members" as "methods + attributes", which >> makes >> it a nice term to use for that purpose. > > That is my understanding / use of the terms as well. On 6/27/2011 5:45 AM, Oleg Broytman wrote: > That's my feeling too. Whereas the actual existing usage is that attributes = members + methods. 'Member' came from 'member of a data structure', as in complex numbers have real and imag members, whereas lists have methods but no (user visible) members. The fact that so many people get the Python usage of 'member' so wrong is a good reason to retire it. -- Terry Jan Reedy From tjreedy at udel.edu Tue Jun 28 00:18:29 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 27 Jun 2011 18:18:29 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> Message-ID: On 6/27/2011 2:33 PM, Terry Reedy wrote: > Let me repeat that that is historically wrong for Python, and illustrate > why the term 'members' should not be used. From the 1.5 Language > Reference, 3.2 Standard type hierarchy: "There are also some 'generic' > special attributes, not listed with the individual objects: __methods__ > is a list of the method names of a built-in object, if it has any; > __members__ is a list of the data attribute names of a built-in object, > if it has any." This sentence was left untouched until 2.2. What's new 2.2 has "In previous versions of Python, there was no consistent way to discover what attributes and methods were supported by an object. There were some informal conventions, such as defining __members__ and __methods__ attributes that were lists of names, but often the author of an extension type or a class wouldn't bother to define them." This is a section on descriptors, but the real replacement is, I believe, dir(). -- Terry Jan Reedy From fuzzyman at voidspace.org.uk Tue Jun 28 00:46:53 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 27 Jun 2011 23:46:53 +0100 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> Message-ID: <4E09085D.80202@voidspace.org.uk> On 27/06/2011 23:18, Terry Reedy wrote: > On 6/27/2011 2:33 PM, Terry Reedy wrote: > >> Let me repeat that that is historically wrong for Python, and illustrate >> why the term 'members' should not be used. From the 1.5 Language >> Reference, 3.2 Standard type hierarchy: "There are also some 'generic' >> special attributes, not listed with the individual objects: __methods__ >> is a list of the method names of a built-in object, if it has any; >> __members__ is a list of the data attribute names of a built-in object, >> if it has any." > > This sentence was left untouched until 2.2. What's new 2.2 has "In > previous versions of Python, there was no consistent way to discover > what attributes and methods were supported by an object. There were > some informal conventions, such as defining __members__ and > __methods__ attributes that were lists of names, but often the author > of an extension type or a class wouldn't bother to define them." So the Python 2.2 what's new talks about attributes and methods as different things.... Of course the context makes it clear, but this mirrors how I use the terms in discussion and how I see others generally using them. Great topic for bikeshedding. :-) Michael > This is a section on descriptors, but the real replacement is, I > believe, dir(). > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From guido at python.org Tue Jun 28 01:32:16 2011 From: guido at python.org (Guido van Rossum) Date: Mon, 27 Jun 2011 16:32:16 -0700 Subject: [Python-Dev] pdb and generators Message-ID: When debugging generators (especially when using them as coroutines) I often want 'next' to step over a yield statement/expression, instead of the current behavior, which is to step out of the frame, since pdb doesn't seem to see the difference between yield and return. Or I could live with a separate command to step over the yield. Has anyone looked into this yet? I found a question about this on StackOverflow.com but the answers were particularly unhelpful, so I'm guessing I'm not the first to want this but it hasn't been solved yet. It might be a good project for someone looking into hacking pdb. -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Tue Jun 28 02:10:36 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jun 2011 10:10:36 +1000 Subject: [Python-Dev] PEP 380 acceptance (was Re: EuroPython Language Summit report) In-Reply-To: References: Message-ID: On Tue, Jun 28, 2011 at 1:09 AM, renaud wrote: > Nick Coghlan gmail.com> writes: > >> I hit a snag with this. The real tests of the PEP 380 functionality >> aren't currently part of the patch - they're a big set of "golden >> output" tests in the zipfile hosted on Greg's site. Those need to be >> refactored into proper unittest or doctest based additions to the test >> suite and incorporated into the patch before I could commit this with >> a clear conscience. > > let me know if i can help. It would be good if you could take a look at Greg's original test suite, consider ways of bringing it into the main regression tests and then update the patch queue on bitbucket accordingly. My preference is for something unittest based, essentially taking the "golden output" comparisons and turning them into appropriate self.assert* invocations. Given the number of tests Greg has, it will probably make more sense to do it as a new test subdirectory rather than as a single test file (although that depends on how many tests are in each file - if there are only a few, or if they overlap a lot, then having them as separate test cases within a single file may be a better choice). >> Renaud's patch mostly applies cleanly at the moment - the only change >> is that the "#endif" for the Py_LIMITED_API check needs to be moved in >> pyerrors.h so it also covers the new StopIteration struct definition. > > if this helps, i've updated the patch to fix this. > https://bitbucket.org/rndblnch/cpython-pep380/changeset/6014d1720625 Yep, that does help. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From steve at pearwood.info Tue Jun 28 02:14:57 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Jun 2011 10:14:57 +1000 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <20110627192217.66A3F2505A3@webabinitio.net> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E089340.7060004@voidspace.org.uk> <20110627192726.51455b5d@msiwind> <20110627192217.66A3F2505A3@webabinitio.net> Message-ID: <4E091D01.4090105@pearwood.info> R. David Murray wrote: > So, the correct generic term for something that can be accessed > via attribute notation is attribute. The more specific term for an > attribute that is a method is method. We don't currently have a more > specific collective term for attributes that aren't methods. *That* > is the problem. "Attribute" also encompasses both instance attributes and class attributes. Rather than having two different words, we simply qualify the word when we need to distinguish them. Likewise, in the cases where it is important to distinguish methods from other attributes, we should qualify the word: data attribute vs method attribute. (I'm not suggesting that we should routinely refer to "method attribute" rather than simply method, but only when we wish to emphasize that methods are a kind of attribute and not a completely different kind of thing.) -- Steven From ncoghlan at gmail.com Tue Jun 28 02:30:14 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jun 2011 10:30:14 +1000 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E09085D.80202@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <4E09085D.80202@voidspace.org.uk> Message-ID: On Tue, Jun 28, 2011 at 8:46 AM, Michael Foord wrote: > So the Python 2.2 what's new talks about attributes and methods as different > things.... Of course the context makes it clear, but this mirrors how I use > the terms in discussion and how I see others generally using them. > > Great topic for bikeshedding. :-) Yep, as David said, "attribute" is legitimately used to mean *both* "all attributes (i.e. both data attributes and methods)" and "data attributes (i.e. excluding methods)". In general, context makes it clear which meaning is intended, and when that isn't the case, more verbose phrasing such as that in the previous sentence can make it explicit. Rather than fighting that convention, we should probably just confront the ambiguity head on and update http://docs.python.org/dev/glossary.html#term-attribute to describe both uses of the term (and add a separate entry for "data attribute", with a definition which basically says "attributes which are not methods"). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ben+python at benfinney.id.au Tue Jun 28 04:11:18 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 28 Jun 2011 12:11:18 +1000 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> Message-ID: <87r56en1rt.fsf@benfinney.id.au> Rob Cliffe writes: > On 27/06/2011 15:08, R. David Murray wrote: > > I guess what I'm saying is that I am more comfortable calling them > > all attributes than calling them all members. The term 'members' > > isn't used anywhere in the language itself, as far as I can recall, > > whereas getattr and setattr are evidence that the language considers > > them all attributes. I think we do the documentation readers a > > disservice by obscuring that fact by using other terminology. +1 > 'function attributes' ? 'def attributes' ? ?1. They don't have to be functions, and hence don't have to be created by ?def?. > Or just stick with method attributes' ? ?callable attributes? describes exactly what they are, in terms that will remain useful to the person learning Python. -- \ ??Did you sleep well?? ?No, I made a couple of mistakes.?? | `\ ?Steven Wright | _o__) | Ben Finney From ericsnowcurrently at gmail.com Tue Jun 28 05:47:12 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Mon, 27 Jun 2011 21:47:12 -0600 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <87r56en1rt.fsf@benfinney.id.au> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> Message-ID: On Mon, Jun 27, 2011 at 8:11 PM, Ben Finney wrote: > Rob Cliffe writes: > >> 'function attributes' ? 'def attributes' ? > > -1. They don't have to be functions, and hence don't have to be created > by 'def'. > >> Or just stick with method attributes' ? > > "callable attributes" describes exactly what they are, in terms that > will remain useful to the person learning Python. > The usage of the object determines what we call it then, so what about "state attributes" in the same vein as "callable attributes" (data vs. method). But it would be nice to have the names consistent across the different contexts. ABCMeta tests __isabstractmethod__ on each attribute of a class, not just the methods, rather that __isabstractattribute__. Perhaps calling something a method attribute even when it isn't a function is still okay. Thus the pair could be "method and data attributes". "method attribute" is a little redundant but calling it a "function attribute" seems less consistent -eric > -- > \ "'Did you sleep well?' 'No, I made a couple of mistakes.'" | > `\ --Steven Wright | > _o__) | > Ben Finney > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com > From ben+python at benfinney.id.au Tue Jun 28 07:05:03 2011 From: ben+python at benfinney.id.au (Ben Finney) Date: Tue, 28 Jun 2011 15:05:03 +1000 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> Message-ID: <87mxh2mtq8.fsf@benfinney.id.au> Eric Snow writes: > On Mon, Jun 27, 2011 at 8:11 PM, Ben Finney wrote: > > "callable attributes" describes exactly what they are, in terms that > > will remain useful to the person learning Python. > > The usage of the object determines what we call it then No, the capability of the attribute is what determines that difference, not how it's used. I still don't have a good term for ?non-callable attribute?, though. -- \ ?I'm not a bad guy! I work hard, and I love my kids. So why | `\ should I spend half my Sunday hearing about how I'm going to | _o__) Hell?? ?Homer Simpson | Ben Finney From martin at v.loewis.de Tue Jun 28 08:21:29 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 28 Jun 2011 08:21:29 +0200 Subject: [Python-Dev] Snow Leopard buildbot failing again... In-Reply-To: <30385.1309201538@parc.com> References: <26766.1309194073@parc.com> <20110627173641.1894.764887772.divmod.xquotient.459@localhost.localdomain> <30385.1309201538@parc.com> Message-ID: <4E0972E9.5080400@v.loewis.de> Am 27.06.2011 21:05, schrieb Bill Janssen: > I also find interesting, because > it seems a good explanation of why the build slave might go into the > zombie state of attempting to reconnect to the master. That wouldn't apply to our buildslaves, though, since we don't have FileUpload build steps (except for a single builder run by David Bolen). Regards, Martin From ncoghlan at gmail.com Tue Jun 28 08:33:23 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jun 2011 16:33:23 +1000 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <87mxh2mtq8.fsf@benfinney.id.au> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> Message-ID: On Tue, Jun 28, 2011 at 3:05 PM, Ben Finney wrote: > I still don't have a good term for ?non-callable attribute?, though. The two terms I've got out of this thread are "callable attributes" (instance/static/class methods, etc) and "data attributes" (everything else). Both seem reasonable to me, creating two largely disjoint sets that together cover all the different kinds of attribute you're likely to encounter. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From nad at acm.org Tue Jun 28 10:17:16 2011 From: nad at acm.org (Ned Deily) Date: Tue, 28 Jun 2011 01:17:16 -0700 Subject: [Python-Dev] cpython: remove duplicate -I References: Message-ID: In article , benjamin.peterson wrote: > http://hg.python.org/cpython/rev/0d0ca6f95d9d > changeset: 70879:0d0ca6f95d9d > user: Benjamin Peterson > date: Sun Jun 19 17:17:30 2011 -0500 > summary: > remove duplicate -I > > files: > Makefile.pre.in | 2 +- > 1 files changed, 1 insertions(+), 1 deletions(-) > > > diff --git a/Makefile.pre.in b/Makefile.pre.in > --- a/Makefile.pre.in > +++ b/Makefile.pre.in > @@ -73,7 +73,7 @@ > # Both CPPFLAGS and LDFLAGS need to contain the shell's value for setup.py > to > # be able to build extension modules using the directories specified in the > # environment variables > -PY_CPPFLAGS= -I. -IInclude -I$(srcdir)/Include $(CONFIGURE_CPPFLAGS) > $(CPPFLAGS) > +PY_CPPFLAGS= -I. -IInclude $(CONFIGURE_CPPFLAGS) $(CPPFLAGS) > PY_LDFLAGS= $(CONFIGURE_LDFLAGS) $(LDFLAGS) > NO_AS_NEEDED= @NO_AS_NEEDED@ > LDLAST= @LDLAST@ I think someone has already mentioned that this change breaks builds with a source directory outside the build directory and that breaks my OS X installer builds. Please revert or make the following change which works for me (but not tested elsewhere): -PY_CPPFLAGS= -I. -IInclude $(CONFIGURE_CPPFLAGS) $(CPPFLAGS) +PY_CPPFLAGS= -I. -I$(srcdir)/Include $(CONFIGURE_CPPFLAGS) $(CPPFLAGS) -- Ned Deily, nad at acm.org From solipsis at pitrou.net Tue Jun 28 10:24:31 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Jun 2011 10:24:31 +0200 Subject: [Python-Dev] Snow Leopard buildbot failing again... References: <26766.1309194073@parc.com> Message-ID: <20110628102431.2fd841d4@pitrou.net> On Mon, 27 Jun 2011 10:01:13 PDT Bill Janssen wrote: > I see that parc-snowleopard-1 went down again. I've done a software > update, rebooted, and installed the latest buildslave, 0.8.4. I can > ping dinsdale.python.org successfully from the machine. However, when I > start the buildslave, I get this: Well, doesn't the slave work correctly now? Or is there a problem? From fdrake at acm.org Tue Jun 28 12:44:08 2011 From: fdrake at acm.org (Fred Drake) Date: Tue, 28 Jun 2011 06:44:08 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> Message-ID: On Tue, Jun 28, 2011 at 2:33 AM, Nick Coghlan wrote: > The two terms I've got out of this thread are "callable attributes" > (instance/static/class methods, etc) and "data attributes" (everything > else). Both seem reasonable to me, creating two largely disjoint sets > that together cover all the different kinds of attribute you're likely > to encounter. But "callable attributes" aren't the same thing as methods; most are methods, but not all. Sometimes, they're data used by the object. The fact that data attributes can be callable is irrelevant. -Fred -- Fred L. Drake, Jr.? ? "Give me the luxuries of life and I will willingly do without the necessities." ?? --Frank Lloyd Wright From fuzzyman at voidspace.org.uk Tue Jun 28 12:54:39 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Jun 2011 11:54:39 +0100 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> Message-ID: <4E09B2EF.9030304@voidspace.org.uk> On 28/06/2011 11:44, Fred Drake wrote: > On Tue, Jun 28, 2011 at 2:33 AM, Nick Coghlan wrote: >> The two terms I've got out of this thread are "callable attributes" >> (instance/static/class methods, etc) and "data attributes" (everything >> else). Both seem reasonable to me, creating two largely disjoint sets >> that together cover all the different kinds of attribute you're likely >> to encounter. > But "callable attributes" aren't the same thing as methods; most are methods, > but not all. Sometimes, they're data used by the object. The fact that > data attributes can be callable is irrelevant. Added to which there are other descriptors, notably property, that are not directly callable but are not provided as normal "data attributes" (although the access syntax is the same). Properties are much closer to methods as they are implemented on the class and fetched via the descriptor protocol. Instead of "data attributes" I prefer the term "instance attributes" although that doesn't include "class attributes" (or more precisely it doesn't cover "class attributes that aren't descriptors"). The problem with "data attributes" is that it doesn't mean *anything*, which I suppose is useful for invented terminology, but it means it doesn't convey anything precise to those who haven't heard the term before. If it becomes widely used then that changes I guess. I'd still normally just use "attributes" though... All the best, Michael Foord > > -Fred > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From fdrake at acm.org Tue Jun 28 13:04:11 2011 From: fdrake at acm.org (Fred Drake) Date: Tue, 28 Jun 2011 07:04:11 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E09B2EF.9030304@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> Message-ID: On Tue, Jun 28, 2011 at 6:54 AM, Michael Foord wrote: > Added to which there are other descriptors, notably property, that are not > directly callable but are not provided as normal "data attributes" (although > the access syntax is the same). Properties are much closer to methods as > they are implemented on the class and fetched via the descriptor protocol. > Instead of "data attributes" I prefer the term "instance attributes" > although that doesn't include "class attributes" (or more precisely it > doesn't cover "class attributes that aren't descriptors"). Given the availability of __getattr__ and __getattribute__, I consider properties an implementation detail for some attributes. The fact that Python code is called on access is only marginally interesting. > The problem with "data attributes" is that it doesn't mean *anything*, which > I suppose is useful for invented terminology, but it means it doesn't convey > anything precise to those who haven't heard the term before. If it becomes > widely used then that changes I guess. I'd still normally just use > "attributes" though... I'd read "data attributes" the same as "non-method attributes". For readers, calling them "attributes" is typically sufficient. It's rare to need to distinguish them from methods. -Fred -- Fred L. Drake, Jr.? ? "Give me the luxuries of life and I will willingly do without the necessities." ?? --Frank Lloyd Wright From fuzzyman at voidspace.org.uk Tue Jun 28 13:35:40 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Jun 2011 12:35:40 +0100 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> Message-ID: <4E09BC8C.7090807@voidspace.org.uk> On 28/06/2011 12:04, Fred Drake wrote: > On Tue, Jun 28, 2011 at 6:54 AM, Michael Foord > wrote: >> Added to which there are other descriptors, notably property, that are not >> directly callable but are not provided as normal "data attributes" (although >> the access syntax is the same). Properties are much closer to methods as >> they are implemented on the class and fetched via the descriptor protocol. >> Instead of "data attributes" I prefer the term "instance attributes" >> although that doesn't include "class attributes" (or more precisely it >> doesn't cover "class attributes that aren't descriptors"). > Given the availability of __getattr__ and __getattribute__, I consider > properties an implementation detail for some attributes. The fact that > Python code is called on access is only marginally interesting. > Well, they're *usually* implemented as methods and backed by a real instance attribute. Usually (but not always) it makes more sense (IME) to group them with methods. The fact that they're *accessed* as an attribute is the uninteresting detail. __getattr__ and __getattribute__ are interesting - they allow you to use attribute access syntax for things that *aren't* attributes. I appreciate the fact that the Python data-model means methods are just object attributes, but they're not *instance attributes* and sometimes you need to make a distinction. (And yes it is helpful if the standard terminology leads people into a better understanding of the Python data model, but that still doesn't change the need on occasion for terminology that doesn't need to be explained whenever it is used.) Given that how even methods are to be described depends on the context (if you're fetching bound methods as objects then it makes perfect sense to just talk about them as attributes) it doesn't seem an area amenable to one-size-fits-all terminology. >> The problem with "data attributes" is that it doesn't mean *anything*, which >> I suppose is useful for invented terminology, but it means it doesn't convey >> anything precise to those who haven't heard the term before. If it becomes >> widely used then that changes I guess. I'd still normally just use >> "attributes" though... > I'd read "data attributes" the same as "non-method attributes". For readers, > calling them "attributes" is typically sufficient. It's rare to need to > distinguish them from methods. Yeah, this is all a grand bikeshed. I'm not sure I would understand "data attributes" unless it was clear from the context. I would wonder what qualifies something as "data". It is an interesting question what terminology we should use in the documentation if we need to distinguish them, but I think that is still wandering away from the original question that was posed. All the best, Michael > > -Fred > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From rdmurray at bitdance.com Tue Jun 28 13:51:35 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 28 Jun 2011 07:51:35 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E09B2EF.9030304@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> Message-ID: <20110628115136.248822505A3@webabinitio.net> On Tue, 28 Jun 2011 11:54:39 +0100, Michael Foord wrote: > On 28/06/2011 11:44, Fred Drake wrote: > > But "callable attributes" aren't the same thing as methods; most are methods, > > but not all. Sometimes, they're data used by the object. The fact that > > data attributes can be callable is irrelevant. > > Added to which there are other descriptors, notably property, that are > not directly callable but are not provided as normal "data attributes" > (although the access syntax is the same). Properties are much closer to > methods as they are implemented on the class and fetched via the > descriptor protocol. Instead of "data attributes" I prefer the term > "instance attributes" although that doesn't include "class attributes" > (or more precisely it doesn't cover "class attributes that aren't > descriptors"). Also, instances can have methods as instance attributes. Trying to use 'instance attributes' for non-method attributes is a bad idea, I think. Given that there is no one thing that covers all non-method attributes, I suspect 'non-method attributes' is as good as we're going to manage. -- R. David Murray http://www.bitdance.com From fuzzyman at voidspace.org.uk Tue Jun 28 14:03:32 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Jun 2011 13:03:32 +0100 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <20110628115136.248822505A3@webabinitio.net> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> <20110628115136.248822505A3@webabinitio.net> Message-ID: <4E09C314.3090809@voidspace.org.uk> On 28/06/2011 12:51, R. David Murray wrote: > On Tue, 28 Jun 2011 11:54:39 +0100, Michael Foord wrote: >> On 28/06/2011 11:44, Fred Drake wrote: >>> But "callable attributes" aren't the same thing as methods; most are methods, >>> but not all. Sometimes, they're data used by the object. The fact that >>> data attributes can be callable is irrelevant. >> Added to which there are other descriptors, notably property, that are >> not directly callable but are not provided as normal "data attributes" >> (although the access syntax is the same). Properties are much closer to >> methods as they are implemented on the class and fetched via the >> descriptor protocol. Instead of "data attributes" I prefer the term >> "instance attributes" although that doesn't include "class attributes" >> (or more precisely it doesn't cover "class attributes that aren't >> descriptors"). > Also, instances can have methods as instance attributes. > > Trying to use 'instance attributes' for non-method attributes is a bad > idea, I think. I would use instance attributes for members that are held in the instance dict (or have specific slots). As this can't be a normal "method" (it could be any object including a callable one - but won't be a standard method descriptor) it seems uncontroversial. (Or more to the point it seems to be *precise* in its meaning in the context of the Python data model.) What do you mean by "instances can have methods as instance attributes"? Once you attach a bound method directly to an instance it becomes a slightly different beast I think. (On top of which that is pretty rare behaviour.) > Given that there is no one thing that covers all non-method > attributes, I suspect 'non-method attributes' is as good as > we're going to manage. Hehe, yeah - that's not bad... It's hard to see how it could be misunderstood too. Michael > -- > R. David Murray http://www.bitdance.com -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From ncoghlan at gmail.com Tue Jun 28 14:31:33 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jun 2011 22:31:33 +1000 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E09B2EF.9030304@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> Message-ID: On Tue, Jun 28, 2011 at 8:54 PM, Michael Foord wrote: > The problem with "data attributes" is that it doesn't mean *anything*, which > I suppose is useful for invented terminology, but it means it doesn't convey > anything precise to those who haven't heard the term before. If it becomes > widely used then that changes I guess. I'd still normally just use > "attributes" though... In this context, precision is an illusion. There is *no* precise terminology, not only because Python blurs the boundaries by design in many respects (callable or not, method or not, attribute or not, etc), but also because *people* use the same words to mean slightly different things. The best we can hope for is to encourage the right way of thinking about the situation, and in that regard you have the complete set of attributes accessible via an object (i.e. via __getattribute__), some of which are callables (and may or may not act like instance methods) and the remainder of which are data attributes (some of which may incidentally be callable, even if they aren't used that way). How a *particular* attribute is classified is not an inherent property of the attribute, but also an artifact of the way it is used by the application. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tseaver at palladion.com Tue Jun 28 14:43:35 2011 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 28 Jun 2011 08:43:35 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/28/2011 06:44 AM, Fred Drake wrote: > On Tue, Jun 28, 2011 at 2:33 AM, Nick Coghlan wrote: >> The two terms I've got out of this thread are "callable attributes" >> (instance/static/class methods, etc) and "data attributes" (everything >> else). Both seem reasonable to me, creating two largely disjoint sets >> that together cover all the different kinds of attribute you're likely >> to encounter. > > But "callable attributes" aren't the same thing as methods; most are methods, > but not all. Sometimes, they're data used by the object. The fact that > data attributes can be callable is irrelevant. Isn't it fuzzy / incorrect to refer to any callable attribute as a method until it have been extracted via the dot operator / getattr, and therefore bound via descriptor semantics? In this sense, 'staticmethod' doesn't create a "method" at all -- it just defeats the default creation of a method-yielding descriptor. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4JzHcACgkQ+gerLs4ltQ6I5ACZAXlBsZkrzQowKYBmJE4NQb4V F14AnRtkWByqwpRATan4OOTMgPqwyjxH =hru9 -----END PGP SIGNATURE----- From fuzzyman at voidspace.org.uk Tue Jun 28 14:56:40 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Jun 2011 13:56:40 +0100 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> Message-ID: <4E09CF88.4090307@voidspace.org.uk> On 28/06/2011 13:31, Nick Coghlan wrote: > On Tue, Jun 28, 2011 at 8:54 PM, Michael Foord > wrote: >> The problem with "data attributes" is that it doesn't mean *anything*, which >> I suppose is useful for invented terminology, but it means it doesn't convey >> anything precise to those who haven't heard the term before. If it becomes >> widely used then that changes I guess. I'd still normally just use >> "attributes" though... > In this context, precision is an illusion. There is *no* precise > terminology, "instance members" is precise, (whether or not an attribute is fetched "from the instance" is one thing that is clear in the python data model - there are just lots of other ways of fetching attributes too). "instance members" just doesn't cover all the cases where you may want to group attributes together though. However, providing "__getattr__" and "__getattribute__" is not the common case and even properties are *usually* backed by a real instance attribute. So "instance attributes" as opposed to other types of attributes is *often* a useful distinction. I don't think "data attributes" is clear or precise. Is a property a data attribute (well it depends how it is implemented and what it does), is a staticmethod a data attribute (no - but then Tres' question - it isn't a normal method either so if you define data attribute to mean "all non method attributes" then its potentially an open question). All the best, Michael Foord > not only because Python blurs the boundaries by design in > many respects (callable or not, method or not, attribute or not, etc), > but also because *people* use the same words to mean slightly > different things. > > The best we can hope for is to encourage the right way of thinking > about the situation, and in that regard you have the complete set of > attributes accessible via an object (i.e. via __getattribute__), some > of which are callables (and may or may not act like instance methods) > and the remainder of which are data attributes (some of which may > incidentally be callable, even if they aren't used that way). How a > *particular* attribute is classified is not an inherent property of > the attribute, but also an artifact of the way it is used by the > application. > > Cheers, > Nick. > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From steve at pearwood.info Tue Jun 28 15:20:34 2011 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 Jun 2011 23:20:34 +1000 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E09C314.3090809@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> <20110628115136.248822505A3@webabinitio.net> <4E09C314.3090809@voidspace.org.uk> Message-ID: <4E09D522.7000009@pearwood.info> Michael Foord wrote: > What do you mean by "instances can have methods as instance attributes"? > Once you attach a bound method directly to an instance it becomes a > slightly different beast I think. (On top of which that is pretty rare > behaviour.) >>> class C: ... def method(self, x): ... return x+1 ... >>> c = C() >>> c.method = types.MethodType(lambda self, x: x+101, c) >>> c.method(1) 102 I don't know how rare it is, but it's a useful trick for customising the behaviour of instances. As I see it, there are three dichotomies we sometimes need to make: (1) Instance attributes vs class (shared) attributes. Broadly speaking, whether the attribute is in instance.__dict__ or type(instance).__dict__. (2) Computed vs non-computed attributes. Attributes which are computed by __getattr__ or via the descriptor protocol (which includes properties) are all computed attributes; everything else is non-computed. (3) Method attributes (methods) vs non-method/data attributes. Broadly speaking, methods are callable, non-method (data) attributes are not. The three are orthogonal: e.g. a staticmethod is a method by virtue of being callable, computed by virtue of being generated by a descriptor, and a class attribute by virtue of existing in the type __dict__ rather than the instance __dict__. Strictly speaking, (3) is not truly a dichotomy, since functions and methods are first class-objects in Python. E.g. one may store a function as an attribute with the intention of using it as data rather than as a method. But that's a moderately obscure corner case, and in my opinion it's not worth obscuring the practical distinction between "methods are things you call, data are not" for the sake of it. Leave the functions-as-data case for a footnote. -- Steven From fuzzyman at voidspace.org.uk Tue Jun 28 15:25:13 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Jun 2011 14:25:13 +0100 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E09CF88.4090307@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> <4E09CF88.4090307@voidspace.org.uk> Message-ID: <4E09D639.5000403@voidspace.org.uk> On 28/06/2011 13:56, Michael Foord wrote: > On 28/06/2011 13:31, Nick Coghlan wrote: >> On Tue, Jun 28, 2011 at 8:54 PM, Michael Foord >> wrote: >>> The problem with "data attributes" is that it doesn't mean >>> *anything*, which >>> I suppose is useful for invented terminology, but it means it >>> doesn't convey >>> anything precise to those who haven't heard the term before. If it >>> becomes >>> widely used then that changes I guess. I'd still normally just use >>> "attributes" though... >> In this context, precision is an illusion. There is *no* precise >> terminology, > "instance members" is precise, (whether or not an attribute is fetched > "from the instance" is one thing that is clear in the python data > model - there are just lots of other ways of fetching attributes too). > "instance members" just doesn't cover all the cases where you may want > to group attributes together though. > > However, providing "__getattr__" and "__getattribute__" is not the > common case and even properties are *usually* backed by a real > instance attribute. So "instance attributes" as opposed to other types > of attributes is *often* a useful distinction. > > I don't think "data attributes" is clear or precise. Although to be fair "data attributes" mirrors the terminology for descriptors where we have "data descriptors" (which have both __set__ and __get__) and "non data descriptors" (__get__ only). Unfortunately there are also non-non-data decriptors too (only __set__ and / or __delete__) so that terminology is at least slightly confusing / imprecise [1]... There was a genuine Python "bug" caused by that confusion at one point (unfortunately I forget what it was). All the best, Michael [1] From: http://users.rcn.com/python/download/Descriptor.htm If an object defines both __get__ and __set__, it is considered a data descriptor. Descriptors that only define __get__ are called non-data descriptors (they are typically used for methods but other uses are possible). > Is a property a data attribute (well it depends how it is implemented > and what it does), is a staticmethod a data attribute (no - but then > Tres' question - it isn't a normal method either so if you define data > attribute to mean "all non method attributes" then its potentially an > open question). > > All the best, > > Michael Foord > >> not only because Python blurs the boundaries by design in >> many respects (callable or not, method or not, attribute or not, etc), >> but also because *people* use the same words to mean slightly >> different things. >> >> The best we can hope for is to encourage the right way of thinking >> about the situation, and in that regard you have the complete set of >> attributes accessible via an object (i.e. via __getattribute__), some >> of which are callables (and may or may not act like instance methods) >> and the remainder of which are data attributes (some of which may >> incidentally be callable, even if they aren't used that way). How a >> *particular* attribute is classified is not an inherent property of >> the attribute, but also an artifact of the way it is used by the >> application. >> >> Cheers, >> Nick. >> > > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From fuzzyman at voidspace.org.uk Tue Jun 28 15:27:32 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Jun 2011 14:27:32 +0100 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E09D522.7000009@pearwood.info> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> <20110628115136.248822505A3@webabinitio.net> <4E09C314.3090809@voidspace.org.uk> <4E09D522.7000009@pearwood.info> Message-ID: <4E09D6C4.1050505@voidspace.org.uk> On 28/06/2011 14:20, Steven D'Aprano wrote: > Michael Foord wrote: > >> What do you mean by "instances can have methods as instance >> attributes"? Once you attach a bound method directly to an instance >> it becomes a slightly different beast I think. (On top of which that >> is pretty rare behaviour.) > > >>> class C: > ... def method(self, x): > ... return x+1 > ... > >>> c = C() > >>> c.method = types.MethodType(lambda self, x: x+101, c) > >>> c.method(1) > 102 > > I don't know how rare it is, but it's a useful trick for customising > the behaviour of instances. > Right - that method is an instance attribute. > > As I see it, there are three dichotomies we sometimes need to make: > > > (1) Instance attributes vs class (shared) attributes. > > Broadly speaking, whether the attribute is in instance.__dict__ or > type(instance).__dict__. > > (2) Computed vs non-computed attributes. > > Attributes which are computed by __getattr__ or via the descriptor > protocol (which includes properties) are all computed attributes; > everything else is non-computed. Technically also via __getattribute__ when overridden. > > (3) Method attributes (methods) vs non-method/data attributes. > > Broadly speaking, methods are callable, non-method (data) attributes > are not. > > > The three are orthogonal: e.g. a staticmethod is a method by virtue of > being callable, computed by virtue of being generated by a descriptor, > and a class attribute by virtue of existing in the type __dict__ > rather than the instance __dict__. > > Strictly speaking, (3) is not truly a dichotomy, since functions and > methods are first class-objects in Python. E.g. one may store a > function as an attribute with the intention of using it as data rather > than as a method. But that's a moderately obscure corner case, and in > my opinion it's not worth obscuring the practical distinction between > "methods are things you call, data are not" for the sake of it. Leave > the functions-as-data case for a footnote. > Yep, useful summary. Michael > > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From ncoghlan at gmail.com Tue Jun 28 15:28:28 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jun 2011 23:28:28 +1000 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E09CF88.4090307@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> <4E09CF88.4090307@voidspace.org.uk> Message-ID: On Tue, Jun 28, 2011 at 10:56 PM, Michael Foord wrote: > I don't think "data attributes" is clear or precise. Is a property a data > attribute (well it depends how it is implemented and what it does), is a > staticmethod a data attribute (no - but then Tres' question - it isn't a > normal method either so if you define data attribute to mean "all non method > attributes" then its potentially an open question). "callable attributes" and "data attributes" are purely about the API exposed by the object in question. If we're going for object model neutral terminology, that's the only view that makes sense. They're descriptions about how attributes are intended to be *used* that are completely silent on the question of how they're *implemented*. So staticmethod would fall into the first group, while property would fall into the latter. >From an implementation point of view, you carve up the world differently, so it makes sense to worry about class attributes, instance attributes, dynamic attributes, etc. (the class vs instance distinction can also matter to some degree from the usage point of view, since it affects the scope of any mutable attributes, and the static vs dynamic distinction can also matter, especially for introspection purposes). This goes back to the original point about all of this being highly context dependent - how you carve up the set of all attributes is going to change based on what you're trying to explain (e.g. the distinction between enforced 'data' descriptors, descriptors that allow shadowing in the instance dict, class attributes that aren't descriptors at all, instance attributes and dynamic attributes retrieved via __getattr__ is another way of dividing them) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Tue Jun 28 15:37:34 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Jun 2011 23:37:34 +1000 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E09D6C4.1050505@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> <20110628115136.248822505A3@webabinitio.net> <4E09C314.3090809@voidspace.org.uk> <4E09D522.7000009@pearwood.info> <4E09D6C4.1050505@voidspace.org.uk> Message-ID: On Tue, Jun 28, 2011 at 11:27 PM, Michael Foord wrote: > Technically also via __getattribute__ when overridden. Since object.__getattribute__ is the hook that implements the entire attribute lookup protocol, *all* attributes are technically retrieved via __getattribute__ (which is why overriding it correctly can be such a PITA). That's also the hook type() overrides to make class attribute lookup differ from ordinary instance lookup. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From fuzzyman at voidspace.org.uk Tue Jun 28 15:41:14 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Jun 2011 14:41:14 +0100 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: <20110624145516.84DA32506EA@webabinitio.net> References: <20110624145516.84DA32506EA@webabinitio.net> Message-ID: <4E09D9FA.7060308@voidspace.org.uk> On 24/06/2011 15:55, R. David Murray wrote: > On Fri, 24 Jun 2011 10:52:40 +0200, Mark Dickinson wrote: >> EuroPython 2011 Language Summit >> =============================== > [...] >> Unicode character classes is a particular need). [Subtopic: what needs >> to be done to get the new regex module into Python? Should it replace >> the existing module? What about backwards compatibility issues?] > I'm pretty sure regex has backward compatibility as a goal for just this > reason (so it can replace the current module). > The new regex library has some great improvements: http://bugs.python.org/issue2636 It also has users and committed maintainers, so I hope we can bring it into 3.3. It wasn't easy to tell from skimming the change notes that Unicode character classes are amongst the new features. Is that the case? Michael >> PEP 0380: Syntax for delegating to a subgenerator. >> >> PEP 3150: Statement local namespaces. >> >> PEP 3152: Cofunctions. >> >> For all three of the above PEPs, there was some feeling that additional syntax >> changes to the language are unnecessary and make it harder to learn; where >> possible, we should prefer using 3rd partly libraries. > I disagree with this with respect to 380. Haven't looked at the others. > >> Python 2.7 >> ---------- >> >> How long will this be maintained? >> Original decision was 5 years. > As I recall, the actual decision was "at *least* 5 years". It's only > been one so far...was the discussion that five years was too short or > too long? > > As the code bases continue to drift apart, and we fix the fixable bugs, > the number of patches going in to 2.7 will decrease over time, so I > don't think the burden of continuing to support it will be too heavy. > Currently about half of the non-feature-request issues (ie: bugs) in > the tracker are tagged 2.7. I expect to see that percentage continue > to decrease over time. > > -- > R. David Murray http://www.bitdance.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From victor.stinner at haypocalc.com Tue Jun 28 15:43:05 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 28 Jun 2011 15:43:05 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? Message-ID: <1309268585.21006.22.camel@marge> In Python 2, open() opens the file in binary mode (e.g. file.readline() returns a byte string). codecs.open() opens the file in binary mode by default, you have to specify an encoding name to open it in text mode. In Python 3, open() opens the file in text mode by default. (It only opens the binary mode if the file mode contains "b".) The problem is that open() uses the locale encoding if the encoding is not specified, which is the case *by default*. The locale encoding can be: - UTF-8 on Mac OS X, most Linux distributions - ISO-8859-1 os some FreeBSD systems - ANSI code page on Windows, e.g. cp1252 (close to ISO-8859-1) in Western Europe, cp952 in Japan, ... - ASCII if the locale is manually set to an empty string or to "C", or if the environment is empty, or by default on some systems - something different depending on the system and user configuration... If you develop under Mac OS X or Linux, you may have surprises when you run your program on Windows on the first non-ASCII character. You may not detect the problem if you only write text in english... until someone writes the first letter with a diacritic. As discussed before on this list, I propose to set the default encoding of open() to UTF-8 in Python 3.3, and add a warning in Python 3.2 if open() is called without an explicit encoding and if the locale encoding is not UTF-8. Using the warning, you will quickly notice the potential problem (using Python 3.2.2 and -Werror) on Windows or by using a different locale encoding (.e.g using LANG="C"). I expect a lot of warnings from the Python standard library, and as many in third party modules and applications. So do you think that it is too late to change that in Python 3.3? One argument for changing it directly in Python 3.3 is that most users will not notice the change because their locale encoding is already UTF-8. An alternative is to: - Python 3.2: use the locale encoding but emit a warning if the locale encoding is not UTF-8 - Python 3.3: use UTF-8 and emit a warning if the locale encoding is not UTF-8... or maybe always emit a warning? - Python 3.3: use UTF-8 (but don't emit warnings anymore) I don't think that Windows developer even know that they are writing files into the ANSI code page. MSDN documentation of WideCharToMultiByte() warns developer that the ANSI code page is not portable, even accross Windows computers: "The ANSI code pages can be different on different computers, or can be changed for a single computer, leading to data corruption. For the most consistent results, applications should use Unicode, such as UTF-8 or UTF-16, instead of a specific code page, unless legacy standards or data formats prevent the use of Unicode. If using Unicode is not possible, applications should tag the data stream with the appropriate encoding name when protocols allow it. HTML and XML files allow tagging, but text files do not." It will always be possible to use ANSI code page using encoding="mbcs" (only work on Windows), or an explicit code page number (e.g. encoding="cp2152"). -- The two other (rejetected?) options to improve open() are: - raise an error if the encoding argument is not set: will break most programs - emit a warning if the encoding argument is not set -- Should I convert this email into a PEP, or is it not required? Victor From mal at egenix.com Tue Jun 28 16:02:52 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 28 Jun 2011 16:02:52 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <1309268585.21006.22.camel@marge> References: <1309268585.21006.22.camel@marge> Message-ID: <4E09DF0C.90200@egenix.com> Victor Stinner wrote: > In Python 2, open() opens the file in binary mode (e.g. file.readline() > returns a byte string). codecs.open() opens the file in binary mode by > default, you have to specify an encoding name to open it in text mode. > > In Python 3, open() opens the file in text mode by default. (It only > opens the binary mode if the file mode contains "b".) The problem is > that open() uses the locale encoding if the encoding is not specified, > which is the case *by default*. The locale encoding can be: > > - UTF-8 on Mac OS X, most Linux distributions > - ISO-8859-1 os some FreeBSD systems > - ANSI code page on Windows, e.g. cp1252 (close to ISO-8859-1) in > Western Europe, cp952 in Japan, ... > - ASCII if the locale is manually set to an empty string or to "C", or > if the environment is empty, or by default on some systems > - something different depending on the system and user configuration... > > If you develop under Mac OS X or Linux, you may have surprises when you > run your program on Windows on the first non-ASCII character. You may > not detect the problem if you only write text in english... until > someone writes the first letter with a diacritic. How about a more radical change: have open() in Py3 default to opening the file in binary mode, if no encoding is given (even if the mode doesn't include 'b') ? That'll make it compatible to the Py2 world again and avoid all the encoding guessing. Making such default encodings depend on the locale has already failed to work when we first introduced a default encoding in Py2, so I don't understand why we are repeating the same mistake again in Py3 (only in a different area). Note that in Py2, Unix applications often leave out the 'b' mode, since there's no difference between using it or not. Only on Windows, you'll see a difference. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 28 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From victor.stinner at haypocalc.com Tue Jun 28 16:06:58 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 28 Jun 2011 16:06:58 +0200 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: <4E09D9FA.7060308@voidspace.org.uk> References: <20110624145516.84DA32506EA@webabinitio.net> <4E09D9FA.7060308@voidspace.org.uk> Message-ID: <1309270018.21006.26.camel@marge> Le mardi 28 juin 2011 ? 14:41 +0100, Michael Foord a ?crit : > On 24/06/2011 15:55, R. David Murray wrote: > > On Fri, 24 Jun 2011 10:52:40 +0200, Mark Dickinson wrote: > >> EuroPython 2011 Language Summit > >> =============================== > > [...] > >> Unicode character classes is a particular need). [Subtopic: what needs > >> to be done to get the new regex module into Python? Should it replace > >> the existing module? What about backwards compatibility issues?] > > I'm pretty sure regex has backward compatibility as a goal for just this > > reason (so it can replace the current module). > > > The new regex library has some great improvements: > > http://bugs.python.org/issue2636 This issue is open since April 2008, has also the longest list of attached files, and has a very long history. What is the status of the issue? I see that there is now a third party project on: http://code.google.com/p/mrab-regex-hg/ -- There is also the re2 library from Google and especially this project: http://pypi.python.org/pypi/re2/ "pyre2 is a Python extension that wraps Google's RE2 regular expression library. This version of pyre2 is similar to the one you'd find at facebook's github repository except that the stated goal of *this version is to be a drop-in replacement for the re module*.)" Victor From solipsis at pitrou.net Tue Jun 28 16:06:17 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Jun 2011 16:06:17 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? References: <1309268585.21006.22.camel@marge> Message-ID: <20110628160617.56d11422@pitrou.net> On Tue, 28 Jun 2011 15:43:05 +0200 Victor Stinner wrote: > - ISO-8859-1 os some FreeBSD systems > - ANSI code page on Windows, e.g. cp1252 (close to ISO-8859-1) in > Western Europe, cp952 in Japan, ... > - ASCII if the locale is manually set to an empty string or to "C", or > if the environment is empty, or by default on some systems > - something different depending on the system and user configuration... Why would utf-8 be the right thing in these cases? Regards Antoine. From tjreedy at udel.edu Tue Jun 28 16:24:58 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 28 Jun 2011 10:24:58 -0400 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <1309268585.21006.22.camel@marge> References: <1309268585.21006.22.camel@marge> Message-ID: On 6/28/2011 9:43 AM, Victor Stinner wrote: > In Python 2, open() opens the file in binary mode (e.g. file.readline() > returns a byte string). codecs.open() opens the file in binary mode by > default, you have to specify an encoding name to open it in text mode. > > In Python 3, open() opens the file in text mode by default. (It only > opens the binary mode if the file mode contains "b".) The problem is > that open() uses the locale encoding if the encoding is not specified, > which is the case *by default*. The locale encoding can be: > > - UTF-8 on Mac OS X, most Linux distributions > - ISO-8859-1 os some FreeBSD systems > - ANSI code page on Windows, e.g. cp1252 (close to ISO-8859-1) in > Western Europe, cp952 in Japan, ... > - ASCII if the locale is manually set to an empty string or to "C", or > if the environment is empty, or by default on some systems > - something different depending on the system and user configuration... > > If you develop under Mac OS X or Linux, you may have surprises when you > run your program on Windows on the first non-ASCII character. You may > not detect the problem if you only write text in english... until > someone writes the first letter with a diacritic. > > > > As discussed before on this list, I propose to set the default encoding > of open() to UTF-8 in Python 3.3, and add a warning in Python 3.2 if > open() is called without an explicit encoding and if the locale encoding > is not UTF-8. Using the warning, you will quickly notice the potential > problem (using Python 3.2.2 and -Werror) on Windows or by using a > different locale encoding (.e.g using LANG="C"). > > I expect a lot of warnings from the Python standard library, and as many > in third party modules and applications. So do you think that it is too > late to change that in Python 3.3? One argument for changing it directly > in Python 3.3 is that most users will not notice the change because > their locale encoding is already UTF-8. > > An alternative is to: > - Python 3.2: use the locale encoding but emit a warning if the locale > encoding is not UTF-8 > - Python 3.3: use UTF-8 and emit a warning if the locale encoding is > not UTF-8... or maybe always emit a warning? > - Python 3.3: use UTF-8 (but don't emit warnings anymore) > > I don't think that Windows developer even know that they are writing > files into the ANSI code page. MSDN documentation of > WideCharToMultiByte() warns developer that the ANSI code page is not > portable, even accross Windows computers: > > "The ANSI code pages can be different on different computers, or can be > changed for a single computer, leading to data corruption. For the most > consistent results, applications should use Unicode, such as UTF-8 or > UTF-16, instead of a specific code page, unless legacy standards or data > formats prevent the use of Unicode. If using Unicode is not possible, > applications should tag the data stream with the appropriate encoding > name when protocols allow it. HTML and XML files allow tagging, but text > files do not." > > It will always be possible to use ANSI code page using > encoding="mbcs" (only work on Windows), or an explicit code page number > (e.g. encoding="cp2152"). > > -- > > The two other (rejetected?) options to improve open() are: > > - raise an error if the encoding argument is not set: will break most > programs > - emit a warning if the encoding argument is not set > > -- > > Should I convert this email into a PEP, or is it not required? I think a PEP is needed. -- Terry Jan Reedy From tjreedy at udel.edu Tue Jun 28 16:36:23 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 28 Jun 2011 10:36:23 -0400 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <4E09DF0C.90200@egenix.com> References: <1309268585.21006.22.camel@marge> <4E09DF0C.90200@egenix.com> Message-ID: On 6/28/2011 10:02 AM, M.-A. Lemburg wrote: > How about a more radical change: have open() in Py3 default to > opening the file in binary mode, if no encoding is given (even > if the mode doesn't include 'b') ? > > That'll make it compatible to the Py2 world again I disagree. I believe S = open('myfile.txt').read() now return a text string in both Py2 and Py3 and a subsequent 'abc' in S works in both. > and avoid all the encoding guessing. > Making such default encodings depend on the locale has already > failed to work when we first introduced a default encoding in > Py2, so I don't understand why we are repeating the same > mistake again in Py3 (only in a different area). I do not remember any proposed change during the Py3 design discussions. > Note that in Py2, Unix applications often leave out the 'b' > mode, since there's no difference between using it or not. I believe it makes a difference now as to whether one gets str or bytes. > Only on Windows, you'll see a difference. I believe the only difference now on Windows is the decoding used, not the return type. -- Terry Jan Reedy From benjamin at python.org Tue Jun 28 16:40:22 2011 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 28 Jun 2011 09:40:22 -0500 Subject: [Python-Dev] cpython: remove duplicate -I In-Reply-To: References: Message-ID: 2011/6/28 Ned Deily : > In article , > ?benjamin.peterson wrote: > >> http://hg.python.org/cpython/rev/0d0ca6f95d9d >> changeset: ? 70879:0d0ca6f95d9d >> user: ? ? ? ?Benjamin Peterson >> date: ? ? ? ?Sun Jun 19 17:17:30 2011 -0500 >> summary: >> ? remove duplicate -I >> >> files: >> ? Makefile.pre.in | ?2 +- >> ? 1 files changed, 1 insertions(+), 1 deletions(-) >> >> >> diff --git a/Makefile.pre.in b/Makefile.pre.in >> --- a/Makefile.pre.in >> +++ b/Makefile.pre.in >> @@ -73,7 +73,7 @@ >> ?# Both CPPFLAGS and LDFLAGS need to contain the shell's value for setup.py >> ?to >> ?# be able to build extension modules using the directories specified in the >> ?# environment variables >> -PY_CPPFLAGS= -I. -IInclude -I$(srcdir)/Include $(CONFIGURE_CPPFLAGS) >> $(CPPFLAGS) >> +PY_CPPFLAGS= -I. -IInclude $(CONFIGURE_CPPFLAGS) $(CPPFLAGS) >> ?PY_LDFLAGS= ?$(CONFIGURE_LDFLAGS) $(LDFLAGS) >> ?NO_AS_NEEDED= ? ? ? ?@NO_AS_NEEDED@ >> ?LDLAST= ? ? ? ? ? ? ?@LDLAST@ > > I think someone has already mentioned that this change breaks builds > with a source directory outside the build directory and that breaks my > OS X installer builds. ? Please revert or make the following change > which works for me (but not tested elsewhere): > > -PY_CPPFLAGS= ? -I. -IInclude $(CONFIGURE_CPPFLAGS) $(CPPFLAGS) > +PY_CPPFLAGS= ? -I. -I$(srcdir)/Include $(CONFIGURE_CPPFLAGS) $(CPPFLAGS) Committed. Thanks for the review. -- Regards, Benjamin From tjreedy at udel.edu Tue Jun 28 16:41:38 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 28 Jun 2011 10:41:38 -0400 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <20110628160617.56d11422@pitrou.net> References: <1309268585.21006.22.camel@marge> <20110628160617.56d11422@pitrou.net> Message-ID: On 6/28/2011 10:06 AM, Antoine Pitrou wrote: > On Tue, 28 Jun 2011 15:43:05 +0200 > Victor Stinner wrote: >> - ISO-8859-1 os some FreeBSD systems >> - ANSI code page on Windows, e.g. cp1252 (close to ISO-8859-1) in >> Western Europe, cp952 in Japan, ... >> - ASCII if the locale is manually set to an empty string or to "C", or >> if the environment is empty, or by default on some systems >> - something different depending on the system and user configuration... > > Why would utf-8 be the right thing in these cases? Because utf-8 is the only way to write out any Python 3 text. By default, writing and reading an str object should work on all Python installations. And because other apps are (increasingly) using it for exactly the same reason. -- Terry Jan Reedy From p.f.moore at gmail.com Tue Jun 28 16:46:12 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Jun 2011 15:46:12 +0100 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <1309268585.21006.22.camel@marge> References: <1309268585.21006.22.camel@marge> Message-ID: On 28 June 2011 14:43, Victor Stinner wrote: > As discussed before on this list, I propose to set the default encoding > of open() to UTF-8 in Python 3.3, and add a warning in Python 3.2 if > open() is called without an explicit encoding and if the locale encoding > is not UTF-8. Using the warning, you will quickly notice the potential > problem (using Python 3.2.2 and -Werror) on Windows or by using a > different locale encoding (.e.g using LANG="C"). -1. This will make things harder for simple scripts which are not intended to be cross-platform. I use Windows, and come from the UK, so 99% of my text files are ASCII. So the majority of my code will be unaffected. But in the occasional situation where I use a ? sign, I'll get encoding errors, where currently things will "just work". And the failures will be data dependent, and hence intermittent (the worst type of problem). I'll write a quick script, use it once and it'll be fine, then use it later on some different data and get an error. :-( I appreciate that the point here is to make sure that people think a bit more carefully about encoding issues. But doing so by making Python less friendly for casual, adhoc script use, seems to me to be a mistake. > I don't think that Windows developer even know that they are writing > files into the ANSI code page. MSDN documentation of > WideCharToMultiByte() warns developer that the ANSI code page is not > portable, even accross Windows computers: Probably true. But for many uses they also don't care. If you're writing something solely for a one-off job on your own PC, the ANSI code page is fine, and provides interoperability with other programs on your PC, which is really what you care about. (UTF-8 without BOM displays incorrectly in Vim, wordpad, and powershell get-content. MBCS works fine in all of these. It also displays incorrectly in CMD type, but in a less familiar form than the incorrect display mbcs produces, for what that's worth...) > It will always be possible to use ANSI code page using > encoding="mbcs" (only work on Windows), or an explicit code page number > (e.g. encoding="cp2152"). So, in effect, you propose making the default favour writing multiplatform portable code at the expense of quick and dirty scripts? My personal view is that this is the wrong choice ("practicality beats purity") but I guess it's ultimately a question of Python's design philosophy. > The two other (rejetected?) options to improve open() are: > > - raise an error if the encoding argument is not set: will break most > programs > - emit a warning if the encoding argument is not set IMHO, you missed another option - open() does not need improving, the current behaviour is better than any of the 3 options noted. Paul. From fuzzyman at voidspace.org.uk Tue Jun 28 16:48:09 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Jun 2011 15:48:09 +0100 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> <4E09DF0C.90200@egenix.com> Message-ID: <4E09E9A9.5020702@voidspace.org.uk> On 28/06/2011 15:36, Terry Reedy wrote: > On 6/28/2011 10:02 AM, M.-A. Lemburg wrote: > >> How about a more radical change: have open() in Py3 default to >> opening the file in binary mode, if no encoding is given (even >> if the mode doesn't include 'b') ? >> >> That'll make it compatible to the Py2 world again > > I disagree. I believe > S = open('myfile.txt').read() > now return a text string in both Py2 and Py3 and a subsequent > 'abc' in S > works in both. Nope, it returns a bytestring in Python 2. Mistakenly treating bytestrings as text is one of the things we aimed to correct in the transition to Python 3. Michael > > > and avoid all the encoding guessing. > >> Making such default encodings depend on the locale has already >> failed to work when we first introduced a default encoding in >> Py2, so I don't understand why we are repeating the same >> mistake again in Py3 (only in a different area). > > I do not remember any proposed change during the Py3 design discussions. > >> Note that in Py2, Unix applications often leave out the 'b' >> mode, since there's no difference between using it or not. > > I believe it makes a difference now as to whether one gets str or bytes. > >> Only on Windows, you'll see a difference. > > I believe the only difference now on Windows is the decoding used, not > the return type. > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From solipsis at pitrou.net Tue Jun 28 17:02:52 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Jun 2011 17:02:52 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? References: <1309268585.21006.22.camel@marge> <20110628160617.56d11422@pitrou.net> Message-ID: <20110628170252.7260dc65@pitrou.net> On Tue, 28 Jun 2011 10:41:38 -0400 Terry Reedy wrote: > On 6/28/2011 10:06 AM, Antoine Pitrou wrote: > > On Tue, 28 Jun 2011 15:43:05 +0200 > > Victor Stinner wrote: > >> - ISO-8859-1 os some FreeBSD systems > >> - ANSI code page on Windows, e.g. cp1252 (close to ISO-8859-1) in > >> Western Europe, cp952 in Japan, ... > >> - ASCII if the locale is manually set to an empty string or to "C", or > >> if the environment is empty, or by default on some systems > >> - something different depending on the system and user configuration... > > > > Why would utf-8 be the right thing in these cases? > > Because utf-8 is the only way to write out any Python 3 text. Er, no, you also have utf-16, utf-32, utf-7 (and possibly others, including home-baked encodings). > By default, writing and reading an str object should work on all Python > installations. But that's only half of the problem. If the text is supposed to be read or processed by some other program, then writing it in some encoding that the other program doesn't expect doesn't really help. That's why we use the locale encoding: because it's a good guess as to what the system (and its users) expects text to be encoded in. Regards Antoine. From sdaoden at googlemail.com Tue Jun 28 17:06:41 2011 From: sdaoden at googlemail.com (Steffen Daode Nurpmeso) Date: Tue, 28 Jun 2011 17:06:41 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> Message-ID: <20110628150641.GA29535@sherwood.local> @ Paul Moore wrote (2011-06-28 16:46+0200): > UTF-8 without BOM displays incorrectly in vim(1) Stop right now (you're oh so wrong)! :-) (By the way: UTF-8 and BOM? Interesting things i learn on this list. And i hope in ten years we can laugh about this -> UTF-8 transition all over the place, 'cause it's simply working.) -- Ciao, Steffen sdaoden(*)(gmail.com) () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments From tjreedy at udel.edu Tue Jun 28 17:23:33 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 28 Jun 2011 11:23:33 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <20110628115136.248822505A3@webabinitio.net> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> <20110628115136.248822505A3@webabinitio.net> Message-ID: On 6/28/2011 7:51 AM, R. David Murray wrote: > Also, instances can have methods as instance attributes. Functions that are instance attributes do not act like methods (instance.func() does not automagically turn instance in the first arg of func) and have never, to my knowledge, been called methods. In Python 2, they are not wrapped as methods whereas functions attached to classes are. So-called 'staticmethods' are not really methods either, but are class function attributes that are just functions and not treated as methods. The decorator that negates normal method treatment could/should have been called 'non_method'. Using 'function' is its generic 'callable' sense ... Method: a class function attribute that in its intended and normal use automagically turns the object it is called on into its first arg. 'Method' is a useful and needed subcategory of class attribute precisely because of this behavior. Instance method: a class function attribute that is (normally) called on instances of the class or subclasses. Class method: a class function attribute that is (normally) called on the class or subclasses. Bound method: a method that has already has the first-arg object bundled with it, so that it can be used as a normal (partial or curried) function. Except for 'classmethod', which was added later, these have been the meanings as I understood them since at least Py 1.4, 15 years ago, and they are exactly what one needs to know to use Python. Any object can be an attribute. However, function attributes of classes normally get special 'method' treatment which alters the meaning of syntax. -- Terry Jan Reedy From ericsnowcurrently at gmail.com Tue Jun 28 17:35:01 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 28 Jun 2011 09:35:01 -0600 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E09BC8C.7090807@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> <4E09BC8C.7090807@voidspace.org.uk> Message-ID: On Tue, Jun 28, 2011 at 5:35 AM, Michael Foord wrote: > > Well, they're *usually* implemented as methods and backed by a real instance > attribute. Usually (but not always) it makes more sense (IME) to group them > with methods. The fact that they're *accessed* as an attribute is the > uninteresting detail. > > __getattr__ and __getattribute__ are interesting - they allow you to use > attribute access syntax for things that *aren't* attributes. > > I appreciate the fact that the Python data-model means methods are just > object attributes, but they're not *instance attributes* and sometimes you > need to make a distinction. (And yes it is helpful if the standard > terminology leads people into a better understanding of the Python data > model, but that still doesn't change the need on occasion for terminology > that doesn't need to be explained whenever it is used.) > While the distinction between class-specific attributes and instance-specific is important, I don't think merging it with the method/data distinction is as helpful. The valuable information here is the expectation of how the attributes are used. In my mind, that boils down to data (holds static or dymanic state) and methods (does something on the class or instance). -eric > Given that how even methods are to be described depends on the context (if > you're fetching bound methods as objects then it makes perfect sense to just > talk about them as attributes) it doesn't seem an area amenable to > one-size-fits-all terminology. > >>> The problem with "data attributes" is that it doesn't mean *anything*, >>> which >>> I suppose is useful for invented terminology, but it means it doesn't >>> convey >>> anything precise to those who haven't heard the term before. If it >>> becomes >>> widely used then that changes I guess. I'd still normally just use >>> "attributes" though... >> >> I'd read "data attributes" the same as "non-method attributes". ?For >> readers, >> calling them "attributes" is typically sufficient. ?It's rare to need to >> distinguish them from methods. > > Yeah, this is all a grand bikeshed. I'm not sure I would understand "data > attributes" unless it was clear from the context. I would wonder what > qualifies something as "data". > > It is an interesting question what terminology we should use in the > documentation if we need to distinguish them, but I think that is still > wandering away from the original question that was posed. > > All the best, > > Michael > >> >> ? -Fred >> > > > -- > http://www.voidspace.org.uk/ > > May you do good and not evil > May you find forgiveness for yourself and forgive others > May you share freely, never taking more than you give. > -- the sqlite blessing http://www.sqlite.org/different.html > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com > From fuzzyman at voidspace.org.uk Tue Jun 28 17:35:38 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Jun 2011 16:35:38 +0100 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> <20110628115136.248822505A3@webabinitio.net> Message-ID: <4E09F4CA.3060002@voidspace.org.uk> On 28/06/2011 16:23, Terry Reedy wrote: > On 6/28/2011 7:51 AM, R. David Murray wrote: > >> Also, instances can have methods as instance attributes. > > Functions that are instance attributes do not act like methods > (instance.func() does not automagically turn instance in the first arg > of func) and have never, to my knowledge, been called methods. In > Python 2, they are not wrapped as methods whereas functions attached > to classes are. > > So-called 'staticmethods' are not really methods either, but are class > function attributes that are just functions and not treated as > methods. The decorator that negates normal method treatment > could/should have been called 'non_method'. > > Using 'function' is its generic 'callable' sense ... > > Method: a class function attribute that in its intended and normal use > automagically turns the object it is called on into its first arg. > 'Method' is a useful and needed subcategory of class attribute > precisely because of this behavior. > > Instance method: a class function attribute that is (normally) called > on instances of the class or subclasses. > So what is the difference between "Instance method" and "Method" above? Is it just that "Method" is broader and includes class methods and bound methods? If anyone said "instance method" to me I would assume they meant bound method. (A normal method fetched from an instance.) All the best, Michael > Class method: a class function attribute that is (normally) called on > the class or subclasses. > > Bound method: a method that has already has the first-arg object > bundled with it, so that it can be used as a normal (partial or > curried) function. > > Except for 'classmethod', which was added later, these have been the > meanings as I understood them since at least Py 1.4, 15 years ago, and > they are exactly what one needs to know to use Python. Any object can > be an attribute. However, function attributes of classes normally get > special 'method' treatment which alters the meaning of syntax. > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From ericsnowcurrently at gmail.com Tue Jun 28 17:42:46 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 28 Jun 2011 09:42:46 -0600 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E09D522.7000009@pearwood.info> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> <20110628115136.248822505A3@webabinitio.net> <4E09C314.3090809@voidspace.org.uk> <4E09D522.7000009@pearwood.info> Message-ID: On Tue, Jun 28, 2011 at 7:20 AM, Steven D'Aprano wrote: > Michael Foord wrote: > >> What do you mean by "instances can have methods as instance attributes"? >> Once you attach a bound method directly to an instance it becomes a slightly >> different beast I think. (On top of which that is pretty rare behaviour.) > >>>> class C: > ... ? ? def method(self, x): > ... ? ? ? ? ? ? return x+1 > ... >>>> c = C() >>>> c.method = types.MethodType(lambda self, x: x+101, c) >>>> c.method(1) > 102 > > I don't know how rare it is, but it's a useful trick for customising the > behaviour of instances. > > > As I see it, there are three dichotomies we sometimes need to make: > > > (1) Instance attributes vs class (shared) attributes. > > Broadly speaking, whether the attribute is in instance.__dict__ or > type(instance).__dict__. > > (2) Computed vs non-computed attributes. > > Attributes which are computed by __getattr__ or via the descriptor protocol > (which includes properties) are all computed attributes; everything else is > non-computed. > > (3) Method attributes (methods) vs non-method/data attributes. > > Broadly speaking, methods are callable, non-method (data) attributes are > not. > For terminology, is it important that data attributes are [usually] not callable, or is it that being callable is not relevant to their use as attributes of the class/instance? The same for "methods". We have precedent for where the the terminology represents an expectation rather than a firm constraint (__isabstractmethod__). > > The three are orthogonal: e.g. a staticmethod is a method by virtue of being > callable, computed by virtue of being generated by a descriptor, and a class > attribute by virtue of existing in the type __dict__ rather than the > instance __dict__. > > Strictly speaking, (3) is not truly a dichotomy, since functions and methods > are first class-objects in Python. E.g. one may store a function as an > attribute with the intention of using it as data rather than as a method. > But that's a moderately obscure corner case, and in my opinion it's not > worth obscuring the practical distinction between "methods are things you > call, data are not" for the sake of it. Leave the functions-as-data case for > a footnote. > +1 The "three dichotomies" is a great way to look at it. -eric > > > -- > Steven > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com > From p.f.moore at gmail.com Tue Jun 28 17:44:02 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Jun 2011 16:44:02 +0100 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <20110628150641.GA29535@sherwood.local> References: <1309268585.21006.22.camel@marge> <20110628150641.GA29535@sherwood.local> Message-ID: On 28 June 2011 16:06, Steffen Daode Nurpmeso wrote: > @ Paul Moore wrote (2011-06-28 16:46+0200): >> UTF-8 without BOM displays incorrectly in vim(1) > > Stop right now (you're oh so wrong)! ?:-) Sorry. Please add "using the default settings of gvim on Windows". My context throughout was Windows not Unix. Sorry I didn't make that clear. > (By the way: UTF-8 and BOM? Windows uses it, I believe. My tests specifically used files with no BOM, just utf8-encoded text. I made this statement to head off people assuming that UTF8 can be detected in Windows by looking at the first few bytes. > Interesting things i learn on this list. :-) > And i hope in ten years we can laugh about this -> UTF-8 > transition all over the place, 'cause it's simply working.) That would be good... Paul. From tjreedy at udel.edu Tue Jun 28 18:12:24 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 28 Jun 2011 12:12:24 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E09D522.7000009@pearwood.info> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> <20110628115136.248822505A3@webabinitio.net> <4E09C314.3090809@voidspace.org.uk> <4E09D522.7000009@pearwood.info> Message-ID: On 6/28/2011 9:20 AM, Steven D'Aprano wrote: > >>> class C: > ... def method(self, x): > ... return x+1 > ... > >>> c = C() > >>> c.method = types.MethodType(lambda self, x: x+101, c) types.MethodType creates a bound method, not a method. A bound method is a partial or curried function, which is to say, a function. Herw and below, I am ignoring the fact that the example ignores self. My comments would be the same for "lambda self,x: return self.value+x" > >>> c.method(1) The fact that you make the bound function an attribute of the same object to which it is bound is irrelevant to how it *acts* when called. It only affect how you *access* it. You could bind it to anything else, including another instance or a plain name: d = C() d.method = types.MethodType(lambda self, x: x+101, c) d.method(1) m = types.MethodType(lambda self, x: x+101, c) m(1) > 102 > I don't know how rare it is, but it's a useful trick for customising the > behaviour of instances. > > > As I see it, there are three dichotomies we sometimes need to make: > > > (1) Instance attributes vs class (shared) attributes. > (2) Computed vs non-computed attributes. > (3) Method attributes (methods) vs non-method/data attributes. Nicely put so far... > The three are orthogonal: Non-class instance function attributes are not methods in any useful sense. > a staticmethod is a method by virtue of being callable A 'staticmethod' is a non-method class function attribute. The name is unfortunate. 'Static' mean 'does not get the dynamic method treatment'. > Strictly speaking, (3) is not truly a dichotomy, I disagree here. A method is usefully defined as a class function attribute that gets an automagic first arg when accessed and called 'normally'. In other words, the dichotomy is needed because being a method affects the meaning of syntax. > since functions and > methods are first class-objects in Python. E.g. one may store a function > as an attribute with the intention of using it as data rather than as a > method. The fact that one can *also* access a method as a function does not negate the effect on syntax. > "methods are things you call" Way too broad, even for attributes. Callable things are methods when 'a.b(c)' causes a to be the first arg of b. -- Terry Jan Reedy From a.badger at gmail.com Tue Jun 28 18:33:51 2011 From: a.badger at gmail.com (Toshio Kuratomi) Date: Tue, 28 Jun 2011 09:33:51 -0700 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> Message-ID: <20110628163351.GK27252@unaka.lan> On Tue, Jun 28, 2011 at 03:46:12PM +0100, Paul Moore wrote: > On 28 June 2011 14:43, Victor Stinner wrote: > > As discussed before on this list, I propose to set the default encoding > > of open() to UTF-8 in Python 3.3, and add a warning in Python 3.2 if > > open() is called without an explicit encoding and if the locale encoding > > is not UTF-8. Using the warning, you will quickly notice the potential > > problem (using Python 3.2.2 and -Werror) on Windows or by using a > > different locale encoding (.e.g using LANG="C"). > > -1. This will make things harder for simple scripts which are not > intended to be cross-platform. > > I use Windows, and come from the UK, so 99% of my text files are > ASCII. So the majority of my code will be unaffected. But in the > occasional situation where I use a ? sign, I'll get encoding errors, > where currently things will "just work". And the failures will be data > dependent, and hence intermittent (the worst type of problem). I'll > write a quick script, use it once and it'll be fine, then use it later > on some different data and get an error. :-( > I don't think this change would make things "harder". It will just move where the pain occurs. Right now, the failures are intermittent on A) computers other than the one that you're using. or B) intermittent when run under a different user than yourself. Sys admins where I'm at are constantly writing ad hoc scripts in python that break because you stick something in a cron job and the locale settings suddenly become "C" and therefore the script suddenly only deals with ASCII characters. I don't know that Victor's proposed solution is the best (I personally would like it a whole lot more than the current guessing but I never develop on Windows so I can certainly see that your environment can lead to the opposite assumption :-) but something should change here. Issuing a warning like "open used without explicit encoding may lead to errors" if open() is used without an explicit encoding would help a little (at least, people who get errors would then have an inkling that the culprit might be an open() call). If I read Victor's previous email correctly, though, he said this was previously rejected. Another brainstorming solution would be to use different default encodings on different platforms. For instance, for writing files, utf-8 on *nix systems (including macosX) and utf-16 on windows. For reading files, check for a utf-16 BOM, if not present, operate as utf-8. That would seem to address your issue with detection by vim, etc but I'm not sure about getting "?" in your input stream. I don't know where your input is coming from and how Windows equivalent of locale plays into that. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From tjreedy at udel.edu Tue Jun 28 18:34:56 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 28 Jun 2011 12:34:56 -0400 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <4E09E9A9.5020702@voidspace.org.uk> References: <1309268585.21006.22.camel@marge> <4E09DF0C.90200@egenix.com> <4E09E9A9.5020702@voidspace.org.uk> Message-ID: On 6/28/2011 10:48 AM, Michael Foord wrote: > On 28/06/2011 15:36, Terry Reedy wrote: >> S = open('myfile.txt').read() >> now return a text string in both Py2 and Py3 and a subsequent >> 'abc' in S >> works in both. > > Nope, it returns a bytestring in Python 2. Which, in Py2 is a str() object. In both Pythons, .read() in default mode returns an object of type str() and 'abc' is an object of type str() and so expressions involving undecorated string literals and input just work, but would not work if input defaulted to bytes in Py 3. Sorry if I was not clear enough. -- Terry Jan Reedy From nad at acm.org Tue Jun 28 18:37:28 2011 From: nad at acm.org (Ned Deily) Date: Tue, 28 Jun 2011 09:37:28 -0700 Subject: [Python-Dev] svn.python.org confusion Message-ID: Not surprisingly, people are still attempting to browse and/or checkout Python source from svn.python.org. They could be following out-of-date instructions from somewhere or might be expecting the svn repos are mirroring the hg ones. This causes wasted time and frustration for users and for us following up on issues. Could some text be added to the svn browser pages to at least indicate that the repo is no longer being updated with a link to the hg.python.org browser? I'm not sure what to do about the repos themselves if people attempt to do an svn co. Perhaps that should just be disabled totally for python? -- Ned Deily, nad at acm.org From fuzzyman at voidspace.org.uk Tue Jun 28 18:45:44 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Jun 2011 17:45:44 +0100 Subject: [Python-Dev] svn.python.org confusion In-Reply-To: References: Message-ID: <4E0A0538.5000401@voidspace.org.uk> On 28/06/2011 17:37, Ned Deily wrote: > Not surprisingly, people are still attempting to browse and/or checkout > Python source from svn.python.org. They could be following out-of-date > instructions from somewhere or might be expecting the svn repos are > mirroring the hg ones. This causes wasted time and frustration for > users and for us following up on issues. > > Could some text be added to the svn browser pages to at least indicate > that the repo is no longer being updated with a link to the > hg.python.org browser? I'm not sure what to do about the repos > themselves if people attempt to do an svn co. Perhaps that should just > be disabled totally for python? > For what it's worth we've had a couple of emails about this to webmaster at python.org (in particular because the viewc svn browser breaks on some latin-1 in source files). All the best, Michael Foord -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From janssen at parc.com Tue Jun 28 18:49:13 2011 From: janssen at parc.com (Bill Janssen) Date: Tue, 28 Jun 2011 09:49:13 PDT Subject: [Python-Dev] Snow Leopard buildbot failing again... In-Reply-To: <20110628102431.2fd841d4@pitrou.net> References: <26766.1309194073@parc.com> <20110628102431.2fd841d4@pitrou.net> Message-ID: <43935.1309279753@parc.com> Antoine Pitrou wrote: > On Mon, 27 Jun 2011 10:01:13 PDT > Bill Janssen wrote: > > I see that parc-snowleopard-1 went down again. I've done a software > > update, rebooted, and installed the latest buildslave, 0.8.4. I can > > ping dinsdale.python.org successfully from the machine. However, when I > > start the buildslave, I get this: > > Well, doesn't the slave work correctly now? Or is there a problem? It does seem to have connected, but the error message from buildslave is new. Let's see if it goes into the zombie state again running 0.8.4. The Tiger buildbot (an old dual-1GHz G4 running buildbot-slave 0.8.1p1) is the only one which seems to run reliably :-). Bill From fuzzyman at voidspace.org.uk Tue Jun 28 18:50:29 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Jun 2011 17:50:29 +0100 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> <4E09DF0C.90200@egenix.com> <4E09E9A9.5020702@voidspace.org.uk> Message-ID: <4E0A0655.40207@voidspace.org.uk> On 28/06/2011 17:34, Terry Reedy wrote: > On 6/28/2011 10:48 AM, Michael Foord wrote: >> On 28/06/2011 15:36, Terry Reedy wrote: > >>> S = open('myfile.txt').read() >>> now return a text string in both Py2 and Py3 and a subsequent >>> 'abc' in S >>> works in both. >> >> Nope, it returns a bytestring in Python 2. > > Which, in Py2 is a str() object. Yes, but not a "text string". The equivalent of the Python 2 str in Python 3 is bytes. Irrelevant discussion anyway. > In both Pythons, .read() in default mode returns an object of type > str() and 'abc' is an object of type str() and so expressions > involving undecorated string literals and input just work, but would > not work if input defaulted to bytes in Py 3. Sorry if I was not clear > enough. > Well, I think you're both right. Both semantics break some assumption or other. All the best, Michael -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From janssen at parc.com Tue Jun 28 18:52:19 2011 From: janssen at parc.com (Bill Janssen) Date: Tue, 28 Jun 2011 09:52:19 PDT Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <4E09DF0C.90200@egenix.com> References: <1309268585.21006.22.camel@marge> <4E09DF0C.90200@egenix.com> Message-ID: <44221.1309279939@parc.com> M.-A. Lemburg wrote: > How about a more radical change: have open() in Py3 default to > opening the file in binary mode, if no encoding is given (even > if the mode doesn't include 'b') ? +1. > That'll make it compatible to the Py2 world again and avoid > all the encoding guessing. Yep. Bill From janssen at parc.com Tue Jun 28 18:53:26 2011 From: janssen at parc.com (Bill Janssen) Date: Tue, 28 Jun 2011 09:53:26 PDT Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> <4E09DF0C.90200@egenix.com> Message-ID: <44243.1309280006@parc.com> Terry Reedy wrote: > > Making such default encodings depend on the locale has already > > failed to work when we first introduced a default encoding in > > Py2, so I don't understand why we are repeating the same > > mistake again in Py3 (only in a different area). > > I do not remember any proposed change during the Py3 design discussions. I certainly proposed it, more than once. Bill From tjreedy at udel.edu Tue Jun 28 19:06:44 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 28 Jun 2011 13:06:44 -0400 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> Message-ID: On 6/28/2011 10:46 AM, Paul Moore wrote: > I use Windows, and come from the UK, so 99% of my text files are > ASCII. So the majority of my code will be unaffected. But in the > occasional situation where I use a ? sign, I'll get encoding errors, I do not understand this. With utf-8 you would never get a string encoding error. > where currently things will "just work". As long as you only use the machine-dependent restricted character set. > And the failures will be data dependent, and hence intermittent > (the worst type of problem). That is the situation now, with platform/machine dependencies added in. Some people share code with other machines, even locally. > So, in effect, you propose making the default favour writing > multiplatform portable code at the expense of quick and dirty scripts? Let us frame it another way. Should Python installations be compatible with other Python installations, or with the other apps on the same machine? Part of the purpose of Python is to cover up platform differences, to the extent possible (and perhaps sensible -- there is the argument). This was part of the purpose of writing our own io module instead of using the compiler stdlib. The evolution of floating point math has gone in the same direction. For instance, float now expects uniform platform-independent Python-dependent names for infinity and nan instead of compiler-dependent names. As for practicality. Notepad++ on Windows offers ANSI, utf-8 (w,w/o BOM), utf-16 (big/little endian). I believe that ODF documents are utf-8 encoded xml (compressed or not). My original claim for this proposal was/is that even Windows apps are moving to uft-8 and that someday making that the default for Python everywhere will be the obvious and sensible thing. -- Terry Jan Reedy From janssen at parc.com Tue Jun 28 19:08:14 2011 From: janssen at parc.com (Bill Janssen) Date: Tue, 28 Jun 2011 10:08:14 PDT Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: <4E09D9FA.7060308@voidspace.org.uk> References: <20110624145516.84DA32506EA@webabinitio.net> <4E09D9FA.7060308@voidspace.org.uk> Message-ID: <44642.1309280894@parc.com> Michael Foord wrote: > The new regex library has some great improvements: > > http://bugs.python.org/issue2636 > > It also has users and committed maintainers, so I hope we can bring it > into 3.3. It wasn't easy to tell from skimming the change notes that > Unicode character classes are amongst the new features. Is that the > case? According to http://effbot.org/zone/unicode-objects.htm (from 2004), the existing re module already supports Unicode character classes, so the regex module will as well. But the support has been updated, according to the change notes. Bill From fuzzyman at voidspace.org.uk Tue Jun 28 19:22:38 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Jun 2011 18:22:38 +0100 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> Message-ID: <4E0A0DDE.5070403@voidspace.org.uk> On 28/06/2011 18:06, Terry Reedy wrote: > On 6/28/2011 10:46 AM, Paul Moore wrote: > >> I use Windows, and come from the UK, so 99% of my text files are >> ASCII. So the majority of my code will be unaffected. But in the >> occasional situation where I use a ? sign, I'll get encoding errors, > > I do not understand this. With utf-8 you would never get a string > encoding error. > I assumed he meant that files written out as utf-8 by python would then be read in using the platform encoding (i.e. not utf-8 on Windows) by the other applications he is inter-operating with. The error would not be in Python but in those applications. >> where currently things will "just work". > > As long as you only use the machine-dependent restricted character set. > Which is the situation he is describing. You do go into those details below, and which choice is "correct" depends on which trade-off you want to make. For the sake of backwards compatibility we are probably stuck with the current trade-off however - unless we deprecate using open(...) without an explicit encoding. All the best, Michael > > And the failures will be data dependent, and hence intermittent > > (the worst type of problem). > > That is the situation now, with platform/machine dependencies added in. > Some people share code with other machines, even locally. > >> So, in effect, you propose making the default favour writing >> multiplatform portable code at the expense of quick and dirty scripts? > > Let us frame it another way. Should Python installations be compatible > with other Python installations, or with the other apps on the same > machine? Part of the purpose of Python is to cover up platform > differences, to the extent possible (and perhaps sensible -- there is > the argument). This was part of the purpose of writing our own io > module instead of using the compiler stdlib. The evolution of floating > point math has gone in the same direction. For instance, float now > expects uniform platform-independent Python-dependent names for > infinity and nan instead of compiler-dependent names. > > As for practicality. Notepad++ on Windows offers ANSI, utf-8 (w,w/o > BOM), utf-16 (big/little endian). I believe that ODF documents are > utf-8 encoded xml (compressed or not). My original claim for this > proposal was/is that even Windows apps are moving to uft-8 and that > someday making that the default for Python everywhere will be the > obvious and sensible thing. > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From fuzzyman at voidspace.org.uk Tue Jun 28 19:23:42 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 28 Jun 2011 18:23:42 +0100 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: <44642.1309280894@parc.com> References: <20110624145516.84DA32506EA@webabinitio.net> <4E09D9FA.7060308@voidspace.org.uk> <44642.1309280894@parc.com> Message-ID: <4E0A0E1E.2040207@voidspace.org.uk> On 28/06/2011 18:08, Bill Janssen wrote: > Michael Foord wrote: > >> The new regex library has some great improvements: >> >> http://bugs.python.org/issue2636 >> >> It also has users and committed maintainers, so I hope we can bring it >> into 3.3. It wasn't easy to tell from skimming the change notes that >> Unicode character classes are amongst the new features. Is that the >> case? > According to http://effbot.org/zone/unicode-objects.htm (from 2004), the > existing re module already supports Unicode character classes, so the > regex module will as well. But the support has been updated, according > to the change notes. Thanks. Support for Unicode character classes was one of the improvements needed in the re module reported from the language summit - so I wonder if the changes in regex are sufficient. Michael > Bill -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From janssen at parc.com Tue Jun 28 19:55:45 2011 From: janssen at parc.com (Bill Janssen) Date: Tue, 28 Jun 2011 10:55:45 PDT Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: <4E0A0E1E.2040207@voidspace.org.uk> References: <20110624145516.84DA32506EA@webabinitio.net> <4E09D9FA.7060308@voidspace.org.uk> <44642.1309280894@parc.com> <4E0A0E1E.2040207@voidspace.org.uk> Message-ID: <46579.1309283745@parc.com> Michael Foord wrote: > Thanks. Support for Unicode character classes was one of the > improvements needed in the re module reported from the language summit > - > so I wonder if the changes in regex are sufficient. I guess it depends on what you're asking -- what does "support for Unicode character classes" mean? Apparently something different from what it meant in 2004 :-). Bill From solipsis at pitrou.net Tue Jun 28 19:56:33 2011 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 Jun 2011 19:56:33 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? References: <1309268585.21006.22.camel@marge> Message-ID: <20110628195633.3b474c45@pitrou.net> On Tue, 28 Jun 2011 13:06:44 -0400 Terry Reedy wrote: > > As for practicality. Notepad++ on Windows offers ANSI, utf-8 (w,w/o > BOM), utf-16 (big/little endian). Well, that's *one* application. We would need much more data than that. > I believe that ODF documents are utf-8 > encoded xml (compressed or not). XML doesn't matter for this discussion, since it explicitly declares the encoding. What we are talking about is "raw" text files that don't have an encoding declaration and for which the data format doesn't specify any default encoding (which also rules out Python source code, by the way). > My original claim for this proposal > was/is that even Windows apps are moving to uft-8 > and that someday > making that the default for Python everywhere will be the obvious and > sensible thing. True, but that may be 5 or 10 years from now. Regards Antoine. From tjreedy at udel.edu Tue Jun 28 19:58:54 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 28 Jun 2011 13:58:54 -0400 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: <4E09F4CA.3060002@voidspace.org.uk> References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> <20110628115136.248822505A3@webabinitio.net> <4E09F4CA.3060002@voidspace.org.uk> Message-ID: <4E0A165E.5080808@udel.edu> On 6/28/2011 11:35 AM, Michael Foord wrote: > On 28/06/2011 16:23, Terry Reedy wrote: >> So-called 'staticmethods' are not really methods either, but are class >> function attributes that are just functions and not treated as >> methods. The decorator that negates normal method treatment >> could/should have been called 'non_method'. >> >> Using 'function' is its generic 'callable' sense ... >> >> Method: a class function attribute that in its intended and normal use >> automagically turns the object it is called on into its first arg. >> 'Method' is a useful and needed subcategory of class attribute >> precisely because of this behavior. >> >> Instance method: a class function attribute that is (normally) called >> on instances of the class or subclasses. Before 2.2, these were the only methods. > So what is the difference between "Instance method" and "Method" above? > Is it just that "Method" is broader and includes class methods Since 2.2, yes. The current glossary entry starts "method A function which is defined inside a class body." This includes 'staticmethods', but as I have said, I think that is a mistake. Static methods are functions without special method treatment. A class staticmethod function act the same as any other function. Also, 'defined inside' is not necessary. I would change the above to "A function that get bound to a instance or class when called as an attribute of the instance or class. Methods are usually defined inside a class body." > and bound methods? The result of accessing an instance or class method via an instance or class. Accessing a static method does not create a bound method. Bound methods are usually anonymous and ephemeral, being used for one call and then deleted. > If anyone said "instance method" to me I would assume they meant bound > method. (A normal method fetched from an instance.) Instance methods are the 'permanent' class function attributes, not the ephemeral object that implements a.b(c). Bound methods would be an implementation detail, except that a.b has to evaluate to something and saving bound methods is quite handy when calling a method or methods repeatedly on the same instance. Functools.partial is a generalization of bound methods, which were Python's first (special-case) implementation of the partial function idea. Leaving out implementation details, if b is a function attribute of type(a), 'a.b' is an abbreviated way of writing 'functools.partial(type(a).b,a)' (and yes, I have tested an example of this). >> Class method: a class function attribute that is (normally) called on >> the class or subclasses. >> >> Bound method: a method that has already has the first-arg object >> bundled with it, so that it can be used as a normal (partial or >> curried) function. --- Terry Jan Reedy From g.brandl at gmx.net Tue Jun 28 20:13:42 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 28 Jun 2011 20:13:42 +0200 Subject: [Python-Dev] EuroPython Language Summit report In-Reply-To: <1309270018.21006.26.camel@marge> References: <20110624145516.84DA32506EA@webabinitio.net> <4E09D9FA.7060308@voidspace.org.uk> <1309270018.21006.26.camel@marge> Message-ID: On 28.06.2011 16:06, Victor Stinner wrote: > Le mardi 28 juin 2011 ? 14:41 +0100, Michael Foord a ?crit : >> On 24/06/2011 15:55, R. David Murray wrote: >> > On Fri, 24 Jun 2011 10:52:40 +0200, Mark Dickinson wrote: >> >> EuroPython 2011 Language Summit >> >> =============================== >> > [...] >> >> Unicode character classes is a particular need). [Subtopic: what needs >> >> to be done to get the new regex module into Python? Should it replace >> >> the existing module? What about backwards compatibility issues?] >> > I'm pretty sure regex has backward compatibility as a goal for just this >> > reason (so it can replace the current module). >> > >> The new regex library has some great improvements: >> >> http://bugs.python.org/issue2636 > > This issue is open since April 2008, has also the longest list of > attached files, and has a very long history. What is the status of the > issue? I see that there is now a third party project on: > > http://code.google.com/p/mrab-regex-hg/ This should be the same module as in the issue (and thankfully, because code management of such a big project does not belong exclusively in a tracker issue). "mrab" stands for Matthew Barnett, who is the author of regex. > -- > > There is also the re2 library from Google and especially this project: > > http://pypi.python.org/pypi/re2/ > > "pyre2 is a Python extension that wraps Google's RE2 regular expression > library. > > This version of pyre2 is similar to the one you'd find at facebook's > github repository except that the stated goal of *this version is to be > a drop-in replacement for the re module*.)" Well, while it can be called drop-in, it is hardly a good replacement: """ That being said, there are features of the re module that this module may never have. For example, RE2 does not handle lookahead assertions ((?=...)). """ It falls back to old re in these cases. Georg From g.brandl at gmx.net Tue Jun 28 20:18:00 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 28 Jun 2011 20:18:00 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> Message-ID: On 28.06.2011 19:06, Terry Reedy wrote: > On 6/28/2011 10:46 AM, Paul Moore wrote: > >> I use Windows, and come from the UK, so 99% of my text files are >> ASCII. So the majority of my code will be unaffected. But in the >> occasional situation where I use a ? sign, I'll get encoding errors, > > I do not understand this. With utf-8 you would never get a string > encoding error. Yes, but you'll get plenty of *decoding* errors. Georg From ethan at stoneleaf.us Tue Jun 28 20:55:02 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 28 Jun 2011 11:55:02 -0700 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <4E0A0655.40207@voidspace.org.uk> References: <1309268585.21006.22.camel@marge> <4E09DF0C.90200@egenix.com> <4E09E9A9.5020702@voidspace.org.uk> <4E0A0655.40207@voidspace.org.uk> Message-ID: <4E0A2386.1090502@stoneleaf.us> Michael Foord wrote: > On 28/06/2011 17:34, Terry Reedy wrote: >> On 6/28/2011 10:48 AM, Michael Foord wrote: >>> On 28/06/2011 15:36, Terry Reedy wrote: >> >>>> S = open('myfile.txt').read() >>>> now return a text string in both Py2 and Py3 and a subsequent >>>> 'abc' in S >>>> works in both. >>> >>> Nope, it returns a bytestring in Python 2. >> >> Which, in Py2 is a str() object. > > Yes, but not a "text string". The equivalent of the Python 2 str in > Python 3 is bytes. Irrelevant discussion anyway. Irrelevant to the OP, yes, but a Python 2 string *is not* the same as Python 3 bytes. If you don't believe me fire up your Python 3 shell and try b'xyz'[1] == 'y'. ~Ethan~ From ethan at stoneleaf.us Tue Jun 28 21:06:01 2011 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 28 Jun 2011 12:06:01 -0700 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <4E0A2386.1090502@stoneleaf.us> References: <1309268585.21006.22.camel@marge> <4E09DF0C.90200@egenix.com> <4E09E9A9.5020702@voidspace.org.uk> <4E0A0655.40207@voidspace.org.uk> <4E0A2386.1090502@stoneleaf.us> Message-ID: <4E0A2619.8020605@stoneleaf.us> Ethan Furman wrote: > Michael Foord wrote: >> On 28/06/2011 17:34, Terry Reedy wrote: >>> On 6/28/2011 10:48 AM, Michael Foord wrote: >>>> On 28/06/2011 15:36, Terry Reedy wrote: >>> >>>>> S = open('myfile.txt').read() >>>>> now return a text string in both Py2 and Py3 and a subsequent >>>>> 'abc' in S >>>>> works in both. >>>> >>>> Nope, it returns a bytestring in Python 2. >>> >>> Which, in Py2 is a str() object. >> >> Yes, but not a "text string". The equivalent of the Python 2 str in >> Python 3 is bytes. Irrelevant discussion anyway. > > Irrelevant to the OP, yes, but a Python 2 string *is not* the same as > Python 3 bytes. If you don't believe me fire up your Python 3 shell and > try b'xyz'[1] == 'y'. er, make that b'xyz'[1] == b'y' :( From brett at python.org Tue Jun 28 21:05:39 2011 From: brett at python.org (Brett Cannon) Date: Tue, 28 Jun 2011 12:05:39 -0700 Subject: [Python-Dev] Issue10403 - using 'attributes' instead of members in documentation In-Reply-To: References: <20110626185242.GB2710@mathmagic> <20110627102428.6fc1955b@msiwind> <20110627140826.877032505A3@webabinitio.net> <4E08917E.3020509@btinternet.com> <87r56en1rt.fsf@benfinney.id.au> <87mxh2mtq8.fsf@benfinney.id.au> <4E09B2EF.9030304@voidspace.org.uk> <4E09CF88.4090307@voidspace.org.uk> Message-ID: On Tue, Jun 28, 2011 at 06:28, Nick Coghlan wrote: > On Tue, Jun 28, 2011 at 10:56 PM, Michael Foord > wrote: > > I don't think "data attributes" is clear or precise. Is a property a data > > attribute (well it depends how it is implemented and what it does), is a > > staticmethod a data attribute (no - but then Tres' question - it isn't a > > normal method either so if you define data attribute to mean "all non > method > > attributes" then its potentially an open question). > > "callable attributes" and "data attributes" are purely about the API > exposed by the object in question. If we're going for object model > neutral terminology, that's the only view that makes sense. They're > descriptions about how attributes are intended to be *used* that are > completely silent on the question of how they're *implemented*. So > staticmethod would fall into the first group, while property would > fall into the latter. > These two terms also have the benefit of being understandable by non Python programmers. Since we can't rely on people read the glossary or any other specific doc before reading the stdlib docs we need to make sure that the terminology that we use can be understood by newbies. Thus I like data and callable attributes since it makes sense even if you come from another programming language since you can easily reason it out. And now I am done adding my graffiti to the shed. -Brett > > >From an implementation point of view, you carve up the world > differently, so it makes sense to worry about class attributes, > instance attributes, dynamic attributes, etc. (the class vs instance > distinction can also matter to some degree from the usage point of > view, since it affects the scope of any mutable attributes, and the > static vs dynamic distinction can also matter, especially for > introspection purposes). > > This goes back to the original point about all of this being highly > context dependent - how you carve up the set of all attributes is > going to change based on what you're trying to explain (e.g. the > distinction between enforced 'data' descriptors, descriptors that > allow shadowing in the instance dict, class attributes that aren't > descriptors at all, instance attributes and dynamic attributes > retrieved via __getattr__ is another way of dividing them) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Jun 28 22:11:50 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 Jun 2011 21:11:50 +0100 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <4E0A0DDE.5070403@voidspace.org.uk> References: <1309268585.21006.22.camel@marge> <4E0A0DDE.5070403@voidspace.org.uk> Message-ID: On 28 June 2011 18:22, Michael Foord wrote: > On 28/06/2011 18:06, Terry Reedy wrote: >> >> On 6/28/2011 10:46 AM, Paul Moore wrote: >> >>> I use Windows, and come from the UK, so 99% of my text files are >>> ASCII. So the majority of my code will be unaffected. But in the >>> occasional situation where I use a ? sign, I'll get encoding errors, >> >> I do not understand this. With utf-8 you would never get a string encoding >> error. >> > > I assumed he meant that files written out as utf-8 by python would then be > read in using the platform encoding (i.e. not utf-8 on Windows) by the other > applications he is inter-operating with. The error would not be in Python > but in those applications. That is correct. Or files written out (as platform encoding) by other applications, will later be read in as UTF-8 by Python, and be seen as incorrect characters, or worse raise decoding errors. (Sorry, in my original post I said "encoding" where I meant "decoding"...) I'm not interested in allocating "blame" for the "error". I'm not convinced that it *is* an error, merely 2 programs with incompatible assumptions. What I'm saying is that compatibility between various programs on a single machine can, in some circumstances, be more important than compatibility between (the same, or different) programs running on different machines or OSes. And that I, personally, am in that situation. >>> where currently things will "just work". >> >> As long as you only use the machine-dependent restricted character set. >> > > Which is the situation he is describing. You do go into those details below, > and which choice is "correct" depends on which trade-off you want to make. > > For the sake of backwards compatibility we are probably stuck with the > current trade-off however - unless we deprecate using open(...) without an > explicit encoding. Backward compatibility is another relevant point. But other than that, it's a design trade-off, agreed. All I'm saying is that I see the current situation (which is in favour of quick script use and beginner friendly at the expense of conceptual correctness and forcing the user to think about his choices) as being preferable (and arguably more "Pythonic", in the sense that I see it as a case of "practicality beats purity" - although it's easy to argue that "in the face of ambiguity..." also applies here :-)) Paul. From victor.stinner at haypocalc.com Tue Jun 28 22:51:38 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 28 Jun 2011 22:51:38 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <4E09DF0C.90200@egenix.com> References: <1309268585.21006.22.camel@marge> <4E09DF0C.90200@egenix.com> Message-ID: <1309294298.23387.12.camel@marge> Le mardi 28 juin 2011 ? 16:02 +0200, M.-A. Lemburg a ?crit : > How about a more radical change: have open() in Py3 default to > opening the file in binary mode, if no encoding is given (even > if the mode doesn't include 'b') ? I tried your suggested change: Python doesn't start. sysconfig uses the implicit locale encoding to read sysconfig.cfg, the Makefile and pyconfig.h. I think that it is correct to use the locale encoding for Makefile and pyconfig.h, but maybe not for sysconfig.cfg. Python require more changes just to run "make". I was able to run "make" by using encoding='utf-8' in various functions (of distutils and setup.py). I didn't try the test suite, I expect too many failures. -- Then I tried my suggestion (use "utf-8" by default): Python starts correctly, I can build it (run "make") and... the full test suite pass without any change. (I'm testing on Linux, my locale encoding is UTF-8.) Victor From g.brandl at gmx.net Tue Jun 28 23:42:11 2011 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 28 Jun 2011 21:42:11 +0000 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> Message-ID: On 28.06.2011 14:24, Terry Reedy wrote: >> As discussed before on this list, I propose to set the default encoding >> of open() to UTF-8 in Python 3.3, and add a warning in Python 3.2 if >> open() is called without an explicit encoding and if the locale encoding >> is not UTF-8. Using the warning, you will quickly notice the potential >> problem (using Python 3.2.2 and -Werror) on Windows or by using a >> different locale encoding (.e.g using LANG="C"). [...] >> Should I convert this email into a PEP, or is it not required? > > I think a PEP is needed. Absolutely. And I hope the hypothetical PEP would be rejected in this form. We need to stop making incompatible changes to Python 3. We had the chance and took it to break all kinds of stuff, some of it gratuitous, with 3.0 and even 3.1. Now the users need a period of compatibility and stability (just like the language moratorium provided for one aspect of Python). Think about porting: Python 3 uptake is not ahead of time (I don't want to say it's too slow, but it's certainly not too fast.) For the sake of porters' sanity, 3.x should not be a moving target. New features are not so much of a problem, but incompatibilities like this one certainly are. At the very least, a change like this needs a transitional strategy, like it has been used during the 2.x series: * In 3.3, accept "locale" as the encoding parameter, meaning the locale encoding * In 3.4, warn if encoding isn't given and the locale encoding isn't UTF-8 * In 3.5, change default encoding to UTF-8 It might be just enough to stress in the documentation that usage of the encoding parameter is recommended for cross-platform consistency. cheers, Georg From victor.stinner at haypocalc.com Wed Jun 29 00:00:45 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 29 Jun 2011 00:00:45 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> Message-ID: <1309298445.23387.33.camel@marge> > > I don't think that Windows developer even know that they are writing > > files into the ANSI code page. MSDN documentation of > > WideCharToMultiByte() warns developer that the ANSI code page is not > > portable, even accross Windows computers: > > Probably true. But for many uses they also don't care. If you're > writing something solely for a one-off job on your own PC, the ANSI > code page is fine, and provides interoperability with other programs > on your PC, which is really what you care about. (UTF-8 without BOM > displays incorrectly in Vim, wordpad, and powershell get-content. I tried to open a text file encoded to UTF-8 (without BOM) on Windows Seven. The default application displays it correctly, it's the well known builtin notepad program. gvim is unable to detect the encoding, it reads the file using the ANSI code page (WTF? UTF-8 is correctly detected on Linux!?). Wordpad reads the file using the ANSI code page, it is unable to detect the UTF-8 encoding. The "type" command in a MS-Dos shell (cmd.exe) dosen't display the UTF-8 correctly, but a file encoded to ANSI code is also display incorrectly. I suppose that the problem is that the terminal uses the OEM code page, not the ANSI code page. Visual C++ 2008 detects the UTF-8 encoding. I don't have other applications to test on my Windows Seven. I agree that UTF-8 is not well supported by "standard" Windows applications. I would at least expect that Wordpad and gvim are able to detect the UTF-8 encoding. > MBCS works fine in all of these. It also displays incorrectly in CMD type, > but in a less familiar form than the incorrect display mbcs produces, > for what that's worth...) True, the encoding of a text file encoded to the ANSI code page is correctly detected by all applications (except "type" in a shell, it should be the OEM/ANSI code page conflict). > IMHO, you missed another option - open() does not need improving, the > current behaviour is better than any of the 3 options noted. My original need is to detect that my program will behave differently on Linux and Windows, because open() uses the implicit locale encoding. Antoine suggested me to monkeypatch __builtins__.open to do that. Victor From tseaver at palladion.com Wed Jun 29 00:29:16 2011 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 28 Jun 2011 18:29:16 -0400 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <44221.1309279939@parc.com> References: <1309268585.21006.22.camel@marge> <4E09DF0C.90200@egenix.com> <44221.1309279939@parc.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/28/2011 12:52 PM, Bill Janssen wrote: > M.-A. Lemburg wrote: > >> How about a more radical change: have open() in Py3 default to >> opening the file in binary mode, if no encoding is given (even >> if the mode doesn't include 'b') ? > > +1. > >> That'll make it compatible to the Py2 world again and avoid >> all the encoding guessing. > > Yep. +1 from me, as well: "in the face of ambiguity, refuse the temptation sto guess." Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4KVbwACgkQ+gerLs4ltQ4gPACgoWGjAhmOg9IGQgMht2KsZYn5 mKUAnjcLP6BGCVFSudm1v77ZHere3VHw =4SPu -----END PGP SIGNATURE----- From victor.stinner at haypocalc.com Wed Jun 29 00:34:54 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 29 Jun 2011 00:34:54 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <20110628163351.GK27252@unaka.lan> References: <1309268585.21006.22.camel@marge> <20110628163351.GK27252@unaka.lan> Message-ID: <1309300494.23387.49.camel@marge> Le mardi 28 juin 2011 ? 09:33 -0700, Toshio Kuratomi a ?crit : > Issuing a warning like "open used without explicit encoding may lead > to errors" if open() is used without an explicit encoding would help > a little (at least, people who get errors would then have an inkling > that the culprit might be an open() call). If I read Victor's previous > email correctly, though, he said this was previously rejected. Oh sorry, I used the wrong word. I listed two other possible solutions, but there were not really rejetected. I just thaugh that changing the default encoding to UTF-8 was the most well accepted idea. If I mix different suggestions together: another solution is to emit a warning if the encoding is not specified (not only if the locale encoding is different than UTF-8). Using encoding="locale" would make it quiet. It would be annoying if the warning would be displayed by default ("This will make things harder for simple scripts which are not intended to be cross-platform." wrote Paul Moore). It only makes sense if we use the same policy than unclosed files/sockets: hidden by default, but it can be configured using command line options (-Werror, yeah!). > Another brainstorming solution would be to use different default encodings on > different platforms. For instance, for writing files, utf-8 on *nix systems > (including macosX) and utf-16 on windows. I don't think that UTF-16 is a better choice than UTF-8 on Windows :-( > For reading files, check for a utf-16 BOM, if not present, operate as utf-8. Oh oh. I already suggested to read the BOM. See http://bugs.python.org/issue7651 and read the email thread "Improve open() to support reading file starting with an unicode BOM" http://mail.python.org/pipermail/python-dev/2010-January/097102.html Reading the BOM is a can of worm, everybody expects something different. I forgot the idea of changing that. Victor From tjreedy at udel.edu Wed Jun 29 02:22:59 2011 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 28 Jun 2011 20:22:59 -0400 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> Message-ID: On 6/28/2011 5:42 PM, Georg Brandl wrote: > At the very least, a change like this needs a transitional strategy, like > it has been used during the 2.x series: > > * In 3.3, accept "locale" as the encoding parameter, meaning the locale encoding > * In 3.4, warn if encoding isn't given and the locale encoding isn't UTF-8 > * In 3.5, change default encoding to UTF-8 3.5 should be 4-5 years off. I actually would not propose anything faster than that. -- Terry Jan Reedy From rndblnch at gmail.com Wed Jun 29 02:34:59 2011 From: rndblnch at gmail.com (renaud) Date: Wed, 29 Jun 2011 00:34:59 +0000 (UTC) Subject: [Python-Dev] PEP 380 acceptance (was Re: EuroPython Language Summit report) References: Message-ID: Nick Coghlan gmail.com> writes: > > On Tue, Jun 28, 2011 at 1:09 AM, renaud gmail.com> wrote: > > Nick Coghlan gmail.com> writes: > > > >> I hit a snag with this. The real tests of the PEP 380 functionality > >> aren't currently part of the patch - they're a big set of "golden > >> output" tests in the zipfile hosted on Greg's site. Those need to be > >> refactored into proper unittest or doctest based additions to the test > >> suite and incorporated into the patch before I could commit this with > >> a clear conscience. > > > > let me know if i can help. > > It would be good if you could take a look at Greg's original test > suite, consider ways of bringing it into the main regression tests and > then update the patch queue on bitbucket accordingly. > > My preference is for something unittest based, essentially taking the > "golden output" comparisons and turning them into appropriate > self.assert* invocations. > > Given the number of tests Greg has, it will probably make more sense > to do it as a new test subdirectory rather than as a single test file > (although that depends on how many tests are in each file - if there > are only a few, or if they overlap a lot, then having them as separate > test cases within a single file may be a better choice). ok, i've generated a single test_pep380.py using greg tests wrapped to be run by unittest. it's ugly, but it does the job. some things that may not be desirable: - line numbers in the tracebacks do rely on the position of the tests in the file, so any editing before the last test case will probably screw up everything; - the test touches sys.stdout & sys.stderr the whole thing is available here: renaud ps. i had to edit the test24 (Test parser module) expected output to make it match the actual output, i guess that the parser module has changed since greg wrote the tests. From vinay_sajip at yahoo.co.uk Wed Jun 29 03:05:18 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jun 2011 01:05:18 +0000 (UTC) Subject: [Python-Dev] svn.python.org confusion References: Message-ID: Ned Deily acm.org> writes: > Could some text be added to the svn browser pages to at least indicate > that the repo is no longer being updated with a link to the > hg.python.org browser? I'm not sure what to do about the repos > themselves if people attempt to do an svn co. Perhaps that should just > be disabled totally for python? At least some of the stuff in the SVN repo is still needed, AFAICT. I recently did a build of the Python Windows binary installer, and I understand (from the buildbot instructions) that the best source of some of the external dependencies (bz2, Tcl/Tk, openssl etc.) is still the SVN repo. Regards, Vinay Sajip From stefan_ml at behnel.de Wed Jun 29 08:28:41 2011 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 29 Jun 2011 08:28:41 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <1309268585.21006.22.camel@marge> References: <1309268585.21006.22.camel@marge> Message-ID: Victor Stinner, 28.06.2011 15:43: > In Python 2, open() opens the file in binary mode (e.g. file.readline() > returns a byte string). codecs.open() opens the file in binary mode by > default, you have to specify an encoding name to open it in text mode. > > In Python 3, open() opens the file in text mode by default. (It only > opens the binary mode if the file mode contains "b".) The problem is > that open() uses the locale encoding if the encoding is not specified, > which is the case *by default*. The locale encoding can be: > > - UTF-8 on Mac OS X, most Linux distributions > - ISO-8859-1 os some FreeBSD systems > - ANSI code page on Windows, e.g. cp1252 (close to ISO-8859-1) in > Western Europe, cp952 in Japan, ... > - ASCII if the locale is manually set to an empty string or to "C", or > if the environment is empty, or by default on some systems > - something different depending on the system and user configuration... > > If you develop under Mac OS X or Linux, you may have surprises when you > run your program on Windows on the first non-ASCII character. You may > not detect the problem if you only write text in english... until > someone writes the first letter with a diacritic. I agree that this is a *very* common source of problems. People write code that doesn't care about encodings all over the place, and are then surprised when it stops working at some point, either by switching environments or by changing the data. I've seen this in virtually all projects I've ever come to work in[1]. So, eventually, all of that code was either thrown away or got debugged and fixed to use an explicit (and usually configurable) encoding. Consequently, I don't think it's a bad idea to break out of this ever recurring development cycle by either requiring an explicit encoding right from the start, or by making the default encoding platform independent. The opportunity to fix this was very unfortunately missed in Python 3.0. Personally, I don't buy the argument that it's harder to write quick scripts if an explicit encoding is required. Most code that gets written is not just quick scripts, and even those tend to live longer than initially intended. Stefan [1] Admittedly, most of those projects were in Java, where the situation is substantially worse than in Python. Java entirely lacks a way to define a per-module source encoding, and it even lacks a straight forward way to encode/decode a file with an explicit encoding. So, by default, *both* input encodings are platform dependent, whereas in Python it's only the default file encoding, and properly decoding a file is straight forward there. From ncoghlan at gmail.com Wed Jun 29 08:52:37 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jun 2011 16:52:37 +1000 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> Message-ID: On Wed, Jun 29, 2011 at 7:42 AM, Georg Brandl wrote: > On 28.06.2011 14:24, Terry Reedy wrote: >> I think a PEP is needed. > > Absolutely. ?And I hope the hypothetical PEP would be rejected in this form. > > We need to stop making incompatible changes to Python 3. ?We had the chance > and took it to break all kinds of stuff, some of it gratuitous, with 3.0 and > even 3.1. ?Now the users need a period of compatibility and stability (just > like the language moratorium provided for one aspect of Python). +1 to everything Georg said. - nothing can change in 3.2 - perhaps provide a way for an application to switch the default behaviour between 'locale' and 'utf-8' in 3.3 - if this is done, also provide a way to explicitly request the 'locale' behaviour (likely via a locale dependent codec alias) - maybe start thinking about an actual transition to 'utf-8' as default in the 3.4/5 time frame Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From barry at python.org Wed Jun 29 08:53:39 2011 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Jun 2011 07:53:39 +0100 Subject: [Python-Dev] svn.python.org confusion In-Reply-To: References: Message-ID: <20110629075339.4bd05b1e@snowdog> On Jun 29, 2011, at 01:05 AM, Vinay Sajip wrote: >Ned Deily acm.org> writes: > >> Could some text be added to the svn browser pages to at least indicate >> that the repo is no longer being updated with a link to the >> hg.python.org browser? I'm not sure what to do about the repos >> themselves if people attempt to do an svn co. Perhaps that should just >> be disabled totally for python? > >At least some of the stuff in the SVN repo is still needed, AFAICT. I recently >did a build of the Python Windows binary installer, and I understand (from the >buildbot instructions) that the best source of some of the external dependencies >(bz2, Tcl/Tk, openssl etc.) is still the SVN repo. In fact, it's also not entirely true that svn isn't being updated. We're still cutting any 2.5 and 2.6 releases from svn, though we're also porting the changes into hg. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From sandro.tosi at gmail.com Wed Jun 29 08:59:01 2011 From: sandro.tosi at gmail.com (Sandro Tosi) Date: Wed, 29 Jun 2011 08:59:01 +0200 Subject: [Python-Dev] svn.python.org confusion In-Reply-To: References: Message-ID: On Wed, Jun 29, 2011 at 03:05, Vinay Sajip wrote: > Ned Deily acm.org> writes: > >> Could some text be added to the svn browser pages to at least indicate >> that the repo is no longer being updated with a link to the >> hg.python.org browser? ? I'm not sure what to do about the repos >> themselves if people attempt to do an svn co. ?Perhaps that should just >> be disabled totally for python? > > At least some of the stuff in the SVN repo is still needed, AFAICT. I recently > did a build of the Python Windows binary installer, and I understand (from the > buildbot instructions) that the best source of some of the external dependencies > (bz2, Tcl/Tk, openssl etc.) is still the SVN repo. SVN is also used to fetch third-party tools to build documentation, such as: sphinx, docutils, jinja, and pygments. These locations are also advertised on http://docs.python.org/py3k/documenting/building.html (that contains outdated info, but that's another story). Cheers, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From devel at baptiste-carvello.net Wed Jun 29 09:21:19 2011 From: devel at baptiste-carvello.net (Baptiste Carvello) Date: Wed, 29 Jun 2011 09:21:19 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> Message-ID: Le 28/06/2011 16:46, Paul Moore a ?crit : > > -1. This will make things harder for simple scripts which are not > intended to be cross-platform. > +1 to all you said. I frequently use the python command prompt or "python -c" for various quick tasks (mostly on linux). I would hate to replace my ugly, but working >>> open('example.txt').read() with the unnecessarily verbose >>> open('example.txt',encoding='utf-8').read() When using python that way as a "swiss army knife", typing does matter. My preferred solution would be: >> - emit a warning if the encoding argument is not set By the way, I just thought that for real programming, I would love to have a -Wcrossplatform command switch, which would warn for all unportable constructs in one go. That way, I don't have to remember which parts of 'os' wrap posix-only functionnality. Baptiste From barry at python.org Wed Jun 29 09:58:51 2011 From: barry at python.org (Barry Warsaw) Date: Wed, 29 Jun 2011 08:58:51 +0100 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> Message-ID: <20110629085851.3df6f39b@snowdog> On Jun 28, 2011, at 09:42 PM, Georg Brandl wrote: >We need to stop making incompatible changes to Python 3. We had the chance >and took it to break all kinds of stuff, some of it gratuitous, with 3.0 and >even 3.1. Now the users need a period of compatibility and stability (just >like the language moratorium provided for one aspect of Python). +1. I think this is the #1 complaint I hear about Python in talking to users. I think in general we do a pretty good job of maintaining backward compatibility between releases, but not a perfect job, and the places where we miss can be painful for folks. It may be difficult to achieve in all cases, but compatibility should be carefully and thoroughly considered for all changes, especially in the stdlib, and clearly documented where deliberate decisions to break that are adopted. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From mal at egenix.com Wed Jun 29 10:18:59 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 29 Jun 2011 10:18:59 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <1309294298.23387.12.camel@marge> References: <1309268585.21006.22.camel@marge> <4E09DF0C.90200@egenix.com> <1309294298.23387.12.camel@marge> Message-ID: <4E0ADFF3.50100@egenix.com> Victor Stinner wrote: > Le mardi 28 juin 2011 ? 16:02 +0200, M.-A. Lemburg a ?crit : >> How about a more radical change: have open() in Py3 default to >> opening the file in binary mode, if no encoding is given (even >> if the mode doesn't include 'b') ? > > I tried your suggested change: Python doesn't start. No surprise there: it's an incompatible change, but one that undoes a wart introduced in the Py3 transition. Guessing encodings should be avoided whenever possible. > sysconfig uses the implicit locale encoding to read sysconfig.cfg, the > Makefile and pyconfig.h. I think that it is correct to use the locale > encoding for Makefile and pyconfig.h, but maybe not for sysconfig.cfg. > > Python require more changes just to run "make". I was able to run "make" > by using encoding='utf-8' in various functions (of distutils and > setup.py). I didn't try the test suite, I expect too many failures. This demonstrates that Python's stdlib is still not being explicit about the encoding issues. I suppose that things just happen to work because we mostly use ASCII files for configuration and setup. > -- > > Then I tried my suggestion (use "utf-8" by default): Python starts > correctly, I can build it (run "make") and... the full test suite pass > without any change. (I'm testing on Linux, my locale encoding is UTF-8.) I bet it would also with "ascii" in most cases. Which then just means that the Python build process and test suite is not a good test case for choosing a default encoding. Linux is also a poor test candidate for this, since most user setups will use UTF-8 as locale encoding. Windows, OTOH, uses all sorts of code page encodings (usually not UTF-8), so you are likely to hit the real problem cases a lot easier. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 29 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From nad at acm.org Wed Jun 29 10:25:42 2011 From: nad at acm.org (Ned Deily) Date: Wed, 29 Jun 2011 01:25:42 -0700 Subject: [Python-Dev] svn.python.org confusion References: Message-ID: In article , Sandro Tosi wrote: > On Wed, Jun 29, 2011 at 03:05, Vinay Sajip wrote: > > Ned Deily acm.org> writes: > >> Could some text be added to the svn browser pages to at least indicate > >> that the repo is no longer being updated with a link to the > >> hg.python.org browser? ? I'm not sure what to do about the repos > >> themselves if people attempt to do an svn co. ?Perhaps that should just > >> be disabled totally for python? > > At least some of the stuff in the SVN repo is still needed, AFAICT. I > > recently > > did a build of the Python Windows binary installer, and I understand (from > > the > > buildbot instructions) that the best source of some of the external > > dependencies > > (bz2, Tcl/Tk, openssl etc.) is still the SVN repo. > > SVN is also used to fetch third-party tools to build documentation, > such as: sphinx, docutils, jinja, and pygments. These locations are > also advertised on > http://docs.python.org/py3k/documenting/building.html (that contains > outdated info, but that's another story). Yes, by "for python", I meant the active python development "branches" (py3k->default, 3.2, 2.7). -- Ned Deily, nad at acm.org From victor.stinner at haypocalc.com Wed Jun 29 11:50:57 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 29 Jun 2011 11:50:57 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <4E0ADFF3.50100@egenix.com> References: <1309268585.21006.22.camel@marge> <4E09DF0C.90200@egenix.com> <1309294298.23387.12.camel@marge> <4E0ADFF3.50100@egenix.com> Message-ID: <1309341057.8003.9.camel@marge> Le mercredi 29 juin 2011 ? 10:18 +0200, M.-A. Lemburg a ?crit : > Victor Stinner wrote: > > Le mardi 28 juin 2011 ? 16:02 +0200, M.-A. Lemburg a ?crit : > >> How about a more radical change: have open() in Py3 default to > >> opening the file in binary mode, if no encoding is given (even > >> if the mode doesn't include 'b') ? > > > > I tried your suggested change: Python doesn't start. > > No surprise there: it's an incompatible change, but one that undoes > a wart introduced in the Py3 transition. Guessing encodings should > be avoided whenever possible. It means that all programs written for Python 3.0, 3.1, 3.2 will stop working with the new 3.x version (let say 3.3). Users will have to migrate from Python 2 to Python 3.2, and then migration from Python 3.2 to Python 3.3 :-( I would prefer a ResourceWarning (emited if the encoding is not specified), hidden by default: it doesn't break compatibility, and -Werror gives exactly the same behaviour that you expect. > This demonstrates that Python's stdlib is still not being explicit > about the encoding issues. I suppose that things just happen to work > because we mostly use ASCII files for configuration and setup. I did more tests. I found some mistakes and sometimes the binary mode can be used, but most function really expect the locale encoding (it is the correct encoding to read-write files). I agree that it would be to have an explicit encoding="locale", but make it mandatory is a little bit rude. > > Then I tried my suggestion (use "utf-8" by default): Python starts > > correctly, I can build it (run "make") and... the full test suite pass > > without any change. (I'm testing on Linux, my locale encoding is UTF-8.) > > I bet it would also with "ascii" in most cases. Which then just > means that the Python build process and test suite is not a good > test case for choosing a default encoding. > > Linux is also a poor test candidate for this, since most user setups > will use UTF-8 as locale encoding. Windows, OTOH, uses all sorts of > code page encodings (usually not UTF-8), so you are likely to hit > the real problem cases a lot easier. I also ran the test suite on my patched Python (open uses UTF-8 by default) with ASCII locale encoding (LANG=C), the test suite does also pass. Many tests uses non-ASCII characters, some of them are skipped if the locale encoding is unable to encode the tested text. Victor From victor.stinner at haypocalc.com Wed Jun 29 12:01:28 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 29 Jun 2011 12:01:28 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: References: <1309268585.21006.22.camel@marge> Message-ID: <1309341688.8003.19.camel@marge> Le mercredi 29 juin 2011 ? 09:21 +0200, Baptiste Carvello a ?crit : > By the way, I just thought that for real programming, I would love to have a > -Wcrossplatform command switch, which would warn for all unportable constructs > in one go. That way, I don't have to remember which parts of 'os' wrap > posix-only functionnality. When I developed using PHP, error_reporting(E_ALL) was really useful. I would like a same function on Python, but I realized that it is not necessary. Python is already strict *by default*. Python can help developers to warn them about some "corner cases". We have already the -bb option for bytes/str warnings (in Python 3), -Werror to convert warnings to exceptions, and ResourceWarning (since Python 3.2) for unclosed files/sockets. I "just" would like a new warning for an implicit locale encoding, so -Wcrossplatform would be as easy as -Werror. -Werror is like Perl "use strict;" or PHP error_reporting(E_ALL). Use -Wd if you prefer to display warnings instead of raising exceptions. See issue #11455 and #11470 for a new "CompatibilityWarning", it's not "cross platform" but "cross Python" :-) It warns about implementation details like non strings keys in a type dict. Victor From mal at egenix.com Wed Jun 29 12:20:42 2011 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 29 Jun 2011 12:20:42 +0200 Subject: [Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3? In-Reply-To: <1309341057.8003.9.camel@marge> References: <1309268585.21006.22.camel@marge> <4E09DF0C.90200@egenix.com> <1309294298.23387.12.camel@marge> <4E0ADFF3.50100@egenix.com> <1309341057.8003.9.camel@marge> Message-ID: <4E0AFC7A.502@egenix.com> Victor Stinner wrote: > Le mercredi 29 juin 2011 ? 10:18 +0200, M.-A. Lemburg a ?crit : >> Victor Stinner wrote: >>> Le mardi 28 juin 2011 ? 16:02 +0200, M.-A. Lemburg a ?crit : >>>> How about a more radical change: have open() in Py3 default to >>>> opening the file in binary mode, if no encoding is given (even >>>> if the mode doesn't include 'b') ? >>> >>> I tried your suggested change: Python doesn't start. >> >> No surprise there: it's an incompatible change, but one that undoes >> a wart introduced in the Py3 transition. Guessing encodings should >> be avoided whenever possible. > > It means that all programs written for Python 3.0, 3.1, 3.2 will stop > working with the new 3.x version (let say 3.3). Users will have to > migrate from Python 2 to Python 3.2, and then migration from Python 3.2 > to Python 3.3 :-( I wasn't suggesting doing this for 3.3, but we may want to start the usual feature change process to make the change eventually happen. > I would prefer a ResourceWarning (emited if the encoding is not > specified), hidden by default: it doesn't break compatibility, and > -Werror gives exactly the same behaviour that you expect. ResourceWarning is the wrong type of warning for this. I'd suggest to use a UnicodeWarning or perhaps create a new EncodingWarning instead. >> This demonstrates that Python's stdlib is still not being explicit >> about the encoding issues. I suppose that things just happen to work >> because we mostly use ASCII files for configuration and setup. > > I did more tests. I found some mistakes and sometimes the binary mode > can be used, but most function really expect the locale encoding (it is > the correct encoding to read-write files). I agree that it would be to > have an explicit encoding="locale", but make it mandatory is a little > bit rude. Again: Using a locale based default encoding will not work out in the long run. We've had those discussions many times in the past. I don't think there's anything bad with having the user require to set an encoding if he wants to read text. It makes him/her think twice about the encoding issue, which is good. And, of course, the stdlib should start using this explicit-is-better-than-implicit approach as well. >>> Then I tried my suggestion (use "utf-8" by default): Python starts >>> correctly, I can build it (run "make") and... the full test suite pass >>> without any change. (I'm testing on Linux, my locale encoding is UTF-8.) >> >> I bet it would also with "ascii" in most cases. Which then just >> means that the Python build process and test suite is not a good >> test case for choosing a default encoding. >> >> Linux is also a poor test candidate for this, since most user setups >> will use UTF-8 as locale encoding. Windows, OTOH, uses all sorts of >> code page encodings (usually not UTF-8), so you are likely to hit >> the real problem cases a lot easier. > > I also ran the test suite on my patched Python (open uses UTF-8 by > default) with ASCII locale encoding (LANG=C), the test suite does also > pass. Many tests uses non-ASCII characters, some of them are skipped if > the locale encoding is unable to encode the tested text. Thanks for checking. So the build process and test suite are indeed not suitable test cases for the problem at hand. With just ASCII files to decode, Python will simply never fail to decode the content, regardless of whether you use an ASCII, UTF-8 or some Windows code page as locale encoding. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jun 29 2011) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ncoghlan at gmail.com Wed Jun 29 14:37:00 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 Jun 2011 22:37:00 +1000 Subject: [Python-Dev] [Python-checkins] cpython: Issue #12400: fix test_faulthandler if regrtest captures sys.stderr In-Reply-To: References: Message-ID: On Wed, Jun 29, 2011 at 9:43 PM, victor.stinner wrote: > http://hg.python.org/cpython/rev/80c7d63325f6 > changeset: ? 71065:80c7d63325f6 > user: ? ? ? ?Victor Stinner > date: ? ? ? ?Wed Jun 29 13:44:05 2011 +0200 > summary: > ?Issue #12400: fix test_faulthandler if regrtest captures sys.stderr > > faulthandler.enable() requires that sys.stderr has a fileno() method. Would there be any value in falling back to sys.__stderr__ in the case where sys.stderr has been redirected? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Wed Jun 29 19:00:13 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 29 Jun 2011 17:00:13 +0000 (UTC) Subject: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation Message-ID: PEP 397 (Python launcher for Windows) has a reference implementation in Python. Does anyone know of a C implementation, or is planning/working on one? I realise this is the final objective, so such implementation might be premature, but perhaps someone has been experimenting ... Regards, Vinay Sajip From victor.stinner at haypocalc.com Wed Jun 29 20:05:27 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 29 Jun 2011 20:05:27 +0200 Subject: [Python-Dev] [Python-checkins] cpython (3.2): Issue #12400: runtest() truncates the StringIO stream before a new test In-Reply-To: <4E0B5B49.7010905@udel.edu> References: <4E0B5B49.7010905@udel.edu> Message-ID: <1309370727.7968.3.camel@marge> Le mercredi 29 juin 2011 ? 13:05 -0400, Terry Reedy a ?crit : > On 6/29/2011 11:30 AM, victor.stinner wrote: > > > summary: > > Issue #12400: runtest() truncates the StringIO stream before a new test > > > > files: > > Lib/test/regrtest.py | 1 + > > 1 files changed, 1 insertions(+), 0 deletions(-) > > > > > > diff --git a/Lib/test/regrtest.py b/Lib/test/regrtest.py > > --- a/Lib/test/regrtest.py > > +++ b/Lib/test/regrtest.py > > @@ -793,6 +793,7 @@ > > if runtest.stringio is None: > > runtest.stringio = io.StringIO() > > stream = runtest.stringio > > + stream.truncate(0) > > You *MUST* seek to 0 to reset the file position, which I presume is your > intention. 'Resize' does not mean what you probably think is does (and > what I thought once either ;-). > > "truncate(size=None) > Resize the stream to the given size in bytes (or the current position if > size is not specified). The current stream position isn?t changed." > > >>> s=io.StringIO() > >>> s.write('abc') > 3 > >>> s.truncate(0) > 0 > >>> s.tell() > 3 > >>> s.write('abc') > 3 > >>> s.getvalue() > '\x00\x00\x00abc' Oh crap. I read _pyio source code and .truncate(0) was looking for the right method to "reset" a StringIO. I tried .truncate(0) in a terminal, but the zeros ('\x00') are "hidden". sys.stdout.write("\0") doesn't print anything in my Linux terminal. Fixed by commit 450209efe272, thank you. Victor From martin at v.loewis.de Wed Jun 29 23:38:48 2011 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 29 Jun 2011 23:38:48 +0200 Subject: [Python-Dev] svn.python.org confusion In-Reply-To: References: Message-ID: <4E0B9B68.6090207@v.loewis.de> > Could some text be added to the svn browser pages to at least indicate > that the repo is no longer being updated with a link to the > hg.python.org browser? Adding some text to only some pages (namely those that have been migrated to mercurial) is difficult - so difficult that I won't have time for it within the next 12 months. Contributions are welcome. > I'm not sure what to do about the repos > themselves if people attempt to do an svn co. Perhaps that should just > be disabled totally for python? It would certainly be possible to "svn rm" the trunk directory, if that is any help. Of course, this would also break all URL links into this tree, of which there are still plenty (and they likely won't go away in the years to come). So I would rather check a file into the directory with the name THIS_SUBVERSION_REPOSITORY_HAS_BEEN_MIGRATED_TO_MERCURIAL_DO_NOT_USE_IT_FOR_ANYTHING_BUT_ARCHEOLOGICAL_RESEARCH.txt See also http://python.cvs.sourceforge.net/viewvc/python/python/ http://python.cvs.sourceforge.net/viewvc/python/python/dist/src/ Regards, Martin From ezio.melotti at gmail.com Thu Jun 30 03:14:20 2011 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Thu, 30 Jun 2011 04:14:20 +0300 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Fixup repr for dict_proxy objects. In-Reply-To: References: Message-ID: Hi, On Thu, Jun 30, 2011 at 2:51 AM, raymond.hettinger < python-checkins at python.org> wrote: > http://hg.python.org/cpython/rev/9a0b6c07b488 > changeset: 71086:9a0b6c07b488 > branch: 2.7 > parent: 71055:cdfcd00873cd > user: Raymond Hettinger > date: Thu Jun 30 00:44:36 2011 +0100 > summary: > Fixup repr for dict_proxy objects. > This was already fixed in a slightly different way in 3.x. Unless there's a valid reason to have two different implementations of dict_proxy.__repr__ in 2.x and 3.x, I would suggest to pick the best one and use it on all the branches. See also http://bugs.python.org/issue5587 Best Regards, Ezio Melotti -------------- next part -------------- An HTML attachment was scrubbed... URL: From jnoller at gmail.com Thu Jun 30 06:05:32 2011 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 30 Jun 2011 00:05:32 -0400 Subject: [Python-Dev] speed.python.org machine online Message-ID: I've posted a more expansive entry on my blog: http://jessenoller.com/2011/06/29/announcing-the-new-speed-python-org-machine/ But the short version, that as discussed at the VM and language summit, we now have a hosted machine dedicated to the running of cross-interpreter speed tests, etc. The hardware was generously donate by HP and the hosting provided, again, free, by OSU/OSL. DL380 HP DL380G7 X5670 LFF (2U) Dual HP NC382i Dual Port Mul?ti?func?tion Giga?bit Server Adapters HP Smart Array P410i/1GB FBWC Controller 4x 4GB (1x4GB) Dual Rank x4 PC3-10600 (DDR3-1333) Reg?is?tered CAS-9 Mem?ory Kit 2x HP 750W Com?mon Slot Gold Hot Plug Power Sup?ply Kit HP iLO Advanced includ?ing 1yr 24x7 Tech?ni?cal Sup?port and Updates Elec?tronic License 4x HP 300GB 6G SAS 15K rpm LFF (3.5-inch) Dual Port Enter?prise 3yr War?ranty Hard Drive 2 HP DL380 G7 Intel? Xeon? X5680 (3.33GHz/6-core/130W/12MB) FIO Proces?sor Kit With hyperthreading on, the machine has 24 cores, and handily translates pypy using cpython 2.7 in about half the time it typically takes. I am looking forward to handing this over to the team who will be running with the project from here on out - special thanks to Van, Bob Gobeille at HP and the entire OSU/OSL team. jesse From skippy.hammond at gmail.com Thu Jun 30 06:23:13 2011 From: skippy.hammond at gmail.com (Mark Hammond) Date: Thu, 30 Jun 2011 14:23:13 +1000 Subject: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation In-Reply-To: References: Message-ID: <4E0BFA31.8020200@gmail.com> On 30/06/2011 3:00 AM, Vinay Sajip wrote: > PEP 397 (Python launcher for Windows) has a reference implementation in Python. > Does anyone know of a C implementation, or is planning/working on one? I realise > this is the final objective, so such implementation might be premature, but > perhaps someone has been experimenting ... Not yet - my last update of the PEP has made the existing reference implementation out-of-date, so I want to work on that before starting on the C version. However, seeing as my note about the most recent PEP update attracted zero comments, I admit I lost any sense of urgency on actually doing this... I'll make an effort to update that reference impl in the next week or so (but obviously anyone else is free to help ;) Cheers, Mark From ericsnowcurrently at gmail.com Thu Jun 30 06:55:14 2011 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 29 Jun 2011 22:55:14 -0600 Subject: [Python-Dev] speed.python.org machine online In-Reply-To: References: Message-ID: On Wed, Jun 29, 2011 at 10:05 PM, Jesse Noller wrote: > I've posted a more expansive entry on my blog: > http://jessenoller.com/2011/06/29/announcing-the-new-speed-python-org-machine/ > > But the short version, that as discussed at the VM and language > summit, we now have a hosted machine dedicated to the running of > cross-interpreter speed tests, etc. The hardware was generously donate > by HP and the hosting provided, again, free, by OSU/OSL. > > DL380 HP DL380G7 X5670 LFF (2U) > Dual HP NC382i Dual Port Mul?ti?func?tion Giga?bit Server Adapters > HP Smart Array P410i/1GB FBWC Controller > 4x 4GB (1x4GB) Dual Rank x4 PC3-10600 (DDR3-1333) Reg?is?tered CAS-9 Mem?ory Kit > 2x HP 750W Com?mon Slot Gold Hot Plug Power Sup?ply Kit > HP iLO Advanced includ?ing 1yr 24x7 Tech?ni?cal Sup?port and Updates > Elec?tronic ?License > 4x HP 300GB 6G SAS 15K rpm LFF (3.5-inch) Dual Port Enter?prise 3yr > War?ranty Hard Drive > 2 ? HP DL380 G7 Intel? Xeon? X5680 (3.33GHz/6-core/130W/12MB) FIO Proces?sor Kit > > With hyperthreading on, the machine has 24 cores, and handily > translates pypy using cpython 2.7 in about half the time it typically > takes. > > I am looking forward to handing this over to the team who will be > running with the project from here on out - special thanks to Van, Bob > Gobeille at HP and the entire OSU/OSL team. > Thanks for the continuing effort, Jesse, Van, and everyone! This is why Python is great and not just good. I can't say thank you enough to all the people that work so hard on tracker issues, documentation, conferences, outreach, infrastructure, and all the pieces that make Python more than just a good idea. Not to discount the effort leading up to the ideas, but the Python community realizes them through the hard work of so many individuals willing to "sacrifice" for something they love. I've been following this list for a year and have not seen much recognition, so it likely goes without saying (or I've just broken an unwritten rule :), but sometimes it needs to be said out loud regardless and sometimes loudly. So thanks everyone! -eric > jesse > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ericsnowcurrently%40gmail.com > From raymond.hettinger at gmail.com Thu Jun 30 08:41:57 2011 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 30 Jun 2011 07:41:57 +0100 Subject: [Python-Dev] [Python-checkins] cpython (2.7): Fixup repr for dict_proxy objects. In-Reply-To: References: Message-ID: On Jun 30, 2011, at 2:14 AM, Ezio Melotti wrote: > Hi, > > On Thu, Jun 30, 2011 at 2:51 AM, raymond.hettinger wrote: > Fixup repr for dict_proxy objects. > > > This was already fixed in a slightly different way in 3.x. The %R formatting code is not available in 2.x Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at timgolden.me.uk Thu Jun 30 09:34:19 2011 From: mail at timgolden.me.uk (Tim Golden) Date: Thu, 30 Jun 2011 08:34:19 +0100 Subject: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation In-Reply-To: <4E0BFA31.8020200@gmail.com> References: <4E0BFA31.8020200@gmail.com> Message-ID: <4E0C26FB.9040704@timgolden.me.uk> On 30/06/2011 05:23, Mark Hammond wrote: > On 30/06/2011 3:00 AM, Vinay Sajip wrote: >> PEP 397 (Python launcher for Windows) has a reference implementation >> in Python. >> Does anyone know of a C implementation, or is planning/working on one? >> I realise >> this is the final objective, so such implementation might be >> premature, but >> perhaps someone has been experimenting ... > > Not yet - my last update of the PEP has made the existing reference > implementation out-of-date, so I want to work on that before starting on > the C version. However, seeing as my note about the most recent PEP > update attracted zero comments.. Sorry, didn't realise you couldn't see me silently nodding my head from the other side of the world ;) I really must work on those communication skills... TJG From ncoghlan at gmail.com Thu Jun 30 12:58:52 2011 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Jun 2011 20:58:52 +1000 Subject: [Python-Dev] [Python-checkins] cpython: store the current scope on the stack right away In-Reply-To: References: Message-ID: On Thu, Jun 30, 2011 at 1:53 PM, benjamin.peterson wrote: > http://hg.python.org/cpython/rev/151142c0c5b1 > changeset: ? 71087:151142c0c5b1 > parent: ? ? ?71084:9aa3fcab76f0 > user: ? ? ? ?Benjamin Peterson > date: ? ? ? ?Wed Jun 29 22:52:39 2011 -0500 > summary: > ?store the current scope on the stack right away Heh, I was going to comment that the spate of refleaks after your previous commit looked mighty suspicious :) End result is simpler, cleaner code overall, though. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From fuzzyman at voidspace.org.uk Thu Jun 30 13:13:25 2011 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 30 Jun 2011 12:13:25 +0100 Subject: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation In-Reply-To: <4E0C26FB.9040704@timgolden.me.uk> References: <4E0BFA31.8020200@gmail.com> <4E0C26FB.9040704@timgolden.me.uk> Message-ID: <4E0C5A55.6000706@voidspace.org.uk> On 30/06/2011 08:34, Tim Golden wrote: > On 30/06/2011 05:23, Mark Hammond wrote: >> On 30/06/2011 3:00 AM, Vinay Sajip wrote: >>> PEP 397 (Python launcher for Windows) has a reference implementation >>> in Python. >>> Does anyone know of a C implementation, or is planning/working on one? >>> I realise >>> this is the final objective, so such implementation might be >>> premature, but >>> perhaps someone has been experimenting ... >> >> Not yet - my last update of the PEP has made the existing reference >> implementation out-of-date, so I want to work on that before starting on >> the C version. However, seeing as my note about the most recent PEP >> update attracted zero comments.. > > Sorry, didn't realise you couldn't see me silently nodding my > head from the other side of the world ;) I really must work > on those communication skills... I have that email (the update one from Mark not the silent nodding from Tim) still sitting in my inbox waiting for me to properly read through and comment on... Sorry for being useless, I'll try and move it up the priority list. I really like the PEP and think it will be a *huge* step forward for Windows users - so it's purely the details that need thrashing out (heh). In the latest update Mark also addressed my main concern, making the launcher configurable so it can also be used by alternative implementations (particularly IronPython for Windows). I've copied Jeff Hardy and Dino (IronPython maintainers) in on this, in the hope that they might take a look at the latest version of the PEP too... All the best, Michael > > TJG > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From vinay_sajip at yahoo.co.uk Thu Jun 30 14:09:43 2011 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 30 Jun 2011 12:09:43 +0000 (UTC) Subject: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation References: <4E0BFA31.8020200@gmail.com> Message-ID: Mark Hammond gmail.com> writes: > Not yet - my last update of the PEP has made the existing reference > implementation out-of-date, so I want to work on that before starting on > the C version. However, seeing as my note about the most recent PEP > update attracted zero comments, I admit I lost any sense of urgency on > actually doing this... I'll make an effort to update that reference > impl in the next week or so (but obviously anyone else is free to help ;) Sorry, I just recently came across the PEP when Tim Golden pointed it out to me, when I asked a question about executable script support in packaging. I think I was probably not so focused on Windows at the time of the original announcement ... There's a lot to like in the PEP, and I have some questions relating to the latest version: 1. In the section on shebang line parsing, it says "If the command starts with the definition of a customized command followed by a space character, the customized command will be used." Sorry if I'm being dense, but what's the significance of the trailing space character? In fact, your vpython example in the customeised comments section doesn't show a trailing space - you've used '#! vpython' as the shebang line. 2. The section on .ini files says that "Commands specified in the [.ini file in the] "application directory" [APPDATA] will have precedence over the one next to the [launcher] executable." What's the benefit of this? If you have only one launcher executable of one type (say console, 32-bit) on a system, then there's no point in having two locations for .ini files. If you have multiple copies of the launcher and multiple .ini files next to them, then with this precedence order, you can't specialise the behaviour of anything in a specific launcher .ini that's also specified in the APPDATA .ini. Doesn't it make more sense to look for a setting in any file next to the launcher, and if not found there, look in the APPDATA? That way common things can be defined in the APPDATA .ini and overridden for special cases in the launcher-adjacent .ini. Or have I got completely the wrong end of the stick? By the way, I've already converted the existing Python reference implementation to C (I did it before I saw your response to my post). It basically works in that it does what the Python version does, but doesn't include any handling of -32 suffixes or .ini files. I can post this on BitBucket if anyone's interested, but it's a work in progress. I'm working on some simple tests. Regards, Vinay Sajip From p.f.moore at gmail.com Thu Jun 30 14:50:11 2011 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 30 Jun 2011 13:50:11 +0100 Subject: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation In-Reply-To: <4E0C5A55.6000706@voidspace.org.uk> References: <4E0BFA31.8020200@gmail.com> <4E0C26FB.9040704@timgolden.me.uk> <4E0C5A55.6000706@voidspace.org.uk> Message-ID: On 30 June 2011 12:13, Michael Foord wrote: > I have that email (the update one from Mark not the silent nodding from Tim) > still sitting in my inbox waiting for me to properly read through and > comment on... Sorry for being useless, I'll try and move it up the priority > list. > > I really like the PEP and think it will be a *huge* step forward for Windows > users - so it's purely the details that need thrashing out (heh). That's my situation, too. I'll try to look through it properly in the next day or two. Paul. From merwok at netwok.org Thu Jun 30 16:17:32 2011 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 30 Jun 2011 16:17:32 +0200 Subject: [Python-Dev] svn.python.org confusion In-Reply-To: <4E0B9B68.6090207@v.loewis.de> References: <4E0B9B68.6090207@v.loewis.de> Message-ID: <4E0C857C.6040906@netwok.org> Adding a THIS_REPOSITORY_HAS_MOVED file to recent branches looks good. Regards From barry at python.org Thu Jun 30 16:32:51 2011 From: barry at python.org (Barry Warsaw) Date: Thu, 30 Jun 2011 15:32:51 +0100 Subject: [Python-Dev] svn.python.org confusion In-Reply-To: <4E0C857C.6040906@netwok.org> References: <4E0B9B68.6090207@v.loewis.de> <4E0C857C.6040906@netwok.org> Message-ID: <20110630153251.751dddfd@snowdog> On Jun 30, 2011, at 04:17 PM, ?ric Araujo wrote: >Adding a THIS_REPOSITORY_HAS_MOVED file to recent branches looks good. I'm not against adding this to svn, but please be sure these files don't leak into the tarballs should we need to cut another 2.5 or 2.6 release. I think that just means tweaking sandbox/release/release.py. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From tseaver at palladion.com Thu Jun 30 17:40:30 2011 From: tseaver at palladion.com (Tres Seaver) Date: Thu, 30 Jun 2011 11:40:30 -0400 Subject: [Python-Dev] svn.python.org confusion In-Reply-To: <20110630153251.751dddfd@snowdog> References: <4E0B9B68.6090207@v.loewis.de> <4E0C857C.6040906@netwok.org> <20110630153251.751dddfd@snowdog> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 06/30/2011 10:32 AM, Barry Warsaw wrote: > On Jun 30, 2011, at 04:17 PM, ?ric Araujo wrote: > >> Adding a THIS_REPOSITORY_HAS_MOVED file to recent branches looks good. > > I'm not against adding this to svn, but please be sure these files don't leak > into the tarballs should we need to cut another 2.5 or 2.6 release. I think > that just means tweaking sandbox/release/release.py. The fact that releases might / will still be made from those branches argues against including the file on them at all: they are in fact the "canonical" repository locations, even if most of the work is done in hg and patched into them. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk4MmO4ACgkQ+gerLs4ltQ7ybwCgqIkXCdYWIdPM9dHFDa658nLe RYkAnRccCcuCjxQvYSn/jseLJU/ZTmcQ =+fEZ -----END PGP SIGNATURE----- From jdhardy at gmail.com Thu Jun 30 19:31:34 2011 From: jdhardy at gmail.com (Jeff Hardy) Date: Thu, 30 Jun 2011 10:31:34 -0700 Subject: [Python-Dev] PEP 397 (Python launcher for Windows) reference implementation In-Reply-To: <4E0C5A55.6000706@voidspace.org.uk> References: <4E0BFA31.8020200@gmail.com> <4E0C26FB.9040704@timgolden.me.uk> <4E0C5A55.6000706@voidspace.org.uk> Message-ID: On Thu, Jun 30, 2011 at 4:13 AM, Michael Foord wrote: > In the latest update Mark also addressed my main concern, making the > launcher configurable so it can also be used by alternative implementations > (particularly IronPython for Windows). I've copied Jeff Hardy and Dino > (IronPython maintainers) in on this, in the hope that they might take a look > at the latest version of the PEP too... I skimmed it a while back but haven't had the time to give an in-depth study. On the surface it seems fine, but the devil is in the details... - Jeff From ulrich.eckhardt at dominolaser.com Thu Jun 30 21:13:46 2011 From: ulrich.eckhardt at dominolaser.com (Ulrich Eckhardt) Date: Thu, 30 Jun 2011 21:13:46 +0200 Subject: [Python-Dev] time.sleep(-1) behaviour Message-ID: <201106302113.47144.ulrich.eckhardt@dominolaser.com> Hi! This is a request for clarification for the thread "how to call a function for evry 10 seconds" from the user mailinglist/newsgroup. The gist is this: 1. On Linux/Python 2.6, time.sleep(-1.0) raises an IOError. 2. On MS Windows/Python 2.5 or 2.7 this sleeps forever. It seems that converting this to a 32-bit integer for Sleep() causes an underflow. Questions: 1. Is the IOError somehow explainable? 2. Is the IOError acceptable or a bug? 3. Is the behaviour under MS Windows acceptable or a bug? 4. Since time.sleep() is documented to possibly sleep a bit longer than specified, someone suggested that the overall sleep time could also be truncated to a minimum of zero, i.e. treating negative values as zero. Greetings! Uli ************************************************************************************** Domino Laser GmbH, Fangdieckstra?e 75a, 22547 Hamburg, Deutschland Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at http://www.dominolaser.com ************************************************************************************** Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden. E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Domino Laser GmbH ist f?r diese Folgen nicht verantwortlich. ************************************************************************************** From rdmurray at bitdance.com Thu Jun 30 21:21:44 2011 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 30 Jun 2011 15:21:44 -0400 Subject: [Python-Dev] time.sleep(-1) behaviour In-Reply-To: <201106302113.47144.ulrich.eckhardt@dominolaser.com> References: <201106302113.47144.ulrich.eckhardt@dominolaser.com> Message-ID: <20110630192145.7C83D2506C1@webabinitio.net> On Thu, 30 Jun 2011 21:13:46 +0200, Ulrich Eckhardt wrote: > 1. Is the IOError somehow explainable? > 2. Is the IOError acceptable or a bug? > 3. Is the behaviour under MS Windows acceptable or a bug? > 4. Since time.sleep() is documented to possibly sleep a bit longer than > specified, someone suggested that the overall sleep time could also be > truncated to a minimum of zero, i.e. treating negative values as zero. Please file a bug report at bugs.python.org. -- R. David Murray http://www.bitdance.com From victor.stinner at haypocalc.com Thu Jun 30 22:11:42 2011 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 30 Jun 2011 22:11:42 +0200 Subject: [Python-Dev] time.sleep(-1) behaviour In-Reply-To: <201106302113.47144.ulrich.eckhardt@dominolaser.com> References: <201106302113.47144.ulrich.eckhardt@dominolaser.com> Message-ID: <1309464702.29934.2.camel@marge> Le jeudi 30 juin 2011 ? 21:13 +0200, Ulrich Eckhardt a ?crit : > Hi! > > This is a request for clarification for the thread "how to call a function for > evry 10 seconds" from the user mailinglist/newsgroup. > > > The gist is this: > 1. On Linux/Python 2.6, time.sleep(-1.0) raises an IOError. > 2. On MS Windows/Python 2.5 or 2.7 this sleeps forever. It seems that > converting this to a 32-bit integer for Sleep() causes an underflow. select.select() was changed recently to raise an exception if the timeout is negative to get the same behaviour on all platforms. signal.sigtimedwait() (recently added function) raises also an exception if the timeout is negative. If it's an overflow, Python should raise an OverflowError, or raise another error, but not sleep for 2^32-1 (or 2^64-1 ?) seconds. Victor