From nnorwitz at gmail.com Tue Apr 1 06:34:10 2008 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 31 Mar 2008 21:34:10 -0700 Subject: [Python-Dev] refleaks on trunk Message-ID: test_io is the only leaky test on trunk that I know of. I narrowed down the leaks to the code below. It's possible there are other leaks in test_io. n -- import sys, gc import _fileio, io class FileIO(_fileio._FileIO, io.RawIOBase): def close(self): io.RawIOBase.close(self) def main(): class MyFileIO(FileIO): pass f = MyFileIO('tt-test', "w") f.close() for i in range(10): main() print(gc.collect()) print(sys.gettotalrefcount()) From nnorwitz at gmail.com Tue Apr 1 08:42:36 2008 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 31 Mar 2008 23:42:36 -0700 Subject: [Python-Dev] fixing tests on windows Message-ID: The Windows buildbots are still failing because some tests keep files opened. This causes subsequent tests which use the same file to fail. Here is a recent run which had a failure early on: http://www.python.org/dev/buildbot/stable/x86%20XP-3%20trunk/builds/1209/step-test/0 I'm assuming the first failure (test_bufio) was due to an open file. (Can't tell, no error msg.) That means the problem was in that test or an earlier test. The only earlier tests are: test_import test_sys test_descr test_xdrlib test_urllibnet test_binhex test_strptime test_importhooks test_copy test_hmac test_genericpath test_complex test_timeout test_quopri test_marshal test_zipfile test_mutants test_csv test_ucn test_hash test_wsgiref test_mmap test_ftplib test_pickletools test_codecmaps_cn Of those tests, only some of them reference TESTFN which is the probable file: test_import 22 test_descr 3 test_urllibnet 2 test_binhex 2 test_genericpath 44 test_complex 3 test_marshal 33 test_zipfile 136 test_mutants 7 test_mmap 32 I inspected test_mmap which had been modified somewhat recently and tried to fix a few things. I'm not sure I helped and there are probably more problems lurking. This issue may not be noticed in the default test run. It definitely occurs when the tests are run in a random order as they are on the buildbots. Try running: ./python.exe ./Lib/test/regrtest.py -r That should help provoke more errors. As always, patches are greatly appreciated. Cheers, n From amauryfa at gmail.com Tue Apr 1 09:37:12 2008 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 1 Apr 2008 09:37:12 +0200 Subject: [Python-Dev] refleaks on trunk In-Reply-To: References: Message-ID: On Tue, Apr 1, 2008 at 6:34 AM, Neal Norwitz wrote: > test_io is the only leaky test on trunk that I know of. I narrowed > down the leaks to the code below. > > It's possible there are other leaks in test_io. > > n > -- > import sys, gc > import _fileio, io > > class FileIO(_fileio._FileIO, io.RawIOBase): > def close(self): > io.RawIOBase.close(self) > > def main(): > class MyFileIO(FileIO): pass > f = MyFileIO('tt-test', "w") > f.close() > > for i in range(10): > main() > print(gc.collect()) > print(sys.gettotalrefcount()) The problem is that the MyFileIO class is registered in io.RawIOBase._abc_cache, and never freed. This is a problem with ABCs: _abc_cache should be changed to a WeakSet, like python3.0 did. I filed http://bugs.python.org/issue2521 about this problem. -- Amaury Forgeot d'Arc From mhammond at skippinet.com.au Tue Apr 1 11:45:57 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Tue, 1 Apr 2008 20:45:57 +1100 Subject: [Python-Dev] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <002401c892ff$a8254630$f86fd290$@com.au> References: <002401c892ff$a8254630$f86fd290$@com.au> Message-ID: <00c801c893dd$32d03320$98709960$@com.au> I wrote: > FYI, I've uploaded a patch that provides for cross-compilation on > Windows between 32 and 64 bit platforms - all comments invited! While I have some people's attention I'd like to re-raise another issue I foresee for x64 builds. I've mentioned this over the last couple of months, but haven't got much of a response, so this seems like a reasonable opportunity to try one more time :) Currently, the "official" (by way of being de-facto) directory structure for a build tree is 'PCBuild/.' for x86 builds and 'PCBuild/amd64' for x64 platforms. I believe this might cause problems for people trying to port their applications to 64bit platforms. My proposal is that we change this subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' exist, while 'PCBuild/.' is always the 'native' platform - ie, that 'PCBuild/.' be a copy of one of the platform subdirectories. The rationale is fairly simple: I'm quite certain that this new directory layout will break existing "native only" build processes (ie, those that aren't aware of cross-compilation). I'm quite certain that Mozilla will break, for example, and no cross-compilation process exists that I am aware of. Existing build tools already know about the PCBuild directory, and are focused almost exclusively towards native compilation - all such tools on non-x86 platforms are currently broken. My proposal would help avoid breakage for existing build processes or tools that try to natively target x64, and give a reasonable story for build processes that explicitly support cross-compilation. I believe the costs are fairly small - disk space is cheap and the extra complexity in the VS project files should be reasonable. What do people think? I think it's time to take the approach of "silent ascent", so unless I hear objections I'll upload a new patch which implements this and also takes it into account for Distutils builds. Cheers, Mark From steven.bethard at gmail.com Tue Apr 1 17:00:26 2008 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 1 Apr 2008 09:00:26 -0600 Subject: [Python-Dev] fixing tests on windows In-Reply-To: References: Message-ID: On Tue, Apr 1, 2008 at 12:42 AM, Neal Norwitz wrote: > The Windows buildbots are still failing because some tests keep files > opened. This causes subsequent tests which use the same file to fail. > > Here is a recent run which had a failure early on: > http://www.python.org/dev/buildbot/stable/x86%20XP-3%20trunk/builds/1209/step-test/0 > > I'm assuming the first failure (test_bufio) was due to an open file. > (Can't tell, no error msg.) That means the problem was in that test > or an earlier test. The only earlier tests are: > > test_import > test_sys > test_descr > test_xdrlib > test_urllibnet > test_binhex > test_strptime > test_importhooks > test_copy > test_hmac > test_genericpath > test_complex > test_timeout > test_quopri > test_marshal > test_zipfile > test_mutants > test_csv > test_ucn > test_hash > test_wsgiref > test_mmap > test_ftplib > test_pickletools > test_codecmaps_cn > > Of those tests, only some of them reference TESTFN which is the probable file: > > test_import 22 > test_descr 3 > test_urllibnet 2 > test_binhex 2 > test_genericpath 44 > test_complex 3 > test_marshal 33 > test_zipfile 136 > test_mutants 7 > test_mmap 32 > > I inspected test_mmap which had been modified somewhat recently and > tried to fix a few things. I'm not sure I helped and there are > probably more problems lurking. > > This issue may not be noticed in the default test run. It definitely > occurs when the tests are run in a random order as they are on the > buildbots. Try running: ./python.exe ./Lib/test/regrtest.py -r > > That should help provoke more errors. At the sprints, I ran into a bunch of similar errors running the test suite on my Windows Vista box, even on tests that were properly cleaning up after themselves in tearDown(). I even tried putting in sleeps as long as 1 second, to no avail. The only way to get the test suite to run without these errors was to stop the Windows Search Service, fully disable Icon Overlays for TortoiseSVN, and then close down all open folders. Any chance the boxes the tests are being run on are running the Windows Search Service or have Icon Overlays enabled for TortoiseSVN? (If anyone has any ideas of how to get around these problems, I'd love to hear them. Seems like I shouldn't have to disable these services.) Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From skip at pobox.com Tue Apr 1 17:05:28 2008 From: skip at pobox.com (skip at pobox.com) Date: Tue, 1 Apr 2008 10:05:28 -0500 Subject: [Python-Dev] test_signal on osx g4 Message-ID: <18418.20280.404970.167329@montanaro-dyndns-org.local> test_signal is failing on osx g4: test test_signal failed -- Traceback (most recent call last): File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 151, in test_main self.fail(tb) AssertionError: Traceback (most recent call last): File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 134, in test_main self.run_test() File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 80, in run_test child = subprocess.Popen(['kill', '-HUP', str(pid)]) File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", line 593, in __init__ errread, errwrite) File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", line 1078, in _execute_child data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB OSError: [Errno 4] Interrupted system call This is the code which reads stderr from the child process: data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB os.close(errpipe_read) if data != "": os.waitpid(self.pid, 0) child_exception = pickle.loads(data) raise child_exception I'm not an expert in this stuff my any stretch of the imagination, but shouldn't subprocess try harder to read this output? Something like: while True: try: data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB except OSError, err: if err.errno == errno.EINTR: continue else: raise else: os.close(errpipe_read) if data != "": os.waitpid(self.pid, 0) child_exception = pickle.loads(data) raise child_exception break Maybe not while True, but try a few times at least. Or is the system supposed to automatically restart interrupted system calls? Skip From mail at timgolden.me.uk Tue Apr 1 17:20:46 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 01 Apr 2008 16:20:46 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: References: Message-ID: <47F252CE.1020702@timgolden.me.uk> Steven Bethard wrote: > At the sprints, I ran into a bunch of similar errors running the test > suite on my Windows Vista box, even on tests that were properly > cleaning up after themselves in tearDown(). I even tried putting in > sleeps as long as 1 second, to no avail. The only way to get the test > suite to run without these errors was to stop the Windows Search > Service, fully disable Icon Overlays for TortoiseSVN, and then close > down all open folders. > > Any chance the boxes the tests are being run on are running the > Windows Search Service or have Icon Overlays enabled for TortoiseSVN? > > (If anyone has any ideas of how to get around these problems, I'd love > to hear them. Seems like I shouldn't have to disable these services.) I'm not sure I'm going to help here, but the reason that this happens is that certain services -- and the Desktop Search tools are notorious for this -- get file handles with FILE_SHARE_DELETE, which means that the file isn't actually removed when it's deleted; only when the last of those handles closes. I did see a suggestion somewhere that, to circumvent this issue, instead of removing it a file you need to move it to %TEMP% (or wherever) and delete it there. I'll try to knock up a test case to see if this works. TJG From mail at timgolden.me.uk Tue Apr 1 18:13:34 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 01 Apr 2008 17:13:34 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F252CE.1020702@timgolden.me.uk> References: <47F252CE.1020702@timgolden.me.uk> Message-ID: <47F25F2E.4070408@timgolden.me.uk> Tim Golden wrote: > Steven Bethard wrote: >> At the sprints, I ran into a bunch of similar errors running the test >> suite on my Windows Vista box, even on tests that were properly >> cleaning up after themselves in tearDown(). I even tried putting in >> sleeps as long as 1 second, to no avail. The only way to get the test >> suite to run without these errors was to stop the Windows Search >> Service, fully disable Icon Overlays for TortoiseSVN, and then close >> down all open folders. >> >> Any chance the boxes the tests are being run on are running the >> Windows Search Service or have Icon Overlays enabled for TortoiseSVN? >> >> (If anyone has any ideas of how to get around these problems, I'd love >> to hear them. Seems like I shouldn't have to disable these services.) > > I'm not sure I'm going to help here, but the reason that > this happens is that certain services -- and the Desktop > Search tools are notorious for this -- get file handles with > FILE_SHARE_DELETE, which means that the file isn't actually > removed when it's deleted; only when the last of those handles > closes. > > I did see a suggestion somewhere that, to circumvent this issue, > instead of removing it a file you need to move it to %TEMP% > (or wherever) and delete it there. I'll try to knock up a test > case to see if this works. OK: to confirm, the following test seems to indicate that doing a (local - same volume) rename followed by a remove will do the right thing, even when a FILE_SHARE_DELETE handle is held. If this is the thing to do, presumably test_support should grow a "remove_file" which does something of this sort? TJG import os, sys import win32file FILENAME = "test" def rename_and_remove (filename): os.rename (filename, filename + ".deleted") os.remove (filename + ".deleted") def remove_only (filename): os.remove (filename) def test (remove): open (FILENAME, "w").close () hFile = win32file.CreateFile ( FILENAME, win32file.GENERIC_READ, win32file.FILE_SHARE_DELETE, None, win32file.OPEN_EXISTING, 0, 0 ) try: remove (FILENAME) try: open (FILENAME, "w").close () except IOError: print "Couldn't recreate" else: print "Could recreate" finally: hFile.Close () try: open (FILENAME, "w").close () except IOError: print "Couldn't recreate" else: print "Could recreate" if __name__ =='__main__': print print "Should not work" test (remove_only) print print "Should work" test (rename_and_remove) From facundobatista at gmail.com Tue Apr 1 18:20:10 2008 From: facundobatista at gmail.com (Facundo Batista) Date: Tue, 1 Apr 2008 13:20:10 -0300 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F25F2E.4070408@timgolden.me.uk> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> Message-ID: 2008/4/1, Tim Golden : > If this is the thing to do, presumably test_support should > grow a "remove_file" which does something of this sort? +1 (I was thinking exactly that). Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From steven.bethard at gmail.com Tue Apr 1 18:27:49 2008 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 1 Apr 2008 10:27:49 -0600 Subject: [Python-Dev] fixing tests on windows In-Reply-To: References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> Message-ID: On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista wrote: > 2008/4/1, Tim Golden : > > If this is the thing to do, presumably test_support should > > grow a "remove_file" which does something of this sort? > > +1 (I was thinking exactly that). +1 here too. That looks like a great solution. Of course, once it's in test_support, we need to fix *all* file removals in the test suite. ;-) Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From sesquile at gmail.com Tue Apr 1 21:02:13 2008 From: sesquile at gmail.com (m h) Date: Tue, 1 Apr 2008 13:02:13 -0600 Subject: [Python-Dev] Tracing at a more granular level (branch coverage) Message-ID: Howdy Folks- I hope this is not too off topic. Sadly the current sys.settrace only allows tracing at the line level. This isn't sufficient to do branch and path coverage. The main problem is that many boolean operations can appear on a single line, and so you can't be sure which conditional branch was taken using the current tracing method. After speaking about code complexity/testing at Pycon [0], with a few people there, and seeing a few people have made some strides or shown interested in metrics/code graphs/flow and branch coverage I think that there is interest in this. One proposed method of getting branch coverage was to use Dalke's Python4Ply [1] to translate code so that all branches occur on their own line. Then using line coverage on that and converting it back to branch coverage of the original code. The maintainer of coverage.py suggested that we look into patching python instead to trace at a more granular level. His feeling was that there would be too many corner cases and the translation would get hairy quite quickly. Sadly in my 8 years of python experience I've yet to touch any c based guts of python. I'm looking for advice on how to get finer grain tracing with sys.settrace. Any advice or suggestions? There is a quorum of people (at least 5 others) who would be very interested in this functionality because it could lead to some cool tools built on top of it. (With the end goal that python code be cleaner, simpler and better tested). thanks much, -matt harrison 0 - http://panela.blog-city.com/pycon_2008_managing_complexity_slides.htm 1 - http://www.dalkescientific.com/Python/python4ply.html From gnewsg at gmail.com Tue Apr 1 21:03:54 2008 From: gnewsg at gmail.com (Giampaolo Rodola') Date: Tue, 1 Apr 2008 12:03:54 -0700 (PDT) Subject: [Python-Dev] fixing tests on windows In-Reply-To: References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> Message-ID: <02680572-1a12-4281-8abe-63b8a97b2ac9@i12g2000prf.googlegroups.com> On 1 Apr, 18:27, "Steven Bethard" wrote: > On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista > > wrote: > > 2008/4/1, Tim Golden : > > ?> ?If this is the thing to do, presumably test_support should > > ?> ?grow a "remove_file" which does something of this sort? > > > ?+1 (I was thinking exactly that). > > +1 here too. That looks like a great solution. ?Of course, once it's > in test_support, we need to fix *all* file removals in the test suite. > ;-) > > Steve Why not just modifying test_support.unlink() like this? Surely more convenient than modifying the whole suite. def unlink(filename): try: if os.name == 'nt': os.rename(filename, filename + ".deleted") filename = filename + ".deleted" os.unlink(filename) except OSError: pass From mail at timgolden.me.uk Tue Apr 1 21:06:11 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 01 Apr 2008 20:06:11 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <02680572-1a12-4281-8abe-63b8a97b2ac9@i12g2000prf.googlegroups.com> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <02680572-1a12-4281-8abe-63b8a97b2ac9@i12g2000prf.googlegroups.com> Message-ID: <47F287A3.3090703@timgolden.me.uk> Giampaolo Rodola' wrote: > > On 1 Apr, 18:27, "Steven Bethard" wrote: >> On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista >> >> wrote: >>> 2008/4/1, Tim Golden : >>> > If this is the thing to do, presumably test_support should >>> > grow a "remove_file" which does something of this sort? >>> +1 (I was thinking exactly that). >> +1 here too. That looks like a great solution. Of course, once it's >> in test_support, we need to fix *all* file removals in the test suite. >> ;-) >> >> Steve > > Why not just modifying test_support.unlink() like this? > Surely more convenient than modifying the whole suite. > > > def unlink(filename): > try: > if os.name == 'nt': > os.rename(filename, filename + ".deleted") > filename = filename + ".deleted" > os.unlink(filename) > except OSError: > pass Funnily enough, that's exactly what the patch I've put together does. Have to look out, though for anywhere which just calls os.unlink rather than going through test_support. TJG From gnewsg at gmail.com Tue Apr 1 21:09:56 2008 From: gnewsg at gmail.com (Giampaolo Rodola') Date: Tue, 1 Apr 2008 12:09:56 -0700 (PDT) Subject: [Python-Dev] fixing tests on windows In-Reply-To: <02680572-1a12-4281-8abe-63b8a97b2ac9@i12g2000prf.googlegroups.com> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <02680572-1a12-4281-8abe-63b8a97b2ac9@i12g2000prf.googlegroups.com> Message-ID: On 1 Apr, 21:03, "Giampaolo Rodola'" wrote: > On 1 Apr, 18:27, "Steven Bethard" wrote: > > > On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista > > > wrote: > > > 2008/4/1, Tim Golden : > > > ?> ?If this is the thing to do, presumably test_support should > > > ?> ?grow a "remove_file" which does something of this sort? > > > > ?+1 (I was thinking exactly that). > > > +1 here too. That looks like a great solution. ?Of course, once it's > > in test_support, we need to fix *all* file removals in the test suite. > > ;-) > > > Steve > > Why not just modifying test_support.unlink() like this? > Surely more convenient than modifying the whole suite. > > def unlink(filename): > ? ? try: > ? ? ? ? if os.name == 'nt': > ? ? ? ? ? ? os.rename(filename, filename + ".deleted") > ? ? ? ? ? ? filename = filename + ".deleted" > ? ? ? ? os.unlink(filename) > ? ? except OSError: > ? ? ? ? pass Another solution, probably better: def unlink(filename): try: os.unlink(filename) except OSError: if os.name == 'nt': try: os.rename(filename, filename + ".deleted") os.unlink(filename + ".deleted") except OSError: pass From gnewsg at gmail.com Tue Apr 1 21:26:17 2008 From: gnewsg at gmail.com (Giampaolo Rodola') Date: Tue, 1 Apr 2008 12:26:17 -0700 (PDT) Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F287A3.3090703@timgolden.me.uk> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <02680572-1a12-4281-8abe-63b8a97b2ac9@i12g2000prf.googlegroups.com> <47F287A3.3090703@timgolden.me.uk> Message-ID: <0a4b21eb-50cf-45fb-bc86-0e6a490bf84f@y24g2000hsd.googlegroups.com> On 1 Apr, 21:06, Tim Golden wrote: > Giampaolo Rodola' wrote: > > > On 1 Apr, 18:27, "Steven Bethard" wrote: > >> On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista > > >> wrote: > >>> 2008/4/1, Tim Golden : > >>> ?> ?If this is the thing to do, presumably test_support should > >>> ?> ?grow a "remove_file" which does something of this sort? > >>> ?+1 (I was thinking exactly that). > >> +1 here too. That looks like a great solution. ?Of course, once it's > >> in test_support, we need to fix *all* file removals in the test suite. > >> ;-) > > >> Steve > > > Why not just modifying test_support.unlink() like this? > > Surely more convenient than modifying the whole suite. > > > def unlink(filename): > > ? ? try: > > ? ? ? ? if os.name == 'nt': > > ? ? ? ? ? ? os.rename(filename, filename + ".deleted") > > ? ? ? ? ? ? filename = filename + ".deleted" > > ? ? ? ? os.unlink(filename) > > ? ? except OSError: > > ? ? ? ? pass > > Funnily enough, that's exactly what the patch I've > put together does. Sorry but maybe I misunderstood what you said above. It seems to me you proposed to add a new "remove_file" function to test_support while the solution I suggested was modifying the current test_support.unlink() function in a similar manner you proposed and have all tests use it wherever it is possible. From mail at timgolden.me.uk Tue Apr 1 21:47:29 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 01 Apr 2008 20:47:29 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <0a4b21eb-50cf-45fb-bc86-0e6a490bf84f@y24g2000hsd.googlegroups.com> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <02680572-1a12-4281-8abe-63b8a97b2ac9@i12g2000prf.googlegroups.com> <47F287A3.3090703@timgolden.me.uk> <0a4b21eb-50cf-45fb-bc86-0e6a490bf84f@y24g2000hsd.googlegroups.com> Message-ID: <47F29151.3000603@timgolden.me.uk> Giampaolo Rodola' wrote: >>> Why not just modifying test_support.unlink() like this? >>> Surely more convenient than modifying the whole suite. >>> def unlink(filename): >>> try: >>> if os.name == 'nt': >>> os.rename(filename, filename + ".deleted") >>> filename = filename + ".deleted" >>> os.unlink(filename) >>> except OSError: >>> pass Tim Golden >> Funnily enough, that's exactly what the patch I've >> put together does. > > Sorry but maybe I misunderstood what you said above. > It seems to me you proposed to add a new "remove_file" function to > test_support while the solution I suggested was modifying the current > test_support.unlink() function in a similar manner you proposed and > have all tests use it wherever it is possible. I wasn't very clear. What I posted earlier was by way of a proof-of-concept and did indeed use a "remove" function. After other people's +1 I have put together a possible patch which does what you're suggesting. I haven't posted it yet and I need to work through the test files checking what the impact would be. I like your second suggestion BTW, altho' I'd probably skip the "if os.name == 'nt'" line and just give it a try in any case. TJG From mail at timgolden.me.uk Tue Apr 1 21:58:53 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 01 Apr 2008 20:58:53 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F287A3.3090703@timgolden.me.uk> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <02680572-1a12-4281-8abe-63b8a97b2ac9@i12g2000prf.googlegroups.com> <47F287A3.3090703@timgolden.me.uk> Message-ID: <47F293FD.5010809@timgolden.me.uk> Tim Golden wrote: > Giampaolo Rodola' wrote: >> On 1 Apr, 18:27, "Steven Bethard" wrote: >>> On Tue, Apr 1, 2008 at 10:20 AM, Facundo Batista >>> >>> wrote: >>>> 2008/4/1, Tim Golden : >>>> > If this is the thing to do, presumably test_support should >>>> > grow a "remove_file" which does something of this sort? >>>> +1 (I was thinking exactly that). >>> +1 here too. That looks like a great solution. Of course, once it's >>> in test_support, we need to fix *all* file removals in the test suite. >>> ;-) >>> >>> Steve >> Why not just modifying test_support.unlink() like this? >> Surely more convenient than modifying the whole suite. >> >> >> def unlink(filename): >> try: >> if os.name == 'nt': >> os.rename(filename, filename + ".deleted") >> filename = filename + ".deleted" >> os.unlink(filename) >> except OSError: >> pass > > Funnily enough, that's exactly what the patch I've > put together does. Have to look out, though for > anywhere which just calls os.unlink rather than > going through test_support. Which turns out to be at least 117 instances: C:\work-in-progress\python26\Lib\test>grep "os.unlink" *.py | wc -l 117 (including the patch I just posted for os.access. Oops). TJG From schmir at gmail.com Tue Apr 1 22:34:41 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Tue, 1 Apr 2008 22:34:41 +0200 Subject: [Python-Dev] test_signal on osx g4 In-Reply-To: <18418.20280.404970.167329@montanaro-dyndns-org.local> References: <18418.20280.404970.167329@montanaro-dyndns-org.local> Message-ID: <932f8baf0804011334n722a36ecgb581af9eb1e98447@mail.gmail.com> this is http://bugs.python.org/issue1068268 On Tue, Apr 1, 2008 at 5:05 PM, wrote: > test_signal is failing on osx g4: > > test test_signal failed -- Traceback (most recent call last): > File > "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 151, > in test_main > self.fail(tb) > AssertionError: Traceback (most recent call last): > File > "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 134, > in test_main > self.run_test() > File > "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 80, > in run_test > child = subprocess.Popen(['kill', '-HUP', str(pid)]) > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", > line 593, in __init__ > errread, errwrite) > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", > line 1078, in _execute_child > data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB > OSError: [Errno 4] Interrupted system call > > This is the code which reads stderr from the child process: > > data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB > os.close(errpipe_read) > if data != "": > os.waitpid(self.pid, 0) > child_exception = pickle.loads(data) > raise child_exception > > I'm not an expert in this stuff my any stretch of the imagination, but > shouldn't subprocess try harder to read this output? Something like: > > while True: > try: > data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 > MB > except OSError, err: > if err.errno == errno.EINTR: > continue > else: > raise > else: > os.close(errpipe_read) > if data != "": > os.waitpid(self.pid, 0) > child_exception = pickle.loads(data) > raise child_exception > break > > Maybe not while True, but try a few times at least. > > Or is the system supposed to automatically restart interrupted system > calls? > > Skip > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/schmir%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080401/75d5b1c0/attachment.htm From schmir at gmail.com Tue Apr 1 22:45:56 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Tue, 1 Apr 2008 22:45:56 +0200 Subject: [Python-Dev] xmlrpclib and dates before 1900 Message-ID: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> Hi all, anyone care to take a look at: http://bugs.python.org/issue2014 It's about xmlrpclib not being able to send datetime objects with dates before 1900. I would like to see this go in and would also like to work on http://bugs.python.org/issue1745722 (xmlrpc wsgi support). But this only makes sense if someone also commits it... Regards, - Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080401/47257f05/attachment.htm From amauryfa at gmail.com Tue Apr 1 22:49:25 2008 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Tue, 1 Apr 2008 22:49:25 +0200 Subject: [Python-Dev] Tracing at a more granular level (branch coverage) In-Reply-To: References: Message-ID: Hello, On Tue, Apr 1, 2008 at 9:02 PM, m h wrote: > Howdy Folks- > > I hope this is not too off topic. Sadly the current sys.settrace only > allows tracing at the line level. This isn't sufficient to do branch > and path coverage. The main problem is that many boolean operations > can appear on a single line, and so you can't be sure which > conditional branch was taken using the current tracing method. > > After speaking about code complexity/testing at Pycon [0], with a few > people there, and seeing a few people have made some strides or shown > interested in metrics/code graphs/flow and branch coverage I think > that there is interest in this. > > One proposed method of getting branch coverage was to use Dalke's > Python4Ply [1] to translate code so that all branches occur on their > own line. Then using line coverage on that and converting it back to > branch coverage of the original code. > > The maintainer of coverage.py suggested that we look into patching > python instead to trace at a more granular level. His feeling was > that there would be too many corner cases and the translation would > get hairy quite quickly. > > Sadly in my 8 years of python experience I've yet to touch any c based > guts of python. I'm looking for advice on how to get finer grain > tracing with sys.settrace. > > Any advice or suggestions? There is a quorum of people (at least 5 > others) who would be very interested in this functionality because it > could lead to some cool tools built on top of it. (With the end goal > that python code be cleaner, simpler and better tested). You can start by looking at the file Python/ceval.c, which contains the interpreter loop. More precisely, the function maybe_call_line_trace() is responsible to call sys.settrace. I think you will have to change the logic there. -- Amaury Forgeot d'Arc From martin at v.loewis.de Tue Apr 1 23:48:35 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 01 Apr 2008 23:48:35 +0200 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> Message-ID: <47F2ADB3.8050802@v.loewis.de> > anyone care to take a look at: > http://bugs.python.org/issue2014 > It's about xmlrpclib not being able to send datetime objects with dates > before 1900. > I would like to see this go in and would also like to work on > http://bugs.python.org/issue1745722 > (xmlrpc wsgi support). > But this only makes sense if someone also commits it... Can you please explain why this is an important problem? Dates before 1900 have all passed long ago, so they shouldn't occur that often in real applications. Regards, Martin From martin at v.loewis.de Wed Apr 2 01:12:35 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 02 Apr 2008 01:12:35 +0200 Subject: [Python-Dev] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <00c801c893dd$32d03320$98709960$@com.au> References: <002401c892ff$a8254630$f86fd290$@com.au> <00c801c893dd$32d03320$98709960$@com.au> Message-ID: <47F2C163.9000402@v.loewis.de> > Currently, the "official" (by way of being de-facto) directory structure for > a build tree is 'PCBuild/.' for x86 builds and 'PCBuild/amd64' for x64 > platforms. I believe this might cause problems for people trying to port > their applications to 64bit platforms. My proposal is that we change this > subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' exist, while > 'PCBuild/.' is always the 'native' platform - ie, that 'PCBuild/.' be a copy > of one of the platform subdirectories. Can you explain what you mean by "native platform"? I usually perform cross-builds for amd64, so would then this directory remain empty? > The rationale is fairly simple: I'm quite certain that this new directory > layout will break existing "native only" build processes (ie, those that > aren't aware of cross-compilation). I'm quite certain that Mozilla will > break, for example, and no cross-compilation process exists that I am aware > of. Existing build tools already know about the PCBuild directory, and are > focused almost exclusively towards native compilation - all such tools on > non-x86 platforms are currently broken. The reverse may also be true: some tools may expect PCbuild to contain always x86 binaries, even on AMD64 - as they don't support non-x86 at all. Those tools might break if PCbuild suddenly starts containing AMD64 binaries. If this is all about *just* Mozilla, then I'd be much more in favour than if it was for some theoretical package. So if Mozilla builds with that kind of setup, and works correctly - go for it. If you have to fix Mozilla anyway, consider fixing it so it looks in the amd64 directory. Regards, Martin From guido at python.org Wed Apr 2 02:09:24 2008 From: guido at python.org (Guido van Rossum) Date: Tue, 1 Apr 2008 17:09:24 -0700 Subject: [Python-Dev] Thread-safe file objects, the return In-Reply-To: References: Message-ID: This is not something that keeps me awake at night, but I am aware of it. Your solution (a counter) seems fine except I think perhaps the close() call should not raise IOError -- instead, it should set a flag so that the thread that makes the counter go to zero can close the thread (after all the file got closed while it was being used). There are of course other concurrency issues besides close -- what if two threads both try to do I/O on the file? What will the C stdio library do in that case? Are stdio files thread-safe at the C level? So (classically contradicting myself while I think the problem over more) perhaps any I/O operation should be disallowed while the file is in use by another thread? --Guido On Mon, Mar 31, 2008 at 1:09 PM, Antoine Pitrou wrote: > > Hello, > > It seems this subject has had quite a bit of history. Tim Peters demonstrated > the problem in 2003 in this message: > http://mail.python.org/pipermail/python-dev/2003-June/036537.html > > In short, Python file objects release the GIL before calling any C stdlib > function on their embedded FILE pointer. Unfortunately, if another thread > calls fclose on the FILE pointer concurrently, the contents pointed to can > become garbage and the interpreter process crashes. Just by using the same > file object in two threads running pure Python code, you can crash the > interpreter. > > (another, easier-to-solve problem is that the FILE pointer stored in the > file object could become NULL at the point it is used by another thread. > If that was the only problem you could just store the FILE pointer in a > local variable before releasing the GIL et voil?) > > There was some discussion at the time about the possible resolution. I've > tried to fix the problem, and I've come to what I think is a satisfying > solution, which I can sum up as the following bullet points: > * Each file object gets a dedicated counter, which is incremented before > the bject releases the GIL and decremented after the GIL is taken again; thus > this counter keeps track of how many running "unlocked" sections of code are > using that particular file object. (please note the counter doesn't need its > own lock, since it is only modified in GIL-protected sections) > * In the close() method, if the aforementioned counter is greater than 0, > we refuse to call fclose and instead raise an IOError. > > This may seem like a worrying semantic change, but I don't think it is, for the > following reasons: > 1) if we closed the FILE pointer anyway, the interpreter would likely crash > because another thread would be using garbage data (that's what we are trying > to fix after all!) > 2) if close() raises an IOError, it can be called again later, or at worse > fclose will be called when the file object is garbage collected > 3) close() can already raise an IOError if fclose fails for whatever reason > (although for sure it's probably very rare) > 4) it doesn't seem wrong to notify the programmer that his code is very > unsafe > > The patch is attached at http://bugs.python.org/issue815646 . It addresses > (or at least I hope it does) all potential problems with pure Python code, > threads, and the file object. It doesn't try to fix C extensions using the > PyFile_AsFile API and doing their own dirty things with the FILE pointer. It > could be a second step if the approach is accepted, but as noted in the 2003 > discussions it would probably involve a new API. Whether we want to introduce > such an API in Python 2.x while Python 3.0 has a different IO model anyway > is left open to discussion :) > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mhammond at skippinet.com.au Wed Apr 2 03:26:22 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Wed, 2 Apr 2008 12:26:22 +1100 Subject: [Python-Dev] [Distutils] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <47F2C163.9000402@v.loewis.de> References: <002401c892ff$a8254630$f86fd290$@com.au> <00c801c893dd$32d03320$98709960$@com.au> <47F2C163.9000402@v.loewis.de> Message-ID: <016001c89460$944f10c0$bced3240$@com.au> Martin quoting me: > > Currently, the "official" (by way of being de-facto) directory > > structure for a build tree is 'PCBuild/.' for x86 builds and > > 'PCBuild/amd64' for x64 > > platforms. I believe this might cause problems for people trying to > > port their applications to 64bit platforms. My proposal is that we > > change this subtly - that both 'PCBuild/x86', and 'PCBuild/amd64' > > exist, while 'PCBuild/.' is always the 'native' platform - ie, > > that 'PCBuild/.' be a copy of one of the platform subdirectories. > > Can you explain what you mean by "native platform"? I usually perform > cross-builds for amd64, so would then this directory remain empty? By "native platform", I mean it contains binaries that are native for the current platform. IOW, on x86 platforms, PCBuild would contain x86 binaries, and binaries compiled for AMD64 would live in the AMD64 directory. If in a given directory tree, you *only* did a cross-compile, then yes, PCBuild would remain empty - but I don't think that is a problem; tools that know how to cross-compile (such as the distutils after my patch is applied) will not look in the PCBuild directory, and tools that do not cross-compile will be unable to use the x64 binaries from the x86 platform anyway. In other words, I can't think of anything that would break by having an empty PCBuild directory in a tree that only contains binaries that can't be executed on the current platform. > > The rationale is fairly simple: I'm quite certain that this new > > directory layout will break existing "native only" build processes > > (ie, those that aren't aware of cross-compilation). I'm quite > > certain that Mozilla will break, for example, and no > > cross-compilation process exists that I am aware > > of. Existing build tools already know about the PCBuild directory, > > and are focused almost exclusively towards native compilation - all > > such tools on non-x86 platforms are currently broken. > > The reverse may also be true: some tools may expect PCbuild to contain > always x86 binaries, even on AMD64 - as they don't support non-x86 at > all. Those tools might break if PCbuild suddenly starts containing > AMD64 binaries. I agree. However, it is my assertion that there are very few build tools which expect the layout you describe, simply as the scheme has only been around for a few months, and only in Python 2.6 builds. Further, I assert that there are a greater number of build tools which do not support cross-compilation, but will build natively on x64 and expect 'PCBuild' to have libraries they can link with to create an x64 binary. I accept my assertions may be incorrect - in which case keeping the existing layout is indeed the best options. But I assert that my assertions are correct :) > If this is all about *just* Mozilla, then I'd be much more in favour > than if it was for some theoretical package. So if Mozilla builds with > that kind of setup, and works correctly - go for it. If you have to > fix Mozilla anyway, consider fixing it so it looks in the amd64 > directory. Nope - it's not about *just* Mozilla at all, and I don't expect that I personally will be involved in any 64bit issues for that platform. That just happens to be one build process that I know about, but I except it does some "typical" things other tools might do - like executing the python executable to sniff sys.prefix and looking for (eg) "Libs" and "PCBuild" to locate libraries to link with seems fairly likely to me. Even looking for the executable itself directly in PCBuild seems likely. On an x64 platform, such tools may well find the 32bit version, and it may even seem to work - but it is unlikely to be doing what they expect. Or they may find no binary at all. It seems that you previously agreed with this. From http://mail.python.org/pipermail/distutils-sig/2007-May/007668.html; you are replying to (I think) Kristj?n: | > I am baffled about why the build environment's layout matters, | > but once an .msi install can place the binaries in any | > old place it wants. The build structure doesn't have to | > reflect the final installed structure at all. | No. But still, people like to be able to "run" Python out of | a source check-out. This has been supported for a long time, | and more and more stuff was added to support it. For examples | within Python itself, see the support in distutils, getpathp.c, | PCbuild/rt.bat, and Tools/buildbot/*.bat. Reportedly (by | Mark), building Mozilla (the web browser) also "knows" | about PCbuild. Your comments exactly reflect my concern, and I believe them relevant for people working with native binaries on x64. But as implied above, I actually have zero personal interest in this at the moment - unlike the distutils cross-compile patch, which does scratch an itch of mine. It just seems it would make life easier for people down the track - but I'm happy to let it go if your position has changed or I have mis-understood it. Thanks, Mark From martin at v.loewis.de Wed Apr 2 04:15:22 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 02 Apr 2008 04:15:22 +0200 Subject: [Python-Dev] [Distutils] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <016001c89460$944f10c0$bced3240$@com.au> References: <002401c892ff$a8254630$f86fd290$@com.au> <00c801c893dd$32d03320$98709960$@com.au> <47F2C163.9000402@v.loewis.de> <016001c89460$944f10c0$bced3240$@com.au> Message-ID: <47F2EC3A.9090300@v.loewis.de> >> The reverse may also be true: some tools may expect PCbuild to contain >> always x86 binaries, even on AMD64 - as they don't support non-x86 at >> all. Those tools might break if PCbuild suddenly starts containing >> AMD64 binaries. > > I agree. However, it is my assertion that there are very few build tools > which expect the layout you describe, simply as the scheme has only been > around for a few months, and only in Python 2.6 builds. That PCbuild exists and contains x86 binaries? This scheme has been in place ever since the PCbuild directory got created, several years ago! > Nope - it's not about *just* Mozilla at all, and I don't expect that I > personally will be involved in any 64bit issues for that platform. That > just happens to be one build process that I know about, but I except it does > some "typical" things other tools might do - like executing the python > executable to sniff sys.prefix and looking for (eg) "Libs" and "PCBuild" to > locate libraries to link with seems fairly likely to me. Even looking for > the executable itself directly in PCBuild seems likely. On an x64 platform, > such tools may well find the 32bit version, and it may even seem to work - > but it is unlikely to be doing what they expect. Unless the *want* the x86 binaries, of course, which always had been in this place. > Your comments exactly reflect my concern, and I believe them relevant for > people working with native binaries on x64. No, I never cared about what binaries are "native". Instead, I wanted to see those binaries in PCbuild which had been built last. If you just built the x86 binaries, PCbuild should contain x86 binaries, even if you were running Win64. This is different from what you propose, but only slightly so (but perhaps importantly, depending on the scenario). > But as implied above, I > actually have zero personal interest in this at the moment - unlike the > distutils cross-compile patch, which does scratch an itch of mine. It just > seems it would make life easier for people down the track - but I'm happy to > let it go if your position has changed or I have mis-understood it. OK, if we don't have an actual use case, I suggest to leave things as is. People running into this problem should propose solutions themselves. Regards, Martin From jyasskin at gmail.com Wed Apr 2 06:10:22 2008 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Tue, 1 Apr 2008 21:10:22 -0700 Subject: [Python-Dev] test_signal on osx g4 In-Reply-To: <18418.20280.404970.167329@montanaro-dyndns-org.local> References: <18418.20280.404970.167329@montanaro-dyndns-org.local> Message-ID: <5d44f72f0804012110s32b43d5erc14a7b3afeb08f3b@mail.gmail.com> I've tried to fix test_signal at least. For those particular calls, EINTR means that the kill I launched finished before I was expecting, so we can ignore it without retrying. Figuring out subprocess in general is a worthy goal but shouldn't block making the test less flaky. :) r62102. On Tue, Apr 1, 2008 at 8:05 AM, wrote: > test_signal is failing on osx g4: > > test test_signal failed -- Traceback (most recent call last): > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 151, in test_main > self.fail(tb) > AssertionError: Traceback (most recent call last): > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 134, in test_main > self.run_test() > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/test/test_signal.py", line 80, in run_test > child = subprocess.Popen(['kill', '-HUP', str(pid)]) > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", line 593, in __init__ > errread, errwrite) > File "/Users/buildslave/bb/trunk.psf-g4/build/Lib/subprocess.py", line 1078, in _execute_child > data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB > OSError: [Errno 4] Interrupted system call > > This is the code which reads stderr from the child process: > > data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB > os.close(errpipe_read) > if data != "": > os.waitpid(self.pid, 0) > child_exception = pickle.loads(data) > raise child_exception > > I'm not an expert in this stuff my any stretch of the imagination, but > shouldn't subprocess try harder to read this output? Something like: > > while True: > try: > data = os.read(errpipe_read, 1048576) # Exceptions limited to 1 MB > except OSError, err: > if err.errno == errno.EINTR: > continue > else: > raise > else: > os.close(errpipe_read) > if data != "": > os.waitpid(self.pid, 0) > child_exception = pickle.loads(data) > raise child_exception > break > > Maybe not while True, but try a few times at least. > > Or is the system supposed to automatically restart interrupted system calls? > > Skip > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com > -- Namast?, Jeffrey Yasskin http://jeffrey.yasskin.info/ From greg at krypto.org Wed Apr 2 08:19:17 2008 From: greg at krypto.org (Gregory P. Smith) Date: Tue, 1 Apr 2008 23:19:17 -0700 Subject: [Python-Dev] Thread-safe file objects, the return In-Reply-To: References: Message-ID: <52dc1c820804012319q6b548fb1r28df92ab7dcc319e@mail.gmail.com> On Tue, Apr 1, 2008 at 5:09 PM, Guido van Rossum wrote: > This is not something that keeps me awake at night, but I am aware of > it. Your solution (a counter) seems fine except I think perhaps the > close() call should not raise IOError -- instead, it should set a flag > so that the thread that makes the counter go to zero can close the > thread (after all the file got closed while it was being used). > No, raising IOError is the right thing to do here. The problem is that calling close on a file implies that the close actually completed in the OS when it returns. The call should not return until the file object has actually been closed in the underlying layers. You can't leave it for later to be done by one of the other currently operating threads as you violate the close semantics and lose direct indication of an error that occurred during close. > There are of course other concurrency issues besides close -- what if > two threads both try to do I/O on the file? What will the C stdio > library do in that case? Are stdio files thread-safe at the C level? > So (classically contradicting myself while I think the problem over > more) perhaps any I/O operation should be disallowed while the file is > in use by another thread? > that does sound like the safest thing to do... -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080401/908e21e2/attachment-0001.htm From greg at krypto.org Wed Apr 2 08:26:20 2008 From: greg at krypto.org (Gregory P. Smith) Date: Tue, 1 Apr 2008 23:26:20 -0700 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F25F2E.4070408@timgolden.me.uk> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> Message-ID: <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> On Tue, Apr 1, 2008 at 9:13 AM, Tim Golden wrote: > Tim Golden wrote: > > Steven Bethard wrote: > >> At the sprints, I ran into a bunch of similar errors running the test > >> suite on my Windows Vista box, even on tests that were properly > >> cleaning up after themselves in tearDown(). I even tried putting in > >> sleeps as long as 1 second, to no avail. The only way to get the test > >> suite to run without these errors was to stop the Windows Search > >> Service, fully disable Icon Overlays for TortoiseSVN, and then close > >> down all open folders. > >> > >> Any chance the boxes the tests are being run on are running the > >> Windows Search Service or have Icon Overlays enabled for TortoiseSVN? > >> > >> (If anyone has any ideas of how to get around these problems, I'd love > >> to hear them. Seems like I shouldn't have to disable these services.) > > > > I'm not sure I'm going to help here, but the reason that > > this happens is that certain services -- and the Desktop > > Search tools are notorious for this -- get file handles with > > FILE_SHARE_DELETE, which means that the file isn't actually > > removed when it's deleted; only when the last of those handles > > closes. > > > > I did see a suggestion somewhere that, to circumvent this issue, > > instead of removing it a file you need to move it to %TEMP% > > (or wherever) and delete it there. I'll try to knock up a test > > case to see if this works. > > OK: to confirm, the following test seems to indicate that > doing a (local - same volume) rename followed by a remove > will do the right thing, even when a FILE_SHARE_DELETE handle > is held. > > If this is the thing to do, presumably test_support should > grow a "remove_file" which does something of this sort? > > TJG > > > import os, sys > import win32file > > FILENAME = "test" > > def rename_and_remove (filename): > os.rename (filename, filename + ".deleted") > os.remove (filename + ".deleted") Isn't this still going to run into problems when the rename fails because the earlier tests remove still left the .deleted file around due to some other running desktop search service that now has the .deleted file open? +1 on not supporting OSes with such bad filesystem semantics? ;) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080401/c87dfe84/attachment.htm From tnelson at onresolve.com Wed Apr 2 09:00:41 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 2 Apr 2008 00:00:41 -0700 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381B1@EXMBX04.exchhosting.com> > def rename_and_remove (filename): > os.rename (filename, filename + ".deleted") > os.remove (filename + ".deleted") > Isn't this still going to run into problems when the rename > fails because the earlier tests remove still left the .deleted > file around due to some other running desktop search service > that now has the .deleted file open? I haven't looked into all the various places the tests write temp files to, but if we could localise everything to a common root directory, i.e. %TEMP%\python-regrtest, we could then attempt to blow this away at the start of regrtest.py before any tests run, and refuse to run if this fails. This would be in combination with the unlinking/renaming approach discussed. This approach seems like it would cover all bases a bit more effectively. Trent. From tnelson at onresolve.com Wed Apr 2 09:12:05 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 2 Apr 2008 00:12:05 -0700 Subject: [Python-Dev] No time for svn merge In-Reply-To: <47F07068.3060503@v.loewis.de> References: <47EFFD64.8020006@v.loewis.de> <1afaf6160803301416n6e37df7cw5ff39a55ddb7341@mail.gmail.com> <47F00C33.1050503@v.loewis.de> <5c6f2a5d0803301507m25455babm7254b9cb62905dfe@mail.gmail.com> <47F0113D.8020408@v.loewis.de> <5c6f2a5d0803301533l1d79be9bh6033bb3dfafb485a@mail.gmail.com> <47F07068.3060503@v.loewis.de> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381B6@EXMBX04.exchhosting.com> > > Yes, that's all I meant: make it the committer's job > > to merge or block as appropriate. I just wasn't sure if > > there was some reason that this would be difficult or > > undesirable. > > Ah, yes. It is indeed difficult or undesirable, or was > so in the past: Some committers don't care (didn't care) > at all about 3k. They would have to setup sandboxes, > learn what the nature of changes is, and invest some > regular time into forward-porting. Is this *really* the case still? Who are these rogue committers? ;-) I think holding a developer accountable for merging or blocking to py3k when they commit to trunk is a great idea. Who better to pass judgement on such an activity than the person closest to it? Trent. From tnelson at onresolve.com Wed Apr 2 09:37:14 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 2 Apr 2008 00:37:14 -0700 Subject: [Python-Dev] [Distutils] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <016001c89460$944f10c0$bced3240$@com.au> References: <002401c892ff$a8254630$f86fd290$@com.au> <00c801c893dd$32d03320$98709960$@com.au> <47F2C163.9000402@v.loewis.de> <016001c89460$944f10c0$bced3240$@com.au> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381BA@EXMBX04.exchhosting.com> > Further, I > assert that there are a greater number of build tools which do not support > cross-compilation, but will build natively on x64 and expect 'PCBuild' > to have libraries they can link with to create an x64 binary. I'm with Martin on this one as well I think. If I understand correctly, you're proposing: PCbuild - may have contents of x86 or x64 depending on the build machine's architecture PCbuild/amd64 - always x64 PCbuild/x86 - always x86 And what we've currently got is: PCbuild/ - always x86 PCbuild/amd64 - always x64 I think this is fine; we don't really have a notion of compiling for a native platform, nor is the build machine's architecture factored into the equation. I believe this is a Good Thing. If you want a 32-bit build, use the 'Win32' target platform in VS; if you want a 64-bit build, use 'x64'. Trent. From tnelson at onresolve.com Wed Apr 2 09:58:35 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 2 Apr 2008 00:58:35 -0700 Subject: [Python-Dev] [Python-3000] the release gods are angry at python In-Reply-To: References: <47EA72D4.8000709@cheimes.de> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381BF@EXMBX04.exchhosting.com> > > In the py3k branch I've assigned the audio resource to the winsound > > tests. Only regrtest.py -uall or -uaudio runs the winsound test. > Reason: > > the test sound was freaking out my poor cat. :/ > > I feel with your cat ;-). > This would not help on the buildbot since it runs 'rt.bat -d -q -uall - > rw'. I feel for the poor NOC engineers at my colo that freak out when some random server in a farm of thousands starts making bizarre sounds. I detest test_winsound. There are so many corner cases you need to account for that makes the test pointless as you end up wrapping everything in except: pass blocks. Does the system have a legacy beep driver? Is it enabled? Is it disabled? Is there a sound card? Is it enabled or disabled? Pah! +1 to removing audio out of -uall, if only for the sake of cats, erroneously red buildbots, and poor ServerCentral NOC engineers. Trent. From mail at timgolden.me.uk Wed Apr 2 10:03:51 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Wed, 02 Apr 2008 09:03:51 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> Message-ID: <47F33DE7.3030702@timgolden.me.uk> Gregory P. Smith wrote: > On Tue, Apr 1, 2008 at 9:13 AM, Tim Golden wrote: > >> Tim Golden wrote: >> >> import os, sys >> import win32file >> >> FILENAME = "test" >> >> def rename_and_remove (filename): >> os.rename (filename, filename + ".deleted") >> os.remove (filename + ".deleted") > Isn't this still going to run into problems when the rename fails because > the earlier tests remove still left the .deleted file around due to some > other running desktop search service that now has the .deleted file open? I admit: this did occur to me on the train this am. While I try to think of a robust way to handle this, other people have proposed variations on pid-based / tempdir based filenames instead of the same name for each test. In principle this sounds good to me, but I'm not at all well-placed to assess the impact it might have on the unit tests in general. TJG From lists at cheimes.de Wed Apr 2 10:57:41 2008 From: lists at cheimes.de (Christian Heimes) Date: Wed, 02 Apr 2008 10:57:41 +0200 Subject: [Python-Dev] No time for svn merge In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381B6@EXMBX04.exchhosting.com> References: <47EFFD64.8020006@v.loewis.de> <1afaf6160803301416n6e37df7cw5ff39a55ddb7341@mail.gmail.com> <47F00C33.1050503@v.loewis.de> <5c6f2a5d0803301507m25455babm7254b9cb62905dfe@mail.gmail.com> <47F0113D.8020408@v.loewis.de> <5c6f2a5d0803301533l1d79be9bh6033bb3dfafb485a@mail.gmail.com> <47F07068.3060503@v.loewis.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381B6@EXMBX04.exchhosting.com> Message-ID: <47F34A85.5020909@cheimes.de> Trent Nelson schrieb: > I think holding a developer accountable for merging or blocking to py3k when they commit to trunk is a great idea. Who better to pass judgement on such an activity than the person closest to it? Blocking a revision makes my job as The Merger easier. I'm not so sure about the merging part. It takes some experience with the Python 3.0 code base to know the subtle differences in the C API. Most merges are straight forward for me. If you enforce the forward merging on every developer it may slow down development. Each regular merge takes me about 45 minutes of computer time but less than 15 supervisor time. The computer time is mostly compile and test time in the background. If everybody merges her own code to 3.0 it still takes *everybody* about 10 minutes of time and 45 minutes of computer time. Christian From amauryfa at gmail.com Wed Apr 2 11:07:03 2008 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 2 Apr 2008 11:07:03 +0200 Subject: [Python-Dev] [Python-3000] the release gods are angry at python In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381BF@EXMBX04.exchhosting.com> References: <47EA72D4.8000709@cheimes.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381BF@EXMBX04.exchhosting.com> Message-ID: On Wed, Apr 2, 2008 at 9:58 AM, Trent Nelson wrote: > > > In the py3k branch I've assigned the audio resource to the winsound > > > tests. Only regrtest.py -uall or -uaudio runs the winsound test. > > Reason: > > > the test sound was freaking out my poor cat. :/ > > > > I feel with your cat ;-). > > This would not help on the buildbot since it runs 'rt.bat -d -q -uall - > > rw'. > > I feel for the poor NOC engineers at my colo that freak out when some random server in a farm of thousands starts making bizarre sounds. > > I detest test_winsound. There are so many corner cases you need to account for that makes the test pointless as you end up wrapping everything in except: pass blocks. Does the system have a legacy beep driver? Is it enabled? Is it disabled? Is there a sound card? Is it enabled or disabled? Pah! > > +1 to removing audio out of -uall, if only for the sake of cats, erroneously red buildbots, and poor ServerCentral NOC engineers. And I would not mind removing this module altogether, and provide a ctypes implementation. -- Amaury Forgeot d'Arc From ncoghlan at gmail.com Wed Apr 2 11:13:24 2008 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 02 Apr 2008 19:13:24 +1000 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F33DE7.3030702@timgolden.me.uk> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk> Message-ID: <47F34E34.9010507@gmail.com> Tim Golden wrote: > I admit: this did occur to me on the train this am. While I > try to think of a robust way to handle this, other people have > proposed variations on pid-based / tempdir based filenames instead > of the same name for each test. In principle this sounds good to me, > but I'm not at all well-placed to assess the impact it might have > on the unit tests in general. Personally, I've never really understood the purpose of test_support.TESTFN. Whenever I've needed a temporary file for a test, I just use the tempfile module (e.g. test_cmd_line_script, test_runpy). Tests using that module don't care if the old files take 'a while' to get deleted on Windows, as tempfile uses a different name each time anyway. Is using a fixed TESTFN just an old approach that predates the existence of a robust tempfile module in the standard library? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From mail at timgolden.me.uk Wed Apr 2 11:24:22 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Wed, 02 Apr 2008 10:24:22 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F34E34.9010507@gmail.com> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk> <47F34E34.9010507@gmail.com> Message-ID: <47F350C6.3010209@timgolden.me.uk> Nick Coghlan wrote: > Tim Golden wrote: >> I admit: this did occur to me on the train this am. While I >> try to think of a robust way to handle this, other people have >> proposed variations on pid-based / tempdir based filenames instead >> of the same name for each test. In principle this sounds good to me, >> but I'm not at all well-placed to assess the impact it might have >> on the unit tests in general. > > Personally, I've never really understood the purpose of > test_support.TESTFN. Whenever I've needed a temporary file for a test, I > just use the tempfile module (e.g. test_cmd_line_script, test_runpy). > Tests using that module don't care if the old files take 'a while' to > get deleted on Windows, as tempfile uses a different name each time anyway. > > Is using a fixed TESTFN just an old approach that predates the existence > of a robust tempfile module in the standard library? I'm a neophyte when it comes to core development, so I've simply cloned existing tests, assumed that there was some kind of (possibly unwritten) standard which used test_support.TESTFN. As I look at it, though it seems a slightly odd choice, although it has variants for testing unicode filenames specifically which I imagine are useful in some places. I'm perfectly happy to run through the test suite, patching it one way or another. The trouble is that I've little confidence that I can assess whether or not such a change will have affected the actual meaning of a test. And, since these are tests, Quis custodiet...? TJG From lists at cheimes.de Wed Apr 2 11:42:04 2008 From: lists at cheimes.de (Christian Heimes) Date: Wed, 02 Apr 2008 11:42:04 +0200 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: <47F2ADB3.8050802@v.loewis.de> References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> <47F2ADB3.8050802@v.loewis.de> Message-ID: Martin v. L?wis schrieb: > Can you please explain why this is an important problem? > Dates before 1900 have all passed long ago, so they shouldn't > occur that often in real applications. Does xmlrpc support dates for 1900? For historic dates the Julian Day Number family (MJD or JDN) or Rata Die are more appropriate and much easier to use. I wish somebody could add both to the datetime module. Christian From mail at timgolden.me.uk Wed Apr 2 12:00:13 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Wed, 02 Apr 2008 11:00:13 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F33DE7.3030702@timgolden.me.uk> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk> Message-ID: <47F3592D.3080700@timgolden.me.uk> Tim Golden wrote: > Gregory P. Smith wrote: >> On Tue, Apr 1, 2008 at 9:13 AM, Tim Golden wrote: >> >>> Tim Golden wrote: > >>> >>> import os, sys >>> import win32file >>> >>> FILENAME = "test" >>> >>> def rename_and_remove (filename): >>> os.rename (filename, filename + ".deleted") >>> os.remove (filename + ".deleted") > >> Isn't this still going to run into problems when the rename fails because >> the earlier tests remove still left the .deleted file around due to some >> other running desktop search service that now has the .deleted file open? I've looked around this issue as far as I can, and I can't come up with a foolproof way of guaranteeing that this won't happen. I considered, eg, trying to open the test file with semantics which preclude FILE_SHARE_DELETE from succeeding (ie stopping the search services / TSVN from getting a handle at all) but I can't find anything which doesn't cause more problems than it solves. Lest this discussion descend into inactivity, I've done a fresh checkout and I'm producing a patch which will add open_testfile and close_testfile functions to test_support (or something like that) and which will update all the tests to use those. I'm inclined to go with the tempfile approach but at least having the functions means that that could change. This is obviously a wide-reaching change, but unless anyone's got better ideas, the alternative seems to be intermittently-failing test runs on the Windows buildbots. TJG From solipsis at pitrou.net Wed Apr 2 12:17:48 2008 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 2 Apr 2008 10:17:48 +0000 (UTC) Subject: [Python-Dev] Thread-safe file objects, the return References: Message-ID: Guido van Rossum python.org> writes: > Your solution (a counter) seems fine except I think perhaps the > close() call should not raise IOError -- instead, it should set a flag > so that the thread that makes the counter go to zero can close the > thread (after all the file got closed while it was being used). I agree with Gregory: we should be explicit about what happens. I wonder what we would gain from that approach - apart from encouraging dangerous coding practices :) It also depends how far we want to go: I am merely proposing to fix the crashes, do we want to provide a "smarter" close() variation that does what you suggest for people that want (or need) to take the risk? > There are of course other concurrency issues besides close -- what if > two threads both try to do I/O on the file? What will the C stdio > library do in that case? Are stdio files thread-safe at the C level? According to the glibc documentation, at http://www.gnu.org/software/libc/manual/html_node/Streams-and-Threads.html : ? The POSIX standard requires that by default the stream operations are atomic. I.e., issuing two stream operations for the same stream in two threads at the same time will cause the operations to be executed as if they were issued sequentially. The buffer operations performed while reading or writing are protected from other uses of the same stream. To do this each stream has an internal lock object which has to be (implicitly) acquired before any work can be done. ? So according to POSIX rules it should be perfectly safe. In any case, someone would have to try my patch under Windows and OS X and see if test_file.py passes without crashing. Regards Antoine. From schmir at gmail.com Wed Apr 2 12:45:31 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Wed, 2 Apr 2008 12:45:31 +0200 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> <47F2ADB3.8050802@v.loewis.de> Message-ID: <932f8baf0804020345k7fdbb0c6k9f930fd88631f134@mail.gmail.com> On Wed, Apr 2, 2008 at 11:42 AM, Christian Heimes wrote: > Martin v. L?wis schrieb: > > Can you please explain why this is an important problem? > > Dates before 1900 have all passed long ago, so they shouldn't > > occur that often in real applications. > In the application where I needed it, the customer wanted to send/store dates for e.g. the date of birth of some people. > > Does xmlrpc support dates for 1900? For historic dates the Julian Day The xmlrpc spec says dates should be sent in the following format: 19980717T14:08:55 1900 is a rather arbitrary limit with this format. Note that the unpatched xmlrpclib is able to receive datetime objects with dates before 1900: ~/ /usr/bin/python2.5 ralf at redok Python 2.5.2 (r252:60911, Mar 9 2008, 11:14:55) [GCC 4.2.3 (Debian 4.2.3-2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import xmlrpclib, datetime >>> xmlrpclib.loads('\n\n18500101T00:00:00\n\n\n', use_datetime=True) ((datetime.datetime(1850, 1, 1),), None) Dumping however doesn't work: >>> xmlrpclib.dumps((datetime.datetime(1850, 1, 1),)) Traceback (most recent call last): File "", line 1, in File "/usr/lib/python2.5/xmlrpclib.py", line 1080, in dumps data = m.dumps(params) File "/usr/lib/python2.5/xmlrpclib.py", line 623, in dumps dump(v, write) File "/usr/lib/python2.5/xmlrpclib.py", line 635, in __dump f(self, value, write) File "/usr/lib/python2.5/xmlrpclib.py", line 725, in dump_datetime write(value.strftime("%Y%m%dT%H:%M:%S")) File "datetime.py", line 791, in strftime return _wrap_strftime(self, fmt, self.timetuple()) File "datetime.py", line 181, in _wrap_strftime "methods require year >= 1900" % year) ValueError: year=1850 is before 1900; the datetime strftime() methods require year >= 1900 This ValueError just shows an implementation detail. Note that it's also possible to send and receive dates before 1900 using xmlrpclib.DateTime objects. > Number family (MJD or JDN) or Rata Die are more appropriate and much > easier to use. I wish somebody could add both to the datetime module. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080402/1d4bf590/attachment.htm From tnelson at onresolve.com Wed Apr 2 14:16:51 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 2 Apr 2008 05:16:51 -0700 Subject: [Python-Dev] No time for svn merge In-Reply-To: <47F34A85.5020909@cheimes.de> References: <47EFFD64.8020006@v.loewis.de> <1afaf6160803301416n6e37df7cw5ff39a55ddb7341@mail.gmail.com> <47F00C33.1050503@v.loewis.de> <5c6f2a5d0803301507m25455babm7254b9cb62905dfe@mail.gmail.com> <47F0113D.8020408@v.loewis.de> <5c6f2a5d0803301533l1d79be9bh6033bb3dfafb485a@mail.gmail.com> <47F07068.3060503@v.loewis.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381B6@EXMBX04.exchhosting.com> <47F34A85.5020909@cheimes.de> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381F9@EXMBX04.exchhosting.com> Christian Heimes [mailto:lists at cheimes.de]: > Trent Nelson schrieb: > > I think holding a developer accountable for merging or blocking to > py3k when they commit to trunk is a great idea. Who better to pass > judgement on such an activity than the person closest to it? > > Blocking a revision makes my job as The Merger easier. > > I'm not so sure about the merging part. It takes some experience with > the Python 3.0 code base to know the subtle differences in the C API. > Most merges are straight forward for me. If you enforce the forward > merging on every developer it may slow down development. > Each regular merge takes me about 45 minutes of computer time but less > than 15 supervisor time. The computer time is mostly compile and test > time in the background. If everybody merges her own code to 3.0 it > still takes *everybody* about 10 minutes of time and 45 minutes of computer > time. Ah, right, I wasn't thinking about the implication of code affecting the C base for some reason, but that's entirely reasonable. Perhaps each developer should be accountable for either: a) blocking b) merging, if they're able to do so c) if they're unable to merge, replying to the relevant python-checkins@ e-mail indicating that they're unable to handle trunk -> py3k for whatever reason (e.g. not familiar with py3k code base) Other developers could then pitch in and help merge if someone requests it via e-mail. I'd think that would make The Merger's life easier. Trent. From tnelson at onresolve.com Wed Apr 2 18:20:21 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 2 Apr 2008 09:20:21 -0700 Subject: [Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E38501@EXMBX04.exchhosting.com> Looking into some of the recent Windows buildbot failures, I see things like this: sqlite3 : error PRJ0008 : Could not delete file 'c:\buildbot\trunk.heller-windows-amd64\build\PCbuild\amd64\sqlite3_d.dll'. build-amd64.bat doesn't go through the kill_python.c hoopla, so I figure the above error is being caused by the fact that an erroneous/stalled python_d.exe from a previous run is still open. I was looking at modifying kill_python.c to accept an 'x64' argument if we want to kill amd64\python_d.exe instead of the usual 32-bit exe, however, this caught my attention: if ((strstr(path, "pcbuild\\python_d.exe") != NULL) || (strstr(path, "\\build\\python.exe") != NULL)) { printf("Terminating %s (pid %d)\n", path, pids[i]); if (!TerminateProcess(hProcess, 1)) { printf("Termination failed: %d\n", GetLastError()); return 1; } return 0; That'll kill the first python_d.exe instance it finds matching the given path; given that our buildbots run trunk/release25-maint/py3k in parallel, it seems as though it wouldn't be hard for us to get into a situation where kill_python.exe ends up killing the wrong python_d.exe (i.e. trunk checkin, trunk builds, starts testing, py3k checkin, calls kill_python.exe, kills trunk's python_d.exe that was in the middle of testing). That can't be helping our cause, unless I'm missing something... Unless anyone advises otherwise, I'll start on a patch. Trent. From martin at v.loewis.de Wed Apr 2 20:39:41 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 02 Apr 2008 20:39:41 +0200 Subject: [Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E38501@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E38501@EXMBX04.exchhosting.com> Message-ID: <47F3D2ED.4070109@v.loewis.de> > That'll kill the first python_d.exe instance it finds matching the > given path; given that our buildbots run trunk/release25-maint/py3k > in parallel That's actually not a given: we currently *don't* run multiple builds simultaneously on the same slave. > Unless anyone advises otherwise, I'll start on a patch. If you can make it less error-prone, sure, go ahead. Regards, Martin From guido at python.org Wed Apr 2 20:47:02 2008 From: guido at python.org (Guido van Rossum) Date: Wed, 2 Apr 2008 11:47:02 -0700 Subject: [Python-Dev] Thread-safe file objects, the return In-Reply-To: References: Message-ID: On Wed, Apr 2, 2008 at 3:17 AM, Antoine Pitrou wrote: > Guido van Rossum python.org> writes: > > Your solution (a counter) seems fine except I think perhaps the > > close() call should not raise IOError -- instead, it should set a flag > > so that the thread that makes the counter go to zero can close the > > thread (after all the file got closed while it was being used). > > I agree with Gregory: we should be explicit about what happens. I wonder > what we would gain from that approach - apart from encouraging dangerous > coding practices :) > It also depends how far we want to go: I am merely proposing to fix the > crashes, do we want to provide a "smarter" close() variation that does what > you suggest for people that want (or need) to take the risk? I also agree with Gregory now -- at least on the issue of fclose(). I think that for other (non-closing) operations we should be fine, given the Posix requirement that streams have an internal lock. While multiple threads reading from a file sounds insane, multiple files writing to a file is pretty common (think of stdout or stderr) and should be supported. > > There are of course other concurrency issues besides close -- what if > > two threads both try to do I/O on the file? What will the C stdio > > library do in that case? Are stdio files thread-safe at the C level? > > According to the glibc documentation, at > http://www.gnu.org/software/libc/manual/html_node/Streams-and-Threads.html : > > ? The POSIX standard requires that by default the stream operations are > atomic. I.e., issuing two stream operations for the same stream in two > threads at the same time will cause the operations to be executed as if > they were issued sequentially. The buffer operations performed while > reading or writing are protected from other uses of the same stream. To do > this each stream has an internal lock object which has to be (implicitly) > acquired before any work can be done. ? > > So according to POSIX rules it should be perfectly safe. Cool. > In any case, someone would have to try my patch under Windows and OS X and > see if test_file.py passes without crashing. I know Windows has internal locks on stdio. I trust that OSX, being a BSD descendant, is posix compliant. So I'm not worried about these. +1 on your patch, assuming some other developer reviews it and submits it. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Wed Apr 2 20:59:40 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 02 Apr 2008 20:59:40 +0200 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F34E34.9010507@gmail.com> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk> <47F34E34.9010507@gmail.com> Message-ID: <47F3D79C.3020307@v.loewis.de> > Personally, I've never really understood the purpose of > test_support.TESTFN. Whenever I've needed a temporary file for a test, I > just use the tempfile module (e.g. test_cmd_line_script, test_runpy). > Tests using that module don't care if the old files take 'a while' to > get deleted on Windows, as tempfile uses a different name each time anyway. > > Is using a fixed TESTFN just an old approach that predates the existence > of a robust tempfile module in the standard library? No. I believe the rationale for TESTFN is to provide a fixed name, precisely so that the test suite doesn't leave tons of garbage around. Regards, Martin From tnelson at onresolve.com Wed Apr 2 21:12:17 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 2 Apr 2008 12:12:17 -0700 Subject: [Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause In-Reply-To: <47F3D2ED.4070109@v.loewis.de> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E38501@EXMBX04.exchhosting.com> <47F3D2ED.4070109@v.loewis.de> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17F1F2EC@EXMBX04.exchhosting.com> > > That'll kill the first python_d.exe instance it finds matching the > > given path; given that our buildbots run trunk/release25-maint/py3k > > in parallel > > That's actually not a given: we currently *don't* run multiple builds > simultaneously on the same slave. I thought the slave lock only applied per branch, not per host? > > Unless anyone advises otherwise, I'll start on a patch. > > If you can make it less error-prone, sure, go ahead. Spent a bit of time on it this evening; as it happens, in order to enumerate 64-bit processes, you need to be a 64-bit process yourself. As it's much easier managing 32-bit vs. x64 binaries when they're a .vcproj part of pcbuild.sln, I'm looking into adding kill_python as a .vcproj and configure the solution such that it builds and runs this before any other project. That'll automatically take care of choosing the right version to run depending on whether 'Win32' or 'x64' is selected as the platform. It'll also simplify the verification logic that checks if it's the right python_d.exe -- the path of the .exe needs to match the path of the running kill_python.exe. Trent. From martin at v.loewis.de Wed Apr 2 21:29:02 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 02 Apr 2008 21:29:02 +0200 Subject: [Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17F1F2EC@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E38501@EXMBX04.exchhosting.com> <47F3D2ED.4070109@v.loewis.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E17F1F2EC@EXMBX04.exchhosting.com> Message-ID: <47F3DE7E.7070808@v.loewis.de> Trent Nelson wrote: >>> That'll kill the first python_d.exe instance it finds matching the >>> given path; given that our buildbots run trunk/release25-maint/py3k >>> in parallel >> That's actually not a given: we currently *don't* run multiple builds >> simultaneously on the same slave. > > I thought the slave lock only applied per branch, not per host? As the name suggests, it's per slave :-) a slave being the name/password pair, along with the buildbot.tac file. The sequencing of builds per builder is not a locking mechanism, actually; a single builder just won't schedule a new build as long as a build is already running. So we have currently three builders per slave, and these are serialized with the slave lock (hence the occasional long sequences of yellow "Build nnn": the build starts, tries to acquire the slave lock, which then a different builder still holds). Now, on your machine, I understand you also have two slaves running. They might, in principle, kill each other's python_d.exe processes, except that the paths will be different there (amd64 vs. not). Regards, Martin From priyarp.tech at gmail.com Wed Apr 2 22:12:04 2008 From: priyarp.tech at gmail.com (Pree Raj) Date: Wed, 2 Apr 2008 13:12:04 -0700 Subject: [Python-Dev] Python on ThreadX Message-ID: <8bb4faa80804021312p6713db38i32373f1cc20a3632@mail.gmail.com> Hi, I have an application that used python on linux. I want to port it to run on ThreadX. Can you tell me if this can be possible. If yes, how can I get started. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080402/07a93647/attachment.htm From alex.neundorf at kitware.com Wed Apr 2 22:37:26 2008 From: alex.neundorf at kitware.com (Alexander Neundorf) Date: Wed, 2 Apr 2008 22:37:26 +0200 Subject: [Python-Dev] Python on ThreadX In-Reply-To: <8bb4faa80804021312p6713db38i32373f1cc20a3632@mail.gmail.com> References: <8bb4faa80804021312p6713db38i32373f1cc20a3632@mail.gmail.com> Message-ID: <806d41050804021337x7faff46ap4be31c14fd969006@mail.gmail.com> On Wed, Apr 2, 2008 at 10:12 PM, Pree Raj wrote: > Hi, > I have an application that used python on linux. > I want to port it to run on ThreadX. > Can you tell me if this can be possible. > If yes, how can I get started. I managed to get Python to work under eCos, which is also a small RTOS (one binary firmware image, no shared libraries, one process space). So I think it should be possible for ThreadX too. I did this with the cmake-based build. Alex From musiccomposition at gmail.com Wed Apr 2 22:55:13 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Wed, 2 Apr 2008 15:55:13 -0500 Subject: [Python-Dev] No time for svn merge In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381F9@EXMBX04.exchhosting.com> References: <1afaf6160803301416n6e37df7cw5ff39a55ddb7341@mail.gmail.com> <47F00C33.1050503@v.loewis.de> <5c6f2a5d0803301507m25455babm7254b9cb62905dfe@mail.gmail.com> <47F0113D.8020408@v.loewis.de> <5c6f2a5d0803301533l1d79be9bh6033bb3dfafb485a@mail.gmail.com> <47F07068.3060503@v.loewis.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381B6@EXMBX04.exchhosting.com> <47F34A85.5020909@cheimes.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381F9@EXMBX04.exchhosting.com> Message-ID: <1afaf6160804021355o8c97cf0le839b6623971dba8@mail.gmail.com> On Wed, Apr 2, 2008 at 7:16 AM, Trent Nelson wrote: > Christian Heimes [mailto:lists at cheimes.de]: > > Trent Nelson schrieb: > > > I think holding a developer accountable for merging or blocking to > > py3k when they commit to trunk is a great idea. Who better to pass > > judgement on such an activity than the person closest to it? > > > > Blocking a revision makes my job as The Merger easier. > > > > I'm not so sure about the merging part. It takes some experience with > > the Python 3.0 code base to know the subtle differences in the C API. > > Most merges are straight forward for me. If you enforce the forward > > merging on every developer it may slow down development. > > Each regular merge takes me about 45 minutes of computer time but less > > than 15 supervisor time. The computer time is mostly compile and test > > time in the background. If everybody merges her own code to 3.0 it > > still takes *everybody* about 10 minutes of time and 45 minutes of > computer > > time. > > Ah, right, I wasn't thinking about the implication of code affecting the C > base for some reason, but that's entirely reasonable. Perhaps each > developer should be accountable for either: > > a) blocking > b) merging, if they're able to do so > c) if they're unable to merge, replying to the relevant python-checkins@ > e-mail indicating that they're unable to handle trunk -> py3k for whatever > reason (e.g. not familiar with py3k code base) > > Other developers could then pitch in and help merge if someone requests it > via e-mail. I'd think that would make The Merger's life easier. I think we should let The Merger decide what makes his life easier. :) > > > Trent. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com > -- Cheers, Benjamin Peterson -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080402/abc94f45/attachment.htm From scott+python-dev at scottdial.com Wed Apr 2 22:58:03 2008 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 02 Apr 2008 16:58:03 -0400 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F3D79C.3020307@v.loewis.de> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk> <47F34E34.9010507@gmail.com> <47F3D79C.3020307@v.loewis.de> Message-ID: <47F3F35B.1010005@scottdial.com> Martin v. L?wis wrote: >> Is using a fixed TESTFN just an old approach that predates the existence >> of a robust tempfile module in the standard library? > > No. I believe the rationale for TESTFN is to provide a fixed name, > precisely so that the test suite doesn't leave tons of garbage around. I figured this was going to be the answer, and I wonder if it would be better to use the tempfile module to generate unique filenames that are all contained in a special directory. It would seem functionally equivalent in the sense that removing the garbage is easy "rmtree(TESTDIR)", and it would also solve the windows problem of files that can't be unlinked. I suppose one would have to periodically clean-up their temp folder of the files left behind, but that's still an improvement over what is currently required if TESTFN is not unlinked. The only reason I could see the switch being undesirable is if you wanted to go read the test file by hand, but I'm not sure anyone is doing that. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From barry at python.org Thu Apr 3 00:00:09 2008 From: barry at python.org (Barry Warsaw) Date: Wed, 2 Apr 2008 18:00:09 -0400 Subject: [Python-Dev] Building next alphas Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 This is a reminder that I am going to start building the next alpha releases for Python 2.6 and 3.0 now. Please, no checkins unless you get approval from me, and until you hear that the freeze is lifted. I am now on freenode #python-dev, IM, and Jabber if you need to contact me. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBR/QB6XEjvBPtnXfVAQLn8QQArynMsNeeb6gqjUCSYuupM2XXAbwP5XOX LbTeGN+vM13uNK32fI47rDaPEfudfGnrd3Ttc1pg6/S/MOo5T41zs/TX2jdMEQ4g 6zCtk6xJiexGbExKioiTVdYgiqA8C6u+XY8aU2ogklD1h7kfEOWKw5urXkValFhG Iymq6mrEyJQ= =d/L3 -----END PGP SIGNATURE----- From mhammond at skippinet.com.au Thu Apr 3 00:50:10 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Thu, 3 Apr 2008 09:50:10 +1100 Subject: [Python-Dev] [Distutils] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381BA@EXMBX04.exchhosting.com> References: <002401c892ff$a8254630$f86fd290$@com.au> <00c801c893dd$32d03320$98709960$@com.au> <47F2C163.9000402@v.loewis.de> <016001c89460$944f10c0$bced3240$@com.au> <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381BA@EXMBX04.exchhosting.com> Message-ID: <022b01c89513$ef278140$cd7683c0$@com.au> > I think this is fine; we don't really have a notion of compiling for a > native platform, nor is the build machine's architecture factored into > the equation. Actually, I think it is slightly. IIUC, the AMD64 build currently assumes it can execute x86 executables in various places. To fix this, the build process for each of the platforms would be slightly different. My proposal would allow (eg) 'PCBuild\python.exe' or 'PCBuild\make_versioninfo.exe' to specified in build processes, with the knowledge it will work, regardless of what platform it is being run on. For example, with my idea, you could write code such as: exe = os.path.join(os.getenv("PYTHON_HOME"), "PCBuild", "python.exe")) popen(exe, ...) and expect it to work on all platforms. If I understand correctly, I now need to write something like: if "AMD64" in sys.platform: exe = os.path.join(os.getenv("PYTHON_HOME"), "PCBuild", "amd64", "python.exe")) else if "32 bit" in sys.platform: exe = os.path.join(os.getenv("PYTHON_HOME"), "PCBuild", "python.exe")) else: # hrm - I obviously don?t know about new platforms yet... popen(exe, ...) In other words, I think there is real advantage to simple scripts knowing that './python' or './PCBuild/python.exe' will always be executable (and always give the "native" version if the platform does emulation) But I accept I'm waving appendages in the wind, and without a concrete use-case I will take Martin's advice and let it go. Cheers, Mark From skip at pobox.com Thu Apr 3 05:36:24 2008 From: skip at pobox.com (skip at pobox.com) Date: Wed, 2 Apr 2008 22:36:24 -0500 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> Message-ID: <18420.20664.231996.696687@montanaro-dyndns-org.local> Ralf> anyone care to take a look at: Ralf> http://bugs.python.org/issue2014 Ralf> It's about xmlrpclib not being able to send datetime objects with Ralf> dates before 1900. It's actually not xmlrpclib which has the limitation, but datetime.strftime(). That's a known limitation. Here's the comment in the datetime code: /* Give up if the year is before 1900. * Python strftime() plays games with the year, and different * games depending on whether envar PYTHON2K is set. This makes * years before 1900 a nightmare, even if the platform strftime * supports them (and not all do). * We could get a lot farther here by avoiding Python's strftime * wrapper and calling the C strftime() directly, but that isn't * an option in the Python implementation of this module. */ That, in turn, calls time.strftime() which calls the underlying system's strftime() library function. Personally, I don't think patching xmlrpclib is the right place to "fix" this problem. It's possible that the datetime comment is no longer correct and that limitation should be reconsidered. I see no other mention of PYTHON2K in any .c, .h or .py files in the trunk. Skip From guido at python.org Thu Apr 3 05:52:52 2008 From: guido at python.org (Guido van Rossum) Date: Wed, 2 Apr 2008 20:52:52 -0700 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: <18420.20664.231996.696687@montanaro-dyndns-org.local> References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> <18420.20664.231996.696687@montanaro-dyndns-org.local> Message-ID: On Wed, Apr 2, 2008 at 8:36 PM, wrote: > Ralf> anyone care to take a look at: > Ralf> http://bugs.python.org/issue2014 > Ralf> It's about xmlrpclib not being able to send datetime objects with > Ralf> dates before 1900. > > It's actually not xmlrpclib which has the limitation, but > datetime.strftime(). That's a known limitation. Here's the comment in the > datetime code: > > /* Give up if the year is before 1900. > * Python strftime() plays games with the year, and different > * games depending on whether envar PYTHON2K is set. This makes > * years before 1900 a nightmare, even if the platform strftime > * supports them (and not all do). > * We could get a lot farther here by avoiding Python's strftime > * wrapper and calling the C strftime() directly, but that isn't > * an option in the Python implementation of this module. > */ > > That, in turn, calls time.strftime() which calls the underlying system's > strftime() library function. > > Personally, I don't think patching xmlrpclib is the right place to "fix" > this problem. It's possible that the datetime comment is no longer correct > and that limitation should be reconsidered. I see no other mention of > PYTHON2K in any .c, .h or .py files in the trunk. I'd like to see this fixed if possible, but I'm not sure how -- the C level 'struct tm' has (year - 1900) in the tm_year member, and I'm not sure that implementations are required to do anything with negative values there. We'd have to reimplement all of strftime ourselves. I'm not against that, but there are higher priorities. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Thu Apr 3 06:21:11 2008 From: barry at python.org (Barry Warsaw) Date: Thu, 3 Apr 2008 00:21:11 -0400 Subject: [Python-Dev] Building next alphas In-Reply-To: References: Message-ID: <5C3C4C91-88B9-49A7-924C-21ACB5225A46@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 2, 2008, at 6:00 PM, Barry Warsaw wrote: > > This is a reminder that I am going to start building the next alpha > releases for Python 2.6 and 3.0 now. Please, no checkins unless you > get approval from me, and until you hear that the freeze is lifted. > > I am now on freenode #python-dev, IM, and Jabber if you need to > contact me. I've been battling the flu and got distracted for a few hours tonight, so I'm not quite done with the releases. However, I've tagged and tar'd 'em so it should just be a matter of uploading the files and updating the site. I should finish tomorrow. I'm thawing the trees, so you can go ahead and start committing things again, but /please/ be especially conservative over the next 24-48 hours. Make sure your changes don't break anything, just in case my virus-addled brain screwed something up and I need to cut another release. - -Barry P.S. Huge thanks to Benjamin Peterson, both for a quick last minute fix to the 3.0 NEWS file via IRC, and his wonderful release.py script. I've hacked it up a bit, but it was exactly what I was looking for, and it made things go much smoother this time. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBR/RbOHEjvBPtnXfVAQKCOwQArrYXC9X3lOvqRTwWQ9SYPH+6n1VN4MrT WNm+jhsbiwZq8EuNslCBW3/52HP/wM7jlYizKZCL+cbcFaevNhWjjbPtwSTkJjVy /uKG/NcDYQsPH3n4mET3/XlF5JrfS51avLSD7YebucTph9+otzI8LkK0Unvdbtq+ /86m3lEZAlY= =q005 -----END PGP SIGNATURE----- From martin at v.loewis.de Thu Apr 3 08:41:44 2008 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 03 Apr 2008 08:41:44 +0200 Subject: [Python-Dev] [Distutils] FW: [issue2513] 64bit cross compilation on windows In-Reply-To: <022b01c89513$ef278140$cd7683c0$@com.au> References: <002401c892ff$a8254630$f86fd290$@com.au> <00c801c893dd$32d03320$98709960$@com.au> <47F2C163.9000402@v.loewis.de> <016001c89460$944f10c0$bced3240$@com.au> <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E381BA@EXMBX04.exchhosting.com> <022b01c89513$ef278140$cd7683c0$@com.au> Message-ID: <47F47C28.9090203@v.loewis.de> > Actually, I think it is slightly. IIUC, the AMD64 build currently assumes > it can execute x86 executables in various places. To fix this, the build > process for each of the platforms would be slightly different. Why does that need fixing? The AMD64 build *can* execute x86 binaries, whether it is performed on Win32 or Win64. A problem only arises if the AMD64 build assumes that it can execute AMD64 binaries, and it is a cross-compilation. As a consequence, we must never execute the python.exe that we just built during the build process. > My proposal > would allow (eg) 'PCBuild\python.exe' or 'PCBuild\make_versioninfo.exe' to > specified in build processes, with the knowledge it will work, regardless of > what platform it is being run on. No to python.exe, yes to make_versioninfo.exe. make_versioninfo.exe is *always* an x86 binary, so it is always safe to execute on Windows. If you tried to execute PCBuild\python.exe, it would break my builds: I don't *have* x86 binaries in PCBuild when I cross-compile for AMD64. Instead, I've changed the build process to run HOST_PYTHON if that is set, and only fall back to PCBuild\python.exe if it isn't. Regards, Martin From schmir at gmail.com Thu Apr 3 09:29:57 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Thu, 3 Apr 2008 09:29:57 +0200 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: <18420.20664.231996.696687@montanaro-dyndns-org.local> References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> <18420.20664.231996.696687@montanaro-dyndns-org.local> Message-ID: <932f8baf0804030029y262d7cfeicc22b5d94b7ffc08@mail.gmail.com> On Thu, Apr 3, 2008 at 5:36 AM, wrote: > > It's actually not xmlrpclib which has the limitation, but > datetime.strftime(). That's a known limitation. Here's the comment in > the > datetime code: > [snip] > Personally, I don't think patching xmlrpclib is the right place to "fix" > this problem. It's possible that the datetime comment is no longer > correct > yes, you're right. but I didn't feel like writing a strftime implementation (which has probably even less chance of being committed). This patch is rather tiny, it's easy to understand and it works now. > and that limitation should be reconsidered. I see no other mention of > PYTHON2K in any .c, .h or .py files in the trunk. > > Skip > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080403/ed8e1e56/attachment.htm From schmir at gmail.com Thu Apr 3 09:34:27 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Thu, 3 Apr 2008 09:34:27 +0200 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> <18420.20664.231996.696687@montanaro-dyndns-org.local> Message-ID: <932f8baf0804030034g625d877ej7dd5f727ad41cc80@mail.gmail.com> On Thu, Apr 3, 2008 at 5:52 AM, Guido van Rossum wrote: > > I'd like to see this fixed if possible, but I'm not sure how -- the C > level 'struct tm' has (year - 1900) in the tm_year member, and I'm not > sure that implementations are required to do anything with negative > values there. We'd have to reimplement all of strftime ourselves. I'm > not against that, but there are higher priorities. > Have you considered using the pure python datetime implementation from the pypy project for py3k? It's even based on your own code :) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080403/51c43b85/attachment.htm From heshan.suri at gmail.com Thu Apr 3 13:47:11 2008 From: heshan.suri at gmail.com (Heshan Suriyaarachchi) Date: Thu, 3 Apr 2008 17:17:11 +0530 Subject: [Python-Dev] Annotations support in python Message-ID: <12e5d0d90804030447u67391abby361d0cabcb919018@mail.gmail.com> Hi, I need to work with annotations as it is said in [1]. Currently I am using python 2.5.1. I would like to know whether the next release of python will support this feature. If the next version support this feature I would like to know when are you planning to release it. I used the __future__ module but I could not get the annotations with it. [1] - http://www.python.org/dev/peps/pep-3107/ Thanx Heshan Suriyaarachchi -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080403/0c2792e4/attachment.htm From thomas at python.org Thu Apr 3 14:20:48 2008 From: thomas at python.org (Thomas Wouters) Date: Thu, 3 Apr 2008 14:20:48 +0200 Subject: [Python-Dev] Annotations support in python In-Reply-To: <12e5d0d90804030447u67391abby361d0cabcb919018@mail.gmail.com> References: <12e5d0d90804030447u67391abby361d0cabcb919018@mail.gmail.com> Message-ID: <9e804ac0804030520i4c3201r51f72ff1a26d3314@mail.gmail.com> On Thu, Apr 3, 2008 at 1:47 PM, Heshan Suriyaarachchi wrote: > Hi, > I need to work with annotations as it is said in [1]. Currently I am > using python 2.5.1. I would like to know whether the > next release of python will support this feature. If the next version > support this feature I would like to know when are you > planning to release it. I used the __future__ module but I could not get > the annotations with it. > > [1] - http://www.python.org/dev/peps/pep-3107/ > The 3xxx PEPs are all about Python 3.0. Python 3.0 is the version that will support function annotations. Python 2.6 will almost certainly not, and 2.5 will most assuredly not -- features don't get added after the major version has been released, we only fix bugs. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080403/e7dc8204/attachment.htm From mail at timgolden.me.uk Thu Apr 3 15:39:47 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Thu, 03 Apr 2008 14:39:47 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F3592D.3080700@timgolden.me.uk> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk> <47F3592D.3080700@timgolden.me.uk> Message-ID: <47F4DE23.3020202@timgolden.me.uk> [re tests which fail because something's holding a file open with SHARE_DELETE] Well I've tried my best, but I can't come up with a solution which guarantees to overcome this obstacle. I set up a fairly aggressive directory watcher which, when it sees a test file being created, takes a SHARE_DELETE handle on it and releases it immediately. (Guessing that this is what the other apps are doing). Running the test suite from HEAD, this generates all manner of access-denied type errors as per the original output. I used tempfile to generate a random TESTFN in the current directory rather than the static @test. And implemented the rename-shim discussed previously, renaming to a different random name, courtesy of mktemp. With those in place, most tests run without error. But I'm still getting errors in the same sort of areas which Steven B originally reported. The one error I can't see a way round is the access denied (which manifests as Permission Error) which is the documented result of attempting to open a file with a pending delete -- ie the delete succeeded but hasn't completed yet. An additional complication is that there are hundreds of instances throughout the tests where the test simply calls os.unlink/os.remove to undo the test file. To have some more robust central deletion I had to go through and update 68 tests. I'll keep trying, but in the current state I'm not convinced the situation's improved enough for me to bother uploading a patch. TJG From tnelson at onresolve.com Thu Apr 3 19:44:51 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Thu, 3 Apr 2008 10:44:51 -0700 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F4DE23.3020202@timgolden.me.uk> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk> <47F3592D.3080700@timgolden.me.uk>,<47F4DE23.3020202@timgolden.me.uk> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB27E@EXMBX04.exchhosting.com> [Disclaimer: thought dump e-mail, signal to noise ratio may be subpar.] Sounds like you're at least making steps forward in the right direction, despite the activity probably being quite disheartening. Based on what you've said below and the rest of the conversation, here are my thoughts for an approach: 1. For a given python[_d].exe, always use the same test directory, but hash it against the entire python process path such that it's unique only for a given python instance. 2. Make sure every time a test wants a temp file, it gets a unique one in this directory. Sounds like your TESTFN modification would take care of this for those tests using TESTFN; if TESTFN is the preferred approach then any other tests using tempfile or hardcoding file names would then be changed to use this instead. 3. In order to address tests that either call the test_support methods for removing files/directories, or those that call os.(unlink|remove), do what ever needs to be done to make these no-ops on Windows if an error occurs. 4. At the end of the regrtest.py run, create a suspended arbitrary process (i.e. explorer.exe), hijack the main thread context of the process and inject a routine (i.e. overwrite the thread context's instruction pointers) that takes care of removing the temporary directory that was used for testing -- patiently re-trying if any failures occur until all rogue processes also accessing the file(s) stop doing so. Resume the thread before exiting python. Heh. Sounds crazy? It did to me as well, until I came across http://www.catch22.net/tuts/selfdel.asp, which documents the approach. It's not particularly necessary in our case, we could simply spawn another python process at the end of regrtest.py that patiently attempts to remove the test directory we just used when the python process that was executing regrtest.py exits. We could then modify regrtest.py such that it will use the same approach if the hashed test directory already exists at the start of a run and it can't remove it via os.unlink. If we *still* run into issues here on the buildbots, say if regrtest.py blocks on our helper process, which for the life of it can't remove some/all of the test files -- it'd be interesting to write something that grok's all open handles for all processes and attempts to figure out what it is that keeps these files open -- i.e. same thing that procexp.exe does when you search for a handle. Or, keeping it simple, rather than a separate process and hashed test directory based on python process path, just have a common directory, i.e. %TEMP%\python-regrtest, and use an incrementing sequence number for each test run's test directory, i.e. if there are directories 001, 002 and 003 in the temp dir, when regrtest.py starts, it'll try delete all of these -- if it can't (which is what we'd want if another test is already running), it adds 1 to the highest numbered directory it couldn't delete. Guess it all depends on how much effort we want to put into cleaning up our test directory really -- just ensuring tests get a clean area and unique file names each run is the easy part. Trent. ________________________________________ From: python-dev-bounces+tnelson=onresolve.com at python.org [python-dev-bounces+tnelson=onresolve.com at python.org] On Behalf Of Tim Golden [mail at timgolden.me.uk] Sent: 03 April 2008 09:39 To: Python-Dev Subject: Re: [Python-Dev] fixing tests on windows [re tests which fail because something's holding a file open with SHARE_DELETE] Well I've tried my best, but I can't come up with a solution which guarantees to overcome this obstacle. I set up a fairly aggressive directory watcher which, when it sees a test file being created, takes a SHARE_DELETE handle on it and releases it immediately. (Guessing that this is what the other apps are doing). Running the test suite from HEAD, this generates all manner of access-denied type errors as per the original output. I used tempfile to generate a random TESTFN in the current directory rather than the static @test. And implemented the rename-shim discussed previously, renaming to a different random name, courtesy of mktemp. With those in place, most tests run without error. But I'm still getting errors in the same sort of areas which Steven B originally reported. The one error I can't see a way round is the access denied (which manifests as Permission Error) which is the documented result of attempting to open a file with a pending delete -- ie the delete succeeded but hasn't completed yet. An additional complication is that there are hundreds of instances throughout the tests where the test simply calls os.unlink/os.remove to undo the test file. To have some more robust central deletion I had to go through and update 68 tests. I'll keep trying, but in the current state I'm not convinced the situation's improved enough for me to bother uploading a patch. TJG _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com From guido at python.org Thu Apr 3 20:17:06 2008 From: guido at python.org (Guido van Rossum) Date: Thu, 3 Apr 2008 11:17:06 -0700 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: <932f8baf0804030029y262d7cfeicc22b5d94b7ffc08@mail.gmail.com> References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> <18420.20664.231996.696687@montanaro-dyndns-org.local> <932f8baf0804030029y262d7cfeicc22b5d94b7ffc08@mail.gmail.com> Message-ID: On Thu, Apr 3, 2008 at 12:29 AM, Ralf Schmitt wrote: > yes, you're right. but I didn't feel like writing a strftime implementation > (which has probably even less chance of being committed). This patch is > rather tiny, it's easy to understand and it works now. Thinking about it more, given the slim chances that we'll reimplement strftime, I think it's okay to fix this for xmlrpc specifically. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 3 20:17:48 2008 From: guido at python.org (Guido van Rossum) Date: Thu, 3 Apr 2008 11:17:48 -0700 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: <932f8baf0804030034g625d877ej7dd5f727ad41cc80@mail.gmail.com> References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> <18420.20664.231996.696687@montanaro-dyndns-org.local> <932f8baf0804030034g625d877ej7dd5f727ad41cc80@mail.gmail.com> Message-ID: On Thu, Apr 3, 2008 at 12:34 AM, Ralf Schmitt wrote: > Have you considered using the pure python datetime implementation from the > pypy project for py3k? I wouldn't dream of it. datetime is considered performance critical by many. > It's even based on your own code :) Which was meant as a prototype only. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tnelson at onresolve.com Thu Apr 3 20:33:23 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Thu, 3 Apr 2008 11:33:23 -0700 Subject: [Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause In-Reply-To: <47F3D2ED.4070109@v.loewis.de> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E17E38501@EXMBX04.exchhosting.com>, <47F3D2ED.4070109@v.loewis.de> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB27F@EXMBX04.exchhosting.com> Committed new version of kill_python to trunk in r62129. Trent. ________________________________________ From: "Martin v. L?wis" [martin at v.loewis.de] Sent: 02 April 2008 14:39 To: Trent Nelson Cc: python-dev at python.org Subject: Re: [Python-Dev] Tools\buildbot\kill_python.c can't be helping our cause > That'll kill the first python_d.exe instance it finds matching the > given path; given that our buildbots run trunk/release25-maint/py3k > in parallel That's actually not a given: we currently *don't* run multiple builds simultaneously on the same slave. > Unless anyone advises otherwise, I'll start on a patch. If you can make it less error-prone, sure, go ahead. Regards, Martin From tjreedy at udel.edu Thu Apr 3 23:09:35 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 3 Apr 2008 17:09:35 -0400 Subject: [Python-Dev] fixing tests on windows References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk><47F3592D.3080700@timgolden.me.uk> <47F4DE23.3020202@timgolden.me.uk> Message-ID: "Tim Golden" wrote in message news:47F4DE23.3020202 at timgolden.me.uk... | [re tests which fail because something's holding a file | open with SHARE_DELETE] There are a couple of things one can do in a directory's Properties box (right click) to reduce interference. 1. Under General/Advanced, uncheck 'allow Indexing Service' and click Yes to recursively apply to subdirs and files. 2. Under Sharing, check 'Make private'. This is recursive, but only works for dirs under My Documents or on Desktop. This would be most useful, I think, if buildbots run as a separate user. Have these already been done? tjr From tnelson at onresolve.com Thu Apr 3 23:40:04 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Thu, 3 Apr 2008 14:40:04 -0700 Subject: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB280@EXMBX04.exchhosting.com> I started looking into this: http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/289/step-test/0 Pertinent part: test_asyncore test_asynchat command timed out: 1200 seconds without output SIGKILL failed to kill process using fake rc=-1 program finished with exit code -1 remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last): Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process ] I tried to replicate it on the buildbot in order to debug, which, surprisingly, I could do consistently by just running rt.bat -q -d -uall test_asynchat. As the log above indicates, the python process becomes completely and utterly wedged, to the point that I can't even attach a remote debugger and step into it. Digging further, I noticed that if I ran the following code in two different python consoles, EADDRINUSE was *NOT* being raised by socket.bind(): import socket s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.bind(('127.0.0.1', 54322)) However, take out the setsockopt line, and wallah, the second s.bind() will raise EADDRINUSE, as expected. This manifests into a really bizarre issue with test_asynchat in particualr, as subsequent sock.accept() calls on the socket put python into the uber wedged state (can't even ctrl-c out at the console, need to kill the process directly). Have to leave the office and head home so I don't have any more time to look at it tonight -- just wanted to post here for others to mull over. Trent. From steven.bethard at gmail.com Thu Apr 3 23:52:39 2008 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 3 Apr 2008 15:52:39 -0600 Subject: [Python-Dev] fixing tests on windows In-Reply-To: References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk> <47F3592D.3080700@timgolden.me.uk> <47F4DE23.3020202@timgolden.me.uk> Message-ID: On Thu, Apr 3, 2008 at 3:09 PM, Terry Reedy wrote: > "Tim Golden" wrote in message > news:47F4DE23.3020202 at timgolden.me.uk... > > | [re tests which fail because something's holding a file > | open with SHARE_DELETE] > > There are a couple of things one can do in a directory's Properties box > (right click) to reduce interference. > 1. Under General/Advanced, uncheck 'allow Indexing Service' and click Yes > to recursively apply to subdirs and files. > 2. Under Sharing, check 'Make private'. This is recursive, but only works > for dirs under My Documents or on Desktop. This would be most useful, I > think, if buildbots run as a separate user. > Have these already been done? We already know how to stop the errors from arising by modifying the Windows environment. In my particular case, it was sufficient to disable Windows Search Service and the TortoiseSVN Image Overlays. However, it would be *really* nice if the Python test suite just worked out of the box, and you didn't have to change your normal Windows environment. Or were you suggesting that there is some programmatic way for the test suite to create directories that disallow the Search Service, etc.? Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From mail at timgolden.me.uk Thu Apr 3 23:58:24 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Thu, 03 Apr 2008 22:58:24 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk><47F3592D.3080700@timgolden.me.uk> <47F4DE23.3020202@timgolden.me.uk> Message-ID: <47F55300.4060603@timgolden.me.uk> Terry Reedy wrote: > "Tim Golden" wrote in message > news:47F4DE23.3020202 at timgolden.me.uk... > | [re tests which fail because something's holding a file > | open with SHARE_DELETE] > > There are a couple of things one can do in a directory's Properties box > (right click) to reduce interference. > 1. Under General/Advanced, uncheck 'allow Indexing Service' and click Yes > to recursively apply to subdirs and files. > 2. Under Sharing, check 'Make private'. This is recursive, but only works > for dirs under My Documents or on Desktop. This would be most useful, I > think, if buildbots run as a separate user. > Have these already been done? I realise that there are options on any given machine to reduce the interference of known applications. (And I assume that the people who own buildbots are in a position to do what they can in that direction). I was working more towards a situation where they wouldn't have to, where the test suite could confidently be run without having to allow for artefacts of this type. Still trying: some new ideas from Trent elsewhere in this thread which I'll look into tomorrow. TJG From tjreedy at udel.edu Fri Apr 4 02:34:18 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 3 Apr 2008 20:34:18 -0400 Subject: [Python-Dev] fixing tests on windows References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk><52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com><47F33DE7.3030702@timgolden.me.uk> <47F3592D.3080700@timgolden.me.uk><47F4DE23.3020202@timgolden.me.uk> Message-ID: "Steven Bethard" wrote in message news:d11dcfba0804031452m911e1f3w4ec68a5967d69656 at mail.gmail.com... | On Thu, Apr 3, 2008 at 3:09 PM, Terry Reedy wrote: | > "Tim Golden" wrote in message | > news:47F4DE23.3020202 at timgolden.me.uk... | > | > | [re tests which fail because something's holding a file | > | open with SHARE_DELETE] | > | > There are a couple of things one can do in a directory's Properties box | > (right click) to reduce interference. | > 1. Under General/Advanced, uncheck 'allow Indexing Service' and click Yes | > to recursively apply to subdirs and files. | > 2. Under Sharing, check 'Make private'. This is recursive, but only works | > for dirs under My Documents or on Desktop. This would be most useful, I | > think, if buildbots run as a separate user. | > Have these already been done? | | We already know how to stop the errors from arising by modifying the | Windows environment. In my particular case, it was sufficient to | disable Windows Search Service and the TortoiseSVN Image Overlays. | However, it would be *really* nice if the Python test suite just | worked out of the box, and you didn't have to change your normal | Windows environment. If the testdir disallows the search indexer, then there should be no need to disable Windows Search Service. If privatizing the dir kept other programs out, then likewise. | Or were you suggesting that there is some programmatic way for the | test suite to create directories that disallow the Search Service, | etc.? I suspect, but do not know, that the dialog box effects changes through user-programmable interfaces. So while I would start with manual changes to see if that solves the problem, I presume there must be a system call for changing dir attributes. tjr From tonynelson at georgeanelson.com Fri Apr 4 02:22:07 2008 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Thu, 3 Apr 2008 20:22:07 -0400 Subject: [Python-Dev] fixing tests on windows In-Reply-To: References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk> <47F3592D.3080700@timgolden.me.uk> <47F4DE23.3020202@timgolden.me.uk> Message-ID: At 3:52 PM -0600 4/3/08, Steven Bethard wrote: >On Thu, Apr 3, 2008 at 3:09 PM, Terry Reedy wrote: ... >Or were you suggesting that there is some programmatic way for the >test suite to create directories that disallow the Search Service, >etc.? I'd think that files and directories created in the TEMP directory would normally not be indexed on any OS, including MSWindows. But this is just a guess. -- ____________________________________________________________________ TonyN.:' ' From barry at python.org Fri Apr 4 03:47:01 2008 From: barry at python.org (Barry Warsaw) Date: Thu, 3 Apr 2008 21:47:01 -0400 Subject: [Python-Dev] RELEASED Python 2.6a2 and 3.0a4 Message-ID: <3C3C0150-ED65-4381-9F54-BB437DD6DFB9@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team and the Python community, I'm happy to announce the second alpha release of Python 2.6, and the fourth alpha release of Python 3.0. Please note that these are alpha releases, and as such are not suitable for production environments. We continue to strive for a high degree of quality, but there are still some known problems and the feature sets have not been finalized. These alphas are being released to solicit feedback and hopefully discover bugs, as well as allowing you to determine how changes in 2.6 and 3.0 might impact you. If you find things broken or incorrect, please submit a bug report at http://bugs.python.org For more information and downloadable distributions, see the Python 2.6 web site: http://www.python.org/download/releases/2.6/ and the Python 3.0 web site: http://www.python.org/download/releases/3.0/ We are planning one more alpha release of each version, followed by two beta releases, with the final releases planned for August 2008. See PEP 361 for release details: http://www.python.org/dev/peps/pep-0361/ Enjoy, - -Barry Barry Warsaw barry at python.org Python 2.6/3.0 Release Manager (on behalf of the entire python-dev team) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBR/WImHEjvBPtnXfVAQJmoQP+MzqNDI+Xt8zua/FE7Ca4TVXoIIy2uoOm I1i3+vmevZ9vtAb9hcGwfEgPY4LSwb9Js4KnJJWMPaMuFJK4NgGoiMdj+t42zDbQ bEzfBUOCoVkejLRxIQnWeJf1Hu8JocYyCHIRffv57/QdKpHuiSs8aE8GIT3STo3o I88H5NY1GgI= =WT2z -----END PGP SIGNATURE----- From skip at pobox.com Fri Apr 4 05:48:06 2008 From: skip at pobox.com (skip at pobox.com) Date: Thu, 3 Apr 2008 22:48:06 -0500 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> <18420.20664.231996.696687@montanaro-dyndns-org.local> <932f8baf0804030029y262d7cfeicc22b5d94b7ffc08@mail.gmail.com> Message-ID: <18421.42230.77841.513592@montanaro-dyndns-org.local> Guido> Thinking about it more, given the slim chances that we'll Guido> reimplement strftime, I think it's okay to fix this for xmlrpc Guido> specifically. Is there some reason we can't incorporate a suitable open source implementation of strftime into the Python core? Here's one example that might provide a suitable starting point (BSD 4.4, then Tcl, then Apple heritage): http://www.opensource.apple.com/darwinsource/WWDC2003/tcl-10/tcl/compat/strftime.c It would probably need some tweaks for Windows, and some changes to the macro usage. Then there is the 1900 base year issue. I'm certainly no licensing expert, but it seems like we should be able to redistribute it. Skip From foom at fuhm.net Fri Apr 4 05:28:51 2008 From: foom at fuhm.net (James Y Knight) Date: Thu, 3 Apr 2008 23:28:51 -0400 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> <18420.20664.231996.696687@montanaro-dyndns-org.local> Message-ID: <5F06CBDC-6DF2-4621-99FC-0087D809E471@fuhm.net> On Apr 2, 2008, at 11:52 PM, Guido van Rossum wrote: > I'd like to see this fixed if possible, but I'm not sure how -- the C > level 'struct tm' has (year - 1900) in the tm_year member, and I'm not > sure that implementations are required to do anything with negative > values there. We'd have to reimplement all of strftime ourselves. I'm > not against that, but there are higher priorities. At least on linux, negative values work great. I have no idea if that's required by the spec or not. Someone would have to test on other platforms I guess... James From guido at python.org Fri Apr 4 05:59:13 2008 From: guido at python.org (Guido van Rossum) Date: Thu, 3 Apr 2008 20:59:13 -0700 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: <18421.42230.77841.513592@montanaro-dyndns-org.local> References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> <18420.20664.231996.696687@montanaro-dyndns-org.local> <932f8baf0804030029y262d7cfeicc22b5d94b7ffc08@mail.gmail.com> <18421.42230.77841.513592@montanaro-dyndns-org.local> Message-ID: Someone else will have to do a thorough code review. Last time we got something off the web it turned out to be awful (the float formatting code -- I'm still reeling from that one). On Thu, Apr 3, 2008 at 8:48 PM, wrote: > > Guido> Thinking about it more, given the slim chances that we'll > Guido> reimplement strftime, I think it's okay to fix this for xmlrpc > Guido> specifically. > > Is there some reason we can't incorporate a suitable open source > implementation of strftime into the Python core? Here's one example that > might provide a suitable starting point (BSD 4.4, then Tcl, then Apple > heritage): > > http://www.opensource.apple.com/darwinsource/WWDC2003/tcl-10/tcl/compat/strftime.c > > It would probably need some tweaks for Windows, and some changes to the > macro usage. Then there is the 1900 base year issue. I'm certainly no > licensing expert, but it seems like we should be able to redistribute it. > > Skip > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Fri Apr 4 06:23:00 2008 From: skip at pobox.com (skip at pobox.com) Date: Thu, 3 Apr 2008 23:23:00 -0500 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> <18420.20664.231996.696687@montanaro-dyndns-org.local> <932f8baf0804030029y262d7cfeicc22b5d94b7ffc08@mail.gmail.com> <18421.42230.77841.513592@montanaro-dyndns-org.local> Message-ID: <18421.44324.892060.577502@montanaro-dyndns-org.local> Guido> Someone else will have to do a thorough code review. Last time we Guido> got something off the web it turned out to be awful (the float Guido> formatting code -- I'm still reeling from that one). This isn't some oddball weekend project from an out-of-work programmer. It's derived from BSD 4.4 and then looks like Tcl absorbed it as their implementation of strftime. The version I posted looks like it's part of Darwin. Should be fairly well wrung out. I downloaded it and with a couple mods I was able to use it as a substitute for strftime() in timemodule.c. It compiles cleanly and seems to pass all tests. My thought would be to only use this for Python 3.0. Skip From skip at pobox.com Fri Apr 4 06:30:05 2008 From: skip at pobox.com (skip at pobox.com) Date: Thu, 3 Apr 2008 23:30:05 -0500 Subject: [Python-Dev] xmlrpclib and dates before 1900 In-Reply-To: References: <932f8baf0804011345m27206411o2033498a72738149@mail.gmail.com> <18420.20664.231996.696687@montanaro-dyndns-org.local> <932f8baf0804030029y262d7cfeicc22b5d94b7ffc08@mail.gmail.com> <18421.42230.77841.513592@montanaro-dyndns-org.local> Message-ID: <18421.44749.206803.852140@montanaro-dyndns-org.local> > I downloaded it and with a couple mods I was able to use it as a > substitute for strftime() in timemodule.c. It compiles cleanly and seems > to pass all tests. I misspoke. test_strptime fails with timezone issues. That's probably just my misunderstanding of how Python deals with timezones. I make a quick replacement of Tcl's tzname stuff to get it to compile, but the test fails. It would probably not be much work for someone who understands how the Python datetime/time code handles timezone names. Skip From nnorwitz at gmail.com Fri Apr 4 06:49:22 2008 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 3 Apr 2008 21:49:22 -0700 Subject: [Python-Dev] Annotations support in python In-Reply-To: <12e5d0d90804030447u67391abby361d0cabcb919018@mail.gmail.com> References: <12e5d0d90804030447u67391abby361d0cabcb919018@mail.gmail.com> Message-ID: The release schedule for 2.6/3.0 is http://www.python.org/dev/peps/pep-0361/ 3.0 will have the feature, 2.6 may or may not. n On Thu, Apr 3, 2008 at 4:47 AM, Heshan Suriyaarachchi wrote: > Hi, > I need to work with annotations as it is said in [1]. Currently I am > using python 2.5.1. I would like to know whether the > next release of python will support this feature. If the next version > support this feature I would like to know when are you > planning to release it. I used the __future__ module but I could not get > the annotations with it. > > [1] - http://www.python.org/dev/peps/pep-3107/ > > Thanx > Heshan Suriyaarachchi > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > > From mail at timgolden.me.uk Fri Apr 4 10:04:25 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Fri, 04 Apr 2008 09:04:25 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk><52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com><47F33DE7.3030702@timgolden.me.uk> <47F3592D.3080700@timgolden.me.uk><47F4DE23.3020202@timgolden.me.uk> Message-ID: <47F5E109.4090608@timgolden.me.uk> Terry Reedy wrote: > If the testdir disallows the search indexer, then there should be no need > to disable Windows Search Service. If privatizing the dir kept other > programs out, then likewise. > > | Or were you suggesting that there is some programmatic way for the > | test suite to create directories that disallow the Search Service, > | etc.? > > I suspect, but do not know, that the dialog box effects changes through > user-programmable interfaces. So while I would start with manual changes > to see if that solves the problem, I presume there must be a system call > for changing dir attributes. The problem is, I think, that it isn't just the Indexing Service which generates this issue. TortoiseSVN is well known for doing the same thing, and there could be now and potentially will be in the future other programs. I don't think that hunting down and turning off their interference case by case is a viable solution in the long-term. Although it would obviously be a way forward in the short term, _faute de mieux_. [Tony Nelson] > I'd think that files and directories created in the TEMP > directory would normally not be indexed on any OS, including > MSWindows. But this is just a guess. I'm inclined to think you're right. And a first pass I had at producing a solution simply used tempfile to do everything. Unfortunately that's far more invasive than I was really comfortable with at the time: at the very least, you have to patch several tests which fail if there's an (escaped) backslash in the path. However, it's clear that my attempt to cause the minimum damage isn't enough to clear the problem 100%. So I think the next move is indeed to turn test_support.TESTFN into a function (in some way) which generates a unique tempfile reference, possibly with a context manager to clean up. Or something. The complication is that, while most test simply want a handy file to exist or be written to, and don't really care what happens afterwards, some tests are testing the very mechanism of creating/deleting a file etc. So a wholesale replacement isn't necessarily straightforward. On we go. TJG From mail at timgolden.me.uk Fri Apr 4 10:04:25 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Fri, 04 Apr 2008 09:04:25 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk><52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com><47F33DE7.3030702@timgolden.me.uk> <47F3592D.3080700@timgolden.me.uk><47F4DE23.3020202@timgolden.me.uk> Message-ID: <47F5E109.4090608@timgolden.me.uk> <<< No Message Collected >>> From mail at timgolden.me.uk Fri Apr 4 10:29:32 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Fri, 04 Apr 2008 09:29:32 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB27E@EXMBX04.exchhosting.com> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk> <47F3592D.3080700@timgolden.me.uk>, <47F4DE23.3020202@timgolden.me.uk> <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB27E@EXMBX04.exchhosting.com> Message-ID: <47F5E6EC.6030402@timgolden.me.uk> Trent Nelson wrote: > 1. For a given python[_d].exe, always use the same test directory, > but hash it against the entire python process path such that it's > unique only for a given python instance. This is to guard against several build runs in parallel, presumably? > 2. Make sure every time a test wants a temp file, it gets a > unique one in this directory. Sounds like your TESTFN > modification would take care of this for those tests using > TESTFN Not quite; at present TESTFN is constant for one set of test runs. It would need to be a function call to change for every test within the run. ; if TESTFN is the preferred approach then any other > tests using tempfile or hardcoding file names would then be > changed to use this instead. Part of the problem (for me): I don't know if it is "the preferred approach", merely that it's there and widely but not universally used and is the likely cause of the locking problems we're seeing. If anything's rolling its own tempfile solution, I'm inclined to leave that alone. As someone's pointed out elsewhere, the temp directories are almost certainly not monitored. > 3. In order to address tests that either call the > test_support methods for removing files/directories, > or those that call os.(unlink|remove), do what ever > needs to be done to make these no-ops on Windows if an > error occurs. I'm reluctant to monkey patch os.(unlink|remove). I'd rather provide and encourage the use of a test_support.remove_test_file function, or even a context manager which did the same, failing silently if needs be. > 4. At the end of the regrtest.py run, create a suspended arbitrary > process (i.e. explorer.exe), hijack the main thread context of the > process and inject a routine (i.e. overwrite the thread context's > instruction pointers) that takes care of removing the temporary > directory that was used for testing -- patiently re-trying if > any failures occur until all rogue processes also accessing the > file(s) stop doing so. Resume the thread before exiting python. OK. That was scary. (Interesting site, though). [... snip ...] > Guess it all depends on how much effort we want to put into cleaning > up our test directory really -- just ensuring tests get a clean area > and unique file names each run is the easy part. Yes. I'm trying desperately hard to stick to a narrow remit of getting tests to run consistently in the face of messy file-locking semantics under Windows. I really don't want to wade into the minor minefield of making all the tests run with consistent temp file semantics. But I may have to. TJG From tnelson at onresolve.com Fri Apr 4 10:29:16 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Fri, 4 Apr 2008 01:29:16 -0700 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F5E109.4090608@timgolden.me.uk> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk><52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com><47F33DE7.3030702@timgolden.me.uk> <47F3592D.3080700@timgolden.me.uk><47F4DE23.3020202@timgolden.me.uk> ,<47F5E109.4090608@timgolden.me.uk> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB281@EXMBX04.exchhosting.com> I agree with Tim, you can jump through as many hoops as you want (setting directories private, using %TEMP% exclusively, etc), but I doubt anything is going to change the behaviour of things like virus scanners, for example. Tim, let me know if you need help with anything, perhaps we could set up a temporary branch outside of trunk to play around with various approaches to see what works best. This'll ensure we don't adversely affect the main buildbots, but also give us the option to get different Windows boxes to build our test branch on demand. Trent. ________________________________________ From: python-dev-bounces+tnelson=onresolve.com at python.org [python-dev-bounces+tnelson=onresolve.com at python.org] On Behalf Of Tim Golden [mail at timgolden.me.uk] Sent: 04 April 2008 04:04 Cc: python-dev at python.org Subject: Re: [Python-Dev] fixing tests on windows Terry Reedy wrote: > If the testdir disallows the search indexer, then there should be no need > to disable Windows Search Service. If privatizing the dir kept other > programs out, then likewise. > > | Or were you suggesting that there is some programmatic way for the > | test suite to create directories that disallow the Search Service, > | etc.? > > I suspect, but do not know, that the dialog box effects changes through > user-programmable interfaces. So while I would start with manual changes > to see if that solves the problem, I presume there must be a system call > for changing dir attributes. The problem is, I think, that it isn't just the Indexing Service which generates this issue. TortoiseSVN is well known for doing the same thing, and there could be now and potentially will be in the future other programs. I don't think that hunting down and turning off their interference case by case is a viable solution in the long-term. Although it would obviously be a way forward in the short term, _faute de mieux_. [Tony Nelson] > I'd think that files and directories created in the TEMP > directory would normally not be indexed on any OS, including > MSWindows. But this is just a guess. I'm inclined to think you're right. And a first pass I had at producing a solution simply used tempfile to do everything. Unfortunately that's far more invasive than I was really comfortable with at the time: at the very least, you have to patch several tests which fail if there's an (escaped) backslash in the path. However, it's clear that my attempt to cause the minimum damage isn't enough to clear the problem 100%. So I think the next move is indeed to turn test_support.TESTFN into a function (in some way) which generates a unique tempfile reference, possibly with a context manager to clean up. Or something. The complication is that, while most test simply want a handy file to exist or be written to, and don't really care what happens afterwards, some tests are testing the very mechanism of creating/deleting a file etc. So a wholesale replacement isn't necessarily straightforward. On we go. TJG _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com From schmir at gmail.com Fri Apr 4 11:46:46 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Fri, 4 Apr 2008 11:46:46 +0200 Subject: [Python-Dev] __eq__ vs hash Message-ID: <932f8baf0804040246s261c1cd2ic977934cbd049731@mail.gmail.com> Hi all, the news file for python 2.6 does not mention that you need to define __hash__ in case you define __eq__ for a class. This breaks some code (for me: mercurial and pyparsing). Shouldn't this be documented somewhere (I also cannot find it in the whatsnew file). - Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080404/a00b2a48/attachment-0001.htm From tnelson at onresolve.com Fri Apr 4 12:57:15 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Fri, 4 Apr 2008 03:57:15 -0700 Subject: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbot/bui... In-Reply-To: <47F5E5EF.5010005@cheimes.de> References: <20080403182707.506A41E400D@bag.python.org>, <47F5E5EF.5010005@cheimes.de> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB286@EXMBX04.exchhosting.com> > I don't like the part where the solution kills the Python process during > a rebuild. It's too surprising for the user. Hmmm. When you say rebuild, I assume you mean the change I made to the pythoncore project's pre-link step to call kill_python.exe, and not to the post-build step of kill_python that runs itself? Personally, when I'm doing development, if I've got the pcbuild\python_d.exe console open, it's usually to test one liners, I'm not using it to do anything important. If I forget to close it before I kick off a build, it's annoying running into errors at the link stage, I'd certainly prefer the build system to kill off anything that'll inhibit a successful link before actually linking. What do others that do Windows development think? I don't have a problem changing the build behaviour if the approach I've taken is generally disliked. Trent. ________________________________________ From: Christian Heimes [lists at cheimes.de] Sent: 04 April 2008 09:25 To: python-checkins at python.org; Trent Nelson Subject: Re: r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbot/bui... trent.nelson schrieb: > Author: trent.nelson > Date: Thu Apr 3 20:27:06 2008 > New Revision: 62129 > > Added: > python/trunk/PCbuild/kill_python.c (contents, props changed) > python/trunk/PCbuild/kill_python.vcproj > Removed: > python/trunk/Tools/buildbot/Makefile > python/trunk/Tools/buildbot/kill_python.bat > python/trunk/Tools/buildbot/kill_python.c > python/trunk/Tools/buildbot/kill_python.mak > Modified: > python/trunk/PCbuild/debug.vsprops > python/trunk/PCbuild/pcbuild.sln > python/trunk/PCbuild/pythoncore.vcproj > python/trunk/PCbuild/release.vsprops > python/trunk/Tools/buildbot/build-amd64.bat > python/trunk/Tools/buildbot/build.bat > python/trunk/Tools/buildbot/buildmsi.bat > Log: > Reimplement kill_python. The existing version had a number of flaws, namely, it didn't work for x64 and it wasn't precise about which python_d.exe it was killing -- it just killed the first one it came across that happened to have 'pcbuild\python_d.exe' or 'build\python_d.exe' in it's path. The new version has been rewritten from the ground up and now lives in PCbuild, instead of Tools\buildbot, and it has also been incorporated into the Visual Studio solution (pcbuild.sln) as 'kill_python'. The solution has also been altered such that kill_python is called where necessary in the build process in order to prevent any linking errors due to open file locks. In lieu of this, all of the existing bits and pieces in Tools\buildbot that called out to kill_python at various points have also be > en removed as they are now obsolete. Tested on both Win32 and x64. I don't like the part where the solution kills the Python process during a rebuild. It's too surprising for the user. Christian From amauryfa at gmail.com Fri Apr 4 14:55:38 2008 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 4 Apr 2008 14:55:38 +0200 Subject: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/build Message-ID: On Fri, Apr 4, 2008 at 12:57 PM, Trent Nelson wrote: > > > I don't like the part where the solution kills the Python process during > > a rebuild. It's too surprising for the user. > > Hmmm. When you say rebuild, I assume you mean the change I made to the pythoncore project's pre-link step to call kill_python.exe, and not to the post-build step of kill_python that runs itself? Personally, when I'm doing development, if I've got the pcbuild\python_d.exe console open, it's usually to test one liners, I'm not using it to do anything important. If I forget to close it before I kick off a build, it's annoying running into errors at the link stage, I'd certainly prefer the build system to kill off anything that'll inhibit a successful link before actually linking. > > What do others that do Windows development think? I don't have a problem changing the build behaviour if the approach I've taken is generally disliked. I agree with Christian: in interactive sessions, the F7 key should not kill my running testsuite... I prefer the linker errors. Please do this only for buildbot builds! Or maybe have it controlled by an enviroment variable. -- Amaury Forgeot d'Arc From amauryfa at gmail.com Fri Apr 4 14:55:38 2008 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 4 Apr 2008 14:55:38 +0200 Subject: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/build Message-ID: On Fri, Apr 4, 2008 at 12:57 PM, Trent Nelson wrote: > > > I don't like the part where the solution kills the Python process during > > a rebuild. It's too surprising for the user. > > Hmmm. When you say rebuild, I assume you mean the change I made to the pythoncore project's pre-link step to call kill_python.exe, and not to the post-build step of kill_python that runs itself? Personally, when I'm doing development, if I've got the pcbuild\python_d.exe console open, it's usually to test one liners, I'm not using it to do anything important. If I forget to close it before I kick off a build, it's annoying running into errors at the link stage, I'd certainly prefer the build system to kill off anything that'll inhibit a successful link before actually linking. > > What do others that do Windows development think? I don't have a problem changing the build behaviour if the approach I've taken is generally disliked. I agree with Christian: in interactive sessions, the F7 key should not kill my running testsuite... I prefer the linker errors. Please do this only for buildbot builds! Or maybe have it controlled by an enviroment variable. -- Amaury Forgeot d'Arc From musiccomposition at gmail.com Fri Apr 4 15:38:21 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Fri, 4 Apr 2008 08:38:21 -0500 Subject: [Python-Dev] Annotations support in python In-Reply-To: References: <12e5d0d90804030447u67391abby361d0cabcb919018@mail.gmail.com> Message-ID: <1afaf6160804040638m73a2ea3fjff046a17c30f66bb@mail.gmail.com> On Thu, Apr 3, 2008 at 11:49 PM, Neal Norwitz wrote: > The release schedule for 2.6/3.0 is > http://www.python.org/dev/peps/pep-0361/ > 3.0 will have the feature, 2.6 may or may not. There is an issue for this: http://bugs.python.org/issue2331 > > > n > > On Thu, Apr 3, 2008 at 4:47 AM, Heshan Suriyaarachchi > wrote: > > Hi, > > I need to work with annotations as it is said in [1]. Currently I > am > > using python 2.5.1. I would like to know whether the > > next release of python will support this feature. If the next version > > support this feature I would like to know when are you > > planning to release it. I used the __future__ module but I could not > get > > the annotations with it. > > > > [1] - http://www.python.org/dev/peps/pep-3107/ > > > > Thanx > > Heshan Suriyaarachchi > > > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com > -- Cheers, Benjamin Peterson -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080404/abaec5a5/attachment.htm From gh at ghaering.de Fri Apr 4 16:26:40 2008 From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=) Date: Fri, 04 Apr 2008 16:26:40 +0200 Subject: [Python-Dev] RELEASED Python 2.6a2 and 3.0a4 In-Reply-To: <3C3C0150-ED65-4381-9F54-BB437DD6DFB9@python.org> References: <3C3C0150-ED65-4381-9F54-BB437DD6DFB9@python.org> Message-ID: <47F63AA0.5040106@ghaering.de> Hello Barry, Great job with the releases! Barry Warsaw wrote: > On behalf of the Python development team and the Python community, I'm > happy to announce the second alpha release of Python 2.6, and the > fourth alpha release of Python 3.0. [...] In case you don't know already, the website is not fully updated, yet: http://www.python.org/download/releases/2.6/ Top of the page says "2.6a1" instead of "2.6a2". This file is apparently not yet updated for 2.6alpha2. http://www.python.org/download/releases/2.6/NEWS.txt For the 3.0 version: Top of the page still says "3.0a3". Cheers, -- Gerhard From guido at python.org Fri Apr 4 16:38:04 2008 From: guido at python.org (Guido van Rossum) Date: Fri, 4 Apr 2008 07:38:04 -0700 Subject: [Python-Dev] __eq__ vs hash In-Reply-To: <932f8baf0804040246s261c1cd2ic977934cbd049731@mail.gmail.com> References: <932f8baf0804040246s261c1cd2ic977934cbd049731@mail.gmail.com> Message-ID: On Fri, Apr 4, 2008 at 2:46 AM, Ralf Schmitt wrote: > the news file for python 2.6 does not mention that you need to define > __hash__ in case you define __eq__ for a class. > This breaks some code (for me: mercurial and pyparsing). > Shouldn't this be documented somewhere (I also cannot find it in the > whatsnew file). Well, technically this has always been the requirement. What specific code breaks? Maybe we need to turn this into a warning in order to be more backwards compatible? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From schmir at gmail.com Fri Apr 4 16:53:52 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Fri, 4 Apr 2008 16:53:52 +0200 Subject: [Python-Dev] __eq__ vs hash In-Reply-To: References: <932f8baf0804040246s261c1cd2ic977934cbd049731@mail.gmail.com> Message-ID: <932f8baf0804040753g74cc5200j738dc29b0b5a4c1c@mail.gmail.com> On Fri, Apr 4, 2008 at 4:38 PM, Guido van Rossum wrote: > On Fri, Apr 4, 2008 at 2:46 AM, Ralf Schmitt wrote: > > the news file for python 2.6 does not mention that you need to define > > __hash__ in case you define __eq__ for a class. > > This breaks some code (for me: mercurial and pyparsing). > > Shouldn't this be documented somewhere (I also cannot find it in the > > whatsnew file). > > Well, technically this has always been the requirement. > > What specific code breaks? Maybe we need to turn this into a warning > in order to be more backwards compatible? > I don't feel like digging into mercurial or pyparsing code currently. sorry. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080404/44c2b722/attachment.htm From guido at python.org Fri Apr 4 17:05:44 2008 From: guido at python.org (Guido van Rossum) Date: Fri, 4 Apr 2008 08:05:44 -0700 Subject: [Python-Dev] __eq__ vs hash In-Reply-To: <932f8baf0804040753g74cc5200j738dc29b0b5a4c1c@mail.gmail.com> References: <932f8baf0804040246s261c1cd2ic977934cbd049731@mail.gmail.com> <932f8baf0804040753g74cc5200j738dc29b0b5a4c1c@mail.gmail.com> Message-ID: Understood. Neither do I. :-) But maybe you could get the authors of that code into this discussion? On Fri, Apr 4, 2008 at 7:53 AM, Ralf Schmitt wrote: > > > > On Fri, Apr 4, 2008 at 4:38 PM, Guido van Rossum wrote: > > > > > > > > On Fri, Apr 4, 2008 at 2:46 AM, Ralf Schmitt wrote: > > > the news file for python 2.6 does not mention that you need to define > > > __hash__ in case you define __eq__ for a class. > > > This breaks some code (for me: mercurial and pyparsing). > > > Shouldn't this be documented somewhere (I also cannot find it in the > > > whatsnew file). > > > > Well, technically this has always been the requirement. > > > > > > What specific code breaks? Maybe we need to turn this into a warning > > in order to be more backwards compatible? > > > > I don't feel like digging into mercurial or pyparsing code currently. sorry. > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jason.orendorff at gmail.com Fri Apr 4 17:08:26 2008 From: jason.orendorff at gmail.com (Jason Orendorff) Date: Fri, 4 Apr 2008 10:08:26 -0500 Subject: [Python-Dev] __eq__ vs hash In-Reply-To: References: <932f8baf0804040246s261c1cd2ic977934cbd049731@mail.gmail.com> Message-ID: On Fri, Apr 4, 2008 at 9:38 AM, Guido van Rossum wrote: > What specific code breaks? Maybe we need to turn this into a warning > in order to be more backwards compatible? I looked at Mercurial. It doesn't use __hash__ at all. It uses __eq__ in two files, three total uses: http://hg.intevation.org/mercurial/crew/file/6c4e12682fb9/mercurial/commands.py http://hg.intevation.org/mercurial/crew/file/6c4e12682fb9/mercurial/context.py -j From status at bugs.python.org Fri Apr 4 18:06:30 2008 From: status at bugs.python.org (Tracker) Date: Fri, 4 Apr 2008 18:06:30 +0200 (CEST) Subject: [Python-Dev] Summary of Tracker Issues Message-ID: <20080404160630.91C05780F1@psf.upfronthosting.co.za> ACTIVITY SUMMARY (03/28/08 - 04/04/08) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1799 open (+34) / 12577 closed (+13) / 14376 total (+47) Open issues with patches: 536 Average duration of open issues: 716 days. Median duration of open issues: 1257 days. Open Issues Breakdown open 1774 (+34) pending 25 ( +0) Issues Created Or Reopened (47) _______________________________ Add gettext.pgettext() and variants support 03/29/08 http://bugs.python.org/issue2504 created genepi Easier creation of _ast nodes 03/29/08 CLOSED http://bugs.python.org/issue2505 created georg.brandl patch Line tracing of continue after always-taken if is incorrect 03/29/08 CLOSED http://bugs.python.org/issue2506 created nedbat Exception state lives too long in 3.0 03/29/08 http://bugs.python.org/issue2507 created pitrou patch When you create a file using file(path, "w") if the filename is 03/29/08 CLOSED http://bugs.python.org/issue2508 created kobipe3 Bazaar ignore file 03/29/08 CLOSED http://bugs.python.org/issue2509 created benjamin.peterson patch, easy Bazaar ignore file 03/29/08 http://bugs.python.org/issue2510 created benjamin.peterson patch, easy Give AST's excepthandler proper attributes 03/29/08 CLOSED http://bugs.python.org/issue2511 created georg.brandl patch decide what to do with gettext API 03/29/08 http://bugs.python.org/issue2512 created benjamin.peterson patch 64bit cross compilation on windows 03/30/08 http://bugs.python.org/issue2513 created mhammond patch, 64bit new AST init breaks on Store and other fieldless Nodes 03/30/08 CLOSED http://bugs.python.org/issue2514 created aronacher patch Segfault while operating on closed sqlite3 cursor. 03/30/08 http://bugs.python.org/issue2515 created pjdavis Instance methods are misreporting the number of arguments 03/30/08 http://bugs.python.org/issue2516 created belopolsky patch Error when printing an exception containing a Unicode string 03/31/08 http://bugs.python.org/issue2517 reopened benjamin.peterson patch smtpd.py to handle huge email 03/31/08 http://bugs.python.org/issue2518 created kawai patch Typing 'modules' in the interactive help system fails when impor 03/31/08 http://bugs.python.org/issue2519 created dennis macerrors.py cannot be imported due to non-ascii characters in c 03/31/08 http://bugs.python.org/issue2520 created thecube patch ABC caches should use weak refs 03/31/08 http://bugs.python.org/issue2521 created amaury.forgeotdarc 26backport locale.format() problems with decimal separator 03/31/08 http://bugs.python.org/issue2522 created mishok13 patch binary buffered reading is quadratic 03/31/08 http://bugs.python.org/issue2523 created pitrou IDLE 3.0 03/31/08 CLOSED http://bugs.python.org/issue2524 created kbk class USTimeZone in Doc/includes/tzinfo-examples.py is out of da 04/01/08 http://bugs.python.org/issue2525 created gvanrossum patch, easy str.format() :n format does not appear to work 04/01/08 http://bugs.python.org/issue2526 created mark Pass a namespace to timeit 04/01/08 http://bugs.python.org/issue2527 created potten Change os.access to check ACLs under Windows 04/01/08 http://bugs.python.org/issue2528 created tim.golden patch list/generator comprehension parser doesn't match spec 04/01/08 CLOSED http://bugs.python.org/issue2529 created kousu Document IO module 04/01/08 http://bugs.python.org/issue2530 created benjamin.peterson float compared to decimal is silently incorrect. 04/01/08 CLOSED http://bugs.python.org/issue2531 created jdunck easy file that breaks 2to3 (despite being correct python) 04/02/08 http://bugs.python.org/issue2532 created fijal Invalid TypeError with class method decorator and class method p 04/02/08 CLOSED http://bugs.python.org/issue2533 created tdimson Speed up isinstance and issubclass 04/02/08 http://bugs.python.org/issue2534 created theller patch, patch duplicate Misc.lower 04/02/08 http://bugs.python.org/issue2535 created mkiever patch itertools.permutations docstring is misleading 04/02/08 http://bugs.python.org/issue2536 created ajaksu2 re.compile(r'((x|y+)*)*') should fail 04/02/08 http://bugs.python.org/issue2537 created jorendorff memoryview of bytes is not readonly 04/03/08 http://bugs.python.org/issue2538 created amaury.forgeotdarc patch Windows Registry issue with 2.5.2 AMD64 msi 04/03/08 http://bugs.python.org/issue2539 created iggyboo If HAVE_LONG_LONG is not defined, longval will not be initialize 04/03/08 http://bugs.python.org/issue2540 created ocean-city patch, patch Unicode escape sequences not parsed in raw strings. 04/03/08 CLOSED http://bugs.python.org/issue2541 reopened amaury.forgeotdarc PyErr_ExceptionMatches must not fail 04/03/08 http://bugs.python.org/issue2542 created theller patch, patch Make ctypes compatible with Python 2.3, 2.4, and 2.5, as per PEP 04/03/08 CLOSED http://bugs.python.org/issue2543 created theller patch, patch link with "gcc -shared" on HP systems 04/03/08 CLOSED http://bugs.python.org/issue2544 created theller patch, patch sphinx.ext.autodoc fails to expand tabs in docstrings 04/04/08 http://bugs.python.org/issue2545 created jmillikin Python-2.5.2: crash in visit_decref () at Modules/gcmodule.c:270 04/04/08 http://bugs.python.org/issue2546 created garikvm Py30a4 RELNOTES only cover 30a1 and 30a2 04/04/08 http://bugs.python.org/issue2547 created mark Undetected error in exception handling 04/04/08 http://bugs.python.org/issue2548 created theller shutil: NameError (WindowsError) when moving from ext3 to fat32 04/04/08 http://bugs.python.org/issue2549 created jerome.chabod easy SO_REUSEADDR doesn't have the same semantics on Windows as on Un 04/04/08 http://bugs.python.org/issue2550 created Trent.Nelson patch, 26backport Issues Now Closed (45) ______________________ %f format for datetime objects 198 days http://bugs.python.org/issue1158 georg.brandl patch pythonstartup addition of minor error checking 136 days http://bugs.python.org/issue1442 georg.brandl patch py3k: test_mailbox fails on Windows 119 days http://bugs.python.org/issue1561 amaury.forgeotdarc patch logging documentation is unclear 110 days http://bugs.python.org/issue1579 vsajip easy Backport of io.py 65 days http://bugs.python.org/issue1919 georg.brandl Undocumented lastrowid attribute i sqlite3 cursor class 35 days http://bugs.python.org/issue2176 georg.brandl patch test_sqlite fails in 2.6a1 on MacOS X 28 days http://bugs.python.org/issue2215 ghaering TimedRotatingFileHandler does not account for daylight savings t 17 days http://bugs.python.org/issue2315 vsajip TimedRotatingFileHandler names files incorrectly if nothing is l 17 days http://bugs.python.org/issue2316 vsajip TimedRotatingFileHandler logic for removing files wrong 17 days http://bugs.python.org/issue2317 vsajip TimedRotatingFileHandler: rotate every month, or every year 17 days http://bugs.python.org/issue2318 vsajip Backport keyword-only arguments to 2.6 11 days http://bugs.python.org/issue2327 georg.brandl 26backport Logging module hides user code errors (bare except) 14 days http://bugs.python.org/issue2424 vsajip pysqlite provides no interface for database SQL dump 9 days http://bugs.python.org/issue2426 kippesp patch Patch for test_urllib2_net moving tests to use a local server 9 days http://bugs.python.org/issue2429 gregory.p.smith patch Stop test_format.py from executing as side effect of import 9 days http://bugs.python.org/issue2430 georg.brandl patch, patch Adding __iter__ to class Values of module optparse 12 days http://bugs.python.org/issue2444 benjamin.peterson patch logging.LogRecord should know how to handle dict-like args 5 days http://bugs.python.org/issue2473 vsajip parser support for future import of unicode_strings 4 days http://bugs.python.org/issue2477 nnorwitz patch Bytearray and io backport 4 days http://bugs.python.org/issue2479 georg.brandl 26backport bdb modernized 1 days http://bugs.python.org/issue2498 benjamin.peterson patch Sync _sqlite module code 1 days http://bugs.python.org/issue2500 ghaering Add enum() example for named tuples 1 days http://bugs.python.org/issue2502 rhettinger patch Replace "== None/True/False" with "is" 1 days http://bugs.python.org/issue2503 benjamin.peterson patch Easier creation of _ast nodes 1 days http://bugs.python.org/issue2505 georg.brandl patch Line tracing of continue after always-taken if is incorrect 3 days http://bugs.python.org/issue2506 nedbat When you create a file using file(path, "w") if the filename is 0 days http://bugs.python.org/issue2508 loewis Bazaar ignore file 0 days http://bugs.python.org/issue2509 benjamin.peterson patch, easy Give AST's excepthandler proper attributes 1 days http://bugs.python.org/issue2511 georg.brandl patch new AST init breaks on Store and other fieldless Nodes 0 days http://bugs.python.org/issue2514 georg.brandl patch IDLE 3.0 0 days http://bugs.python.org/issue2524 kbk list/generator comprehension parser doesn't match spec 1 days http://bugs.python.org/issue2529 kousu float compared to decimal is silently incorrect. 1 days http://bugs.python.org/issue2531 facundobatista easy Invalid TypeError with class method decorator and class method p 0 days http://bugs.python.org/issue2533 amaury.forgeotdarc Unicode escape sequences not parsed in raw strings. 0 days http://bugs.python.org/issue2541 benjamin.peterson Make ctypes compatible with Python 2.3, 2.4, and 2.5, as per PEP 1 days http://bugs.python.org/issue2543 theller patch, patch link with "gcc -shared" on HP systems 1 days http://bugs.python.org/issue2544 theller patch, patch file (& socket) I/O are not thread safe 2052 days http://bugs.python.org/issue595601 georg.brandl compiler.transformer: correct lineno attribute when possible 1311 days http://bugs.python.org/issue1015989 benjamin.peterson patch automatic XMLRPC boxcarring via multicall for xmlrpclib 1250 days http://bugs.python.org/issue1054434 loewis patch Segfaults with concurrent sqlite db access and schema change 582 days http://bugs.python.org/issue1546263 ghaering Python is dumping core after the test test_ctypes 528 days http://bugs.python.org/issue1582742 theller sqlite3 documentation omits: close(), commit(), autocommit 454 days http://bugs.python.org/issue1625205 ghaering VC6 build patch for trunk 353 days http://bugs.python.org/issue1700463 ocean-city patch RuntimeWarning: tp_compare didn't return -1 or -2 300 days http://bugs.python.org/issue1733757 amaury.forgeotdarc Top Issues Most Discussed (10) ______________________________ 14 Error when printing an exception containing a Unicode string 4 days open http://bugs.python.org/issue2517 14 Line tracing of continue after always-taken if is incorrect 3 days closed http://bugs.python.org/issue2506 13 Bazaar ignore file 6 days open http://bugs.python.org/issue2510 11 64bit cross compilation on windows 5 days open http://bugs.python.org/issue2513 10 Fast BytesIO implementation + misc changes 88 days open http://bugs.python.org/issue1751 8 thread unsafe file objects cause crash 1647 days open http://bugs.python.org/issue815646 6 Instance methods are misreporting the number of arguments 5 days open http://bugs.python.org/issue2516 6 run_setup can fail if the setup script uses __file__ 17 days open http://bugs.python.org/issue2385 6 urllib2 header capitalization 24 days open http://bugs.python.org/issue2275 6 implement warnings module in C 451 days pending http://bugs.python.org/issue1631171 From tnelson at onresolve.com Fri Apr 4 18:07:22 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Fri, 4 Apr 2008 09:07:22 -0700 Subject: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB280@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB280@EXMBX04.exchhosting.com> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB28D@EXMBX04.exchhosting.com> I've raised issue 2550 to track this problem. I've also provided a patch on the tracker to test_socket.py that reproduces the issue. Anyone mind if I commit this to trunk? I'd like to observe if any other platforms exhibit different behaviour via buildbots. It'll cause all the Windows slaves to fail on test_socket though. (I can revert it once I've seen how the buildbots behave until I can come up with an actual patch for Windows that fixes the issue.) http://bugs.python.org/issue2550 http://bugs.python.org/file9939/test_socket.py.patch Trent. ________________________________________ From: python-dev-bounces+tnelson=onresolve.com at python.org [python-dev-bounces+tnelson=onresolve.com at python.org] On Behalf Of Trent Nelson [tnelson at onresolve.com] Sent: 03 April 2008 22:40 To: python-dev at python.org Subject: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) I started looking into this: http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/289/step-test/0 Pertinent part: test_asyncore test_asynchat command timed out: 1200 seconds without output SIGKILL failed to kill process using fake rc=-1 program finished with exit code -1 remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last): Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process ] I tried to replicate it on the buildbot in order to debug, which, surprisingly, I could do consistently by just running rt.bat -q -d -uall test_asynchat. As the log above indicates, the python process becomes completely and utterly wedged, to the point that I can't even attach a remote debugger and step into it. Digging further, I noticed that if I ran the following code in two different python consoles, EADDRINUSE was *NOT* being raised by socket.bind(): import socket s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.bind(('127.0.0.1', 54322)) However, take out the setsockopt line, and wallah, the second s.bind() will raise EADDRINUSE, as expected. This manifests into a really bizarre issue with test_asynchat in particualr, as subsequent sock.accept() calls on the socket put python into the uber wedged state (can't even ctrl-c out at the console, need to kill the process directly). Have to leave the office and head home so I don't have any more time to look at it tonight -- just wanted to post here for others to mull over. Trent. _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com From guido at python.org Fri Apr 4 18:08:16 2008 From: guido at python.org (Guido van Rossum) Date: Fri, 4 Apr 2008 09:08:16 -0700 Subject: [Python-Dev] __eq__ vs hash In-Reply-To: References: <932f8baf0804040246s261c1cd2ic977934cbd049731@mail.gmail.com> Message-ID: On Fri, Apr 4, 2008 at 8:08 AM, Jason Orendorff wrote: > On Fri, Apr 4, 2008 at 9:38 AM, Guido van Rossum wrote: > > > What specific code breaks? Maybe we need to turn this into a warning > > in order to be more backwards compatible? > > I looked at Mercurial. > > It doesn't use __hash__ at all. It uses __eq__ in two files, three total uses: > http://hg.intevation.org/mercurial/crew/file/6c4e12682fb9/mercurial/commands.py > http://hg.intevation.org/mercurial/crew/file/6c4e12682fb9/mercurial/context.py If it breaks, perhaps the instances of one of those classes that define __eq__ but not __hash__ are used in a set or as dict keys? That's the difference between 2.5 and 2.6 in this area. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From p.f.moore at gmail.com Fri Apr 4 18:30:57 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 4 Apr 2008 17:30:57 +0100 Subject: [Python-Dev] __eq__ vs hash In-Reply-To: References: <932f8baf0804040246s261c1cd2ic977934cbd049731@mail.gmail.com> Message-ID: <79990c6b0804040930y4d29d6a8s1805c4879719c2a@mail.gmail.com> On 04/04/2008, Guido van Rossum wrote: > > It doesn't use __hash__ at all. It uses __eq__ in two files, three total uses: > > http://hg.intevation.org/mercurial/crew/file/6c4e12682fb9/mercurial/commands.py > > http://hg.intevation.org/mercurial/crew/file/6c4e12682fb9/mercurial/context.py > > > If it breaks, perhaps the instances of one of those classes that > define __eq__ but not __hash__ are used in a set or as dict keys? > That's the difference between 2.5 and 2.6 in this area. That looks like it. I'll work up a patch and submit it to the Mercurial developers. Paul. From dirkjan at ochtman.nl Fri Apr 4 19:29:58 2008 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Fri, 04 Apr 2008 19:29:58 +0200 Subject: [Python-Dev] __eq__ vs hash In-Reply-To: <79990c6b0804040930y4d29d6a8s1805c4879719c2a@mail.gmail.com> References: <932f8baf0804040246s261c1cd2ic977934cbd049731@mail.gmail.com> <79990c6b0804040930y4d29d6a8s1805c4879719c2a@mail.gmail.com> Message-ID: <47F66596.2020507@ochtman.nl> Paul Moore wrote: > That looks like it. I'll work up a patch and submit it to the > Mercurial developers. I've already got one going. Cheers, Dirkjan From theller at ctypes.org Fri Apr 4 20:52:24 2008 From: theller at ctypes.org (Thomas Heller) Date: Fri, 04 Apr 2008 20:52:24 +0200 Subject: [Python-Dev] osx x86 buildbot offer Message-ID: I can offer an OS X x86 machine to run a buildbot on. This is a physical machine, running OS X 10.5 Leopard. Thomas From tjreedy at udel.edu Fri Apr 4 20:54:28 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 4 Apr 2008 14:54:28 -0400 Subject: [Python-Dev] fixing tests on windows References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk><52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com><47F33DE7.3030702@timgolden.me.uk> <47F3592D.3080700@timgolden.me.uk><47F4DE23.3020202@timgolden.me.uk> <47F5E109.4090608@timgolden.me.uk> Message-ID: "Tim Golden" wrote in message news:47F5E109.4090608 at timgolden.me.uk... | Terry Reedy wrote: | > I suspect, but do not know, that the dialog box effects changes through | > user-programmable interfaces. So while I would start with manual changes | > to see if that solves the problem, I presume there must be a system call | > for changing dir attributes. | | The problem is, I think, that it isn't just the Indexing Service | which generates this issue. TortoiseSVN is well known for doing | the same thing, and there could be now and potentially will be | in the future other programs. I don't think that hunting down | and turning off their interference case by case is a viable | solution in the long-term. And that is exactly not what I was suggesting. I do not know how much privitizing a directory actually blocks, but if Microsoft says 'only you can access the folder', perhaps it would block some things. Don't most scanners read files rapidly, close, and move on? Perhaps testall could either hang around at the end trying to make sure all is removed. Perhaps display a message, wait five seconds, try again, up to a minute or so. tjr From martin at v.loewis.de Fri Apr 4 20:57:08 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 04 Apr 2008 20:57:08 +0200 Subject: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbot/bui... In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB286@EXMBX04.exchhosting.com> References: <20080403182707.506A41E400D@bag.python.org>, <47F5E5EF.5010005@cheimes.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB286@EXMBX04.exchhosting.com> Message-ID: <47F67A04.4020805@v.loewis.de> > What do others that do Windows development think? I don't have a > problem changing the build behaviour if the approach I've taken is > generally disliked. I think kill_python should only ever be invoked in the build slaves; it should *not* be part of the regular build. If developers find they can't build because some files are still open, they should kill the processes themselves. Regards, Martin From tjreedy at udel.edu Fri Apr 4 21:03:14 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 4 Apr 2008 15:03:14 -0400 Subject: [Python-Dev] RELEASED Python 2.6a2 and 3.0a4 References: <3C3C0150-ED65-4381-9F54-BB437DD6DFB9@python.org> <47F63AA0.5040106@ghaering.de> Message-ID: "Gerhard H?ring" wrote in message news:47F63AA0.5040106 at ghaering.de... | In case you don't know already, the website is not fully updated, yet: | | http://www.python.org/download/releases/2.6/ | | Top of the page says "2.6a1" instead of "2.6a2". | | This file is apparently not yet updated for 2.6alpha2. | | http://www.python.org/download/releases/2.6/NEWS.txt | | | For the 3.0 version: | | Top of the page still says "3.0a3". The tops of the pages are fixed, but the left sidebar DOWNLOAD/Releases list still says 6a1 and 0a3. From tnelson at onresolve.com Fri Apr 4 21:41:43 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Fri, 4 Apr 2008 12:41:43 -0700 Subject: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbo... In-Reply-To: <47F67A04.4020805@v.loewis.de> References: <20080403182707.506A41E400D@bag.python.org>, <47F5E5EF.5010005@cheimes.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB286@EXMBX04.exchhosting.com>, <47F67A04.4020805@v.loewis.de> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB290@EXMBX04.exchhosting.com> Ok, I'll change the approach this weekend. Trent. ________________________________________ From: "Martin v. L?wis" [martin at v.loewis.de] Sent: 04 April 2008 19:57 To: Trent Nelson Cc: Christian Heimes; python-dev at python.org Subject: Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbo... > What do others that do Windows development think? I don't have a > problem changing the build behaviour if the approach I've taken is > generally disliked. I think kill_python should only ever be invoked in the build slaves; it should *not* be part of the regular build. If developers find they can't build because some files are still open, they should kill the processes themselves. Regards, Martin From barry at python.org Fri Apr 4 21:54:58 2008 From: barry at python.org (Barry Warsaw) Date: Fri, 4 Apr 2008 15:54:58 -0400 Subject: [Python-Dev] RELEASED Python 2.6a2 and 3.0a4 In-Reply-To: References: <3C3C0150-ED65-4381-9F54-BB437DD6DFB9@python.org> <47F63AA0.5040106@ghaering.de> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 4, 2008, at 3:03 PM, Terry Reedy wrote: > > "Gerhard H?ring" wrote in message > news:47F63AA0.5040106 at ghaering.de... > | In case you don't know already, the website is not fully updated, > yet: > | > | http://www.python.org/download/releases/2.6/ > | > | Top of the page says "2.6a1" instead of "2.6a2". > | > | This file is apparently not yet updated for 2.6alpha2. > | > | http://www.python.org/download/releases/2.6/NEWS.txt > | > | > | For the 3.0 version: > | > | Top of the page still says "3.0a3". > > The tops of the pages are fixed, but the left sidebar DOWNLOAD/ > Releases > list still says 6a1 and 0a3. I think I fixed these -- at least a grep of my local tree turns up no other references to 2.6a1 or 3.0a3. It might take a little while for the site to get rebuilt though. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBR/aHknEjvBPtnXfVAQJ3TQP+IrKnzpxWlJoEDovpg4hb2h4mcVhB5h6+ cpKczLE6DWlC4H4t76hZdRIOf/unAX6FeaTzlA7xLiXNkIlBT455k39jL98ZUmPy OjOw6n96WyhWL6zmVBnF4YuBFKo2D1Sj9rzYF8i1qtwY3Xq0rvtSSTNmVSnqXDOu x6H5Z5lm1AY= =Qzkl -----END PGP SIGNATURE----- From gregor.lingl at aon.at Fri Apr 4 22:07:07 2008 From: gregor.lingl at aon.at (Gregor Lingl) Date: Fri, 04 Apr 2008 22:07:07 +0200 Subject: [Python-Dev] RELEASED Python 2.6a2 and 3.0a4 In-Reply-To: <3C3C0150-ED65-4381-9F54-BB437DD6DFB9@python.org> References: <3C3C0150-ED65-4381-9F54-BB437DD6DFB9@python.org> Message-ID: <47F68A6B.9060901@aon.at> Hi, something doesn't work as usual, at least for me: When I try to download the Python 2.6a2 release for Windows by clicking * Windows x86 MSI Installer (2.6a2) (sig) instead of the usual download dialog I get an Error 404: File not found for the url http://www.python.org/ftp/python/2.6/python-2.6a2.msi . The same is true when trying to download Python 3.0 a4. Embarassed, Gregor Barry Warsaw schrieb: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On behalf of the Python development team and the Python community, I'm > happy to announce the second alpha release of Python 2.6, and the > fourth alpha release of Python 3.0. > > Please note that these are alpha releases, and as such are not > suitable for production environments. We continue to strive for a > high degree of quality, but there are still some known problems and > the feature sets have not been finalized. These alphas are being > released to solicit feedback and hopefully discover bugs, as well as > allowing you to determine how changes in 2.6 and 3.0 might impact > you. If you find things broken or incorrect, please submit a bug > report at > > http://bugs.python.org > > For more information and downloadable distributions, see the Python > 2.6 web > site: > > http://www.python.org/download/releases/2.6/ > > and the Python 3.0 web site: > > http://www.python.org/download/releases/3.0/ > > We are planning one more alpha release of each version, followed by > two beta releases, with the final releases planned for August 2008. > See PEP 361 for release details: > > http://www.python.org/dev/peps/pep-0361/ > > Enjoy, > - -Barry > > Barry Warsaw > barry at python.org > Python 2.6/3.0 Release Manager > (on behalf of the entire python-dev team) > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.8 (Darwin) > > iQCVAwUBR/WImHEjvBPtnXfVAQJmoQP+MzqNDI+Xt8zua/FE7Ca4TVXoIIy2uoOm > I1i3+vmevZ9vtAb9hcGwfEgPY4LSwb9Js4KnJJWMPaMuFJK4NgGoiMdj+t42zDbQ > bEzfBUOCoVkejLRxIQnWeJf1Hu8JocYyCHIRffv57/QdKpHuiSs8aE8GIT3STo3o > I88H5NY1GgI= > =WT2z > -----END PGP SIGNATURE----- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/gregor.lingl%40aon.at > > > From martin at v.loewis.de Fri Apr 4 22:23:37 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 04 Apr 2008 22:23:37 +0200 Subject: [Python-Dev] [Python-3000] RELEASED Python 2.6a2 and 3.0a4 In-Reply-To: <47F68A6B.9060901@aon.at> References: <3C3C0150-ED65-4381-9F54-BB437DD6DFB9@python.org> <47F68A6B.9060901@aon.at> Message-ID: <47F68E49.30306@v.loewis.de> > Error 404: File not found That has a simple explanation: the file is not there because it just doesn't exist yet, which in turn is because I have problems creating it (which is in turn due to switching to Visual Studio 2008). Regards, Martin From tnelson at onresolve.com Fri Apr 4 22:24:49 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Fri, 4 Apr 2008 13:24:49 -0700 Subject: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB28D@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB280@EXMBX04.exchhosting.com>, <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB28D@EXMBX04.exchhosting.com> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB291@EXMBX04.exchhosting.com> Interesting results! I committed the patch to test_socket.py in r62152. I was expecting all other platforms except for Windows to behave consistently (i.e. pass). That is, given the following: import socket host = '127.0.0.1' sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock.bind((host, 0)) port = sock.getsockname()[1] sock.close() del sock sock1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) sock1.bind((host, port)) sock2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM) sock2.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) sock2.bind((host, port)) ^^^^ ....the second bind should fail with EADDRINUSE, at least according to the 'SO_REUSEADDR and SO_REUSEPORT Socket Options' section in chapter 7.5 of Stevens' UNIX Network Programming Volume 1 (2nd Ed): "With TCP, we are never able to start multiple servers that bind the same IP address and same port: a completely duplicate binding. That is, we cannot start one server that binds 198.69.10.2 port 80 and start another that also binds 198.69.10.2 port 80, even if we set the SO_REUSEADDR socket option for the second server." The results: both Windows *and* Linux fail the patched test; none of the buildbots for either platform encountered an EADDRINUSE socket.error after the second bind(). FreeBSD, OS X, Solaris and Tru64 pass the test -- EADDRINUSE is raised on the second bind. (Interesting that all the ones that passed have a BSD lineage.) I've just reverted the test in r62156 as planned. The real issue now is that there are tests that are calling test_support.bind_socket() with the assumption that the port returned by this method is 'unbound', when in fact, the current implementation can't guarantee this: def bind_port(sock, host='', preferred_port=54321): for port in [preferred_port, 9907, 10243, 32999, 0]: try: sock.bind((host, port)) if port == 0: port = sock.getsockname()[1] return port except socket.error, (err, msg): if err != errno.EADDRINUSE: raise print >>sys.__stderr__, \ ' WARNING: failed to listen on port %d, trying another' % port This logic is only correct for platforms other than Windows and Linux. I haven't looked into all the networking test cases that rely on bind_port(), but I would think an implementation such as this would be much more reliable than what we've got for returning an unused port: def bind_port(sock, host='127.0.0.1', *args): s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.bind((host, 0)) port = s.getsockname()[1] s.close() del s sock.bind((host, port)) return port Actually, FWIW, I just ran a full regrtest.py against trunk on Win32 with this change in place and all the tests still pass. Thoughts? Trent. ________________________________________ From: python-dev-bounces+tnelson=onresolve.com at python.org [python-dev-bounces+tnelson=onresolve.com at python.org] On Behalf Of Trent Nelson [tnelson at onresolve.com] Sent: 04 April 2008 17:07 To: python-dev at python.org Subject: Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) I've raised issue 2550 to track this problem. I've also provided a patch on the tracker to test_socket.py that reproduces the issue. Anyone mind if I commit this to trunk? I'd like to observe if any other platforms exhibit different behaviour via buildbots. It'll cause all the Windows slaves to fail on test_socket though. (I can revert it once I've seen how the buildbots behave until I can come up with an actual patch for Windows that fixes the issue.) http://bugs.python.org/issue2550 http://bugs.python.org/file9939/test_socket.py.patch Trent. ________________________________________ From: python-dev-bounces+tnelson=onresolve.com at python.org [python-dev-bounces+tnelson=onresolve.com at python.org] On Behalf Of Trent Nelson [tnelson at onresolve.com] Sent: 03 April 2008 22:40 To: python-dev at python.org Subject: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) I started looking into this: http://www.python.org/dev/buildbot/all/x86%20W2k8%20trunk/builds/289/step-test/0 Pertinent part: test_asyncore test_asynchat command timed out: 1200 seconds without output SIGKILL failed to kill process using fake rc=-1 program finished with exit code -1 remoteFailed: [Failure instance: Traceback from remote host -- Traceback (most recent call last): Failure: buildbot.slave.commands.TimeoutError: SIGKILL failed to kill process ] I tried to replicate it on the buildbot in order to debug, which, surprisingly, I could do consistently by just running rt.bat -q -d -uall test_asynchat. As the log above indicates, the python process becomes completely and utterly wedged, to the point that I can't even attach a remote debugger and step into it. Digging further, I noticed that if I ran the following code in two different python consoles, EADDRINUSE was *NOT* being raised by socket.bind(): import socket s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) s.bind(('127.0.0.1', 54322)) However, take out the setsockopt line, and wallah, the second s.bind() will raise EADDRINUSE, as expected. This manifests into a really bizarre issue with test_asynchat in particualr, as subsequent sock.accept() calls on the socket put python into the uber wedged state (can't even ctrl-c out at the console, need to kill the process directly). Have to leave the office and head home so I don't have any more time to look at it tonight -- just wanted to post here for others to mull over. Trent. _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com From barry at python.org Fri Apr 4 22:39:09 2008 From: barry at python.org (Barry Warsaw) Date: Fri, 4 Apr 2008 16:39:09 -0400 Subject: [Python-Dev] Python source code on Bazaar vcs In-Reply-To: <18409.16093.799920.286191@montanaro-dyndns-org.local> References: <20C0AC37-D748-450E-B690-FBCA2ACFFC4E@python.org> <18408.27695.339064.649345@montanaro-dyndns-org.local> <08E8188C-AEA2-4E78-B74C-AF213757DEA0@python.org> <18409.16093.799920.286191@montanaro-dyndns-org.local> Message-ID: <99AE707E-C992-42B9-A238-F2751A691DFA@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Mar 25, 2008, at 2:05 PM, skip at pobox.com wrote: > >>> Did I misread the directions or do I really need the --create-prefix >>> arg? > > Barry> You do, the first time you push a user branch because > users/skip > Barry> doesn't exist yet. It's mentioned in the docs, but it's > pretty > Barry> easy to overlook ;). > > Well, I noticed the mention in .../dev/bazaar, where it reads, "the > first > time you do this, you might need to add --create-prefix". Perhaps > that > should read "... you will need to ...". > > It pushed fine with --create-prefix. Thanks Skip. Fixed. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBR/aR7XEjvBPtnXfVAQIoXgQAsN6Hfs1JjdFMOI1/Ef2kLeeBfTPxf+Ys K9y81yEHUNonaQEIF9ptnyOIEyic5uX+Ig4cYO20i1LgvGEIIiCg191EJtYFc9jr s1dTgmE3PQfiR7J2m2SWS06bYMsanBdAAW/ZnMpgmUMZixYEX43z7Q+kjFibwTn+ UbGz2uLeW+o= =Fx0S -----END PGP SIGNATURE----- From exarkun at divmod.com Sat Apr 5 00:01:53 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Fri, 4 Apr 2008 17:01:53 -0500 Subject: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB291@EXMBX04.exchhosting.com> Message-ID: <20080404220153.6859.1817508967.divmod.quotient.25765@ohm> On Fri, 4 Apr 2008 13:24:49 -0700, Trent Nelson wrote: >Interesting results! I committed the patch to test_socket.py in r62152. I was expecting all other platforms except for Windows to behave consistently (i.e. pass). That is, given the following: > > import socket > host = '127.0.0.1' > sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > sock.bind((host, 0)) > port = sock.getsockname()[1] > sock.close() > del sock > > sock1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > sock1.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) > sock1.bind((host, port)) > sock2 = socket.socket(socket.AF_INET, socket.SOCK_STREAM) > sock2.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) > sock2.bind((host, port)) > ^^^^ > >....the second bind should fail with EADDRINUSE, at least according to the 'SO_REUSEADDR and SO_REUSEPORT Socket Options' section in chapter 7.5 of Stevens' UNIX Network Programming Volume 1 (2nd Ed): > >"With TCP, we are never able to start multiple servers that bind > the same IP address and same port: a completely duplicate binding. > That is, we cannot start one server that binds 198.69.10.2 port 80 > and start another that also binds 198.69.10.2 port 80, even if we > set the SO_REUSEADDR socket option for the second server." > >The results: both Windows *and* Linux fail the patched test; none of the buildbots for either platform encountered an EADDRINUSE socket.error after the second bind(). FreeBSD, OS X, Solaris and Tru64 pass the test -- EADDRINUSE is raised on the second bind. (Interesting that all the ones that passed have a BSD lineage.) Notice that the quoted text explains that you cannot start multiple servers that etc. Since you didn't call listen on either socket, it's arguable that you didn't start any servers, so there should be no surprise regarding the behavior. Try adding listen calls at various places in the example and you'll see something different happen. FWIW, AIUI, SO_REUSEADDR behaves just as described in the above quote on Linux/BSD/UNIX/etc. On Windows, however, that option actually means something quite different. It means that the address should be stolen from any process which happens to be using it at the moment. There is another option, SO_EXCLUSIVEADDRUSE, only on Windows I think, which, AIUI, makes it impossible for another process to steal the port using SO_REUSEADDR. Hope this helps, Jean-Paul From martin at martinthomas.net Sat Apr 5 00:58:53 2008 From: martin at martinthomas.net (Martin Thomas) Date: Fri, 4 Apr 2008 17:58:53 -0500 Subject: [Python-Dev] Dates wrong on front page of pydotorg Message-ID: Did anyone else notice that the dates are incorrect in the news items on the front page? As an example: Published: Mon, 4 Apr 2008 which should be Published: Fri, 4 Apr 2008 I'll poke around and see if I can figure it out. Cheers //M From ggpolo at gmail.com Sat Apr 5 01:37:14 2008 From: ggpolo at gmail.com (Guilherme Polo) Date: Fri, 4 Apr 2008 20:37:14 -0300 Subject: [Python-Dev] Dates wrong on front page of pydotorg In-Reply-To: References: Message-ID: 2008/4/4, Martin Thomas : > Did anyone else notice that the dates are incorrect in the news items > on the front page? As an example: > Published: Mon, 4 Apr 2008 > which should be > Published: Fri, 4 Apr 2008 > > I'll poke around and see if I can figure it out. > There is a file (newsindex.yml) which contains the news items. The publication date is hand written so it is just a typo from whoever commited that news item with that publication date. I just fixed it (as you pointed out). Thanks ;) Anyway, did you send this to the wrong list or was it intentional ? Regards, > Cheers //M > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ggpolo%40gmail.com > -- -- Guilherme H. Polo Goncalves From exarkun at divmod.com Sat Apr 5 01:37:23 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Fri, 4 Apr 2008 18:37:23 -0500 Subject: [Python-Dev] __eq__ vs hash In-Reply-To: Message-ID: <20080404233723.6859.1733915494.divmod.quotient.25786@ohm> On Fri, 4 Apr 2008 07:38:04 -0700, Guido van Rossum wrote: >On Fri, Apr 4, 2008 at 2:46 AM, Ralf Schmitt wrote: >> the news file for python 2.6 does not mention that you need to define >> __hash__ in case you define __eq__ for a class. >> This breaks some code (for me: mercurial and pyparsing). >> Shouldn't this be documented somewhere (I also cannot find it in the >> whatsnew file). > >Well, technically this has always been the requirement. > >What specific code breaks? Maybe we need to turn this into a warning >in order to be more backwards compatible? > There was some code in Twisted (one class, specifically) which was broken/ revealed to be broken by this Python 2.6 change. The code assumed identity hashing if no __hash__ method was implemented. This ended up only working if you only had a singleton instance of the class, but the class also went out of its way to make sure that was the case. We have since changed the code to work on Python 2.6. If you're curious about the details, here's the code after the fix: http://twistedmatrix.com/trac/browser/trunk/twisted/web2/dav/element/base.py?rev=22305#L345 Here's the changeset that fixed it: http://twistedmatrix.com/trac/changeset/22305 And here's the same class before the fix: http://twistedmatrix.com/trac/browser/trunk/twisted/web2/dav/element/base.py?rev=22304#L344 Jean-Paul From martin at martinthomas.net Sat Apr 5 01:47:11 2008 From: martin at martinthomas.net (Martin Thomas) Date: Fri, 4 Apr 2008 18:47:11 -0500 Subject: [Python-Dev] Dates wrong on front page of pydotorg In-Reply-To: References: Message-ID: On Apr 4, 2008, at 6:37 PM, Guilherme Polo wrote: > 2008/4/4, Martin Thomas : >> Did anyone else notice that the dates are incorrect in the news items >> on the front page? As an example: >> Published: Mon, 4 Apr 2008 >> which should be >> Published: Fri, 4 Apr 2008 >> >> I'll poke around and see if I can figure it out. >> > > There is a file (newsindex.yml) which contains the news items. The > publication date is hand written so it is just a typo from whoever > commited that news item with that publication date. > > I just fixed it (as you pointed out). Thanks ;) Welcome :-) > > > Anyway, did you send this to the wrong list or was it intentional ? I sent it to the wrong list.. should have pydotorg. Sorry. //M From tjreedy at udel.edu Sat Apr 5 03:11:09 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 4 Apr 2008 21:11:09 -0400 Subject: [Python-Dev] RELEASED Python 2.6a2 and 3.0a4 References: <3C3C0150-ED65-4381-9F54-BB437DD6DFB9@python.org><47F63AA0.5040106@ghaering.de> Message-ID: "Barry Warsaw" wrote in message news:E7B09BA2-A4A6-4D65-AE87-DFE2EB37EBCB at python.org... I think I fixed these -- at least a grep of my local tree turns up no other references to 2.6a1 or 3.0a3. It might take a little while for the site to get rebuilt though. ============= Fixed now. Now I am only waiting for the Windows binaries to be built and added From skip at pobox.com Sat Apr 5 02:37:47 2008 From: skip at pobox.com (skip at pobox.com) Date: Fri, 4 Apr 2008 19:37:47 -0500 Subject: [Python-Dev] inittimezone - shouldn't it be static? Message-ID: <18422.51675.824719.583407@montanaro-dyndns-org.local> Looks to me like the inittimezone function in timemodule.c should be static. I don't see it called from any code other than within that module. If it's not supposed to be static it needs a more pythonic name. (Can't dig into this just now as I'm flying from Denver to Chicago a.t.m...) Skip From nnorwitz at gmail.com Sat Apr 5 05:14:04 2008 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 4 Apr 2008 20:14:04 -0700 Subject: [Python-Dev] osx x86 buildbot offer In-Reply-To: References: Message-ID: On Fri, Apr 4, 2008 at 11:52 AM, Thomas Heller wrote: > I can offer an OS X x86 machine to run a buildbot on. This is a physical machine, > running OS X 10.5 Leopard. Thanks Thomas! Martin and I will coordinate with you off-list. n From antonyjoseph89 at gmail.com Sat Apr 5 07:49:21 2008 From: antonyjoseph89 at gmail.com (Antony Joseph) Date: Fri, 4 Apr 2008 22:49:21 -0700 Subject: [Python-Dev] Antony Joseph wants to chat Message-ID: ----------------------------------------------------------------------- Antony Joseph wants to stay in better touch using some of Google's coolest new products. If you already have Gmail or Google Talk, visit: http://mail.google.com/mail/b-29833b0a8-b7e4c018a7-1e5ec176e7509c87 You'll need to click this link to be able to chat with Antony Joseph. To get Gmail - a free email account from Google with over 2,800 megabytes of storage - and chat with Antony Joseph, visit: http://mail.google.com/mail/a-29833b0a8-b7e4c018a7-e7105acb3e Gmail offers: - Instant messaging right inside Gmail - Powerful spam protection - Built-in search for finding your messages and a helpful way of organizing emails into "conversations" - No pop-up ads or untargeted banners - just text ads and related information that are relevant to the content of your messages All this, and its yours for free. But wait, there's more! By opening a Gmail account, you also get access to Google Talk, Google's instant messaging service: http://www.google.com/talk/ Google Talk offers: - Web-based chat that you can use anywhere, without a download - A contact list that's synchronized with your Gmail account - Free, high quality PC-to-PC voice calls when you download the Google Talk client Gmail and Google Talk are still in beta. We're working hard to add new features and make improvements, so we might also ask for your comments and suggestions periodically. We appreciate your help in making our products even better! Thanks, The Google Team To learn more about Gmail and Google Talk, visit: http://mail.google.com/mail/help/about.html http://www.google.com/talk/about.html (If clicking the URLs in this message does not work, copy and paste them into the address bar of your browser). From guido at python.org Sat Apr 5 16:46:58 2008 From: guido at python.org (Guido van Rossum) Date: Sat, 5 Apr 2008 07:46:58 -0700 Subject: [Python-Dev] inittimezone - shouldn't it be static? In-Reply-To: <18422.51675.824719.583407@montanaro-dyndns-org.local> References: <18422.51675.824719.583407@montanaro-dyndns-org.local> Message-ID: Sounds like an oversight. Please fix! On Fri, Apr 4, 2008 at 5:37 PM, wrote: > > Looks to me like the inittimezone function in timemodule.c should be static. > I don't see it called from any code other than within that module. If it's > not supposed to be static it needs a more pythonic name. (Can't dig into > this just now as I'm flying from Denver to Chicago a.t.m...) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From eikeon at eikeon.com Sat Apr 5 18:48:21 2008 From: eikeon at eikeon.com (Daniel Krech) Date: Sat, 5 Apr 2008 12:48:21 -0400 Subject: [Python-Dev] __eq__ vs hash In-Reply-To: References: <932f8baf0804040246s261c1cd2ic977934cbd049731@mail.gmail.com> Message-ID: On Apr 4, 2008, at 10:38 AM, Guido van Rossum wrote: > On Fri, Apr 4, 2008 at 2:46 AM, Ralf Schmitt wrote: >> the news file for python 2.6 does not mention that you need to define >> __hash__ in case you define __eq__ for a class. >> This breaks some code (for me: mercurial and pyparsing). >> Shouldn't this be documented somewhere (I also cannot find it in the >> whatsnew file). > > Well, technically this has always been the requirement. > > What specific code breaks? Maybe we need to turn this into a warning > in order to be more backwards compatible? I think a warning would be nice. I've run into a number of places that are breaking due to the change. Including parts of the standard lib, in particular, unittest.TestSuite as used by nose [1]. There is already an open issue with patches for this in the issue tracker [2]. I've been running trunk with the patch installed and am able to get past the breakage due to the change. I don't believe the current patch is kicking out a warning however. [1] http://code.google.com/p/python-nose/issues/detail?id=161 [2] http://bugs.python.org/issue2235 From tnelson at onresolve.com Sat Apr 5 19:22:44 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Sat, 5 Apr 2008 10:22:44 -0700 Subject: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) In-Reply-To: <20080404220153.6859.1817508967.divmod.quotient.25765@ohm> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB291@EXMBX04.exchhosting.com> <20080404220153.6859.1817508967.divmod.quotient.25765@ohm> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E226D8C7C@EXMBX04.exchhosting.com> > >"With TCP, we are never able to start multiple servers that bind > > the same IP address and same port: a completely duplicate binding. > > That is, we cannot start one server that binds 198.69.10.2 port 80 > > and start another that also binds 198.69.10.2 port 80, even if we > > set the SO_REUSEADDR socket option for the second server." > Notice that the quoted text explains that you cannot start multiple > servers that etc. Since you didn't call listen on either socket, it's > arguable that you didn't start any servers, so there should be no > surprise regarding the behavior. Try adding listen calls at various > places in the example and you'll see something different happen. I agree in principle, Stevens says nothing about what happens if you *do* try and bind two sockets on two identical host/port addresses. Even so, test_support.bind_port() makes an assumption that bind() will raise EADDRINUSE if the port is not available, which, as has been demonstrated, won't be the case on Windows or Linux. > FWIW, AIUI, SO_REUSEADDR behaves just as described in the above quote > on Linux/BSD/UNIX/etc. On Windows, however, that option actually means > something quite different. It means that the address should be stolen > from any process which happens to be using it at the moment. Probably explains why the python process wedges when this happens on Windows... > There is another option, SO_EXCLUSIVEADDRUSE, only on Windows I think, > which, AIUI, makes it impossible for another process to steal the port > using SO_REUSEADDR. Nod, if SO_EXCLUSIVEADDRUSE is used instead in the code I posted, Windows raises EADDRINUSE on the second bind(). I don't have access to any Linux boxes at the moment, so I can't test what sort of error is raised with the example I posted if listen() and accept() are called on the two sockets bound to identical addresses. Can anyone else shed some light on this? I'd be interested in knowing if the process wedges on Linux as badly as it does on Windows (to the point where it's not respecting ctrl-c or sigkill). Trent. From skip at pobox.com Sat Apr 5 21:17:19 2008 From: skip at pobox.com (skip at pobox.com) Date: Sat, 5 Apr 2008 14:17:19 -0500 Subject: [Python-Dev] configure error: "rm: conftest.dSYM: is a directory" Message-ID: <18423.53311.150471.667891@montanaro-dyndns-org.local> I just noticed this error message during configure: checking whether gcc accepts -Olimit 1500... no checking whether gcc supports ParseTuple __format__... no checking whether pthreads are available without options... yes checking whether g++ also accepts flags for thread support... no checking for ANSI C header files... rm: conftest.dSYM: is a directory rm: conftest.dSYM: is a directory yes checking for sys/types.h... yes checking for sys/stat.h... yes checking for stdlib.h... yes checking for string.h... yes Note the "rm: conftest.dSYM: is a directory". This occurred a few times during the configure process. Didn't cause it to conk out, but is annoying. Skip From skip at pobox.com Sat Apr 5 21:49:08 2008 From: skip at pobox.com (skip at pobox.com) Date: Sat, 5 Apr 2008 14:49:08 -0500 Subject: [Python-Dev] inittimezone - shouldn't it be static? In-Reply-To: References: <18422.51675.824719.583407@montanaro-dyndns-org.local> Message-ID: <18423.55220.469345.843015@montanaro-dyndns-org.local> >> Looks to me like the inittimezone function in timemodule.c should be >> static. Guido> Sounds like an oversight. Please fix! Done. inittimezone is now declared static. Skip From mhammond at skippinet.com.au Sun Apr 6 04:34:54 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Sun, 6 Apr 2008 12:34:54 +1000 Subject: [Python-Dev] Remove "current" Windows executables from Lib/distutils/command in svn? Message-ID: <018701c8978e$ce620660$6b261320$@com.au> As I was preparing to check-in my cross-compilation patch, which includes a new x64 executable in the Lib/distutils/command directory, it occurs to me that we don't actually need it in subversion - and nor do we need wininst-9.0.exe. I believe these executables are in svn for historical reasons. The build process for these executables are now integrated into the core Python build process (although they are disabled by default). There is a good use-case for keeping executables built with older MSVC versions, but I don't see one for executables built with the current version. I'd like to propose we delete Lib/Distutils/command/wininst-9.0.exe, and enable the building of that project by default in the standard build process (and I'll setup the x64 build of the executable similarly). They will then need to be re-added when a future version moves away from Visual Studio 2008, but that seems like a reasonable way to approach things. Are there any objections to this, or should we stick with the status-quo and I'll add the new x64 executable? Or do things that way *just* for x64? Thanks, Mark. From martin at v.loewis.de Sun Apr 6 08:15:32 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 06 Apr 2008 08:15:32 +0200 Subject: [Python-Dev] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <018701c8978e$ce620660$6b261320$@com.au> References: <018701c8978e$ce620660$6b261320$@com.au> Message-ID: <47F86A84.6000204@v.loewis.de> > I'd like to propose we delete Lib/Distutils/command/wininst-9.0.exe, and > enable the building of that project by default in the standard build process > (and I'll setup the x64 build of the executable similarly). There are two issues here: a) how does the binary get into the release tarball? You might argue that it doesn't have to, as it is sufficient when it is included in the Windows installer, however, as currently implemented, bdist_wininst also runs on Unix, and people use it that way. b) IIRC, upx was used to compress these executables. I don't think the the current build process covers this, yet (but then, the current binaries might not be compressed with upx anymore, either - not sure whether that would be a bug). Of course, upx would probably not currently apply to Win64-amd64. As another alternative to providing a separate AMD64 binary, you could also try to make the existing binary work properly on Win64, and tell it with a config flag whether to look for Win32 or Win64 (or either) on the target system. Regards, Martin From greg at krypto.org Sun Apr 6 08:29:29 2008 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 5 Apr 2008 23:29:29 -0700 Subject: [Python-Dev] Thread-safe file objects, the return In-Reply-To: References: Message-ID: <52dc1c820804052329i10fe9c75ka6572e37d603ad70@mail.gmail.com> I've reviewed the patch on http://bugs.python.org/issue815646 and have uploaded my modified version (mostly test improvements and some formatting to keep C code under 80 columns with proper 8 space tabs). I would have committed it already but I have a sneaking suspicion that its unit test will barf on windows since it could depend on some posix-like file system semantics. Could someone with a windows build environment please test it as asked in the issue and report back in the tracker? thanks! -gps On Wed, Apr 2, 2008 at 11:47 AM, Guido van Rossum wrote: > On Wed, Apr 2, 2008 at 3:17 AM, Antoine Pitrou > wrote: > > Guido van Rossum python.org> writes: > > > Your solution (a counter) seems fine except I think perhaps the > > > close() call should not raise IOError -- instead, it should set a > flag > > > so that the thread that makes the counter go to zero can close the > > > thread (after all the file got closed while it was being used). > > > > I agree with Gregory: we should be explicit about what happens. I > wonder > > what we would gain from that approach - apart from encouraging > dangerous > > coding practices :) > > It also depends how far we want to go: I am merely proposing to fix the > > crashes, do we want to provide a "smarter" close() variation that does > what > > you suggest for people that want (or need) to take the risk? > > I also agree with Gregory now -- at least on the issue of fclose(). > > I think that for other (non-closing) operations we should be fine, > given the Posix requirement that streams have an internal lock. While > multiple threads reading from a file sounds insane, multiple files > writing to a file is pretty common (think of stdout or stderr) and > should be supported. > > > > There are of course other concurrency issues besides close -- what if > > > two threads both try to do I/O on the file? What will the C stdio > > > library do in that case? Are stdio files thread-safe at the C level? > > > > According to the glibc documentation, at > > > http://www.gnu.org/software/libc/manual/html_node/Streams-and-Threads.html: > > > > ? The POSIX standard requires that by default the stream operations are > > atomic. I.e., issuing two stream operations for the same stream in two > > threads at the same time will cause the operations to be executed as if > > they were issued sequentially. The buffer operations performed while > > reading or writing are protected from other uses of the same stream. To > do > > this each stream has an internal lock object which has to be > (implicitly) > > acquired before any work can be done. ? > > > > So according to POSIX rules it should be perfectly safe. > > Cool. > > > In any case, someone would have to try my patch under Windows and OS X > and > > see if test_file.py passes without crashing. > > I know Windows has internal locks on stdio. I trust that OSX, being a > BSD descendant, is posix compliant. So I'm not worried about these. > > +1 on your patch, assuming some other developer reviews it and submits it. > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/ > ) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080405/0427aa16/attachment-0001.htm From brett at python.org Sun Apr 6 10:13:57 2008 From: brett at python.org (Brett Cannon) Date: Sun, 6 Apr 2008 10:13:57 +0200 Subject: [Python-Dev] configure error: "rm: conftest.dSYM: is a directory" In-Reply-To: <18423.53311.150471.667891@montanaro-dyndns-org.local> References: <18423.53311.150471.667891@montanaro-dyndns-org.local> Message-ID: On Sat, Apr 5, 2008 at 9:17 PM, wrote: > I just noticed this error message during configure: > > checking whether gcc accepts -Olimit 1500... no > checking whether gcc supports ParseTuple __format__... no > checking whether pthreads are available without options... yes > checking whether g++ also accepts flags for thread support... no > checking for ANSI C header files... rm: conftest.dSYM: is a directory > rm: conftest.dSYM: is a directory > yes > checking for sys/types.h... yes > checking for sys/stat.h... yes > checking for stdlib.h... yes > checking for string.h... yes > > Note the "rm: conftest.dSYM: is a directory". This occurred a few times > during the configure process. Didn't cause it to conk out, but is annoying. I am assuming this is on your OS X machine, Skip? I have been getting that message for a while on my OS X machine as well. Same with some linking message when building ctypes. Never bothered to debug them since they never seemed to affect anything. -Brett From mhammond at skippinet.com.au Sun Apr 6 15:19:21 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Sun, 6 Apr 2008 23:19:21 +1000 Subject: [Python-Dev] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <47F86A84.6000204@v.loewis.de> References: <018701c8978e$ce620660$6b261320$@com.au> <47F86A84.6000204@v.loewis.de> Message-ID: <01c701c897e8$d8ce3a60$8a6aaf20$@com.au> > As another alternative to providing a separate AMD64 binary, you could > also try to make the existing binary work properly on Win64, and tell > it with a config flag whether to look for Win32 or Win64 (or either) > on the target system. I'm afraid that isn't an option for me in the short term. Should I just check-in my cross-compilation patch as it stands then? Mark From skip at pobox.com Sun Apr 6 15:20:47 2008 From: skip at pobox.com (skip at pobox.com) Date: Sun, 6 Apr 2008 08:20:47 -0500 Subject: [Python-Dev] configure error: "rm: conftest.dSYM: is a directory" In-Reply-To: References: <18423.53311.150471.667891@montanaro-dyndns-org.local> Message-ID: <18424.52783.238721.876293@montanaro-dyndns-org.local> >> Note the "rm: conftest.dSYM: is a directory". This occurred a few >> times during the configure process. Didn't cause it to conk out, but >> is annoying. Brett> I am assuming this is on your OS X machine, Skip? Yes, sorry. I forgot to mention that. As long as someone else sees it and feels fine to ignore it, I'm fine with it. I suspect it's a configure problem anyway, not a Python problem. Skip From martin at v.loewis.de Sun Apr 6 15:54:48 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 06 Apr 2008 15:54:48 +0200 Subject: [Python-Dev] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <01c701c897e8$d8ce3a60$8a6aaf20$@com.au> References: <018701c8978e$ce620660$6b261320$@com.au> <47F86A84.6000204@v.loewis.de> <01c701c897e8$d8ce3a60$8a6aaf20$@com.au> Message-ID: <47F8D628.1030404@v.loewis.de> >> As another alternative to providing a separate AMD64 binary, you could >> also try to make the existing binary work properly on Win64, and tell >> it with a config flag whether to look for Win32 or Win64 (or either) >> on the target system. > > I'm afraid that isn't an option for me in the short term. Should I just > check-in my cross-compilation patch as it stands then? Sure, go ahead. Regards, Martin From tnelson at onresolve.com Sun Apr 6 22:51:28 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Sun, 6 Apr 2008 13:51:28 -0700 Subject: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbo... In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB290@EXMBX04.exchhosting.com> References: <20080403182707.506A41E400D@bag.python.org>, <47F5E5EF.5010005@cheimes.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB286@EXMBX04.exchhosting.com>, <47F67A04.4020805@v.loewis.de>, <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB290@EXMBX04.exchhosting.com> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22939EE1@EXMBX04.exchhosting.com> Fixed in r62193. ________________________________________ From: python-dev-bounces+tnelson=onresolve.com at python.org [python-dev-bounces+tnelson=onresolve.com at python.org] On Behalf Of Trent Nelson [tnelson at onresolve.com] Sent: 04 April 2008 20:41 To: "Martin v. L?wis" Cc: Christian Heimes; python-dev at python.org Subject: Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbo... Ok, I'll change the approach this weekend. Trent. ________________________________________ From: "Martin v. L?wis" [martin at v.loewis.de] Sent: 04 April 2008 19:57 To: Trent Nelson Cc: Christian Heimes; python-dev at python.org Subject: Re: [Python-Dev] r62129 - in python/trunk: PCbuild/debug.vsprops PCbuild/kill_python.c PCbuild/kill_python.vcproj PCbuild/pcbuild.sln PCbuild/pythoncore.vcproj PCbuild/release.vsprops Tools/buildbot/Makefile Tools/buildbot/build-amd64.bat Tools/buildbo... > What do others that do Windows development think? I don't have a > problem changing the build behaviour if the approach I've taken is > generally disliked. I think kill_python should only ever be invoked in the build slaves; it should *not* be part of the regular build. If developers find they can't build because some files are still open, they should kill the processes themselves. Regards, Martin _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com From tnelson at onresolve.com Sun Apr 6 23:34:20 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Sun, 6 Apr 2008 14:34:20 -0700 Subject: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E226D8C7C@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB291@EXMBX04.exchhosting.com> <20080404220153.6859.1817508967.divmod.quotient.25765@ohm>, <87D3F9C72FBF214DB39FA4E3FE618CDC6E226D8C7C@EXMBX04.exchhosting.com> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22939EE2@EXMBX04.exchhosting.com> I've attached a patch (http://bugs.python.org/file9966/trunk.2550.patch) to issue 2550 that addresses the original problem here: test_support.bind_port() potentially returning ports that have already been bound to. The patch updates the tests that relied on this method, such that they call it with the new calling convention (test_ftplib, test_httplib, test_socket, test_ssl_socket, test_asynchat, test_telnetlib). Any objections to the patch? Would like to commit it sooner rather than later, as it'll fix my buildbots from wedging on test_asynchat at the very least. Trent. ________________________________________ From: python-dev-bounces+tnelson=onresolve.com at python.org [python-dev-bounces+tnelson=onresolve.com at python.org] On Behalf Of Trent Nelson [tnelson at onresolve.com] Sent: 05 April 2008 18:22 To: Jean-Paul Calderone; python-dev at python.org Subject: Re: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) > >"With TCP, we are never able to start multiple servers that bind > > the same IP address and same port: a completely duplicate binding. > > That is, we cannot start one server that binds 198.69.10.2 port 80 > > and start another that also binds 198.69.10.2 port 80, even if we > > set the SO_REUSEADDR socket option for the second server." > Notice that the quoted text explains that you cannot start multiple > servers that etc. Since you didn't call listen on either socket, it's > arguable that you didn't start any servers, so there should be no > surprise regarding the behavior. Try adding listen calls at various > places in the example and you'll see something different happen. I agree in principle, Stevens says nothing about what happens if you *do* try and bind two sockets on two identical host/port addresses. Even so, test_support.bind_port() makes an assumption that bind() will raise EADDRINUSE if the port is not available, which, as has been demonstrated, won't be the case on Windows or Linux. > FWIW, AIUI, SO_REUSEADDR behaves just as described in the above quote > on Linux/BSD/UNIX/etc. On Windows, however, that option actually means > something quite different. It means that the address should be stolen > from any process which happens to be using it at the moment. Probably explains why the python process wedges when this happens on Windows... > There is another option, SO_EXCLUSIVEADDRUSE, only on Windows I think, > which, AIUI, makes it impossible for another process to steal the port > using SO_REUSEADDR. Nod, if SO_EXCLUSIVEADDRUSE is used instead in the code I posted, Windows raises EADDRINUSE on the second bind(). I don't have access to any Linux boxes at the moment, so I can't test what sort of error is raised with the example I posted if listen() and accept() are called on the two sockets bound to identical addresses. Can anyone else shed some light on this? I'd be interested in knowing if the process wedges on Linux as badly as it does on Windows (to the point where it's not respecting ctrl-c or sigkill). Trent. _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/tnelson%40onresolve.com From garg1 at ualberta.ca Mon Apr 7 02:48:50 2008 From: garg1 at ualberta.ca (Rahul Garg) Date: Sun, 06 Apr 2008 18:48:50 -0600 Subject: [Python-Dev] New project : Spyke python-to-C compiler Message-ID: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> Note this message has been posted to numpy-discussion and python-dev. Sorry for the multiple posting but I thought both python devs and numpy users will be interested. If you believe your list should not receive this email, let me know. Also I just wanted to introduce myself since I may ask doubts about Python and Numpy internals from time to time :) Hi. I am a student at Univ of Alberta doing my masters in computing science. I am writing a Python-to-C compiler as one part of my thesis. The compiler, named Spyke, will be made available in a couple of weeks and is geared towards scientific applications and will therefore focus mostly on needs of scientific app developers. What is Spyke? In many performance critical projects, it is often necessary to rewrite parts of the application in C. However writing C wrappers can be time consuming. Spyke offers an alternative approach. You add annotations to your Python code as strings. These strings are discarded by the Python interpreter but these are interpreted as types by Spyke compiler to convert to C. Example : "int -> int" def f(x): return 2*x In this case the Spyke compiler will consider the string "int -> int" as a decalration that the function accepts int as parameter and returns int. Spyke will then generate a C function and a wrapper function. This idea is directly copied from PLW (Python Language Wrapper) project. Once Python3k arrives, much of these declarations will be moved to function annotations and class decorators. This way you can do all your development and debugging interactively using the standard Python interpreter. When you need to compile to C, you just add type annotations to places that you want to convert and invoke spyke on the annotated module. This is different from Pyrex because Pyrex does not accept Python code. With Spyke, your code is 100% pure python. Spyke has basic support for functions and classes. Spyke can do very basic type inference for local variables in function bodies. Spyke also has partial support for homogenous lists and dictionaries and fixed length tuples. One big advantage of Spyke is that it understands at least part of numpy. Numpy arrays are treated as fundamental types and Spyke knows what C code to generate for slicing/indexing of numpy arrays etc. This should help a lot in scientific applications. Note that Spyke can handle only a subset of Python. Exceptions, iterators, generators, runtime code generation of any kind etc is not handled. Nested functions will be added soon. I will definitely add some of these missing features based on what is actually required for real world Python codes. Currently if Spyke does not understand a function, it just leaves it as Python code. Classes can be handled but special methods are not currently supported. The support of classes is a little brittle because I am trying to resolve some issues b/w old and new style of classes. Where is Spyke? Spyke will be available as a binary only release in a couple of weeks. I intend to make it open source after a few months. Spyke is written in Python and Java and should be platform independant. I do intend to make the source open in a few months. Right now its undergoing very rapid development and has negligible amounts of documentation so the source code right now is pretty useless to anyone else anyway. I need help: However I need a bit of help. I am having a couple of problems : a) I am finding it hard to get pure Python+NumPy testing codes. I need more codes to test the compiler. Developing a compiler without a test-suite is kind of useless. If you have some pure Python codes which need better performance, please contact me. I guarantee that your codes will not be released to public without your permission but might be referenced in academic publications. I can also make the compiler available to you hopefully after 10th of April. Its kind of unstable currently. I will also need your help in annotating the provided testing codes since I probably wont know what your application is doing. b) Libraries which interface with C/C++ : Many codes in SciPy for instance have mixed language codes. Part of the code is written in C/C++. Spyke only knows how to annotated Python codes. For C/C++ libraries wrapped into Python modules, Spyke will therefore need to know at least 2 things : i) The mapping of a C function name/struct etc to Python ii) The type information of the said C function. There are many many ways that people interact with C code. People either write wrappers manually, or use autogenerated wrappers using SWIG or SIP Boost.Python etc., use Pyrex or Cython while some people use ctypes. I dont have the time or resources to support these multitude of methods. I considered trying to parse the C code implementing wrappers but its "non-trivial" to put it mildly. Parsing only SWIG generated code is another possibility but its still hard. Another approach that I am seriously considering is to support a subset of ctypes (with additional restriction) instead. But my question is : Is ctypes good enough for most of you? Ctypes cannot interface with C++ code but its pure Python. However I have not seen too many instances of people using ctypes. c) Strings as type declarations : Do you think I should use decorators instead at least for function type declarations? thanks for patiently reading this, comments and inquiries sought. rahul From theller at ctypes.org Mon Apr 7 08:45:43 2008 From: theller at ctypes.org (Thomas Heller) Date: Mon, 07 Apr 2008 08:45:43 +0200 Subject: [Python-Dev] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <47F86A84.6000204@v.loewis.de> References: <018701c8978e$ce620660$6b261320$@com.au> <47F86A84.6000204@v.loewis.de> Message-ID: <47F9C317.9060701@ctypes.org> Martin v. L?wis schrieb: >> I'd like to propose we delete Lib/Distutils/command/wininst-9.0.exe, and >> enable the building of that project by default in the standard build process >> (and I'll setup the x64 build of the executable similarly). > > There are two issues here: > a) how does the binary get into the release tarball? You might argue > that it doesn't have to, as it is sufficient when it is included in > the Windows installer, however, as currently implemented, > bdist_wininst also runs on Unix, and people use it that way. > b) IIRC, upx was used to compress these executables. I don't think the > the current build process covers this, yet (but then, the current > binaries might not be compressed with upx anymore, either - not > sure whether that would be a bug). Of course, upx would probably > not currently apply to Win64-amd64. I did not compress them with upx anymore because of licensing issues with the upx-created executables, but I do not remember the details anymore. Thomas From lists at cheimes.de Mon Apr 7 11:57:27 2008 From: lists at cheimes.de (Christian Heimes) Date: Mon, 07 Apr 2008 11:57:27 +0200 Subject: [Python-Dev] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <47F86A84.6000204@v.loewis.de> References: <018701c8978e$ce620660$6b261320$@com.au> <47F86A84.6000204@v.loewis.de> Message-ID: <47F9F007.2060903@cheimes.de> Martin v. L?wis schrieb: > b) IIRC, upx was used to compress these executables. I don't think the > the current build process covers this, yet (but then, the current > binaries might not be compressed with upx anymore, either - not > sure whether that would be a bug). Of course, upx would probably > not currently apply to Win64-amd64. The 9.0 binary hasn't been compressed with upx (yet). Christian From lists at cheimes.de Mon Apr 7 11:57:27 2008 From: lists at cheimes.de (Christian Heimes) Date: Mon, 07 Apr 2008 11:57:27 +0200 Subject: [Python-Dev] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <47F86A84.6000204@v.loewis.de> References: <018701c8978e$ce620660$6b261320$@com.au> <47F86A84.6000204@v.loewis.de> Message-ID: <47F9F007.2060903@cheimes.de> Martin v. L?wis schrieb: > b) IIRC, upx was used to compress these executables. I don't think the > the current build process covers this, yet (but then, the current > binaries might not be compressed with upx anymore, either - not > sure whether that would be a bug). Of course, upx would probably > not currently apply to Win64-amd64. The 9.0 binary hasn't been compressed with upx (yet). Christian From draghuram at gmail.com Mon Apr 7 15:52:42 2008 From: draghuram at gmail.com (Raghuram Devarakonda) Date: Mon, 7 Apr 2008 09:52:42 -0400 Subject: [Python-Dev] socket.SOL_REUSEADDR: different semantics between Windows vs Unix (or why test_asynchat is sometimes dying on Windows) In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E226D8C7C@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB291@EXMBX04.exchhosting.com> <20080404220153.6859.1817508967.divmod.quotient.25765@ohm> <87D3F9C72FBF214DB39FA4E3FE618CDC6E226D8C7C@EXMBX04.exchhosting.com> Message-ID: <2c51ecee0804070652y2bd30d7apcebb4a96fd7685cb@mail.gmail.com> On Sat, Apr 5, 2008 at 1:22 PM, Trent Nelson wrote: > Nod, if SO_EXCLUSIVEADDRUSE is used instead in the code I posted, Windows raises EADDRINUSE on the second bind(). I don't have access to any Linux boxes at the moment, so I can't test what sort of error is raised with the example I posted if listen() and accept() are called on the two sockets bound to identical addresses. Can anyone else shed some light on this? I'd be interested in knowing if the process wedges on Linux as badly as it does on Windows (to the point where it's not respecting ctrl-c or sigkill). When I call sock1.listen(5) after sock1.bind(), the test passes for me on SuSE Linux 10.1 Thanks, Raghu From tjreedy at udel.edu Mon Apr 7 18:35:00 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 7 Apr 2008 12:35:00 -0400 Subject: [Python-Dev] New project : Spyke python-to-C compiler References: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> Message-ID: "Rahul Garg" wrote in message news:20080406184850.xiqr9t0hwgco004s at webmail.ualberta.ca... | Note this message has been posted to numpy-discussion and python-dev. | Sorry for the multiple posting but I thought both python devs and | numpy users will be interested. If you believe your list should not | receive this email, let me know. Also I just wanted to introduce | myself since I may ask doubts about Python and Numpy internals from | time to time :) Pydev is for discussion about development of future versions of Python (ie, 2.6 and some of 3.0). I think this would have been better posted on comp.lang.python (or gmane.comp.python.general or the corresponding mailing list). You can get answers about current Python internals there. From robert.kern at gmail.com Mon Apr 7 23:14:16 2008 From: robert.kern at gmail.com (Robert Kern) Date: Mon, 07 Apr 2008 14:14:16 -0700 Subject: [Python-Dev] configure error: "rm: conftest.dSYM: is a directory" In-Reply-To: <18424.52783.238721.876293@montanaro-dyndns-org.local> References: <18423.53311.150471.667891@montanaro-dyndns-org.local> <18424.52783.238721.876293@montanaro-dyndns-org.local> Message-ID: skip at pobox.com wrote: > >> Note the "rm: conftest.dSYM: is a directory". This occurred a few > >> times during the configure process. Didn't cause it to conk out, but > >> is annoying. > > Brett> I am assuming this is on your OS X machine, Skip? > > Yes, sorry. I forgot to mention that. As long as someone else sees it and > feels fine to ignore it, I'm fine with it. I suspect it's a configure > problem anyway, not a Python problem. I've seen it with just about every ./configure script on OS X 10.5. It's not specific to Python, and I haven't seen it cause any actual problems. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From guido at python.org Mon Apr 7 23:44:24 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 7 Apr 2008 14:44:24 -0700 Subject: [Python-Dev] Python 3000: Special type for object attributes & map keys In-Reply-To: References: Message-ID: Without an implementation and supporting profile data nobody is going to believe that you can do name lookup faster than with the built-in dict type in CPython. Note that names seen by the parser are already interned, so most of what you seem to be proposing is already implemented... On Wed, Mar 5, 2008 at 2:27 PM, Henrik Vendelbo wrote: > It appears to me that if you can make mapping mechanisms faster in > Python you can make significant > overall speed improvements. I also think the proposed concept could > add flexibility to persistence formats > and RMI interfaces. > > My basic idea is to have a constant string type with an interpreter > globally unique hash. If the original constant > is created in a manner different from string constants, it can be > tracked and handled differently by the interpreter. > > Obviously most object attribute references are done with the dot > operator, so I guess the interpreter already has > an efficient mapping mechanism. But there must be a crossover with > __getattr__ etc, where a map of some sort is > used. I imagine that having a global namespace to translate attribute > names into integers could be used for several > purposes by the interpreter as well as an application exchanging > objects with other applications. > > I imagine these expressions to be supported: > * attrname(string) - creates an attrname value from the string > * int(attrname) - gets the hash value > * string(attrname) - gets the string value > > Hope this makes sense > > Henrik > > P.S. I originally thought of this in a JavaScript context so forgive > me if this would make little difference in Python. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From d at 0x1.org Mon Apr 7 03:51:45 2008 From: d at 0x1.org (David Arnold) Date: Sun, 6 Apr 2008 21:51:45 -0400 Subject: [Python-Dev] [Distutils] Remove "current" Windows executables from Lib/distutils/command in svn? In-Reply-To: <47F86A84.6000204@v.loewis.de> References: <018701c8978e$ce620660$6b261320$@com.au> <47F86A84.6000204@v.loewis.de> Message-ID: On 06/04/2008, at 2:15 AM, Martin v. L?wis wrote: > a) how does the binary get into the release tarball? You might argue > that it doesn't have to, as it is sufficient when it is included in > the Windows installer, however, as currently implemented, > bdist_wininst also runs on Unix, and people use it that way. It would be a shame to lose this ability. It's very handy. d From avikohn at gmail.com Mon Apr 7 06:58:40 2008 From: avikohn at gmail.com (Avi Kohn) Date: Mon, 7 Apr 2008 00:58:40 -0400 Subject: [Python-Dev] python source code Message-ID: I am interested in understanding the python source code. Can someone direct me to resources (documentation,book,archive of mailling lists,etc) that will assist me ? Thank you, Avi -- Avi Kohn -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080407/8bf71990/attachment.htm From dagss at student.matnat.uio.no Mon Apr 7 11:33:05 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 7 Apr 2008 11:33:05 +0200 (CEST) Subject: [Python-Dev] [Numpy-discussion] New project : Spyke python-to-C compiler In-Reply-To: <50807.193.157.243.12.1207560672.squirrel@webmail.uio.no> References: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> <50807.193.157.243.12.1207560672.squirrel@webmail.uio.no> Message-ID: <50810.193.157.243.12.1207560785.squirrel@webmail.uio.no> > (Though as the saying goes, little duplication is normal (and perhaps > wanted) for open source software.) Sorry! I meant "a little", completely reversing the meaning of my sentence. Dag Sverre From dagss at student.matnat.uio.no Mon Apr 7 11:31:12 2008 From: dagss at student.matnat.uio.no (Dag Sverre Seljebotn) Date: Mon, 7 Apr 2008 11:31:12 +0200 (CEST) Subject: [Python-Dev] [Numpy-discussion] New project : Spyke python-to-C compiler In-Reply-To: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> References: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> Message-ID: <50807.193.157.243.12.1207560672.squirrel@webmail.uio.no> > What is Spyke? > In many performance critical projects, it is often necessary to > rewrite parts of the application in C. However writing C wrappers can > be time consuming. Spyke offers an alternative approach. You add > annotations to your Python code as strings. These strings are > discarded by the Python > interpreter but these are interpreted as types by Spyke compiler to > convert to C. Have you had a look at Cython? http://cython.org. >From seeing what you write, it looks like we have almost exactly the same long-term goals; one could almost say that the two pieces of software will be complete duplicates in functionality. (Cython isn't "there" just yet though.) (Though as the saying goes, little duplication is normal (and perhaps wanted) for open source software.) Dag Sverre From musiccomposition at gmail.com Tue Apr 8 01:01:44 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Mon, 7 Apr 2008 18:01:44 -0500 Subject: [Python-Dev] python source code In-Reply-To: References: Message-ID: <1afaf6160804071601q71d43dd0x39c278188373203a@mail.gmail.com> On Sun, Apr 6, 2008 at 11:58 PM, Avi Kohn wrote: > I am interested in understanding the python source code. Can someone > direct me to resources (documentation,book,archive of mailling lists,etc) > that will assist me ? > What part(s) do you want to learn about? The CPython interpreter is quite huge. I recommend you ask the comp.lang.python mailing list for further advice. > Thank you, > Avi > > -- > Avi Kohn > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com > > -- Cheers, Benjamin Peterson -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080407/07f9d09f/attachment-0001.htm From musiccomposition at gmail.com Tue Apr 8 01:04:52 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Mon, 7 Apr 2008 18:04:52 -0500 Subject: [Python-Dev] New project : Spyke python-to-C compiler In-Reply-To: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> References: <20080406184850.xiqr9t0hwgco004s@webmail.ualberta.ca> Message-ID: <1afaf6160804071604g1b7f8eeanda6673444b6e88e0@mail.gmail.com> [snip] > > > > c) Strings as type declarations : Do you think I should use decorators > instead at least for function type declarations? You might be interested in 3.0's (and maybe 2.6's) function annotations. See PEP 3107. > > > thanks for patiently reading this, > comments and inquiries sought. > rahul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com > -- Cheers, Benjamin Peterson -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080407/8875ab2f/attachment.htm From bonelake at gmail.com Tue Apr 8 01:14:53 2008 From: bonelake at gmail.com (Brad Miller) Date: Mon, 7 Apr 2008 18:14:53 -0500 Subject: [Python-Dev] string representation of range in 3.0 Message-ID: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> Hi, I use Python in my CS1 and CS2 curriculum and I have a question. As I've been using the Python 3.0 alphas one of the things that I am bothered by is that I cannot see the sequence produced by range without introducing students to the list() function. I typically introduce range on day 1 of class and show students what it produces without making a big deal out of the fact that it creates a list. They all accept this and things work out nicely when I introduce lists for real in a week or two. My question is why couldn't the __str__ method for the range object be more friendly and show a representation of the sequence? I understand why __repr__ should return range(0,10) for an object created using range(10) but couldn't print(range(10)) produce [0, 1, 2, ... 9] The ... could even be used if the sequence were excessively wrong. If this is acceptable, I would be happy to accept the challenge of providing a patch. Thanks, Brad From guido at python.org Tue Apr 8 01:24:19 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 7 Apr 2008 16:24:19 -0700 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> Message-ID: I'd object to it returning something that resembles a list too closely, but I could live with str(range(3)) return <0, 1, 2>. We should probably have a cutoff so that if there are more than 6 values it'll show the first 3 values, then dots, then the last 2 values. (The cutoff would be computed so that '...' always represents at least 2 values. On Mon, Apr 7, 2008 at 4:14 PM, Brad Miller wrote: > Hi, > > I use Python in my CS1 and CS2 curriculum and I have a question. > As I've been using the Python 3.0 alphas one of the things that I am > bothered by is that I cannot see the sequence produced by range > without introducing students to the list() function. > > I typically introduce range on day 1 of class and show students what > it produces without making a big deal out of the fact that it creates > a list. They all accept this and things work out nicely when I > introduce lists for real in a week or two. > > My question is why couldn't the __str__ method for the range object be > more friendly and show a representation of the sequence? I understand > why __repr__ should return range(0,10) for an object created using > range(10) but couldn't print(range(10)) produce [0, 1, 2, ... 9] > The ... could even be used if the sequence were excessively wrong. > > If this is acceptable, I would be happy to accept the challenge of > providing a patch. > > Thanks, > > Brad > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Apr 8 02:37:54 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 7 Apr 2008 17:37:54 -0700 Subject: [Python-Dev] unittest's redundant assertions: asserts vs. failIf/Unlesses In-Reply-To: <5d44f72f0803191716k330e1249kce353d173fe503@mail.gmail.com> References: <5d44f72f0803190120n700de143m5d67b8164f7c57ae@mail.gmail.com> <644FD103-F31C-4A7E-8172-374B57BF3BC4@python.org> <20080319211556.21558.2085930557.divmod.xquotient.8213@joule.divmod.com> <5d44f72f0803191716k330e1249kce353d173fe503@mail.gmail.com> Message-ID: On Wed, Mar 19, 2008 at 5:16 PM, Jeffrey Yasskin wrote: > On Wed, Mar 19, 2008 at 2:15 PM, wrote: > > > > On 02:21 pm, murman at gmail.com wrote: > > >>OTOH, I'd rather there be OOWTDI so whatever the consensus is is fine > > >>with me. > > > > > >This strikes me as a gratuitous API change of the kind Guido was > > >warning about in his recent post: "Don't change your APIs incompatibly > > >when porting to Py3k" > > > > I agree emphatically. Actually I think this is the most extreme case. > > The unit test stuff should be as stable as humanly possible between 2 > > and 3, moreso than any other library. > > This is convincing for me. Move my +1 back to 3.1. Same here; let's tread carefully here and not change this with 3.0. Starting to deprecate in 3.1 and killing in 3.3 would be soon enough. I like using only the assertKeyword variants, removing assert_, fail*, and assertEquals. However I don't like changing assertTrue and assertFalse to insist that the value is exactly True or False -- if you really care that much, let's add assertIs(x, y) which asserts that x and y are the same object. I also think that all tests should use the operator their name implies, e.g. assertEqual(x, y) should do something like if x == y: pass else: raise AssertionError(...) rather than if x != y: raise AssertionError(...) Someone please open a bug for this task. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From musiccomposition at gmail.com Tue Apr 8 03:02:08 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Mon, 7 Apr 2008 20:02:08 -0500 Subject: [Python-Dev] unittest's redundant assertions: asserts vs. failIf/Unlesses In-Reply-To: References: <5d44f72f0803190120n700de143m5d67b8164f7c57ae@mail.gmail.com> <644FD103-F31C-4A7E-8172-374B57BF3BC4@python.org> <20080319211556.21558.2085930557.divmod.xquotient.8213@joule.divmod.com> <5d44f72f0803191716k330e1249kce353d173fe503@mail.gmail.com> Message-ID: <1afaf6160804071802j28a247f0t70af39044c691e14@mail.gmail.com> On Mon, Apr 7, 2008 at 7:37 PM, Guido van Rossum wrote: > On Wed, Mar 19, 2008 at 5:16 PM, Jeffrey Yasskin wrote: > > On Wed, Mar 19, 2008 at 2:15 PM, wrote: > > > > > > On 02:21 pm, murman at gmail.com wrote: > > > >>OTOH, I'd rather there be OOWTDI so whatever the consensus is is fine > > > >>with me. > > > > > > > >This strikes me as a gratuitous API change of the kind Guido was > > > >warning about in his recent post: "Don't change your APIs incompatibly > > > >when porting to Py3k" > > > > > > I agree emphatically. Actually I think this is the most extreme case. > > > The unit test stuff should be as stable as humanly possible between 2 > > > and 3, moreso than any other library. > > > > This is convincing for me. Move my +1 back to 3.1. > > Same here; let's tread carefully here and not change this with 3.0. > Starting to deprecate in 3.1 and killing in 3.3 would be soon enough. > I like using only the assertKeyword variants, removing assert_, fail*, > and assertEquals. However I don't like changing assertTrue and > assertFalse to insist that the value is exactly True or False -- if > you really care that much, let's add assertIs(x, y) which asserts that > x and y are the same object. I also think that all tests should use > the operator their name implies, e.g. assertEqual(x, y) should do > something like > > if x == y: > pass > else: > raise AssertionError(...) > > rather than > > if x != y: > raise AssertionError(...) > > Someone please open a bug for this task. See 2578. > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com > -- Cheers, Benjamin Peterson From fuzzyman at voidspace.org.uk Tue Apr 8 10:58:46 2008 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 08 Apr 2008 09:58:46 +0100 Subject: [Python-Dev] unittest's redundant assertions: asserts vs. failIf/Unlesses In-Reply-To: References: <5d44f72f0803190120n700de143m5d67b8164f7c57ae@mail.gmail.com> <644FD103-F31C-4A7E-8172-374B57BF3BC4@python.org> <20080319211556.21558.2085930557.divmod.xquotient.8213@joule.divmod.com> <5d44f72f0803191716k330e1249kce353d173fe503@mail.gmail.com> Message-ID: <47FB33C6.4080008@voidspace.org.uk> Guido van Rossum wrote: > On Wed, Mar 19, 2008 at 5:16 PM, Jeffrey Yasskin wrote: > >> On Wed, Mar 19, 2008 at 2:15 PM, wrote: >> > >> > On 02:21 pm, murman at gmail.com wrote: >> > >>OTOH, I'd rather there be OOWTDI so whatever the consensus is is fine >> > >>with me. >> > > >> > >This strikes me as a gratuitous API change of the kind Guido was >> > >warning about in his recent post: "Don't change your APIs incompatibly >> > >when porting to Py3k" >> > >> > I agree emphatically. Actually I think this is the most extreme case. >> > The unit test stuff should be as stable as humanly possible between 2 >> > and 3, moreso than any other library. >> >> This is convincing for me. Move my +1 back to 3.1. >> > > Same here; let's tread carefully here and not change this with 3.0. > Starting to deprecate in 3.1 and killing in 3.3 would be soon enough. > I like using only the assertKeyword variants, removing assert_, fail*, > and assertEquals. However I don't like changing assertTrue and > assertFalse to insist that the value is exactly True or False -- if > you really care that much, let's add assertIs(x, y) which asserts that > x and y are the same object. I also think that all tests should use > the operator their name implies, e.g. assertEqual(x, y) should do > something like > > if x == y: > pass > else: > raise AssertionError(...) > > rather than > > if x != y: > raise AssertionError(...) > > Someone please open a bug for this task. > > This sounds like a good compromise and I'm happy to take on the cleanup - unless someone else beats me to it. I guess it should wait until 3.0 final is out of the door before adding the DeprecationWarnings. Michael Foord From tnelson at onresolve.com Tue Apr 8 14:00:12 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Tue, 8 Apr 2008 05:00:12 -0700 Subject: [Python-Dev] Changing all network-oriented tests to facilitate (virtually unlimited) parallel test execution [Issue#: 2550] Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22939EEF@EXMBX04.exchhosting.com> I've forwarding my most recent update to issue 2550 here such that the proposed patch (and in general, the approach to network-oriented test cases) can be vetted by a wider audience: http://bugs.python.org/file9980/trunk.2550-2.patch This patch works towards fixing a large proportion of the tests that were written in such a way that often leads to buildbot errors when port conflicts arise. With this patch applied, I can run many instances of the test suite in parallel and none of the network-oriented tests fail (which they do currently if you try and do this). Plenty of discussion (mostly me replying to my own comments) on the subject at: http://bugs.python.org/issue2550. Anyone have any issues with this new approach? I'm particularily interested in whether or not people disagree with the assumption I've drawn regarding never using SO_REUSEADDR in tests for TCP/IP sockets (see below). Trent. ________________________________________ From: python-bugs-list-bounces+tnelson=onresolve.com at python.org [python-bugs-list-bounces+tnelson=onresolve.com at python.org] On Behalf Of Trent Nelson [report at bugs.python.org] Sent: 08 April 2008 12:49 To: python-bugs-list at python.org Subject: [issue2550] SO_REUSEADDR doesn't have the same semantics on Windows as on Unix Trent Nelson added the comment: Invested quite a few cycles on this issue last night. The more time I spent on it, the more I became convinced that every single test working with sockets should be changed in one fell swoop in order to facilitate (virtually unlimited) parallel test execution without fear of port conflicts. I've attached a second patch, trunk.2550-2.patch, which is my progress so far on doing just this. The main changes can be expressed by the following two points: a) do whatever it takes in network-oriented tests to ensure unique ports are obtained (relying on the bind_port() and find_unused_port() methods exposed by test_support) b) never, ever, ever call SO_REUSEADDR on a socket from a test; because we're putting so much effort into obtaining a unique port, this should never be necessary -- in the rare cases that our attempts to obtain a unique port fail, then we absolutely should fail with EADDRINUSE, as the ability to obtain a unique port for the duration of a client/server test is an invariant that we *must* be able to depend upon. If the invariant is broken, fail immediately (don't mask the problem with SO_REUSEADDR). With this patch applied, I can spawn a handful of Python processes and run the entire test suite (without -r, ensuring all tests are run in the same order, which should encourage port conflicts (if there were any)) without any errors*. Doing that now is completely and utterly impossible. [*] Well, almost without error. All the I/O related tests that try and open @test fail. I believe there's still outstanding work to do with this patch with regards to how the intracacies of SO_REUSEADDR and SO_EXCLUSIVEADDRUSE should be handled in the rest of the stdlib. I'm still thinking about the best approach for this. However, the patch as it currently stands is still quite substantial so I wanted to get it out sooner rather than later for review. (I'll forward this to python-dev@ to try and encourage more eyes from people with far more network-fu than I.) Added file: http://bugs.python.org/file9980/trunk.2550-2.patch __________________________________ Tracker __________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/tnelson%40onresolve.com From alessandro.guido at gmail.com Tue Apr 8 15:22:50 2008 From: alessandro.guido at gmail.com (Alessandro Guido) Date: Tue, 8 Apr 2008 15:22:50 +0200 Subject: [Python-Dev] py3k: print function treats sep=None and end=None in an unintuitive way Message-ID: <200804081522.50764.alessandro.guido@gmail.com> Can anybody please point me why print('a', 'b', sep=None, end=None) should produce "a b\n" instead of "ab"? I've read http://docs.python.org/dev/3.0/library/functions.html#print, pep-3105 and some ml threads but did not find a good reason justifying such a strange behaviour. Thanks. -Alessandro Guido From ncoghlan at gmail.com Tue Apr 8 15:57:08 2008 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 08 Apr 2008 23:57:08 +1000 Subject: [Python-Dev] py3k: print function treats sep=None and end=None in an unintuitive way In-Reply-To: <200804081522.50764.alessandro.guido@gmail.com> References: <200804081522.50764.alessandro.guido@gmail.com> Message-ID: <47FB79B4.5020605@gmail.com> Alessandro Guido wrote: > Can anybody please point me why print('a', 'b', sep=None, end=None) should > produce "a b\n" instead of "ab"? > I've read http://docs.python.org/dev/3.0/library/functions.html#print, pep-3105 and some > ml threads but did not find a good reason justifying such a strange behaviour. So that print(a, b) does the right thing (i.e. matches the Python 2.x print statement's behaviour) sep=None, end=None means "use the default separator and line ending" rather than "don't use a separator or line ending" (this is the same as for most functions with optional arguments - you need to look at the documentation of the function to find out what default values are used when passing None for an optional argument). That said, it does read oddly, so I'd advise against writing it out explicitly like that - if you want the default, just leave out the relevant argument. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From eric+python-dev at trueblade.com Tue Apr 8 15:38:26 2008 From: eric+python-dev at trueblade.com (Eric Smith) Date: Tue, 08 Apr 2008 09:38:26 -0400 Subject: [Python-Dev] py3k: print function treats sep=None and end=None in an unintuitive way In-Reply-To: <200804081522.50764.alessandro.guido@gmail.com> References: <200804081522.50764.alessandro.guido@gmail.com> Message-ID: <47FB7552.4020109@trueblade.com> Alessandro Guido wrote: > Can anybody please point me why print('a', 'b', sep=None, end=None) should > produce "a b\n" instead of "ab"? > I've read http://docs.python.org/dev/3.0/library/functions.html#print, pep-3105 and some > ml threads but did not find a good reason justifying such a strange behaviour. > > Thanks. > > -Alessandro Guido Because None mean 'use the default value'. You probably want: print('a', 'b', sep='', end='') From eric+python-dev at trueblade.com Tue Apr 8 15:40:26 2008 From: eric+python-dev at trueblade.com (Eric Smith) Date: Tue, 08 Apr 2008 09:40:26 -0400 Subject: [Python-Dev] py3k: print function treats sep=None and end=None in an unintuitive way In-Reply-To: <200804081522.50764.alessandro.guido@gmail.com> References: <200804081522.50764.alessandro.guido@gmail.com> Message-ID: <47FB75CA.6070601@trueblade.com> Alessandro Guido wrote: > Can anybody please point me why print('a', 'b', sep=None, end=None) should > produce "a b\n" instead of "ab"? > I've read http://docs.python.org/dev/3.0/library/functions.html#print, pep-3105 and some > ml threads but did not find a good reason justifying such a strange behaviour. > > Thanks. > > -Alessandro Guido Because None mean 'use the default value'. You probably want: print('a', 'b', sep='', end='') From eric+python-dev at trueblade.com Tue Apr 8 15:44:02 2008 From: eric+python-dev at trueblade.com (Eric Smith) Date: Tue, 08 Apr 2008 09:44:02 -0400 Subject: [Python-Dev] py3k: print function treats sep=None and end=None in an unintuitive way In-Reply-To: <200804081522.50764.alessandro.guido@gmail.com> References: <200804081522.50764.alessandro.guido@gmail.com> Message-ID: <47FB76A2.7020600@trueblade.com> Alessandro Guido wrote: > Can anybody please point me why print('a', 'b', sep=None, end=None) should > produce "a b\n" instead of "ab"? > I've read http://docs.python.org/dev/3.0/library/functions.html#print, pep-3105 and some > ml threads but did not find a good reason justifying such a strange behaviour. > > Thanks. > > -Alessandro Guido Because None means 'use the default value'. You probably want: print('a', 'b', sep='', end='') >>> import io >>> s = io.StringIO() >>> print('a', 'b', end='', sep='', file=s) >>> s.getvalue() 'ab' >>> From eric+python-dev at trueblade.com Tue Apr 8 16:22:45 2008 From: eric+python-dev at trueblade.com (Eric Smith) Date: Tue, 08 Apr 2008 10:22:45 -0400 Subject: [Python-Dev] py3k: print function treats sep=None and end=None in an unintuitive way In-Reply-To: <47FB76A2.7020600@trueblade.com> References: <200804081522.50764.alessandro.guido@gmail.com> <47FB76A2.7020600@trueblade.com> Message-ID: <47FB7FB5.4010506@trueblade.com> [Sorry for the dupes. Lesson: never try and send mail from a moving train.] Eric Smith wrote: > Alessandro Guido wrote: >> Can anybody please point me why print('a', 'b', sep=None, end=None) should >> produce "a b\n" instead of "ab"? >> I've read http://docs.python.org/dev/3.0/library/functions.html#print, pep-3105 and some >> ml threads but did not find a good reason justifying such a strange behaviour. >> >> Thanks. >> >> -Alessandro Guido > > Because None means 'use the default value'. You probably want: > print('a', 'b', sep='', end='') > > >>> import io > >>> s = io.StringIO() > >>> print('a', 'b', end='', sep='', file=s) > >>> s.getvalue() > 'ab' > >>> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/eric%2Bpython-dev%40trueblade.com > From tnelson at onresolve.com Tue Apr 8 16:47:43 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Tue, 8 Apr 2008 07:47:43 -0700 Subject: [Python-Dev] [OT] Wingware IDE key for sprinters at PyCon Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22939EF5@EXMBX04.exchhosting.com> Anyone happen to have the key handy that Wingware were giving out to sprinters at PyCon? For the life of me, I can't find what I did with that piece of paper. If someone could forward me it off list, that'd be great. Trent. From alessandro.guido at gmail.com Tue Apr 8 18:01:15 2008 From: alessandro.guido at gmail.com (Alessandro Guido) Date: Tue, 8 Apr 2008 18:01:15 +0200 Subject: [Python-Dev] =?iso-8859-1?q?py3k=3A_print_function_treats_sep=3DN?= =?iso-8859-1?q?one_and_end=3DNone_in=09an_unintuitive_way?= In-Reply-To: <47FB79B4.5020605@gmail.com> References: <200804081522.50764.alessandro.guido@gmail.com> <47FB79B4.5020605@gmail.com> Message-ID: <200804081801.15687.alessandro.guido@gmail.com> Nick Coghlan wrote: > So that print(a, b) does the right thing (i.e. matches the Python 2.x > print statement's behaviour) AFAICS print(a, b) does the right thing because default values of "sep" and "end" are ' ' and '\n' respectively, doesn't it? Eric Smith wrote: > Because None means 'use the default value'. You probably want: > print('a', 'b', sep='', end='') I think this is a "not optimally designed" API because you have to read the documentation to understand why print('a', 'b', sep=None, end=None) does not translate into the obvious: ?print strings 'a' and 'b' using no separator and no terminator? but into: ?print strings 'a' and 'b' using the default separator and the default terminator? However i'll just cope with it, Python is still the best language ;) Thank you all for replying! -Alessandro Guido From zooko at zooko.com Tue Apr 8 19:01:23 2008 From: zooko at zooko.com (zooko) Date: Tue, 8 Apr 2008 10:01:23 -0700 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <47EB07A6.6040800@plope.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> Message-ID: <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> On Mar 26, 2008, at 7:34 PM, Chris McDonough wrote: > zooko wrote: http://mail.python.org/pipermail/python-dev/2008-March/078243.html >> Here is a simple proposal: make the standard Python "import" >> mechanism notice eggs on the PYTHONPATH and insert them (into the >> *same* location) on the sys.path. >> This eliminates the #1 problem with eggs -- that they don't >> easily work when installing them into places other than your site- >> packages and that if you allow any of them to be installed on >> your system then they take precedence over your non-egg packages >> even you explicitly put those other packages earlier in your >> PYTHONPATH. (That latter behavior is very disagreeable to more >> than a few prorgammers.) > > Sorry if I'm out of the loop and there's some subtlety here that > I'm disregarding, but it doesn't appear that either of the issues > you mention is a actually problem with eggs. These are instead > problems with how eggs get installed by easy_install (which uses > a .pth file to extend sys.path). It's reasonable to put eggs on > the PYTHONPATH manually (e.g. sys.path.append('/path/to/some.egg')) > instead of using easy_install to install them. Yes, you are missing something. While many programmers, such as yourself and Lennart Regebro (who posted to this thread) find the current eggs system to be perfectly convenient and to Just Work, many others, such as Glyph Lefkowitz (who posted to a related thread) find them to be so annoying that they actively ensure that no eggs are ever allowed to touch their system. The reasons for this latter problem are two: 1. You can't conveniently install eggs into a non-system directory, such as ~/my-python-stuff. 2. If you allow even a single egg to be installed into your PYTHONPATH, it will change the semantics of your PYTHONPATH. Both of these problems are directly caused by the need for eggs to hack your site.py. If Python automatically added eggs found in the PYTHONPATH to the sys.path, both of these problems would go away. I am skeptical that the current proposals to define a new database for installed packages will fare any better than the current eggs scheme does in this respect. This issue is important to me, because the benefits of eggs grow superlinearly with the number of good programmers who use them. They are a tool for re-using source code -- a tool for cooperation between programmers. To gain the greatest benefits at this point we do not need to add new features to eggs, we need to make them more palatable to more good programmers. I am skeptical that prorgammers are going to be willing to use a new database format. They already have a database -- their filesystem -- and they already have the tools to control it -- mv, rm, and PYTHONPATH. Many of them already hate the existence the "easy_instlal.pth" database file, and I don't see why a new database file would be any different. My proposal makes the current benefits of eggs -- clean, easy code re- use among programmers -- more compatible with their current tools -- mv, rm, and PYTHONPATH. It is also forward-compatible with more sophisticated proposals to add features like uninstall and operating system integration. By the way, since I posted my proposal two weeks ago I have pointed a couple of Python hackers who currently refuse to use eggs at the URL: http://mail.python.org/pipermail/python-dev/2008-March/078243.html They both agreed that it made perfect sense. I told one of them about the alternate proposal to define a new database file to contain a list of installed packages, and he sighed and rolled his eyes and said "So they are planning to reinvent apt!". Regards, Zooko From ncoghlan at gmail.com Tue Apr 8 19:15:58 2008 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 09 Apr 2008 03:15:58 +1000 Subject: [Python-Dev] py3k: print function treats sep=None and end=None in an unintuitive way In-Reply-To: <200804081801.15687.alessandro.guido@gmail.com> References: <200804081522.50764.alessandro.guido@gmail.com> <47FB79B4.5020605@gmail.com> <200804081801.15687.alessandro.guido@gmail.com> Message-ID: <47FBA84E.3050608@gmail.com> Alessandro Guido wrote: > ?print strings 'a' and 'b' using the default separator and the default terminator? > > However i'll just cope with it, Python is still the best language ;) I definitely recommend getting used to this idiom - None is used to indicate missing (i.e. 'use the default value') arguments in many more cases than just the print function. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From pje at telecommunity.com Tue Apr 8 19:42:08 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 08 Apr 2008 13:42:08 -0400 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> Message-ID: <20080408174307.79E453A4108@sparrow.telecommunity.com> At 10:01 AM 4/8/2008 -0700, zooko wrote: >On Mar 26, 2008, at 7:34 PM, Chris McDonough wrote: > > zooko wrote: > >http://mail.python.org/pipermail/python-dev/2008-March/078243.html > > >> Here is a simple proposal: make the standard Python "import" > >> mechanism notice eggs on the PYTHONPATH and insert them (into the > >> *same* location) on the sys.path. > >> This eliminates the #1 problem with eggs -- that they don't > >> easily work when installing them into places other than your site- > >> packages and that if you allow any of them to be installed on > >> your system then they take precedence over your non-egg packages > >> even you explicitly put those other packages earlier in your > >> PYTHONPATH. (That latter behavior is very disagreeable to more > >> than a few prorgammers.) > > > > Sorry if I'm out of the loop and there's some subtlety here that > > I'm disregarding, but it doesn't appear that either of the issues > > you mention is a actually problem with eggs. These are instead > > problems with how eggs get installed by easy_install (which uses > > a .pth file to extend sys.path). It's reasonable to put eggs on > > the PYTHONPATH manually (e.g. sys.path.append('/path/to/some.egg')) > > instead of using easy_install to install them. > >Yes, you are missing something. While many programmers, such as >yourself and Lennart Regebro (who posted to this thread) find the >current eggs system to be perfectly convenient and to Just Work, many >others, such as Glyph Lefkowitz (who posted to a related thread) find >them to be so annoying that they actively ensure that no eggs are >ever allowed to touch their system. > >The reasons for this latter problem are two: > >1. You can't conveniently install eggs into a non-system directory, >such as ~/my-python-stuff. Wha? >2. If you allow even a single egg to be installed into your >PYTHONPATH, it will change the semantics of your PYTHONPATH. Only in the same way that manually putting an egg on the front of PYTHONPATH can be considered to "change the semantics" of your PYTHONPATH. >Both of these problems are directly caused by the need for eggs to >hack your site.py. If Python automatically added eggs found in the >PYTHONPATH to the sys.path, both of these problems would go away. And add new ones. >I am skeptical that the current proposals to define a new database >for installed packages will fare any better than the current eggs >scheme does in this respect. The purpose for the installation database is to allow easy_install to eschew the use of .egg files or directories for anything other than multi-version installs. Thus, no need to add those .egg files or directories to the head of the PYTHONPATH. Conflicts would be handled at install time rather than runtime, in other words. >I am skeptical that prorgammers are going to be willing to use a new >database format. They already have a database -- their filesystem -- >and they already have the tools to control it -- mv, rm, and >PYTHONPATH. Many of them already hate the existence the >"easy_instlal.pth" database file, and I don't see why a new database >file would be any different. PEP 262 does not propose a database file -- it proposes the inclusion of a metadata file for each installed distribution. >My proposal makes the current benefits of eggs -- clean, easy code re- >use among programmers -- more compatible with their current tools -- >mv, rm, and PYTHONPATH. It is also forward-compatible with more >sophisticated proposals to add features like uninstall and operating >system integration. Actually, your current proposal doesn't work, unless you at least have some way to indicate which *version* of an egg should be automatically added to sys.path -- and some way to change that. Otherwise, you might as well use the -m option to easy_install, and require() the eggs at runtime. (Which needs neither .pth files nor site.py hacking.) Meanwhile, my understanding is that the people who dislike eggs, dislike them because when they install a setuptools-based package, it's installed as an egg by default. The installation database proposal (and by the way, people really should read and understand PEP 262, including the open issues, before trying to compete with it), will allow setuptools-based packages to install the "old-fashioned" way by default. That is, not as eggs. Similarly, easy_install would be able to skip installing .eggs unless you wanted multi-version support. So, people who don't like eggs would never see them, since the only way you'd ever get them would be via easy_install -m, and they would never use it. >By the way, since I posted my proposal two weeks ago I have pointed a >couple of Python hackers who currently refuse to use eggs at the URL: > >http://mail.python.org/pipermail/python-dev/2008-March/078243.html > >They both agreed that it made perfect sense. I told one of them >about the alternate proposal to define a new database file to contain >a list of installed packages, and he sighed and rolled his eyes and >said "So they are planning to reinvent apt!". No, we're planning to make it possible for easy_install not to overwrite things that would break your system, and allow distutils and setuptools to uninstall what they installed. That's a considerably less ambitious goal, by far. :) From p.f.moore at gmail.com Tue Apr 8 19:43:55 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 8 Apr 2008 18:43:55 +0100 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> Message-ID: <79990c6b0804081043ld93ea2fua01b9eb96be9f5c@mail.gmail.com> On 08/04/2008, zooko wrote: > By the way, since I posted my proposal two weeks ago I have pointed a > couple of Python hackers who currently refuse to use eggs at the URL: > > http://mail.python.org/pipermail/python-dev/2008-March/078243.html > > They both agreed that it made perfect sense. I told one of them > about the alternate proposal to define a new database file to contain > a list of installed packages, and he sighed and rolled his eyes and > said "So they are planning to reinvent apt!". I do think that a simple solution like that has some merit. It has two significant downsides, however: 1. It requires that core Python "bless" the egg format to some extent - something Guido has said he is unwilling to do. 2. It ignores the issue of package management completely. Personally, I avoid anything that doesn't integrate with a unified package manager (whether that be the Windows add/remove feature, or an as-yet-to-be-built custom Python package manager is not important). Filesystem commands do not a package manager make... Paul. From chrism at plope.com Tue Apr 8 19:52:19 2008 From: chrism at plope.com (Chris McDonough) Date: Tue, 08 Apr 2008 13:52:19 -0400 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> Message-ID: <47FBB0D3.2090405@plope.com> zooko wrote: > > On Mar 26, 2008, at 7:34 PM, Chris McDonough wrote: >> zooko wrote: > > http://mail.python.org/pipermail/python-dev/2008-March/078243.html > >>> Here is a simple proposal: make the standard Python "import" >>> mechanism notice eggs on the PYTHONPATH and insert them (into the >>> *same* location) on the sys.path. >>> This eliminates the #1 problem with eggs -- that they don't easily >>> work when installing them into places other than your site-packages >>> and that if you allow any of them to be installed on your system >>> then they take precedence over your non-egg packages even you >>> explicitly put those other packages earlier in your PYTHONPATH. >>> (That latter behavior is very disagreeable to more than a few >>> prorgammers.) >> >> Sorry if I'm out of the loop and there's some subtlety here that I'm >> disregarding, but it doesn't appear that either of the issues you >> mention is a actually problem with eggs. These are instead problems >> with how eggs get installed by easy_install (which uses a .pth file to >> extend sys.path). It's reasonable to put eggs on the PYTHONPATH >> manually (e.g. sys.path.append('/path/to/some.egg')) instead of using >> easy_install to install them. > > Yes, you are missing something. While many programmers, such as > yourself and Lennart Regebro (who posted to this thread) find the > current eggs system to be perfectly convenient and to Just Work, many > others, such as Glyph Lefkowitz (who posted to a related thread) find > them to be so annoying that they actively ensure that no eggs are ever > allowed to touch their system. > > The reasons for this latter problem are two: > > 1. You can't conveniently install eggs into a non-system directory, > such as ~/my-python-stuff. That's just not true. $ export PYTHONPATH=/home/you/my-python-stuff/foo-1.3.egg $ python >>> import foo Eggs are directories (or zipfiles) that contain packages. They happen to contain other metadata directories too, but these can be ignored if you just want to *use* them (as opposed to wanting to introspect metadata about them). > 2. If you allow even a single egg to be installed into your PYTHONPATH, > it will change the semantics of your PYTHONPATH. I think you are mistaken. The use of the .pth file that changes sys.path is a feature of easy_install, not of eggs. You don't need to use any .pth file to put eggs on your PYTHONPATH. Note that zc.buildout is a framework that installs eggs, and it doesn't rely at all on .pth files to automatically hack sys.path. Instead, it munges console scripts to add each egg dir to sys.path. This is pretty nasty too, but it does prove the point. It is however true that you need to change sys.path to use an egg. Is that what you're objecting to fundamentally? You just don't want to have to change sys.path at all to use an egg? Maybe you're objecting to the notion that an egg can contain more than one package. There is functionally no difference between an egg and a directory full of packages. > Both of these problems are directly caused by the need for eggs to hack > your site.py. If Python automatically added eggs found in the > PYTHONPATH to the sys.path, both of these problems would go away. I'm not sure what you mean. Eggs don't hack site.py. Eggs are just a packaging format. easy_install doesn't hack site.py either. It just puts a .pth file (the parsing of which is a feature of "core" Python itself, not any setuptools magic) in site packages and manages it. It seems like you're advocating adding magic that you can't turn off (magical detection of eggs in an already site.py-approved packages directory) to defeat magic that you can turn off (by not using easy_install or .pth files). At some level the magic you want to see built in to Python would almost certainly wind up doing what you hate by hacking sys.path unless you wanted to restrict eggs to containing a single package only. And you wouldn't be able to turn it off except through some obscure environment variable setting. > By the way, since I posted my proposal two weeks ago I have pointed a > couple of Python hackers who currently refuse to use eggs at the URL: > > http://mail.python.org/pipermail/python-dev/2008-March/078243.html > > They both agreed that it made perfect sense. I told one of them about > the alternate proposal to define a new database file to contain a list > of installed packages, and he sighed and rolled his eyes and said "So > they are planning to reinvent apt!". I think changing the Python core is the worst possible answer to this problem. "Don't use easy_install" is currently the best, AFAICT. - C From tnelson at onresolve.com Tue Apr 8 20:36:47 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Tue, 8 Apr 2008 11:36:47 -0700 Subject: [Python-Dev] [OT] Wingware IDE key for sprinters at PyCon In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22939EF5@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22939EF5@EXMBX04.exchhosting.com> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E226D9B7E@EXMBX04.exchhosting.com> All sorted, thanks. > -----Original Message----- > From: python-dev-bounces+tnelson=onresolve.com at python.org > [mailto:python-dev-bounces+tnelson=onresolve.com at python.org] > On Behalf Of Trent Nelson > Sent: 08 April 2008 15:48 > To: python-dev at python.org > Subject: [Python-Dev] [OT] Wingware IDE key for sprinters at PyCon > > Anyone happen to have the key handy that Wingware were giving > out to sprinters at PyCon? For the life of me, I can't find > what I did with that piece of paper. If someone could > forward me it off list, that'd be great. > > Trent. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/tnelson%40on resolve.com > From zooko at zooko.com Tue Apr 8 20:46:27 2008 From: zooko at zooko.com (zooko) Date: Tue, 8 Apr 2008 11:46:27 -0700 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <1207679260.3474.74.camel@localhost.localdomain> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> <1207679260.3474.74.camel@localhost.localdomain> Message-ID: <236B9285-BE59-45CA-B727-74B3B3B764BA@zooko.com> On Apr 8, 2008, at 11:27 AM, Lloyd Kvam wrote: > > When I wear my sysadmin hat, eggs become a nuisance. ... > As a developer, eggs are great. ... > Fortunately, distutils includes tools like bdist_rpm so that python > modules can be packaged for easy processing by the system package > manager. So once I need to switch back to a sysadmin role, I can use > the system tools to install and track packages. This is the same experience I have. I rely on setuptools and eggs extensively in developing our software, and I use setuptools and eggs as the primary method of giving our source code to other programmers. But no software is ever installed on our production servers unless that software is in .deb form in an apt-gettable repository, and this requirement is unlikely to change for the forseeable future. Regards, Zooko From greg at krypto.org Tue Apr 8 21:58:01 2008 From: greg at krypto.org (Gregory P. Smith) Date: Tue, 8 Apr 2008 12:58:01 -0700 Subject: [Python-Dev] Changing all network-oriented tests to facilitate (virtually unlimited) parallel test execution [Issue#: 2550] In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22939EEF@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22939EEF@EXMBX04.exchhosting.com> Message-ID: <52dc1c820804081258r73d5491chf0a79f0f7de8c3f4@mail.gmail.com> On Tue, Apr 8, 2008 at 5:00 AM, Trent Nelson wrote: > > I've forwarding my most recent update to issue 2550 here such that the > proposed patch (and in general, the approach to network-oriented test cases) > can be vetted by a wider audience: > > http://bugs.python.org/file9980/trunk.2550-2.patch > > This patch works towards fixing a large proportion of the tests that were > written in such a way that often leads to buildbot errors when port > conflicts arise. With this patch applied, I can run many instances of the > test suite in parallel and none of the network-oriented tests fail (which > they do currently if you try and do this). > > Plenty of discussion (mostly me replying to my own comments) on the > subject at: http://bugs.python.org/issue2550. > > Anyone have any issues with this new approach? I'm particularily > interested in whether or not people disagree with the assumption I've drawn > regarding never using SO_REUSEADDR in tests for TCP/IP sockets (see below). > > Trent. I think your assumptions are fair. Note that not using SO_REUSEADDR can reserve the port for a few minutes beyond the actual unbinding destruction of the server socket on some OSes but with tens of thousands of ports available and likely not more than 1-200 being needed by a full test suite run and how long the test suite takes to run I think that is acceptable and is not a problem we're likely to ever run into (fix it only iffwe ever do). -gps > > > > ________________________________________ > From: python-bugs-list-bounces+tnelson=onresolve.com at python.org[python-bugs-list-bounces+tnelson= > onresolve.com at python.org] On Behalf Of Trent Nelson [ > report at bugs.python.org] > Sent: 08 April 2008 12:49 > To: python-bugs-list at python.org > Subject: [issue2550] SO_REUSEADDR doesn't have the same semantics on > Windows as on Unix > > Trent Nelson added the comment: > > Invested quite a few cycles on this issue last night. The more time I > spent on it, the more I became convinced that every single test working > with sockets should be changed in one fell swoop in order to facilitate > (virtually unlimited) parallel test execution without fear of port > conflicts. > > I've attached a second patch, trunk.2550-2.patch, which is my progress > so far on doing just this. The main changes can be expressed by the > following two points: > > a) do whatever it takes in network-oriented tests to ensure > unique ports are obtained (relying on the bind_port() and > find_unused_port() methods exposed by test_support) > > b) never, ever, ever call SO_REUSEADDR on a socket from a test; > because we're putting so much effort into obtaining a unique > port, this should never be necessary -- in the rare cases that > our attempts to obtain a unique port fail, then we absolutely > should fail with EADDRINUSE, as the ability to obtain a unique > port for the duration of a client/server test is an invariant > that we *must* be able to depend upon. If the invariant is > broken, fail immediately (don't mask the problem with > SO_REUSEADDR). > > With this patch applied, I can spawn a handful of Python processes and > run the entire test suite (without -r, ensuring all tests are run in > the same order, which should encourage port conflicts (if there were > any)) without any errors*. Doing that now is completely and utterly > impossible. > > [*] Well, almost without error. All the I/O related tests that try and > open @test fail. > > I believe there's still outstanding work to do with this patch with > regards to how the intracacies of SO_REUSEADDR and SO_EXCLUSIVEADDRUSE > should be handled in the rest of the stdlib. I'm still thinking about > the best approach for this. However, the patch as it currently stands > is still quite substantial so I wanted to get it out sooner rather than > later for review. > > (I'll forward this to python-dev@ to try and encourage more eyes from > people with far more network-fu than I.) > > Added file: http://bugs.python.org/file9980/trunk.2550-2.patch > > __________________________________ > Tracker > > __________________________________ > _______________________________________________ > Python-bugs-list mailing list > Unsubscribe: > http://mail.python.org/mailman/options/python-bugs-list/tnelson%40onresolve.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080408/537bb057/attachment-0001.htm From greg.ewing at canterbury.ac.nz Wed Apr 9 01:36:20 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 09 Apr 2008 11:36:20 +1200 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> Message-ID: <47FC0174.3070605@canterbury.ac.nz> zooko wrote: > 1. You can't conveniently install eggs into a non-system directory, > such as ~/my-python-stuff. > > 2. If you allow even a single egg to be installed into your > PYTHONPATH, it will change the semantics of your PYTHONPATH. I discovered another annoyance with eggs the other day -- it seems that tracebacks referring to egg-resident files contain the pathname of some temporary directory that existed when the egg was being packaged, rather than the one it actually exists in at run time. -- Greg From tnelson at onresolve.com Wed Apr 9 02:01:26 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Tue, 8 Apr 2008 17:01:26 -0700 Subject: [Python-Dev] Changing all network-oriented tests to facilitate (virtually unlimited) parallel test execution [Issue#: 2550] In-Reply-To: <52dc1c820804081258r73d5491chf0a79f0f7de8c3f4@mail.gmail.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22939EEF@EXMBX04.exchhosting.com> <52dc1c820804081258r73d5491chf0a79f0f7de8c3f4@mail.gmail.com> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22BEB575@EXMBX04.exchhosting.com> Committed the patch in r62234. Hopefully the work paid off! (He says moments before all the buildbots turn red...) ________________________________ From: Gregory P. Smith [mailto:greg at krypto.org] Sent: 08 April 2008 20:58 To: Trent Nelson Cc: python-dev at python.org Subject: Re: [Python-Dev] Changing all network-oriented tests to facilitate (virtually unlimited) parallel test execution [Issue#: 2550] On Tue, Apr 8, 2008 at 5:00 AM, Trent Nelson > wrote: I've forwarding my most recent update to issue 2550 here such that the proposed patch (and in general, the approach to network-oriented test cases) can be vetted by a wider audience: http://bugs.python.org/file9980/trunk.2550-2.patch This patch works towards fixing a large proportion of the tests that were written in such a way that often leads to buildbot errors when port conflicts arise. With this patch applied, I can run many instances of the test suite in parallel and none of the network-oriented tests fail (which they do currently if you try and do this). Plenty of discussion (mostly me replying to my own comments) on the subject at: http://bugs.python.org/issue2550. Anyone have any issues with this new approach? I'm particularily interested in whether or not people disagree with the assumption I've drawn regarding never using SO_REUSEADDR in tests for TCP/IP sockets (see below). Trent. I think your assumptions are fair. Note that not using SO_REUSEADDR can reserve the port for a few minutes beyond the actual unbinding destruction of the server socket on some OSes but with tens of thousands of ports available and likely not more than 1-200 being needed by a full test suite run and how long the test suite takes to run I think that is acceptable and is not a problem we're likely to ever run into (fix it only iffwe ever do). -gps ________________________________________ From: python-bugs-list-bounces+tnelson=onresolve.com at python.org [python-bugs-list-bounces+tnelson=onresolve.com at python.org] On Behalf Of Trent Nelson [report at bugs.python.org] Sent: 08 April 2008 12:49 To: python-bugs-list at python.org Subject: [issue2550] SO_REUSEADDR doesn't have the same semantics on Windows as on Unix Trent Nelson > added the comment: Invested quite a few cycles on this issue last night. The more time I spent on it, the more I became convinced that every single test working with sockets should be changed in one fell swoop in order to facilitate (virtually unlimited) parallel test execution without fear of port conflicts. I've attached a second patch, trunk.2550-2.patch, which is my progress so far on doing just this. The main changes can be expressed by the following two points: a) do whatever it takes in network-oriented tests to ensure unique ports are obtained (relying on the bind_port() and find_unused_port() methods exposed by test_support) b) never, ever, ever call SO_REUSEADDR on a socket from a test; because we're putting so much effort into obtaining a unique port, this should never be necessary -- in the rare cases that our attempts to obtain a unique port fail, then we absolutely should fail with EADDRINUSE, as the ability to obtain a unique port for the duration of a client/server test is an invariant that we *must* be able to depend upon. If the invariant is broken, fail immediately (don't mask the problem with SO_REUSEADDR). With this patch applied, I can spawn a handful of Python processes and run the entire test suite (without -r, ensuring all tests are run in the same order, which should encourage port conflicts (if there were any)) without any errors*. Doing that now is completely and utterly impossible. [*] Well, almost without error. All the I/O related tests that try and open @test fail. I believe there's still outstanding work to do with this patch with regards to how the intracacies of SO_REUSEADDR and SO_EXCLUSIVEADDRUSE should be handled in the rest of the stdlib. I'm still thinking about the best approach for this. However, the patch as it currently stands is still quite substantial so I wanted to get it out sooner rather than later for review. (I'll forward this to python-dev@ to try and encourage more eyes from people with far more network-fu than I.) Added file: http://bugs.python.org/file9980/trunk.2550-2.patch __________________________________ Tracker > __________________________________ _______________________________________________ Python-bugs-list mailing list Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/tnelson%40onresolve.com _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/greg%40krypto.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080408/253b6341/attachment.htm From bignose+hates-spam at benfinney.id.au Wed Apr 9 03:37:07 2008 From: bignose+hates-spam at benfinney.id.au (Ben Finney) Date: Wed, 09 Apr 2008 11:37:07 +1000 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> Message-ID: <87bq4jolks.fsf@benfinney.id.au> zooko writes: > I am skeptical that prorgammers are going to be willing to use a new > database format. They already have a database -- their filesystem -- > and they already have the tools to control it -- mv, rm, and > PYTHONPATH. Many of them already hate the existence the > "easy_instlal.pth" database file, and I don't see why a new database > file would be any different. Moreover, many of us already have a database of *all* packages on the system, not just Python-language ones: the package database of our operating system. Adding another, parallel, database which needs separate maintenance, and only applies to Python packages, is not a step forward in such a situation. > They both agreed that it made perfect sense. I told one of them > about the alternate proposal to define a new database file to > contain a list of installed packages, and he sighed and rolled his > eyes and said "So they are planning to reinvent apt!". That's pretty much my reaction, too. -- \ "Contentment is a pearl of great price, and whosoever procures | `\ it at the expense of ten thousand desires makes a wise and | _o__) happy purchase." -- J. Balguy | Ben Finney From Scott.Daniels at Acm.Org Wed Apr 9 05:01:10 2008 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Tue, 08 Apr 2008 20:01:10 -0700 Subject: [Python-Dev] py3k: print function treats sep=None and end=None in an unintuitive way In-Reply-To: <200804081801.15687.alessandro.guido@gmail.com> References: <200804081522.50764.alessandro.guido@gmail.com> <47FB79B4.5020605@gmail.com> <200804081801.15687.alessandro.guido@gmail.com> Message-ID: Alessandro Guido wrote: > Nick Coghlan wrote: > Eric Smith wrote: >> Because None means 'use the default value'. You probably want: >> print('a', 'b', sep='', end='') > > I think this is a "not optimally designed" API > because you have to read the documentation to understand why Excuse me, I don't know about you, but I don't mind a language with a document to consult. I actually wasn't born understanding _any_ computer (or for that matter natural) language. 0Scott David Daniels Scott.Daniels at Acm.Org From tjreedy at udel.edu Wed Apr 9 05:47:34 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 8 Apr 2008 23:47:34 -0400 Subject: [Python-Dev] unittest's redundant assertions: assertsvs. failIf/Unlesses References: <5d44f72f0803190120n700de143m5d67b8164f7c57ae@mail.gmail.com> <644FD103-F31C-4A7E-8172-374B57BF3BC4@python.org> <20080319211556.21558.2085930557.divmod.xquotient.8213@joule.divmod.com> <5d44f72f0803191716k330e1249kce353d173fe503@mail.gmail.com> <47FB33C6.4080008@voidspace.org.uk> Message-ID: "Michael Foord" wrote in message news:47FB33C6.4080008 at voidspace.org.uk... | > Someone please open a bug for this task. | > | > | This sounds like a good compromise and I'm happy to take on the cleanup | - unless someone else beats me to it. I guess it should wait until 3.0 | final is out of the door before adding the DeprecationWarnings. I think, however, that the docs should be revised now, for 2.6/3.0. In the list of methods under TestCase Objects, the preferred method of each pair (or triplit) should be given first and the others marked as future deprecations, not recommended for new test code. The explanations of the methods should use the preferred names. So failIf/assertFalse should be reversed in order (assuming that failIf is the one to go) and the text "The inverse of the failUnless() method" should be changed to "The inverse of the assertTrue() method" (if that is the one to be kept). I would also (lesser priority) revise examples to only use the preferred names. The would make things easiest for new unittest users that are not Java refugees. tjr From pje at telecommunity.com Wed Apr 9 06:41:32 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 00:41:32 -0400 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you Message-ID: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> At 10:49 PM 4/8/2008 -0400, Stanley A. Klein wrote: >On Tue, April 8, 2008 9:37 pm, Ben Finney > wrote: > > Date: Wed, 09 Apr 2008 11:37:07 +1000 > > From: Ben Finney > > Subject: Re: [Distutils] how to easily consume just the parts of eggs > > that are good for you > > To: Distutils-Sig at Python.Org > > > > > > zooko writes: > >> eyes and said "So they are planning to reinvent apt!". > > > > That's pretty much my reaction, too. > >I have the same reaction. I'm curious. Have any of you actually read PEP 262 in any detail? I have seen precious little discussion so far that doesn't appear to be based on significant misunderstandings of either the purpose of reviving the PEP, or the mechanics of its proposed implementation. >I have tried in the past to use easy_install, but have run into problems >because there is no communication between easy_install and the rpm >database, resulting in failure of easy_install to recognize that >dependencies have already been installed using rpms. This problem doesn't exist with Python 2.5, unless you're using a platform that willfully strips out the installation information that Python 2.5 provides for these packages. >A database focused only on Python packages is highly inappropriate for >Linux systems, violates the Linux standards, and creates problems because >eggs are not coordinated with the operating system package manager. The revamp of PEP 262 is aimed at removing .egg files and directories from the process, by allowing system packagers to tell Python what files belong to them and should not be messed with. And conversely, allowing systems and installation targets *without* package managers to safely manage their Python installations. > The >way to achieve a database for Python would be to provide tools for >conversion of eggs to rpms and debs, Such tools already exist, although the conversion takes place from source distributions rather than egg distributions. >to have eggs support conformance to >the LSB and FHS, Applying LSB and FHS to the innards of Python packages makes as much sense as applying them to the contents of Java .jar files -- i.e., none. If it's unchanging data that's part of a program or library, then it's a program or library, just like static data declared in a C program or library. Whether the file extension is .py, .so, or even .png is irrelevant. From fdrake at acm.org Wed Apr 9 06:56:18 2008 From: fdrake at acm.org (Fred Drake) Date: Wed, 9 Apr 2008 00:56:18 -0400 Subject: [Python-Dev] unittest's redundant assertions: assertsvs. failIf/Unlesses In-Reply-To: References: <5d44f72f0803190120n700de143m5d67b8164f7c57ae@mail.gmail.com> <644FD103-F31C-4A7E-8172-374B57BF3BC4@python.org> <20080319211556.21558.2085930557.divmod.xquotient.8213@joule.divmod.com> <5d44f72f0803191716k330e1249kce353d173fe503@mail.gmail.com> <47FB33C6.4080008@voidspace.org.uk> Message-ID: <3FC7EDB8-A5A8-48B1-91D0-455DEC90405D@acm.org> On Apr 8, 2008, at 11:47 PM, Terry Reedy wrote: > I think, however, that the docs should be revised now, for 2.6/3.0. > In the list of methods under TestCase Objects, the preferred method > of each > pair (or triplit) should be given first and the others marked as > future > deprecations, not recommended for new test code. +1 ! This kind of information, once known, should be included in all maintained versions of the documentation so that users can build good coding habits, or have more time to adjust existing code. -Fred -- Fred Drake From tnelson at onresolve.com Wed Apr 9 12:00:52 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 9 Apr 2008 03:00:52 -0700 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47F5E6EC.6030402@timgolden.me.uk> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk> <47F3592D.3080700@timgolden.me.uk>,<47F4DE23.3020202@timgolden.me.uk> <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB27E@EXMBX04.exchhosting.com> <47F5E6EC.6030402@timgolden.me.uk> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22BEB610@EXMBX04.exchhosting.com> > -----Original Message----- > From: Tim Golden [mailto:mail at timgolden.me.uk] > Sent: 04 April 2008 09:30 > To: Trent Nelson > Cc: Python-Dev > Subject: Re: [Python-Dev] fixing tests on windows > Yes. I'm trying desperately hard to stick to a narrow remit > of getting tests to run consistently in the face of messy > file-locking semantics under Windows. I really don't want to > wade into the minor minefield of making all the tests run > with consistent temp file semantics. But I may have to. Hey Tim, any progress on this? I'd like to start working on this towards the weekend... Trent. From pje at telecommunity.com Wed Apr 9 15:00:01 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 09:00:01 -0400 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409080016.GB1761@phare.normalesup.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <20080409080016.GB1761@phare.normalesup.org> Message-ID: <20080409125953.99EE63A40D7@sparrow.telecommunity.com> At 10:00 AM 4/9/2008 +0200, Gael Varoquaux wrote: >On Wed, Apr 09, 2008 at 12:41:32AM -0400, Phillip J. Eby wrote: > > >The way to achieve a database for Python would be to provide tools for > > >conversion of eggs to rpms and debs, > > > Such tools already exist, although the conversion takes place from > > source distributions rather than egg distributions. > >What is the status of the deb backend? The only one I know is unofficial >maintained by Andrew Straw, but my information my be lagging behind. I was under the impression that there were 2 .deb tools, neither one "official" in any sense, any more than 'bdist_rpm' is really "official" for RPM-based systems. >By the way, if these tools work well, they are priceless! I haven't had need to use any of them, so I don't really know. From mail at timgolden.me.uk Wed Apr 9 15:04:57 2008 From: mail at timgolden.me.uk (Tim Golden) Date: Wed, 09 Apr 2008 14:04:57 +0100 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22BEB610@EXMBX04.exchhosting.com> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk> <47F3592D.3080700@timgolden.me.uk>, <47F4DE23.3020202@timgolden.me.uk> <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB27E@EXMBX04.exchhosting.com> <47F5E6EC.6030402@timgolden.me.uk> <87D3F9C72FBF214DB39FA4E3FE618CDC6E22BEB610@EXMBX04.exchhosting.com> Message-ID: <47FCBEF9.9080308@timgolden.me.uk> Trent Nelson wrote: > >> -----Original Message----- >> From: Tim Golden [mailto:mail at timgolden.me.uk] >> Sent: 04 April 2008 09:30 >> To: Trent Nelson >> Cc: Python-Dev >> Subject: Re: [Python-Dev] fixing tests on windows > >> Yes. I'm trying desperately hard to stick to a narrow remit >> of getting tests to run consistently in the face of messy >> file-locking semantics under Windows. I really don't want to >> wade into the minor minefield of making all the tests run >> with consistent temp file semantics. But I may have to. > > Hey Tim, any progress on this? I'd like to start working on this towards the weekend... Thanks for the prod, Trent. In short, yes, I spent some time on this over the weekend but haven't had time since. I found myself becoming more and more worried at the amount I had to change, especially given rumblings on the list concerning not changing the tests if possible. The possibilities I've been pursuing are these: 1) Leave TESTFN as-is and make the .unlink more robust 2) Leave TESTFN static but instead of "./@TEST" use the tempfile module to generate a random name in whatever temp dir applies for the test system. And make the .unlink more robust. 3) Drop TESTFN and have a function testfn () which generates a new tempdir filename every time. And make the .unlink more robust. Each of these approaches has had its problems, and I'm additionally hampered by the fact that I don't run *nix or OSX (or any other platform). I spent far more time than I really had setting up a VMWare player with Ubuntu but I haven't got it to the point of building the tests which won't run under Windows. Pros and cons: 1) Running with a fairly aggressive directory watcher, this doesn't do enough to make the tests worth. Not worth it. 2) A bit better, especially since the betting is that most of the suspect apps (local search engines etc.) won't index %TEMP%. But... certain tests run into problems because they assume that the filename of TESTFN has no escape-needy characters such as the backslash. Need to update at least those tests. 3) [This is what I was trying over the weekend]. Loads of work and quite intrusive. I'll try to get hold of my patch this evening and mail it over to you. Part of the problem is working out where each test is expecting TESTFN to be the same file each time and where it doesn't care. I was heartened to see that you'd gone ahead with the -- almost as invasive -- socket test changes. If you'd like to take on what I've got so far, or simply to adopt your own strategy, please feel free. TJG From tnelson at onresolve.com Wed Apr 9 16:12:58 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 9 Apr 2008 07:12:58 -0700 Subject: [Python-Dev] Next monthly sprint/bugfix day? Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22BEB730@EXMBX04.exchhosting.com> Hi, Is there another online sprint/bugfix day in the pipeline? If not, can there be? ;-) Trent. From tnelson at onresolve.com Wed Apr 9 16:44:50 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 9 Apr 2008 07:44:50 -0700 Subject: [Python-Dev] fixing tests on windows In-Reply-To: <47FCBEF9.9080308@timgolden.me.uk> References: <47F252CE.1020702@timgolden.me.uk> <47F25F2E.4070408@timgolden.me.uk> <52dc1c820804012326y1f8df17ega2955b8c220785ce@mail.gmail.com> <47F33DE7.3030702@timgolden.me.uk> <47F3592D.3080700@timgolden.me.uk>,<47F4DE23.3020202@timgolden.me.uk> <87D3F9C72FBF214DB39FA4E3FE618CDC6E18FBB27E@EXMBX04.exchhosting.com> <47F5E6EC.6030402@timgolden.me.uk> <87D3F9C72FBF214DB39FA4E3FE618CDC6E22BEB610@EXMBX04.exchhosting.com> <47FCBEF9.9080308@timgolden.me.uk> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22BEB7A8@EXMBX04.exchhosting.com> > Thanks for the prod, Trent. In short, yes, I spent some time > on this over the weekend but haven't had time since. I found > myself becoming more and more worried at the amount I had to > change, especially given rumblings on the list concerning not > changing the tests if possible. > I was heartened to see that you'd gone ahead with the -- > almost as invasive -- socket test changes. If you'd like to > take on what I've got so far, or simply to adopt your own > strategy, please feel free. I don't hold any weight to rumblings regarding "not changing the tests if possible". The tests need to change. They were never written with the intent to run in parallel without any resource contention. That's an important goal, now. Please send over any patches you've got, everything will be useful! Now that the dust has (sort of) settled on the network-oriented tests, I'm ready to start attacking the I/O ones. Trent. From guido at python.org Wed Apr 9 18:37:50 2008 From: guido at python.org (Guido van Rossum) Date: Wed, 9 Apr 2008 09:37:50 -0700 Subject: [Python-Dev] unittest's redundant assertions: assertsvs. failIf/Unlesses In-Reply-To: <3FC7EDB8-A5A8-48B1-91D0-455DEC90405D@acm.org> References: <5d44f72f0803190120n700de143m5d67b8164f7c57ae@mail.gmail.com> <644FD103-F31C-4A7E-8172-374B57BF3BC4@python.org> <20080319211556.21558.2085930557.divmod.xquotient.8213@joule.divmod.com> <5d44f72f0803191716k330e1249kce353d173fe503@mail.gmail.com> <47FB33C6.4080008@voidspace.org.uk> <3FC7EDB8-A5A8-48B1-91D0-455DEC90405D@acm.org> Message-ID: On Tue, Apr 8, 2008 at 9:56 PM, Fred Drake wrote: > On Apr 8, 2008, at 11:47 PM, Terry Reedy wrote: > > I think, however, that the docs should be revised now, for 2.6/3.0. > > In the list of methods under TestCase Objects, the preferred method > > of each > > pair (or triplit) should be given first and the others marked as > > future > > deprecations, not recommended for new test code. > +1 ! > > This kind of information, once known, should be included in all > maintained versions of the documentation so that users can build good > coding habits, or have more time to adjust existing code. Great point, Fred! This sounds like a general strategy that we might employ more often -- when deprecating something, the first step ought to be to document the best practice. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From amcnabb at mcnabbs.org Wed Apr 9 18:24:22 2008 From: amcnabb at mcnabbs.org (Andrew McNabb) Date: Wed, 9 Apr 2008 10:24:22 -0600 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <87bq4jolks.fsf@benfinney.id.au> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> <87bq4jolks.fsf@benfinney.id.au> Message-ID: <20080409162422.GB2442@mcnabbs.org> On Wed, Apr 09, 2008 at 11:37:07AM +1000, Ben Finney wrote: > > Moreover, many of us already have a database of *all* packages on the > system, not just Python-language ones: the package database of our > operating system. Adding another, parallel, database which needs > separate maintenance, and only applies to Python packages, is not a > step forward in such a situation. I agree with you completely. There are three things I can see myself wanting from an easy_install command: 1) To download and install an RPM/dpkg to a system-wide location. The package is then removable using the normal system package manager. 2) To download the source of a package. 3) To install a package to my home directory. I would much rather that this hypothetical easy_install did `rpm --prefix "$HOME/.local" -ivh some_package.rpm` than have a Python-specific database. > > They both agreed that it made perfect sense. I told one of them > > about the alternate proposal to define a new database file to > > contain a list of installed packages, and he sighed and rolled his > > eyes and said "So they are planning to reinvent apt!". > > That's pretty much my reaction, too. Me, too. -- Andrew McNabb http://www.mcnabbs.org/andrew/ PGP Fingerprint: 8A17 B57C 6879 1863 DE55 8012 AB4D 6098 8826 6868 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20080409/deba6c62/attachment.pgp From p.f.moore at gmail.com Wed Apr 9 19:23:50 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Apr 2008 18:23:50 +0100 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> Message-ID: <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> On 09/04/2008, Stanley A. Klein wrote: > IMHO, the main system without a package manager is Windows. A reasonable > way to deal with Windows would be to create a package manager for it that > could be used by Python and anyone else who wanted to use it. The package > manager could establish a file hierarchy similar to the Unix FHS and > install files appropriately, except for what is needed to satisfy the > Windows OS. That would probably go a long way to addressing the issues > being discussed here. This is primarily a Windows problem, not a Python > problem. Windows does have a package manager - the add/remove programs application. It's extremely limited, and doesn't make any attempt at doing dependency resolution, certainly - but that's a separate issue. I don't know if you use Windows (as in, develop programs using Python on Windows). If you do, then I'd be interested in your views on bdist_wininst and bdist_msi installers, and how they fit into the setuptools/egg environment, particularly with regard to the package manager you are proposing. If you don't use Windows, then I don't see how you can usefully comment. Personally, as I've said before, I don't have a problem with a Python-only package manager, as long as it replaces or integrates bdist_wininst and bdist_msi. Having two package managers is far worse than having none - and claiming that add/remove programs "isn't a package manager" is just ignoring reality (if it isn't, then why do bdist_wininst and bdist_msi exist?). Are the Linux users happy with having a Python package manager that ignores RPM/apt? Why should Windows users be any happier? Sorry - I'm feeling a little grumpy. I've read one too many "Windows is so broken that people who use it obviously don't care about doing things right" postings this week :-( Paul. From zooko at zooko.com Wed Apr 9 19:30:55 2008 From: zooko at zooko.com (zooko) Date: Wed, 9 Apr 2008 10:30:55 -0700 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> Message-ID: <593FD433-F1A0-4445-AA9D-BD7B75DE8125@zooko.com> On Apr 8, 2008, at 9:41 PM, Phillip J. Eby wrote: > > I'm curious. Have any of you actually read PEP 262 in any detail? I read it, though not in fine detail. I didn't write that you are planning to reinvent apt. I wrote that when programmers hear about this PEP they exclaim "They are planning to reinvent apt!". That is a matter of perception and marketing -- the value that I want to gain from Python packages is the value of a critical mass of good programmers using compatible tools for code re-use. If a lot of programmers hate an idea, then it doesn't matter what the details are -- it isn't going to provide this value to me. I think part of our disagreement is that we are talking about two overlapping use cases: programmer and sysadmin (and "end user" which is much like sysadmin). I am not, at this time, interested in the sysadmin use case. As I've mentioned, my sysadmin needs are currently well satisfied by apt (and sometimes by GNU stow), and my fellow sysadmins with whom I work are absolutely not going to relax their "apt-only policy" for the forseeable future, so I cannot use such a tool unless it is named "apt" and written by Debian/Ubuntu. On the other hand I am very interested in the programmer use case, because setuptools/easy_install already works pretty well for that, and we are already very close to achieving a critical mass of good programmers. Recently several more packages that my project [1] relies on have become easy_installable -- Twisted, pywin32 (thanks to you, PJE), and foolscap -- and several more have had bugfixes which make them work better with easy_install/setuptools -- Nevow and zope.interface -- and there are some patches in the queue to make another one compatible with setuptools -- pyflakes [2, 3]. So setuptools/easy_install is already (slowly) winning. I want to accelerate that process by reducing the degree to which it is incompatible, inconvenient, or objectionable to other programmers. PEP 262 sounds like a non-starter to me because 1. It appears to be backwards-incompatible with setuptools/ easy_install/eggs, thus losing all the recently gained cooperation that I mentioned in the previous paragraph, and 2. It defines a new database file, where I would prefer either: a. Doing away with database files entirely and relying on the filesystem alone to hold that information, or b. Continuing to use the current ".pth" database file format, possibly improved by having native support for .pth files in the Python import machinery. 3. Because of #2, it triggers programmers to exclaim "They are planning to reinvent apt!", thus making it unlikely that the new proposal will recapture the cooperation that setuptools has already (slowly) gained. I'm sorry, PJE -- I know it must be frustrating to you to have your proposal criticized on social rather than technical grounds -- but social benefits are what I am seeking right now. Perhaps PEP 262 and my proposal are not actually alternatives, but are complementary. I do not object to Python maintaining a database of installed packages for itself (although I cannot *rely* upon such behavior, not least because I will be maintaining backwards compatibility with Python 2.4 for at least the next several years, and with Python 2.5 for at least the next several years after that). What I want is for the already implemented, tested, and deployed code- re-use features of setuptools/easy_install to be more widely accepted. This is best and most easily achieved by fixing the two most frequent objections to setuptools/easy_install: 1. That you can't conveniently install into an arbitrary directory, and 2. that it subverts the meaning of your PYTHONPATH. Regards, Zooko [1] http://allmydata.org/source/tahoe/trunk/docs/install.html [2] http://divmod.org/trac/ticket/2535 [3] http://divmod.org/trac/ticket/2048 From zooko at zooko.com Wed Apr 9 21:30:17 2008 From: zooko at zooko.com (zooko) Date: Wed, 9 Apr 2008 12:30:17 -0700 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <47FC0174.3070605@canterbury.ac.nz> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> <47FC0174.3070605@canterbury.ac.nz> Message-ID: <2869C14C-A204-4A71-9C5E-8E202E1DC42D@zooko.com> On Apr 8, 2008, at 4:36 PM, Greg Ewing wrote: > > I discovered another annoyance with eggs the other day -- it > seems that tracebacks referring to egg-resident files contain the > pathname of some temporary directory that existed when the egg > was being packaged, rather than the one it actually exists in > at run time. Brian Warner and I discovered that issue yesterday, too. We determined that if you install the egg (with easy_install or with a setuptools-powered ./setup.py install) in unzipped form then the source file names get rewritten so that your stack traces come with source lines. If you have a package which requires stack traces to come with source lines, then you could pass "zip_safe=False" to the call to setup(). I would prefer that zip_safe=False were the default and that either the producer or the consumer of a package had to specifically choose zip_safe=True in order to install eggs in zipped form. I've opened a ticket on my setuptools trac: http://allmydata.org/trac/setuptools/ticket/4 Regards, Zooko From pje at telecommunity.com Wed Apr 9 21:40:53 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 15:40:53 -0400 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> Message-ID: <20080409194044.1A4E03A406A@sparrow.telecommunity.com> At 11:52 AM 4/9/2008 -0400, Stanley A. Klein wrote: >However, are you implying that the installation information for Python egg >packages accesses and coordinates with the rpm database? Yes, when the information isn't stripped out. Try a more recent Fedora. >IMHO, the main system without a package manager is Windows. You're ignoring shared environments and development environments. (Not to mention Mac OS.) > A reasonable >way to deal with Windows would be to create a package manager for it that >could be used by Python and anyone else who wanted to use it. Let us know when you've finished it, along with the one for Mac OS. :) Of course this still won't do anything for shared environments and development environments. >You are talking here about bdist_rpm and not about a tool that would take >a Python package distributed as an egg file and convert the egg to an rpm >or a deb. Unfortunately, some Python packagers are beginning to limit >their focus only to egg distribution. That creates a problem for users >who have native operating system package management. That is indeed a problem -- but it's a social one, not a technical one. It's trivial for the publisher of an egg to change their command line from "setup.py bdist_egg upload" to "setup.py sdist bdist_egg upload", as soon as their users (politely) request that they do so. > > Applying LSB and FHS to the innards of Python packages makes as much > > sense as applying them to the contents of Java .jar files -- i.e., > > none. If it's unchanging data that's part of a program or library, > > then it's a program or library, just like static data declared in a C > > program or library. Whether the file extension is .py, .so, or even > > .png is irrelevant. > >The FHS defines places to put specific kinds of files, such as command >scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation >(/usr/share/doc/package-name), and configuration files (/etc). There are >several kinds of files identified and places defined to put them. >Distribution by eggs has a tendency to scoop up all of those files and put >them in /usr/lib/python/site-packages, regardless of where they belong. Eggs don't include documentation or configuration files, and they install scripts in script directories, so I don't get what you're talking about here. For any other data that a package accesses at runtime, my earlier comments apply. >Having eggs support conformance to FHS would mean recognizing and tagging >the relevant files. A tool for converting eggs to rpms or debs would >essentially reformat the egg to rpm or deb and put files where they >belong. No, because such files as you describe don't exist. If you think they do, then either you have misunderstood the nature of the files in question, or the developer has incorrectly placed non-runtime files in their installation tree. From pje at telecommunity.com Wed Apr 9 21:57:13 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 15:57:13 -0400 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you Message-ID: <20080409195704.32E5F3A406A@sparrow.telecommunity.com> At 10:30 AM 4/9/2008 -0700, zooko wrote: >PEP 262 sounds like a non-starter to me because > >1. It appears to be backwards-incompatible with setuptools/ >easy_install/eggs, thus losing all the recently gained cooperation >that I mentioned in the previous paragraph, and No. It provides a forward-compatibility path for the distutils, so that easy_install doesn't have to install things in .egg format in the future. There's no compatibility breakage at all. >2. It defines a new database file, No, it defines *files*. One file per installed distribution, containing (among other things) an installation manifest. >where I would prefer either: > a. Doing away with database files entirely and relying on the >filesystem alone to hold that information, or ...which is what PEP 262 *does*. Unfortunately, PEP 262's title is bad for marketing, as you've effectively pointed out. It would be better titled something "package installation manifests" or "package contents files", or something of that sort. > b. Continuing to use the current ".pth" database file format, >possibly improved by having native support for .pth files in the >Python import machinery. These mechanisms are orthogonal to this issue. >3. Because of #2, it triggers programmers to exclaim "They are >planning to reinvent apt!", thus making it unlikely that the new >proposal will recapture the cooperation that setuptools has already >(slowly) gained. Yeah, we need a new name. Everybody is going off of "database of installed packages" and thinking "apt", because they aren't paying any closer attention. However, given that we are discussing this on Python-Dev and distutils-sig, I do think it's reasonable to expect (if perhaps not reasonable to receive) that people discussing the PEP have *read* the freaking PEP first, prior to trashing it or offering alternatives. And it's not like I'm personally offended or anything -- I didn't even write the PEP in question. But what's the point of having PEPs if people read nothing but their titles? We could just delete everything but PEP 0. :) >Perhaps PEP 262 and my proposal are not actually alternatives, but >are complementary. As I've already pointed out, your proposal does not address multiple installed versions of a package, and I see no sane way to modify it to do so. >What I want is for the already implemented, tested, and deployed >code- re-use features of setuptools/easy_install to be more widely >accepted. This is best and most easily achieved by fixing the two >most frequent objections to setuptools/easy_install: 1. That you >can't conveniently install into an arbitrary directory, and 2. that >it subverts the meaning of your PYTHONPATH. As I've already stated, the only way for these problems to be fixed is for easy_install to not install files in .egg form -- which also solves the general objection to using .eggs in the first place. And the only way to do that, is to have a way to keep track of what files are installed. Rather than have easy_install come up with its own way of doing that, I would prefer to share a standard with the distutils. Hence, the PEP discussion. For earlier versions of Python, it will still be possible to install and uninstall with setuptools using this approach. You just won't be able to uninstall pure distutils-based packages, unless you installed them using easy_install. Meanwhile, it has occurred to me that the easiest way of handling compatibility is not to require that other packaging tools mark their files for non-removability, but simply not allow easy_install to remove or overwrite anything that *isn't* claimed by a manifest. In that way, easy_install would be immediately usable in the new mode, without any updates to Python or to system packaging tools. From pje at telecommunity.com Wed Apr 9 22:06:14 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 16:06:14 -0400 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <2869C14C-A204-4A71-9C5E-8E202E1DC42D@zooko.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> <47FC0174.3070605@canterbury.ac.nz> <2869C14C-A204-4A71-9C5E-8E202E1DC42D@zooko.com> Message-ID: <20080409200605.DD2783A406A@sparrow.telecommunity.com> At 12:30 PM 4/9/2008 -0700, zooko wrote: >On Apr 8, 2008, at 4:36 PM, Greg Ewing wrote: > > > > I discovered another annoyance with eggs the other day -- it > > seems that tracebacks referring to egg-resident files contain the > > pathname of some temporary directory that existed when the egg > > was being packaged, rather than the one it actually exists in > > at run time. > >Brian Warner and I discovered that issue yesterday, too. We >determined that if you install the egg (with easy_install or with a >setuptools-powered ./setup.py install) in unzipped form then the >source file names get rewritten so that your stack traces come with >source lines. Are you using Python 2.5? As of 2.5, the linecache module should correctly read the source line from the present location of the source file the module was loaded from, regardless of the file name specified in the traceback. >If you have a package which requires stack traces to come with source >lines, then you could pass "zip_safe=False" to the call to setup(). > >I would prefer that zip_safe=False were the default and that either >the producer or the consumer of a package had to specifically choose >zip_safe=True in order to install eggs in zipped form. A better fix would be for Python to use relative paths in co_filename, as this would fix similar problems that occur whenever you move .pyc/.pyo files around. From sergiodj at linux.vnet.ibm.com Wed Apr 9 22:44:14 2008 From: sergiodj at linux.vnet.ibm.com (=?ISO-8859-1?Q?S=E9rgio?= Durigan =?ISO-8859-1?Q?J=FAnior?=) Date: Wed, 09 Apr 2008 17:44:14 -0300 Subject: [Python-Dev] Compiling Python on a biarch (PPC/PPC64) Message-ID: <1207773854.5788.23.camel@miki> Hi everybody :-) I'm running a PPC64 biarch here (i.e., a 64-bit kernel with 32-bit userspace), and trying to compile Python on it. Well, as I have a GCC compiler that generates natively 32-bit executables, everything goes well when I *don't* use any specific 64-bit flag (./configure --enable-shared && make && make test && make install). My problem happens when I try to compile Python for PPC64 using some configure flags and environment variables, like --target and CFLAGS. To be more specific, that's what I tried: ./configure --enable-shared --target=powerpc64-unknown-linux CFLAGS='-m64' The configure runs fine. But when I do a "make": sergio at elm3b188 ~/work/w/Python-2.5.2 $ make gcc -pthread -c -fno-strict-aliasing -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I. -IInclude -I./Include -fPIC -DPy_BUILD_CORE -o Modules/python.o ./Modules/python.c In file included from Include/Python.h:57, from ./Modules/python.c:3: Include/pyport.h:761:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)." make: *** [Modules/python.o] Error 1 Well, now starts the part where I try to understand things, so please correct me if I'm wrong ;-). After some investigation, I thought that the error was the (absence of) detection of the building target, so I wrote a little piece of code in configure.in which detects PPC64 in the $target variable and appends to $BASECFLAGS the string '-m64'. It seems to be more than that, though (because that approach didn't work completely)... Can anyone give me a hand on this? As far as I could investigate, I must change setup.py in order to get this issue fixed, right? Thanks in advance, -- S?rgio Durigan J?nior Linux on Power Toolchain - Software Engineer Linux Technology Center - LTC IBM Brazil From jimjjewett at gmail.com Wed Apr 9 23:15:16 2008 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 9 Apr 2008 17:15:16 -0400 Subject: [Python-Dev] windows (was: how to easily consume just the parts of eggs that are good for you) Message-ID: > Are the Linux users happy with having a Python > package manager that ignores RPM/apt? Why > should Windows users be any happier? Because, as you noted, the add/remove programs application is severely limited. > I've read one too many "Windows is so broken > that people who use it obviously don't care about > doing things right" postings this week I'm honestly not sure that such fine-grained control is the right user interface, particularly for a non-shared system. But even if it were, Windows doesn't really have it, and it isn't so valuable that a solution which works only for python could do much better than the existing 3rd-party setup tools. As a windows user, I don't want python packages showing up in the add/remove programs list, because it won't help me, and will make the few times I do use that tool even more awkward. That said, I agree that if python does package management, offering windows users the choice of using that application is probably a good idea. The catch is that "package managers" seem to offer far more fine-grained power (even without dependency resolution) than windows. Duplicating this would add lots of complexity just for windows -- and still might not be all that useful. I'm already used to looking for an uninstall.exe in the directory of anything I can actually uninstall, and accepting that most things just don't go away cleanly. As a programmer, this feels wrong, but ... it is probably a good tradeoff for the time I don't want to spend maintaining things. If I really wanted a fancy tool that took care of dependencies and alternate versions, I would be willing to run something python-specific, or to treat each package as a subcomponent that I managed through "Change an existing program" applied to python. But realistically, I don't see such a tool being used often enough to justify inclusion in the core. -jJ From pje at telecommunity.com Wed Apr 9 23:25:31 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 17:25:31 -0400 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <12121.207.188.248.157.1207773812.squirrel@www.cpcug.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <12121.207.188.248.157.1207773812.squirrel@www.cpcug.org> Message-ID: <20080409212522.9CDE13A409E@sparrow.telecommunity.com> At 04:43 PM 4/9/2008 -0400, Stanley A. Klein wrote: >I don't understand what you mean by "shared environments and development > environments". I mean that in a shared or development environment, a system packager isn't useful, since it expects things to live in *one* place, and usually to have only one *version*, as well. >I agree that we are dealing with a combination of technical and social >issues here. However, I think it takes a lot more understanding for a >publisher to get everything straight. If they provide you with the source distribution, you can make any sort of package you want. > > Eggs don't include documentation or configuration files, and they > > install scripts in script directories, so I don't get what you're > > talking about here. For any other data that a package accesses at > > runtime, my earlier comments apply. > > > >But rpms and debs do include these files, plus manual pages, localization >files and a lot of other ancillary stuff. ...just one of the many reasons that eggs are not a replacement for rpms and debs. :) >Most of the Python tarballs I have downloaded have all kinds of files in >their installation trees. This is a major pain in the you-know-what for >someone trying to use bdist_rpm and get proper, FHS-compliant rpms. If >eggs are supposed to be strictly runtime files, I think very few >developers actually understand that. Better yet, how do you define what >should be included in an installation? It sounds like the egg concept >doesn't include several kinds of files that rpm and deb would include in >an installation. I think that may be an important issue here. It would be, if .eggs were a packaging format, rather than a binary distribution/runtime format. Remember "eggs are to Python as jars are to Java" -- a Java .jar doesn't contain documentation either, unless it's needed at runtime. Same for configuration files. They're not system packages, in other words. The assumption that they are is another marketing failure, due to conflation of "package == distribution of python code" and "package == thing you manage with a system packager". People see, "I put my package in an .egg" and think it's the latter definition, when it's barely even the former. :) From martin at v.loewis.de Wed Apr 9 23:27:49 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 09 Apr 2008 23:27:49 +0200 Subject: [Python-Dev] Compiling Python on a biarch (PPC/PPC64) In-Reply-To: <1207773854.5788.23.camel@miki> References: <1207773854.5788.23.camel@miki> Message-ID: <47FD34D5.2060607@v.loewis.de> > Can anyone give me a hand on this? As far as I could > investigate, I must change setup.py in order to get this issue fixed, > right? Wrong. I didn't read your entire message, but this is definitely wrong. setup.py isn't invoked until python itself is built, and Modules/python.c is part of python itself. Regards, Martin From zooko at zooko.com Thu Apr 10 00:11:01 2008 From: zooko at zooko.com (zooko) Date: Wed, 9 Apr 2008 15:11:01 -0700 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409125953.99EE63A40D7@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <20080409080016.GB1761@phare.normalesup.org> <20080409125953.99EE63A40D7@sparrow.telecommunity.com> Message-ID: <943A6CC1-6A89-4761-AB7A-332DCDCC7F81@zooko.com> On Apr 9, 2008, at 6:00 AM, Phillip J. Eby wrote: >> >> By the way, if these tools work well, they are priceless! > > I haven't had need to use any of them, so I don't really know. They are easydeb [1] and stddeb [2]. The former appears to be incomplete and unmaintained. The latter appears to be usable, but somewhat incomplete -- substantial manual labor is required to use it successfully, as documented by my programming partner Brian Warner in this ticket: [3]. Regards, Zooko [1] http://easy-deb.sourceforge.net/ [2] http://stdeb.python-hosting.com/ [3] http://allmydata.org/trac/tahoe/ticket/251 From zooko at zooko.com Thu Apr 10 00:20:30 2008 From: zooko at zooko.com (zooko) Date: Wed, 9 Apr 2008 15:20:30 -0700 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409194044.1A4E03A406A@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> Message-ID: <390014FF-5088-42C1-8213-17D0B25D2FFB@zooko.com> On Apr 9, 2008, at 12:40 PM, Phillip J. Eby wrote: >> >> You are talking here about bdist_rpm and not about a tool that >> would take >> a Python package distributed as an egg file and convert the egg to >> an rpm >> or a deb. Unfortunately, some Python packagers are beginning to >> limit >> their focus only to egg distribution. That creates a problem for >> users >> who have native operating system package management. > > That is indeed a problem -- but it's a social one, not a technical > one. It's trivial for the publisher of an egg to change their > command line from "setup.py bdist_egg upload" to "setup.py sdist > bdist_egg upload", as soon as their users (politely) request that > they do so. In general, it would be good if eggs came with .py files by default instead of .pyc files. I've opened a ticket on my setuptools trac about this proposal: http://allmydata.org/trac/setuptools/ticket/5 # binary eggs should come with .py files by default, rather than .pyc files Regards, Zooko From p.f.moore at gmail.com Thu Apr 10 00:46:19 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Apr 2008 23:46:19 +0100 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> Message-ID: <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> On 09/04/2008, Stanley A. Klein wrote: > I think you raise an interesting issue: What is a package manager? My (very simplistic) answer is that it's whatever someone uses to manage packages. What level of functionality it has is irrelevant, as long as it suits an individual's way of working. > I don't think I ever said that Windows is broken in the area of package > management. I hope I didn't say you had - but it is an often-raised point, and it does grate on me (as by implication, it says that what I do isn't "real" package management). It's just another flavour of the old Windows vs Linux OS wars, which I don't want to participate in :-) > My own experience is that the files of Windows programs tend > to be put in a directory devoted to the program, rather than put in > directories with other files having similar purposes. At one time, the > default location in Windows for word processing files was even in a > sub-directory of the word processing program. That changed to having a > form of user home directory, but it didn't change much for the program > files themselves. Unix/Linux puts the files in specific areas of the file > system having functional commonality. One could almost say that the > Windows default approach to structuring its filesystem avoids or minimizes > the need for package management. Agreed. The minimal package manager on Windows is arguably reasonable precisely because the standard layout doesn't require much more. On the other hand, Microsoft has a bad habit of changing their "standards", and in particular Vista changes a lot of things. But that's very off-topic, so I'll avoid going into detail here. > I repeat that this issue mainly arises because Windows doesn't have the > same kind of filesystem structure (and therefore the need for package > management) that other systems have. I don't know what Windows add/remove > programs function does, but all it might do is to run the executable to > install packages and record the installation (as was previously done by > third party programs) to facilitate clean removal. Precisely. I could argue that the Linix filesystem structure is over complex, and causes the need for complex package managers, precisely because it's impossible to manually keep track of what file is owned by what package. But this way lies OS wars, so I won't make a major point of this, but just ask that people consider it as a possibility. (I believe that Mac OS X goes for an even simpler structure - applications store *everything* in the one directory, so that install/uninstall is simply a directory copy/remove. I won't comment on what that proves...) > Unless you can perform > more of the other functions I listed above, I doubt I would call > add/remove a package manager. OK. I would, though, as I don't really want much more. Later, you said: > I just don't do Windows. :-) And you've been pretty clear that you're looking for information rather than trying to explain how you think Windows should work. But many people aren't so balanced in their views, and that makes it hard to have a discussion - I'd suggest that people without Windows experience leave the Windows side to the Windows people, and accept that when they say "XXX won't work for us", that the statement is probably true. I find this whole discussion hugely confusing, because a lot of people are stating opinions about environments which it seems they don't use, or know much about. I don't know how to avoid this, but it does make it highly unlikely that any practical progress will get made. Paul. From p.f.moore at gmail.com Thu Apr 10 00:48:57 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Apr 2008 23:48:57 +0100 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409212522.9CDE13A409E@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <12121.207.188.248.157.1207773812.squirrel@www.cpcug.org> <20080409212522.9CDE13A409E@sparrow.telecommunity.com> Message-ID: <79990c6b0804091548p533e4dear5fb500341e4b87af@mail.gmail.com> On 09/04/2008, Phillip J. Eby wrote: > It would be, if .eggs were a packaging format, rather than a binary > distribution/runtime format. > > Remember "eggs are to Python as jars are to Java" -- a Java .jar > doesn't contain documentation either, unless it's needed at > runtime. Same for configuration files. And yet, Java doesn't have an equivalent of easy_install for jar files, to my knowledge. If Python had eggs but no easy_install, maybe this whole discussion wouldn't be taking place. (I know I personally like the idea of egg-as-jar-file, but *hate* the idea of egg-as-dependency-handling-tool-and-everything-else). Paul. From bignose+hates-spam at benfinney.id.au Thu Apr 10 00:47:06 2008 From: bignose+hates-spam at benfinney.id.au (Ben Finney) Date: Thu, 10 Apr 2008 08:47:06 +1000 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> Message-ID: <87iqyqir2t.fsf@benfinney.id.au> "Stanley A. Klein" writes: > IMHO, the main system without a package manager is Windows. AFAICT the MacOS platform also lacks in this area. > A reasonable way to deal with Windows would be to create a package > manager for it that could be used by Python and anyone else who > wanted to use it. [...] This is primarily a Windows problem, not a > Python problem. I'd rephrase this as: If you *must* re-invent package management for those legacy systems without it, please *don't* make it specific to Python. -- \ ?The right to search for truth implies also a duty; one must | `\ not conceal any part of what one has recognized to be true.? | _o__) ?Albert Einstein | Ben Finney From p.f.moore at gmail.com Thu Apr 10 00:52:08 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Apr 2008 23:52:08 +0100 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <87iqyqir2t.fsf@benfinney.id.au> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <87iqyqir2t.fsf@benfinney.id.au> Message-ID: <79990c6b0804091552h4ee7f9f6t5512ed5f5aa70d2a@mail.gmail.com> On 09/04/2008, Ben Finney wrote: > "Stanley A. Klein" writes: > > A reasonable way to deal with Windows would be to create a package > > manager for it that could be used by Python and anyone else who > > wanted to use it. [...] This is primarily a Windows problem, not a > > Python problem. > > I'd rephrase this as: If you *must* re-invent package management for > those legacy systems without it, please *don't* make it specific to > Python. And I would say that Windows doesn't have a problem. Are any Windows users proposing building a package management system for Windows (Python-specific or otherwise)? It's a genuine question - is this something that Windows users are after, or is it just Linux users trying to show Windows users what they are missing? (BTW, it's unreasonable to call Windows "legacy" - whatever that word means, Windows ain't it (yet...)) Paul. From pje at telecommunity.com Thu Apr 10 01:12:55 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 19:12:55 -0400 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <390014FF-5088-42C1-8213-17D0B25D2FFB@zooko.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <390014FF-5088-42C1-8213-17D0B25D2FFB@zooko.com> Message-ID: <20080409231247.7F3A23A406A@sparrow.telecommunity.com> At 03:20 PM 4/9/2008 -0700, zooko wrote: >I've opened a ticket on my setuptools trac about this proposal: > >http://allmydata.org/trac/setuptools/ticket/5 # binary eggs should >come with .py files by default, rather than .pyc files Filling your tracker with already-rejected proposals isn't likely to encourage me to look at it, especially when I've personally rejected them to you in IRC. That goes for your ticket #4 as well. From pje at telecommunity.com Thu Apr 10 01:16:20 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 19:16:20 -0400 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <79990c6b0804091548p533e4dear5fb500341e4b87af@mail.gmail.co m> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <12121.207.188.248.157.1207773812.squirrel@www.cpcug.org> <20080409212522.9CDE13A409E@sparrow.telecommunity.com> <79990c6b0804091548p533e4dear5fb500341e4b87af@mail.gmail.com> Message-ID: <20080409231611.646C53A406A@sparrow.telecommunity.com> At 11:48 PM 4/9/2008 +0100, Paul Moore wrote: >On 09/04/2008, Phillip J. Eby wrote: > > It would be, if .eggs were a packaging format, rather than a binary > > distribution/runtime format. > > > > Remember "eggs are to Python as jars are to Java" -- a Java .jar > > doesn't contain documentation either, unless it's needed at > > runtime. Same for configuration files. > >And yet, Java doesn't have an equivalent of easy_install for jar >files, to my knowledge. Actually, OSGi and Eclipse plugins and "feature sites" come quite close, and setuptools rips off many of its features from them. OSGi is basically a standard for additional .jar metadata to encompass dependencies and other info. From dpeterson at enthought.com Thu Apr 10 01:17:49 2008 From: dpeterson at enthought.com (Dave Peterson) Date: Wed, 09 Apr 2008 18:17:49 -0500 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409194044.1A4E03A406A@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> Message-ID: <47FD4E9D.8060600@enthought.com> Phillip J. Eby wrote: > >>> Applying LSB and FHS to the innards of Python packages makes as much >>> sense as applying them to the contents of Java .jar files -- i.e., >>> none. If it's unchanging data that's part of a program or library, >>> then it's a program or library, just like static data declared in a C >>> program or library. Whether the file extension is .py, .so, or even >>> .png is irrelevant. >>> >> The FHS defines places to put specific kinds of files, such as command >> scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation >> (/usr/share/doc/package-name), and configuration files (/etc). There are >> several kinds of files identified and places defined to put them. >> Distribution by eggs has a tendency to scoop up all of those files and put >> them in /usr/lib/python/site-packages, regardless of where they belong. >> > > Eggs don't include documentation or configuration files, and they > install scripts in script directories, so I don't get what you're > talking about here. For any other data that a package accesses at > runtime, my earlier comments apply. > We've talked a bit about this before, and IIRC, at that time you (Phillip) were willing to consider patches to setuptools that allowed for the inclusion of documentation in eggs such that it was placed into an LSB/FHS appropriate directory (or some standard dir for non-LSB systems) during the install process. I'm assuming that something similar for config files wouldn't be a problem either? Or is this whole idea out the window given the way the discussion has trended and the reiteration above that eggs are meant to be similar in principal to jars? Not that I have a patch yet, but we've been working on it in our "spare time" over here at Enthought. I'm now wondering if we're wasting our time. :-) I think the biggest use-case confusion in the current discussion is whether we're talking about applications or libraries? If we're talking about libraries, then clearly distribution of only executables is sufficient because anything else should be handled by the application distribution when that library is used in an app. Whereas if we're talking about applications, you probably *do* want to include documentation, configuration info, etc. in your distribution. I think I can sum up any further points by simply asking: "Should it be safe to assume I can distribute my application via eggs / easy_install just because it is written in Python?" -- Dave -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080409/95acebba/attachment-0001.htm From pje at telecommunity.com Thu Apr 10 01:24:46 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 09 Apr 2008 19:24:46 -0400 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409225138.GH28763@phare.normalesup.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> <20080409225138.GH28763@phare.normalesup.org> Message-ID: <20080409232437.7D74F3A406A@sparrow.telecommunity.com> At 12:51 AM 4/10/2008 +0200, Gael Varoquaux wrote: >On Wed, Apr 09, 2008 at 11:46:19PM +0100, Paul Moore wrote: > > I find this whole discussion hugely confusing, because a lot of people > > are stating opinions about environments which it seems they don't use, > > or know much about. I don't know how to avoid this, but it does make > > it highly unlikely that any practical progress will get made. > >I find that something that doesn't help at all the discussion move >forward is that everybody has different usecases in mind, on different >platforms, and is not interested in other people's usecases. > >Hopefuly I am wrong, You're not wrong at all. I have to deal with *all* the platforms and use cases, which makes it quite frustrating when people who haven't even read the requirements are making proposals to "solve" things in ways that break for everyone except their own niche platform+usecase combination. Guido, can I borrow the time machine and go back and NOT try to improve on the distutils? Or is there already too much collateral damage to the timeline? ;-) From greg.ewing at canterbury.ac.nz Thu Apr 10 01:27:50 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 10 Apr 2008 11:27:50 +1200 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <2869C14C-A204-4A71-9C5E-8E202E1DC42D@zooko.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> <47FC0174.3070605@canterbury.ac.nz> <2869C14C-A204-4A71-9C5E-8E202E1DC42D@zooko.com> Message-ID: <47FD50F6.1070109@canterbury.ac.nz> zooko wrote: > > We determined > that if you install the egg (with easy_install or with a > setuptools-powered ./setup.py install) in unzipped form then the source > file names get rewritten so that your stack traces come with source lines. That wouldn't have helped me with my problem, because I was trying to use the traceback to track down where the file was in the first place. I wasn't even aware that it was in an egg at first. I wouldn't have been using eggs at all if I hadn't been tricked into it by some package telling me I had to use easy_install to install it. If I'd known that setuptools would make intrusive changes to the behaviour of my system, I would never have touched it. Is there a way of *un*installing setuptools? I'd like to put my Python back the way it used to be. -- Greg From greg.ewing at canterbury.ac.nz Thu Apr 10 01:58:53 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 10 Apr 2008 11:58:53 +1200 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409200605.DD2783A406A@sparrow.telecommunity.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> <47FC0174.3070605@canterbury.ac.nz> <2869C14C-A204-4A71-9C5E-8E202E1DC42D@zooko.com> <20080409200605.DD2783A406A@sparrow.telecommunity.com> Message-ID: <47FD583D.30900@canterbury.ac.nz> Phillip J. Eby wrote: > Are you using Python 2.5? As of 2.5, the linecache module should > correctly read the source line from the present location of the source > file the module was loaded from, regardless of the file name specified > in the traceback. I think it was doing that, but I was trying to find the file, and the traceback wasn't helping. > A better fix would be for Python to use relative paths in co_filename, > as this would fix similar problems that occur whenever you move > .pyc/.pyo files around. Yes, it must be possible to do something better. The actual path is known when the module is loaded, so it should just be a matter of storing it somewhere appropriate. -- Greg From zooko at zooko.com Thu Apr 10 02:27:22 2008 From: zooko at zooko.com (zooko) Date: Wed, 9 Apr 2008 17:27:22 -0700 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409231247.7F3A23A406A@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <390014FF-5088-42C1-8213-17D0B25D2FFB@zooko.com> <20080409231247.7F3A23A406A@sparrow.telecommunity.com> Message-ID: <09411577-DD0F-4E0C-AE69-6CE4F1D97CD5@zooko.com> On Apr 9, 2008, at 4:12 PM, Phillip J. Eby wrote: >> http://allmydata.org/trac/setuptools/ticket/5 # binary eggs should >> come with .py files by default, rather than .pyc files > > Filling your tracker with already-rejected proposals isn't likely > to encourage me to look at it, especially when I've personally > rejected them to you in IRC. That goes for your ticket #4 as well. Part of my motivation in maintaining this tracker is to take issue discussions from IRC, and from mailing lists, and make them more permanent and structured. This part is useful even for rejected proposals, as an historical record that other people interested in those issues can consult. I will mark those two tickets as "rejected by PJE". Could you please repeat (so that I don't misrepresent you due to my faulty memory of our IRC discussion from more than a year ago) your reason for rejecting these two: http://allmydata.org/trac/setuptools/ticket/4 (when considering whether to zip, err on the side of safety rather than performance) http://allmydata.org/trac/setuptools/ticket/5 (binary eggs should come with .py files by default, rather than .pyc files) You are of course welcome to log in to that trac and update those tickets yourself, but if you prefer not to then I will paste your reasons into those tickets. Regards, Zooko From lxander.m at gmail.com Thu Apr 10 03:54:05 2008 From: lxander.m at gmail.com (Alexander Michael) Date: Wed, 9 Apr 2008 21:54:05 -0400 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409194044.1A4E03A406A@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> Message-ID: <525f23e80804091854g19460b02pd426016b56177272@mail.gmail.com> On Wed, Apr 9, 2008 at 3:40 PM, Phillip J. Eby wrote: > That is indeed a problem -- but it's a social one, not a technical > one. It's trivial for the publisher of an egg to change their > command line from "setup.py bdist_egg upload" to "setup.py sdist > bdist_egg upload", as soon as their users (politely) request that they do so. I know you say it in , but perhaps you should be more explicit about the best practice being to distribute an sdist as well as one or more eggs and highlight this with some standout text like: """ When uploading your project to PyPI, always upload your sdist in addition to any eggs unless you have a good idiosyncratic reason not to. The easy_install tool can also download and install sdist's on all platforms (universally for pure-python packages and provided compilers are available for distributions containing python extensions), so uploading them gives your project the widest possible audience. """ Perhaps this will help some the social challenges. Regards, Alex From stephen at xemacs.org Thu Apr 10 06:12:40 2008 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 10 Apr 2008 13:12:40 +0900 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <87iqyqir2t.fsf@benfinney.id.au> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <87iqyqir2t.fsf@benfinney.id.au> Message-ID: <87ej9eic07.fsf@uwakimon.sk.tsukuba.ac.jp> Ben Finney writes: > "Stanley A. Klein" writes: > > > IMHO, the main system without a package manager is Windows. > > AFAICT the MacOS platform also lacks in this area. Actually, they both have them. Windows has Cygwin (rpm-based), while for MacOS Fink (deb-based), MacPorts (FreeBSD ports-like), and NetBSD's pkgsrc are all viable options if you want packaging support for 3rd-party packages. From steven.bethard at gmail.com Thu Apr 10 08:11:22 2008 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 10 Apr 2008 00:11:22 -0600 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <79990c6b0804091548p533e4dear5fb500341e4b87af@mail.gmail.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <12121.207.188.248.157.1207773812.squirrel@www.cpcug.org> <20080409212522.9CDE13A409E@sparrow.telecommunity.com> <79990c6b0804091548p533e4dear5fb500341e4b87af@mail.gmail.com> Message-ID: On Wed, Apr 9, 2008 at 4:48 PM, Paul Moore wrote: > On 09/04/2008, Phillip J. Eby wrote: > > It would be, if .eggs were a packaging format, rather than a binary > > distribution/runtime format. > > > > Remember "eggs are to Python as jars are to Java" -- a Java .jar > > doesn't contain documentation either, unless it's needed at > > runtime. Same for configuration files. > > And yet, Java doesn't have an equivalent of easy_install for jar > files, to my knowledge. If Python had eggs but no easy_install, maybe > this whole discussion wouldn't be taking place. > > (I know I personally like the idea of egg-as-jar-file, but *hate* the > idea of egg-as-dependency-handling-tool-and-everything-else). So then do you really need anything more than the 2.6 support for executing directories (and zipfiles) with a __main__ file? $ python_d.exe --version Python 2.6a1+ $ more foo\__main__.py print 'here I am!' $ python_d.exe foo here I am! [8698 refs] $ python_d.exe foo.zip here I am! [8563 refs] STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From apt.shansen at gmail.com Thu Apr 10 09:12:43 2008 From: apt.shansen at gmail.com (Stephen Hansen) Date: Thu, 10 Apr 2008 00:12:43 -0700 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <87ej9eic07.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <87iqyqir2t.fsf@benfinney.id.au> <87ej9eic07.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <7a9c25c20804100012y1a9d797cv1ed4c6d0739c5ba0@mail.gmail.com> > > > > IMHO, the main system without a package manager is Windows. > > > > AFAICT the MacOS platform also lacks in this area. > > Actually, they both have them. Windows has Cygwin (rpm-based), while > for MacOS Fink (deb-based), MacPorts (FreeBSD ports-like), and > NetBSD's pkgsrc are all viable options if you want packaging support > for 3rd-party packages. > Er, excuse me for cutting in but-- that's just not at all the same thing. For people who are using a Red Had derivative, or a Debian derivative, or .. whatever .. the package manager isn't something they go out of their way to add and then have to struggle with. It's simply *how* their world works. By a large measure, everything they want is there, managed by their package management system. For those users, I understand well that they don't want some Python package management system to be a second system that they have to deal with. But I'm sorry: the world is bigger then Linux and such things. I'm a mac user, who has had extensive experience in Linux; but on the mac? Fink and MacPorts added on top of MacOSX is not even remotely comparable to using an operating system which has a standard package manager that is a part of every users daily life. My operating system is Unixy, and comes pre-installed with a number of things, including Python, wxWidgets, sqlite-- many things that make the programs I make for my customers easy to use. But for those products that are *not* available standard, where are my mac customers left? Their options are to install something like Fink, or MacPorts... and then we come into issues of it wanting to install its *own* version of python, or its *own* version of these third party things, on top of what's already there? The alternative is that users have to install, manually, these third-party requirements themselves. I've found that it is in general far easier for me to just download and install stuff manually then to rely on these "Add-on" package managers. At least if I'm thinking of providing as minimally and least intrusive as possible experience for my users to install my product. Power users, especially those familiar with the Linux world, may relish in the existence of MacPorts and Fink... Regular people, even IT managers of companies I have to deal with-- will not. I love easy_install/setuptools because it lets me get my *Python* applications and products out to people, regardless of OS, in a way that just *works*. I do think its valuable to do so in a way that will integrate with native package managers on those operating systems that they are a native and integrated part of -- but to say, "Let's not re-create apt!" is a sorrowful stance. It's saying, "Screw Windows, because it isn't as good as what we have." and "Screw Mac, because its not as good as we have." Or even, "Screw the people who aren't power users and are just not going to be able to go through the effort of adding *on* a non-standard package management system to their operating system." There's a whole wide world out there that simply does *not* have a "package management"(*) system. Python is beautiful, making Python programs is blissful. I'd be far, far more concerned with making it easy to distribute Python-based programs to *any* operating system then I would be concerned with partially redoing what a *minority* of systems out there have done to make package management (with dependencies and all) easy for its users. Python is a cross-platform development environment. Let's not forget that most people just... don't have Linux... and don't have the equally blissful world of apt or rpm available to them natively. Its very cool to *integrate* -- to make a way for those RPM and DEB distributors to deliver an app in their own way that will satisfy their needs. But what about the people *without* that native capability? Having a Python-only distribution/management system like easy_install is a *huge* boon to getting real products to real people. I think PJE's idea here is very good. Just include certain files and such in the RPM/DEB that will satisfy the "python-package-management" system. For RPM/DEB users and their OS's database of packages, its irrelevant largely-- they'll still keep using their own system. But if a product needs something without a .deb or .rpm, or if someone's on an operating system without a native system-- they can still gather everything they need. Anyways. My 2 cents. --Stephen -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080410/be208ce6/attachment-0001.htm From him at online.de Thu Apr 10 09:58:27 2008 From: him at online.de (=?ISO-8859-1?Q?Joachim_K=F6nig?=) Date: Thu, 10 Apr 2008 09:58:27 +0200 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409212522.9CDE13A409E@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <12121.207.188.248.157.1207773812.squirrel@www.cpcug.org> <20080409212522.9CDE13A409E@sparrow.telecommunity.com> Message-ID: <47FDC8A3.7090300@online.de> Phillip J. Eby wrote: > It would be, if .eggs were a packaging format, rather than a binary > distribution/runtime format. > > Remember "eggs are to Python as jars are to Java" -- a Java .jar > doesn't contain documentation either, unless it's needed at > runtime. Same for configuration files. > But there's generally no need to easily have a look into a .class file with a tool the user is used to whereas for python, we're often interested in knowing the details. And having a zip file in my way to the source has left me frustrated often enough. If you want to be consequent and consistent leave out the .py files from eggs, a jar file normally doesn't contain .java sources files either. > They're not system packages, in other words. The assumption that > they are is another marketing failure, due to conflation of "package > == distribution of python code" and "package == thing you manage with > a system packager". People see, "I put my package in an .egg" and > think it's the latter definition, when it's barely even the former. :) > I agree that they are not system packages, but I would have prefered to install multiple versions of a package to separate "site-packages" directories, something that is really well understood by most unsofisticated python programmers. The selection of the version could then be made at runtime by a PYTHONPATH setting and not by fiddling with .pth files (something that could be autmated by a tool and persisted in batch files). From stephen at xemacs.org Thu Apr 10 10:30:41 2008 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 10 Apr 2008 17:30:41 +0900 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <7a9c25c20804100012y1a9d797cv1ed4c6d0739c5ba0@mail.gmail.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <87iqyqir2t.fsf@benfinney.id.au> <87ej9eic07.fsf@uwakimon.sk.tsukuba.ac.jp> <7a9c25c20804100012y1a9d797cv1ed4c6d0739c5ba0@mail.gmail.com> Message-ID: <87zls2glhq.fsf@uwakimon.sk.tsukuba.ac.jp> Stephen Hansen writes: > > > > > > IMHO, the main system without a package manager is Windows. > > > > > > AFAICT the MacOS platform also lacks in this area. > > > > Actually, they both have them. Windows has Cygwin (rpm-based), while > > for MacOS Fink (deb-based), MacPorts (FreeBSD ports-like), and > > NetBSD's pkgsrc are all viable options if you want packaging support > > for 3rd-party packages. > > Er, excuse me for cutting in but-- that's just not at all the same > thing. [...] > I do think its valuable to do so in a way that will integrate with native > package managers on those operating systems that they are a native and > integrated part of Why restrict it to those? That's my point. If you're going to make it apt-compatible, you should do all the others, too, because there are lots of people using them. Whether or not they are OS-provided. As far as I'm concerned, asking a Python tool to integrate into any pms is a non-starter because it really means asking for integration into all of them. > but to say, "Let's not re-create apt!" is a sorrowful stance. It's > saying, "Screw Windows, because it isn't as good as what we have." > and "Screw Mac, because its not as good as we have." It's worse than that. One point I hoped to make is that "Let's not recreate apt!" is a way of saying "Screw Windows and Mac and Red Hat and Gentoo and NetBSD and FreeBSD and ...." I think it's worth working out a standard format for documenting dependencies so that a downstream distributor can easily create a dependency graph (perhaps several of them, since system bootstrap, package build-time, and package run-time may have different requirements). And standard Python distribution media should have a "manual mode" so that distros can decide where to put things. But that's as far as Python itself should ever go; fitting those things into any given pms is the downstream distro's job. From pje at telecommunity.com Thu Apr 10 16:51:13 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 10 Apr 2008 10:51:13 -0400 Subject: [Python-Dev] how to easily consume just the parts of eggs that are good for you In-Reply-To: <7a9c25c20804100012y1a9d797cv1ed4c6d0739c5ba0@mail.gmail.co m> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <87iqyqir2t.fsf@benfinney.id.au> <87ej9eic07.fsf@uwakimon.sk.tsukuba.ac.jp> <7a9c25c20804100012y1a9d797cv1ed4c6d0739c5ba0@mail.gmail.com> Message-ID: <20080410145106.051E53A406A@sparrow.telecommunity.com> At 12:12 AM 4/10/2008 -0700, Stephen Hansen wrote: >I think PJE's idea here is very good. Just include certain files and >such in the RPM/DEB that will satisfy the >"python-package-management" system. For RPM/DEB users and their OS's >database of packages, its irrelevant largely-- they'll still keep >using their own system. But if a product needs something without a >.deb or .rpm, or if someone's on an operating system without a >native system-- they can still gather everything they need. I've narrowed it a bit from that, actually. It's safest if easy_install simply refuses to touch any files that it can't tell were installed by it (or a compatible system, eg. distutils.) While that won't solve Paul Moore's desire for the One True Package Manager, it will at least make it possible to move forward. From rbp at isnomore.net Thu Apr 10 17:10:47 2008 From: rbp at isnomore.net (Rodrigo Bernardo Pimentel) Date: Thu, 10 Apr 2008 12:10:47 -0300 Subject: [Python-Dev] Next monthly sprint/bugfix day? In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22BEB730@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22BEB730@EXMBX04.exchhosting.com> Message-ID: <20080410151046.GA11041@isnomore.net> On Wed, Apr 09 2008 at 11:12:58AM BRT, Trent Nelson wrote: > Hi, > > Is there another online sprint/bugfix day in the pipeline? If not, can > there be? ;-) +1. The Sao Paulo User's Group gathered to work on the last Bug Day, and I think it was very productive (both for generating patches/comments and for us to get more involved with Python core development). So, actually, assuming I can speak for at least half of the people from the PUG who participated in the last Bug Day, +~8 :). rbp -- Rodrigo Bernardo Pimentel | GPG: <0x0DB14978> From sergiodj at linux.vnet.ibm.com Thu Apr 10 19:50:46 2008 From: sergiodj at linux.vnet.ibm.com (=?ISO-8859-1?Q?S=E9rgio?= Durigan =?ISO-8859-1?Q?J=FAnior?=) Date: Thu, 10 Apr 2008 14:50:46 -0300 Subject: [Python-Dev] Compiling Python on a biarch (PPC/PPC64) In-Reply-To: <47FD34D5.2060607@v.loewis.de> References: <1207773854.5788.23.camel@miki> <47FD34D5.2060607@v.loewis.de> Message-ID: <1207849847.6411.6.camel@miki> Hi Martin, On Wed, 2008-04-09 at 23:27 +0200, "Martin v. L?wis" wrote: > > Can anyone give me a hand on this? As far as I could > > investigate, I must change setup.py in order to get this issue fixed, > > right? > > Wrong. I didn't read your entire message, but this is definitely wrong. > setup.py isn't invoked until python itself is built, and > Modules/python.c is part of python itself. If I understood correctly, you're saying that Python's building process involves only the C compiler, right? So, in order to fix this issue that I've found, I only need to modify configure.in file (and maybe Makefile.pre.in). Is that correct? Regards, -- S?rgio Durigan J?nior Linux on Power Toolchain - Software Engineer Linux Technology Center - LTC IBM Brazil From martin at v.loewis.de Thu Apr 10 21:23:56 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 10 Apr 2008 21:23:56 +0200 Subject: [Python-Dev] Compiling Python on a biarch (PPC/PPC64) In-Reply-To: <1207849847.6411.6.camel@miki> References: <1207773854.5788.23.camel@miki> <47FD34D5.2060607@v.loewis.de> <1207849847.6411.6.camel@miki> Message-ID: <47FE694C.70607@v.loewis.de> > If I understood correctly, you're saying that Python's building process > involves only the C compiler, right? So, in order to fix this issue that > I've found, I only need to modify configure.in file (and maybe > Makefile.pre.in). Is that correct? Correct. There are also some additional shell scripts involved, but I don't think they matter. Regards, Martin From musiccomposition at gmail.com Fri Apr 11 01:42:41 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Thu, 10 Apr 2008 18:42:41 -0500 Subject: [Python-Dev] generated NEWS files Message-ID: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> I know that doing merges is rather painful because of the Misc/NEWS files. Would it be possible to have the NEWS log generated from commit messages? I was thinking something like this in a message: Fixed this and that **NEWS:Library** #1234 X is now known as Y. It should be fairly simple to write a script to extract and compile these messages into NEWS before release. -- Cheers, Benjamin Peterson From guido at python.org Fri Apr 11 02:55:15 2008 From: guido at python.org (Guido van Rossum) Date: Thu, 10 Apr 2008 17:55:15 -0700 Subject: [Python-Dev] generated NEWS files In-Reply-To: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> References: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> Message-ID: In the past we haven't done that because often the change logs are either too chatty (they do administrative stuff that doesn't deserve a NEWS entry) or they are too brief (in the heat of the battle developers don't always write great changelog entries). While it's easy to fix NEWS its not easy to fix changelog entries (only a few svn super-users can do it, and it's a pain). --Guido On Thu, Apr 10, 2008 at 4:42 PM, Benjamin Peterson wrote: > I know that doing merges is rather painful because of the Misc/NEWS > files. Would it be possible to have the NEWS log generated from commit > messages? I was thinking something like this in a message: > > Fixed this and that > > **NEWS:Library** > #1234 X is now known as Y. > > It should be fairly simple to write a script to extract and compile > these messages into NEWS before release. > > -- > Cheers, > Benjamin Peterson > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From musiccomposition at gmail.com Fri Apr 11 02:59:35 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Thu, 10 Apr 2008 19:59:35 -0500 Subject: [Python-Dev] generated NEWS files In-Reply-To: References: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> Message-ID: <1afaf6160804101759j2ff0c90ep400cd096ea31f64f@mail.gmail.com> On Thu, Apr 10, 2008 at 7:55 PM, Guido van Rossum wrote: > In the past we haven't done that because often the change logs are > either too chatty (they do administrative stuff that doesn't deserve a > NEWS entry) or they are too brief (in the heat of the battle > developers don't always write great changelog entries). While it's > easy to fix NEWS its not easy to fix changelog entries (only a few svn > super-users can do it, and it's a pain). A NEWs entry wouldn't be added if the special **NEWS** marking wasn't present. I think we'd have to come up with some way to let people change/edit by overriding those in the commit messages, but I'm not sure how that would be done. > > --Guido -- Cheers, Benjamin Peterson From guido at python.org Fri Apr 11 03:24:18 2008 From: guido at python.org (Guido van Rossum) Date: Thu, 10 Apr 2008 18:24:18 -0700 Subject: [Python-Dev] generated NEWS files In-Reply-To: <1afaf6160804101759j2ff0c90ep400cd096ea31f64f@mail.gmail.com> References: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> <1afaf6160804101759j2ff0c90ep400cd096ea31f64f@mail.gmail.com> Message-ID: On Thu, Apr 10, 2008 at 5:59 PM, Benjamin Peterson wrote: > On Thu, Apr 10, 2008 at 7:55 PM, Guido van Rossum wrote: > > In the past we haven't done that because often the change logs are > > either too chatty (they do administrative stuff that doesn't deserve a > > NEWS entry) or they are too brief (in the heat of the battle > > developers don't always write great changelog entries). While it's > > easy to fix NEWS its not easy to fix changelog entries (only a few svn > > super-users can do it, and it's a pain). > A NEWs entry wouldn't be added if the special **NEWS** marking wasn't > present. I think we'd have to come up with some way to let people > change/edit by overriding those in the commit messages, but I'm not > sure how that would be done. It would be fine if that was *one* way of adding a NEWS entry, but I don't think it should be the only way, and I don't think it would work well to regenerate the NEWS file from those entries. In practice we see corrections on NEWS entries all the time. Also, there's the issue of which section of the NEWS file an entry should be added. That's often subjective. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at krypto.org Fri Apr 11 07:53:14 2008 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 10 Apr 2008 22:53:14 -0700 Subject: [Python-Dev] generated NEWS files In-Reply-To: References: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> <1afaf6160804101759j2ff0c90ep400cd096ea31f64f@mail.gmail.com> Message-ID: <52dc1c820804102253i390c9912i6a84bbe074ae8738@mail.gmail.com> On Thu, Apr 10, 2008 at 6:24 PM, Guido van Rossum wrote: > > Also, there's the issue of which section of the NEWS file an entry > should be added. That's often subjective. > Agreed. For example we have both Library and Extension Module categories in the NEWS file. I always end up listing changes in the "wrong" one of those half the time. In my mind those are the same thing. A reader is not likely to care if the change happened to be implemented in the Lib/ directory vs the Modules/ directory. From python's point of view they're both just an import. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080410/62043f0f/attachment.htm From martin at v.loewis.de Fri Apr 11 09:23:34 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 11 Apr 2008 09:23:34 +0200 Subject: [Python-Dev] generated NEWS files In-Reply-To: References: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> Message-ID: <47FF11F6.5010808@v.loewis.de> > While it's > easy to fix NEWS its not easy to fix changelog entries (only a few svn > super-users can do it, and it's a pain). Actually, any committer can do that: svn pe --revprop -rrev svn:log It may be difficult to remember, but it's fairly easy to execute. Regards, Martin From lists at cheimes.de Fri Apr 11 11:52:15 2008 From: lists at cheimes.de (Christian Heimes) Date: Fri, 11 Apr 2008 11:52:15 +0200 Subject: [Python-Dev] generated NEWS files In-Reply-To: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> References: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> Message-ID: Benjamin Peterson schrieb: > I know that doing merges is rather painful because of the Misc/NEWS > files. Would it be possible to have the NEWS log generated from commit > messages? I was thinking something like this in a message: No, it's not painful ;) The NEWS files doesn't get merged from trunk to 3.0. $ svnmerge.py merge $ svn revert Misc/NEWS resolve conflicts, compile, test $ svn ci -F svn.... Christian From asmodai at in-nomine.org Fri Apr 11 17:10:24 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Fri, 11 Apr 2008 17:10:24 +0200 Subject: [Python-Dev] Python 2.6a2 execution times with various compilers Message-ID: <20080411151024.GJ51167@nexus.in-nomine.org> I did some performance comparisons with various compilers and the resulting Python 2.6a2 and pybench. I put the details on http://www.in-nomine.org/2008/04/11/python-26a2-execution-times-with-various-compilers/ Of course, take benchmark results with a grain of salt, but it seems ICC can provide people with an added performance edge when using Python. In short: I measured a speedup between ~21%-29% with ICC compared to GCC. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ I was searching through the Heavens and somehow I slipped... From status at bugs.python.org Fri Apr 11 18:06:40 2008 From: status at bugs.python.org (Tracker) Date: Fri, 11 Apr 2008 18:06:40 +0200 (CEST) Subject: [Python-Dev] Summary of Tracker Issues Message-ID: <20080411160640.59DC07810A@psf.upfronthosting.co.za> ACTIVITY SUMMARY (04/04/08 - 04/11/08) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1826 open (+45) / 12616 closed (+21) / 14442 total (+66) Open issues with patches: 544 Average duration of open issues: 709 days. Median duration of open issues: 1287 days. Open Issues Breakdown open 1801 (+45) pending 25 ( +0) Issues Created Or Reopened (68) _______________________________ setitimer, getitimer wrapper 04/05/08 http://bugs.python.org/issue2240 reopened loewis patch Unicode escape sequences not parsed in raw strings. 04/05/08 http://bugs.python.org/issue2541 reopened gvanrossum patch Python 2.6a2 on Solaris 10 built with SUN C 04/04/08 CLOSED http://bugs.python.org/issue2551 created MrJean1 test_ctypes failed Python 2.6a2 Solaris 10 SUN C 04/04/08 http://bugs.python.org/issue2552 created MrJean1 patch test_ioctrl failed Python 2.6a2 Solaris 10 SUN C 04/04/08 CLOSED http://bugs.python.org/issue2553 created MrJean1 test_ioctl failed Python 2.6a2 Solaris 10 SUN C 04/04/08 http://bugs.python.org/issue2554 created MrJean1 test_parser failed Python 2.6a2 Solaris 10 SUN C 04/04/08 CLOSED http://bugs.python.org/issue2555 created MrJean1 changing sys.dont_write_bytecode has not effect 04/05/08 CLOSED http://bugs.python.org/issue2556 created benjamin.peterson \u and \U in raw strings have regressed in 3.0a4 04/05/08 CLOSED http://bugs.python.org/issue2557 created gvanrossum Document pickle protocol 3 04/05/08 CLOSED http://bugs.python.org/issue2558 created georg.brandl patch atom sorting error when buiding ctypes 04/05/08 http://bugs.python.org/issue2559 created skip.montanaro removal of stale code from myreadline.c 04/06/08 http://bugs.python.org/issue2560 created JosephArmbruster patch Instance remembers old values from former life 04/06/08 CLOSED http://bugs.python.org/issue2561 created warnhold Cannot use non-ascii letters in disutils if setuptools is used. 04/06/08 http://bugs.python.org/issue2562 reopened loewis patch embed manifest in windows extensions 04/06/08 http://bugs.python.org/issue2563 created mhammond patch, patch Python hangs on FreeBSD7 in test_capi 04/06/08 CLOSED http://bugs.python.org/issue2564 created trepan Change 'type' to 'class' in repr/str(builtin-class) 04/07/08 CLOSED http://bugs.python.org/issue2565 created tjreedy easy Py3.0a4 wsgiref simple_server failed to start 04/07/08 http://bugs.python.org/issue2566 created delimy Section "New-style and classic classes" needs to be removed/rewr 04/07/08 CLOSED http://bugs.python.org/issue2567 created loewis Seconds range in time unit 04/07/08 http://bugs.python.org/issue2568 created datacompboy default_scheme in urlparse.urlparse() useless 04/07/08 http://bugs.python.org/issue2569 created pk backport 3.0-style \u/\U processing in raw strings when unicode_ 04/07/08 http://bugs.python.org/issue2570 created gvanrossum 26backport cmd.py always uses raw_input, even when another stdin is specifi 04/07/08 http://bugs.python.org/issue2571 created rickbking 3.0 pickle docs -- what about old-style classes? 04/07/08 http://bugs.python.org/issue2572 created georg.brandl Can't change the framework name on OS X builds 04/07/08 http://bugs.python.org/issue2573 created habnabit patch Add RFC 3768 SSM Multicast support to "socket" 04/07/08 http://bugs.python.org/issue2574 created bms patch Reference manual: production for integer literals is missing "bi 04/07/08 CLOSED http://bugs.python.org/issue2575 created marketdickinson patch, patch httplib read() very slow due to lack of socket buffer 04/07/08 http://bugs.python.org/issue2576 created reacocard patch cmd.py should track input file objects so macros with submacros 04/07/08 http://bugs.python.org/issue2577 created rickbking Figure out what to do with unittest's redundant APIs 04/08/08 http://bugs.python.org/issue2578 created benjamin.peterson Misleading 'toctree references unknown document' error 04/08/08 CLOSED http://bugs.python.org/issue2579 created henryl patch Revise builtin class int library manual entry 04/08/08 CLOSED http://bugs.python.org/issue2580 created tjreedy Vista UAC/elevation support for bdist_wininst 04/08/08 http://bugs.python.org/issue2581 created mhammond patch, patch Unpickling of range objects fail in Py3k 04/08/08 http://bugs.python.org/issue2582 created pythonhacker urlparse normalize URL path 04/08/08 http://bugs.python.org/issue2583 created monk.e.boy numeric overflow in IDLE 04/08/08 http://bugs.python.org/issue2584 created Qodosh urllib2.HTTPError broken due to urllib.addinfourl changes 04/08/08 CLOSED http://bugs.python.org/issue2585 created djc patch Integer signedness bugs in zlib modules 04/08/08 CLOSED http://bugs.python.org/issue2586 created jnferguson PyString_FromStringAndSize() to be considered unsafe 04/10/08 http://bugs.python.org/issue2587 reopened gvanrossum PyOS_vsnprintf() underflow leads to memory corruption 04/08/08 http://bugs.python.org/issue2588 created jnferguson PyOS_vsnprintf() potential integer overflow leads to memory corr 04/08/08 http://bugs.python.org/issue2589 created jnferguson S_unpack_from() Read Access Violation 04/08/08 http://bugs.python.org/issue2590 created jnferguson ErrorHandler buffer overflow in ?unused? SGI extension module al 04/08/08 http://bugs.python.org/issue2591 created jnferguson weakref.proxy fails to delegate tp_index slot 04/08/08 http://bugs.python.org/issue2592 created ncoghlan easy alp_ReadFrames() integer overflow leads to buffer overflow 04/08/08 http://bugs.python.org/issue2593 created jnferguson alp_readsamps() overflow leads to memory corruption in ?unused? 04/08/08 http://bugs.python.org/issue2594 created jnferguson Multiple integer overflows in imgfile extension module lead to b 04/08/08 http://bugs.python.org/issue2595 created jnferguson 2to3's fix_xrange needs fix_dict's context information 04/08/08 CLOSED http://bugs.python.org/issue2596 created collinwinter "python2.6 -3" should report list.sort(cmp) as DeprecationWarnin 04/08/08 CLOSED http://bugs.python.org/issue2597 created dangyogi "{ +(}".format(**{' +(': 44}) should produce ValueError: invalid 04/08/08 http://bugs.python.org/issue2598 created dangyogi allow field_name in format strings to default to next positional 04/08/08 http://bugs.python.org/issue2599 created dangyogi BindingHTTPConnectionWithTimeout and BindingHTTPHandlerWithTimeo 04/08/08 http://bugs.python.org/issue2600 created pau [regression] reading from a urllib2 file descriptor happens byte 04/08/08 http://bugs.python.org/issue2601 created doko importing gtk hangs (freebsd 7.0) 04/09/08 CLOSED http://bugs.python.org/issue2602 created trepan Make range __eq__ work 04/09/08 http://bugs.python.org/issue2603 created benjamin.peterson patch doctest.DocTestCase fails when run repeatedly 04/10/08 http://bugs.python.org/issue2604 created pjd Descriptor instance attributes not interpreted consistently 04/10/08 http://bugs.python.org/issue2605 created pjd trace module crashes due to using wrong sort idiom 04/10/08 CLOSED http://bugs.python.org/issue2606 created mark patch why is (default)dict so slow on tuples??? 04/10/08 CLOSED http://bugs.python.org/issue2607 created eisele manpages for sphinx-build & sphinx-quickstart 04/10/08 http://bugs.python.org/issue2608 created dottedmag Tests fail if ./@test is not writeable 04/10/08 http://bugs.python.org/issue2609 created belopolsky string representation of range 04/10/08 http://bugs.python.org/issue2610 created bmiller patch Extend buildbot web interface to allow for forced tests to be ru 04/10/08 http://bugs.python.org/issue2611 created Trent.Nelson file.tell() returns Long usually, Int if subclassed 04/11/08 CLOSED http://bugs.python.org/issue2612 created esrever_otua inconsistency with bare * in parameter list 04/11/08 http://bugs.python.org/issue2613 created mark Console UnicodeDecodeError s once more 04/11/08 http://bugs.python.org/issue2614 created techtonik xml.dom.minidom documentation consistency and update 04/11/08 http://bugs.python.org/issue2615 created asmodai patch ctypes.pointer(), ctypes.POINTER() speedup 04/11/08 http://bugs.python.org/issue2616 created theller patch, patch Issues Now Closed (47) ______________________ socket.socket.getsockname() has inconsistent UNIX/Windows behavi 226 days http://bugs.python.org/issue1049 Trent.Nelson pyconfig.h not compatible with MS VC++ Express Edition 175 days http://bugs.python.org/issue1297 fdrake sys.argv is wrong for unicode strings 51 days http://bugs.python.org/issue2128 benjamin.peterson patch tokenize: does not allow CR for a newline 43 days http://bugs.python.org/issue2182 jaredgrubb Further simplify dict literal bytecode 41 days http://bugs.python.org/issue2197 rhettinger patch Py30a3: eval in threaded code raises SystemError 38 days http://bugs.python.org/issue2221 amaury.forgeotdarc Install failure of 2.6a1 on Windows XP without VS8 installed 34 days http://bugs.python.org/issue2256 loewis [PATCH] Tcl/Tk 8.4.16 patches needed to get an x64 Windows build 20 days http://bugs.python.org/issue2296 Trent.Nelson patch, 64bit Compiler warnings when using UCS4 20 days http://bugs.python.org/issue2388 loewis easy types module can be implemented only in Python 21 days http://bugs.python.org/issue2408 amaury.forgeotdarc patch absolute import doesn't work for standard python modules 21 days http://bugs.python.org/issue2410 ncoghlan Issues with getargs_n() and PyNumber_Index. 21 days http://bugs.python.org/issue2440 benjamin.peterson patch, patch, 64bit Line tracing of continue after always-taken if is incorrect 7 days http://bugs.python.org/issue2506 tjreedy 64bit cross compilation on windows 10 days http://bugs.python.org/issue2513 loewis patch, 64bit Segfault while operating on closed sqlite3 cursor. 7 days http://bugs.python.org/issue2515 pjdavis class USTimeZone in Doc/includes/tzinfo-examples.py is out of da 7 days http://bugs.python.org/issue2525 georg.brandl patch, easy Document IO module 8 days http://bugs.python.org/issue2530 benjamin.peterson patch sphinx.ext.autodoc fails to expand tabs in docstrings 2 days http://bugs.python.org/issue2545 georg.brandl Python-2.5.2: crash in visit_decref () at Modules/gcmodule.c:270 1 days http://bugs.python.org/issue2546 garikvm Python 2.6a2 on Solaris 10 built with SUN C 0 days http://bugs.python.org/issue2551 loewis test_ioctrl failed Python 2.6a2 Solaris 10 SUN C 0 days http://bugs.python.org/issue2553 loewis test_parser failed Python 2.6a2 Solaris 10 SUN C 1 days http://bugs.python.org/issue2555 benjamin.peterson changing sys.dont_write_bytecode has not effect 0 days http://bugs.python.org/issue2556 benjamin.peterson \u and \U in raw strings have regressed in 3.0a4 2 days http://bugs.python.org/issue2557 gvanrossum Document pickle protocol 3 1 days http://bugs.python.org/issue2558 georg.brandl patch Instance remembers old values from former life 0 days http://bugs.python.org/issue2561 georg.brandl Python hangs on FreeBSD7 in test_capi 2 days http://bugs.python.org/issue2564 amaury.forgeotdarc Change 'type' to 'class' in repr/str(builtin-class) 0 days http://bugs.python.org/issue2565 loewis easy Section "New-style and classic classes" needs to be removed/rewr 1 days http://bugs.python.org/issue2567 georg.brandl Reference manual: production for integer literals is missing "bi 2 days http://bugs.python.org/issue2575 georg.brandl patch, patch Misleading 'toctree references unknown document' error 2 days http://bugs.python.org/issue2579 georg.brandl patch Revise builtin class int library manual entry 2 days http://bugs.python.org/issue2580 georg.brandl urllib2.HTTPError broken due to urllib.addinfourl changes 1 days http://bugs.python.org/issue2585 georg.brandl patch Integer signedness bugs in zlib modules 2 days http://bugs.python.org/issue2586 belopolsky 2to3's fix_xrange needs fix_dict's context information 0 days http://bugs.python.org/issue2596 collinwinter "python2.6 -3" should report list.sort(cmp) as DeprecationWarnin 0 days http://bugs.python.org/issue2597 rhettinger importing gtk hangs (freebsd 7.0) 0 days http://bugs.python.org/issue2602 benjamin.peterson trace module crashes due to using wrong sort idiom 0 days http://bugs.python.org/issue2606 loewis patch why is (default)dict so slow on tuples??? 1 days http://bugs.python.org/issue2607 eisele file.tell() returns Long usually, Int if subclassed 0 days http://bugs.python.org/issue2612 benjamin.peterson urllib.urlopen results.readline is slow 2265 days http://bugs.python.org/issue508157 reacocard file (& socket) I/O are not thread safe 2061 days http://bugs.python.org/issue595601 gregory.p.smith thread unsafe file objects cause crash 1650 days http://bugs.python.org/issue815646 gregory.p.smith patch Tkdnd.py crashes due to read-only attributes 1116 days http://bugs.python.org/issue1164742 benjamin.peterson non-admin install may fail (win xp pro) 1007 days http://bugs.python.org/issue1232947 loewis Missing builtin help for with and as 610 days http://bugs.python.org/issue1536059 ncoghlan Add RegEnableReflectionKey and RegDisableReflectionKey 268 days http://bugs.python.org/issue1753245 mhammond patch Top Issues Most Discussed (10) ______________________________ 24 PyString_FromStringAndSize() to be considered unsafe 1 days open http://bugs.python.org/issue2587 18 Cannot use non-ascii letters in disutils if setuptools is used. 5 days open http://bugs.python.org/issue2562 17 Unicode escape sequences not parsed in raw strings. 6 days open http://bugs.python.org/issue2541 15 test_ctypes failed Python 2.6a2 Solaris 10 SUN C 7 days open http://bugs.python.org/issue2552 14 Missing *-unpacking generalizations 27 days open http://bugs.python.org/issue2292 10 SO_REUSEADDR doesn't have the same semantics on Windows as on U 7 days open http://bugs.python.org/issue2550 9 string representation of range 1 days open http://bugs.python.org/issue2610 9 Make range __eq__ work 2 days open http://bugs.python.org/issue2603 9 Document IO module 8 days closed http://bugs.python.org/issue2530 8 PyOS_vsnprintf() underflow leads to memory corruption 3 days open http://bugs.python.org/issue2588 From divinekid at gmail.com Fri Apr 11 18:35:47 2008 From: divinekid at gmail.com (Haoyu Bai) Date: Sat, 12 Apr 2008 00:35:47 +0800 Subject: [Python-Dev] Need help for SWIG's Python 3.0 backend Message-ID: <47FF9363.7060005@gmail.com> Hello, I am a Google Summer of Code student who preparing a SWIG's Python 3.0 support proposal. Here's detail of my proposal: http://www.dabeaz.com/cgi-bin/wiki.pl?GSoCPython3Proposal And abstract shown below for convenient: This project adds Python 3.0 support for SWIG. We will add a "-3" option to SWIG's current backend, which indicates SWIG to generate wrapper for Python 3. We also make SWIG generate more efficient code and more clear proxy by utilizing Python 3's new features. The considered features are as follows: * Function Annotations * Mutable Buffer Support * Abstract Base Classes I have read PEPs and Python 3's document, then did some experiment on the API. I have modified a SWIG generated wrapper code by hand so it can running with Python 3.0. However, there still some API changes I can't handle. SWIG used some undocumented C API, for example the _PyInstance_Lookup(). And some API disappeared, I can't found the alternative of them, for example PyInstance_NewRaw(). I think I will need a lot of help from Python developers if my proposal is accepted. So I post this here to make sure if I can get help when doing this project. And I really appreciate if you can give any advice about how to solve the problems I mentioned before. Thank you! Best regards, Haoyu Bai 4/12/2008 From sergiodj at linux.vnet.ibm.com Fri Apr 11 19:10:02 2008 From: sergiodj at linux.vnet.ibm.com (=?ISO-8859-1?Q?S=E9rgio?= Durigan =?ISO-8859-1?Q?J=FAnior?=) Date: Fri, 11 Apr 2008 14:10:02 -0300 Subject: [Python-Dev] Python 32- and 64-bit living together Message-ID: <1207933802.6420.6.camel@miki> Hi all, My question is simple: is there any problem when installing/using both 32- and 64-bit Python's on the same machine? I'm more concerned about header files (those installed under /usr/include/python-2.x), because as far as I could see there's nothing similar to a "#ifdef USE_64BIT" or something on them. Thanks, -- S?rgio Durigan J?nior Linux on Power Toolchain - Software Engineer Linux Technology Center - LTC IBM Brazil From fwierzbicki at gmail.com Fri Apr 11 19:38:57 2008 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Fri, 11 Apr 2008 13:38:57 -0400 Subject: [Python-Dev] Reserving an arg space for Jython Message-ID: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> Hi all, I was wondering if it might be possible for Jython to get an "arg space" for command line execution. We try to deliver the same switches that Python delivers (so for example -c means "program passed in as string" on Python and Jython). We have some need for arguments that would be Jython-specific, for example we would like to be able to pass arguments to the Java process that starts Jython. I would just make one up, but it would be best if it was future compatible with Python (so -J would be great, but it would be annoying if -J started to mean something in a future Python). So what do guys say? Can we hava -J? I don't think it is being used yet... -Frank From guido at python.org Fri Apr 11 19:40:50 2008 From: guido at python.org (Guido van Rossum) Date: Fri, 11 Apr 2008 10:40:50 -0700 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> Message-ID: Works for me. We should have a patch to CPython that looks for -J and rejects it with "-J is reserved for Jython". On Fri, Apr 11, 2008 at 10:38 AM, Frank Wierzbicki wrote: > Hi all, > > I was wondering if it might be possible for Jython to get an "arg > space" for command line execution. We try to deliver the same > switches that Python delivers (so for example -c means "program passed > in as string" on Python and Jython). We have some need for arguments > that would be Jython-specific, for example we would like to be able to > pass arguments to the Java process that starts Jython. I would just > make one up, but it would be best if it was future compatible with > Python (so -J would be great, but it would be annoying if -J started > to mean something in a future Python). > > So what do guys say? Can we hava -J? I don't think it is being used yet... > > -Frank > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fwierzbicki at gmail.com Fri Apr 11 19:43:39 2008 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Fri, 11 Apr 2008 13:43:39 -0400 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> Message-ID: <4dab5f760804111043x998c271yce8b5da8a909b8d4@mail.gmail.com> On Fri, Apr 11, 2008 at 1:40 PM, Guido van Rossum wrote: > Works for me. We should have a patch to CPython that looks for -J and > rejects it with "-J is reserved for Jython". Great! Knowing this crowd it is probably already implemented -- but if not I'd love to dust off my C skills and upload a patch, I doubt I would cause harm in this area of Python :) -Frank From mal at egenix.com Fri Apr 11 19:38:18 2008 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 11 Apr 2008 19:38:18 +0200 Subject: [Python-Dev] Python 32- and 64-bit living together In-Reply-To: <1207933802.6420.6.camel@miki> References: <1207933802.6420.6.camel@miki> Message-ID: <47FFA20A.4070606@egenix.com> On 2008-04-11 19:10, S?rgio Durigan J?nior wrote: > Hi all, > > My question is simple: is there any problem when installing/using both > 32- and 64-bit Python's on the same machine? I'm more concerned about > header files (those installed under /usr/include/python-2.x), because as > far as I could see there's nothing similar to a "#ifdef USE_64BIT" or > something on them. The include files are all static and can be used on both 32-bit and 64-bit platforms or installations. Only the /usr/lib/python2.x files differ between 32-bit and 64-bit (the configuration files are in /usr/lib/python2.x/config). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 11 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 From sergiodj at linux.vnet.ibm.com Fri Apr 11 20:21:44 2008 From: sergiodj at linux.vnet.ibm.com (=?ISO-8859-1?Q?S=E9rgio?= Durigan =?ISO-8859-1?Q?J=FAnior?=) Date: Fri, 11 Apr 2008 15:21:44 -0300 Subject: [Python-Dev] Python 32- and 64-bit living together In-Reply-To: <47FFA20A.4070606@egenix.com> References: <1207933802.6420.6.camel@miki> <47FFA20A.4070606@egenix.com> Message-ID: <1207938104.6420.15.camel@miki> Hi Lemburg, On Fri, 2008-04-11 at 19:38 +0200, M.-A. Lemburg wrote: > On 2008-04-11 19:10, S?rgio Durigan J?nior wrote: > > Hi all, > > > > My question is simple: is there any problem when installing/using both > > 32- and 64-bit Python's on the same machine? I'm more concerned about > > header files (those installed under /usr/include/python-2.x), because as > > far as I could see there's nothing similar to a "#ifdef USE_64BIT" or > > something on them. > > The include files are all static and can be used on both 32-bit and > 64-bit platforms or installations. Thanks :-). > Only the /usr/lib/python2.x files differ between 32-bit and 64-bit > (the configuration files are in /usr/lib/python2.x/config). Hmm, right. I tried to modify the installation path (using --libdir in ./configure) to /usr/lib64, but some *.pyo objects still are installed under /usr/lib. AFAIK, these objects are bitness-dependent (i.e., if they were generated by a 32-bit Python, they can only be execute by a 32-bit Python - and vice-versa), right? Is there any way to separate these arch-dependent files in /usr/lib and /usr/lib64 depending on their bitness? Thanks, P.S.: I think this "misbehaviour" of --libdir is a bug. IMHO, it should put every arch-dependent file in the path that the user provided. -- S?rgio Durigan J?nior Linux on Power Toolchain - Software Engineer Linux Technology Center - LTC IBM Brazil From fwierzbicki at gmail.com Fri Apr 11 20:49:17 2008 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Fri, 11 Apr 2008 14:49:17 -0400 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <4dab5f760804111043x998c271yce8b5da8a909b8d4@mail.gmail.com> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111043x998c271yce8b5da8a909b8d4@mail.gmail.com> Message-ID: <4dab5f760804111149y5357e875nbbadf3bdd593673@mail.gmail.com> Patch is here: http://bugs.python.org/issue2617 -Frank From brett at python.org Fri Apr 11 20:51:18 2008 From: brett at python.org (Brett Cannon) Date: Fri, 11 Apr 2008 20:51:18 +0200 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> Message-ID: On Fri, Apr 11, 2008 at 7:40 PM, Guido van Rossum wrote: > Works for me. We should have a patch to CPython that looks for -J and > rejects it with "-J is reserved for Jython". > Do we want it to be Jython-specific, or should it be available to any alternative VM? I don't know if the IronPython folks need anything for .NET, but maybe they would like one? -Brett > > > On Fri, Apr 11, 2008 at 10:38 AM, Frank Wierzbicki > wrote: > > Hi all, > > > > I was wondering if it might be possible for Jython to get an "arg > > space" for command line execution. We try to deliver the same > > switches that Python delivers (so for example -c means "program passed > > in as string" on Python and Jython). We have some need for arguments > > that would be Jython-specific, for example we would like to be able to > > pass arguments to the Java process that starts Jython. I would just > > make one up, but it would be best if it was future compatible with > > Python (so -J would be great, but it would be annoying if -J started > > to mean something in a future Python). > > > > So what do guys say? Can we hava -J? I don't think it is being used yet... > > > > -Frank > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From fwierzbicki at gmail.com Fri Apr 11 21:13:44 2008 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Fri, 11 Apr 2008 15:13:44 -0400 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> Message-ID: <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> On Fri, Apr 11, 2008 at 2:51 PM, Brett Cannon wrote: > On Fri, Apr 11, 2008 at 7:40 PM, Guido van Rossum wrote: > > > Works for me. We should have a patch to CPython that looks for -J and > > rejects it with "-J is reserved for Jython". > > > > Do we want it to be Jython-specific, or should it be available to any > alternative VM? I don't know if the IronPython folks need anything for > .NET, but maybe they would like one? -X was suggested on Jython's irc. I kind of like -J, but -X would work for us too. -Frank From dinov at exchange.microsoft.com Fri Apr 11 21:39:58 2008 From: dinov at exchange.microsoft.com (Dino Viehland) Date: Fri, 11 Apr 2008 12:39:58 -0700 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> Message-ID: <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> IronPython already uses -X:OptionName for special IronPython only options so +1 for -X. -----Original Message----- From: python-dev-bounces+dinov=microsoft.com at python.org [mailto:python-dev-bounces+dinov=microsoft.com at python.org] On Behalf Of Frank Wierzbicki Sent: Friday, April 11, 2008 12:14 PM To: Brett Cannon Cc: Guido van Rossum; python-dev at python.org Subject: Re: [Python-Dev] Reserving an arg space for Jython On Fri, Apr 11, 2008 at 2:51 PM, Brett Cannon wrote: > On Fri, Apr 11, 2008 at 7:40 PM, Guido van Rossum wrote: > > > Works for me. We should have a patch to CPython that looks for -J and > > rejects it with "-J is reserved for Jython". > > > > Do we want it to be Jython-specific, or should it be available to any > alternative VM? I don't know if the IronPython folks need anything for > .NET, but maybe they would like one? -X was suggested on Jython's irc. I kind of like -J, but -X would work for us too. -Frank _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/dinov%40microsoft.com From fuzzyman at voidspace.org.uk Fri Apr 11 21:52:36 2008 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 11 Apr 2008 20:52:36 +0100 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> Message-ID: <47FFC184.5000009@voidspace.org.uk> Frank Wierzbicki wrote: > On Fri, Apr 11, 2008 at 2:51 PM, Brett Cannon wrote: > >> On Fri, Apr 11, 2008 at 7:40 PM, Guido van Rossum wrote: >> >> >>> Works for me. We should have a patch to CPython that looks for -J and >>> >> > rejects it with "-J is reserved for Jython". >> > >> >> Do we want it to be Jython-specific, or should it be available to any >> alternative VM? I don't know if the IronPython folks need anything for >> .NET, but maybe they would like one? >> > -X was suggested on Jython's irc. I kind of like -J, but -X would > work for us too. > IronPython has a host of -X:Something command line switches - so reserving -X would be helpful. Michael Foord > -Frank > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > From nnorwitz at gmail.com Fri Apr 11 22:03:25 2008 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 11 Apr 2008 13:03:25 -0700 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> Message-ID: I was also going to suggest a platform independent option. I like -Xwhat-follows-is-impl-dependent. n On Fri, Apr 11, 2008 at 12:39 PM, Dino Viehland wrote: > IronPython already uses -X:OptionName for special IronPython only options so +1 for -X. > > > > -----Original Message----- > From: python-dev-bounces+dinov=microsoft.com at python.org [mailto:python-dev-bounces+dinov=microsoft.com at python.org] On Behalf Of Frank Wierzbicki > Sent: Friday, April 11, 2008 12:14 PM > To: Brett Cannon > Cc: Guido van Rossum; python-dev at python.org > Subject: Re: [Python-Dev] Reserving an arg space for Jython > > On Fri, Apr 11, 2008 at 2:51 PM, Brett Cannon wrote: > > On Fri, Apr 11, 2008 at 7:40 PM, Guido van Rossum wrote: > > > > > Works for me. We should have a patch to CPython that looks for -J and > > > rejects it with "-J is reserved for Jython". > > > > > > > Do we want it to be Jython-specific, or should it be available to any > > alternative VM? I don't know if the IronPython folks need anything for > > .NET, but maybe they would like one? > -X was suggested on Jython's irc. I kind of like -J, but -X would > work for us too. > > -Frank > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/dinov%40microsoft.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > From mal at egenix.com Fri Apr 11 22:06:19 2008 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 11 Apr 2008 22:06:19 +0200 Subject: [Python-Dev] Python 32- and 64-bit living together In-Reply-To: <1207938104.6420.15.camel@miki> References: <1207933802.6420.6.camel@miki> <47FFA20A.4070606@egenix.com> <1207938104.6420.15.camel@miki> Message-ID: <47FFC4BB.3070209@egenix.com> On 2008-04-11 20:21, S?rgio Durigan J?nior wrote: > Hi Lemburg, > > On Fri, 2008-04-11 at 19:38 +0200, M.-A. Lemburg wrote: >> On 2008-04-11 19:10, S?rgio Durigan J?nior wrote: >>> Hi all, >>> >>> My question is simple: is there any problem when installing/using both >>> 32- and 64-bit Python's on the same machine? I'm more concerned about >>> header files (those installed under /usr/include/python-2.x), because as >>> far as I could see there's nothing similar to a "#ifdef USE_64BIT" or >>> something on them. >> The include files are all static and can be used on both 32-bit and >> 64-bit platforms or installations. > > Thanks :-). > >> Only the /usr/lib/python2.x files differ between 32-bit and 64-bit >> (the configuration files are in /usr/lib/python2.x/config). > > Hmm, right. I tried to modify the installation path (using --libdir > in ./configure) to /usr/lib64, but some *.pyo objects still are > installed under /usr/lib. AFAIK, these objects are bitness-dependent > (i.e., if they were generated by a 32-bit Python, they can only be > execute by a 32-bit Python - and vice-versa), right? Right. > Is there any way to separate these arch-dependent files in /usr/lib > and /usr/lib64 depending on their bitness? There's no need for that. Only the config/ dir which is included in the Python lib dir is dependent on the Python configuration. > Thanks, > > P.S.: I think this "misbehaviour" of --libdir is a bug. IMHO, it should > put every arch-dependent file in the path that the user provided. You should probably have a look at how RedHat or openSUSE solve these problems. Some of them have patched Python to fit their needs. You may have to do that as well. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 11 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 From sergiodj at linux.vnet.ibm.com Fri Apr 11 22:25:08 2008 From: sergiodj at linux.vnet.ibm.com (=?ISO-8859-1?Q?S=E9rgio?= Durigan =?ISO-8859-1?Q?J=FAnior?=) Date: Fri, 11 Apr 2008 17:25:08 -0300 Subject: [Python-Dev] Python 32- and 64-bit living together In-Reply-To: <47FFC4BB.3070209@egenix.com> References: <1207933802.6420.6.camel@miki> <47FFA20A.4070606@egenix.com> <1207938104.6420.15.camel@miki> <47FFC4BB.3070209@egenix.com> Message-ID: <1207945509.6420.30.camel@miki> On Fri, 2008-04-11 at 22:06 +0200, M.-A. Lemburg wrote: > > Hmm, right. I tried to modify the installation path (using --libdir > > in ./configure) to /usr/lib64, but some *.pyo objects still are > > installed under /usr/lib. AFAIK, these objects are bitness-dependent > > (i.e., if they were generated by a 32-bit Python, they can only be > > execute by a 32-bit Python - and vice-versa), right? > > Right. > > > Is there any way to separate these arch-dependent files in /usr/lib > > and /usr/lib64 depending on their bitness? > > There's no need for that. Only the config/ dir which is included > in the Python lib dir is dependent on the Python configuration. I'm afraid I still don't understand your point. I mean, if the *.pyo file *is* dependent on the bitness of the Python interpreter (as you confirmed in my first question), therefore when I decide to have both 32- and 64-bit Python on my system I *must* have two versions of every .pyo file: one for 32- and another for 64-bit Python. What I've missed? > You should probably have a look at how RedHat or openSUSE solve > these problems. Some of them have patched Python to fit their > needs. You may have to do that as well. I'll sure take a look at them. Thanks! -- S?rgio Durigan J?nior Linux on Power Toolchain - Software Engineer Linux Technology Center - LTC IBM Brazil From mal at egenix.com Fri Apr 11 22:42:09 2008 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 11 Apr 2008 22:42:09 +0200 Subject: [Python-Dev] Python 32- and 64-bit living together In-Reply-To: <1207945509.6420.30.camel@miki> References: <1207933802.6420.6.camel@miki> <47FFA20A.4070606@egenix.com> <1207938104.6420.15.camel@miki> <47FFC4BB.3070209@egenix.com> <1207945509.6420.30.camel@miki> Message-ID: <47FFCD21.1060206@egenix.com> On 2008-04-11 22:25, S?rgio Durigan J?nior wrote: > On Fri, 2008-04-11 at 22:06 +0200, M.-A. Lemburg wrote: > >>> Hmm, right. I tried to modify the installation path (using --libdir >>> in ./configure) to /usr/lib64, but some *.pyo objects still are >>> installed under /usr/lib. AFAIK, these objects are bitness-dependent >>> (i.e., if they were generated by a 32-bit Python, they can only be >>> execute by a 32-bit Python - and vice-versa), right? >> Right. Sorry, I misread you question. PYO and PYC files are *not* dependent on 32/64 bit sizes. >>> Is there any way to separate these arch-dependent files in /usr/lib >>> and /usr/lib64 depending on their bitness? >> There's no need for that. Only the config/ dir which is included >> in the Python lib dir is dependent on the Python configuration. > > > I'm afraid I still don't understand your point. I mean, if the *.pyo > file *is* dependent on the bitness of the Python interpreter (as you > confirmed in my first question), therefore when I decide to have both > 32- and 64-bit Python on my system I *must* have two versions of > every .pyo file: one for 32- and another for 64-bit Python. What I've > missed? Sorry for the confusion. >> You should probably have a look at how RedHat or openSUSE solve >> these problems. Some of them have patched Python to fit their >> needs. You may have to do that as well. > > I'll sure take a look at them. Thanks! -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 11 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 From musiccomposition at gmail.com Fri Apr 11 23:28:56 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Fri, 11 Apr 2008 16:28:56 -0500 Subject: [Python-Dev] generated NEWS files In-Reply-To: <47FF11F6.5010808@v.loewis.de> References: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> <47FF11F6.5010808@v.loewis.de> Message-ID: <1afaf6160804111428m24e9e94fv25d56f9bbb994b6a@mail.gmail.com> On Fri, Apr 11, 2008 at 2:23 AM, "Martin v. L?wis" wrote: > > While it's > > easy to fix NEWS its not easy to fix changelog entries (only a few svn > > super-users can do it, and it's a pain). > > Actually, any committer can do that: > > svn pe --revprop -rrev svn:log Wow, I'm adding that to my list of things I never dreamed Subversion could do! :P I just tried it, and although it works, I get this output: svn: 'post-revprop-change' hook failed; no error output available Significant? > > It may be difficult to remember, but it's fairly easy to execute. > > Regards, > Martin > -- Cheers, Benjamin Peterson From martin at v.loewis.de Fri Apr 11 23:53:20 2008 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 11 Apr 2008 23:53:20 +0200 Subject: [Python-Dev] generated NEWS files In-Reply-To: <1afaf6160804111428m24e9e94fv25d56f9bbb994b6a@mail.gmail.com> References: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> <47FF11F6.5010808@v.loewis.de> <1afaf6160804111428m24e9e94fv25d56f9bbb994b6a@mail.gmail.com> Message-ID: <47FFDDD0.10900@v.loewis.de> > I just tried it, and although it works, I get this output: > svn: 'post-revprop-change' hook failed; no error output available > > Significant? It's the mailer.py hook failing. I'm not quite sure why it fails, and had no time to look it up. It reads #!/bin/sh REPOS="$1" REV="$2" USER="$3" PROPNAME="$4" /data/repos/projects/hooks/mailer.py propchange "$@" The script is the same mailer.py that is also invoked in post-commit. Can anybody see a problem with that? Regards, Martin From brett at python.org Fri Apr 11 23:54:36 2008 From: brett at python.org (Brett Cannon) Date: Fri, 11 Apr 2008 14:54:36 -0700 Subject: [Python-Dev] generated NEWS files In-Reply-To: <1afaf6160804111428m24e9e94fv25d56f9bbb994b6a@mail.gmail.com> References: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> <47FF11F6.5010808@v.loewis.de> <1afaf6160804111428m24e9e94fv25d56f9bbb994b6a@mail.gmail.com> Message-ID: On Fri, Apr 11, 2008 at 2:28 PM, Benjamin Peterson wrote: > On Fri, Apr 11, 2008 at 2:23 AM, "Martin v. L?wis" wrote: > > > While it's > > > easy to fix NEWS its not easy to fix changelog entries (only a few svn > > > super-users can do it, and it's a pain). > > > > Actually, any committer can do that: > > > > svn pe --revprop -rrev svn:log > Wow, I'm adding that to my list of things I never dreamed Subversion > could do! :P Then you should read http://python.org/dev/faq/ which is essentially a svn FAQ for Python development. -Brett From martin at v.loewis.de Fri Apr 11 23:58:37 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 11 Apr 2008 23:58:37 +0200 Subject: [Python-Dev] Python 32- and 64-bit living together In-Reply-To: <1207933802.6420.6.camel@miki> References: <1207933802.6420.6.camel@miki> Message-ID: <47FFDF0D.6090307@v.loewis.de> > My question is simple: It's also off-topic for python-dev, which is about the development of Python, not the development with Python. > is there any problem when installing/using both > 32- and 64-bit Python's on the same machine? I'm more concerned about > header files (those installed under /usr/include/python-2.x), because as > far as I could see there's nothing similar to a "#ifdef USE_64BIT" or > something on them. You can't install them both into the same prefix (obviously, as they would overwrite their binary files). With two installation, use the header files you got from the 32-bit install for 32-bit extensions/embedding, and the header files you got from the 64-bit install for 64-bit extensions/embedding. The only header file that will actually differ is pyconfig.h. Regards, Martin From sergiodj at linux.vnet.ibm.com Sat Apr 12 00:24:17 2008 From: sergiodj at linux.vnet.ibm.com (=?ISO-8859-1?Q?S=E9rgio?= Durigan =?ISO-8859-1?Q?J=FAnior?=) Date: Fri, 11 Apr 2008 19:24:17 -0300 Subject: [Python-Dev] Python 32- and 64-bit living together In-Reply-To: <47FFDF0D.6090307@v.loewis.de> References: <1207933802.6420.6.camel@miki> <47FFDF0D.6090307@v.loewis.de> Message-ID: <1207952657.6420.43.camel@miki> Hi Martin, On Fri, 2008-04-11 at 23:58 +0200, "Martin v. L?wis" wrote: > > My question is simple: > > It's also off-topic for python-dev, which is about the development > of Python, not the development with Python. With all respect, I think you're confusing things. I'm here trying to find some solutions for Python's build system (which, AFAIK, is about the development *of* Python), which I think is bogus on some systems. I'm just trying to better understand how it works here, and maybe send some patches if I find some solution for my problems. > > is there any problem when installing/using both > > 32- and 64-bit Python's on the same machine? I'm more concerned about > > header files (those installed under /usr/include/python-2.x), because as > > far as I could see there's nothing similar to a "#ifdef USE_64BIT" or > > something on them. > > You can't install them both into the same prefix (obviously, as they > would overwrite their binary files). I was thinking about renaming the binary files, since they're (apparently) the only things that get overwritten *and* are dependent of the bit size (32/64). The rest of the files that would be overwritten, AFAIK, is not dependent of bitness so there's no problem at all. I think this may work (actually, I did the same thing successfuly with Perl). > > With two installation, use the header files you got from the 32-bit > install for 32-bit extensions/embedding, and the header files you > got from the 64-bit install for 64-bit extensions/embedding. The > only header file that will actually differ is pyconfig.h. If I correctly understand what you're saying, you're proposing installing the header files in different places, right? Since these files are the same (except for pyconfig.h, as you said), I assume that the reason for this separation is only for safety. I'll consider your proposal. Regards, -- S?rgio Durigan J?nior Linux on Power Toolchain - Software Engineer Linux Technology Center - LTC IBM Brazil From greg at krypto.org Sat Apr 12 03:31:10 2008 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 11 Apr 2008 18:31:10 -0700 Subject: [Python-Dev] strange import encodings error in trunk? Message-ID: <52dc1c820804111831n78b55960kddf51011f01953bf@mail.gmail.com> has anyone ever seen this error? this is a pristine --with-pydebug build of trunk: >>> msg = 'ABC' >>> x = msg.decode('utf8') Traceback (most recent call last): File "", line 1, in File "/home/gps/python/trunk/Lib/encodings/__init__.py", line 100, in search_function level=0) TypeError: SetupPathsAndImport() got an unexpected keyword argument 'level' -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080411/0283d266/attachment.htm From thomas at python.org Sat Apr 12 04:01:48 2008 From: thomas at python.org (Thomas Wouters) Date: Sat, 12 Apr 2008 04:01:48 +0200 Subject: [Python-Dev] strange import encodings error in trunk? In-Reply-To: <52dc1c820804111831n78b55960kddf51011f01953bf@mail.gmail.com> References: <52dc1c820804111831n78b55960kddf51011f01953bf@mail.gmail.com> Message-ID: <9e804ac0804111901j1cde50abj6761636e5276ade3@mail.gmail.com> On Sat, Apr 12, 2008 at 3:31 AM, Gregory P. Smith wrote: > has anyone ever seen this error? this is a pristine --with-pydebug build > of trunk: > > >>> msg = 'ABC' > >>> x = msg.decode('utf8') > Traceback (most recent call last): > File "", line 1, in > File "/home/gps/python/trunk/Lib/encodings/__init__.py", line 100, in > search_function > level=0) > TypeError: SetupPathsAndImport() got an unexpected keyword argument > 'level' > Your Google sitecustomize is in the way :) Run python with -E. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080412/5dfe1a5e/attachment.htm From zooko at zooko.com Sat Apr 12 06:02:04 2008 From: zooko at zooko.com (zooko) Date: Fri, 11 Apr 2008 21:02:04 -0700 Subject: [Python-Dev] an example of setuptools being used to good effect -- allmydata.org Tahoe References: Message-ID: <3C9044AE-D292-41B6-976A-4313CA8AC174@zooko.com> Folks: I'm sorry, but I am not caught up on the current conversation about packaging. I'm very busy with exciting Python development -- http:// allmydata.com and http://allmydata.org . I understand from PJE's message that he thinks I misunderstand some things about PEP 262; this is entirely possible. I intend to catch up on reading the emails of this conversation and to read carefully PJE's messages about PEP 262 in the coming days. Meanwhile, here is the last message that I wrote before I got swamped with the aforementioned excitement: On Apr 9, 2008, at 5:59 PM, Greg Ewing wrote: > Paul Moore wrote: > >> I believe that Mac OS X goes for an even simpler structure - >> applications store *everything* in the one directory, so that >> install/uninstall is simply a directory copy/remove. > > Yep, and thereby cuts the whole gordian knot, throws the > pieces on the fire and burns them. :-) > > Package managers have always seemed to me to be an > excessively complex solution to a problem that needn't > have existed in the first place. Yes! I love the Zen of the Mac OS X packaging approach. The best install is none at all! (Of course, I also love apt.) > I keep hoping that someday Linux will support something > like MacOSX application bundles and frameworks, but I > haven't seen any sign of it yet. We're slowly approaching this level of simplicity in packaging of the *source code* of Allmydata.org "Tahoe", using setuptools. http://allmydata.org/source/tahoe/trunk/docs/install.html The list of dependencies which are automatically resolved by setuptools is visible here: [1]. It currently includes zfec, foolscap, simplejson, pycryptopp, nevow, zope.interface, twisted, and pywin32. This automatic resolution of dependencies works while fully preserving the user's ability to use their own packages and their own packaging tools. That is: 1. If any of these dependencies are already installed, such as in a Debian package or if the user has installed them by hand, then installing Tahoe will use the already-installed versions and not install anything new, and 2. For any of these dependencies that are not already installed, installing Tahoe will *not* write these dependencies into your standard system directory (which is potentially a sacred place where only your own packaging tool or your root account is allowed to tread) but will instead write them into a local, newly-created install directory from which you can then run Tahoe. (This part is similar in spirit to the Mac OS packaging technique.) Also, this install process never downloads anything from the Internet at install time, per our policy [2, 3], which also happens to be a policy some other people have, e.g. [4, 5]. This works on all of our supported platforms, which includes Linux, Solaris, Windows, Cygwin, and Mac OS X. Oh yes, we also have our buildbot [6] automatically produce Debian packages for edgy, feisty, etch, and gutsy. As far as I know, all of this is accomplished without breaking any of the use cases traditionally associated with setuptools / easy_install, for example "easy_install allmydata-tahoe" works, and if you want "setup.py install" to install into your standard system directory, it will. The reason that I am posting this is to let other programmers know that setuptools is actually a pretty useful tool, even if the use cases that you want to support are incompatible with certain easy_install traditions such as fetching new packages from the internet at buildtime or installing into your system directory. Regards, Zooko P.S. Two days ago I was able to remove twisted from the list of "Manual Dependencies" that people have to be aware of in order to try out Allmydata Tahoe from the source tarball. I think I can safely remove pyOpenSSL now, but that remains to be properly tested by our buildbot. I will be able to remove Crypto++ soon, due to the pycryptopp [7] library. If I can figure out a hack to work-around one of the major frustrations of setuptools (that you can't simply run "./setup.py install --prefix=$FOO"), and if I finish my setuptools plugin to run Twisted trial instead pyunit when "./setup.py test", then I'll be able to remove GNU make from the dependencies. That will leave only g++, Python, and OpenSSL as packages that a programmer has to manually deal with in order to try out Allmydata Tahoe from source. [1] http://allmydata.org/trac/tahoe/browser/_auto_deps.py [2] http://allmydata.org/trac/tahoe/wiki/Packaging [3] http://allmydata.org/trac/tahoe/ticket/229 [4] http://bytes.com/forum/thread666455.html [5] http://fedoraproject.org/wiki/Packaging/Python/Eggs [6] http://allmydata.org/buildbot/waterfall?show_events=false [7] http://allmydata.org/trac/pycryptopp From nnorwitz at gmail.com Sat Apr 12 06:34:06 2008 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 11 Apr 2008 21:34:06 -0700 Subject: [Python-Dev] generated NEWS files In-Reply-To: <47FFDDD0.10900@v.loewis.de> References: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> <47FF11F6.5010808@v.loewis.de> <1afaf6160804111428m24e9e94fv25d56f9bbb994b6a@mail.gmail.com> <47FFDDD0.10900@v.loewis.de> Message-ID: On Fri, Apr 11, 2008 at 2:53 PM, "Martin v. L?wis" wrote: > > I just tried it, and although it works, I get this output: > > svn: 'post-revprop-change' hook failed; no error output available > > > > Significant? > > It's the mailer.py hook failing. I'm not quite sure why it fails, > and had no time to look it up. It reads > > #!/bin/sh > REPOS="$1" > REV="$2" > USER="$3" > PROPNAME="$4" > > /data/repos/projects/hooks/mailer.py propchange "$@" > > The script is the same mailer.py that is also invoked in post-commit. > Can anybody see a problem with that? No, but how many parameters are passed to mailer.py? if cmd == 'commit': if len(sys.argv) > 5: usage() if len(sys.argv) > 4: config_fname = sys.argv[4] elif cmd == 'propchange': if len(sys.argv) < 6 or len(sys.argv) > 7: usage() # ... For commit, sys.argv must be <= 5, but for propchange sys.argv must == 6, unless my tired eyes are reading that wrong. I don't know if the # of args passed are the same in both cases or not. n From asmodai at in-nomine.org Sat Apr 12 09:32:26 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Sat, 12 Apr 2008 09:32:26 +0200 Subject: [Python-Dev] configure on FreeBSD 7 Message-ID: <20080412073226.GL51167@nexus.in-nomine.org> I do not have this problem on FreeBSD 6.3-STABLE, but on my FreeBSD 7.0-STABLE I get this problem after running an identical ./configure: [09:11] [asmodai at oni] (0) {0} % make "Makefile", line 1192: warning: duplicate script for target "Modules/" ignored "Makefile", line 1194: warning: duplicate script for target "Modules/" ignored "Makefile", line 1196: warning: duplicate script for target "Modules/" ignored "Makefile", line 1198: warning: duplicate script for target "Modules/" ignored "Makefile", line 1200: warning: duplicate script for target "Modules/" ignored "Makefile", line 1202: warning: duplicate script for target "Modules/" ignored "Makefile", line 1204: warning: duplicate script for target "Modules/" ignored "Makefile", line 1206: warning: duplicate script for target "Modules/" ignored "Makefile", line 1208: warning: duplicate script for target "Modules/" ignored Graph cycles through thread make: don't know how to make threadmodule.c. Stop line 1192 is: Modules/ thread threadmodule.o: $(srcdir)/Modules/ thread threadmodule.c; $(CC) $(PY_CFLAGS) -c $(srcdir)/Modules/ thread threadmodule.c -o Modules/ thread threadmodule.o On my 6.3 box it became: Modules/threadmodule.o: $(srcdir)/Modules/threadmodule.c; $(CC) $(PY_CFLAGS) -c $(srcdir)/Modules/threadmodule.c -o Modules/threadmodule.o As you can see, some spurious 'thread' mentions got inserted. Apparently this section is made by makedepend. Has anyone else encountered this? -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ No man is good enough to govern another man without the other's consent... From asmodai at in-nomine.org Sat Apr 12 11:10:26 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Sat, 12 Apr 2008 11:10:26 +0200 Subject: [Python-Dev] configure on FreeBSD 7 In-Reply-To: <20080412073226.GL51167@nexus.in-nomine.org> References: <20080412073226.GL51167@nexus.in-nomine.org> Message-ID: <20080412091026.GM51167@nexus.in-nomine.org> -On [20080412 09:32], Jeroen Ruigrok van der Werven (asmodai at in-nomine.org) wrote: >Apparently this section is made by makedepend. I had no updates for makedepend or autotools, but somewhere along the full upgrade of all my ports makedepend suddenly started to behave, so some dependency must have caused the previous breakage. I wonder which though, since I forced a recompile of makedepend before emailing. Peculiar. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ The only trust in this world is that of steel... From asmodai at in-nomine.org Sat Apr 12 12:00:27 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Sat, 12 Apr 2008 12:00:27 +0200 Subject: [Python-Dev] weird configure (autotools) setup Message-ID: <20080412100027.GN51167@nexus.in-nomine.org> Why is CFLAGS in Makefile.pre.in specified as CFLAGS= $(BASECFLAGS) $(OPT) $(EXTRA_CFLAGS) whereas that will negate any CFLAGS you pass to configure? A normal call to configure (as ./configure --help also explains) can contain a CFLAGS specification, e.g.: CFLAGS="-compiler_options" ./configure --config_options Configure happily compiles and tests everything with the working CFLAGS (as evident in config.log), but when it comes to the output substitution it will, of course, not be overridden. Now, I realize that configure.in talks about BASECFLAGS and OPT, but neither of these are documented in the ./configure --help output, nor are they standard when it comes to autotooling. Passing CFLAGS is the defacto standard. Furthermore, when passing compiler options to OPT, these are NOT taken along in the configure tests, which means you might have passed faulty options but configure will not detect any problems with them. You will only encounter this when you start building. So is there any rationale for all this? -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ Contentment that derives from knowing when to be content is eternal contentment... From martin at v.loewis.de Sat Apr 12 13:45:08 2008 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 12 Apr 2008 13:45:08 +0200 Subject: [Python-Dev] weird configure (autotools) setup In-Reply-To: <20080412100027.GN51167@nexus.in-nomine.org> References: <20080412100027.GN51167@nexus.in-nomine.org> Message-ID: <4800A0C4.1020601@v.loewis.de> Jeroen Ruigrok van der Werven wrote: > Why is CFLAGS in Makefile.pre.in specified as > CFLAGS= $(BASECFLAGS) $(OPT) $(EXTRA_CFLAGS) > whereas that will negate any CFLAGS you pass to configure? There is a long history to that. The short version is that configure decides on its own what flags to pass to the compiler; that overrides any CFLAGS passed by the users. There have been multiple patches over the years to resolve this, eventually arriving at the status quo. Regards, Martin From fwierzbicki at gmail.com Sat Apr 12 14:42:00 2008 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Sat, 12 Apr 2008 08:42:00 -0400 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> Message-ID: <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> On Fri, Apr 11, 2008 at 4:03 PM, Neal Norwitz wrote: > I was also going to suggest a platform independent option. I like > -Xwhat-follows-is-impl-dependent. This would work just fine for us, and it makes sense to have it available for all implementations. If everyone likes this I will change the patch for CPython accordingly. How is -X is reserved for non-standard arguments -Frank From ajaksu at gmail.com Sat Apr 12 15:12:18 2008 From: ajaksu at gmail.com (Daniel (ajax) Diniz) Date: Sat, 12 Apr 2008 10:12:18 -0300 Subject: [Python-Dev] Next monthly sprint/bugfix day? In-Reply-To: <20080410151046.GA11041@isnomore.net> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22BEB730@EXMBX04.exchhosting.com> <20080410151046.GA11041@isnomore.net> Message-ID: <2d75d7660804120612r1b1677bfx74747942c52d8111@mail.gmail.com> On Thu, Apr 10, 2008 at 12:10 PM, Rodrigo Bernardo Pimentel wrote: > On Wed, Apr 09 2008 at 11:12:58AM BRT, Trent Nelson wrote: > > Hi, > > > > Is there another online sprint/bugfix day in the pipeline? If not, can > > there be? ;-) > > +1. > > The Sao Paulo User's Group gathered to work on the last Bug Day, and I think > it was very productive (both for generating patches/comments and for us to > get more involved with Python core development). +1 from me. Rodrigo: Do you think GruPy-SP could coordinate with python-brasil to promote local participation in the next Bug Day? Regards, Daniel (from Bras?lia-DF) From rbp at isnomore.net Sat Apr 12 16:04:32 2008 From: rbp at isnomore.net (Rodrigo Bernardo Pimentel) Date: Sat, 12 Apr 2008 11:04:32 -0300 Subject: [Python-Dev] Next monthly sprint/bugfix day? In-Reply-To: <2d75d7660804120612r1b1677bfx74747942c52d8111@mail.gmail.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22BEB730@EXMBX04.exchhosting.com> <20080410151046.GA11041@isnomore.net> <2d75d7660804120612r1b1677bfx74747942c52d8111@mail.gmail.com> Message-ID: <20080412140431.GB10294@isnomore.net> On Sat, Apr 12 2008 at 10:12:18AM BRT, "Daniel (ajax) Diniz" wrote: > On Thu, Apr 10, 2008 at 12:10 PM, Rodrigo Bernardo Pimentel > wrote: > > On Wed, Apr 09 2008 at 11:12:58AM BRT, Trent Nelson wrote: > > > Hi, > > > > > > Is there another online sprint/bugfix day in the pipeline? If not, can > > > there be? ;-) > > > > +1. > > > > The Sao Paulo User's Group gathered to work on the last Bug Day, and I think > > it was very productive (both for generating patches/comments and for us to > > get more involved with Python core development). > > +1 from me. > > Rodrigo: Do you think GruPy-SP could coordinate with python-brasil to > promote local participation in the next Bug Day? Sure. When the next Bug Day is confirmed, we'll take it to python-brasil and the local user's groups. rbp -- Rodrigo Bernardo Pimentel | GPG: <0x0DB14978> From brett at python.org Sat Apr 12 18:12:19 2008 From: brett at python.org (Brett Cannon) Date: Sat, 12 Apr 2008 18:12:19 +0200 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> Message-ID: On Sat, Apr 12, 2008 at 5:42 AM, Frank Wierzbicki wrote: > On Fri, Apr 11, 2008 at 4:03 PM, Neal Norwitz wrote: > > I was also going to suggest a platform independent option. I like > > -Xwhat-follows-is-impl-dependent. > This would work just fine for us, and it makes sense to have it > available for all implementations. If everyone likes this I will > change the patch for CPython accordingly. How is > > -X is reserved for non-standard arguments > Fine by me. -Brett From lists at cheimes.de Sat Apr 12 18:39:01 2008 From: lists at cheimes.de (Christian Heimes) Date: Sat, 12 Apr 2008 18:39:01 +0200 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> Message-ID: <4800E5A5.4040600@cheimes.de> Brett Cannon schrieb: >> -X is reserved for non-standard arguments > > Fine by me. And implemented in r62293 (trunk) Christian From martin at v.loewis.de Sat Apr 12 19:34:58 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 12 Apr 2008 19:34:58 +0200 Subject: [Python-Dev] generated NEWS files In-Reply-To: References: <1afaf6160804101642la035a44k65625bda0ca893e5@mail.gmail.com> <47FF11F6.5010808@v.loewis.de> <1afaf6160804111428m24e9e94fv25d56f9bbb994b6a@mail.gmail.com> <47FFDDD0.10900@v.loewis.de> Message-ID: <4800F2C2.30709@v.loewis.de> >> #!/bin/sh >> REPOS="$1" >> REV="$2" >> USER="$3" >> PROPNAME="$4" >> >> /data/repos/projects/hooks/mailer.py propchange "$@" >> >> The script is the same mailer.py that is also invoked in post-commit. >> Can anybody see a problem with that? > > No, but how many parameters are passed to mailer.py? > > if cmd == 'commit': > if len(sys.argv) > 5: > usage() > if len(sys.argv) > 4: > config_fname = sys.argv[4] > elif cmd == 'propchange': > if len(sys.argv) < 6 or len(sys.argv) > 7: > usage() > # ... > > For commit, sys.argv must be <= 5, but for propchange sys.argv must == > 6, unless my tired eyes are reading that wrong. I don't know if the # > of args passed are the same in both cases or not. I've now looked into this; in Subversion 1.2, the number of arguments passed to post-revprop-change had changed, giving an addition "action" as $5. This broke the old mailer.py, which expected only 4 arguments. The rationale for this additional argument was to be able to tell whether a property had been added, modified, or deleted; if it was added, the old value was supplied as stdin (since revprops themselves aren't versioned). I've now updated mailer.py, merging our (few) changes into it. If anything seems broken now, please let me know. Regards, Martin From skip at pobox.com Sat Apr 12 20:02:02 2008 From: skip at pobox.com (skip at pobox.com) Date: Sat, 12 Apr 2008 13:02:02 -0500 Subject: [Python-Dev] Python 2.4.4/2.4.5 test_pty failure on Solaris 10 Message-ID: <18432.63770.323783.565994@montanaro-dyndns-org.local> I know this is old stuff, but... I want to update our Python 2.4 installation at work from 2.4.2 to 2.4.5 (the latest 2.4 source release). I get a test failure for test_pty, an extra ^M at the end of one line. I don't get a failure in the 2.4.2 installation, but the 2.4.4 and 2.4.5 both fail this test. Looking at the code in test_pty.py, it appears to me that r43570 fixed things for OSF/1 and IRIX which both do weird things with output while breaking things for any other platform by suppressing the \r\n -> \n mapping which used to be performed for all platforms. So, for Solaris, that mapping doesn't happen and the actual and expected outputs don't agree. I'm not suggesting this needs to be fixed, I'm just looking for confirmation of my hypothesis if someone has a moment to compare the two revisions: http://svn.python.org/view/python/branches/release24-maint/Lib/test/test_pty.py?rev=43570&r1=42233&r2=43570 Thanks, Skip From asmodai at in-nomine.org Sat Apr 12 20:09:36 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Sat, 12 Apr 2008 20:09:36 +0200 Subject: [Python-Dev] Python 2.6a2 execution times with various compilers In-Reply-To: <20080411151024.GJ51167@nexus.in-nomine.org> References: <20080411151024.GJ51167@nexus.in-nomine.org> Message-ID: <20080412180936.GP51167@nexus.in-nomine.org> I did some more tests concentrating on GCC, partly based on the feedback I got, results at http://www.in-nomine.org/2008/04/12/python-26-compiler-options-results/ Executive summary: Python needs to be compiled with -O2 or -O3. Not doing so, no optimization level, results with GCC 4.2.1 in a doubling of execution time. Using just -O1 is still ~15% slower than using -O2. Using -mtune=native -march=native can shave of 0,1/0,2 seconds, but otherwise I did not find much difference in using having march or mfpmath present. Profile-guided optimization did not help much, as might be expected, it pushed about the same kind of optimization as the mtune/march combination. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ All know that the drop merges into the ocean but few know that the ocean merges into the drop. From fwierzbicki at gmail.com Sat Apr 12 20:27:57 2008 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Sat, 12 Apr 2008 14:27:57 -0400 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <4800E5A5.4040600@cheimes.de> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> <4800E5A5.4040600@cheimes.de> Message-ID: <4dab5f760804121127l3713a329v1927519864b33d6b@mail.gmail.com> On Sat, Apr 12, 2008 at 12:39 PM, Christian Heimes wrote: > Brett Cannon schrieb: > > >> -X is reserved for non-standard arguments > > > > Fine by me. > > And implemented in r62293 (trunk) Great, thanks! While I'd love to have *both* -X and -J, is that okay with the other devs? -Frank From martin at v.loewis.de Sat Apr 12 22:08:31 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 12 Apr 2008 22:08:31 +0200 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <4dab5f760804121127l3713a329v1927519864b33d6b@mail.gmail.com> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> <4800E5A5.4040600@cheimes.de> <4dab5f760804121127l3713a329v1927519864b33d6b@mail.gmail.com> Message-ID: <480116BF.2040208@v.loewis.de> > Great, thanks! While I'd love to have *both* -X and -J, is that okay > with the other devs? +0. If we ever run out of letters for command line options to have to collect -J, we have deeper problems than having to coordinate with Jython whether the letter is still available. Regards, Martin From greg at krypto.org Sun Apr 13 00:01:32 2008 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 12 Apr 2008 15:01:32 -0700 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? Message-ID: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> http://bugs.python.org/issue1481036 Basically as things are now EOFError is on its own but often wants to be handled the same as other I/O errors that EnvironmentError currently covers. Many uses of EOFError in our code base do not provide it any arguments so it doesn't really fit the (errno, message [, filename]) tuple style that EnvironmentError promises. But we could fudge that with a reasonable default (whats reasonable?) if we rerooted this under EnvironmentError. Alternatively the bug suggests a new parent exception for EnvironmentError and EOFError both to inherit from. Last time changing the heirarchy around this came up there was pushback against adding yet another exception type so I'm thinking the simple re-rooting may be the best answer if anything is done at all. any thoughts? -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080412/569d5180/attachment.htm From greg at krypto.org Sun Apr 13 00:21:32 2008 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 12 Apr 2008 15:21:32 -0700 Subject: [Python-Dev] Python 2.6a2 execution times with various compilers In-Reply-To: <20080412180936.GP51167@nexus.in-nomine.org> References: <20080411151024.GJ51167@nexus.in-nomine.org> <20080412180936.GP51167@nexus.in-nomine.org> Message-ID: <52dc1c820804121521u4c8f69d8q1bf0d0bb9b00e03a@mail.gmail.com> On Sat, Apr 12, 2008 at 11:09 AM, Jeroen Ruigrok van der Werven < asmodai at in-nomine.org> wrote: > I did some more tests concentrating on GCC, partly based on the feedback I > got, results at > http://www.in-nomine.org/2008/04/12/python-26-compiler-options-results/ > > Executive summary: Python needs to be compiled with -O2 or -O3. Not doing > so, no optimization level, results with GCC 4.2.1 in a doubling of > execution > time. Using just -O1 is still ~15% slower than using -O2. > > Using -mtune=native -march=native can shave of 0,1/0,2 seconds, but > otherwise I did not find much difference in using having march or mfpmath > present. > > Profile-guided optimization did not help much, as might be expected, it > pushed about the same kind of optimization as the mtune/march combination. > With gcc 4.1.3 i'm finding that profile guided optimization when trained on pybench or regrtest does make a measurable difference (2-5% overall time with 10-20% on some pybench tests). I haven't run benchmarks enough times to be confident in my results yet, I'll report back with data once I have it. I'm testing both pybench and regrtest as profiling training runs. I will check in a special makefile target for easy gcc profile guided compiles shortly so that those who want faster builds easily produce them. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080412/6bfe80c4/attachment.htm From brett at python.org Sun Apr 13 00:52:42 2008 From: brett at python.org (Brett Cannon) Date: Sat, 12 Apr 2008 15:52:42 -0700 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <480116BF.2040208@v.loewis.de> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> <4800E5A5.4040600@cheimes.de> <4dab5f760804121127l3713a329v1927519864b33d6b@mail.gmail.com> <480116BF.2040208@v.loewis.de> Message-ID: On Sat, Apr 12, 2008 at 1:08 PM, "Martin v. L?wis" wrote: > > Great, thanks! While I'd love to have *both* -X and -J, is that okay > > with the other devs? > > +0. If we ever run out of letters for command line options to have > to collect -J, we have deeper problems than having to coordinate > with Jython whether the letter is still available. +0 -Brett From stephen at xemacs.org Sun Apr 13 01:14:14 2008 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 13 Apr 2008 08:14:14 +0900 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> Message-ID: <87abjyis3d.fsf@uwakimon.sk.tsukuba.ac.jp> Frank Wierzbicki writes: > On Fri, Apr 11, 2008 at 4:03 PM, Neal Norwitz wrote: > > I was also going to suggest a platform independent option. I like > > -Xwhat-follows-is-impl-dependent. > This would work just fine for us, and it makes sense to have it > available for all implementations. If everyone likes this I will > change the patch for CPython accordingly. How is > > -X is reserved for non-standard arguments How about "-X is reserved for implementation-specific arguments"? Ie, I suppose that the intent is not that these arguments won't be standardized, it's that they be standardized by the affected implementations. From martin at v.loewis.de Sun Apr 13 07:27:48 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 13 Apr 2008 07:27:48 +0200 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <87abjyis3d.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> <87abjyis3d.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <480199D4.20701@v.loewis.de> > How about "-X is reserved for implementation-specific arguments"? Ie, > I suppose that the intent is not that these arguments won't be > standardized, it's that they be standardized by the affected > implementations. Isn't that bikeshedding? Regards, Martin From stephen at xemacs.org Sun Apr 13 09:00:21 2008 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 13 Apr 2008 16:00:21 +0900 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <480199D4.20701@v.loewis.de> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> <87abjyis3d.fsf@uwakimon.sk.tsukuba.ac.jp> <480199D4.20701@v.loewis.de> Message-ID: <873apqgry2.fsf@uwakimon.sk.tsukuba.ac.jp> "Martin v. L?wis" writes: > > How about "-X is reserved for implementation-specific arguments"? > Isn't that bikeshedding? No. I think "implementation-specific" is definitely more accurate, and I was hoping the suggestion might get an immediate "good idea, implemented", from somebody already looking at that code. From martin at v.loewis.de Sun Apr 13 10:23:59 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 13 Apr 2008 10:23:59 +0200 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <873apqgry2.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> <87abjyis3d.fsf@uwakimon.sk.tsukuba.ac.jp> <480199D4.20701@v.loewis.de> <873apqgry2.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4801C31F.1030300@v.loewis.de> > > > How about "-X is reserved for implementation-specific arguments"? > > > Isn't that bikeshedding? > > No. I think "implementation-specific" is definitely more accurate, > and I was hoping the suggestion might get an immediate "good idea, > implemented", from somebody already looking at that code. It's already committed, so one would have to go back and change it. Regards, Martin From asmodai at in-nomine.org Sun Apr 13 10:25:03 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Sun, 13 Apr 2008 10:25:03 +0200 Subject: [Python-Dev] Python 2.6a2 execution times with various compilers In-Reply-To: <52dc1c820804121521u4c8f69d8q1bf0d0bb9b00e03a@mail.gmail.com> References: <20080411151024.GJ51167@nexus.in-nomine.org> <20080412180936.GP51167@nexus.in-nomine.org> <52dc1c820804121521u4c8f69d8q1bf0d0bb9b00e03a@mail.gmail.com> Message-ID: <20080413082503.GQ51167@nexus.in-nomine.org> -On [20080413 00:47], Gregory P. Smith (greg at krypto.org) wrote: >With gcc 4.1.3 i'm finding that profile guided optimization when trained on >pybench or regrtest does make a measurable difference (2-5% overall time with >10-20% on some pybench tests). I haven't run benchmarks enough times to be >confident in my results yet, I'll report back with data once I have it. I'm >testing both pybench and regrtest as profiling training runs. It seems GCC 4.2.4 yields worse code for Python with the same options as 4.2.1, I measured about ~7%-8% slowdown (~0,5 seconds) on my test. Granted, in general this might all be nitpicking, but for our friends in the calculating departments this might be quite useful to know. The differences are in general not concentrated in specific sections of pybench, but are uniformly distributed. I know my employer can use such additional free optimizations since our jobs spawn in many hours of execution. Next to optimizing the source code, of course, this will also shave off quite a lot of execution time. >I will check in a special makefile target for easy gcc profile guided compiles >shortly so that those who want faster builds easily produce them. That would be interesting I think. I went with -fprofile-generate and -fprofile-use in my small test. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ Put your heart, mind, intellect and soul even to your smallest acts. This is the secret of success... From turnbull at sk.tsukuba.ac.jp Sun Apr 13 19:28:30 2008 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 14 Apr 2008 02:28:30 +0900 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <4801C31F.1030300@v.loewis.de> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> <87abjyis3d.fsf@uwakimon.sk.tsukuba.ac.jp> <480199D4.20701@v.loewis.de> <873apqgry2.fsf@uwakimon.sk.tsukuba.ac.jp> <4801C31F.1030300@v.loewis.de> Message-ID: <87zlrxfyv5.fsf@uwakimon.sk.tsukuba.ac.jp> "Martin v. L?wis" writes: > > > > How about "-X is reserved for implementation-specific arguments"? > > > > > Isn't that bikeshedding? > > > > No. I think "implementation-specific" is definitely more accurate, > > and I was hoping the suggestion might get an immediate "good idea, > > implemented", from somebody already looking at that code. > > It's already committed, so one would have to go back and change it. Now, *you* are bikeshedding. I know that it was committed, so I politely made the suggestion, half-expecting it to get dropped on the floor. If so, no great loss, and I wouldn't have followed up (except maybe with a patch on the tracker). Is that discouraged on this list? (That's a real question.) From martin at v.loewis.de Sun Apr 13 19:59:30 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 13 Apr 2008 19:59:30 +0200 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <87zlrxfyv5.fsf@uwakimon.sk.tsukuba.ac.jp> References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> <87abjyis3d.fsf@uwakimon.sk.tsukuba.ac.jp> <480199D4.20701@v.loewis.de> <873apqgry2.fsf@uwakimon.sk.tsukuba.ac.jp> <4801C31F.1030300@v.loewis.de> <87zlrxfyv5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <48024A02.5070904@v.loewis.de> > > > > Isn't that bikeshedding? > > > > > > No. I think "implementation-specific" is definitely more accurate, > > > and I was hoping the suggestion might get an immediate "good idea, > > > implemented", from somebody already looking at that code. > > > > It's already committed, so one would have to go back and change it. > > Now, *you* are bikeshedding. > > I know that it was committed, so I politely made the suggestion, > half-expecting it to get dropped on the floor. If so, no great loss, > and I wouldn't have followed up (except maybe with a patch on the > tracker). > > Is that discouraged on this list? (That's a real question.) Making suggestions on the list, and then following up with patch, is certainly encouraged, and happens all the time. I just think that *this* specific proposed change is more effort to talk about than it's worth. It may be more accurate, but changing the phrasing of the message that you get when you invoke a command line option that you shouldn't invoke in the first place, and can't know about unless you've been following this thread or read the source ... is such a minor thing that accuracy doesn't really matter, and is primarily a matter of personal taste. Regards, Martin From lists at cheimes.de Sun Apr 13 20:10:18 2008 From: lists at cheimes.de (Christian Heimes) Date: Sun, 13 Apr 2008 20:10:18 +0200 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: References: <4dab5f760804111038u48617a40k20a376213216ea79@mail.gmail.com> <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> <4800E5A5.4040600@cheimes.de> <4dab5f760804121127l3713a329v1927519864b33d6b@mail.gmail.com> <480116BF.2040208@v.loewis.de> Message-ID: Brett Cannon schrieb: >> +0. If we ever run out of letters for command line options to have >> to collect -J, we have deeper problems than having to coordinate >> with Jython whether the letter is still available. > > +0 Shall I remove the reservation of -J again? Christian From asmodai at in-nomine.org Sun Apr 13 21:27:20 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Sun, 13 Apr 2008 21:27:20 +0200 Subject: [Python-Dev] Reserving an arg space for Jython In-Reply-To: <48024A02.5070904@v.loewis.de> References: <4dab5f760804111213o56ce451cxc7315b5fa088f3@mail.gmail.com> <7AD436E4270DD54A94238001769C2227011DAFFE40CE@DF-GRTDANE-MSG.exchange.corp.microsoft.com> <4dab5f760804120542oeae7e7ek2ab84bbfb0c8223c@mail.gmail.com> <87abjyis3d.fsf@uwakimon.sk.tsukuba.ac.jp> <480199D4.20701@v.loewis.de> <873apqgry2.fsf@uwakimon.sk.tsukuba.ac.jp> <4801C31F.1030300@v.loewis.de> <87zlrxfyv5.fsf@uwakimon.sk.tsukuba.ac.jp> <48024A02.5070904@v.loewis.de> Message-ID: <20080413192720.GV51167@nexus.in-nomine.org> -On [20080413 19:59], "Martin v. L?wis" (martin at v.loewis.de) wrote: >Making suggestions on the list, and then following up with patch, >is certainly encouraged, and happens all the time. Here you go. >I just think that *this* specific proposed change is more effort to >talk about than it's worth. So I wonder why it couldn't have been changed then instead of spending another email on the subject. :P -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ Without you, I am nothing... -------------- next part -------------- A non-text attachment was scrubbed... Name: X.diff Type: text/x-diff Size: 736 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20080413/d2f87a49/attachment.diff From hrvoje.niksic at avl.com Mon Apr 14 15:56:47 2008 From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=) Date: Mon, 14 Apr 2008 15:56:47 +0200 Subject: [Python-Dev] Pickle format machine independence Message-ID: <1208181407.28170.80.camel@localhost> Are pickle files intended to be readable across different machine architectures? The documentation states unequivocally that they are compatible across Python versions, but compatibility across machine architectures (wrt to differences in size and layout of primitive C types) is not explicitly addressed. One example where I stumbled upon the incompatibility is the pickling of arrays. While pickle is normally very careful to write out numbers in a platform-independent way, arrays are written out in "tostring" format. This is filed under http://bugs.python.org/issue2389. I can work around this issue in my application, but if this is considered a bug, I'd prefer to fix it in Python instead. From schmir at gmail.com Mon Apr 14 18:12:35 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Mon, 14 Apr 2008 18:12:35 +0200 Subject: [Python-Dev] very bad network performance Message-ID: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> Hi all, I'm using mercurial with the release25-maint branch. I noticed that checking out a local repository now takes more than 5 minutes (it should be around 30s). I've tracked it down to this change: http://hgpy.de/py/release25-maint/rev/e9446c6ab3cd this is svn revision 61009. Here is the diff inline: --- a/Lib/socket.py Fri Mar 23 14:27:29 2007 +0100 +++ b/Lib/socket.py Sat Feb 23 20:30:59 2008 +0100 @@ -305,7 +305,7 @@ self._rbuf = "" while True: left = size - buf_len - recv_size = max(self._rbufsize, left) + recv_size = min(self._rbufsize, left) data = self._sock.recv(recv_size) if not data: break self._rbufsize if 1, and so the code reads one byte at a time. this is clearly wrong, I'm posting it to the mailing list, as I don't want this issue to get lost in the bugtracker. - Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080414/eeff8dfc/attachment.htm From tjreedy at udel.edu Mon Apr 14 19:25:27 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 14 Apr 2008 13:25:27 -0400 Subject: [Python-Dev] very bad network performance References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> Message-ID: "Ralf Schmitt" wrote in message news:932f8baf0804140912u54adc7d5md7261541857f21bd at mail.gmail.com... | Hi all, | | I'm using mercurial with the release25-maint branch. I noticed that checking | out a local repository now takes more than | 5 minutes (it should be around 30s). | | I've tracked it down to this change: | http://hgpy.de/py/release25-maint/rev/e9446c6ab3cd | this is svn revision 61009. Here is the diff inline: | | --- a/Lib/socket.py Fri Mar 23 14:27:29 2007 +0100 | +++ b/Lib/socket.py Sat Feb 23 20:30:59 2008 +0100 | @@ -305,7 +305,7 @@ | self._rbuf = "" | while True: | left = size - buf_len | - recv_size = max(self._rbufsize, left) | + recv_size = min(self._rbufsize, left) | data = self._sock.recv(recv_size) | if not data: | break | | | | self._rbufsize if 1, and so the code reads one byte at a time. this is | clearly wrong, I'm posting it to the mailing list, as I don't want | this issue to get lost in the bugtracker. -------------------------------------------------------------------------------- It is at least as likely to get lost here. There is a mailing list for new tracker items that many devs subscribe to. From guido at python.org Mon Apr 14 19:56:17 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 14 Apr 2008 10:56:17 -0700 Subject: [Python-Dev] very bad network performance In-Reply-To: References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> Message-ID: Ralf, Terry is right. Please file a bug. I do think there may be a problem with that change but I don't have the time to review it in depth. Hopefully others will. I do recall that sockets reading one byte at a time has been a problem before -- I recall a bug about this in the 1.5.2 era for Windows... Too bad it's back. :-( --Guido On Mon, Apr 14, 2008 at 10:25 AM, Terry Reedy wrote: > > "Ralf Schmitt" wrote in message > news:932f8baf0804140912u54adc7d5md7261541857f21bd at mail.gmail.com... > > > | Hi all, > | > | I'm using mercurial with the release25-maint branch. I noticed that > checking > | out a local repository now takes more than > | 5 minutes (it should be around 30s). > | > | I've tracked it down to this change: > | http://hgpy.de/py/release25-maint/rev/e9446c6ab3cd > | this is svn revision 61009. Here is the diff inline: > | > | --- a/Lib/socket.py Fri Mar 23 14:27:29 2007 +0100 > | +++ b/Lib/socket.py Sat Feb 23 20:30:59 2008 +0100 > | @@ -305,7 +305,7 @@ > | self._rbuf = "" > | while True: > | left = size - buf_len > | - recv_size = max(self._rbufsize, left) > | + recv_size = min(self._rbufsize, left) > | data = self._sock.recv(recv_size) > | if not data: > | break > | > | > | > | self._rbufsize if 1, and so the code reads one byte at a time. this is > | clearly wrong, I'm posting it to the mailing list, as I don't want > | this issue to get lost in the bugtracker. > > -------------------------------------------------------------------------------- > > It is at least as likely to get lost here. There is a mailing list for new > tracker items that many devs subscribe to. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Apr 14 19:59:51 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 14 Apr 2008 10:59:51 -0700 Subject: [Python-Dev] Pickle format machine independence In-Reply-To: <1208181407.28170.80.camel@localhost> References: <1208181407.28170.80.camel@localhost> Message-ID: On Mon, Apr 14, 2008 at 6:56 AM, Hrvoje Nik?i? wrote: > Are pickle files intended to be readable across different machine > architectures? The documentation states unequivocally that they are > compatible across Python versions, but compatibility across machine > architectures (wrt to differences in size and layout of primitive C > types) is not explicitly addressed. They're supposed to be compatible across all architectures. > One example where I stumbled upon the incompatibility is the pickling of > arrays. While pickle is normally very careful to write out numbers in a > platform-independent way, arrays are written out in "tostring" format. > This is filed under http://bugs.python.org/issue2389. > > I can work around this issue in my application, but if this is > considered a bug, I'd prefer to fix it in Python instead. It may not be easy to fix this in a backwards-compatible way, but I agree that there's something fishy there. If you can think of a fix, please do submit a patch! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From curt at hagenlocher.org Mon Apr 14 20:10:12 2008 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Mon, 14 Apr 2008 11:10:12 -0700 Subject: [Python-Dev] very bad network performance In-Reply-To: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> Message-ID: On Mon, Apr 14, 2008 at 9:12 AM, Ralf Schmitt wrote: > > I've tracked it down to this change: > http://hgpy.de/py/release25-maint/rev/e9446c6ab3cd > this is svn revision 61009. > [...] > self._rbufsize if 1, and so the code reads one byte at a time The change is correct, but exposes a flaw earlier in the same method. "_rbufsize == 1" represents a request to buffer "by line", which is clearly irrelevant in this context. A request to read n bytes should just use the default buffer size if buffering "by line". Sample patch is attached. -- Curt Hagenlocher curt at hagenlocher.org -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: socket.py.diff Url: http://mail.python.org/pipermail/python-dev/attachments/20080414/4003efd8/attachment.txt From guido at python.org Mon Apr 14 20:22:05 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 14 Apr 2008 11:22:05 -0700 Subject: [Python-Dev] very bad network performance In-Reply-To: References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> Message-ID: Eek! Please use the bug tracker. On Mon, Apr 14, 2008 at 11:10 AM, Curt Hagenlocher wrote: > On Mon, Apr 14, 2008 at 9:12 AM, Ralf Schmitt wrote: > > > > I've tracked it down to this change: > > http://hgpy.de/py/release25-maint/rev/e9446c6ab3cd > > this is svn revision 61009. > > [...] > > > self._rbufsize if 1, and so the code reads one byte at a time > > The change is correct, but exposes a flaw earlier in the same method. > "_rbufsize == 1" represents a request to buffer "by line", which is > clearly irrelevant in this context. A request to read n bytes should > just use the default buffer size if buffering "by line". Sample patch > is attached. > > > -- > Curt Hagenlocher > curt at hagenlocher.org > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From janssen at parc.com Mon Apr 14 20:36:33 2008 From: janssen at parc.com (Bill Janssen) Date: Mon, 14 Apr 2008 11:36:33 PDT Subject: [Python-Dev] very bad network performance In-Reply-To: References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> Message-ID: <08Apr14.113640pdt."58696"@synergy1.parc.xerox.com> There's some really convoluted code in socket._fileobject.__init__() here. When initializing a _fileobject, if the 'bufsize' parameter is explicitly given as zero, that's turned into an _rbufsize of 1, which, combined with the 'min' change, will produce the read-one-byte behavior. The code for setting _rbufsize seems odd; be nice if it was commented with some notes on why these specific selections were made. if bufsize < 0: bufsize = self.default_bufsize if bufsize == 0: self._rbufsize = 1 elif bufsize == 1: self._rbufsize = self.default_bufsize else: self._rbufsize = bufsize self._wbufsize = bufsize It also depends on whether 'read' is called with an explicit # of bytes to read (which appears to be the case here). So, it's not the code in socket.py, necessarily; it's the code which opens the socket, most likely. The only library which seems to use a bufsize of zero is httplib (which has a lot of other problems as well). I think the change cited below (while IMO correct) will affect a number of other HTTP-based services, as well. Bill > Ralf, > > Terry is right. Please file a bug. I do think there may be a problem > with that change but I don't have the time to review it in depth. > Hopefully others will. I do recall that sockets reading one byte at a > time has been a problem before -- I recall a bug about this in the > 1.5.2 era for Windows... Too bad it's back. :-( > > --Guido > > On Mon, Apr 14, 2008 at 10:25 AM, Terry Reedy wrote: > > > > "Ralf Schmitt" wrote in message > > news:932f8baf0804140912u54adc7d5md7261541857f21bd at mail.gmail.com... > > > > > > | Hi all, > > | > > | I'm using mercurial with the release25-maint branch. I noticed that > > checking > > | out a local repository now takes more than > > | 5 minutes (it should be around 30s). > > | > > | I've tracked it down to this change: > > | http://hgpy.de/py/release25-maint/rev/e9446c6ab3cd > > | this is svn revision 61009. Here is the diff inline: > > | > > | --- a/Lib/socket.py Fri Mar 23 14:27:29 2007 +0100 > > | +++ b/Lib/socket.py Sat Feb 23 20:30:59 2008 +0100 > > | @@ -305,7 +305,7 @@ > > | self._rbuf = "" > > | while True: > > | left = size - buf_len > > | - recv_size = max(self._rbufsize, left) > > | + recv_size = min(self._rbufsize, left) > > | data = self._sock.recv(recv_size) > > | if not data: > > | break > > | > > | > > | > > | self._rbufsize if 1, and so the code reads one byte at a time. this is > > | clearly wrong, I'm posting it to the mailing list, as I don't want > > | this issue to get lost in the bugtracker. > > > > -------------------------------------------------------------------------------- > > > > It is at least as likely to get lost here. There is a mailing list for new > > tracker items that many devs subscribe to. From guido at python.org Mon Apr 14 21:12:27 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 14 Apr 2008 12:12:27 -0700 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? In-Reply-To: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> Message-ID: Offhand, -0. I don't think of EOFError as an environmental error. Its primary purpose was to have something raised by raw_input() (in 3.0, input()) when there is no more input. This is quite a different level of error than what EnvironmentError typically means (a problem in the filesystem or network, or a permissions thing). On Sat, Apr 12, 2008 at 3:01 PM, Gregory P. Smith wrote: > http://bugs.python.org/issue1481036 > > Basically as things are now EOFError is on its own but often wants to be > handled the same as other I/O errors that EnvironmentError currently covers. > > Many uses of EOFError in our code base do not provide it any arguments so it > doesn't really fit the (errno, message [, filename]) tuple style that > EnvironmentError promises. But we could fudge that with a reasonable > default (whats reasonable?) if we rerooted this under EnvironmentError. > > Alternatively the bug suggests a new parent exception for EnvironmentError > and EOFError both to inherit from. > > Last time changing the heirarchy around this came up there was pushback > against adding yet another exception type so I'm thinking the simple > re-rooting may be the best answer if anything is done at all. > > any thoughts? > > -gps > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From amk at amk.ca Mon Apr 14 21:17:25 2008 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 14 Apr 2008 15:17:25 -0400 Subject: [Python-Dev] very bad network performance In-Reply-To: References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> Message-ID: <20080414191725.GA23517@amk-desktop.matrixgroup.net> On Mon, Apr 14, 2008 at 11:10:12AM -0700, Curt Hagenlocher wrote: > while True: > left = size - buf_len > ! recv_size = max(self._rbufsize, left) > data = self._sock.recv(recv_size) What version is this patch against? (The last 2.5 release, maybe?) The max() in the above line should be min(), because you want to use the *smaller* number of the buffer size and the # of remaining bytes to read, not the *larger*. This code is using min() in both 25-maint and trunk. --amk From curt at hagenlocher.org Mon Apr 14 21:34:35 2008 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Mon, 14 Apr 2008 12:34:35 -0700 Subject: [Python-Dev] very bad network performance In-Reply-To: <20080414191725.GA23517@amk-desktop.matrixgroup.net> References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> <20080414191725.GA23517@amk-desktop.matrixgroup.net> Message-ID: On Mon, Apr 14, 2008 at 12:17 PM, A.M. Kuchling wrote: > On Mon, Apr 14, 2008 at 11:10:12AM -0700, Curt Hagenlocher wrote: > > while True: > > left = size - buf_len > > ! recv_size = max(self._rbufsize, left) > > data = self._sock.recv(recv_size) > > What version is this patch against? (The last 2.5 release, maybe?) Yes, sorry. I thought I had checked this against the repository -- particularly because the max->min change is what kicked off this thread. -- Curt Hagenlocher curt at hagenlocher.org From bonelake at gmail.com Mon Apr 14 21:45:14 2008 From: bonelake at gmail.com (Brad Miller) Date: Mon, 14 Apr 2008 14:45:14 -0500 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> Message-ID: <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> After posting a patch to implement this some good discussion followed see: http://bugs.python.org/issue2610 It was suggested that a broader discussion might be in order around the issue of iterators and how they are displayed in the command line interpreter. Several new iterators have appeared in Python 3.0 that makes the language less transparent to beginning programmers. The examples that immediately come to mind are shown below, and I would guess there are others I haven't run across yet. >>> range(10) range(0, 10) >>> myd = {chr(i):i for i in range(32,42)} >>> myd.keys() >>> myd.values() >>> myd.items() Although none of the above are a problem for intermediate or advanced programmers I would like to find a way so that beginning students would automatically get a more helpful representation when they evaluate expressions in the interpreter. My solution of implementing the __str__ method for range is one solution, and that could also be done for the dict_xxx objects as well. Other solutions that were suggested were to include some kind of a module that overrides sys.displayhook or to simply make the command line interpreter more intelligence. For example it already handles a return value of None in a special way, maybe it should do something for these iterators as well. Any other comments or ideas? Thanks, Brad On Apr 7, 2008, at 6:24 PM, Guido van Rossum wrote: > I'd object to it returning something that resembles a list too > closely, but I could live with str(range(3)) return <0, 1, 2>. We > should probably have a cutoff so that if there are more than 6 values > it'll show the first 3 values, then dots, then the last 2 values. (The > cutoff would be computed so that '...' always represents at least 2 > values. > > On Mon, Apr 7, 2008 at 4:14 PM, Brad Miller > wrote: >> Hi, >> >> I use Python in my CS1 and CS2 curriculum and I have a question. >> As I've been using the Python 3.0 alphas one of the things that I am >> bothered by is that I cannot see the sequence produced by range >> without introducing students to the list() function. >> >> I typically introduce range on day 1 of class and show students what >> it produces without making a big deal out of the fact that it creates >> a list. They all accept this and things work out nicely when I >> introduce lists for real in a week or two. >> >> My question is why couldn't the __str__ method for the range object >> be >> more friendly and show a representation of the sequence? I >> understand >> why __repr__ should return range(0,10) for an object created using >> range(10) but couldn't print(range(10)) produce [0, 1, 2, ... 9] >> The ... could even be used if the sequence were excessively wrong. >> >> If this is acceptable, I would be happy to accept the challenge of >> providing a patch. >> >> Thanks, >> >> Brad >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080414/be28e179/attachment.htm From lists at cheimes.de Mon Apr 14 23:05:57 2008 From: lists at cheimes.de (Christian Heimes) Date: Mon, 14 Apr 2008 23:05:57 +0200 Subject: [Python-Dev] r62342 - python/branches/py3k/Objects/bytesobject.c In-Reply-To: <20080414205105.B45C11E4005@bag.python.org> References: <20080414205105.B45C11E4005@bag.python.org> Message-ID: <4803C735.10100@cheimes.de> alexandre.vassalotti schrieb: > Author: alexandre.vassalotti > Date: Mon Apr 14 22:51:05 2008 > New Revision: 62342 > > Log: > Improved bytes_extend() to avoid making a full copy of the temporary > buffer. This also makes the code slightly cleaner. Changes to Objects/bytesobject.c should be applied to the trunk and merged into the py3k branch via svnmerge.py. We need to agree on a policy how we are going to sync the trunk and py3k for new code like bytesobject.c and io.py. The former is easy because the file is almost identical. The later is going to be hard because 2.6 doesn't have annotations. Collin: How hard is it to write a fixer that removes all annotations from functions? A set of small 3to2 fixers for annotations and metaclasses would make the syncing job much easier. Christian From collinw at gmail.com Mon Apr 14 23:18:55 2008 From: collinw at gmail.com (Collin Winter) Date: Mon, 14 Apr 2008 14:18:55 -0700 Subject: [Python-Dev] r62342 - python/branches/py3k/Objects/bytesobject.c In-Reply-To: <4803C735.10100@cheimes.de> References: <20080414205105.B45C11E4005@bag.python.org> <4803C735.10100@cheimes.de> Message-ID: <43aa6ff70804141418w414cbe0bl8eb82ea77e8559a1@mail.gmail.com> On Mon, Apr 14, 2008 at 2:05 PM, Christian Heimes wrote: > alexandre.vassalotti schrieb: > > > Author: alexandre.vassalotti > > Date: Mon Apr 14 22:51:05 2008 > > New Revision: 62342 > > > > Log: > > Improved bytes_extend() to avoid making a full copy of the temporary > > buffer. This also makes the code slightly cleaner. > > Changes to Objects/bytesobject.c should be applied to the trunk and > merged into the py3k branch via svnmerge.py. > > We need to agree on a policy how we are going to sync the trunk and py3k > for new code like bytesobject.c and io.py. The former is easy because > the file is almost identical. The later is going to be hard because 2.6 > doesn't have annotations. > > Collin: > How hard is it to write a fixer that removes all annotations from > functions? A set of small 3to2 fixers for annotations and metaclasses > would make the syncing job much easier. It should be pretty easy. I'm working on a 2to3 metaclass fixer, which could provide guidance for a 3to2 fixer. Do we want to take this opportunity to create a real 3to2 project in the sandbox? If so, I'd like to refactor lib2to3 into its own project, then import that into 2to3 and 3to2. I can do the work. Collin From schmir at gmail.com Mon Apr 14 23:18:15 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Mon, 14 Apr 2008 23:18:15 +0200 Subject: [Python-Dev] very bad network performance In-Reply-To: References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> Message-ID: <932f8baf0804141418q2b22bb97k74c6e12c2e9518b@mail.gmail.com> On Mon, Apr 14, 2008 at 8:22 PM, Guido van Rossum wrote: > Eek! Please use the bug tracker. > I 've made some comments on: http://bugs.python.org/issue1092502 (which is the original issue). However I cannot reopen this issue. Regards, - Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080414/36746efb/attachment-0001.htm From martin at v.loewis.de Mon Apr 14 23:26:26 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 14 Apr 2008 23:26:26 +0200 Subject: [Python-Dev] r62342 - python/branches/py3k/Objects/bytesobject.c In-Reply-To: <43aa6ff70804141418w414cbe0bl8eb82ea77e8559a1@mail.gmail.com> References: <20080414205105.B45C11E4005@bag.python.org> <4803C735.10100@cheimes.de> <43aa6ff70804141418w414cbe0bl8eb82ea77e8559a1@mail.gmail.com> Message-ID: <4803CC02.8060400@v.loewis.de> > Do we want to take this opportunity to create a real 3to2 project in > the sandbox? If so, I'd like to refactor lib2to3 into its own project, > then import that into 2to3 and 3to2. I can do the work. In that case, I would propose that the copy in the Python trunk becomes official, and the other copies use svn:externals, or merge tracking. Regards, Martin From schmir at gmail.com Mon Apr 14 23:29:21 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Mon, 14 Apr 2008 23:29:21 +0200 Subject: [Python-Dev] very bad network performance In-Reply-To: References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> Message-ID: <932f8baf0804141429v666672a0j8833d8bcd3cd44d5@mail.gmail.com> On Mon, Apr 14, 2008 at 8:10 PM, Curt Hagenlocher wrote: > On Mon, Apr 14, 2008 at 9:12 AM, Ralf Schmitt wrote: > > > > I've tracked it down to this change: > > http://hgpy.de/py/release25-maint/rev/e9446c6ab3cd > > this is svn revision 61009. > > [...] > > self._rbufsize if 1, and so the code reads one byte at a time > > The change is correct, but exposes a flaw earlier in the same method. > "_rbufsize == 1" represents a request to buffer "by line", which is > clearly irrelevant in this context. A request to read n bytes should > just use the default buffer size if buffering "by line". Sample patch > is attached. > Sorry to reply on the mailing list. But this change is wrong. e.g. if you're using a buffer size of 16 bytes and try to read 256 bytes, it should call recv with a value of 256 and not call recv 16 times with a value of 16. However, there should be an upper limit (as shown by the imap bug). Regards, - Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080414/34506a5b/attachment.htm From collinw at gmail.com Mon Apr 14 23:35:16 2008 From: collinw at gmail.com (Collin Winter) Date: Mon, 14 Apr 2008 14:35:16 -0700 Subject: [Python-Dev] r62342 - python/branches/py3k/Objects/bytesobject.c In-Reply-To: <4803CC02.8060400@v.loewis.de> References: <20080414205105.B45C11E4005@bag.python.org> <4803C735.10100@cheimes.de> <43aa6ff70804141418w414cbe0bl8eb82ea77e8559a1@mail.gmail.com> <4803CC02.8060400@v.loewis.de> Message-ID: <43aa6ff70804141435h2d487790u70e4fed0c7cfbdee@mail.gmail.com> On Mon, Apr 14, 2008 at 2:26 PM, "Martin v. L?wis" wrote: > > Do we want to take this opportunity to create a real 3to2 project in > > the sandbox? If so, I'd like to refactor lib2to3 into its own project, > > then import that into 2to3 and 3to2. I can do the work. > > In that case, I would propose that the copy in the Python trunk becomes > official, and the other copies use svn:externals, or merge tracking. Can do. From schmir at gmail.com Mon Apr 14 23:45:21 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Mon, 14 Apr 2008 23:45:21 +0200 Subject: [Python-Dev] very bad network performance In-Reply-To: <932f8baf0804141418q2b22bb97k74c6e12c2e9518b@mail.gmail.com> References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> <932f8baf0804141418q2b22bb97k74c6e12c2e9518b@mail.gmail.com> Message-ID: <932f8baf0804141445k31ead5baic0201759a49d6714@mail.gmail.com> On Mon, Apr 14, 2008 at 11:18 PM, Ralf Schmitt wrote: > > On Mon, Apr 14, 2008 at 8:22 PM, Guido van Rossum > wrote: > > > Eek! Please use the bug tracker. > > > > I 've made some comments on: http://bugs.python.org/issue1092502 (which is > the original issue). However I cannot reopen this issue. > Curt opened another bug for it: http://bugs.python.org/issue2632 someone should change the priority. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080414/423d92dd/attachment.htm From lists at cheimes.de Mon Apr 14 23:56:41 2008 From: lists at cheimes.de (Christian Heimes) Date: Mon, 14 Apr 2008 23:56:41 +0200 Subject: [Python-Dev] r62342 - python/branches/py3k/Objects/bytesobject.c In-Reply-To: <43aa6ff70804141418w414cbe0bl8eb82ea77e8559a1@mail.gmail.com> References: <20080414205105.B45C11E4005@bag.python.org> <4803C735.10100@cheimes.de> <43aa6ff70804141418w414cbe0bl8eb82ea77e8559a1@mail.gmail.com> Message-ID: <4803D319.5010000@cheimes.de> Collin Winter schrieb: > It should be pretty easy. I'm working on a 2to3 metaclass fixer, which > could provide guidance for a 3to2 fixer. > > Do we want to take this opportunity to create a real 3to2 project in > the sandbox? If so, I'd like to refactor lib2to3 into its own project, > then import that into 2to3 and 3to2. I can do the work. Good idea! A 3to2 project is going to make backporting io.py and other new stuff less painful and much faster. For the io.py backport I spent most time on removing annotations and replacing "" with u"". What needs to be done? * remove funtion annotation * Add object to all empty class definition * replace class Egg(metaclass=Spam) with class Egg(object):\n__metaclass__ = Spam * Add __future__ imports for print_function and unicode literals Anything else? Christian From curt at hagenlocher.org Tue Apr 15 00:19:35 2008 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Mon, 14 Apr 2008 15:19:35 -0700 Subject: [Python-Dev] very bad network performance In-Reply-To: <932f8baf0804141429v666672a0j8833d8bcd3cd44d5@mail.gmail.com> References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> <932f8baf0804141429v666672a0j8833d8bcd3cd44d5@mail.gmail.com> Message-ID: On Mon, Apr 14, 2008 at 2:29 PM, Ralf Schmitt wrote: > > Sorry to reply on the mailing list. But this change is wrong. > e.g. if you're using a buffer size of 16 bytes and try to read 256 bytes, it > should call recv with a value of 256 and not call recv 16 times with a value > of 16. > However, there should be an upper limit (as shown by the imap bug). There is an upper limit. It's called "the buffer size". If someone specifies a buffer size of 16 bytes, it means "read 16 bytes at a time". I don't know why someone would want such a small buffer size, but presumably they have their reasons. The only reason "min" is a problem is that there's standard library code passing a zero to socket.makefile, which gets turned into a bufsize of 1 by the constructor. I actually agree with Bill Janssen -- __init__ is where the real problem lies. But I think the change to read() is safer. -- Curt Hagenlocher curt at hagenlocher.org From schmir at gmail.com Tue Apr 15 00:57:05 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Tue, 15 Apr 2008 00:57:05 +0200 Subject: [Python-Dev] very bad network performance In-Reply-To: References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> <932f8baf0804141429v666672a0j8833d8bcd3cd44d5@mail.gmail.com> Message-ID: <932f8baf0804141557u5d1a06e6q7112dd4d606e079c@mail.gmail.com> On Tue, Apr 15, 2008 at 12:19 AM, Curt Hagenlocher wrote: > On Mon, Apr 14, 2008 at 2:29 PM, Ralf Schmitt wrote: > > > > Sorry to reply on the mailing list. But this change is wrong. > > e.g. if you're using a buffer size of 16 bytes and try to read 256 > bytes, it > > should call recv with a value of 256 and not call recv 16 times with a > value > > of 16. > > However, there should be an upper limit (as shown by the imap bug). > > There is an upper limit. It's called "the buffer size". If someone > specifies a buffer size of 16 bytes, it means "read 16 bytes at a > time". I don't know why someone would want such a small buffer size, > but presumably they have their reasons. > No, I don't agree. To me buffer size means buffer up to buffer_size bytes in memory. It does not mean that it should read only buffer_size bytes at once when asked to read more bytes than buffer size. The upper limit I was talking about is the buffer size limit of the operating system, i.e. the operating system will at a maximum return N bytes from recv call. It doesn't make sense to ask for more then, and the original problem with imaplip asking for 10MB of data and then realloc'ing that buffer would be gone. > The only reason "min" is a problem is that there's standard library > code passing a zero to socket.makefile, which gets turned into a > bufsize of 1 by the constructor. I actually agree with Bill Janssen > -- __init__ is where the real problem lies. But I think the change to > read() is safer. > again no, if I pass in 4 as buffer size, I don't expect the system to make 1024 calls to recv when I want to read 4096 bytes. Regards, - Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080415/dba04a91/attachment.htm From guido at python.org Tue Apr 15 01:19:44 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 14 Apr 2008 16:19:44 -0700 Subject: [Python-Dev] very bad network performance In-Reply-To: <932f8baf0804141557u5d1a06e6q7112dd4d606e079c@mail.gmail.com> References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> <932f8baf0804141429v666672a0j8833d8bcd3cd44d5@mail.gmail.com> <932f8baf0804141557u5d1a06e6q7112dd4d606e079c@mail.gmail.com> Message-ID: On Mon, Apr 14, 2008 at 3:57 PM, Ralf Schmitt wrote: > > > > On Tue, Apr 15, 2008 at 12:19 AM, Curt Hagenlocher > wrote: > > > > On Mon, Apr 14, 2008 at 2:29 PM, Ralf Schmitt wrote: > > > > > > Sorry to reply on the mailing list. But this change is wrong. > > > e.g. if you're using a buffer size of 16 bytes and try to read 256 > bytes, it > > > should call recv with a value of 256 and not call recv 16 times with a > value > > > of 16. > > > However, there should be an upper limit (as shown by the imap bug). > > > > There is an upper limit. It's called "the buffer size". If someone > > specifies a buffer size of 16 bytes, it means "read 16 bytes at a > > time". I don't know why someone would want such a small buffer size, > > but presumably they have their reasons. > > > > No, I don't agree. To me buffer size means buffer up to buffer_size bytes in > memory. > It does not mean that it should read only buffer_size bytes at once when > asked to read more bytes than buffer size. > > The upper limit I was talking about is the buffer size limit of the > operating system, i.e. the operating system will at a maximum return N bytes > from recv call. It doesn't make sense to ask for more then, and the original > problem with imaplip asking for 10MB of data and then realloc'ing that > buffer would be gone. > > > > > > The only reason "min" is a problem is that there's standard library > > code passing a zero to socket.makefile, which gets turned into a > > bufsize of 1 by the constructor. I actually agree with Bill Janssen > > -- __init__ is where the real problem lies. But I think the change to > > read() is safer. > > > > again no, if I pass in 4 as buffer size, I don't expect the system to make > 1024 calls to recv when I want to read 4096 bytes. But why was imaplib apparently specifying 10MB? Did it know there was that much data? Or did it just not want to bother looping over all the data in smaller buffer increments (e.g. 64K, which is probably the max of what most TCP stacks will give you)? If I'm right with my hunch that the TCP stack will probably clamp at 64K, perhaps we should use min(system limit, max(requested size, buffer size))? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From schmir at gmail.com Tue Apr 15 01:28:06 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Tue, 15 Apr 2008 01:28:06 +0200 Subject: [Python-Dev] very bad network performance In-Reply-To: References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> <932f8baf0804141429v666672a0j8833d8bcd3cd44d5@mail.gmail.com> <932f8baf0804141557u5d1a06e6q7112dd4d606e079c@mail.gmail.com> Message-ID: <932f8baf0804141628k19993582jeabca7d31806f2e1@mail.gmail.com> On Tue, Apr 15, 2008 at 1:19 AM, Guido van Rossum wrote: > > But why was imaplib apparently specifying 10MB? Did it know there was > that much data? Or did it just not want to bother looping over all the > data in smaller buffer increments (e.g. 64K, which is probably the max > of what most TCP stacks will give you)? > Well, calling read with a size of 10MB should be possible. The problem is that this value ended up inside the recv call, which then did the malloc/realloc calls. > > If I'm right with my hunch that the TCP stack will probably clamp at > 64K, perhaps we should use min(system limit, max(requested size, > buffer size))? > yes, this is what I was trying to explain. > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/ > ) > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080415/1271d926/attachment.htm From curt at hagenlocher.org Tue Apr 15 01:41:27 2008 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Mon, 14 Apr 2008 16:41:27 -0700 Subject: [Python-Dev] very bad network performance In-Reply-To: References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> <932f8baf0804141429v666672a0j8833d8bcd3cd44d5@mail.gmail.com> <932f8baf0804141557u5d1a06e6q7112dd4d606e079c@mail.gmail.com> Message-ID: On Mon, Apr 14, 2008 at 4:19 PM, Guido van Rossum wrote: > > But why was imaplib apparently specifying 10MB? Did it know there was > that much data? Or did it just not want to bother looping over all the > data in smaller buffer increments (e.g. 64K, which is probably the max > of what most TCP stacks will give you)? I'm going to guess that the code in question is size = int(self.mo.group('size')) if __debug__: if self.debug >= 4: self._mesg('read literal size %s' % size) data = self.read(size) It's reading however many bytes are reported by the server as the size. > If I'm right with my hunch that the TCP stack will probably clamp at > 64K, perhaps we should use min(system limit, max(requested size, > buffer size))? I have indeed missed the point of the read buffer size. This would work. -- Curt Hagenlocher curt at hagenlocher.org From greg.ewing at canterbury.ac.nz Tue Apr 15 03:59:36 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 15 Apr 2008 13:59:36 +1200 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? In-Reply-To: References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> Message-ID: <48040C08.504@canterbury.ac.nz> Guido van Rossum wrote: > I don't think of EOFError as an environmental error... This is quite > a different level of error than what EnvironmentError typically means I think it depends. Any "expected" EOFErrors are going to be caught by the surrounding code before propagating very far. An *uncaught* EOFError probably means that a file was shorter than you expected it to be, which counts as an environmental error to my way of thinking. My current coding style involves wrapping an "except EnvironmentError" around any major operation and reporting it as a "File could not be read/written/whatever because..." kind of message. Having EOFError get missed by that would be a nuisance. -- Greg From guido at python.org Tue Apr 15 04:11:26 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 14 Apr 2008 19:11:26 -0700 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? In-Reply-To: <48040C08.504@canterbury.ac.nz> References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> <48040C08.504@canterbury.ac.nz> Message-ID: On Mon, Apr 14, 2008 at 6:59 PM, Greg Ewing wrote: > Guido van Rossum wrote: > > I don't think of EOFError as an environmental error... This is quite > > > a different level of error than what EnvironmentError typically means > > I think it depends. Any "expected" EOFErrors are going to be > caught by the surrounding code before propagating very far. > An *uncaught* EOFError probably means that a file was shorter > than you expected it to be, which counts as an environmental > error to my way of thinking. No, that's some kind of parsing error. EnvironmentError doesn't concern itself with the contents of files. > My current coding style involves wrapping an "except EnvironmentError" > around any major operation and reporting it as a "File could not be > read/written/whatever because..." kind of message. Having > EOFError get missed by that would be a nuisance. But what operations raise EOFError? Surely you're not using raw_input()? It's really only there for teaching. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From musiccomposition at gmail.com Tue Apr 15 04:21:40 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Mon, 14 Apr 2008 21:21:40 -0500 Subject: [Python-Dev] NeedsReview keyword Message-ID: <1afaf6160804141921h239689c6s7f37e6af3ed70ab2@mail.gmail.com> I think it would be useful for the tracker to grow a "NeedsReview" keyword. I realize the "patch" keyword does some of this, but it may just represent some initial or trivial work. "NeedsReview" should represent a mature patch that some senior dev needs to look hard at and make the choice. -- Cheers, Benjamin Peterson From nnorwitz at gmail.com Tue Apr 15 06:19:02 2008 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 14 Apr 2008 21:19:02 -0700 Subject: [Python-Dev] Python 2.4.4/2.4.5 test_pty failure on Solaris 10 In-Reply-To: <18432.63770.323783.565994@montanaro-dyndns-org.local> References: <18432.63770.323783.565994@montanaro-dyndns-org.local> Message-ID: On Sat, Apr 12, 2008 at 11:02 AM, wrote: > > I know this is old stuff, but... > > I want to update our Python 2.4 installation at work from 2.4.2 to 2.4.5 > (the latest 2.4 source release). I get a test failure for test_pty, an > extra ^M at the end of one line. I don't get a failure in the 2.4.2 > installation, but the 2.4.4 and 2.4.5 both fail this test. Looking at the > code in test_pty.py, it appears to me that r43570 fixed things for OSF/1 and > IRIX which both do weird things with output while breaking things for any > other platform by suppressing the \r\n -> \n mapping which used to be > performed for all platforms. So, for Solaris, that mapping doesn't happen > and the actual and expected outputs don't agree. This was probably me. Perhaps a fix wasn't backported? I notice the 2.5 version of the test changed from the 2.4 version and does a str.replace rather than changing the last chars of the string. You can try using the 2.5 version and my guess is it will work (ie, the test will pass). The change is in normalize_output. n From nnorwitz at gmail.com Tue Apr 15 07:06:51 2008 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 14 Apr 2008 22:06:51 -0700 Subject: [Python-Dev] r62342 - python/branches/py3k/Objects/bytesobject.c In-Reply-To: <4803D319.5010000@cheimes.de> References: <20080414205105.B45C11E4005@bag.python.org> <4803C735.10100@cheimes.de> <43aa6ff70804141418w414cbe0bl8eb82ea77e8559a1@mail.gmail.com> <4803D319.5010000@cheimes.de> Message-ID: On Mon, Apr 14, 2008 at 2:56 PM, Christian Heimes wrote: > > Good idea! A 3to2 project is going to make backporting io.py and other > new stuff less painful and much faster. For the io.py backport I spent > most time on removing annotations and replacing "" with u"". > > What needs to be done? > > * remove funtion annotation > > * Add object to all empty class definition > > * replace class Egg(metaclass=Spam) with class > Egg(object):\n__metaclass__ = Spam > > * Add __future__ imports for print_function and unicode literals > > Anything else? Iteration with the dict methods (e.g., keys -> iterkeys()), map/zip/filter returning iterator rather than list. int -> (int, long) str -> basestring or (str, unicode) __bool__ -> __nonzero__ exec/execfile input -> rawinput Most things that have a fixer in 2to3 would also require one in 3to2. Only things that work in backwards compatible ways like apply/has_key removal, etc don't need a 3to2 fixer. Although most of these 3to2 fixers are probably pretty low priority as they are not real likely to be used in the python code base. They are still needed for general user code. n From martin at v.loewis.de Tue Apr 15 07:21:07 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 15 Apr 2008 07:21:07 +0200 Subject: [Python-Dev] NeedsReview keyword In-Reply-To: <1afaf6160804141921h239689c6s7f37e6af3ed70ab2@mail.gmail.com> References: <1afaf6160804141921h239689c6s7f37e6af3ed70ab2@mail.gmail.com> Message-ID: <48043B43.7070303@v.loewis.de> > I think it would be useful for the tracker to grow a "NeedsReview" > keyword. I realize the "patch" keyword does some of this, but it may > just represent some initial or trivial work. "NeedsReview" should > represent a mature patch that some senior dev needs to look hard at > and make the choice. Not sure what problem that would solve. Over time, I would expect that any open patch also grows the "NeedsReview" keyword, making the keyword pointless. If somebody specifically should review a certain proposed change, the change should be assigned to that person. If someone in a group should review, they should be contacted by email. Regards, Martin From schmir at gmail.com Tue Apr 15 07:31:07 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Tue, 15 Apr 2008 07:31:07 +0200 Subject: [Python-Dev] NeedsReview keyword In-Reply-To: <48043B43.7070303@v.loewis.de> References: <1afaf6160804141921h239689c6s7f37e6af3ed70ab2@mail.gmail.com> <48043B43.7070303@v.loewis.de> Message-ID: <932f8baf0804142231j611e7803j12103388d4c163c5@mail.gmail.com> On Tue, Apr 15, 2008 at 7:21 AM, "Martin v. L?wis" wrote: > > I think it would be useful for the tracker to grow a "NeedsReview" > > keyword. I realize the "patch" keyword does some of this, but it may > > just represent some initial or trivial work. "NeedsReview" should > > represent a mature patch that some senior dev needs to look hard at > > and make the choice. > > Not sure what problem that would solve. Over time, I would expect that > any open patch also grows the "NeedsReview" keyword, making the keyword > pointless. If somebody specifically should review a certain proposed > change, the change should be assigned to that person. If someone in > a group should review, they should be contacted by email. > I think it would be nice if that patch keyword could be set by non-admins. This would mean I didn't have to write to the mailing list asking for people to look at some specific bug. Like "did someone look at http://bugs.python.org/issue2122. This isssue is about mmap.flush not raising an exception on errors. which I think is a rather severe". (btw. can someone please look at it? :) ) Regards, - Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080415/36aeb655/attachment-0001.htm From nnorwitz at gmail.com Tue Apr 15 07:43:06 2008 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 14 Apr 2008 22:43:06 -0700 Subject: [Python-Dev] Next monthly sprint/bugfix day? In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22BEB730@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22BEB730@EXMBX04.exchhosting.com> Message-ID: On Wed, Apr 9, 2008 at 7:12 AM, Trent Nelson wrote: > Hi, > > Is there another online sprint/bugfix day in the pipeline? If not, can there be? ;-) Trent, I think you just volunteered to lead it. :-) We should either do it this weekend Apr 19-20 or wait until after the release. The first available date should be May 10. The schedule http://www.python.org/dev/peps/pep-0361/ has the upcoming May 7 release as the last alpha. That means we are getting closer to an API freeze. Anything that might change an API for 2.6/3.0 should be addressed sooner rather than later. If we have to change an API before release, please update the bug tracker priority to "release blocker". n From martin at v.loewis.de Tue Apr 15 07:54:48 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 15 Apr 2008 07:54:48 +0200 Subject: [Python-Dev] NeedsReview keyword In-Reply-To: <932f8baf0804142231j611e7803j12103388d4c163c5@mail.gmail.com> References: <1afaf6160804141921h239689c6s7f37e6af3ed70ab2@mail.gmail.com> <48043B43.7070303@v.loewis.de> <932f8baf0804142231j611e7803j12103388d4c163c5@mail.gmail.com> Message-ID: <48044328.6040207@v.loewis.de> > I think it would be nice if that patch keyword could be set by non-admins. > This would mean I didn't have to write to the mailing list asking for > people to look at > some specific bug. Like "did someone look at > http://bugs.python.org/issue2122. Just name your patch files .patch or .diff the next time, not .txt, and the keyword will get automatically set. This isssue is about mmap.flush not > raising an exception on errors. which I think is a rather severe". (btw. > can someone please look at it? :) ) I've added the patch keyword. I don't think the bug is rather severe, as it only affects the mmap module. I also don't see how this could cause data loss if the application works correctly. Regards, Martin From schmir at gmail.com Tue Apr 15 08:19:01 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Tue, 15 Apr 2008 08:19:01 +0200 Subject: [Python-Dev] NeedsReview keyword In-Reply-To: <48044328.6040207@v.loewis.de> References: <1afaf6160804141921h239689c6s7f37e6af3ed70ab2@mail.gmail.com> <48043B43.7070303@v.loewis.de> <932f8baf0804142231j611e7803j12103388d4c163c5@mail.gmail.com> <48044328.6040207@v.loewis.de> Message-ID: <932f8baf0804142319p19ec1565ye364ad98b7674a3e@mail.gmail.com> On Tue, Apr 15, 2008 at 7:54 AM, "Martin v. L?wis" wrote: > > Just name your patch files .patch or .diff the next time, not .txt, and > the keyword will get automatically set. > fine. I used .txt cause I wanted to view it in my browser (without the browser asking me for an application) > > This isssue is about mmap.flush not > > raising an exception on errors. which I think is a rather severe". (btw. > > can someone please look at it? :) ) > > I've added the patch keyword. I don't think the bug is rather severe, > thanks. > as it only affects the mmap module. I also don't see how this could > cause data loss if the application works correctly. > the flush fails but the programs fails to recognize it? i.e. the program assumes the data is written to disk but it isn't? Regards, - Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080415/f40f4f7e/attachment.htm From p.f.moore at gmail.com Tue Apr 15 10:57:52 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 15 Apr 2008 09:57:52 +0100 Subject: [Python-Dev] [Distutils] PEP 365 (Adding the pkg_resources module) In-Reply-To: References: <20080317151637.532D23A409D@sparrow.telecommunity.com> <20080317161305.22B183A409D@sparrow.telecommunity.com> <47DEC0EB.3000404@palladion.com> <440CD260-F6F6-4484-8326-0EA1D7B83132@zooko.com> <79990c6b0803191134had5def2uff0742cc5188b125@mail.gmail.com> Message-ID: <79990c6b0804150157n558c7572x17f33582e6a585a9@mail.gmail.com> On 19/03/2008, Guido van Rossum wrote: > > I'm currently working on an addition to pkgutil to provide this type > > of function. I'm considering going a little further (adding functions > > to get a file-like object, test for existence, and list available > > resources, modelled on the pkg_resources functions - but these extra > > ones are not supported by the loader protocol, so I'm undecided as to > > whether it's worth it, I'll see how complex the code gets). > > I'd only do what __loader__ offers. People can always wrap a StringIO around it. > > > Once I have a patch, I'll post it to the tracker. What's the best > > approach? Code a patch for 3.0 and backport, or code for 2.6 and let > > the merging process do its stuff? > > Code for 2.6, let the merge do its thing. I've had a patch in http://bugs.python.org/issue2439 for a few weeks now. It just adds a get_data function, on the basis that that's all the loader API offers. I think it's complete - Phillip Eby helped thrash out a couple of problems. Is there anything I can do to get it applied, or should I just leave it now for someone to decide if they care enough? (As it's a library change, I don't know to what extent the "no API changes after the next alpha" rule will apply). Paul. From stefan_ml at behnel.de Tue Apr 15 11:21:28 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 15 Apr 2008 11:21:28 +0200 Subject: [Python-Dev] r62342 - python/branches/py3k/Objects/bytesobject.c In-Reply-To: References: <20080414205105.B45C11E4005@bag.python.org> <4803C735.10100@cheimes.de> <43aa6ff70804141418w414cbe0bl8eb82ea77e8559a1@mail.gmail.com> <4803D319.5010000@cheimes.de> Message-ID: Neal Norwitz wrote: > Iteration with the dict methods (e.g., keys -> iterkeys()), > map/zip/filter returning iterator rather than list. That's only an optimisation, it's not functionally required. A list behaves like an iterator in most use cases, so it's rather unlikely that Py3 code will break in the backport because of this (and it's very unlikely that static analysis can catch the cases where it breaks). There should be a rule to optimise "list(map(...))" into "map(...)" and "list(x.keys())" into plain "x.keys()" etc., but I don't think there is much more to do here. > int -> (int, long) Is there any case where this might be required? I don't see any reason why back-converted Py3 code should break here. What would "long()" be needed for in working Py3 code that "int()" doesn't provide in Py2? Although you might have been referring to "isinstance(x, int)" in Py3? > str -> basestring or (str, unicode) This is an issue, although I think that Py3 is explicit enough here to make this easy to handle by static renaming (and maybe optimising "isinstance(s, (str, bytes))" into "..., basestring))"). > __bool__ -> __nonzero__ > exec/execfile > input -> rawinput Also valid issues that can be handled through renaming and static syntactic adjustments. > Most things that have a fixer in 2to3 would also require one in 3to2. I think the more explicit syntax in Py3 will actually make it easier to back-convert the code statically from 3to2 than to migrate it from 2to3. Stefan From ncoghlan at gmail.com Tue Apr 15 11:57:27 2008 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 15 Apr 2008 19:57:27 +1000 Subject: [Python-Dev] [Distutils] PEP 365 (Adding the pkg_resources module) In-Reply-To: <79990c6b0804150157n558c7572x17f33582e6a585a9@mail.gmail.com> References: <20080317151637.532D23A409D@sparrow.telecommunity.com> <20080317161305.22B183A409D@sparrow.telecommunity.com> <47DEC0EB.3000404@palladion.com> <440CD260-F6F6-4484-8326-0EA1D7B83132@zooko.com> <79990c6b0803191134had5def2uff0742cc5188b125@mail.gmail.com> <79990c6b0804150157n558c7572x17f33582e6a585a9@mail.gmail.com> Message-ID: <48047C07.1020807@gmail.com> Paul Moore wrote: > Is there anything I can do to get it applied, or should I just leave > it now for someone to decide if they care enough? (As it's a library > change, I don't know to what extent the "no API changes after the next > alpha" rule will apply). I'm looking into it now - assuming it applies cleanly and the tests run happily afterwards (and I don't see any reason it won't after reading the patch), it should be checked in later tonight. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From sergiodj at linux.vnet.ibm.com Tue Apr 15 19:48:07 2008 From: sergiodj at linux.vnet.ibm.com (=?ISO-8859-1?Q?S=E9rgio?= Durigan =?ISO-8859-1?Q?J=FAnior?=) Date: Tue, 15 Apr 2008 14:48:07 -0300 Subject: [Python-Dev] weird configure (autotools) setup In-Reply-To: <20080412100027.GN51167@nexus.in-nomine.org> References: <20080412100027.GN51167@nexus.in-nomine.org> Message-ID: <1208281687.5817.15.camel@miki> Hi Jeroen On Sat, 2008-04-12 at 12:00 +0200, Jeroen Ruigrok van der Werven wrote: > Why is CFLAGS in Makefile.pre.in specified as > CFLAGS= $(BASECFLAGS) $(OPT) $(EXTRA_CFLAGS) > whereas that will negate any CFLAGS you pass to configure? I suggest you read (and follow) this issue: http://bugs.python.org/issue1628484 Bob and I are discussing the same thing, so if you want to help us in this effort of making Python's build system more flexible and correct, you're really welcome. Regards, -- S?rgio Durigan J?nior Linux on Power Toolchain - Software Engineer Linux Technology Center - LTC IBM Brazil From tnelson at onresolve.com Tue Apr 15 22:13:35 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Tue, 15 Apr 2008 13:13:35 -0700 Subject: [Python-Dev] Deco Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22E7536B@EXMBX04.exchhosting.com> Neal, Martin, We're in the process of decommissioning the box the 'x86 FreeBSD 3' build slave is running on, can you remove it from the list? Our new FreeBSD 7.0 server is up, which we're slowly migrating to, and I'll be able to set a slave up on that probably some time next week once we've moved our production stuff over. Trent. From martin at v.loewis.de Tue Apr 15 22:45:58 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 15 Apr 2008 22:45:58 +0200 Subject: [Python-Dev] NeedsReview keyword In-Reply-To: <932f8baf0804142319p19ec1565ye364ad98b7674a3e@mail.gmail.com> References: <1afaf6160804141921h239689c6s7f37e6af3ed70ab2@mail.gmail.com> <48043B43.7070303@v.loewis.de> <932f8baf0804142231j611e7803j12103388d4c163c5@mail.gmail.com> <48044328.6040207@v.loewis.de> <932f8baf0804142319p19ec1565ye364ad98b7674a3e@mail.gmail.com> Message-ID: <48051406.80501@v.loewis.de> > Just name your patch files .patch or .diff the next time, not .txt, and > the keyword will get automatically set. > > > fine. I used .txt cause I wanted to view it in my browser (without the > browser asking me for an application) .diff would have done the same thing. > as it only affects the mmap module. I also don't see how this could > cause data loss if the application works correctly. > > > the flush fails but the programs fails to recognize it? i.e. the program > assumes the data is written to disk but it isn't? Why would the program fail to recognize it? It should just look at the result being returned. I agree that this is fairly unpythonic, but I also think that there is any data loss that this may cause is the application's fault. Regards, Martin From schmir at gmail.com Tue Apr 15 23:03:57 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Tue, 15 Apr 2008 23:03:57 +0200 Subject: [Python-Dev] NeedsReview keyword In-Reply-To: <48051406.80501@v.loewis.de> References: <1afaf6160804141921h239689c6s7f37e6af3ed70ab2@mail.gmail.com> <48043B43.7070303@v.loewis.de> <932f8baf0804142231j611e7803j12103388d4c163c5@mail.gmail.com> <48044328.6040207@v.loewis.de> <932f8baf0804142319p19ec1565ye364ad98b7674a3e@mail.gmail.com> <48051406.80501@v.loewis.de> Message-ID: <932f8baf0804151403m1b35e1dfvf49e0b218c5862a9@mail.gmail.com> > > > > the flush fails but the programs fails to recognize it? i.e. the program > > assumes the data is written to disk but it isn't? > > Why would the program fail to recognize it? It should just look at the > result being returned. > sorry no. everything else raises an error. this is not documented (as far as I can see) and mmap.flush returns 0 on unix if it succeeds, and raises an exception on unix if it fails. on windows it returns 1 if it succeeds and 0 if it doesn't (One more time: on unix it returns 0 if it succeeds). Regards, - Ralf > I agree that this is fairly unpythonic, but I also think that there is > any data loss that this may cause is the application's fault. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080415/50940dc1/attachment.htm From martin at v.loewis.de Tue Apr 15 23:20:20 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 15 Apr 2008 23:20:20 +0200 Subject: [Python-Dev] NeedsReview keyword In-Reply-To: <932f8baf0804151403m1b35e1dfvf49e0b218c5862a9@mail.gmail.com> References: <1afaf6160804141921h239689c6s7f37e6af3ed70ab2@mail.gmail.com> <48043B43.7070303@v.loewis.de> <932f8baf0804142231j611e7803j12103388d4c163c5@mail.gmail.com> <48044328.6040207@v.loewis.de> <932f8baf0804142319p19ec1565ye364ad98b7674a3e@mail.gmail.com> <48051406.80501@v.loewis.de> <932f8baf0804151403m1b35e1dfvf49e0b218c5862a9@mail.gmail.com> Message-ID: <48051C14.6030002@v.loewis.de> > > the flush fails but the programs fails to recognize it? i.e. the > program > > assumes the data is written to disk but it isn't? > > Why would the program fail to recognize it? It should just look at the > result being returned. > > > sorry no. Sorry no what? Regards, Martin From schmir at gmail.com Tue Apr 15 23:37:32 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Tue, 15 Apr 2008 23:37:32 +0200 Subject: [Python-Dev] mmap.flush [was: NeedsReview keyword] Message-ID: <932f8baf0804151437k3ccf1ea6nb7b1adcaab75e71e@mail.gmail.com> On Tue, Apr 15, 2008 at 11:20 PM, "Martin v. L?wis" wrote: > > > the flush fails but the programs fails to recognize it? i.e. the > > program > > > assumes the data is written to disk but it isn't? > > > > Why would the program fail to recognize it? It should just look at > the > > result being returned. > > > > > > sorry no. > > Sorry no what? > mmap.flush returns different values for windows/unix like platforms in case it succeeds. mmap.flush raises an exception on unix like platforms for errors. mmap.flush returns 0 on windows for errors. This is the value which is returned on unix like platforms for a successful call. The documentation for mmap.flush does not even mention a return value. so, still: sorry no, the application should not just look at the result being returned. The mmap.flush method should be fixed. > Regards, > Martin > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080415/66539465/attachment.htm From martin at v.loewis.de Tue Apr 15 23:49:17 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 15 Apr 2008 23:49:17 +0200 Subject: [Python-Dev] mmap.flush [was: NeedsReview keyword] In-Reply-To: <932f8baf0804151437k3ccf1ea6nb7b1adcaab75e71e@mail.gmail.com> References: <932f8baf0804151437k3ccf1ea6nb7b1adcaab75e71e@mail.gmail.com> Message-ID: <480522DD.2020800@v.loewis.de> > so, still: sorry no, the application should not just look at the result > being returned. If it doesn't want to lose data, it *has* to, because of the way it's currently implemented. There is no other way (other than ignoring the error) in Python 2.5.x and earlier. > The mmap.flush method should be fixed. I never debated that. I debated whether that is a "rather severe" issue, which I still don't think it is. Regards, Martin From greg.ewing at canterbury.ac.nz Tue Apr 15 23:53:22 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 16 Apr 2008 09:53:22 +1200 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> Message-ID: <480523D2.2070705@canterbury.ac.nz> Brad Miller wrote: > It was suggested that a broader discussion might be in order around the > issue of iterators and how they are displayed in the command line > interpreter. Whatever is done, I don't think it would be a good idea to make the str() of these things look *too* much like a list. I think that would make things more confusing for a newcomer rather than less. The way they are, at least it's obvious that they're something special. -- Greg From guido at python.org Wed Apr 16 00:12:52 2008 From: guido at python.org (Guido van Rossum) Date: Tue, 15 Apr 2008 15:12:52 -0700 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: <480523D2.2070705@canterbury.ac.nz> References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> Message-ID: That's why I proposed <0, 1, ..., 9> for repr(range(10)). (And I meant the '...' literally, i.e. if there are more than 4 values, replace all but the first two and the last with three dots. And yes, I mean that str(range(4)) == '<0, 1, 2, 3>' but str(range(5)) == '<0, 1, ..., 4>'. I'm not at all sure that we should go the same way for dict views though. They are quite different beasts -- the fact that they change depending on the underlying dict ought to be somehow emphasized, and I'd be happier to keep these as they are in 3.0a4. --Guido On Tue, Apr 15, 2008 at 2:53 PM, Greg Ewing wrote: > Brad Miller wrote: > > It was suggested that a broader discussion might be in order around the > > issue of iterators and how they are displayed in the command line > > interpreter. > > Whatever is done, I don't think it would be a good idea > to make the str() of these things look *too* much like > a list. I think that would make things more confusing > for a newcomer rather than less. > > The way they are, at least it's obvious that they're > something special. > > -- > Greg > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Wed Apr 16 00:27:13 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 16 Apr 2008 10:27:13 +1200 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? In-Reply-To: References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> <48040C08.504@canterbury.ac.nz> Message-ID: <48052BC1.40202@canterbury.ac.nz> Guido van Rossum wrote: > No, that's some kind of parsing error. EnvironmentError doesn't > concern itself with the contents of files. Often I raise EnvironmentErrors of my own to signal parsing errors. This makes it easy to wrap everything in a try-except that catches anything that's the user's fault rather than the program's. > But what operations raise EOFError? Surely you're not using > raw_input()? It's really only there for teaching. I'm fairly sure there are some others, although I can't point to them on the spur of the moment. However, thinking about it a bit more, anything that calls something that can raise EOFError should be catching it anyway, so an escaped EOFError represents a program bug. So it's probably okay. -- Greg From guido at python.org Wed Apr 16 00:36:39 2008 From: guido at python.org (Guido van Rossum) Date: Tue, 15 Apr 2008 15:36:39 -0700 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? In-Reply-To: <48052BC1.40202@canterbury.ac.nz> References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> <48040C08.504@canterbury.ac.nz> <48052BC1.40202@canterbury.ac.nz> Message-ID: On Tue, Apr 15, 2008 at 3:27 PM, Greg Ewing wrote: > Guido van Rossum wrote: > > > No, that's some kind of parsing error. EnvironmentError doesn't > > concern itself with the contents of files. > > Often I raise EnvironmentErrors of my own to signal > parsing errors. This makes it easy to wrap everything > in a try-except that catches anything that's the user's > fault rather than the program's. Well, that's your problem. That's not what EnvironmentErrors are for. > > But what operations raise EOFError? Surely you're not using > > raw_input()? It's really only there for teaching. > > I'm fairly sure there are some others, although I > can't point to them on the spur of the moment. > > However, thinking about it a bit more, anything that > calls something that can raise EOFError should be > catching it anyway, so an escaped EOFError represents > a program bug. So it's probably okay. Right. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From steve at pearwood.info Wed Apr 16 01:40:39 2008 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 16 Apr 2008 09:40:39 +1000 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <480523D2.2070705@canterbury.ac.nz> Message-ID: <200804160940.40083.steve@pearwood.info> Hi folks, Been lurking for a while, this is my first post. As I recall, this discussion was started because of concern that Python 3 had introduced features that made it less friendly to beginners. If I can quote Brad Miller: "Several new iterators have appeared in Python 3.0 that makes the language less transparent to beginning programmers. The examples that immediately come to mind are shown below..." It hasn't been that many years since I was a beginner myself, and I'm pretty sure that had I seen Guido's suggestion: > That's why I proposed <0, 1, ..., 9> for repr(range(10)). (And I > meant the '...' literally, i.e. if there are more than 4 values, > replace all but the first two and the last with three dots. And yes, > I mean that str(range(4)) == '<0, 1, 2, 3>' but str(range(5)) == '<0, > 1, ..., 4>'. in the interpreter back then, I would have tried writing the literal <0, 1, 2, 3> as an alternative to [0, 1, 2, 3] or (0, 1, 2, 3) and been rather distressed that it didn't do what I expected. As a beginner I was rather hazy on the difference between lists and tuples, and I would have imagined that <> was just another sort of delimiter pair for a list-like thing. I'd like to say that I'd only make the mistake once, but thinking about how often I still write dict(key) instead of dict[key], it's quite possible that some people might take a long time to drive the <> pseudo-delimiters out of their head. I'm -1 on <>. In my experience, real beginners aren't going to using range() except in the context of for loops, hence are unlikely to be printing the range object directly except by accident. Personally, I find it a feature that this happens in Python 2.x: >>> xrange(5) xrange(5) and would say the same thing about range() in Python 3, regardless of whether you wrap it in a str() or a repr(). As for the use of dots to keep stringified-sequences short, I'd vote +1 except I think that four items is rather short and would prefer the dots to come in around six or eight items. That's not entirely an aesthetic preference, I also have a reason for thinking that it might actually make a difference, but to avoid turning this discussion into bike-shedding I'll keep it to myself unless asked. -- Steven From greg.ewing at canterbury.ac.nz Wed Apr 16 05:34:44 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 16 Apr 2008 15:34:44 +1200 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> Message-ID: <480573D4.5040708@canterbury.ac.nz> Guido van Rossum wrote: > That's why I proposed <0, 1, ..., 9> for repr(range(10)). My worry is that this will lead a newcomer into thinking this is some kind of valid expression syntax. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 16 05:40:39 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 16 Apr 2008 15:40:39 +1200 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? In-Reply-To: References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> <48040C08.504@canterbury.ac.nz> <48052BC1.40202@canterbury.ac.nz> Message-ID: <48057537.3060003@canterbury.ac.nz> Guido van Rossum wrote: >> Often I raise EnvironmentErrors of my own to signal >> parsing errors. > > Well, that's your problem. That's not what EnvironmentErrors are for. But it seems like it's what it *should* be for. I'd be happy to use some other standard exception type if one existed at the right point in the hierarchy to catch what I want to catch, but there isn't one. -- Greg From guido at python.org Wed Apr 16 06:39:02 2008 From: guido at python.org (Guido van Rossum) Date: Tue, 15 Apr 2008 21:39:02 -0700 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? In-Reply-To: <48057537.3060003@canterbury.ac.nz> References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> <48040C08.504@canterbury.ac.nz> <48052BC1.40202@canterbury.ac.nz> <48057537.3060003@canterbury.ac.nz> Message-ID: On Tue, Apr 15, 2008 at 8:40 PM, Greg Ewing wrote: > Guido van Rossum wrote: > >> Often I raise EnvironmentErrors of my own to signal > >> parsing errors. > > > > > Well, that's your problem. That's not what EnvironmentErrors are for. > > But it seems like it's what it *should* be for. You are unique in demanding this. > I'd be happy to use some other standard exception > type if one existed at the right point in the hierarchy > to catch what I want to catch, but there isn't one. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 16 06:45:42 2008 From: guido at python.org (Guido van Rossum) Date: Tue, 15 Apr 2008 21:45:42 -0700 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: <480573D4.5040708@canterbury.ac.nz> References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> Message-ID: On Tue, Apr 15, 2008 at 8:34 PM, Greg Ewing wrote: > Guido van Rossum wrote: > > That's why I proposed <0, 1, ..., 9> for repr(range(10)). > > My worry is that this will lead a newcomer into thinking > this is some kind of valid expression syntax. You and Steven D'Aprano both. So if this is indeed a bad idea, I'm stumped -- the 3.0 status quo is the best we can do, and teachers will just have to teach their students to write list(range(10)) if they want to see what it means. Or they can write an explicit for-loop (always useful on the first day): for i in range(10): print i -- --Guido van Rossum (home page: http://www.python.org/~guido/) From schmir at gmail.com Wed Apr 16 08:06:49 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Wed, 16 Apr 2008 08:06:49 +0200 Subject: [Python-Dev] mmap.flush [was: NeedsReview keyword] In-Reply-To: <480522DD.2020800@v.loewis.de> References: <932f8baf0804151437k3ccf1ea6nb7b1adcaab75e71e@mail.gmail.com> <480522DD.2020800@v.loewis.de> Message-ID: <932f8baf0804152306v184201acgf9362c73086825ec@mail.gmail.com> On Tue, Apr 15, 2008 at 11:49 PM, "Martin v. L?wis" wrote: > > so, still: sorry no, the application should not just look at the result > > being returned. > > If it doesn't want to lose data, it *has* to, because of the way it's > currently implemented. There is no other way (other than ignoring the > error) in Python 2.5.x and earlier. > My point is it will ignore this error, because the programmer didn't even know about the return value. So it will possibly use data. But, it's all in the bug report and I'm only repeating myself. > > > The mmap.flush method should be fixed. > > I never debated that. I debated whether that is a "rather severe" issue, > which I still don't think it is. > I didn't recognize that we discussed this phrase. Sorry if I wasted your time. Regards, Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080416/231cafe9/attachment.htm From arigo at tunes.org Wed Apr 16 12:15:44 2008 From: arigo at tunes.org (Armin Rigo) Date: Wed, 16 Apr 2008 12:15:44 +0200 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: <480573D4.5040708@canterbury.ac.nz> References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> Message-ID: <20080416101544.GA10756@code0.codespeak.net> Hi Greg, On Wed, Apr 16, 2008 at 03:34:44PM +1200, Greg Ewing wrote: > > That's why I proposed <0, 1, ..., 9> for repr(range(10)). > > My worry is that this will lead a newcomer into thinking > this is some kind of valid expression syntax. What about the less confusing and more readily generalizable: It would also be helpful IMHO to use this kind of repr for most built-in iterators and iterables, instead of their mosty-useless default repr. A bientot, Armin. From p.f.moore at gmail.com Wed Apr 16 14:59:40 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 16 Apr 2008 13:59:40 +0100 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: <20080416101544.GA10756@code0.codespeak.net> References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> Message-ID: <79990c6b0804160559h2f63bdd0yef374fd6fe8dab7f@mail.gmail.com> On 16/04/2008, Armin Rigo wrote: > What about the less confusing and more readily generalizable: > > > > It would also be helpful IMHO to use this kind of repr for most built-in > iterators and iterables, instead of their mosty-useless default repr. I quite like this. But as a non-beginner, I'll freely admit that my intuitions are suspect :-) Paul. From asmodai at in-nomine.org Wed Apr 16 14:58:45 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Wed, 16 Apr 2008 14:58:45 +0200 Subject: [Python-Dev] mmap.flush [was: NeedsReview keyword] In-Reply-To: <932f8baf0804151437k3ccf1ea6nb7b1adcaab75e71e@mail.gmail.com> References: <932f8baf0804151437k3ccf1ea6nb7b1adcaab75e71e@mail.gmail.com> Message-ID: <20080416125845.GG91329@nexus.in-nomine.org> -On [20080415 23:37], Ralf Schmitt (schmir at gmail.com) wrote: >The documentation for mmap.flush does not even mention a return value. Fixed in revision 62359. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ With a nuclear fire of Love in our Hearts, rest easy baby, rest easy... From schmir at gmail.com Wed Apr 16 15:19:58 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Wed, 16 Apr 2008 15:19:58 +0200 Subject: [Python-Dev] mmap.flush [was: NeedsReview keyword] In-Reply-To: <20080416125845.GG91329@nexus.in-nomine.org> References: <932f8baf0804151437k3ccf1ea6nb7b1adcaab75e71e@mail.gmail.com> <20080416125845.GG91329@nexus.in-nomine.org> Message-ID: <932f8baf0804160619q3446203ax40ee303145782c8f@mail.gmail.com> On Wed, Apr 16, 2008 at 2:58 PM, Jeroen Ruigrok van der Werven < asmodai at in-nomine.org> wrote: > -On [20080415 23:37], Ralf Schmitt (schmir at gmail.com) wrote: > >The documentation for mmap.flush does not even mention a return value. > > Fixed in revision 62359. > I think documenting this insanity is just insane and doesn't help much. Anyway, you can now close http://bugs.python.org/issue2122 as wontfix. Regards, - Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080416/3642075e/attachment.htm From asmodai at in-nomine.org Wed Apr 16 15:11:13 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Wed, 16 Apr 2008 15:11:13 +0200 Subject: [Python-Dev] Unreachable code in mmapmodule.c? Message-ID: <20080416131113.GH91329@nexus.in-nomine.org> Reading through Modules/mmapmodule.c to work on some documentation I encountered this: 24 #ifndef MS_WINDOWS 25 #define UNIX 26 #endif 536 #ifdef MS_WINDOWS 537 return PyInt_FromLong((long) FlushViewOfFile(self->data+offset, siz e)); 538 #elif defined(UNIX) 539 /* XXX semantics of return value? */ 540 /* XXX flags for msync? */ 541 if (-1 == msync(self->data + offset, size, MS_SYNC)) { 542 PyErr_SetFromErrno(mmap_module_error); 543 return NULL; 544 } 545 return PyInt_FromLong(0); 546 #else 547 PyErr_SetString(PyExc_ValueError, "flush not supported on this syst em"); 548 return NULL; 549 #endif The #else will never be reached, unless I read it wrongly? -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ Once sent from the Golden Hall... From asmodai at in-nomine.org Wed Apr 16 15:29:15 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Wed, 16 Apr 2008 15:29:15 +0200 Subject: [Python-Dev] mmap.flush [was: NeedsReview keyword] In-Reply-To: <932f8baf0804160619q3446203ax40ee303145782c8f@mail.gmail.com> References: <932f8baf0804151437k3ccf1ea6nb7b1adcaab75e71e@mail.gmail.com> <20080416125845.GG91329@nexus.in-nomine.org> <932f8baf0804160619q3446203ax40ee303145782c8f@mail.gmail.com> Message-ID: <20080416132915.GI91329@nexus.in-nomine.org> -On [20080416 15:20], Ralf Schmitt (schmir at gmail.com) wrote: >I think documenting this insanity is just insane and doesn't help much. >Anyway, you can now close http://bugs.python.org/issue2122 as wontfix. Why? I added documentation for the current way. I was not aware of the other thread's mention of this issue. When your patch goes in --and looking at it it makes sense to me-- I can easily adjust the documentation to reflect the new state. :) I can understand your apparent frustration at not having gotten someone to look over your issue, but please: relax. I cannot make source changes, only documentation. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ A kiss is a lovely trick designed by nature to stop speech when words become superfluous... From schmir at gmail.com Wed Apr 16 15:35:34 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Wed, 16 Apr 2008 15:35:34 +0200 Subject: [Python-Dev] mmap.flush [was: NeedsReview keyword] In-Reply-To: <20080416132915.GI91329@nexus.in-nomine.org> References: <932f8baf0804151437k3ccf1ea6nb7b1adcaab75e71e@mail.gmail.com> <20080416125845.GG91329@nexus.in-nomine.org> <932f8baf0804160619q3446203ax40ee303145782c8f@mail.gmail.com> <20080416132915.GI91329@nexus.in-nomine.org> Message-ID: <932f8baf0804160635x161e95a9yb85416c8b3106814@mail.gmail.com> On Wed, Apr 16, 2008 at 3:29 PM, Jeroen Ruigrok van der Werven < asmodai at in-nomine.org> wrote: > -On [20080416 15:20], Ralf Schmitt (schmir at gmail.com) wrote: > >I think documenting this insanity is just insane and doesn't help much. > >Anyway, you can now close http://bugs.python.org/issue2122 as wontfix. > > Why? > I added documentation for the current way. I was not aware of the other > thread's mention of this issue. When your patch goes in --and looking at > it > it makes sense to me-- I can easily adjust the documentation to reflect > the > new state. :) > > I can understand your apparent frustration at not having gotten someone to > look over your issue, but please: relax. I cannot make source changes, > only > documentation. > ok. makes sense. sorry, if I offended you. Regards, - Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080416/ec87c425/attachment.htm From ijmorlan at cs.uwaterloo.ca Wed Apr 16 15:37:28 2008 From: ijmorlan at cs.uwaterloo.ca (Isaac Morland) Date: Wed, 16 Apr 2008 09:37:28 -0400 (EDT) Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: <79990c6b0804160559h2f63bdd0yef374fd6fe8dab7f@mail.gmail.com> References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> <79990c6b0804160559h2f63bdd0yef374fd6fe8dab7f@mail.gmail.com> Message-ID: On Wed, 16 Apr 2008, Paul Moore wrote: > On 16/04/2008, Armin Rigo wrote: >> What about the less confusing and more readily generalizable: >> >> >> >> It would also be helpful IMHO to use this kind of repr for most built-in >> iterators and iterables, instead of their mosty-useless default repr. > > I quite like this. But as a non-beginner, I'll freely admit that my > intuitions are suspect :-) I like this too. For iterators, though, would you always show the next couple of elements? The values "contained in" the iterator will change as the iterator iterates. Alternatively, the representation could be "frozen" to reflect the values originally pending in the iterator, but then the representation wouldn't show anything about the current state of the iterator. Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist From wolever at cs.toronto.edu Wed Apr 16 15:54:38 2008 From: wolever at cs.toronto.edu (David Wolever) Date: Wed, 16 Apr 2008 09:54:38 -0400 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> <79990c6b0804160559h2f63bdd0yef374fd6fe8dab7f@mail.gmail.com> Message-ID: <9EC6ABE1-9BA8-4349-B8C3-F332E069AA83@cs.toronto.edu> On 16-Apr-08, at 9:37 AM, Isaac Morland wrote: > On Wed, 16 Apr 2008, Paul Moore wrote: >> On 16/04/2008, Armin Rigo wrote: >>> What about the less confusing and more readily generalizable: >>> >>> >>> It would also be helpful IMHO to use this kind of repr for most >>> built-in >>> iterators and iterables, instead of their mosty-useless default >>> repr. >> >> I quite like this. But as a non-beginner, I'll freely admit that my >> intuitions are suspect :-) > > I like this too. For iterators, though, would you always show the > next > couple of elements? The values "contained in" the iterator will > change as > the iterator iterates. Alternatively, the representation could be > "frozen" to reflect the values originally pending in the iterator, but > then the representation wouldn't show anything about the current > state of > the iterator. So would you mean something like: , <__main__.Foo instance at 0x83620>, ...> Or maybe: , <__main__.Foo instance at 0x83620>, ... > While I agree in theory, I'm not sure I like the looks of it in practise. From steve at holdenweb.com Wed Apr 16 15:58:39 2008 From: steve at holdenweb.com (Steve Holden) Date: Wed, 16 Apr 2008 09:58:39 -0400 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> <79990c6b0804160559h2f63bdd0yef374fd6fe8dab7f@mail.gmail.com> Message-ID: Isaac Morland wrote: > On Wed, 16 Apr 2008, Paul Moore wrote: > >> On 16/04/2008, Armin Rigo wrote: >>> What about the less confusing and more readily generalizable: >>> >>> >>> >>> It would also be helpful IMHO to use this kind of repr for most built-in >>> iterators and iterables, instead of their mosty-useless default repr. >> I quite like this. But as a non-beginner, I'll freely admit that my >> intuitions are suspect :-) > > I like this too. For iterators, though, would you always show the next > couple of elements? The values "contained in" the iterator will change as > the iterator iterates. Alternatively, the representation could be > "frozen" to reflect the values originally pending in the iterator, but > then the representation wouldn't show anything about the current state of > the iterator. > If you consume values from the iterator to display them in the repr() where do you then propose to store them until the application wants them, and how do you distinguish between "real" and "repr" consumption of the values? "The next couple of elements" is a dangerous thing to use unless you don't mind them disappearing. And the last one's right out - you;d end up storing lists anyway! regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ From ncoghlan at gmail.com Wed Apr 16 16:03:46 2008 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 17 Apr 2008 00:03:46 +1000 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> <79990c6b0804160559h2f63bdd0yef374fd6fe8dab7f@mail.gmail.com> Message-ID: <48060742.70101@gmail.com> Isaac Morland wrote: > On Wed, 16 Apr 2008, Paul Moore wrote: > >> On 16/04/2008, Armin Rigo wrote: >>> What about the less confusing and more readily generalizable: >>> >>> >>> >>> It would also be helpful IMHO to use this kind of repr for most built-in >>> iterators and iterables, instead of their mosty-useless default repr. >> I quite like this. But as a non-beginner, I'll freely admit that my >> intuitions are suspect :-) > > I like this too. For iterators, though, would you always show the next > couple of elements? The values "contained in" the iterator will change as > the iterator iterates. Alternatively, the representation could be > "frozen" to reflect the values originally pending in the iterator, but > then the representation wouldn't show anything about the current state of > the iterator. You can't write nice repr's for arbitrary iterators, or things like enumerate and friends that can operate on arbitrary iterator, because you have no way to get at the 'first few' and 'the last' value. You can only have a nice representation like that for iterables like range which have random access to the underlying values. Note that range is NOT an iterator in Py3k as it has no __next__ method (the same hold trues for xrange in 2.x). However, it would be really nice if collections.dict_keys, collections.dict_items and collections.dict_values could provide a nice repr implementation that was the rough equivalent of: def full_repr(self): contents = ', '.join(map(repr, self)) return "<%s: {%s}>" % (self.__class__.__name__, contents) >>> d = dict(a=1, b=2, c=3) >>> d {'a': 1, 'c': 3, 'b': 2} >>> full_repr(d.keys()) "" >>> full_repr(d.values()) '' >>> full_repr(d.items()) "" Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From bonelake at gmail.com Wed Apr 16 16:11:27 2008 From: bonelake at gmail.com (Brad Miller) Date: Wed, 16 Apr 2008 09:11:27 -0500 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: <20080416101544.GA10756@code0.codespeak.net> References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> Message-ID: <4CA553FE-4D6B-40D1-9517-D95756E3C1EC@gmail.com> On Apr 16, 2008, at 5:15 AM, Armin Rigo wrote: > Hi Greg, > > On Wed, Apr 16, 2008 at 03:34:44PM +1200, Greg Ewing wrote: >>> That's why I proposed <0, 1, ..., 9> for repr(range(10)). >> >> My worry is that this will lead a newcomer into thinking >> this is some kind of valid expression syntax. > > What about the less confusing and more readily generalizable: > > > > It would also be helpful IMHO to use this kind of repr for most > built-in > iterators and iterables, instead of their mosty-useless default repr. > I think this is a great compromise. It is much more helpful to the beginner than range(0,10). This would also be a very simple change to the patch I already made :-) I think it works nicely for the dict_keys, dict_values, and dict_items objects also as the student will see: This reinforces that they will be iterating over tuples, which is much more helpful than a hexadecimal address. Since ordering is not important for dictionary objects I wonder whether the ending value(s) are even needed or whether the first two are enough to help the student get an idea of what the object contains. Brad > > A bientot, > > Armin. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/bonelake%40gmail.com From curt at hagenlocher.org Wed Apr 16 16:54:43 2008 From: curt at hagenlocher.org (Curt Hagenlocher) Date: Wed, 16 Apr 2008 07:54:43 -0700 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> <79990c6b0804160559h2f63bdd0yef374fd6fe8dab7f@mail.gmail.com> Message-ID: On Wed, Apr 16, 2008 at 6:58 AM, Steve Holden wrote: > > If you consume values from the iterator to display them in the repr() > where do you then propose to store them until the application wants > them, and how do you distinguish between "real" and "repr" consumption > of the values? "The next couple of elements" is a dangerous thing to use > unless you don't mind them disappearing. Not only that, but you'd have no idea what the performance consequences of accessing the "next" object might be. -- Curt Hagenlocher curt at hagenlocher.org From guido at python.org Wed Apr 16 16:57:06 2008 From: guido at python.org (Guido van Rossum) Date: Wed, 16 Apr 2008 07:57:06 -0700 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: <20080416101544.GA10756@code0.codespeak.net> References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> Message-ID: On Wed, Apr 16, 2008 at 3:15 AM, Armin Rigo wrote: > What about the less confusing and more readily generalizable: > > > > It would also be helpful IMHO to use this kind of repr for most built-in > iterators and iterables, instead of their mosty-useless default repr. We can't do this for iterators, since you can't print the values without consuming the iterator. Printing something shouldn't have a side effect on it. But for iterables (e.g. dict views) it should work fine, and if others can agree with this I'd be happy to accept patches. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ijmorlan at cs.uwaterloo.ca Wed Apr 16 17:39:30 2008 From: ijmorlan at cs.uwaterloo.ca (Isaac Morland) Date: Wed, 16 Apr 2008 11:39:30 -0400 (EDT) Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: <9EC6ABE1-9BA8-4349-B8C3-F332E069AA83@cs.toronto.edu> References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> <79990c6b0804160559h2f63bdd0yef374fd6fe8dab7f@mail.gmail.com> <9EC6ABE1-9BA8-4349-B8C3-F332E069AA83@cs.toronto.edu> Message-ID: On Wed, 16 Apr 2008, David Wolever wrote: > On 16-Apr-08, at 9:37 AM, Isaac Morland wrote: >> On Wed, 16 Apr 2008, Paul Moore wrote: >>> On 16/04/2008, Armin Rigo wrote: >>>> What about the less confusing and more readily generalizable: >>>> >>>> >>>> It would also be helpful IMHO to use this kind of repr for most built-in >>>> iterators and iterables, instead of their mosty-useless default repr. >>> >>> I quite like this. But as a non-beginner, I'll freely admit that my >>> intuitions are suspect :-) >> >> I like this too. For iterators, though, would you always show the next >> couple of elements? The values "contained in" the iterator will change as >> the iterator iterates. Alternatively, the representation could be >> "frozen" to reflect the values originally pending in the iterator, but >> then the representation wouldn't show anything about the current state of >> the iterator. > So would you mean something like: > , <__main__.Foo instance > at 0x83620>, ...> > Or maybe: > , <__main__.Foo instance at > 0x83620>, ... > > > While I agree in theory, I'm not sure I like the looks of it in practise. I was mostly responding to what I saw as a suggestion to change the representation of existing iterators. It's been pointed out in a previous reply to my message that obtaining values from a general iterator for use in the representation is problematic at best, and in general I don't think it can be done in an acceptable fashion, because I can imagine code depending on values not being obtained from an iterator before they are explicitly requested by next(). We wouldn't want a call to __repr__() to change the operation of the iterator, so any idea to pull two values and store them somewhere isn't generally acceptable. For some specific iterators which have access to all the underlying information, an informative representation is possible and frequently feasible. My question simply concerned what it would look like. Would it show the next couple of items, or the first couple of items (or something else)? Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist From tnelson at onresolve.com Wed Apr 16 19:51:53 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 16 Apr 2008 10:51:53 -0700 Subject: [Python-Dev] Global Python Sprint Weekends: May 10th-11th and June 21st-22nd. Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22F389FE@EXMBX04.exchhosting.com> Following on from the success of previous sprint/bugfix weekends and sprinting efforts at PyCon 2008, I'd like to propose the next two Global Python Sprint Weekends take place on the following dates: * May 10th-11th (four days after 2.6a3 and 3.0a5 are released) * June 21st-22nd (~week before 2.6b2 and 3.0b2 are released) It seems there are a few of the Python User Groups keen on meeting up in person and sprinting collaboratively, akin to PyCon, which I highly recommend. I'd like to nominate Saturday across the board as the day for PUGs to meet up in person, with Sunday geared more towards an online collaboration day via IRC, where we can take care of all the little things that got in our way of coding on Saturday (like finalising/preparing/reviewing patches, updating tracker and documentation, writing tests ;-). For User Groups that are planning on meeting up to collaborate, please reply to this thread on python-dev at python.org and let every- one know your intentions! As is commonly the case, #python-dev on irc.freenode.net will be the place to be over the course of each sprint weekend; a large proportion of Python developers with commit access will be present, increasing the amount of eyes available to review and apply patches. For those that have an idea on areas they'd like to sprint on and want to look for other developers to rope in (or just to communicate plans in advance), please also feel free to jump on this thread via python-dev@ and indicate your intentions. For those that haven't the foggiest on what to work on, but would like to contribute, the bugs tracker at http://bugs.python.org is the best place to start. Register an account and start searching for issues that you'd be able to lend a hand with. All contributors that submit code patches or documentation updates will typically get listed in Misc/ACKS.txt; come September when the final release of 2.6 and 3.0 come about, you'll be able to point at the tarball or .msi and exclaim loudly ``I helped build that!'', and actually back it up with hard evidence ;-) Bring on the pizza and Red Bull! Trent. From fuzzyman at voidspace.org.uk Wed Apr 16 20:40:44 2008 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Wed, 16 Apr 2008 19:40:44 +0100 Subject: [Python-Dev] Global Python Sprint Weekends: May 10th-11th and June 21st-22nd. In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22F389FE@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22F389FE@EXMBX04.exchhosting.com> Message-ID: <4806482C.4030305@voidspace.org.uk> Trent Nelson wrote: > Following on from the success of previous sprint/bugfix weekends and > sprinting efforts at PyCon 2008, I'd like to propose the next two > Global Python Sprint Weekends take place on the following dates: > > * May 10th-11th (four days after 2.6a3 and 3.0a5 are released) > * June 21st-22nd (~week before 2.6b2 and 3.0b2 are released) > > It seems there are a few of the Python User Groups keen on meeting > up in person and sprinting collaboratively, akin to PyCon, which I > highly recommend. I'd like to nominate Saturday across the board > as the day for PUGs to meet up in person, with Sunday geared more > towards an online collaboration day via IRC, where we can take care > of all the little things that got in our way of coding on Saturday > (like finalising/preparing/reviewing patches, updating tracker and > documentation, writing tests ;-). > > For User Groups that are planning on meeting up to collaborate, > please reply to this thread on python-dev at python.org and let every- > one know your intentions! > > I should be able to help organise and attend the London contribution. Personally I'd like to work on the documentation changes / clean-up for the unittest module discussed recently. Michael Foord From tjreedy at udel.edu Wed Apr 16 22:16:14 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 16 Apr 2008 16:16:14 -0400 Subject: [Python-Dev] string representation of range in 3.0 References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com><6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com><480523D2.2070705@canterbury.ac.nz><480573D4.5040708@canterbury.ac.nz><20080416101544.GA10756@code0.codespeak.net> Message-ID: "Guido van Rossum" wrote in message news:ca471dc20804160757u6a0257c8rff1ab5f68b5fc698 at mail.gmail.com... | On Wed, Apr 16, 2008 at 3:15 AM, Armin Rigo wrote: | > What about the less confusing and more readily generalizable: | > | > | > | > It would also be helpful IMHO to use this kind of repr for most built-in | > iterators and iterables, instead of their mosty-useless default repr. | | We can't do this for iterators, since you can't print the values | without consuming the iterator. Printing something shouldn't have a | side effect on it. But for iterables (e.g. dict views) it should work | fine, and if others can agree with this I'd be happy to accept | patches. Assuming we are discussing the output of str() rather that repr(), I think we should give Armin's idea a try for iterables that don't get the full display (as with lists, etc) but can harmlessly produce some values. From guido at python.org Wed Apr 16 23:20:24 2008 From: guido at python.org (Guido van Rossum) Date: Wed, 16 Apr 2008 14:20:24 -0700 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> Message-ID: On Wed, Apr 16, 2008 at 1:16 PM, Terry Reedy wrote: > > "Guido van Rossum" wrote in message > news:ca471dc20804160757u6a0257c8rff1ab5f68b5fc698 at mail.gmail.com... > > > | On Wed, Apr 16, 2008 at 3:15 AM, Armin Rigo wrote: > | > What about the less confusing and more readily generalizable: > | > > | > > | > > | > It would also be helpful IMHO to use this kind of repr for most > built-in > | > iterators and iterables, instead of their mosty-useless default repr. > | > | We can't do this for iterators, since you can't print the values > | without consuming the iterator. Printing something shouldn't have a > | side effect on it. But for iterables (e.g. dict views) it should work > | fine, and if others can agree with this I'd be happy to accept > | patches. > > Assuming we are discussing the output of str() rather that repr(), > I think we should give Armin's idea a try for iterables that don't get the > full display > (as with lists, etc) but can harmlessly produce some values. Why only str()? Note that the interactive prompt uses repr() to display values. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at krypto.org Wed Apr 16 23:24:20 2008 From: greg at krypto.org (Gregory P. Smith) Date: Wed, 16 Apr 2008 14:24:20 -0700 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? In-Reply-To: References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> <48040C08.504@canterbury.ac.nz> Message-ID: <52dc1c820804161424g121d76f5n7c144efc0770c804@mail.gmail.com> On Mon, Apr 14, 2008 at 7:11 PM, Guido van Rossum wrote: > > But what operations raise EOFError? Surely you're not using > raw_input()? It's really only there for teaching. > > There are quite a few things in Lib/ that raise EOFError on their own. Most look like reasonable uses. ftplib's getline() is likely the one raising it such that it is ever seen when using urrlib2. nntplib, telnetlib, xdrlib, getpass, aifc, chunk, sunau, gzip, & mailbox also raise it. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080416/99dc1e3b/attachment.htm From lists at cheimes.de Wed Apr 16 23:39:40 2008 From: lists at cheimes.de (Christian Heimes) Date: Wed, 16 Apr 2008 23:39:40 +0200 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> Message-ID: <4806721C.9040407@cheimes.de> Guido van Rossum schrieb: > Why only str()? Note that the interactive prompt uses repr() to display values. Does py3k still use the tp_print slot for the interactive prompt? Christian From guido at python.org Thu Apr 17 00:30:12 2008 From: guido at python.org (Guido van Rossum) Date: Wed, 16 Apr 2008 15:30:12 -0700 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: <4806721C.9040407@cheimes.de> References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> <4806721C.9040407@cheimes.de> Message-ID: No, tp_print is dead, unless I am terribly mistaken. (We didn't remove the slot because that would require updating every single static type initializer.) On Wed, Apr 16, 2008 at 2:39 PM, Christian Heimes wrote: > Guido van Rossum schrieb: > > > Why only str()? Note that the interactive prompt uses repr() to display values. > > Does py3k still use the tp_print slot for the interactive prompt? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Thu Apr 17 02:27:43 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 17 Apr 2008 12:27:43 +1200 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? In-Reply-To: References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> <48040C08.504@canterbury.ac.nz> <48052BC1.40202@canterbury.ac.nz> <48057537.3060003@canterbury.ac.nz> Message-ID: <4806997F.50000@canterbury.ac.nz> Guido van Rossum wrote: > You are unique in demanding this. Really? Nobody else wants a convenient way to distinguish program bugs from exceptions caused by external factors? -- Greg From greg.ewing at canterbury.ac.nz Thu Apr 17 02:30:09 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 17 Apr 2008 12:30:09 +1200 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? In-Reply-To: References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> <48040C08.504@canterbury.ac.nz> <48052BC1.40202@canterbury.ac.nz> <48057537.3060003@canterbury.ac.nz> Message-ID: <48069A11.8060001@canterbury.ac.nz> Guido van Rossum wrote: > You are unique in demanding this. I'm not asking for anything to be changed (I've already agreed that EOFError can stay the way it is), just pointing out what I think is a legitimate use for custom EnvironmentError subclasses. -- Greg From amk at amk.ca Thu Apr 17 03:51:10 2008 From: amk at amk.ca (A.M. Kuchling) Date: Wed, 16 Apr 2008 21:51:10 -0400 Subject: [Python-Dev] Global Python Sprint Weekends: May 10th-11th and June 21st-22nd. In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22F389FE@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22F389FE@EXMBX04.exchhosting.com> Message-ID: <20080417014450.GA8591@amk.local> On Wed, Apr 16, 2008 at 10:51:53AM -0700, Trent Nelson wrote: > Following on from the success of previous sprint/bugfix weekends and > sprinting efforts at PyCon 2008, I'd like to propose the next two > Global Python Sprint Weekends take place on the following dates: A great idea; thanks for organizing this! I've updated the wiki page at http://wiki.python.org/moin/PythonBugDay, so people can link to it, and added a news item on www.python.org. --amk From bonelake at gmail.com Thu Apr 17 03:57:32 2008 From: bonelake at gmail.com (Brad Miller) Date: Wed, 16 Apr 2008 20:57:32 -0500 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> <4806721C.9040407@cheimes.de> Message-ID: <84EB04A4-5443-4C7C-957F-43253CEF1368@gmail.com> I worked up prototype implementation for dict_keys, dict_values, and dict_items Here's an example of what the output looks like: >>> x = {chr(i):i for i in range(68,90)} >>> x.keys() >>> x.values() >>> x.items() >>> comments? Are there other objects in this family that I should look at? Brad On Apr 16, 2008, at 5:30 PM, Guido van Rossum wrote: > No, tp_print is dead, unless I am terribly mistaken. (We didn't remove > the slot because that would require updating every single static type > initializer.) > > On Wed, Apr 16, 2008 at 2:39 PM, Christian Heimes > wrote: >> Guido van Rossum schrieb: >> >>> Why only str()? Note that the interactive prompt uses repr() to >>> display values. >> >> Does py3k still use the tp_print slot for the interactive prompt? > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/bonelake%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080416/58f0230b/attachment-0001.htm From greg.ewing at canterbury.ac.nz Thu Apr 17 04:03:11 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 17 Apr 2008 14:03:11 +1200 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: <4CA553FE-4D6B-40D1-9517-D95756E3C1EC@gmail.com> References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> <20080416101544.GA10756@code0.codespeak.net> <4CA553FE-4D6B-40D1-9517-D95756E3C1EC@gmail.com> Message-ID: <4806AFDF.6030308@canterbury.ac.nz> Brad Miller wrote: > I wouldn't include the word "object" in any of these. Everything in Python is an object, so it's just space-wasting noise. -- Greg From tjreedy at udel.edu Thu Apr 17 04:49:02 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 16 Apr 2008 22:49:02 -0400 Subject: [Python-Dev] string representation of range in 3.0 References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com><6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com><480523D2.2070705@canterbury.ac.nz><480573D4.5040708@canterbury.ac.nz><20080416101544.GA10756@code0.codespeak.net> Message-ID: "Guido van Rossum" wrote in message news:ca471dc20804161420o5e51978csdf99d2427f328c5b at mail.gmail.com... | Why only str()? Note that the interactive prompt uses repr() to display values. I was under the impression that for range, you still wanted to maintain the distinction between an evalable and a friendly representation. If not, then ok, but I would then almost wonder whether to have both functions -- until I remember the reason for two versions of string representation and your proposal (which I approve of) for both a full and abbreviated rep for classes. | From tnelson at onresolve.com Thu Apr 17 12:17:21 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Thu, 17 Apr 2008 03:17:21 -0700 Subject: [Python-Dev] Code signing of Windows .msi/.dll/.exe's for 2.6/3.0 Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD422@EXMBX04.exchhosting.com> Hi Barry, Friendly poke to see if there's been any progress on acquiring the relevant certificates such that we can code sign binaries on Windows for 2.6/3.0 ;-) Trent. From lists at cheimes.de Thu Apr 17 13:13:07 2008 From: lists at cheimes.de (Christian Heimes) Date: Thu, 17 Apr 2008 13:13:07 +0200 Subject: [Python-Dev] Code signing of Windows .msi/.dll/.exe's for 2.6/3.0 In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD422@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD422@EXMBX04.exchhosting.com> Message-ID: <480730C3.5010703@cheimes.de> Trent Nelson schrieb: > Hi Barry, > > Friendly poke to see if there's been any progress on acquiring the relevant certificates such that we can code sign binaries on Windows for 2.6/3.0 ;-) Can you explain: * Why the binaries should be signed? * What is required to sign the binaries? * Which binaries should be signed? Christian From tnelson at onresolve.com Thu Apr 17 13:56:28 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Thu, 17 Apr 2008 04:56:28 -0700 Subject: [Python-Dev] Code signing of Windows .msi/.dll/.exe's for 2.6/3.0 In-Reply-To: <480730C3.5010703@cheimes.de> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD422@EXMBX04.exchhosting.com> <480730C3.5010703@cheimes.de> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD433@EXMBX04.exchhosting.com> > * Why the binaries should be signed? Makes the installation process on Windows Vista and Server 2008 a little nicer; instead of getting an "unknown-executable-could- be-a-virus-aaaaaahhhh-watchout"-type dialog with a big red flag, you get a less threatening message saying that you're about to run something that's been digitally signed by the Python Software Foundation. (I've come across a few entities (NSA, government bodies, etc), who mandate that all installers/binaries they get must be digitally signed.) > * What is required to sign the binaries? 1. Obtain a code signing certificate from someone. I used VeriSign. You end up with an .spc and a .pvk file. You need to combine them into a single .pfx file via a tool called pvk2pfx.exe: Usage: pvk2pfx -pvk [-pi ] -spc [-pfx [-po ] [-f]] -pvk - input PVK file name. -spc - input SPC file name. -pfx - output PFX file name. -pi - PVK password. -po - PFX password; same as -pi if not given. -f - force overwrite existing PFX file. if -pfx option is not given, an export wizard will pop up. in this case, options -po and -f are ignored. C:\..> pvk2pfx.exe -pvk verisign-privatekey.pvk -pi ****** -spc onresolve-verisign.spc -po ****** -pfx onresolve-verisign.pfx 3. The resulting .pfx file, onresolve-verisign.pfx in this case, can then be installed as a 'Personal' certificate in Windows, using the Certificate Management facility (CertMgr.exe). When you install it, you provide a name that the certificate can be referred to by apps; in my case I just used 'VeriSign'. This name is used below by the signtool.exe app. 4. Sign the executable, MSI or DLL as follows: C:\..> signtool.exe sign /i "VeriSign" /d "Python 2.6.0" /du http://www.python.org /t http://timestamp.verisign.com/scripts/timstamp.dll Python-2.6.msi Successfully signed and timestamped: Python-2.6.msi > * Which binaries should be signed? Personally, once I figured out the steps above, I hooked the signing process into all my Visual Studio projects as a post-build step, such that I sign all .exe and .dll files. Not really necessary, but eh, it does have the advantage of looking more professional (users can view properties on the .dll, for example, and see that it's been digitally signed by the PSF). Additionally, it prevents any tampering; Windows can detect if it's been altered in any way since it's been signed, and will flat out prevent it from being loaded/run if that's the case. Trent. From ziade.tarek at gmail.com Thu Apr 17 15:44:23 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 17 Apr 2008 15:44:23 +0200 Subject: [Python-Dev] Global Python Sprint Weekends: May 10th-11th and June 21st-22nd. In-Reply-To: <4806482C.4030305@voidspace.org.uk> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22F389FE@EXMBX04.exchhosting.com> <4806482C.4030305@voidspace.org.uk> Message-ID: <94bdd2610804170644x4db329b0vf49de21a590c51e@mail.gmail.com> On Wed, Apr 16, 2008 at 8:40 PM, Michael Foord wrote: > Trent Nelson wrote: > > Following on from the success of previous sprint/bugfix weekends and > > sprinting efforts at PyCon 2008, I'd like to propose the next two > > Global Python Sprint Weekends take place on the following dates: > > > > * May 10th-11th (four days after 2.6a3 and 3.0a5 are released) > > * June 21st-22nd (~week before 2.6b2 and 3.0b2 are released) > > > > It seems there are a few of the Python User Groups keen on meeting > > up in person and sprinting collaboratively, akin to PyCon, which I > > highly recommend. I'd like to nominate Saturday across the board > > as the day for PUGs to meet up in person, with Sunday geared more > > towards an online collaboration day via IRC, where we can take care > > of all the little things that got in our way of coding on Saturday > > (like finalising/preparing/reviewing patches, updating tracker and > > documentation, writing tests ;-). > > > > For User Groups that are planning on meeting up to collaborate, > > please reply to this thread on python-dev at python.org and let every- > > one know your intentions! > > > > > > I should be able to help organise and attend the London contribution. > Personally I'd like to work on the documentation changes / clean-up for > the unittest module discussed recently. We are trying to set up a team here in Paris, Personnally I would like to continue the work started in distutils (various patches) and some friends here are interested in contributing on documentation. Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From aahz at pythoncraft.com Thu Apr 17 16:25:34 2008 From: aahz at pythoncraft.com (Aahz) Date: Thu, 17 Apr 2008 07:25:34 -0700 Subject: [Python-Dev] string representation of range in 3.0 In-Reply-To: References: <9061B337-C6E0-4826-A5F3-846B2B60E1F8@gmail.com> <6D78AAC5-0C6F-4EA6-A491-AA0504BF0C6A@gmail.com> <480523D2.2070705@canterbury.ac.nz> <480573D4.5040708@canterbury.ac.nz> Message-ID: <20080417142534.GA17709@panix.com> On Tue, Apr 15, 2008, Guido van Rossum wrote: > On Tue, Apr 15, 2008 at 8:34 PM, Greg Ewing wrote: >> Guido van Rossum wrote: >>> >>> That's why I proposed <0, 1, ..., 9> for repr(range(10)). >> >> My worry is that this will lead a newcomer into thinking >> this is some kind of valid expression syntax. > > You and Steven D'Aprano both. > > So if this is indeed a bad idea, I'm stumped -- the 3.0 status quo is > the best we can do, and teachers will just have to teach their > students to write list(range(10)) if they want to see what it means. > Or they can write an explicit for-loop (always useful on the first > day): > > for i in range(10): > print i No, they can't. ;-) (print is a function in 3.0....) This may seem like a minor nit -- and it is, in some ways -- but I think reminding people of the pedagogical issues involved in the transition is important. Granted, this is python-dev, so we should be using 2.x syntax, but we're currently talking about 3.0 and that requires clarifying what is meant/intended. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "It is easier to optimize correct code than to correct optimized code." --Bill Harlan From fuzzyman at voidspace.org.uk Thu Apr 17 15:49:15 2008 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 17 Apr 2008 14:49:15 +0100 Subject: [Python-Dev] Proposed unittest changes Message-ID: <4807555B.2070206@voidspace.org.uk> Hello all, I'm starting to put together a list of cleanups (with documentation changes) for the unittest module. I thought someone had already done this but the issue with the most comments I could find was 2578 which doesn't go into much detail: http://bugs.python.org/issue2578 The thing most people would like is test discovery - but that probably requires some discussion before anything can be added to unittest. What I would like to do is PEP-8'ify the method names (widening the API rather than shrinking it!): assert_true assert_false assert_equals assert_not_equals assert_raises set_up tear_down failure_exception (? - class attribute) TestSuite.add_test (etc) Documenting that these are to be preferred to 'assertEquals' and 'failUnlessEquals' (etc) and that the 'assert' statement should be used instead of 'assert_'. Adding the following new asserts: assert_in (member, container, msg=None) assert_not_in (member, container, msg=None) assert_is (first, second, msg=None) assert_not_is (first, second, msg=None) assert_raises_with_message (exc_class, message, callable, *args, **keywargs) A decorator to turn a test function into a TestCase ("as_test_case" ?). A simple 'RunTests' function that takes a collection of modules, test suites or test cases and runs them with TextTestRunner - passing on keyword arguments to the test runner. This makes running a test suite easier - once you have collected all your test cases together you can just pass them to this function so long as you are happy with the default runner (possibly allowing an alternative runner class to be provided as a keyword argument). I would provide an implementation for discussion of course. I would also like to make the error messages for "assert_equals" and "assert_not_equals" more useful - showing the objects that compare incorrectly even if an explicit message is passed in. Additionally, when comparing lists and tuples that are the same length show the members (and indices?) that were different. I've copied Steve Purcell into this email, but his comments on issue 2578 indicate that he is happy for 'us' to make changes and he no longer has a string sense of "ownership" of this module. Michael Foord From guido at python.org Thu Apr 17 16:48:07 2008 From: guido at python.org (Guido van Rossum) Date: Thu, 17 Apr 2008 07:48:07 -0700 Subject: [Python-Dev] Code signing of Windows .msi/.dll/.exe's for 2.6/3.0 In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD433@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD422@EXMBX04.exchhosting.com> <480730C3.5010703@cheimes.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD433@EXMBX04.exchhosting.com> Message-ID: This is going to be a major administrative hassle. Obviously you have to be extremely cautious with the private part of the certificate, or it's worthless. That means that there will probably be only one person who can sign binaries. That person would have also have to build all the binaries -- signing binaries you retrieved from the internet sounds like a complete bypassing of the security procedures. *And* that person would forever have to be extremely cautious with the machine on which the certificate resides, keeping it turned off and locked away securely most of the time, or else risk that the machine is infected by malware, again bypassing the point. While the chances of ever signing something bad are low, the bad effects could be huge (sort of like the risk of an earthquake as compared to a car crash). I'm not at all sure that we are set up to do this right, and that it is worth the minor inconvenience to users of having to acknowledge a red-flag dialog. After all, they will be ack'ing such dialogs all the time if they are at all used to downloading software from the internet. --Guido On Thu, Apr 17, 2008 at 4:56 AM, Trent Nelson wrote: > > > * Why the binaries should be signed? > > Makes the installation process on Windows Vista and Server 2008 > a little nicer; instead of getting an "unknown-executable-could- > be-a-virus-aaaaaahhhh-watchout"-type dialog with a big red flag, > you get a less threatening message saying that you're about to > run something that's been digitally signed by the Python Software > Foundation. (I've come across a few entities (NSA, government > bodies, etc), who mandate that all installers/binaries they get > must be digitally signed.) > > > > * What is required to sign the binaries? > > 1. Obtain a code signing certificate from someone. I used VeriSign. > You end up with an .spc and a .pvk file. You need to combine them > into a single .pfx file via a tool called pvk2pfx.exe: > > Usage: > pvk2pfx -pvk [-pi ] -spc > [-pfx [-po ] [-f]] > > -pvk - input PVK file name. > -spc - input SPC file name. > -pfx - output PFX file name. > -pi - PVK password. > -po - PFX password; same as -pi if not given. > -f - force overwrite existing PFX file. > > if -pfx option is not given, an export wizard will pop up. in > this case, options -po and -f are ignored. > > C:\..> pvk2pfx.exe -pvk verisign-privatekey.pvk -pi ****** -spc onresolve-verisign.spc -po ****** -pfx onresolve-verisign.pfx > > 3. The resulting .pfx file, onresolve-verisign.pfx in this case, can > then be installed as a 'Personal' certificate in Windows, using the > Certificate Management facility (CertMgr.exe). When you install it, > you provide a name that the certificate can be referred to by apps; > in my case I just used 'VeriSign'. This name is used below by the > signtool.exe app. > > 4. Sign the executable, MSI or DLL as follows: > > C:\..> signtool.exe sign /i "VeriSign" /d "Python 2.6.0" /du http://www.python.org /t http://timestamp.verisign.com/scripts/timstamp.dll Python-2.6.msi > Successfully signed and timestamped: Python-2.6.msi > > > > * Which binaries should be signed? > > Personally, once I figured out the steps above, I hooked the signing > process into all my Visual Studio projects as a post-build step, such > that I sign all .exe and .dll files. Not really necessary, but eh, > it does have the advantage of looking more professional (users can > view properties on the .dll, for example, and see that it's been > digitally signed by the PSF). Additionally, it prevents any tampering; > Windows can detect if it's been altered in any way since it's been > signed, and will flat out prevent it from being loaded/run if that's > the case. > > > Trent. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 17 16:54:32 2008 From: guido at python.org (Guido van Rossum) Date: Thu, 17 Apr 2008 07:54:32 -0700 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: <4807555B.2070206@voidspace.org.uk> References: <4807555B.2070206@voidspace.org.uk> Message-ID: I'm worried that a mass renaming would do anything but inconvenience users during the already stressful 2->3 transition. I'm more in favor of the original proposal of reducing the redundancy post-3.0. If you're looking for useful features, Google has a set of extensions to unittest.py that I find useful: - module-level setUp and tearDown - routines for comparing large lists and strings that produce useful output indicating exactly where the inputs differ. - assertLess etc. - assertSameElements (sort of like assert(set(x) == set(y)) --Guido On Thu, Apr 17, 2008 at 6:49 AM, Michael Foord wrote: > Hello all, > > I'm starting to put together a list of cleanups (with documentation > changes) for the unittest module. I thought someone had already done > this but the issue with the most comments I could find was 2578 which > doesn't go into much detail: > > http://bugs.python.org/issue2578 > > The thing most people would like is test discovery - but that probably > requires some discussion before anything can be added to unittest. > > What I would like to do is PEP-8'ify the method names (widening the API > rather than shrinking it!): > > assert_true > assert_false > assert_equals > assert_not_equals > assert_raises > set_up > tear_down > failure_exception (? - class attribute) > TestSuite.add_test > > (etc) > > Documenting that these are to be preferred to 'assertEquals' and > 'failUnlessEquals' (etc) and that the 'assert' statement should be used > instead of 'assert_'. > > Adding the following new asserts: > > assert_in (member, container, msg=None) > assert_not_in (member, container, msg=None) > assert_is (first, second, msg=None) > assert_not_is (first, second, msg=None) > assert_raises_with_message (exc_class, message, callable, *args, > **keywargs) > > A decorator to turn a test function into a TestCase ("as_test_case" ?). > > A simple 'RunTests' function that takes a collection of modules, test > suites or test cases and runs them with TextTestRunner - passing on > keyword arguments to the test runner. This makes running a test suite > easier - once you have collected all your test cases together you can > just pass them to this function so long as you are happy with the > default runner (possibly allowing an alternative runner class to be > provided as a keyword argument). > > I would provide an implementation for discussion of course. > > I would also like to make the error messages for "assert_equals" and > "assert_not_equals" more useful - showing the objects that compare > incorrectly even if an explicit message is passed in. Additionally, when > comparing lists and tuples that are the same length show the members > (and indices?) that were different. > > I've copied Steve Purcell into this email, but his comments on issue > 2578 indicate that he is happy for 'us' to make changes and he no longer > has a string sense of "ownership" of this module. > > Michael Foord > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Thu Apr 17 17:05:20 2008 From: barry at python.org (Barry Warsaw) Date: Thu, 17 Apr 2008 11:05:20 -0400 Subject: [Python-Dev] Code signing of Windows .msi/.dll/.exe's for 2.6/3.0 In-Reply-To: References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD422@EXMBX04.exchhosting.com> <480730C3.5010703@cheimes.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD433@EXMBX04.exchhosting.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 17, 2008, at 10:48 AM, Guido van Rossum wrote: > This is going to be a major administrative hassle. Obviously you have > to be extremely cautious with the private part of the certificate, or > it's worthless. That means that there will probably be only one person > who can sign binaries. That person would have also have to build all > the binaries -- signing binaries you retrieved from the internet > sounds like a complete bypassing of the security procedures. *And* > that person would forever have to be extremely cautious with the > machine on which the certificate resides, keeping it turned off and > locked away securely most of the time, or else risk that the machine > is infected by malware, again bypassing the point. While the chances > of ever signing something bad are low, the bad effects could be huge > (sort of like the risk of an earthquake as compared to a car crash). > > I'm not at all sure that we are set up to do this right, and that it > is worth the minor inconvenience to users of having to acknowledge a > red-flag dialog. After all, they will be ack'ing such dialogs all the > time if they are at all used to downloading software from the > internet. MvL is leading this effort and is currently trying to get a code signing certificate (my attempts having utterly failed on the Mac ;). Please coordinate with him. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAdnMHEjvBPtnXfVAQKkhQP/frCWs9sLcWyrAmDouCIq0n2X0B0TUbbG +tEUQVBj2hJ/CVnmc6PjyFNjOhlfhIv+BI544QhsvoAlC2OdQe9d8hHX8sqnPUJy lsm0gZ49ldqOhM91Q285RPtoELEEXQtfTYS2RUI/whNy+O/EDiRMVIDtNfaJOMgS j6zeojy8d4E= =m4aj -----END PGP SIGNATURE----- From fuzzyman at voidspace.org.uk Thu Apr 17 16:59:01 2008 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 17 Apr 2008 15:59:01 +0100 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: References: <4807555B.2070206@voidspace.org.uk> Message-ID: <480765B5.2010904@voidspace.org.uk> Guido van Rossum wrote: > I'm worried that a mass renaming would do anything but inconvenience > users during the already stressful 2->3 transition. > > I'm more in favor of the original proposal of reducing the redundancy post-3.0. > > So nix the PEP-8'ifying until after 3.0. So new methods should follow the current naming scheme (assertIn, assertNotIn etc). > If you're looking for useful features, Google has a set of extensions > to unittest.py that I find useful: > > - module-level setUp and tearDown > So when a suite is made from a module the setUp should be called before the first test and tearDown after the last. I can look at that. > - routines for comparing large lists and strings that produce useful > output indicating exactly where the inputs differ. > > - assertLess etc. > By etc I assume you mean: assertLessThan assertGreaterThan assertLessThanOrEquals assertGreaterThanOrEquals Would not variants be useful as well - it seems not as the not of one is always another... (I think 'assertLessThan' reads better than 'assertLess' but will do what I'm told...) > - assertSameElements (sort of like assert(set(x) == set(y)) > Sounds good. Did you look at the other proposals? * Decorator to make a function a TestCase * Convenience RunTests functions taking modules, suites and TestCases and running them * Improved messages for assertEquals and assertNotEquals when an explicit message is passed in * Improved message when comparing lists/tuples with assertEquals * The additional asserts that I suggested (In/NotIn, RaisesWithMessage, Is/NotIs) I think that there is still work I can do on the docs even before any grand renaming... Michael Foord > --Guido > > On Thu, Apr 17, 2008 at 6:49 AM, Michael Foord > wrote: > >> Hello all, >> >> I'm starting to put together a list of cleanups (with documentation >> changes) for the unittest module. I thought someone had already done >> this but the issue with the most comments I could find was 2578 which >> doesn't go into much detail: >> >> http://bugs.python.org/issue2578 >> >> The thing most people would like is test discovery - but that probably >> requires some discussion before anything can be added to unittest. >> >> What I would like to do is PEP-8'ify the method names (widening the API >> rather than shrinking it!): >> >> assert_true >> assert_false >> assert_equals >> assert_not_equals >> assert_raises >> set_up >> tear_down >> failure_exception (? - class attribute) >> TestSuite.add_test >> >> (etc) >> >> Documenting that these are to be preferred to 'assertEquals' and >> 'failUnlessEquals' (etc) and that the 'assert' statement should be used >> instead of 'assert_'. >> >> Adding the following new asserts: >> >> assert_in (member, container, msg=None) >> assert_not_in (member, container, msg=None) >> assert_is (first, second, msg=None) >> assert_not_is (first, second, msg=None) >> assert_raises_with_message (exc_class, message, callable, *args, >> **keywargs) >> >> A decorator to turn a test function into a TestCase ("as_test_case" ?). >> >> A simple 'RunTests' function that takes a collection of modules, test >> suites or test cases and runs them with TextTestRunner - passing on >> keyword arguments to the test runner. This makes running a test suite >> easier - once you have collected all your test cases together you can >> just pass them to this function so long as you are happy with the >> default runner (possibly allowing an alternative runner class to be >> provided as a keyword argument). >> >> I would provide an implementation for discussion of course. >> >> I would also like to make the error messages for "assert_equals" and >> "assert_not_equals" more useful - showing the objects that compare >> incorrectly even if an explicit message is passed in. Additionally, when >> comparing lists and tuples that are the same length show the members >> (and indices?) that were different. >> >> I've copied Steve Purcell into this email, but his comments on issue >> 2578 indicate that he is happy for 'us' to make changes and he no longer >> has a string sense of "ownership" of this module. >> >> Michael Foord >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> > > > > From jnoller at gmail.com Thu Apr 17 17:19:35 2008 From: jnoller at gmail.com (Jesse Noller) Date: Thu, 17 Apr 2008 11:19:35 -0400 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: <480765B5.2010904@voidspace.org.uk> References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> Message-ID: <4222a8490804170819g3f31590fo71ea23f30fce712d@mail.gmail.com> On Thu, Apr 17, 2008 at 10:59 AM, Michael Foord wrote: > Guido van Rossum wrote: > > I'm worried that a mass renaming would do anything but inconvenience > > users during the already stressful 2->3 transition. > > > > I'm more in favor of the original proposal of reducing the redundancy post-3.0. > > > > > > So nix the PEP-8'ifying until after 3.0. > > So new methods should follow the current naming scheme (assertIn, > assertNotIn etc). > > > > If you're looking for useful features, Google has a set of extensions > > to unittest.py that I find useful: > > > > - module-level setUp and tearDown > > > So when a suite is made from a module the setUp should be called before > the first test and tearDown after the last. I can look at that. > > > > - routines for comparing large lists and strings that produce useful > > output indicating exactly where the inputs differ. > > > > - assertLess etc. > > > By etc I assume you mean: > > assertLessThan > assertGreaterThan > assertLessThanOrEquals > assertGreaterThanOrEquals > > Would not variants be useful as well - it seems not as the not of one is > always another... (I think 'assertLessThan' reads better than > 'assertLess' but will do what I'm told...) > > > > - assertSameElements (sort of like assert(set(x) == set(y)) > > > > Sounds good. > > Did you look at the other proposals? > > * Decorator to make a function a TestCase > * Convenience RunTests functions taking modules, suites and TestCases > and running them > * Improved messages for assertEquals and assertNotEquals when an > > explicit message is passed in > * Improved message when comparing lists/tuples with assertEquals > * The additional asserts that I suggested (In/NotIn, RaisesWithMessage, > Is/NotIs) > > I think that there is still work I can do on the docs even before any > grand renaming... > > Michael Foord > > > > > --Guido > > > > On Thu, Apr 17, 2008 at 6:49 AM, Michael Foord > > wrote: > > > >> Hello all, > >> > >> I'm starting to put together a list of cleanups (with documentation > >> changes) for the unittest module. I thought someone had already done > >> this but the issue with the most comments I could find was 2578 which > >> doesn't go into much detail: > >> > >> http://bugs.python.org/issue2578 > >> > >> The thing most people would like is test discovery - but that probably > >> requires some discussion before anything can be added to unittest. > >> > >> What I would like to do is PEP-8'ify the method names (widening the API > >> rather than shrinking it!): > >> > >> assert_true > >> assert_false > >> assert_equals > >> assert_not_equals > >> assert_raises > >> set_up > >> tear_down > >> failure_exception (? - class attribute) > >> TestSuite.add_test > >> > >> (etc) > >> > >> Documenting that these are to be preferred to 'assertEquals' and > >> 'failUnlessEquals' (etc) and that the 'assert' statement should be used > >> instead of 'assert_'. > >> > >> Adding the following new asserts: > >> > >> assert_in (member, container, msg=None) > >> assert_not_in (member, container, msg=None) > >> assert_is (first, second, msg=None) > >> assert_not_is (first, second, msg=None) > >> assert_raises_with_message (exc_class, message, callable, *args, > >> **keywargs) > >> > >> A decorator to turn a test function into a TestCase ("as_test_case" ?). > >> > >> A simple 'RunTests' function that takes a collection of modules, test > >> suites or test cases and runs them with TextTestRunner - passing on > >> keyword arguments to the test runner. This makes running a test suite > >> easier - once you have collected all your test cases together you can > >> just pass them to this function so long as you are happy with the > >> default runner (possibly allowing an alternative runner class to be > >> provided as a keyword argument). > >> > >> I would provide an implementation for discussion of course. > >> > >> I would also like to make the error messages for "assert_equals" and > >> "assert_not_equals" more useful - showing the objects that compare > >> incorrectly even if an explicit message is passed in. Additionally, when > >> comparing lists and tuples that are the same length show the members > >> (and indices?) that were different. > >> > >> I've copied Steve Purcell into this email, but his comments on issue > >> 2578 indicate that he is happy for 'us' to make changes and he no longer > >> has a string sense of "ownership" of this module. > >> > >> Michael Foord +1 on your changes Michael - I really like the decorator idea. Let me know if you want help on adding the new stuff or updating the documentation. What about aiming this at 2.6 instead of 3.0? -jesse From lists at cheimes.de Thu Apr 17 17:27:19 2008 From: lists at cheimes.de (Christian Heimes) Date: Thu, 17 Apr 2008 17:27:19 +0200 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: <480765B5.2010904@voidspace.org.uk> References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> Message-ID: <48076C57.5070601@cheimes.de> Michael Foord schrieb: > By etc I assume you mean: > > assertLessThan > assertGreaterThan > assertLessThanOrEquals > assertGreaterThanOrEquals > > Would not variants be useful as well - it seems not as the not of one is > always another... (I think 'assertLessThan' reads better than > 'assertLess' but will do what I'm told...) Most of the etc. could be simplified with a function assertOp which takes an operator as first argument import operator def assertOp(self, op, a, b, msg): func = getattr(operator, op) self.assert_(func(a, b) ...) assertOp("gt", a, b) == assert a > g I also like to have some assert for is, type, isinstance, issubclass and contains. Christian From steve at pythonconsulting.com Thu Apr 17 17:39:16 2008 From: steve at pythonconsulting.com (Steve Purcell) Date: Thu, 17 Apr 2008 17:39:16 +0200 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: <480765B5.2010904@voidspace.org.uk> References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> Message-ID: <2BC94709-57CD-4607-B4FC-1576A3285C7D@pythonconsulting.com> Great that you're taking this on, Michael! On 17 Apr 2008, at 16:59, Michael Foord wrote: >> If you're looking for useful features, Google has a set of extensions >> to unittest.py that I find useful: >> >> - module-level setUp and tearDown >> > So when a suite is made from a module the setUp should be called > before the first test and tearDown after the last. I can look at that. Note that suites are just clumps of tests, and test runners can choose to run tests in any order, which might complicate the above. In any case, I'd advise against per-module setUp/tearDown because it breaks the fundamental rule that any instance of TestCase can be safely run in isolation, secure in the knowledge that it will set itself up and tear itself down as required. There are usually (always?) superior alternatives to module-level setUp, so I wouldn't suggest encouraging it by providing a sanctioned mechanism. Also, I'd note that assert_() or similar may still be needed: it is provided as a preferred alternative to the assert() built-in because the latter stops working if you run Python with the -O switch, which might be a reasonable thing to do with a test suite. Aside from these points, everything else looks great to me. Better "diff-style" output for assertEquals of strings etc. has been lacking for ages, as well as collection-oriented assertions. -Steve P.S. Hi all! From lists at cheimes.de Thu Apr 17 17:32:14 2008 From: lists at cheimes.de (Christian Heimes) Date: Thu, 17 Apr 2008 17:32:14 +0200 Subject: [Python-Dev] Code signing of Windows .msi/.dll/.exe's for 2.6/3.0 In-Reply-To: References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD422@EXMBX04.exchhosting.com> <480730C3.5010703@cheimes.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD433@EXMBX04.exchhosting.com> Message-ID: <48076D7E.4060301@cheimes.de> Barry Warsaw schrieb: > MvL is leading this effort and is currently trying to get a code > signing certificate (my attempts having utterly failed on the Mac ;). > Please coordinate with him. Apropos certificate, I like to suggest a PGP signing party for the next PyCon. :] Christian From barry at python.org Thu Apr 17 17:56:58 2008 From: barry at python.org (Barry Warsaw) Date: Thu, 17 Apr 2008 11:56:58 -0400 Subject: [Python-Dev] Code signing of Windows .msi/.dll/.exe's for 2.6/3.0 In-Reply-To: <48076D7E.4060301@cheimes.de> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD422@EXMBX04.exchhosting.com> <480730C3.5010703@cheimes.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD433@EXMBX04.exchhosting.com> <48076D7E.4060301@cheimes.de> Message-ID: <9C65BE76-8093-4790-B67F-3AD7F4769AEC@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 17, 2008, at 11:32 AM, Christian Heimes wrote: > Barry Warsaw schrieb: >> MvL is leading this effort and is currently trying to get a code >> signing certificate (my attempts having utterly failed on the Mac ;). >> Please coordinate with him. > > Apropos certificate, I like to suggest a PGP signing party for the > next > PyCon. :] /me smacks his head for not doing one in 2008! - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) iQCVAwUBSAdzS3EjvBPtnXfVAQJEygP/SNB+nQ+XfEnOprAji9pGj5YcmJJSoKb3 A8UFR3LAI3EK3CVzgIXEeFuQOKZV34p0nTabVM9FEzsPrIbvnbzp1k4V4Go1bK/G iBktzoG8F/VCmdbF7OjwBLb7O5GVVK1MRIO1KvdHp00lW311VYYMYdRuZ150lsct doLzOH7kriU= =LCTo -----END PGP SIGNATURE----- From fuzzyman at voidspace.org.uk Thu Apr 17 18:11:32 2008 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 17 Apr 2008 17:11:32 +0100 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: <48076C57.5070601@cheimes.de> References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> <48076C57.5070601@cheimes.de> Message-ID: <480776B4.60503@voidspace.org.uk> Christian Heimes wrote: > Michael Foord schrieb: > >> By etc I assume you mean: >> >> assertLessThan >> assertGreaterThan >> assertLessThanOrEquals >> assertGreaterThanOrEquals >> >> Would not variants be useful as well - it seems not as the not of one is >> always another... (I think 'assertLessThan' reads better than >> 'assertLess' but will do what I'm told...) >> > > Most of the etc. could be simplified with a function assertOp which > takes an operator as first argument > > import operator > def assertOp(self, op, a, b, msg): > func = getattr(operator, op) > self.assert_(func(a, b) ...) > > assertOp("gt", a, b) == assert a > g > > Which choice would be up to consensus. I'm agnostic on this one. > I also like to have some assert for is, type, isinstance, issubclass and > contains. > > is and contains ('in') I listed. type, isinstance, issubclass would be extra. Michael > Christian > From fuzzyman at voidspace.org.uk Thu Apr 17 18:20:49 2008 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 17 Apr 2008 17:20:49 +0100 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: <4222a8490804170819g3f31590fo71ea23f30fce712d@mail.gmail.com> References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> <4222a8490804170819g3f31590fo71ea23f30fce712d@mail.gmail.com> Message-ID: <480778E1.6060908@voidspace.org.uk> Jesse Noller wrote: > On Thu, Apr 17, 2008 at 10:59 AM, Michael Foord > wrote: > >> Guido van Rossum wrote: >> > I'm worried that a mass renaming would do anything but inconvenience >> > users during the already stressful 2->3 transition. >> > >> > I'm more in favor of the original proposal of reducing the redundancy post-3.0. >> > >> > >> >> So nix the PEP-8'ifying until after 3.0. >> >> So new methods should follow the current naming scheme (assertIn, >> assertNotIn etc). >> >> [snip...] >> >> > > +1 on your changes Michael - I really like the decorator idea. Let me > know if you want help on adding the new stuff or updating the > documentation. What about aiming this at 2.6 instead of 3.0? > > -jesse > As they will just be new features rather than API changes I would like to add these to 2.6 - although I think May 10th may be after the last alpha? Michael From rrr at ronadam.com Thu Apr 17 18:38:57 2008 From: rrr at ronadam.com (Ron Adam) Date: Thu, 17 Apr 2008 11:38:57 -0500 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: <480765B5.2010904@voidspace.org.uk> References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> Message-ID: <48077D21.80001@ronadam.com> Michael Foord wrote: > Guido van Rossum wrote: >> I'm worried that a mass renaming would do anything but inconvenience >> users during the already stressful 2->3 transition. >> >> I'm more in favor of the original proposal of reducing the redundancy post-3.0. >> >> > > So nix the PEP-8'ifying until after 3.0. > > So new methods should follow the current naming scheme (assertIn, > assertNotIn etc). > >> If you're looking for useful features, Google has a set of extensions >> to unittest.py that I find useful: >> >> - module-level setUp and tearDown >> > So when a suite is made from a module the setUp should be called before > the first test and tearDown after the last. I can look at that. > >> - routines for comparing large lists and strings that produce useful >> output indicating exactly where the inputs differ. >> >> - assertLess etc. >> > By etc I assume you mean: > > assertLessThan > assertGreaterThan > assertLessThanOrEquals > assertGreaterThanOrEquals > > Would not variants be useful as well - it seems not as the not of one is > always another... (I think 'assertLessThan' reads better than > 'assertLess' but will do what I'm told...) > >> - assertSameElements (sort of like assert(set(x) == set(y)) >> > > Sounds good. > > Did you look at the other proposals? > > * Decorator to make a function a TestCase > * Convenience RunTests functions taking modules, suites and TestCases > and running them > * Improved messages for assertEquals and assertNotEquals when an > explicit message is passed in > * Improved message when comparing lists/tuples with assertEquals > * The additional asserts that I suggested (In/NotIn, RaisesWithMessage, > Is/NotIs) > > I think that there is still work I can do on the docs even before any > grand renaming... Something I found useful to improve messages is to have the tests catch and then report errors with it's own unique exceptions to indicate the failures. What that does is make a very clear distinction between an error found in a module being tested vs test case. An error in the test case results in a regular python exception and a trace back of the test case. An error in the module being tested results in an specific unittest exception indicating what type of error was found in the module followed by the original exception and traceback from the module being tested. Ron class TestCase(unittest.TestCase): failureException = unittest.TestCase.failureException class Unexpected_Exception_Raised(failureException): def __init__(self, exc, ref): self.exc = exc self.ref = ref def __str__(self): return '\n'.join([repr(self.exc), '\nReference:', self.ref]) class Wrong_Exception_Raised(Unexpected_Exception_Raised): pass class No_Exception_Raised(failureException): def __init__(self, result, ref=""): self.result = repr(result) self.ref = ref def __str__(self): return "returned -> " + '\n'.join([self.result, '\nReference:', self.ref]) class Wrong_Result_Returned(No_Exception_Raised): def __str__(self): return '\n'.join([self.result, '\nReference:', self.ref]) def assertTestReturns(self, test, expect, ref=""): try: result = test() except Exception, e: e0, e1, e2 = sys.exc_info() raise self.Unexpected_Exception_Raised, (e, ref), e2 if result != expect: raise self.Wrong_Result_Returned(result, ref) def assertTestRaises(self, test, expect, ref=""): try: result = test() except Exception, e: if isinstance(e, expect): return e else: e0, e1, e2 = sys.exc_info() raise self.Wrong_Exception_Raised, (e, ref), e2 raise self.No_Exception_Raised(result, ref) From rrr at ronadam.com Thu Apr 17 18:38:57 2008 From: rrr at ronadam.com (Ron Adam) Date: Thu, 17 Apr 2008 11:38:57 -0500 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: <480765B5.2010904@voidspace.org.uk> References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> Message-ID: <48077D21.80001@ronadam.com> Michael Foord wrote: > Guido van Rossum wrote: >> I'm worried that a mass renaming would do anything but inconvenience >> users during the already stressful 2->3 transition. >> >> I'm more in favor of the original proposal of reducing the redundancy post-3.0. >> >> > > So nix the PEP-8'ifying until after 3.0. > > So new methods should follow the current naming scheme (assertIn, > assertNotIn etc). > >> If you're looking for useful features, Google has a set of extensions >> to unittest.py that I find useful: >> >> - module-level setUp and tearDown >> > So when a suite is made from a module the setUp should be called before > the first test and tearDown after the last. I can look at that. > >> - routines for comparing large lists and strings that produce useful >> output indicating exactly where the inputs differ. >> >> - assertLess etc. >> > By etc I assume you mean: > > assertLessThan > assertGreaterThan > assertLessThanOrEquals > assertGreaterThanOrEquals > > Would not variants be useful as well - it seems not as the not of one is > always another... (I think 'assertLessThan' reads better than > 'assertLess' but will do what I'm told...) > >> - assertSameElements (sort of like assert(set(x) == set(y)) >> > > Sounds good. > > Did you look at the other proposals? > > * Decorator to make a function a TestCase > * Convenience RunTests functions taking modules, suites and TestCases > and running them > * Improved messages for assertEquals and assertNotEquals when an > explicit message is passed in > * Improved message when comparing lists/tuples with assertEquals > * The additional asserts that I suggested (In/NotIn, RaisesWithMessage, > Is/NotIs) > > I think that there is still work I can do on the docs even before any > grand renaming... Something I found useful to improve messages is to have the tests catch and then report errors with it's own unique exceptions to indicate the failures. What that does is make a very clear distinction between an error found in a module being tested vs test case. An error in the test case results in a regular python exception and a trace back of the test case. An error in the module being tested results in an specific unittest exception indicating what type of error was found in the module followed by the original exception and traceback from the module being tested. Ron class TestCase(unittest.TestCase): failureException = unittest.TestCase.failureException class Unexpected_Exception_Raised(failureException): def __init__(self, exc, ref): self.exc = exc self.ref = ref def __str__(self): return '\n'.join([repr(self.exc), '\nReference:', self.ref]) class Wrong_Exception_Raised(Unexpected_Exception_Raised): pass class No_Exception_Raised(failureException): def __init__(self, result, ref=""): self.result = repr(result) self.ref = ref def __str__(self): return "returned -> " + '\n'.join([self.result, '\nReference:', self.ref]) class Wrong_Result_Returned(No_Exception_Raised): def __str__(self): return '\n'.join([self.result, '\nReference:', self.ref]) def assertTestReturns(self, test, expect, ref=""): try: result = test() except Exception, e: e0, e1, e2 = sys.exc_info() raise self.Unexpected_Exception_Raised, (e, ref), e2 if result != expect: raise self.Wrong_Result_Returned(result, ref) def assertTestRaises(self, test, expect, ref=""): try: result = test() except Exception, e: if isinstance(e, expect): return e else: e0, e1, e2 = sys.exc_info() raise self.Wrong_Exception_Raised, (e, ref), e2 raise self.No_Exception_Raised(result, ref) From ziade.tarek at gmail.com Thu Apr 17 18:52:34 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 17 Apr 2008 18:52:34 +0200 Subject: [Python-Dev] A smarter shutil.copytree ? Message-ID: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> Hi, shutil.copytree is very convenient to make recursive copies, but os.walk has to be used everytime some filtering has to be done on the files copied., if you want to avoid copying some files. The code pattern with os.walk is pretty talkative : --------------------- copying a source folder to a target folder, but the pyc/pyo files os.mkdir(target) for root, dirs, filenames in os.walk(source): root_target = root.replace(source, target) for dir_ in dirs: target_dir = join(root_target, dir_) if os.path.exists(target_dir): continue os.mkdir(target_dir) for filename in filenames: filename_source = join(root, filename) filename_target = join(root_target, filename) if (os.path.exists(filename_target) or os.path.splitext(filename) in ('.pyc', '.pyo')): continue shutil.copyfile(filename_source, filename_target) -------------------- If we could provide a callable argument to shutil.copytree, this would allow simplifying a lot the code: --------------------- copying a source to a target, but the pyc/pyo file def filtering(source, target): return os.path.splitext(filename) not in ('.pyc', '.pyo') shutil.copytree(source, target, filter_=filtering) --------------------- This is a very current pattern in my code, and I think this could be an interesting enhancement to shutil. So if people think it is a good idea, I can work on a patch and submit it to the tracker. Regards Tarek -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From skip at pobox.com Thu Apr 17 19:01:41 2008 From: skip at pobox.com (skip at pobox.com) Date: Thu, 17 Apr 2008 12:01:41 -0500 Subject: [Python-Dev] Code signing of Windows .msi/.dll/.exe's for 2.6/3.0 In-Reply-To: <9C65BE76-8093-4790-B67F-3AD7F4769AEC@python.org> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD422@EXMBX04.exchhosting.com> <480730C3.5010703@cheimes.de> <87D3F9C72FBF214DB39FA4E3FE618CDC6E23EAD433@EXMBX04.exchhosting.com> <48076D7E.4060301@cheimes.de> <9C65BE76-8093-4790-B67F-3AD7F4769AEC@python.org> Message-ID: <18439.33397.636841.820546@montanaro-dyndns-org.local> >> Apropos certificate, I like to suggest a PGP signing party for the >> next PyCon. :] Barry> /me smacks his head for not doing one in 2008! Maybe a bunch of Python local user groups could be coaxed into having mini-signing parties during upcoming meetings, then let natural cross-fertilization between groups (such as Fitz visiting Google Mountain View or a few US folk attending EuroPython (mid-July), EuroSciPy (late July) or PyCon UK (September)) connect those groups. It wouldn't be as convenient as one big key signing party, but probably as effective in the long run. Skip From guido at python.org Thu Apr 17 19:01:51 2008 From: guido at python.org (Guido van Rossum) Date: Thu, 17 Apr 2008 10:01:51 -0700 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: <480765B5.2010904@voidspace.org.uk> References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> Message-ID: On Thu, Apr 17, 2008 at 7:59 AM, Michael Foord wrote: > By etc I assume you mean: > > assertLessThan > assertGreaterThan > assertLessThanOrEquals > assertGreaterThanOrEquals These names are used here: assertListEqual(self, list1, list2, msg=None): assertIn(self, a, b, msg=None): assertNotIn(self, a, b, msg=None): assertDictEqual(self, d1, d2, msg=None): assertSameElements(self, expected_seq, actual_seq, msg=None): assertMultiLineEqual(self, first, second, msg=None): assertLess(self, a, b, msg=None): assertLessEqual(self, a, b, msg=None): assertGreater(self, a, b, msg=None): assertGreaterEqual(self, a, b, msg=None): assertCommandSucceeds(self, command): assertCommandFails(self, command, regexes): I can look into making those open source, but it'd probably more efficient use of everyone's time to just reimplement them; most are only a few lines. You can skip the assertCommand* ones, they're for testing shell commands. > Did you look at the other proposals? Not yet. > * Decorator to make a function a TestCase But what about TOOWTDI? > * Convenience RunTests functions taking modules, suites and TestCases and > running them If it addresses the most common reason why people have to hack this stuff, by all means. > * Improved messages for assertEquals and assertNotEquals when an explicit > message is passed in Sure. > * Improved message when comparing lists/tuples with assertEquals I'd say do that in assertListEqual > * The additional asserts that I suggested (In/NotIn, RaisesWithMessage, > Is/NotIs) Sure. Google has In/NotIn > I think that there is still work I can do on the docs even before any grand > renaming... Go ahead! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ziade.tarek at gmail.com Thu Apr 17 19:02:57 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 17 Apr 2008 19:02:57 +0200 Subject: [Python-Dev] A smarter shutil.copytree ? In-Reply-To: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> References: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> Message-ID: <94bdd2610804171002q637bf590rda1caaa752d3a723@mail.gmail.com> > --------------------- copying a source to a target, but the pyc/pyo file > def filtering(source, target): > return os.path.splitext(filename) not in ('.pyc', '.pyo') > > shutil.copytree(source, target, filter_=filtering) > --------------------- oups, made a mistake in my example: def filtering(source_path, target_path): return os.path.splitext(source_path) not in ('.pyc', '.pyo') shutil.copytree(source, target, filter_=filtering) From guido at python.org Thu Apr 17 19:04:30 2008 From: guido at python.org (Guido van Rossum) Date: Thu, 17 Apr 2008 10:04:30 -0700 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: <48076C57.5070601@cheimes.de> References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> <48076C57.5070601@cheimes.de> Message-ID: On Thu, Apr 17, 2008 at 8:27 AM, Christian Heimes wrote: > Most of the etc. could be simplified with a function assertOp which > takes an operator as first argument > > import operator > def assertOp(self, op, a, b, msg): > func = getattr(operator, op) > self.assert_(func(a, b) ...) > > assertOp("gt", a, b) == assert a > g -1 on this; it requires more thinking and has more opportunities for mistakes (e.g. why "gt" and not ">"?). > I also like to have some assert for is, type, isinstance, issubclass and > contains. Yes. Michael had In/NotIn. I have needed all of the others too at various times! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 17 19:06:17 2008 From: guido at python.org (Guido van Rossum) Date: Thu, 17 Apr 2008 10:06:17 -0700 Subject: [Python-Dev] A smarter shutil.copytree ? In-Reply-To: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> References: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> Message-ID: Sounds like a neat little feature. Looking forward to it. Maybe the most useful use case would be to provide glob-style patterns for skipping files or directories (and their contents). --Guido On Thu, Apr 17, 2008 at 9:52 AM, Tarek Ziad? wrote: > Hi, > > shutil.copytree is very convenient to make recursive copies, but > os.walk has to be used everytime some filtering > has to be done on the files copied., if you want to avoid copying some files. > > The code pattern with os.walk is pretty talkative : > > --------------------- copying a source folder to a target folder, but > the pyc/pyo files > os.mkdir(target) > for root, dirs, filenames in os.walk(source): > root_target = root.replace(source, target) > for dir_ in dirs: > target_dir = join(root_target, dir_) > if os.path.exists(target_dir): > continue > os.mkdir(target_dir) > for filename in filenames: > filename_source = join(root, filename) > filename_target = join(root_target, filename) > if (os.path.exists(filename_target) or > os.path.splitext(filename) in ('.pyc', '.pyo')): > continue > shutil.copyfile(filename_source, filename_target) > -------------------- > > If we could provide a callable argument to shutil.copytree, this would > allow simplifying a lot the code: > > --------------------- copying a source to a target, but the pyc/pyo file > def filtering(source, target): > return os.path.splitext(filename) not in ('.pyc', '.pyo') > > shutil.copytree(source, target, filter_=filtering) > --------------------- > > This is a very current pattern in my code, and I think this could be > an interesting enhancement to > shutil. So if people think it is a good idea, I can work on a patch > and submit it to the tracker. > > Regards > > Tarek > > -- > Tarek Ziad? | Association AfPy | www.afpy.org > Blog FR | http://programmation-python.org > Blog EN | http://tarekziade.wordpress.com/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gjcarneiro at gmail.com Thu Apr 17 19:10:53 2008 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Thu, 17 Apr 2008 18:10:53 +0100 Subject: [Python-Dev] A smarter shutil.copytree ? In-Reply-To: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> References: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> Message-ID: On 17/04/2008, Tarek Ziad? wrote: > > Hi, > > shutil.copytree is very convenient to make recursive copies, but > os.walk has to be used everytime some filtering > has to be done on the files copied., if you want to avoid copying some > files. > > The code pattern with os.walk is pretty talkative : > > --------------------- copying a source folder to a target folder, but > the pyc/pyo files > os.mkdir(target) > for root, dirs, filenames in os.walk(source): > root_target = root.replace(source, target) > for dir_ in dirs: > target_dir = join(root_target, dir_) > if os.path.exists(target_dir): > continue > os.mkdir(target_dir) > for filename in filenames: > filename_source = join(root, filename) > filename_target = join(root_target, filename) > if (os.path.exists(filename_target) or > os.path.splitext(filename) in ('.pyc', '.pyo')): > continue > shutil.copyfile(filename_source, filename_target) > -------------------- > > If we could provide a callable argument to shutil.copytree, this would > allow simplifying a lot the code: > > --------------------- copying a source to a target, but the pyc/pyo file > def filtering(source, target): > return os.path.splitext(filename) not in ('.pyc', '.pyo') > > shutil.copytree(source, target, filter_=filtering) > --------------------- > > This is a very current pattern in my code, and I think this could be > an interesting enhancement to > shutil. So if people think it is a good idea, I can work on a patch > and submit it to the tracker. I also think this is a good idea; I recently was forced to copy-paste and modify shutil.copytree into a project because of this limitation. Regards > > Tarek > > -- > > Tarek Ziad? | Association AfPy | www.afpy.org > Blog FR | http://programmation-python.org > Blog EN | http://tarekziade.wordpress.com/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com > -- Gustavo J. A. M. Carneiro INESC Porto, Telecommunications and Multimedia Unit "The universe is always one step beyond logic." -- Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080417/ffb2ca5b/attachment.htm From tjreedy at udel.edu Thu Apr 17 19:39:17 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 17 Apr 2008 13:39:17 -0400 Subject: [Python-Dev] Proposed unittest changes References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> Message-ID: "Michael Foord" wrote in message news:480765B5.2010904 at voidspace.org.uk... | I think that there is still work I can do on the docs even before any | grand renaming... In a related thread, I proposed and Guido approved that the 2.6/3.0 docs be changed to 1. specify that the Fail... names will disappear in the future 2. list the preferred Assert names first for each pair 3. use the preferred Assert names and only those names in the explanatory text and examples. Have you or are you going to make these changes or should I open a separate tracker issue? Terry From asmodai at in-nomine.org Thu Apr 17 22:46:29 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Thu, 17 Apr 2008 22:46:29 +0200 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: References: <480765B5.2010904@voidspace.org.uk> Message-ID: <20080417204629.GS91329@nexus.in-nomine.org> -On [20080417 19:46], Terry Reedy (tjreedy at udel.edu) wrote: >Have you or are you going to make these changes or should I open a separate >tracker issue? You can always put it on my account in the tracker (asmodai) and I'll get to it in the coming days. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ The administration of justice is the firmest pillar of government... From ziade.tarek at gmail.com Fri Apr 18 00:27:06 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 18 Apr 2008 00:27:06 +0200 Subject: [Python-Dev] A smarter shutil.copytree ? In-Reply-To: References: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> Message-ID: <94bdd2610804171527m6ec08d52g4e05517643541316@mail.gmail.com> On Thu, Apr 17, 2008 at 7:06 PM, Guido van Rossum wrote: > Sounds like a neat little feature. Looking forward to it. Maybe the > most useful use case would be to provide glob-style patterns for > skipping files or directories (and their contents). Alright I will work on it that way, thanks for the advice Tarek From jml at mumak.net Fri Apr 18 00:28:17 2008 From: jml at mumak.net (Jonathan Lange) Date: Fri, 18 Apr 2008 08:28:17 +1000 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: References: <4807555B.2070206@voidspace.org.uk> Message-ID: On Fri, Apr 18, 2008 at 12:54 AM, Guido van Rossum wrote: > I'm worried that a mass renaming would do anything but inconvenience > users during the already stressful 2->3 transition. > > I'm more in favor of the original proposal of reducing the redundancy post-3.0. > > If you're looking for useful features, Google has a set of extensions > to unittest.py that I find useful: > > - module-level setUp and tearDown > > - routines for comparing large lists and strings that produce useful > output indicating exactly where the inputs differ. > > - assertLess etc. > > - assertSameElements (sort of like assert(set(x) == set(y)) > Hi, Some things that Bazaar, Twisted and Launchpad have found helpful: - assertRaises returning the exception object that it catches. This allows for easy testing of exception strings. - Assertion methods for 'in', 'is' and 'isinstance' and their negations. - TestCase.addCleanup. This method takes a function, args and kwargs and pushes them to a stack of cleanups. Before tearDown is run, each of these cleanups is popped off the stack and then run. This makes it easier to acquire certain resources in tests: e.g def make_temp_dir(self): dir = tempfile.mkdtemp() self.addCleanup(shutil.rmtree, dir) return dir Luckily, I have patches (with tests!) for these that I will be filing on the tracker as soon as I get the opportunity. jml From jml at mumak.net Fri Apr 18 00:31:15 2008 From: jml at mumak.net (Jonathan Lange) Date: Fri, 18 Apr 2008 08:31:15 +1000 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: <4807555B.2070206@voidspace.org.uk> References: <4807555B.2070206@voidspace.org.uk> Message-ID: On Thu, Apr 17, 2008 at 11:49 PM, Michael Foord wrote: > assert_raises_with_message (exc_class, message, callable, *args, > **keywargs) > I don't think this one should go in. I think it would be better if assertRaises just returned the exception object that it catches. That way, you can test properties of the exception other than its message. jml From guido at python.org Fri Apr 18 00:34:53 2008 From: guido at python.org (Guido van Rossum) Date: Thu, 17 Apr 2008 15:34:53 -0700 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: References: <4807555B.2070206@voidspace.org.uk> Message-ID: On Thu, Apr 17, 2008 at 3:31 PM, Jonathan Lange wrote: > On Thu, Apr 17, 2008 at 11:49 PM, Michael Foord > wrote: > > assert_raises_with_message (exc_class, message, callable, *args, > > **keywargs) > > > > I don't think this one should go in. > > I think it would be better if assertRaises just returned the exception > object that it catches. That way, you can test properties of the > exception other than its message. Hm. I've got to say that returning the exception object is, um, an odd API in the set of unittest APIs. I can see how it's sometimes more powerful, but I'd say that in many cases assertRaisesWithMessage will be easier to write and read. (And making it a regex match would be even cooler.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jml at mumak.net Fri Apr 18 00:40:21 2008 From: jml at mumak.net (Jonathan Lange) Date: Fri, 18 Apr 2008 08:40:21 +1000 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: References: <4807555B.2070206@voidspace.org.uk> Message-ID: On Fri, Apr 18, 2008 at 8:34 AM, Guido van Rossum wrote: > > On Thu, Apr 17, 2008 at 3:31 PM, Jonathan Lange wrote: > > On Thu, Apr 17, 2008 at 11:49 PM, Michael Foord > > wrote: > > > assert_raises_with_message (exc_class, message, callable, *args, > > > **keywargs) > > > > > > > I don't think this one should go in. > > > > I think it would be better if assertRaises just returned the exception > > object that it catches. That way, you can test properties of the > > exception other than its message. > > Hm. I've got to say that returning the exception object is, um, an odd > API in the set of unittest APIs. I can see how it's sometimes more > powerful, but I'd say that in many cases assertRaisesWithMessage will > be easier to write and read. (And making it a regex match would be > even cooler.) I don't know about odd. It works and it's not obviously terrible. Not having it the unittest API simply means that people who do want to test non-message properties will rewrite assertRaises. Which is, in fact, what we've already done. jml From steve at holdenweb.com Fri Apr 18 00:53:51 2008 From: steve at holdenweb.com (Steve Holden) Date: Thu, 17 Apr 2008 18:53:51 -0400 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: References: <4807555B.2070206@voidspace.org.uk> Message-ID: Guido van Rossum wrote: > On Thu, Apr 17, 2008 at 3:31 PM, Jonathan Lange wrote: >> On Thu, Apr 17, 2008 at 11:49 PM, Michael Foord >> wrote: >> > assert_raises_with_message (exc_class, message, callable, *args, >> > **keywargs) >> > >> >> I don't think this one should go in. >> >> I think it would be better if assertRaises just returned the exception >> object that it catches. That way, you can test properties of the >> exception other than its message. > > Hm. I've got to say that returning the exception object is, um, an odd > API in the set of unittest APIs. I can see how it's sometimes more > powerful, but I'd say that in many cases assertRaisesWithMessage will > be easier to write and read. (And making it a regex match would be > even cooler.) > In which case assertRaisesMatching (and then eventually assert_raises_matching) might be a better name for it? regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ From fuzzyman at voidspace.org.uk Fri Apr 18 00:54:18 2008 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 17 Apr 2008 23:54:18 +0100 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> Message-ID: <4807D51A.2060104@voidspace.org.uk> Terry Reedy wrote: > "Michael Foord" wrote in message > news:480765B5.2010904 at voidspace.org.uk... > | I think that there is still work I can do on the docs even before any > | grand renaming... > > In a related thread, I proposed and Guido approved that the 2.6/3.0 docs > be changed to > 1. specify that the Fail... names will disappear in the future > 2. list the preferred Assert names first for each pair > 3. use the preferred Assert names and only those names in the explanatory > text and examples. > > Have you or are you going to make these changes or should I open a separate > tracker issue? > > Yep - these were the doc changes I mentioned. I haven't started them but will unless someone beats me to it. :-) Michael > Terry > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > From tom at vector-seven.com Fri Apr 18 01:40:06 2008 From: tom at vector-seven.com (Thomas Lee) Date: Fri, 18 Apr 2008 09:40:06 +1000 Subject: [Python-Dev] Global Python Sprint Weekends: May 10th-11th and June 21st-22nd. In-Reply-To: <94bdd2610804170644x4db329b0vf49de21a590c51e@mail.gmail.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22F389FE@EXMBX04.exchhosting.com> <4806482C.4030305@voidspace.org.uk> <94bdd2610804170644x4db329b0vf49de21a590c51e@mail.gmail.com> Message-ID: <4807DFD6.5040800@vector-seven.com> Anybody in Melbourne keen for this? Not sure if I'll be able to make it myself, but I'd be interested to know if there's anybody in the area keen to do the sprint. Cheers, T Tarek Ziad? wrote: > On Wed, Apr 16, 2008 at 8:40 PM, Michael Foord > wrote: > >> Trent Nelson wrote: >> > Following on from the success of previous sprint/bugfix weekends and >> > sprinting efforts at PyCon 2008, I'd like to propose the next two >> > Global Python Sprint Weekends take place on the following dates: >> > >> > * May 10th-11th (four days after 2.6a3 and 3.0a5 are released) >> > * June 21st-22nd (~week before 2.6b2 and 3.0b2 are released) >> > >> > It seems there are a few of the Python User Groups keen on meeting >> > up in person and sprinting collaboratively, akin to PyCon, which I >> > highly recommend. I'd like to nominate Saturday across the board >> > as the day for PUGs to meet up in person, with Sunday geared more >> > towards an online collaboration day via IRC, where we can take care >> > of all the little things that got in our way of coding on Saturday >> > (like finalising/preparing/reviewing patches, updating tracker and >> > documentation, writing tests ;-). >> > >> > For User Groups that are planning on meeting up to collaborate, >> > please reply to this thread on python-dev at python.org and let every- >> > one know your intentions! >> > >> > >> >> I should be able to help organise and attend the London contribution. >> Personally I'd like to work on the documentation changes / clean-up for >> the unittest module discussed recently. >> > > We are trying to set up a team here in Paris, > > Personnally I would like to continue the work started in distutils > (various patches) > and some friends here are interested in contributing on documentation. > > Tarek > > From musiccomposition at gmail.com Fri Apr 18 04:24:29 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Thu, 17 Apr 2008 21:24:29 -0500 Subject: [Python-Dev] Interface to change Py3kWarning in Python Message-ID: <1afaf6160804171924g409c3770tcd4a9061fadeb258@mail.gmail.com> I currently have a patch to make it possible to change py3k warnings in Python through new functions in sys: issue 2458. I realize the functions are rather ugly, but I don't think there is another practical way to do it unless you want to be writing PySys_GetObject and checking it for NULL whenever you add a Py3k warning. -- Cheers, Benjamin Peterson From rbp at isnomore.net Fri Apr 18 06:11:08 2008 From: rbp at isnomore.net (Rodrigo Bernardo Pimentel) Date: Fri, 18 Apr 2008 01:11:08 -0300 Subject: [Python-Dev] Global Python Sprint Weekends: May 10th-11th and June 21st-22nd. In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22F389FE@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22F389FE@EXMBX04.exchhosting.com> Message-ID: <20080418041107.GD23236@isnomore.net> On Wed, Apr 16 2008 at 02:51:53PM BRT, Trent Nelson wrote: > > Following on from the success of previous sprint/bugfix weekends and > sprinting efforts at PyCon 2008, I'd like to propose the next two > Global Python Sprint Weekends take place on the following dates: > > * May 10th-11th (four days after 2.6a3 and 3.0a5 are released) > * June 21st-22nd (~week before 2.6b2 and 3.0b2 are released) > > It seems there are a few of the Python User Groups keen on meeting > up in person and sprinting collaboratively, akin to PyCon, which I > highly recommend. I'd like to nominate Saturday across the board > as the day for PUGs to meet up in person, with Sunday geared more > towards an online collaboration day via IRC, where we can take care > of all the little things that got in our way of coding on Saturday > (like finalising/preparing/reviewing patches, updating tracker and > documentation, writing tests ;-). The Sao Paulo User Group gathered "in person" to participate on the last bug day, everyone enjoyed it and we're looking forward to the next one! I've received strong support for our participation on May 10th/11th so we'll definetly be there. rbp -- Rodrigo Bernardo Pimentel | GPG: <0x0DB14978> From solipsis at pitrou.net Fri Apr 18 11:13:05 2008 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 18 Apr 2008 09:13:05 +0000 (UTC) Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> <48040C08.504@canterbury.ac.nz> <48052BC1.40202@canterbury.ac.nz> <48057537.3060003@canterbury.ac.nz> <4806997F.50000@canterbury.ac.nz> Message-ID: Greg Ewing canterbury.ac.nz> writes: > > Really? Nobody else wants a convenient way to > distinguish program bugs from exceptions caused > by external factors? Why do you want to derive program bugs from EnvironmentError ? Usually I derive them from ValueError, RuntimeError or simply Exception. From lists at cheimes.de Fri Apr 18 14:38:47 2008 From: lists at cheimes.de (Christian Heimes) Date: Fri, 18 Apr 2008 14:38:47 +0200 Subject: [Python-Dev] Interface to change Py3kWarning in Python In-Reply-To: <1afaf6160804171924g409c3770tcd4a9061fadeb258@mail.gmail.com> References: <1afaf6160804171924g409c3770tcd4a9061fadeb258@mail.gmail.com> Message-ID: <48089657.6010907@cheimes.de> Benjamin Peterson schrieb: > I currently have a patch to make it possible to change py3k warnings > in Python through new functions in sys: issue 2458. I realize the > functions are rather ugly, but I don't think there is another > practical way to do it unless you want to be writing PySys_GetObject > and checking it for NULL whenever you add a Py3k warning. In Python we usually have to methods for the job, like set_py3kwarningmode and get_py3kwarningmode. I don't like the enable* and disable* style. Even the get/set methods are an ugly workaround for the fact, Python doesn't support module properties. :/ How do you like a macro or function PyErr_Warn3k(msg) that does all the dirty work? Christian From antonyjoseph89 at gmail.com Fri Apr 18 15:41:56 2008 From: antonyjoseph89 at gmail.com (Antony Joseph) Date: Fri, 18 Apr 2008 19:11:56 +0530 Subject: [Python-Dev] Need help in MAPI Message-ID: Hi, My Code: mapi.MAPIInitialize(None) session = mapi.MAPILogonEx(0, MAPIProfile, None, mapi.MAPI_EXTENDED | mapi.MAPI_USE_DEFAULT) I am trying to send a mail using the extended MAPI interface, I am new to work with MAPI. I am trying to execute your code,i getting the following exception, and a popup message of Either there is no default mail client or the current mail client cannot fullfill the messaging request,please run Microsoftoffice outlookand set it as the defaukt mail client. I am using thunderbird as my default mail client , then i set my outlook as my default mail client.its running fine. can u tell me is there possiblites to run the code with out changing the default mail client to Ms Outlook. if could u find me a solution,that'd really helpfull. Error: C:\Documents and Settings\Administrator\Desktop>python mapisend.py Traceback (most recent call last): File "mapisend.py", line 85, in SendEMAPIMail(SendSubject, SendMessage, SendTo, MAPIProfile=MAPIProfile) File "mapisend.py", line 23, in SendEMAPIMail mapi.MAPIInitialize(None) pywintypes.com_error: (-2147467259, 'Unspecified error', None, None) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080418/43cf6dc5/attachment.htm From asmodai at in-nomine.org Fri Apr 18 16:12:16 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Fri, 18 Apr 2008 16:12:16 +0200 Subject: [Python-Dev] Known doctest bug with unicode? Message-ID: <20080418141216.GF64048@nexus.in-nomine.org> Is it a known doctest bug that when you have a dict with Unicode key values that doctest dies with a KeyError? When I excute my code from the regular python interpreter it works as expected. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ Whenever you meet difficult situations dash forward bravely and joyfully... From varmaa at gmail.com Fri Apr 18 16:17:47 2008 From: varmaa at gmail.com (Atul Varma) Date: Fri, 18 Apr 2008 07:17:47 -0700 Subject: [Python-Dev] Known doctest bug with unicode? In-Reply-To: <20080418141216.GF64048@nexus.in-nomine.org> References: <20080418141216.GF64048@nexus.in-nomine.org> Message-ID: <361b27370804180717nce0cd15y2c131ac18478fb56@mail.gmail.com> Can you provide an example that fails? This seems to work on my end, for instance: >>> mydict = { u'\u2026' : 'ellipsis' } >>> mydict[u'\u2026'] 'ellipsis' - Atul On Fri, Apr 18, 2008 at 7:12 AM, Jeroen Ruigrok van der Werven < asmodai at in-nomine.org> wrote: > Is it a known doctest bug that when you have a dict with Unicode key > values > that doctest dies with a KeyError? > > When I excute my code from the regular python interpreter it works as > expected. > > -- > Jeroen Ruigrok van der Werven / asmodai > $B%$%'%k!<%s(B $B%i%&%U%m%C%/(B $B%t%!%s(B $B%G%k(B $B%&%'%k%t%'%s(B > http://www.in-nomine.org/ | http://www.rangaku.org/ > Whenever you meet difficult situations dash forward bravely and > joyfully... > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/varmaa%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080418/25d0e2d8/attachment.htm From asmodai at in-nomine.org Fri Apr 18 16:27:05 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Fri, 18 Apr 2008 16:27:05 +0200 Subject: [Python-Dev] Known doctest bug with unicode? In-Reply-To: <361b27370804180717nce0cd15y2c131ac18478fb56@mail.gmail.com> References: <20080418141216.GF64048@nexus.in-nomine.org> <361b27370804180717nce0cd15y2c131ac18478fb56@mail.gmail.com> Message-ID: <20080418142705.GG64048@nexus.in-nomine.org> # vim: set fileencoding=utf-8 : kanamap = { u'?': 'a' } def transpose(word): """Convert a word in kana to its equivalent Hepburn romanisation. >>> transpose(u'?') 'a' """ transposed = '' for character in word: transposed += kanamap[character] return transposed if __name__ == '__main__': import doctest doctest.testmod() doctest: [16:24] [ruigrok at akuma] (1) {20} % python trans.py ********************************************************************** File "trans.py", line 11, in __main__.transpose Failed example: transpose(u'?') Exception raised: Traceback (most recent call last): File "doctest.py", line 1212, in __run compileflags, 1) in test.globs File "", line 1, in transpose(u'?') File "trans.py", line 16, in transpose transposed += kanamap[character] KeyError: u'\xe3' ********************************************************************** 1 items had failures: 1 of 1 in __main__.transpose ***Test Failed*** 1 failures. normal interpreter: >>> fromm trans import transpose >>> transpose(u'?') 'a' -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ They have learned nothing, and forgotten nothing... From amauryfa at gmail.com Fri Apr 18 16:32:42 2008 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 18 Apr 2008 16:32:42 +0200 Subject: [Python-Dev] Need help in MAPI In-Reply-To: References: Message-ID: Hello, Antony Joseph wrote: > Hi, > > My Code: > mapi.MAPIInitialize(None) > session = mapi.MAPILogonEx(0, MAPIProfile, None, > mapi.MAPI_EXTENDED | mapi.MAPI_USE_DEFAULT) > > I am trying to send a mail using the extended MAPI interface, I am > new to work with MAPI. > I am trying to execute your code,i getting the following exception, and a > popup message of Either there is no default mail client or the current mail > client cannot fullfill the messaging request,please run Microsoftoffice > outlookand set it as the defaukt mail client. > I am using thunderbird as my default mail client , then i set my outlook as > my default mail client.its running fine. > can u tell me is there possiblites to run the code with out changing the > default mail client to Ms Outlook. First, this mailing list is for the development of the python language, not for development with python. Please ask this kind of questions on the comp.lang.python newsgroup. Then, your problem is not related to python at all. The same call from any other program would return the same error. A quick google search gave: http://www.tech-archive.net/Archive/Development/microsoft.public.win32.programmer.messaging/2007-04/msg00036.html which describes the same problem (and a solution) Otherwise you are welcome ;-) -- Amaury Forgeot d'Arc From rhamph at gmail.com Fri Apr 18 18:05:19 2008 From: rhamph at gmail.com (Adam Olsen) Date: Fri, 18 Apr 2008 10:05:19 -0600 Subject: [Python-Dev] Known doctest bug with unicode? In-Reply-To: <20080418142705.GG64048@nexus.in-nomine.org> References: <20080418141216.GF64048@nexus.in-nomine.org> <361b27370804180717nce0cd15y2c131ac18478fb56@mail.gmail.com> <20080418142705.GG64048@nexus.in-nomine.org> Message-ID: On Fri, Apr 18, 2008 at 8:27 AM, Jeroen Ruigrok van der Werven wrote: > # vim: set fileencoding=utf-8 : > > kanamap = { > u'$B$"(B': 'a' > } > > def transpose(word): > """Convert a word in kana to its equivalent Hepburn romanisation. > > >>> transpose(u'$B$"(B') > 'a' > """ > transposed = '' > for character in word: > transposed += kanamap[character] > return transposed > > if __name__ == '__main__': > import doctest > doctest.testmod() > > doctest: > > [16:24] [ruigrok at akuma] (1) {20} % python trans.py > ********************************************************************** > File "trans.py", line 11, in __main__.transpose > Failed example: > transpose(u'$B$"(B') > Exception raised: > Traceback (most recent call last): > File "doctest.py", line 1212, in __run > compileflags, 1) in test.globs > File "", line 1, in > transpose(u'$B$"(B') > File "trans.py", line 16, in transpose > transposed += kanamap[character] > KeyError: u'\xe3' > ********************************************************************** > 1 items had failures: > 1 of 1 in __main__.transpose > ***Test Failed*** 1 failures. > > normal interpreter: > > >>> fromm trans import transpose > >>> transpose(u'$B$"(B') > 'a' What you've got is an 8-bit string containing a unicode literal. Since this gets past the module's compilation stage, it doctest passes it to the compiler again, and it defaults to iso-8859-1. Thus u'$B$"(B'.encode('utf-8').decode('latin-1') -> u'\xe3\x81\x82'. Possible solutions: 1. Make the docstring itself unicode, assuming doctest allows this. 2. Call doctest explicitly, giving it the correct encoding. 3. See if you can put an encoding declaration in the doctest itself. 4. Make doctest smarter, so that it can grab the original module's encoding. 5. Wait until 3.0, where this is hopefully fixed by making doctests use unicode by default? -- Adam Olsen, aka Rhamphoryncus From status at bugs.python.org Fri Apr 18 18:06:37 2008 From: status at bugs.python.org (Tracker) Date: Fri, 18 Apr 2008 18:06:37 +0200 (CEST) Subject: [Python-Dev] Summary of Tracker Issues Message-ID: <20080418160637.1D64178351@psf.upfronthosting.co.za> ACTIVITY SUMMARY (04/11/08 - 04/18/08) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1839 open (+31) / 12641 closed ( +7) / 14480 total (+38) Open issues with patches: 550 Average duration of open issues: 706 days. Median duration of open issues: 1317 days. Open Issues Breakdown open 1817 (+31) pending 22 ( +0) Issues Created Or Reopened (39) _______________________________ Patch to emit "-J is reserved for Jython" on -J arg 04/11/08 CLOSED http://bugs.python.org/issue2617 created fwierzbicki patch Tile module: Add support for themed widgets 04/11/08 http://bugs.python.org/issue2618 created wordtech Document memoryview 04/11/08 http://bugs.python.org/issue2619 created benjamin.peterson Multiple buffer overflows in unicode processing 04/11/08 http://bugs.python.org/issue2620 created jnferguson rename test_support to support 04/12/08 http://bugs.python.org/issue2621 created benjamin.peterson patch, easy Import errors in email.message.py 04/12/08 http://bugs.python.org/issue2622 created JohnJackson Patch: xmlrpclib client ignores datetime tzinfo when creating is 04/12/08 http://bugs.python.org/issue2623 created lclark patch swig support in distutils should use the build and temp dirs 04/12/08 http://bugs.python.org/issue2624 created afflux patch mailbox.MH.get_message() treats result of get_sequences() as lis 04/12/08 http://bugs.python.org/issue2625 created hcs If compile with gcc 4.3.0 python interpreter itself eats all mem 04/12/08 CLOSED http://bugs.python.org/issue2626 created orivej mark pgen generated files 04/13/08 CLOSED http://bugs.python.org/issue2627 created benjamin.peterson patch, easy ftplib Persistent data connection 04/13/08 http://bugs.python.org/issue2628 created jbell patch _Py_ForgetReference crash when called from _PyUnicode_New on Mem 04/14/08 http://bugs.python.org/issue2629 created gregory.p.smith repr() should not escape non-ASCII characters 04/14/08 http://bugs.python.org/issue2630 created ishimoto patch IMPORT_NAME Documentation is incomplete 04/14/08 http://bugs.python.org/issue2631 created pib performance problem in socket._fileobject.read 04/14/08 http://bugs.python.org/issue2632 created CurtHagenlocher patch Improve subprocess.Popen() documentation ("env" parameter) 04/14/08 http://bugs.python.org/issue2633 created roysmith os.execvpe() docs need to be more specific 04/15/08 http://bugs.python.org/issue2634 created roysmith textwrap: bug in 'fix_sentence_endings' option 04/15/08 http://bugs.python.org/issue2635 created gscelsi Regexp 2.6 (modifications to current re 2.2.2) 04/15/08 http://bugs.python.org/issue2636 created timehorse patch urllib.quote() escapes characters unnecessarily and contrary to 04/15/08 http://bugs.python.org/issue2637 created tlesher tkSimpleDialog Window Flashing 04/15/08 http://bugs.python.org/issue2638 created Longorh shutil.copyfile() documentation is vague 04/15/08 http://bugs.python.org/issue2639 created roysmith "excel" csv option generates multiple lines 04/15/08 CLOSED http://bugs.python.org/issue2640 created laxrulz777 setuptools gets site-packages wrong on Mac 04/15/08 http://bugs.python.org/issue2641 created jorendorff MSVCRT packing in Windows Installer (3.0a4) 04/16/08 http://bugs.python.org/issue2642 created wesley.spikes mmap_object_dealloc does not call FlushViewOfFile on windows 04/16/08 http://bugs.python.org/issue2643 created schmir errors from msync ignored in mmap_object_dealloc 04/16/08 http://bugs.python.org/issue2644 created schmir httplib throws ValueError 04/16/08 http://bugs.python.org/issue2645 created niallo patch Python does not accept unicode keywords 04/16/08 http://bugs.python.org/issue2646 created j5 patch XML munges apos entity in tag content 04/17/08 CLOSED http://bugs.python.org/issue2647 created resplin decimal receipe moneyfmt suppress leading 0 04/17/08 CLOSED http://bugs.python.org/issue2648 created cgrohmann patch poss. patch for fnmatch.py to add {.htm,html} style globbing 04/17/08 http://bugs.python.org/issue2649 created mark re.escape should not escape underscore 04/17/08 http://bugs.python.org/issue2650 created rsc easy Strings passed to KeyError do not round trip 04/17/08 http://bugs.python.org/issue2651 created rharris 64 bit python memory leak usage 04/17/08 http://bugs.python.org/issue2652 created kevin3210 string to float to int 04/18/08 CLOSED http://bugs.python.org/issue2653 created wr Typo at http://docs.python.org/dev/3.0/howto/doanddont.html 04/18/08 http://bugs.python.org/issue2654 created cartman IOBaseError 04/12/08 CLOSED http://bugs.python.org/issue1481036 reopened gregory.p.smith Issues Now Closed (28) ______________________ reduce tarfile memory footprint 64 days http://bugs.python.org/issue2058 lars.gustaebel patch CSV Sniffer does not function properly on single column .csv fil 60 days http://bugs.python.org/issue2078 skip.montanaro patch Patch to add a get_data function to pkgutil 26 days http://bugs.python.org/issue2439 ncoghlan patch int and float accept bytes, complex does not 22 days http://bugs.python.org/issue2483 marketdickinson Bazaar ignore file 15 days http://bugs.python.org/issue2510 benjamin.peterson patch, easy Document IO module 12 days http://bugs.python.org/issue2530 benjamin.peterson patch atom sorting error when building ctypes 9 days http://bugs.python.org/issue2559 theller why is (default)dict so slow on tuples??? 1 days http://bugs.python.org/issue2607 rhettinger xml.dom.minidom documentation consistency and update 1 days http://bugs.python.org/issue2615 georg.brandl patch ctypes.pointer(), ctypes.POINTER() speedup 3 days http://bugs.python.org/issue2616 theller patch, patch Patch to emit "-J is reserved for Jython" on -J arg 1 days http://bugs.python.org/issue2617 fwierzbicki patch If compile with gcc 4.3.0 python interpreter itself eats all mem 0 days http://bugs.python.org/issue2626 loewis mark pgen generated files 0 days http://bugs.python.org/issue2627 benjamin.peterson patch, easy "excel" csv option generates multiple lines 1 days http://bugs.python.org/issue2640 skip.montanaro XML munges apos entity in tag content 0 days http://bugs.python.org/issue2647 loewis decimal receipe moneyfmt suppress leading 0 0 days http://bugs.python.org/issue2648 rhettinger patch string to float to int 0 days http://bugs.python.org/issue2653 rhettinger ^$ won't split on empty line 1595 days http://bugs.python.org/issue852532 mkc Keyword or function to invoke pdb programmatically 1592 days http://bugs.python.org/issue853569 benjamin.peterson readline not implemented for UTF-16 1483 days http://bugs.python.org/issue920680 benjamin.peterson overall index to distributed documentation 1417 days http://bugs.python.org/issue960845 benjamin.peterson Memory leak in socket.py on Mac OS X 1204 days http://bugs.python.org/issue1092502 gregory.p.smith cmd.Cmd().cmdloop() can't read from file 1136 days http://bugs.python.org/issue1156280 georg.brandl IOBaseError 1 days http://bugs.python.org/issue1481036 arigo implement warnings module in C 460 days http://bugs.python.org/issue1631171 brett.cannon patch object.__init__ shouldn't allow args/kwds 393 days http://bugs.python.org/issue1683368 gvanrossum code that writes the PKG-INFO file doesnt handle unicode 330 days http://bugs.python.org/issue1721241 georg.brandl cvs.get_dialect() return a class object 290 days http://bugs.python.org/issue1744580 skip.montanaro Top Issues Most Discussed (10) ______________________________ 19 Make range __eq__ work 9 days open http://bugs.python.org/issue2603 14 repr() should not escape non-ASCII characters 4 days open http://bugs.python.org/issue2630 12 Multiple buffer overflows in unicode processing 7 days open http://bugs.python.org/issue2620 11 string representation of range and dictionary views 8 days open http://bugs.python.org/issue2610 10 performance problem in socket._fileobject.read 4 days open http://bugs.python.org/issue2632 9 Python 2.5 64 bit compile fails on Solaris 10/gcc 4.1.1 469 days open http://bugs.python.org/issue1628484 8 [regression] reading from a urllib2 file descriptor happens byt 10 days open http://bugs.python.org/issue2601 6 Fast BytesIO implementation + misc changes 102 days open http://bugs.python.org/issue1751 6 IOBaseError 1 days closed http://bugs.python.org/issue1481036 5 Import errors in email.message.py 6 days open http://bugs.python.org/issue2622 From ncoghlan at gmail.com Fri Apr 18 19:41:01 2008 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Apr 2008 03:41:01 +1000 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: <4807555B.2070206@voidspace.org.uk> References: <4807555B.2070206@voidspace.org.uk> Message-ID: <4808DD2D.5090605@gmail.com> Michael Foord wrote: > that the 'assert' statement should be used > instead of 'assert_'. assert statements are actually a bad idea in tests - they get skipped when the test suite is run with optimisation switched on. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From musiccomposition at gmail.com Fri Apr 18 22:31:14 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Fri, 18 Apr 2008 15:31:14 -0500 Subject: [Python-Dev] Interface to change Py3kWarning in Python In-Reply-To: <48089657.6010907@cheimes.de> References: <1afaf6160804171924g409c3770tcd4a9061fadeb258@mail.gmail.com> <48089657.6010907@cheimes.de> Message-ID: <1afaf6160804181331q5ef5f41dvec83f6c74758fb24@mail.gmail.com> On Fri, Apr 18, 2008 at 7:38 AM, Christian Heimes wrote: > Benjamin Peterson schrieb: > > > > In Python we usually have to methods for the job, like > set_py3kwarningmode and get_py3kwarningmode. I don't like the enable* > and disable* style. Even the get/set methods are an ugly workaround for > the fact, Python doesn't support module properties. :/ I will look at that. > > How do you like a macro or function PyErr_Warn3k(msg) that does all the > dirty work? Sounds lovely. > > Christian > > -- Cheers, Benjamin Peterson From musiccomposition at gmail.com Sat Apr 19 00:43:01 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Fri, 18 Apr 2008 17:43:01 -0500 Subject: [Python-Dev] unscriptable? Message-ID: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> Consider this error: >>> 3["something"] Traceback (most recent call last): File "", line 1, in TypeError: 'int' object is unsubscriptable "unscriptable" seems rather ambiguous. How about "[object] cannot be indexed"? -- Cheers, Benjamin Peterson From musiccomposition at gmail.com Sat Apr 19 01:03:09 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Fri, 18 Apr 2008 18:03:09 -0500 Subject: [Python-Dev] unscriptable? In-Reply-To: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> Message-ID: <1afaf6160804181603x1c5ecfcagcac4534321a37c6c@mail.gmail.com> On Fri, Apr 18, 2008 at 5:43 PM, Benjamin Peterson wrote: > Consider this error: > >>> 3["something"] > Traceback (most recent call last): > File "", line 1, in > TypeError: 'int' object is unsubscriptable > > "unscriptable" seems rather ambiguous. How about "[object] cannot be indexed"? Titus just noticed that I confused "unscriptable" with "unsubscriptable." :P Still, though, unsubscriptable seems to be a Python invented word. What does (un)subscriptable even mean? > > -- > Cheers, > Benjamin Peterson > -- Cheers, Benjamin Peterson From scott+python-dev at scottdial.com Sat Apr 19 01:15:07 2008 From: scott+python-dev at scottdial.com (Scott Dial) Date: Fri, 18 Apr 2008 19:15:07 -0400 Subject: [Python-Dev] unscriptable? In-Reply-To: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> Message-ID: <48092B7B.1090503@scottdial.com> Benjamin Peterson wrote: > Consider this error: >>>> 3["something"] > Traceback (most recent call last): > File "", line 1, in > TypeError: 'int' object is unsubscriptable > > "unscriptable" seems rather ambiguous. How about "[object] cannot be indexed"? > Although I think this is a bit of bike-shedding, I would point out that CPython is itself inconsistent about this: >>> set([1,2,3])[0] Traceback (most recent call last): File "", line 1, in TypeError: 'set' object is unindexable I tend to agree with Benjamin that "unsubscriptable" is a made-up word, but so is "unindexable". So, whatever. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From fuzzyman at voidspace.org.uk Sat Apr 19 01:19:00 2008 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 19 Apr 2008 00:19:00 +0100 Subject: [Python-Dev] unscriptable? In-Reply-To: <48092B7B.1090503@scottdial.com> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <48092B7B.1090503@scottdial.com> Message-ID: <48092C64.80209@voidspace.org.uk> Scott Dial wrote: > Benjamin Peterson wrote: > >> Consider this error: >> >>>>> 3["something"] >>>>> >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: 'int' object is unsubscriptable >> >> "unscriptable" seems rather ambiguous. How about "[object] cannot be indexed"? >> >> > > Although I think this is a bit of bike-shedding, I would point out that > CPython is itself inconsistent about this: > > >>> set([1,2,3])[0] > Traceback (most recent call last): > File "", line 1, in > TypeError: 'set' object is unindexable > > I tend to agree with Benjamin that "unsubscriptable" is a made-up word, > but so is "unindexable". So, whatever. > It seems to be more common in the 'Python world' to refer to indexing than subscription and I admit that this error message has confused me in the past. Michael > -Scott > > From martin at v.loewis.de Sat Apr 19 02:08:07 2008 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 19 Apr 2008 02:08:07 +0200 Subject: [Python-Dev] unscriptable? In-Reply-To: <48092B7B.1090503@scottdial.com> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <48092B7B.1090503@scottdial.com> Message-ID: <480937E7.8050006@v.loewis.de> > I tend to agree with Benjamin that "unsubscriptable" is a made-up word, > but so is "unindexable". So, whatever. FWIW, here is what other tools give as error message in similar cases: gcc: subscripted value is neither array nor pointer g++: invalid types ?double[int]? for array subscript javac: array required, but X found ruby: undefined method `[]=' for 1:Fixnum (NoMethodError) See also http://tinyurl.com/4nn284 Regards, Martin From greg.ewing at canterbury.ac.nz Sat Apr 19 02:18:47 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 19 Apr 2008 12:18:47 +1200 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? In-Reply-To: References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> <48040C08.504@canterbury.ac.nz> <48052BC1.40202@canterbury.ac.nz> <48057537.3060003@canterbury.ac.nz> <4806997F.50000@canterbury.ac.nz> Message-ID: <48093A67.5070702@canterbury.ac.nz> Antoine Pitrou wrote: > Why do you want to derive program bugs from EnvironmentError ? Usually I derive > them from ValueError, RuntimeError or simply Exception. I'm *not* talking about program bugs, I'm talking about exceptions due to something the user did wrong. I like to be able to do this: try: f = open(somefile) mungulate(f) f.close() except EnvironmentError, e: big_nasty_alert("Couldn't mungulate: %s" % e) I *don't* want this to catch things like TypeError and ValueError that indicate a program bug if they ever escape. I want those to propagate and cause a traceback or get handled at a higher level. By deriving my own external-factors exceptions from EnvironmentError, this all works very nicely. From its position in the exception hierarchy, this seems to be just the sort of thing that EnvironmentError is designed for, and I'm perplexed to be told that it's not. -- Greg From greg.ewing at canterbury.ac.nz Sat Apr 19 02:52:01 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 19 Apr 2008 12:52:01 +1200 Subject: [Python-Dev] unscriptable? In-Reply-To: <48092C64.80209@voidspace.org.uk> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <48092B7B.1090503@scottdial.com> <48092C64.80209@voidspace.org.uk> Message-ID: <48094231.8050607@canterbury.ac.nz> Michael Foord wrote: > It seems to be more common in the 'Python world' to refer to indexing > than subscription Especially since the subscript isn't actually written as a subscript, unless you have a particulary fancy code editor... -- Greg From steve at pearwood.info Sat Apr 19 04:12:55 2008 From: steve at pearwood.info (Steven) Date: Sat, 19 Apr 2008 12:12:55 +1000 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? In-Reply-To: <48093A67.5070702@canterbury.ac.nz> References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> <48040C08.504@canterbury.ac.nz> <48052BC1.40202@canterbury.ac.nz> <48057537.3060003@canterbury.ac.nz> <4806997F.50000@canterbury.ac.nz> <48093A67.5070702@canterbury.ac.nz> Message-ID: <20080419121255.150520e9.steve@pearwood.info> On Sat, 19 Apr 2008 12:18:47 +1200 Greg Ewing wrote: > Antoine Pitrou wrote: > > > Why do you want to derive program bugs from EnvironmentError ? Usually I derive > > them from ValueError, RuntimeError or simply Exception. > > I'm *not* talking about program bugs, I'm talking about > exceptions due to something the user did wrong. > > I like to be able to do this: > > try: > f = open(somefile) > mungulate(f) > f.close() > except EnvironmentError, e: > big_nasty_alert("Couldn't mungulate: %s" % e) It might help if you explain what sort of actual things that the user does wrong that you are talking about. From where I'm sitting, the only thing I can see that the user could do wrong is specify the wrong file. That doesn't sound like an EnvironmentError to me, but I don't know that it should be a ValueError or TypeError either. -- Steven From skip at pobox.com Sat Apr 19 12:39:40 2008 From: skip at pobox.com (skip at pobox.com) Date: Sat, 19 Apr 2008 05:39:40 -0500 Subject: [Python-Dev] test_subprocess failing Message-ID: <18441.52204.719333.678164@montanaro-dyndns-org.local> test_subprocess failed in Neal's regression test overnight. My guess for the culprit would be Christian's checkin on Friday: r62386 | christian.heimes | 2008-04-18 21:23:57 -0500 (Fri, 18 Apr 2008) | 2 lines Added kill, terminate and send_signal to subprocess.Popen. The bits and pieces for the Windows side were already in place. The POSIX side is trivial (as usual) and uses os.kill(). Skip From ncoghlan at gmail.com Sat Apr 19 14:13:19 2008 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 19 Apr 2008 22:13:19 +1000 Subject: [Python-Dev] unscriptable? In-Reply-To: <1afaf6160804181603x1c5ecfcagcac4534321a37c6c@mail.gmail.com> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <1afaf6160804181603x1c5ecfcagcac4534321a37c6c@mail.gmail.com> Message-ID: <4809E1DF.9020306@gmail.com> Benjamin Peterson wrote: > On Fri, Apr 18, 2008 at 5:43 PM, Benjamin Peterson > wrote: >> Consider this error: >> >>> 3["something"] >> Traceback (most recent call last): >> File "", line 1, in >> TypeError: 'int' object is unsubscriptable >> >> "unscriptable" seems rather ambiguous. How about "[object] cannot be indexed"? > Titus just noticed that I confused "unscriptable" with > "unsubscriptable." :P Still, though, unsubscriptable seems to be a > Python invented word. > What does (un)subscriptable even mean? You can't pass the object a subscript (the expression between the square brackets) and get a meaningful answer. Being indexable is subtly different from being subscriptable - the former has stronger connotations of numeric indices and sequence-like behaviour (particularly since the introduction of operator.index), while the latter merely states that the container provides some kinds of mapping from subscripts to values in the container, without provide any implications as to the nature of that mapping. 'Unsubscriptable' may be a bit clumsy, but it's as accurate a description of the error as you're likely to find. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From steve at pearwood.info Sat Apr 19 15:05:08 2008 From: steve at pearwood.info (Steven) Date: Sat, 19 Apr 2008 23:05:08 +1000 Subject: [Python-Dev] unscriptable? In-Reply-To: <4809E1DF.9020306@gmail.com> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <1afaf6160804181603x1c5ecfcagcac4534321a37c6c@mail.gmail.com> <4809E1DF.9020306@gmail.com> Message-ID: <20080419230508.8468355a.steve@pearwood.info> On Sat, 19 Apr 2008 22:13:19 +1000 Nick Coghlan wrote: > Being indexable is subtly different from being subscriptable - the > former has stronger connotations of numeric indices and sequence-like > behaviour I dispute this. Indices aren't necessarily numeric (think of an A-Z file), and I don't believe you are correct about the connotations of indexable being numeric or sequence-like. Think of index cards. With my maths background, I would expect that subscripts are (almost) always numeric, e.g. x-subscript-i means the i-th x value, where i is a natural number. +0 on keeping unsubscriptable. +1 on changing it to unidexable. -1 on keeping the current misspelling. -- Steven D'Aprano From g.brandl at gmx.net Sat Apr 19 16:12:53 2008 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 19 Apr 2008 16:12:53 +0200 Subject: [Python-Dev] Interface to change Py3kWarning in Python In-Reply-To: <48089657.6010907@cheimes.de> References: <1afaf6160804171924g409c3770tcd4a9061fadeb258@mail.gmail.com> <48089657.6010907@cheimes.de> Message-ID: Christian Heimes schrieb: > Benjamin Peterson schrieb: >> I currently have a patch to make it possible to change py3k warnings >> in Python through new functions in sys: issue 2458. I realize the >> functions are rather ugly, but I don't think there is another >> practical way to do it unless you want to be writing PySys_GetObject >> and checking it for NULL whenever you add a Py3k warning. > > In Python we usually have to methods for the job, like > set_py3kwarningmode and get_py3kwarningmode. I don't like the enable* > and disable* style. Even the get/set methods are an ugly workaround for > the fact, Python doesn't support module properties. :/ > > How do you like a macro or function PyErr_Warn3k(msg) that does all the > dirty work? Which additional dirty work would it do? It should never have to do more than check a simple static C variable, for obvious reasons. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From musiccomposition at gmail.com Sat Apr 19 17:37:36 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Sat, 19 Apr 2008 10:37:36 -0500 Subject: [Python-Dev] Interface to change Py3kWarning in Python In-Reply-To: References: <1afaf6160804171924g409c3770tcd4a9061fadeb258@mail.gmail.com> <48089657.6010907@cheimes.de> Message-ID: <1afaf6160804190837s1ac6ecfbxe6073cd813656942@mail.gmail.com> On Sat, Apr 19, 2008 at 9:12 AM, Georg Brandl wrote: > Christian Heimes schrieb: > > > > How do you like a macro or function PyErr_Warn3k(msg) that does all the > > dirty work? > > Which additional dirty work would it do? It should never have to do more than > check a simple static C variable, for obvious reasons. Even if there's no dirty work, it could clean up this: if (Py_Py3kWarningFlag && PyErr_WarnEx(PyErr_DeprecationWarning, "msg", 2) < 0) to if (PyErr_WarnPy3k("msg", 2) < 0) > > Georg -- Cheers, Benjamin Peterson From lists at cheimes.de Sat Apr 19 20:19:29 2008 From: lists at cheimes.de (Christian Heimes) Date: Sat, 19 Apr 2008 20:19:29 +0200 Subject: [Python-Dev] test_subprocess failing In-Reply-To: <18441.52204.719333.678164@montanaro-dyndns-org.local> References: <18441.52204.719333.678164@montanaro-dyndns-org.local> Message-ID: <480A37B1.8000006@cheimes.de> skip at pobox.com schrieb: > test_subprocess failed in Neal's regression test overnight. My guess for > the culprit would be Christian's checkin on Friday: > > r62386 | christian.heimes | 2008-04-18 21:23:57 -0500 (Fri, 18 Apr 2008) | 2 lines > > Added kill, terminate and send_signal to subprocess.Popen. The bits and > pieces for the Windows side were already in place. The POSIX side is trivial > (as usual) and uses os.kill(). The processes were killed too early. The tests now wait until the subprocess sends out a string. Christian From g.brandl at gmx.net Sat Apr 19 23:23:36 2008 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 19 Apr 2008 23:23:36 +0200 Subject: [Python-Dev] unscriptable? In-Reply-To: <20080419230508.8468355a.steve@pearwood.info> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <1afaf6160804181603x1c5ecfcagcac4534321a37c6c@mail.gmail.com> <4809E1DF.9020306@gmail.com> <20080419230508.8468355a.steve@pearwood.info> Message-ID: Steven schrieb: > On Sat, 19 Apr 2008 22:13:19 +1000 > Nick Coghlan wrote: > >> Being indexable is subtly different from being subscriptable - the >> former has stronger connotations of numeric indices and sequence-like >> behaviour > > I dispute this. Indices aren't necessarily numeric (think of an A-Z > file), and I don't believe you are correct about the connotations of > indexable being numeric or sequence-like. Think of index cards. In normal language use, perhaps -- I'm no native speaker. But in the Python world, Nick is right -- think of the __index__ special method that adapts the object to a sequence index; it has to return an integer. > With my maths background, I would expect that subscripts are (almost) > always numeric, e.g. x-subscript-i means the i-th x value, where i > is a natural number. Maths background isn't always directly transferable to computer languages. Think 1 / 2 == 0 -- with terms, it's even worse. An interesting point here is that the special methods are called __getitem__ et al., smartly evading the problem. "'int' object has no items" sounds equally strange to me though. > +0 on keeping unsubscriptable. > +1 on changing it to unidexable. > -1 on keeping the current misspelling. There is currently no misspelling. Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From g.brandl at gmx.net Sat Apr 19 23:26:35 2008 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 19 Apr 2008 23:26:35 +0200 Subject: [Python-Dev] 3k checkin mails to python-checkins Message-ID: Since a few days, checkin notifications for the 3k branch seem to be sent to both the python-checkins and the python-3000-checkins lists. Was that a deliberate decision or has some bug crept into the SVN hook? Georg -- Thus spake the Lord: Thou shalt indent with four spaces. No more, no less. Four shall be the number of spaces thou shalt indent, and the number of thy indenting shall be four. Eight shalt thou not indent, nor either indent thou two, excepting that thou then proceed to four. Tabs are right out. From jimjjewett at gmail.com Sat Apr 19 23:31:12 2008 From: jimjjewett at gmail.com (Jim Jewett) Date: Sat, 19 Apr 2008 17:31:12 -0400 Subject: [Python-Dev] unscriptable? Message-ID: > I dispute this. Indices aren't necessarily numeric > (think of an A-Z file), Python has recently added an __index__ slot which means "as an integer, and I really am an integer, I'm not just rounding like int(3.4) would do" So in the context of python, an index is numeric, whereas "subscript" has already been used for hashtables with arbitrary keys. -jJ From stephen at xemacs.org Sun Apr 20 01:03:17 2008 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 20 Apr 2008 08:03:17 +0900 Subject: [Python-Dev] unscriptable? In-Reply-To: <20080419230508.8468355a.steve@pearwood.info> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <1afaf6160804181603x1c5ecfcagcac4534321a37c6c@mail.gmail.com> <4809E1DF.9020306@gmail.com> <20080419230508.8468355a.steve@pearwood.info> Message-ID: <87zlrpcuru.fsf@uwakimon.sk.tsukuba.ac.jp> Steven writes: > On Sat, 19 Apr 2008 22:13:19 +1000 > Nick Coghlan wrote: > > > Being indexable is subtly different from being subscriptable - the > > former has stronger connotations of numeric indices and sequence-like > > behaviour > With my maths background, I would expect that subscripts are (almost) > always numeric, e.g. x-subscript-i means the i-th x value, where i > is a natural number. I don't think you can get general agreement on which tend to connote numerical indicies more. On the other hand, in many contexts "index" is a general concept, while "subscript" is a typographical convention for representing an index. +1 on indexable +0.5 on subscriptable -0.1 on scriptable From scott+python-dev at scottdial.com Sun Apr 20 01:00:29 2008 From: scott+python-dev at scottdial.com (Scott Dial) Date: Sat, 19 Apr 2008 19:00:29 -0400 Subject: [Python-Dev] unscriptable? In-Reply-To: References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <1afaf6160804181603x1c5ecfcagcac4534321a37c6c@mail.gmail.com> <4809E1DF.9020306@gmail.com> <20080419230508.8468355a.steve@pearwood.info> Message-ID: <480A798D.5040806@scottdial.com> Georg Brandl wrote: > There is currently no misspelling. We should at least make the set object match the others. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From greg.ewing at canterbury.ac.nz Sun Apr 20 01:04:56 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 20 Apr 2008 11:04:56 +1200 Subject: [Python-Dev] unscriptable? In-Reply-To: <4809E1DF.9020306@gmail.com> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <1afaf6160804181603x1c5ecfcagcac4534321a37c6c@mail.gmail.com> <4809E1DF.9020306@gmail.com> Message-ID: <480A7A98.8020407@canterbury.ac.nz> Nick Coghlan wrote: > Being indexable is subtly different from being subscriptable - the > former has stronger connotations of numeric indices and sequence-like > behaviour (particularly since the introduction of operator.index), while > the latter merely states that the container provides some kinds of > mapping from subscripts to values in the container, without provide any > implications as to the nature of that mapping. I don't think everyone would agree with that. For example, databases have these things that are called "indexes" even when the value being used as a key isn't a number. I've never really liked the term "subscript" in this context, because to me it implies a typographical feature, i.e. writing something below the baseline, which isn't normally done in a textual programming language. The term "index", on the other hand, directly suggests the idea of looking something up, without relying on indirection through typography and mathematical convention. -- Greg From ncoghlan at gmail.com Sun Apr 20 03:03:21 2008 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 20 Apr 2008 11:03:21 +1000 Subject: [Python-Dev] unscriptable? In-Reply-To: <87zlrpcuru.fsf@uwakimon.sk.tsukuba.ac.jp> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <1afaf6160804181603x1c5ecfcagcac4534321a37c6c@mail.gmail.com> <4809E1DF.9020306@gmail.com> <20080419230508.8468355a.steve@pearwood.info> <87zlrpcuru.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <480A9659.30704@gmail.com> Stephen J. Turnbull wrote: > Steven writes: > > On Sat, 19 Apr 2008 22:13:19 +1000 > > Nick Coghlan wrote: > > > > > Being indexable is subtly different from being subscriptable - the > > > former has stronger connotations of numeric indices and sequence-like > > > behaviour > > > With my maths background, I would expect that subscripts are (almost) > > always numeric, e.g. x-subscript-i means the i-th x value, where i > > is a natural number. > > I don't think you can get general agreement on which tend to connote > numerical indicies more. Indices and arrays go hand-in-hand for me (probably due to my signal processing background) and I believe PEP 357's choice of __index__ for sequence indexing made that association reasonably official in Python terms. > On the other hand, in many contexts "index" > is a general concept, while "subscript" is a typographical convention > for representing an index. Granted, but the latter is explicitly used as the general term for sequence indexing and mapping in the relevant section of the language reference [1]. So specifically in the context of Python, the error message from 'int' is absolutely correct, and the error message from 'set' isn't technically wrong either (although perhaps not quite as correct as it could be). To suggest yet another colour for the bikeshed, maybe we should ditch both unindexable and unsubscriptable and go with "'int' is not a sequence or mapping". Any object which supports subscript notation will be one or the other. Cheers, Nick. [1] http://docs.python.org/dev/reference/expressions.html#id7 -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From steve at holdenweb.com Sun Apr 20 03:54:46 2008 From: steve at holdenweb.com (Steve Holden) Date: Sat, 19 Apr 2008 21:54:46 -0400 Subject: [Python-Dev] unscriptable? In-Reply-To: <480A9659.30704@gmail.com> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <1afaf6160804181603x1c5ecfcagcac4534321a37c6c@mail.gmail.com> <4809E1DF.9020306@gmail.com> <20080419230508.8468355a.steve@pearwood.info> <87zlrpcuru.fsf@uwakimon.sk.tsukuba.ac.jp> <480A9659.30704@gmail.com> Message-ID: Nick Coghlan wrote: [...]> > To suggest yet another colour for the bikeshed, maybe we should ditch > both unindexable and unsubscriptable and go with "'int' is not a > sequence or mapping". Any object which supports subscript notation will > be one or the other. > All the object needs to to to be "indexable" or "subscriptable" is to implement .__index__() or .__getitem__(). While sequences and mappings are the only built-in types to do so, this says nothing about user-defined types. Couldn't we find a more obvious and direct error message like "Illegal use of [] subscripting/indexing"? one-more-coat-for-the-bikeshed-ly y'rs - steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ From stephen at xemacs.org Sun Apr 20 04:33:57 2008 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 20 Apr 2008 11:33:57 +0900 Subject: [Python-Dev] unscriptable? In-Reply-To: <480A9659.30704@gmail.com> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <1afaf6160804181603x1c5ecfcagcac4534321a37c6c@mail.gmail.com> <4809E1DF.9020306@gmail.com> <20080419230508.8468355a.steve@pearwood.info> <87zlrpcuru.fsf@uwakimon.sk.tsukuba.ac.jp> <480A9659.30704@gmail.com> Message-ID: <87wsmtcl0q.fsf@uwakimon.sk.tsukuba.ac.jp> Nick Coghlan writes: > To suggest yet another colour for the bikeshed, maybe we should ditch > both unindexable and unsubscriptable and go with "'int' is not a > sequence or mapping". I like this best. From asmodai at in-nomine.org Sun Apr 20 10:11:11 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Sun, 20 Apr 2008 10:11:11 +0200 Subject: [Python-Dev] unscriptable? In-Reply-To: <480A9659.30704@gmail.com> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <1afaf6160804181603x1c5ecfcagcac4534321a37c6c@mail.gmail.com> <4809E1DF.9020306@gmail.com> <20080419230508.8468355a.steve@pearwood.info> <87zlrpcuru.fsf@uwakimon.sk.tsukuba.ac.jp> <480A9659.30704@gmail.com> Message-ID: <20080420081111.GO64048@nexus.in-nomine.org> -On [20080420 03:04], Nick Coghlan (ncoghlan at gmail.com) wrote: >To suggest yet another colour for the bikeshed, maybe we should ditch >both unindexable and unsubscriptable and go with "'int' is not a >sequence or mapping". Any object which supports subscript notation will >be one or the other. Sounds even clearer to me. +1 -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B It is necessary to be noble, and yet take humility as a basis. It is necessary to be exalted, and yet take modesty as a foundation... From martin at v.loewis.de Sun Apr 20 11:25:28 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 20 Apr 2008 11:25:28 +0200 Subject: [Python-Dev] 3k checkin mails to python-checkins In-Reply-To: References: Message-ID: <480B0C08.2010501@v.loewis.de> > Since a few days, checkin notifications for the 3k branch seem to be sent > to both the python-checkins and the python-3000-checkins lists. Was that a > deliberate decision or has some bug crept into the SVN hook? It's a bug, I'll look into it (when I find the time). Regards, Martin From ziade.tarek at gmail.com Mon Apr 21 00:15:57 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 21 Apr 2008 00:15:57 +0200 Subject: [Python-Dev] A smarter shutil.copytree ? In-Reply-To: <94bdd2610804171527m6ec08d52g4e05517643541316@mail.gmail.com> References: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> <94bdd2610804171527m6ec08d52g4e05517643541316@mail.gmail.com> Message-ID: <94bdd2610804201515j24a3f7e4uf7ab98cfa915fafb@mail.gmail.com> I have submitted a patch for review here: http://bugs.python.org/issue2663 glob-style patterns or a callable (for complex cases) can be provided to filter out files or directories. Regards Tarek From steven.bethard at gmail.com Mon Apr 21 02:25:34 2008 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 20 Apr 2008 18:25:34 -0600 Subject: [Python-Dev] A smarter shutil.copytree ? In-Reply-To: <94bdd2610804201515j24a3f7e4uf7ab98cfa915fafb@mail.gmail.com> References: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> <94bdd2610804171527m6ec08d52g4e05517643541316@mail.gmail.com> <94bdd2610804201515j24a3f7e4uf7ab98cfa915fafb@mail.gmail.com> Message-ID: On Sun, Apr 20, 2008 at 4:15 PM, Tarek Ziad? wrote: > I have submitted a patch for review here: http://bugs.python.org/issue2663 > > glob-style patterns or a callable (for complex cases) can be provided > to filter out files or directories. I'm not a big fan of the sequence-or-callable argument. Why not just make it a callable argument, and supply a utility function so that you can write something like:: exclude_func = shutil.excluding_patterns('*.tmp', 'test_dir2') shutil.copytree(src_dir, dst_dir, exclude=exclude_func) ? Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From ijmorlan at cs.uwaterloo.ca Mon Apr 21 02:38:03 2008 From: ijmorlan at cs.uwaterloo.ca (Isaac Morland) Date: Sun, 20 Apr 2008 20:38:03 -0400 (EDT) Subject: [Python-Dev] A smarter shutil.copytree ? In-Reply-To: References: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> <94bdd2610804171527m6ec08d52g4e05517643541316@mail.gmail.com> <94bdd2610804201515j24a3f7e4uf7ab98cfa915fafb@mail.gmail.com> Message-ID: On Sun, 20 Apr 2008, Steven Bethard wrote: > On Sun, Apr 20, 2008 at 4:15 PM, Tarek Ziad? wrote: >> I have submitted a patch for review here: http://bugs.python.org/issue2663 >> >> glob-style patterns or a callable (for complex cases) can be provided >> to filter out files or directories. > > I'm not a big fan of the sequence-or-callable argument. Why not just > make it a callable argument, and supply a utility function so that you > can write something like:: > > exclude_func = shutil.excluding_patterns('*.tmp', 'test_dir2') > shutil.copytree(src_dir, dst_dir, exclude=exclude_func) Even if a glob pattern filter is considered useful enough to be worth special-casing, the glob capability should also be exposed via something like your excluding_patterns constructor and additionally as a function that can be called by another function intended for use as a callable argument. If it is not, then doing something like "files matching these glob patterns except for those matching this non-glob-expressible condition and also those files matching this second non-glob-expressible condition" becomes painful because the glob part essentially needs to be re-implemented. Isaac Morland CSCF Web Guru DC 2554C, x36650 WWW Software Specialist From guido at python.org Mon Apr 21 06:33:25 2008 From: guido at python.org (Guido van Rossum) Date: Sun, 20 Apr 2008 21:33:25 -0700 Subject: [Python-Dev] unscriptable? In-Reply-To: <480A9659.30704@gmail.com> References: <1afaf6160804181543g63e5b5acvc5c675b42a3111fc@mail.gmail.com> <1afaf6160804181603x1c5ecfcagcac4534321a37c6c@mail.gmail.com> <4809E1DF.9020306@gmail.com> <20080419230508.8468355a.steve@pearwood.info> <87zlrpcuru.fsf@uwakimon.sk.tsukuba.ac.jp> <480A9659.30704@gmail.com> Message-ID: On Sat, Apr 19, 2008 at 6:03 PM, Nick Coghlan wrote: > To suggest yet another colour for the bikeshed, maybe we should ditch > both unindexable and unsubscriptable and go with "'int' is not a > sequence or mapping". Any object which supports subscript notation will > be one or the other. I wouldn't bet my life on that. __getitem__ overloading for all sorts of nefarious purposes has quite a following. I'd prefer a message that doesn't get into what x "is" but sticks to the point at hand, which is that it doesn't support __getitem__. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ziade.tarek at gmail.com Mon Apr 21 09:48:47 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 21 Apr 2008 09:48:47 +0200 Subject: [Python-Dev] A smarter shutil.copytree ? In-Reply-To: References: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> <94bdd2610804171527m6ec08d52g4e05517643541316@mail.gmail.com> <94bdd2610804201515j24a3f7e4uf7ab98cfa915fafb@mail.gmail.com> Message-ID: <94bdd2610804210048x1afc7b27j5598d5cac0ae3336@mail.gmail.com> The pattern matching uses the src_dir to call glob.glob(), which returns the list of files to be excluded. That's why I added within the copytree() function. To make an excluding_patterns work, it could be coded like this:: def excluding_patterns(*patterns): def _excluding_patterns(filepath): exclude_files = [] dir_ = os.path.dirname(filepath) for pattern in patterns: pattern = os.path.join(dir_, pattern) exclude_files.extend(glob.glob(pattern)) return path in exclude_files return _excluding_patterns But I can see some performance issues, as the glob function will be called within the loop to test each file or folder:: def copytree(src, dst, exclude): ... for name in names: srcname = os.path.join(src, name) if exclude(srcname): continue ... ... Adding it at the beginning of the `copytree` function would then be better for performance, but means that the callable has to return a list of matching files instead of the match result itself:: def excluding_patterns(*patterns): def _excluding_patterns(path): exclude_files = [] for pattern in patterns: pattern = os.path.join(dir_, pattern) exclude_files.extend(glob.glob(pattern)) return exclude_files Then in copytree:: def copytree(src, dst, exclude): ... excluded = exclude(src) ... for name in names: srcname = os.path.join(src, name) if srcname in excluded: continue ... ... But this means that people that wants to implement their own callable will have to provide a function that returns a list of excluded files, therefore they won't be free to implement what they want. We could have two parameters, one for the glob-style sequence and one for the callable, to be able to use them at the appropriate places in the function, but I think this would make the function signature rather heavy:: def copytree(src, dst, exclude_patterns=None, exclude_function=None): ... That's why I would be in favor of sequence-or-callable argument even if I admit that it is not the pretiest way to present an argument. Regards Tarek On Mon, Apr 21, 2008 at 2:38 AM, Isaac Morland wrote: > On Sun, 20 Apr 2008, Steven Bethard wrote: > > > > On Sun, Apr 20, 2008 at 4:15 PM, Tarek Ziad? > wrote: > > > > > I have submitted a patch for review here: > http://bugs.python.org/issue2663 > > > > > > glob-style patterns or a callable (for complex cases) can be provided > > > to filter out files or directories. > > > > > > > I'm not a big fan of the sequence-or-callable argument. Why not just > > make it a callable argument, and supply a utility function so that you > > can write something like:: > > > > exclude_func = shutil.excluding_patterns('*.tmp', 'test_dir2') > > shutil.copytree(src_dir, dst_dir, exclude=exclude_func) > > > > Even if a glob pattern filter is considered useful enough to be worth > special-casing, the glob capability should also be exposed via something > like your excluding_patterns constructor and additionally as a function that > can be called by another function intended for use as a callable argument. > > If it is not, then doing something like "files matching these glob patterns > except for those matching this non-glob-expressible condition and also those > files matching this second non-glob-expressible condition" becomes painful > because the glob part essentially needs to be re-implemented. > > Isaac Morland CSCF Web Guru > DC 2554C, x36650 WWW Software Specialist -- Tarek Ziad? | Association AfPy | www.afpy.org Blog FR | http://programmation-python.org Blog EN | http://tarekziade.wordpress.com/ From ocean at m2.ccsnet.ne.jp Mon Apr 21 09:54:12 2008 From: ocean at m2.ccsnet.ne.jp (ocean) Date: Mon, 21 Apr 2008 16:54:12 +0900 Subject: [Python-Dev] Is Py_WIN_WIDE_FILENAMES still alive? Message-ID: <000901c8a384$e3965570$0300a8c0@whiterabc2znlh> Hello. I noticed when I removes following line in trunk/PC/pyconfig.h #define Py_WIN_WIDE_FILENAMES _fileio.c and posixmodule.c (and maybe more) cannot be compiled on Windows. When Py_WIN_WIDE_FILENAMES is not defined, how should python behave? - call posix functions like open(2) - call ANSI Win32 APIs like MoveFileA Or maybe this macro is not used anymore? Thank you. From jcea at argo.es Mon Apr 21 17:00:11 2008 From: jcea at argo.es (Jesus Cea) Date: Mon, 21 Apr 2008 17:00:11 +0200 Subject: [Python-Dev] BSDDB3 In-Reply-To: <47CF0B7C.109@v.loewis.de> References: <47C32D2D.1030206@free.fr> <47C4813F.2080403@v.loewis.de> <52dc1c820802262146n7b7c211dk973eb44390a4b8f7@mail.gmail.com> <20080227141322.GA5871@amk-desktop.matrixgroup.net> <47CDDF57.1020006@argo.es> <47CF0B7C.109@v.loewis.de> Message-ID: <480CABFB.2080302@argo.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Martin v. L?wis wrote: | I think it would be helpful if you could analyze the crashes that bsddb | caused on Windows. Just go back a few revisions in the subversion tree | to reproduce the crashes. I have no MS Windows machines in my environment :-( | This problem has now been worked-around, by reformulating the test cases | so that the situation doesn't occur anymore, but IMO, it should not be | possible for an extension module to cause an interpreter abort. Agreed. But I can't test Windows builds myself. Could anybody help me in this issue?. - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ ~ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSAyr9Zlgi5GaxT1NAQKzHgP/YKQ/9eLlwQfEKxePb5VtapkbfDri5T6C dulFrvuEEQqsefBopC1K70Tm4XGNmmLpf6U4Ew5ran0dQwjRPfQH+AWo8Cloh9IC ta8KjxHzIdl6myzitDwH1YKkDbrqdd1M5qs2QDKDjVx5c53ePHQNfLS9oOqtoWcc XNg6ro8K4os= =eQC0 -----END PGP SIGNATURE----- From wolever at cs.toronto.edu Mon Apr 21 17:56:54 2008 From: wolever at cs.toronto.edu (David Wolever) Date: Mon, 21 Apr 2008 11:56:54 -0400 Subject: [Python-Dev] Encoding detection in the standard library? Message-ID: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> Is there some sort of text encoding detection module is the standard library? And, if not, is there any reason not to add one? After some googling, I've come across this: http://mail.python.org/pipermail/python-3000/2006-September/003537.html But I can't find any changes that resulted from that thread. From alexander.belopolsky at gmail.com Mon Apr 21 18:37:57 2008 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 21 Apr 2008 16:37:57 +0000 (UTC) Subject: [Python-Dev] unscriptable? References: Message-ID: > ruby: undefined method `[]=' for 1:Fixnum (NoMethodError) I think it will be natural to unify [] error message with the other binary ops: Now: >>> 1+"" Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for +: 'int' and 'str' Proposal: >>> 1[2] Traceback (most recent call last): File "", line 1, in TypeError: unsupported operand type(s) for []: 'int' and 'int' From skip at pobox.com Mon Apr 21 18:44:09 2008 From: skip at pobox.com (skip at pobox.com) Date: Mon, 21 Apr 2008 11:44:09 -0500 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> Message-ID: <18444.50265.618909.751390@montanaro-dyndns-org.local> David> Is there some sort of text encoding detection module is the David> standard library? And, if not, is there any reason not to add David> one? No, there's not. I suspect the fact that you can't correctly determine the encoding of a chunk of text 100% of the time mitigates against it. Skip From fuzzyman at voidspace.org.uk Mon Apr 21 18:50:43 2008 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 21 Apr 2008 17:50:43 +0100 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <18444.50265.618909.751390@montanaro-dyndns-org.local> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> Message-ID: <480CC5E3.1050300@voidspace.org.uk> skip at pobox.com wrote: > David> Is there some sort of text encoding detection module is the > David> standard library? And, if not, is there any reason not to add > David> one? > > No, there's not. I suspect the fact that you can't correctly determine the > encoding of a chunk of text 100% of the time mitigates against it. > The only approach I know of is a heuristic based approach. e.g. http://www.voidspace.org.uk/python/articles/guessing_encoding.shtml (Which was 'borrowed' from docutils in the first place.) Michael Foord > Skip > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > From lists at cheimes.de Mon Apr 21 18:54:47 2008 From: lists at cheimes.de (Christian Heimes) Date: Mon, 21 Apr 2008 18:54:47 +0200 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> Message-ID: <480CC6D7.2020404@cheimes.de> David Wolever schrieb: > Is there some sort of text encoding detection module is the standard > library? > And, if not, is there any reason not to add one? You cannot detect the encoding unless it's explicitly defined through a header (e.g. the UTF BOM). It's technically impossible. The best you can do is an educated guess. Christian From exarkun at divmod.com Mon Apr 21 18:59:49 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Mon, 21 Apr 2008 12:59:49 -0400 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480CC5E3.1050300@voidspace.org.uk> Message-ID: <20080421165949.6859.924459832.divmod.quotient.33138@ohm> On Mon, 21 Apr 2008 17:50:43 +0100, Michael Foord wrote: >skip at pobox.com wrote: >> David> Is there some sort of text encoding detection module is the >> David> standard library? And, if not, is there any reason not to add >> David> one? >> >> No, there's not. I suspect the fact that you can't correctly determine the >> encoding of a chunk of text 100% of the time mitigates against it. >> > >The only approach I know of is a heuristic based approach. e.g. > >http://www.voidspace.org.uk/python/articles/guessing_encoding.shtml > >(Which was 'borrowed' from docutils in the first place.) This isn't the only approach, although you're right that in general you have to rely on heuristics. See the charset detection features of ICU: http://www.icu-project.org/userguide/charsetDetection.html I think OSAF's pyicu exposes these APIs: http://pyicu.osafoundation.org/ Jean-Paul From wolever at cs.toronto.edu Mon Apr 21 19:14:07 2008 From: wolever at cs.toronto.edu (David Wolever) Date: Mon, 21 Apr 2008 13:14:07 -0400 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <18444.50265.618909.751390@montanaro-dyndns-org.local> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> Message-ID: <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> On 21-Apr-08, at 12:44 PM, skip at pobox.com wrote: > > David> Is there some sort of text encoding detection module is the > David> standard library? And, if not, is there any reason not > to add > David> one? > No, there's not. I suspect the fact that you can't correctly > determine the > encoding of a chunk of text 100% of the time mitigates against it. Sorry, I wasn't very clear what I was asking. I was thinking about making an educated guess -- just like chardet (http://chardet.feedparser.org/). This is useful when you get a hunk of data which _should_ be some sort of intelligible text from the Big Scary Internet (say, a posted web form or email message), and you want to do something useful with it (say, search the content). From g.brandl at gmx.net Mon Apr 21 19:27:27 2008 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 21 Apr 2008 19:27:27 +0200 Subject: [Python-Dev] unscriptable? In-Reply-To: References: Message-ID: Alexander Belopolsky schrieb: >> ruby: undefined method `[]=' for 1:Fixnum (NoMethodError) > > I think it will be natural to unify [] error message with > the other binary ops: > > Now: >>>> 1+"" > Traceback (most recent call last): > File "", line 1, in > TypeError: unsupported operand type(s) for +: 'int' and 'str' > > Proposal: >>>> 1[2] > Traceback (most recent call last): > File "", line 1, in > TypeError: unsupported operand type(s) for []: 'int' and 'int' This is misleading, since it suggests that you can remedy this by using another type in the subscript, which isn't possible. Georg From guido at python.org Mon Apr 21 19:29:38 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 21 Apr 2008 10:29:38 -0700 Subject: [Python-Dev] A smarter shutil.copytree ? In-Reply-To: References: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> <94bdd2610804171527m6ec08d52g4e05517643541316@mail.gmail.com> <94bdd2610804201515j24a3f7e4uf7ab98cfa915fafb@mail.gmail.com> Message-ID: On Sun, Apr 20, 2008 at 5:25 PM, Steven Bethard wrote: > > On Sun, Apr 20, 2008 at 4:15 PM, Tarek Ziad? wrote: > > I have submitted a patch for review here: http://bugs.python.org/issue2663 > > > > glob-style patterns or a callable (for complex cases) can be provided > > to filter out files or directories. > > I'm not a big fan of the sequence-or-callable argument. Why not just > make it a callable argument, and supply a utility function so that you > can write something like:: > > exclude_func = shutil.excluding_patterns('*.tmp', 'test_dir2') > shutil.copytree(src_dir, dst_dir, exclude=exclude_func) > > ? Agreed. Type testing is fraught with problems. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From g.brandl at gmx.net Mon Apr 21 19:28:48 2008 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 21 Apr 2008 19:28:48 +0200 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480CC6D7.2020404@cheimes.de> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <480CC6D7.2020404@cheimes.de> Message-ID: Christian Heimes schrieb: > David Wolever schrieb: >> Is there some sort of text encoding detection module is the standard >> library? >> And, if not, is there any reason not to add one? > > You cannot detect the encoding unless it's explicitly defined through a > header (e.g. the UTF BOM). It's technically impossible. The best you can > do is an educated guess. Exactly, and in light of that, I'm -1 for such a standard module. We've enough issues with modules implementing (apparently) fully specified standards. :) Georg From guido at python.org Mon Apr 21 19:38:03 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 21 Apr 2008 10:38:03 -0700 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <480CC6D7.2020404@cheimes.de> Message-ID: To the contrary, an encoding-guessing module is often needed, and guessing can be done with a pretty high success rate. Other Unicode libraries (e.g. ICU) contain guessing modules. I suppose the API could return two values: the guessed encoding and a confidence indicator. Note that the locale settings might figure in the guess. On Mon, Apr 21, 2008 at 10:28 AM, Georg Brandl wrote: > Christian Heimes schrieb: > > > David Wolever schrieb: > >> Is there some sort of text encoding detection module is the standard > >> library? > >> And, if not, is there any reason not to add one? > > > > You cannot detect the encoding unless it's explicitly defined through a > > header (e.g. the UTF BOM). It's technically impossible. The best you can > > do is an educated guess. > > Exactly, and in light of that, I'm -1 for such a standard module. > We've enough issues with modules implementing (apparently) fully > specified standards. :) > > Georg > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Mon Apr 21 19:00:54 2008 From: skip at pobox.com (skip at pobox.com) Date: Mon, 21 Apr 2008 12:00:54 -0500 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480CC5E3.1050300@voidspace.org.uk> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <480CC5E3.1050300@voidspace.org.uk> Message-ID: <18444.51270.116350.591529@montanaro-dyndns-org.local> Michael> The only approach I know of is a heuristic based approach. e.g. Michael> http://www.voidspace.org.uk/python/articles/guessing_encoding.shtml Michael> (Which was 'borrowed' from docutils in the first place.) Yes, I implemented a heuristic approach for the Musi-Cal web server. I was able to rely on domain knowledge to guess correctly almost all the time. The heuristic was that almost all form submissions came from the US and the rest which didn't came from Western Europe. Python could never embed such a narrow-focused heuristic into its core distribution. Skip From skip at pobox.com Mon Apr 21 19:56:27 2008 From: skip at pobox.com (skip at pobox.com) Date: Mon, 21 Apr 2008 12:56:27 -0500 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <480CC6D7.2020404@cheimes.de> Message-ID: <18444.54603.483042.133611@montanaro-dyndns-org.local> Guido> Note that the locale settings might figure in the guess. Alas, locale settings in a web server have little or nothing to do with the locale settings in the client submitting the form. Skip From greg at krypto.org Mon Apr 21 20:10:24 2008 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 21 Apr 2008 11:10:24 -0700 Subject: [Python-Dev] very bad network performance In-Reply-To: References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> <932f8baf0804141429v666672a0j8833d8bcd3cd44d5@mail.gmail.com> <932f8baf0804141557u5d1a06e6q7112dd4d606e079c@mail.gmail.com> Message-ID: <52dc1c820804211110g5827ba59t60fcde7f0c7b571c@mail.gmail.com> On Mon, Apr 14, 2008 at 4:41 PM, Curt Hagenlocher wrote: > On Mon, Apr 14, 2008 at 4:19 PM, Guido van Rossum > wrote: > > > > But why was imaplib apparently specifying 10MB? Did it know there was > > that much data? Or did it just not want to bother looping over all the > > data in smaller buffer increments (e.g. 64K, which is probably the max > > of what most TCP stacks will give you)? > > I'm going to guess that the code in question is > > size = int(self.mo.group('size')) > if __debug__: > if self.debug >= 4: > self._mesg('read literal size %s' % size) > data = self.read(size) > > It's reading however many bytes are reported by the server as the size. > > > If I'm right with my hunch that the TCP stack will probably clamp at > > 64K, perhaps we should use min(system limit, max(requested size, > > buffer size))? > > I have indeed missed the point of the read buffer size. This would work. > The 64K hunch is wrong. The system limit can be found using getsockopt(...SO_RCVBUF...). It can easily be (and often is) set to many megabytes either at a system default level or on a per socket level by the user using setsockopt. When the system default is that large, limiting by the system limit would not help the 10mb read case. Even smaller allocations like 64K cause problems as mentioned in issue 1092502 linking to this twisted http://twistedmatrix.com/trac/ticket/1079bug. twisted's solution was to make the string object returned by a recv as short lived as possible by copying it into a StringIO. We could do the same in _fileobject.read() and readline(). I have attached a patch to issue 2632 that changes socket to use StringIO for its read buffer and keeps the lifetime of strings returned by recv() as short as possible when appropriate. It also refuses to call recv() with a size smaller than default_bufsize within read() [the source of the performance problem]. That changes internal recv() call behavior over the existing code after the issue 1092502 "fix" was applied to use min() rather than max(), but it is -not- a significant change over the pre-1092502 "fix" behavior that exists in all released versions of python (it already chose the larger of two values for recv sizes). The main difference behind the scenes? StringIO is using realloc only to increase its size while recv() was using realloc to shrink the allocation size and many of these recv()ed shrunken strings were being held onto in a list before the final value was constructed. I suggest continuing the discussion within issue 2632 to keep better track of it. My socket-strio patch in 2632 needs more testing (it passed socket, http* and url* tests) and verification that both issue's problems are indeed gone but they should be. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20080421/6f8c4938/attachment.htm From tonynelson at georgeanelson.com Mon Apr 21 20:34:50 2008 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Mon, 21 Apr 2008 14:34:50 -0400 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> Message-ID: At 1:14 PM -0400 4/21/08, David Wolever wrote: >On 21-Apr-08, at 12:44 PM, skip at pobox.com wrote: >> >> David> Is there some sort of text encoding detection module is the >> David> standard library? And, if not, is there any reason not >> to add >> David> one? >> No, there's not. I suspect the fact that you can't correctly >> determine the >> encoding of a chunk of text 100% of the time mitigates against it. >Sorry, I wasn't very clear what I was asking. > >I was thinking about making an educated guess -- just like chardet >(http://chardet.feedparser.org/). > >This is useful when you get a hunk of data which _should_ be some >sort of intelligible text from the Big Scary Internet (say, a posted >web form or email message), and you want to do something useful with >it (say, search the content). Feedparser.org's chardet can't guess 'latin1', so it should be used as a last resort, just as the docs say. -- ____________________________________________________________________ TonyN.:' ' From martin at v.loewis.de Mon Apr 21 23:29:18 2008 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 21 Apr 2008 23:29:18 +0200 Subject: [Python-Dev] BSDDB3 In-Reply-To: <480CABFB.2080302@argo.es> References: <47C32D2D.1030206@free.fr> <47C4813F.2080403@v.loewis.de> <52dc1c820802262146n7b7c211dk973eb44390a4b8f7@mail.gmail.com> <20080227141322.GA5871@amk-desktop.matrixgroup.net> <47CDDF57.1020006@argo.es> <47CF0B7C.109@v.loewis.de> <480CABFB.2080302@argo.es> Message-ID: <480D072E.8080100@v.loewis.de> > | This problem has now been worked-around, by reformulating the test cases > | so that the situation doesn't occur anymore, but IMO, it should not be > | possible for an extension module to cause an interpreter abort. > > Agreed. But I can't test Windows builds myself. Could anybody help me in > this issue?. Sure. If you post a patch, I'm sure someone will be able to test it on Windows and report results; when it's committed, also the buildbots will test it. Regards, Martin From martin at v.loewis.de Mon Apr 21 23:31:06 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 21 Apr 2008 23:31:06 +0200 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> Message-ID: <480D079A.2020904@v.loewis.de> > This is useful when you get a hunk of data which _should_ be some > sort of intelligible text from the Big Scary Internet (say, a posted > web form or email message), and you want to do something useful with > it (say, search the content). I don't think that should be part of the standard library. People will mistake what it tells them for certain. Regards, Martin From rbp at isnomore.net Mon Apr 21 23:37:20 2008 From: rbp at isnomore.net (Rodrigo Bernardo Pimentel) Date: Mon, 21 Apr 2008 18:37:20 -0300 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480D079A.2020904@v.loewis.de> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> Message-ID: <20080421213720.GB16282@isnomore.net> On Mon, Apr 21 2008 at 06:31:06PM BRT, "\"Martin v. L?wis\"" wrote: > > This is useful when you get a hunk of data which _should_ be some > > sort of intelligible text from the Big Scary Internet (say, a posted > > web form or email message), and you want to do something useful with > > it (say, search the content). > > I don't think that should be part of the standard library. People > will mistake what it tells them for certain. Maybe call it "charguess", then? rbp -- Rodrigo Bernardo Pimentel | GPG: <0x0DB14978> From phd at phd.pp.ru Tue Apr 22 00:17:01 2008 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 22 Apr 2008 02:17:01 +0400 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <20080421213720.GB16282@isnomore.net> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <20080421213720.GB16282@isnomore.net> Message-ID: <20080421221701.GA13372@phd.pp.ru> On Mon, Apr 21, 2008 at 06:37:20PM -0300, Rodrigo Bernardo Pimentel wrote: > On Mon, Apr 21 2008 at 06:31:06PM BRT, "\"Martin v. L??wis\"" wrote: > > > This is useful when you get a hunk of data which _should_ be some > > > sort of intelligible text from the Big Scary Internet (say, a posted > > > web form or email message), and you want to do something useful with > > > it (say, search the content). > > > > I don't think that should be part of the standard library. People > > will mistake what it tells them for certain. > > Maybe call it "charguess", then? The famous chardet returns probablity of its guessing: >>> import chardet >>> chardet.detect("dabc") {'confidence': 1.0, 'encoding': 'ascii'} >>> chardet.detect("????") {'confidence': 0.98999999999999999, 'encoding': 'KOI8-R'} Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From greg.ewing at canterbury.ac.nz Tue Apr 22 02:26:49 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 22 Apr 2008 12:26:49 +1200 Subject: [Python-Dev] thoughts on having EOFError inherit from EnvironmentError? In-Reply-To: <20080419121255.150520e9.steve@pearwood.info> References: <52dc1c820804121501k5c3d0190k9e62cb4c47878d58@mail.gmail.com> <48040C08.504@canterbury.ac.nz> <48052BC1.40202@canterbury.ac.nz> <48057537.3060003@canterbury.ac.nz> <4806997F.50000@canterbury.ac.nz> <48093A67.5070702@canterbury.ac.nz> <20080419121255.150520e9.steve@pearwood.info> Message-ID: <480D30C9.2050805@canterbury.ac.nz> Steven wrote: > It might help if you explain what sort of actual things that the user does > wrong that you are talking about. Usually it's some kind of parsing error. Or it might be a network connection getting closed unexpectedly. Essentially it's anything due to factors outside the program's control, but which aren't detected and reported by any of the built-in IOError or OSError exceptions. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 22 02:50:44 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 22 Apr 2008 12:50:44 +1200 Subject: [Python-Dev] A smarter shutil.copytree ? In-Reply-To: References: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> <94bdd2610804171527m6ec08d52g4e05517643541316@mail.gmail.com> <94bdd2610804201515j24a3f7e4uf7ab98cfa915fafb@mail.gmail.com> Message-ID: <480D3664.5080001@canterbury.ac.nz> Steven Bethard wrote: > I'm not a big fan of the sequence-or-callable argument. Why not just > make it a callable argument, and supply a utility function Or have two different keyword arguments, one for a sequence and one for a callable. -- Greg From wolever at cs.toronto.edu Tue Apr 22 03:41:51 2008 From: wolever at cs.toronto.edu (David Wolever) Date: Mon, 21 Apr 2008 21:41:51 -0400 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480D079A.2020904@v.loewis.de> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> Message-ID: <91478133-664C-4F29-90CB-D0985BE01F24@cs.toronto.edu> On 21-Apr-08, at 5:31 PM, Martin v. L?wis wrote: >> This is useful when you get a hunk of data which _should_ be some >> sort of intelligible text from the Big Scary Internet (say, a posted >> web form or email message), and you want to do something useful with >> it (say, search the content). > I don't think that should be part of the standard library. People > will mistake what it tells them for certain. As Oleg mentioned, if the method is called something like 'guess_encoding', I think we could live with clear consciences. IMO, encoding estimation is something that many web programs will have to deal with, so it might as well be built in; I would prefer the option to run `text=input.encode('guess')` (or something similar) than relying on an external dependency or worse yet using a hand- rolled algorithm. From jimjjewett at gmail.com Tue Apr 22 05:30:18 2008 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 21 Apr 2008 23:30:18 -0400 Subject: [Python-Dev] Encoding detection in the standard library? Message-ID: David Wolever wrote: > IMO, encoding estimation is something that > many web programs will have to deal with, > so it might as well be built in; I would prefer > the option to run `text=input.encode('guess')` > (or something similar) than relying on an external > dependency or worse yet using a hand-rolled > algorithm The (still draft) html5 spec is trying to get error-correction standardized, so it includes all sort of "if this fails, do X". Encoding detection will be standardized, so there will be an external standard that we can reference. http://dev.w3.org/html5/spec/Overview.html#determining Note that this portion of the spec is probably not stable yet, as there was some new analysis on which "wrong" answers provided better results on real world web pages. e.g., http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2008-March/014127.html http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2008-March/014190.html There was also a recent analysis of how many characters it takes to sniff successfully X% of the time on today's web, though I'm not finding it at the moment. -jJ From martin at v.loewis.de Tue Apr 22 06:30:33 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 22 Apr 2008 06:30:33 +0200 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <91478133-664C-4F29-90CB-D0985BE01F24@cs.toronto.edu> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <91478133-664C-4F29-90CB-D0985BE01F24@cs.toronto.edu> Message-ID: <480D69E9.7020004@v.loewis.de> > IMO, encoding estimation is something that many web programs will have > to deal with Can you please explain why that is? Web programs should not normally have the need to detect the encoding; instead, it should be specified always - unless you are talking about browsers specifically, which need to support web pages that specify the encoding incorrectly. > so it might as well be built in; I would prefer the option > to run `text=input.encode('guess')` (or something similar) than relying > on an external dependency or worse yet using a hand-rolled algorithm. Ok, let me try differently then. Please feel free to post a patch to bugs.python.org, and let other people rip it apart. For example, I don't think it should be a codec, as I can't imagine it working on streams. Regards, Martin From nnorwitz at gmail.com Tue Apr 22 06:46:10 2008 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 21 Apr 2008 21:46:10 -0700 Subject: [Python-Dev] r62342 - python/branches/py3k/Objects/bytesobject.c In-Reply-To: References: <20080414205105.B45C11E4005@bag.python.org> <4803C735.10100@cheimes.de> <43aa6ff70804141418w414cbe0bl8eb82ea77e8559a1@mail.gmail.com> <4803D319.5010000@cheimes.de> Message-ID: On Tue, Apr 15, 2008 at 2:21 AM, Stefan Behnel wrote: > Neal Norwitz wrote: > > Iteration with the dict methods (e.g., keys -> iterkeys()), > > map/zip/filter returning iterator rather than list. > > That's only an optimisation, it's not functionally required. A list behaves > like an iterator in most use cases, so it's rather unlikely that Py3 code will > break in the backport because of this (and it's very unlikely that static > analysis can catch the cases where it breaks). There should be a rule to > optimise "list(map(...))" into "map(...)" and "list(x.keys())" into plain > "x.keys()" etc., but I don't think there is much more to do here. It's not just an optimization if a copy won't fit in memory. I'd like the solution to be closer to 100% than 95%. > > int -> (int, long) > > Is there any case where this might be required? I don't see any reason why > back-converted Py3 code should break here. What would "long()" be needed for > in working Py3 code that "int()" doesn't provide in Py2? > > Although you might have been referring to "isinstance(x, int)" in Py3? Yes, sorry, I wasn't explicit. isinstance is specifically what I was referring to (or other type checks such as type(x) in (int, long)). > > str -> basestring or (str, unicode) > > This is an issue, although I think that Py3 is explicit enough here to make > this easy to handle by static renaming (and maybe optimising "isinstance(s, > (str, bytes))" into "..., basestring))"). > > > > > __bool__ -> __nonzero__ > > exec/execfile > > input -> rawinput > > Also valid issues that can be handled through renaming and static syntactic > adjustments. > > > > > Most things that have a fixer in 2to3 would also require one in 3to2. > > I think the more explicit syntax in Py3 will actually make it easier to > back-convert the code statically from 3to2 than to migrate it from 2to3. Sure, that's the idea. I haven't seen any action on 3to2 (although I'm very behind on email). Stefan, could you try to implement some of these and report back how it works? n From schmir at gmail.com Tue Apr 22 08:44:48 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Tue, 22 Apr 2008 08:44:48 +0200 Subject: [Python-Dev] very bad network performance In-Reply-To: <52dc1c820804211110g5827ba59t60fcde7f0c7b571c@mail.gmail.com> References: <932f8baf0804140912u54adc7d5md7261541857f21bd@mail.gmail.com> <932f8baf0804141429v666672a0j8833d8bcd3cd44d5@mail.gmail.com> <932f8baf0804141557u5d1a06e6q7112dd4d606e079c@mail.gmail.com> <52dc1c820804211110g5827ba59t60fcde7f0c7b571c@mail.gmail.com> Message-ID: <932f8baf0804212344w3722c1eewe954b59498e708c9@mail.gmail.com> On Mon, Apr 21, 2008 at 8:10 PM, Gregory P. Smith wrote: > > > > The 64K hunch is wrong. The system limit can be found using > getsockopt(...SO_RCVBUF...). It can easily be (and often is) set to many > megabytes either at a system default level or on a per socket level by the > user using setsockopt. When the system default is that large, limiting by > the system limit would not help the 10mb read case. > but it would help in the 100mb read case. > > Even smaller allocations like 64K cause problems as mentioned in issue > 1092502 linking to this twisted http://twistedmatrix.com/trac/ticket/1079bug. twisted's solution was to make the string object returned by a recv as > short lived as possible by copying it into a StringIO. We could do the same > in _fileobject.read() and readline(). > this approach look reasonable to me. - Ralf -------------- next part -------------- An HTML attachment was scrubbed... URL: From ziade.tarek at gmail.com Tue Apr 22 09:56:47 2008 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Tue, 22 Apr 2008 09:56:47 +0200 Subject: [Python-Dev] A smarter shutil.copytree ? In-Reply-To: References: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> <94bdd2610804171527m6ec08d52g4e05517643541316@mail.gmail.com> <94bdd2610804201515j24a3f7e4uf7ab98cfa915fafb@mail.gmail.com> Message-ID: <94bdd2610804220056w24a34d07q16b1151796f1b9d7@mail.gmail.com> On Mon, Apr 21, 2008 at 2:25 AM, Steven Bethard wrote: > > On Sun, Apr 20, 2008 at 4:15 PM, Tarek Ziad? wrote: > > I have submitted a patch for review here: http://bugs.python.org/issue2663 > > > > glob-style patterns or a callable (for complex cases) can be provided > > to filter out files or directories. > > I'm not a big fan of the sequence-or-callable argument. Why not just > make it a callable argument, and supply a utility function so that you > can write something like:: > > exclude_func = shutil.excluding_patterns('*.tmp', 'test_dir2') > shutil.copytree(src_dir, dst_dir, exclude=exclude_func) > > ? I made another draft based on a single callable argument to try out: http://bugs.python.org/file10073/shutil.copytree.filtering.patch The callable takes the src directory + its content as a list, and returns filter eligible for exclusion That makes me wonder, like Alexander said on the bug tracker: In the glob-style patterns callable, do we want to deal with absolute paths ? Tarek From stefan_ml at behnel.de Tue Apr 22 10:45:49 2008 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 22 Apr 2008 10:45:49 +0200 Subject: [Python-Dev] r62342 - python/branches/py3k/Objects/bytesobject.c In-Reply-To: References: <20080414205105.B45C11E4005@bag.python.org> <4803C735.10100@cheimes.de> <43aa6ff70804141418w414cbe0bl8eb82ea77e8559a1@mail.gmail.com> <4803D319.5010000@cheimes.de> Message-ID: <480DA5BD.9080300@behnel.de> Neal Norwitz wrote: > I haven't seen any action on 3to2 (although I'm very behind on email). > Stefan, could you try to implement some of these and report back how > it works? No, sorry, that's too low a priority for me currently. Stefan From mal at egenix.com Tue Apr 22 12:31:34 2008 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 22 Apr 2008 12:31:34 +0200 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480D079A.2020904@v.loewis.de> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> Message-ID: <480DBE86.2000507@egenix.com> On 2008-04-21 23:31, Martin v. L?wis wrote: >> This is useful when you get a hunk of data which _should_ be some >> sort of intelligible text from the Big Scary Internet (say, a posted >> web form or email message), and you want to do something useful with >> it (say, search the content). > > I don't think that should be part of the standard library. People > will mistake what it tells them for certain. +1 I also think that it's better to educate people to add (correct) encoding information to their text data, rather than give them a guess mechanism... http://chardet.feedparser.org/docs/faq.html#faq.yippie chardet is based on the Mozilla algorithm and at least in my experience that algorithm doesn't work too well. The Mozilla algorithm may work for Asian encodings due to the fact that those encodings are usually also bound to a specific language (and you can then use character and word frequency analysis), but for encodings which can encode far more than just a single language (e.g. UTF-8 or Latin-1), the correct detection rate is rather low. The problem becomes completely even more difficult when leaving the normal text domain or when mixing languages in the same text, e.g. when trying to detect source code with comments using a non-ASCII encoding. The "trick" to just pass the text through a codec and see whether it roundtrips also doesn't necessarily help: Latin-1, for example, will always round-trip, since Latin-1 is a subset of Unicode. IMHO, more research has to be done into this area before a "standard" module can be added to the Python's stdlib... and who knows, perhaps we're lucky and by the time everyone is using UTF-8 anyway :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 22 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 From ndbecker2 at gmail.com Tue Apr 22 13:17:24 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Tue, 22 Apr 2008 07:17:24 -0400 Subject: [Python-Dev] pydoc works with eggs? (python-2.5.1) Message-ID: pydoc blew up when I tried to view doc for pytools module, which is an egg: pydoc -p 8082 pydoc server ready at http://localhost:8082/ ---------------------------------------- Exception happened during processing of request from ('127.0.0.1', 52915) Traceback (most recent call last): File "/usr/lib64/python2.5/SocketServer.py", line 222, in handle_request self.process_request(request, client_address) File "/usr/lib64/python2.5/SocketServer.py", line 241, in process_request self.finish_request(request, client_address) File "/usr/lib64/python2.5/SocketServer.py", line 254, in finish_request self.RequestHandlerClass(request, client_address, self) File "/usr/lib64/python2.5/SocketServer.py", line 522, in __init__ self.handle() File "/usr/lib64/python2.5/BaseHTTPServer.py", line 316, in handle self.handle_one_request() File "/usr/lib64/python2.5/BaseHTTPServer.py", line 310, in handle_one_request method() File "/usr/lib64/python2.5/pydoc.py", line 1924, in do_GET self.send_document(describe(obj), html.document(obj, path)) File "/usr/lib64/python2.5/pydoc.py", line 321, in document if inspect.ismodule(object): return self.docmodule(*args) File "/usr/lib64/python2.5/pydoc.py", line 672, in docmodule contents.append(self.document(value, key, name, fdict, cdict)) File "/usr/lib64/python2.5/pydoc.py", line 322, in document if inspect.isclass(object): return self.docclass(*args) File "/usr/lib64/python2.5/pydoc.py", line 807, in docclass lambda t: t[1] == 'method') File "/usr/lib64/python2.5/pydoc.py", line 735, in spill funcs, classes, mdict, object)) File "/usr/lib64/python2.5/pydoc.py", line 323, in document if inspect.isroutine(object): return self.docroutine(*args) File "/usr/lib64/python2.5/pydoc.py", line 891, in docroutine getdoc(object), self.preformat, funcs, classes, methods) File "/usr/lib64/python2.5/pydoc.py", line 79, in getdoc result = inspect.getdoc(object) or inspect.getcomments(object) File "/usr/lib64/python2.5/inspect.py", line 521, in getcomments lines, lnum = findsource(object) File "/usr/lib64/python2.5/inspect.py", line 510, in findsource if pat.match(lines[lnum]): break IndexError: list index out of range ---------------------------------------- From janssen at parc.com Tue Apr 22 17:14:43 2008 From: janssen at parc.com (Bill Janssen) Date: Tue, 22 Apr 2008 08:14:43 PDT Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480DBE86.2000507@egenix.com> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <480DBE86.2000507@egenix.com> Message-ID: <08Apr22.081453pdt."58696"@synergy1.parc.xerox.com> > IMHO, more research has to be done into this area before a > "standard" module can be added to the Python's stdlib... and > who knows, perhaps we're lucky and by the time everyone is > using UTF-8 anyway :-) I walked over to our computational linguistics group and asked. This is often combined with language guessing (which uses a similar approach, but using characters instead of bytes), and apparently can usually be done with high confidence. Of course, they're usually looking at clean texts, not random "stuff". I'll see if I can get some references and report back -- most of the research on this was done in the 90's. Bill From ronaldoussoren at mac.com Tue Apr 22 17:47:53 2008 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 22 Apr 2008 17:47:53 +0200 Subject: [Python-Dev] configure error: "rm: conftest.dSYM: is a directory" In-Reply-To: <18423.53311.150471.667891@montanaro-dyndns-org.local> References: <18423.53311.150471.667891@montanaro-dyndns-org.local> Message-ID: <403F765C-2EFF-4B9B-9780-835EDDC54F43@mac.com> On 5 Apr, 2008, at 21:17, skip at pobox.com wrote: > I just noticed this error message during configure: > > checking whether gcc accepts -Olimit 1500... no > checking whether gcc supports ParseTuple __format__... no > checking whether pthreads are available without options... yes > checking whether g++ also accepts flags for thread support... no > checking for ANSI C header files... rm: conftest.dSYM: is a > directory > rm: conftest.dSYM: is a directory > yes > checking for sys/types.h... yes > checking for sys/stat.h... yes > checking for stdlib.h... yes > checking for string.h... yes > > Note the "rm: conftest.dSYM: is a directory". This occurred a few > times > during the configure process. Didn't cause it to conk out, but is > annoying. I've looked into this issue. It is harmless and caused by an interaction between AC_TRY_RUN and gcc on leopard. Gcc generates '.dSYM' directories when linking with debugging enabled. These directories contain detached debugging information (see man dsymutil). AC_TRY_RUN tries to remove 'conftest.*' using rm, without the -r flag. The end result is an error message during configure and a 'config.dSYM' turd. AFAIK this not easily fixed without changing the definition of AC_TRY_RUN, at least not without crude hacks. Ronald > > > Skip > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2224 bytes Desc: not available URL: From wolever at cs.toronto.edu Tue Apr 22 17:48:07 2008 From: wolever at cs.toronto.edu (David Wolever) Date: Tue, 22 Apr 2008 11:48:07 -0400 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480D69E9.7020004@v.loewis.de> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <91478133-664C-4F29-90CB-D0985BE01F24@cs.toronto.edu> <480D69E9.7020004@v.loewis.de> Message-ID: <091D7293-363C-4766-8931-CFE6A9E72331@cs.toronto.edu> On 22-Apr-08, at 12:30 AM, Martin v. L?wis wrote: >> IMO, encoding estimation is something that many web programs will >> have >> to deal with > Can you please explain why that is? Web programs should not normally > have the need to detect the encoding; instead, it should be specified > always - unless you are talking about browsers specifically, which > need to support web pages that specify the encoding incorrectly. Two cases come immediately to mind: email and web forms. When a web browser POSTs data, there is no standard way of communicating which encoding it's using. There are some hints which make it easier (accept-charset attributes, the encoding used to send the page to the browser), but no guarantees. Email is a smaller problem, because it usually has a helpful content- type header, but that's no guarantee. Now, at the moment, the only data I have to support this claim is my experience with DrProject in non-English locations. If I'm the only one who has had these sorts of problems, I'll go back to "Unicode for Dummies". >> so it might as well be built in; I would prefer the option >> to run `text=input.encode('guess')` (or something similar) than >> relying >> on an external dependency or worse yet using a hand-rolled algorithm. > Ok, let me try differently then. Please feel free to post a patch to > bugs.python.org, and let other people rip it apart. > For example, I don't think it should be a codec, as I can't imagine it > working on streams. As things frequently are, it seems like this is a much larger problem that I originally believed. I'll go back and take another look at the problem, then come back if new revelations appear. From janssen at parc.com Tue Apr 22 18:33:18 2008 From: janssen at parc.com (Bill Janssen) Date: Tue, 22 Apr 2008 09:33:18 PDT Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <08Apr22.081453pdt."58696"@synergy1.parc.xerox.com> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <480DBE86.2000507@egenix.com> <08Apr22.081453pdt."58696"@synergy1.parc.xerox.com> Message-ID: <08Apr22.093328pdt."58696"@synergy1.parc.xerox.com> The 2002 paper "A language and character set determination method based on N-gram statistics" by Izumi Suzuki and Yoshiki Mikami and Ario Ohsato and Yoshihide Chubachi seems to me a pretty good way to go about this. They're looking at "LSE"s, language-script-encoding triples; a "script" is a way of using a particular character set to write in a particular language. Their system has these requirements: R1. the response must be either "correct answer" or "unable to detect" where "unable to detect" includes "other than registered" [the registered set of LSEs]; R2. Applicable to multi-LSE texts; R3. never accept a wrong answer, even when the program does not have enough data on an LSE; and R4. applicable to any LSE text. So, no wrong answers. The biggest disadvantage would seem to be that the registration data for a particular LSE is kind of bulky; on the order of 10,000 shift-codons, each of three bytes, about 30K uncompressed. http://portal.acm.org/ft_gateway.cfm?id=772759&type=pdf Bill > > IMHO, more research has to be done into this area before a > > "standard" module can be added to the Python's stdlib... and > > who knows, perhaps we're lucky and by the time everyone is > > using UTF-8 anyway :-) > > I walked over to our computational linguistics group and asked. This > is often combined with language guessing (which uses a similar > approach, but using characters instead of bytes), and apparently can > usually be done with high confidence. Of course, they're usually > looking at clean texts, not random "stuff". I'll see if I can get > some references and report back -- most of the research on this was > done in the 90's. > > Bill From steven.bethard at gmail.com Tue Apr 22 19:04:45 2008 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 22 Apr 2008 11:04:45 -0600 Subject: [Python-Dev] A smarter shutil.copytree ? In-Reply-To: <94bdd2610804220056w24a34d07q16b1151796f1b9d7@mail.gmail.com> References: <94bdd2610804170952p12f3bffan114d937178ebf046@mail.gmail.com> <94bdd2610804171527m6ec08d52g4e05517643541316@mail.gmail.com> <94bdd2610804201515j24a3f7e4uf7ab98cfa915fafb@mail.gmail.com> <94bdd2610804220056w24a34d07q16b1151796f1b9d7@mail.gmail.com> Message-ID: On Tue, Apr 22, 2008 at 1:56 AM, Tarek Ziad? wrote: > On Mon, Apr 21, 2008 at 2:25 AM, Steven Bethard wrote: > > On Sun, Apr 20, 2008 at 4:15 PM, Tarek Ziad? wrote: > > > I have submitted a patch for review here: http://bugs.python.org/issue2663 > > > > > > glob-style patterns or a callable (for complex cases) can be provided > > > to filter out files or directories. > > > > I'm not a big fan of the sequence-or-callable argument. Why not just > > make it a callable argument, and supply a utility function so that you > > can write something like:: > > > > exclude_func = shutil.excluding_patterns('*.tmp', 'test_dir2') > > shutil.copytree(src_dir, dst_dir, exclude=exclude_func) > > I made another draft based on a single callable argument to try out: > http://bugs.python.org/file10073/shutil.copytree.filtering.patch > > The callable takes the src directory + its content as a list, and > returns filter eligible for exclusion FWIW, that looks better to me. > That makes me wonder, like Alexander said on the bug tracker: > In the glob-style patterns callable, do we want to deal with absolute paths ? I think that it would be okay to document that shutil.ignore_patterns() only accepts patterns matching individual filenames (not complex paths). If someone needs to do something with absolute paths, then they can write their own 'ignore' function, right? Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From martin at v.loewis.de Tue Apr 22 20:06:16 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 22 Apr 2008 20:06:16 +0200 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <091D7293-363C-4766-8931-CFE6A9E72331@cs.toronto.edu> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <91478133-664C-4F29-90CB-D0985BE01F24@cs.toronto.edu> <480D69E9.7020004@v.loewis.de> <091D7293-363C-4766-8931-CFE6A9E72331@cs.toronto.edu> Message-ID: <480E2918.7090006@v.loewis.de> > When a web browser POSTs data, there is no standard way of communicating > which encoding it's using. That's just not true. Web browser should and do use the encoding of the web page that originally contained the form. > There are some hints which make it easier > (accept-charset attributes, the encoding used to send the page to the > browser), but no guarantees. Not true. The latter is guaranteed (unless you assume bugs - but if you do, can you present a specific browser that has that bug?) > Email is a smaller problem, because it usually has a helpful > content-type header, but that's no guarantee. Then assume windows-1252. Mailers who don't use MIME for non-ASCII characters mostly died 10 years ago; those people who continue to use them likely can accept occasional moji-bake (or else they would have switched long ago). > Now, at the moment, the only data I have to support this claim is my > experience with DrProject in non-English locations. > If I'm the only one who has had these sorts of problems, I'll go back to > "Unicode for Dummies". For web forms, I always encode the pages in UTF-8, and that always works. For email, I once added encoding processing to the pipermail (the mailman archiver), and that also always works. > I'll go back and take another look at the problem, then come back if new > revelations appear. Good luck! Martin From tnelson at onresolve.com Tue Apr 22 20:42:10 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Tue, 22 Apr 2008 11:42:10 -0700 Subject: [Python-Dev] BSDDB3 In-Reply-To: <480CABFB.2080302@argo.es> References: <47C32D2D.1030206@free.fr> <47C4813F.2080403@v.loewis.de> <52dc1c820802262146n7b7c211dk973eb44390a4b8f7@mail.gmail.com> <20080227141322.GA5871@amk-desktop.matrixgroup.net> <47CDDF57.1020006@argo.es> <47CF0B7C.109@v.loewis.de> <480CABFB.2080302@argo.es> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E243C7CA1@EXMBX04.exchhosting.com> Hi Jesus, > Martin v. L?wis wrote: > | I think it would be helpful if you could analyze the crashes that > | bsddb caused on Windows. Just go back a few revisions in the > | subversion tree to reproduce the crashes. > > I have no MS Windows machines in my environment :-( I remember those rampant BSDDB crashes on Windows well. I brought this up with Martin at PyCon; I really don't think we can fault BSDDB here -- basically, the tests weren't cleaning up their environment in the right order, so BSDDB was getting passed completely and utterly bogus values. I *think* I managed to persuade Martin that this was indeed our fault, and we can't really hold BSDDB accountable. (My argument being that if a 3rd party app says the behaviour of a method is undefined if you pass it a null pointer, and you pass it a null pointer, and it crashes your program, it's your fault, not theirs.) Once this was addressed, the BSDDB tests ran more or less on Windows 32-bit without error. Windows x64 was another matter though -- I traced the problem down to wildly conflicting compiler and linker flags between our Python build and how we were building BSDDB (or rather how BSDDB builds out of the box on Windows). My solution was to drop our reliance on the Berkeley_DB.sln/db_static.vcproj files completely, and mimic a bsddb44 vcproj in our own pcbuild.sln, which basically meant all the BSDDB source code got built in the exact same fashion as the rest of Python. I also took this approach with sqlite3 and it's worked really well -- there have been no issues with either module since this change. I've also got bsddb45.vcproj and bsddb46.vcproj projects floating around in one of my local branches somewhere. These mimic the corresponding BSDDB projects, with the intent being that when it comes to release time for 2.6 and 3.0, we'd make a decision about which one to ship with, and then set the Python _bsddb module to use that. I should probably pick that up again... Hope this clarifies things... Regards, Trent. From martin at v.loewis.de Tue Apr 22 21:58:08 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 22 Apr 2008 21:58:08 +0200 Subject: [Python-Dev] 3k checkin mails to python-checkins In-Reply-To: References: Message-ID: <480E4350.4060404@v.loewis.de> > Since a few days, checkin notifications for the 3k branch seem to be sent > to both the python-checkins and the python-3000-checkins lists. Was that a > deliberate decision or has some bug crept into the SVN hook? This should be fixed now. The new mailer.py had named some config options differently from the old one. Regards, Martin From mike.klaas at gmail.com Tue Apr 22 22:46:26 2008 From: mike.klaas at gmail.com (Mike Klaas) Date: Tue, 22 Apr 2008 13:46:26 -0700 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480DBE86.2000507@egenix.com> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <480DBE86.2000507@egenix.com> Message-ID: <1A9B6931-5D55-4442-BB50-0B888AA98086@gmail.com> On 22-Apr-08, at 3:31 AM, M.-A. Lemburg wrote: >>> >> I don't think that should be part of the standard library. People >> will mistake what it tells them for certain. > > +1 > > I also think that it's better to educate people to add (correct) > encoding information to their text data, rather than give them a > guess mechanism... That is a fallacious alternative: the programmers that need encoding detection are not the same people who are omitting encoding information. I only have a small opinion on whether charset detection should appear in the stdlib, but I am somewhat perplexed by the arguments in this thread. I don't see how inclusion in the stdlib would make people more inclined to think that the algorithm is always correct. In terms of the need of this functionality: Martin wrote: > Can you please explain why that is? Web programs should not normally > have the need to detect the encoding; instead, it should be specified > always - unless you are talking about browsers specifically, which > need to support web pages that specify the encoding incorrectly. Any program that needs to examine the contents of documents/feeds/ whatever on the web needs to deal with incorrectly-specified encodings (which, sadly, is rather common). The set of programs of programs that need this functionality is probably the same set that needs BeautifulSoup--I think that set is larger than just browsers -Mike From mal at egenix.com Tue Apr 22 22:54:35 2008 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 22 Apr 2008 22:54:35 +0200 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <480DBE86.2000507@egenix.com> Message-ID: <480E508B.2060006@egenix.com> [CCing python-dev again] On 2008-04-22 12:38, Greg Wilson wrote: >>> I don't think that should be part of the standard library. People >>> will mistake what it tells them for certain. >>> [etc] > > These are all good arguments, but the fact remains that we can't control > our inputs (e.g., we're archiving mail messages sent to lists managed by > DrProject), and some of those inputs *don't* tell us how they're encoded. > Under those circumstances, what would you recommend? I haven't done much research into this, but in general, I think it's better to: * first try to look at other characteristics of a text message, e.g. language, origin, topic, etc., * then narrow down the number of encodings which could apply, * rank them to try to avoid ambiguities and * then try to see what percentage of the text you can decode using each of the encodings in reverse ranking order (ie. more specialized encodings should be tested first, latin-1 last). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 22 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 From asmodai at in-nomine.org Tue Apr 22 23:12:18 2008 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Tue, 22 Apr 2008 23:12:18 +0200 Subject: [Python-Dev] Known doctest bug with unicode? In-Reply-To: References: <20080418141216.GF64048@nexus.in-nomine.org> <361b27370804180717nce0cd15y2c131ac18478fb56@mail.gmail.com> <20080418142705.GG64048@nexus.in-nomine.org> Message-ID: <20080422211218.GZ64048@nexus.in-nomine.org> -On [20080418 18:05], Adam Olsen (rhamph at gmail.com) wrote: >4. Make doctest smarter, so that it can grab the original module's encoding. >5. Wait until 3.0, where this is hopefully fixed by making doctests >use unicode by default? Getting rid of the u in front of the strings as required made Python 3 indeed run the doctests as they should. So there's a difference in behaviour between 2.x and 3.0 when it comes to this part. I guess the better behaviour would be for doctest to honour the encoding specified in the file/module? If other people agree I can see what I can to make that work. -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Confutatis maledictis, flammis acribus addictis... From martin at v.loewis.de Tue Apr 22 23:16:02 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 22 Apr 2008 23:16:02 +0200 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <1A9B6931-5D55-4442-BB50-0B888AA98086@gmail.com> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <480DBE86.2000507@egenix.com> <1A9B6931-5D55-4442-BB50-0B888AA98086@gmail.com> Message-ID: <480E5592.3050102@v.loewis.de> >> Can you please explain why that is? Web programs should not normally >> have the need to detect the encoding; instead, it should be specified >> always - unless you are talking about browsers specifically, which >> need to support web pages that specify the encoding incorrectly. > > Any program that needs to examine the contents of > documents/feeds/whatever on the web needs to deal with > incorrectly-specified encodings That's not true. Most programs that need to examine the contents of a web page don't need to guess the encoding. In most such programs, the encoding can be hard-coded if the declared encoding is not correct. Most such programs *know* what page they are webscraping, or else they couldn't extract the information out of it that they want to get at. As for feeds - can you give examples of incorrectly encoded one (I don't ever use feeds, so I honestly don't know whether they are typically encoded incorrectly. I've heard they are often XML, in which case I strongly doubt they are incorrectly encoded) As for "whatever" - can you give specific examples? > (which, sadly, is rather common). The > set of programs of programs that need this functionality is probably the > same set that needs BeautifulSoup--I think that set is larger than just > browsers Again, can you give *specific* examples that are not web browsers? Programs needing BeautifulSoup may still not need encoding guessing, since they still might be able to hard-code the encoding of the web page they want to process. In any case, I'm very skeptical that a general "guess encoding" module would do a meaningful thing when applied to incorrectly encoded HTML pages. Regards, Martin From martin at v.loewis.de Tue Apr 22 23:32:55 2008 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 22 Apr 2008 23:32:55 +0200 Subject: [Python-Dev] Known doctest bug with unicode? In-Reply-To: <20080422211218.GZ64048@nexus.in-nomine.org> References: <20080418141216.GF64048@nexus.in-nomine.org> <361b27370804180717nce0cd15y2c131ac18478fb56@mail.gmail.com> <20080418142705.GG64048@nexus.in-nomine.org> <20080422211218.GZ64048@nexus.in-nomine.org> Message-ID: <480E5987.1030906@v.loewis.de> > So there's a difference in behaviour between 2.x and 3.0 when it comes to > this part. I guess the better behaviour would be for doctest to honour the > encoding specified in the file/module? If other people agree I can see what > I can to make that work. I'm fairly skeptical that you can make that work, whether or not it's a good idea. Regards, Martin From mal at egenix.com Tue Apr 22 23:34:20 2008 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 22 Apr 2008 23:34:20 +0200 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <08Apr22.093328pdt."58696"@synergy1.parc.xerox.com> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <480DBE86.2000507@egenix.com> <08Apr22.081453pdt."58696"@synergy1.parc.xerox.com> <08Apr22.093328pdt."58696"@synergy1.parc.xerox.com> Message-ID: <480E59DC.3010007@egenix.com> On 2008-04-22 18:33, Bill Janssen wrote: > The 2002 paper "A language and character set determination method > based on N-gram statistics" by Izumi Suzuki and Yoshiki Mikami and > Ario Ohsato and Yoshihide Chubachi seems to me a pretty good way to go > about this. Thanks for the reference. Looks like the existing research on this just hasn't made it into the mainstream yet. Here's their current project: http://www.language-observatory.org/ Looks like they are focusing more on language detection. Another interesting paper using n-grams: "Language Identification in Web Pages" by Bruno Martins and M?rio J. Silva http://xldb.fc.ul.pt/data/Publications_attach/ngram-article.pdf And one using compression: "Text Categorization Using Compression Models" by Eibe Frank, Chang Chui, Ian H. Witten http://portal.acm.org/citation.cfm?id=789742 > They're looking at "LSE"s, language-script-encoding > triples; a "script" is a way of using a particular character set to > write in a particular language. > > Their system has these requirements: > > R1. the response must be either "correct answer" or "unable to detect" > where "unable to detect" includes "other than registered" [the > registered set of LSEs]; > > R2. Applicable to multi-LSE texts; > > R3. never accept a wrong answer, even when the program does not have > enough data on an LSE; and > > R4. applicable to any LSE text. > > So, no wrong answers. > > The biggest disadvantage would seem to be that the registration data > for a particular LSE is kind of bulky; on the order of 10,000 > shift-codons, each of three bytes, about 30K uncompressed. > > http://portal.acm.org/ft_gateway.cfm?id=772759&type=pdf For a server based application that doesn't sound too large. Unless you're using a very broad scope, I don't think that you'd need more than a few hundred LSEs for a typical application - nothing you'd want to put in the Python stdlib, though. > Bill > >>> IMHO, more research has to be done into this area before a >>> "standard" module can be added to the Python's stdlib... and >>> who knows, perhaps we're lucky and by the time everyone is >>> using UTF-8 anyway :-) >> I walked over to our computational linguistics group and asked. This >> is often combined with language guessing (which uses a similar >> approach, but using characters instead of bytes), and apparently can >> usually be done with high confidence. Of course, they're usually >> looking at clean texts, not random "stuff". I'll see if I can get >> some references and report back -- most of the research on this was >> done in the 90's. >> >> Bill -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 22 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 From alessandro.guido at gmail.com Tue Apr 8 15:15:24 2008 From: alessandro.guido at gmail.com (Alessandro Guido) Date: Tue, 8 Apr 2008 15:15:24 +0200 Subject: [Python-Dev] py3k: print function treats sep=None and end=None in an unintuitive way Message-ID: <200804081515.24887.alessandro.guido@gmail.com> Can anybody please point me why print('a', 'b', sep=None, end=None) should produce "a b\n" instead of "ab"? I've read http://docs.python.org/dev/3.0/library/functions.html#print, pep-3105 and some ml threads but did not find a good reason justifying such a strange behaviour. Thanks. -Alessandro Guido From coughlan at ski.org Tue Apr 8 19:13:06 2008 From: coughlan at ski.org (James Coughlan) Date: Tue, 08 Apr 2008 10:13:06 -0700 Subject: [Python-Dev] New project : Spyke python-to-C compiler Message-ID: <47FBA7A2.8000701@ski.org> Hello, Your project sounds very interesting! I would be happy to send you some test code -- perhaps you could give me some more details about what you're looking for. (E.g. *small,* self-contained python/numpy functions.) Also, are you aware of Shed Skin? "Shed Skin is an experimental Python-to-C++ compiler. It can convert pure, but implicitly statically typed Python programs into optimized C++ code." http://shed-skin.blogspot.com/ Best, James -- ------------------------------------------------------- James Coughlan, Ph.D., Scientist The Smith-Kettlewell Eye Research Institute Email: coughlan at ski.org URL: http://www.ski.org/Rehab/Coughlan_lab/ Phone: 415-345-2146 Fax: 415-345-8455 ------------------------------------------------------- From python at venix.com Tue Apr 8 20:27:39 2008 From: python at venix.com (Lloyd Kvam) Date: Tue, 08 Apr 2008 14:27:39 -0400 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> Message-ID: <1207679260.3474.74.camel@localhost.localdomain> On Tue, 2008-04-08 at 10:01 -0700, zooko wrote: > They both agreed that it made perfect sense. I told one of them > about the alternate proposal to define a new database file to > contain > a list of installed packages, and he sighed and rolled his eyes and > said "So they are planning to reinvent apt!". When I wear my sysadmin hat, eggs become a nuisance. They are not listed in the system packages; if zipped they won't work when the apache user tries to import them; easy_install can produce unexpected upgrades. The system package manager (apt or yum) is much preferred. As a developer, eggs are great. If a python module is not already available from my system packagers, easy_install will find it, get it, and install it. I waste almost no time with system administration issues while developing. Fortunately, distutils includes tools like bdist_rpm so that python modules can be packaged for easy processing by the system package manager. So once I need to switch back to a sysadmin role, I can use the system tools to install and track packages. -- Lloyd Kvam Venix Corp DLSLUG/GNHLUG library http://www.librarything.com/catalog/dlslug http://www.librarything.com/profile/dlslug http://www.librarything.com/rsshtml/recent/dlslug From gael.varoquaux at normalesup.org Wed Apr 9 09:54:16 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 9 Apr 2008 09:54:16 +0200 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <87bq4jolks.fsf@benfinney.id.au> References: <913B68B1-C70C-4B56-8223-152954B86EBE@zooko.com> <47EB07A6.6040800@plope.com> <73387C02-3D5C-4D9E-A886-82D486AC0C76@zooko.com> <87bq4jolks.fsf@benfinney.id.au> Message-ID: <20080409075416.GA1761@phare.normalesup.org> On Wed, Apr 09, 2008 at 11:37:07AM +1000, Ben Finney wrote: > zooko writes: > > I am skeptical that prorgammers are going to be willing to use a new > > database format. They already have a database -- their filesystem -- > > and they already have the tools to control it -- mv, rm, and > > PYTHONPATH. Many of them already hate the existence the > > "easy_instlal.pth" database file, and I don't see why a new database > > file would be any different. > Moreover, many of us already have a database of *all* packages on the > system, not just Python-language ones: the package database of our > operating system. Adding another, parallel, database which needs > separate maintenance, and only applies to Python packages, is not a > step forward in such a situation. 90 % (at least) of the world does not have such database. I, and probably you, have such a very nice database. I works well, and we can choose to forget the problems our users are facing. It does not solve them though. In addition, packaging is system-specific. I recently had to learn some Debian packaging, because I wanted my Ubuntu and Debian users to be able to use my projects seamlessly. What about RPMs for RHEL, Fedora, Mandriva? ... and coronary packages? and MSIs? ... When do I find time to do development if I have to learn all this packaging. It would be fantastic to have an abstraction on all these packaging systems, including, as you point out, their database. I do agree that reusing the system packaging's database is great, and would be the best option for system-wide install. However one of the very neat features of setuptools and eggs is that you don't need administrator access to install the packages, and that is great in a shared environment, like a computation cluster. The system's database is thus unfortunately not a complete solution to the problem. My 2 cents, Ga?l From gael.varoquaux at normalesup.org Wed Apr 9 10:00:16 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 9 Apr 2008 10:00:16 +0200 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> Message-ID: <20080409080016.GB1761@phare.normalesup.org> On Wed, Apr 09, 2008 at 12:41:32AM -0400, Phillip J. Eby wrote: > >The way to achieve a database for Python would be to provide tools for > >conversion of eggs to rpms and debs, > Such tools already exist, although the conversion takes place from > source distributions rather than egg distributions. What is the status of the deb backend? The only one I know is unofficial maintained by Andrew Straw, but my information my be lagging behind. By the way, if these tools work well, they are priceless! Cheers, Ga?l From sklein at cpcug.org Wed Apr 9 17:52:53 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Wed, 9 Apr 2008 11:52:53 -0400 (EDT) Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> Message-ID: <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> On Wed, April 9, 2008 12:41 am, Phillip J. Eby wrote: > At 10:49 PM 4/8/2008 -0400, Stanley A. Klein wrote: >>On Tue, April 8, 2008 9:37 pm, Ben Finney >> wrote: >> > Date: Wed, 09 Apr 2008 11:37:07 +1000 >> > From: Ben Finney >> > Subject: Re: [Distutils] how to easily consume just the parts of eggs >> > that are good for you >> > To: Distutils-Sig at Python.Org >> > >> > >> > zooko writes: >> >> eyes and said "So they are planning to reinvent apt!". >> > >> > That's pretty much my reaction, too. >> >>I have the same reaction. > > I'm curious. Have any of you actually read PEP 262 in any detail? I > have seen precious little discussion so far that doesn't appear to be > based on significant misunderstandings of either the purpose of > reviving the PEP, or the mechanics of its proposed implementation. I haven't read the PEP at all. I generally don't read PEP's. > > >>I have tried in the past to use easy_install, but have run into problems >>because there is no communication between easy_install and the rpm >>database, resulting in failure of easy_install to recognize that >>dependencies have already been installed using rpms. > > This problem doesn't exist with Python 2.5, unless you're using a > platform that willfully strips out the installation information that > Python 2.5 provides for these packages. > IIRC, I have had the problem with Python 2.5 on Fedora 7. Until recently, Fedora packagers did strip out the egg information included with Python packages they packaged. I left those files in when packaging myself using bdist_rpm. However, are you implying that the installation information for Python egg packages accesses and coordinates with the rpm database? I found myself having to go into the setup.py for the relevant package(s) and delete any statements regarding dependencies. Otherwise, IIRC, the packaging couldn't proceed because the Python packaging tool couldn't find the dependencies that had already been installed as rpms. After installation, Python managed to find the relevant files, but the packaging tool couldn't. > >>A database focused only on Python packages is highly inappropriate for >>Linux systems, violates the Linux standards, and creates problems because >>eggs are not coordinated with the operating system package manager. > > The revamp of PEP 262 is aimed at removing .egg files and directories > from the process, by allowing system packagers to tell Python what > files belong to them and should not be messed with. And conversely, > allowing systems and installation targets *without* package managers > to safely manage their Python installations. IMHO, the main system without a package manager is Windows. A reasonable way to deal with Windows would be to create a package manager for it that could be used by Python and anyone else who wanted to use it. The package manager could establish a file hierarchy similar to the Unix FHS and install files appropriately, except for what is needed to satisfy the Windows OS. That would probably go a long way to addressing the issues being discussed here. This is primarily a Windows problem, not a Python problem. > > >> The >>way to achieve a database for Python would be to provide tools for >>conversion of eggs to rpms and debs, > > Such tools already exist, although the conversion takes place from > source distributions rather than egg distributions. > You are talking here about bdist_rpm and not about a tool that would take a Python package distributed as an egg file and convert the egg to an rpm or a deb. Unfortunately, some Python packagers are beginning to limit their focus only to egg distribution. That creates a problem for users who have native operating system package management. > >>to have eggs support conformance to >>the LSB and FHS, > > Applying LSB and FHS to the innards of Python packages makes as much > sense as applying them to the contents of Java .jar files -- i.e., > none. If it's unchanging data that's part of a program or library, > then it's a program or library, just like static data declared in a C > program or library. Whether the file extension is .py, .so, or even > .png is irrelevant. The FHS defines places to put specific kinds of files, such as command scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation (/usr/share/doc/package-name), and configuration files (/etc). There are several kinds of files identified and places defined to put them. Distribution by eggs has a tendency to scoop up all of those files and put them in /usr/lib/python/site-packages, regardless of where they belong. Having eggs support conformance to FHS would mean recognizing and tagging the relevant files. A tool for converting eggs to rpms or debs would essentially reformat the egg to rpm or deb and put files where they belong. Stan Klein From brianc at temple.edu Wed Apr 9 19:36:38 2008 From: brianc at temple.edu (Brian Cole) Date: Wed, 9 Apr 2008 11:36:38 -0600 Subject: [Python-Dev] Python Leopard DLL Hell In-Reply-To: <47FC2161.5020905@gmail.com> References: <54b165660804081819g4787f4d9g821c7269af7b4c66@mail.gmail.com> <47FC2161.5020905@gmail.com> Message-ID: <54b165660804091036n40aef4c9r98aabe2a059fc19e@mail.gmail.com> I have learned that this is a specific behavior of OS X. I have submitted a formal bug report to Apple about the problem. It appears that this is documented by Apple as acceptable: http://developer.apple.com/documentation/DeveloperTools/Reference/MachOReference/Reference/reference.html#//apple_ref/c/func/dlopen Whereas, linux will respect the fact you gave it a specific shared library: http://linux.die.net/man/3/dlopen If I am provided a workaround by apple I will post a python patch. A little scary that someone can circumvent my application by just setting an environment variable. -Brian Cole On Tue, Apr 8, 2008 at 7:52 PM, Michael Torrie wrote: > Brian Cole wrote: > > That appears to be working correctly at first glance. The argument to > > dlopen is the correct shared library. Unfortunately, either python or > > OS X is lying to me here. If I inspect the python process with OS X's > > Activity Monitor and look at the "Open Files and Ports" tab, it shows > > that the _foo.so shared library is actually the one located inside > > $DYLD_LIBRARY_PATH. > > > > So this problem may not be python's, but I place it here as a first > > shot (maybe I'm using the imp module incorrectly). > > Sounds like you're going to need to learn how to use dtrace. Then you > can more closely monitor exactly what python and the loader are doing. > dtrace is very complicated (borrowed from Solaris) but extremely > powerful. Worth learning anyway, but sounds like it's probably the > debugging tool you need. > > Another thing you can do is check through the python source code and see > how the os x-specific code is handling this type of situation. > > -- > http://mail.python.org/mailman/listinfo/python-list > From sklein at cpcug.org Wed Apr 9 20:26:31 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Wed, 9 Apr 2008 14:26:31 -0400 (EDT) Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> Message-ID: <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> All my development is done on Linux. I use Windows very minimally (such as for tax preparation) and unless forced to do so for specific circumstances (such as submittal to grants.gov) do not expose Windows to the Internet. In the future there may possibly arise a need for us to port some Linux-developed Python code to Windows, but we will have to cross that bridge when we get there. I think you raise an interesting issue: What is a package manager? I have minimal experience installing packages on Windows over the last 5-10 years, but in my experience a Windows package comes as an executable that, when run, installs itself. Unless a third-party program monitors the installation, uninstalling is a nasty chore, as is finding out what files were installed or where they went. The rpm and deb package managers (and their yum and other higher level dependency managers) do a lot of things: 1. They install packages and maintain databases of what packages were installed 2. They manage dependencies 3. They support clean uninstalling of packages 4. They can query packages, both installed (via their databases) and not yet installed (e.g., as rpm or deb files), to determine attributes, such as files they install, dependencies, and other information defined at packaging time. 5. They build packages and (in some cases) can rebuild packages. 6. They can verify packages for integrity and security purposes. 7. They can download package files and maintain archives of installed package files for use as local repositories. There may be other functions, but the above is a top-of-the-head list. I can say that I'm not terribly happy with Python packaging that is only minimally compatible with rpm. I haven't used Ubuntu all that much. I do like Ubuntu's packaging and package management, and I do know that there are programs, such as alias, that can translate from rpm to deb formats. I don't think I ever said that Windows is broken in the area of package management. My own experience is that the files of Windows programs tend to be put in a directory devoted to the program, rather than put in directories with other files having similar purposes. At one time, the default location in Windows for word processing files was even in a sub-directory of the word processing program. That changed to having a form of user home directory, but it didn't change much for the program files themselves. Unix/Linux puts the files in specific areas of the file system having functional commonality. One could almost say that the Windows default approach to structuring its filesystem avoids or minimizes the need for package management. I repeat that this issue mainly arises because Windows doesn't have the same kind of filesystem structure (and therefore the need for package management) that other systems have. I don't know what Windows add/remove programs function does, but all it might do is to run the executable to install packages and record the installation (as was previously done by third party programs) to facilitate clean removal. Unless you can perform more of the other functions I listed above, I doubt I would call add/remove a package manager. Stan Klein On Wed, April 9, 2008 1:23 pm, Paul Moore wrote: > On 09/04/2008, Stanley A. Klein wrote: >> IMHO, the main system without a package manager is Windows. A >> reasonable >> way to deal with Windows would be to create a package manager for it >> that >> could be used by Python and anyone else who wanted to use it. The >> package >> manager could establish a file hierarchy similar to the Unix FHS and >> install files appropriately, except for what is needed to satisfy the >> Windows OS. That would probably go a long way to addressing the issues >> being discussed here. This is primarily a Windows problem, not a >> Python >> problem. > > Windows does have a package manager - the add/remove programs > application. It's extremely limited, and doesn't make any attempt at > doing dependency resolution, certainly - but that's a separate issue. > > I don't know if you use Windows (as in, develop programs using Python > on Windows). If you do, then I'd be interested in your views on > bdist_wininst and bdist_msi installers, and how they fit into the > setuptools/egg environment, particularly with regard to the package > manager you are proposing. If you don't use Windows, then I don't see > how you can usefully comment. > > Personally, as I've said before, I don't have a problem with a > Python-only package manager, as long as it replaces or integrates > bdist_wininst and bdist_msi. Having two package managers is far worse > than having none - and claiming that add/remove programs "isn't a > package manager" is just ignoring reality (if it isn't, then why do > bdist_wininst and bdist_msi exist?). > > Are the Linux users happy with having a Python package manager that > ignores RPM/apt? Why should Windows users be any happier? > > Sorry - I'm feeling a little grumpy. I've read one too many "Windows > is so broken that people who use it obviously don't care about doing > things right" postings this week :-( > > Paul. > -- From gael.varoquaux at normalesup.org Wed Apr 9 21:19:48 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Wed, 9 Apr 2008 21:19:48 +0200 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> Message-ID: <20080409191948.GA11856@phare.normalesup.org> On Wed, Apr 09, 2008 at 02:26:31PM -0400, Stanley A. Klein wrote: > The rpm and deb package managers (and their yum and other higher level > dependency managers) do a lot of things: > 1. They install packages and maintain databases of what packages were > installed > 2. They manage dependencies > 3. They support clean uninstalling of packages > 4. They can query packages, both installed (via their databases) and not > yet installed (e.g., as rpm or deb files), to determine attributes, such > as files they install, dependencies, and other information defined at > packaging time. > 5. They build packages and (in some cases) can rebuild packages. > 6. They can verify packages for integrity and security purposes. > 7. They can download package files and maintain archives of installed > package files for use as local repositories. You are collapsing three different functionalities in one: * Dealing with repositories and downloading: yum/apt * Installing + uninstalling packages, and dealing with system consistency (thus checking the dependencies are available): rpm/dpkg * Building For me it is important that the 3 are separated: * I may want to download the dependencies of a package to burn to a CD for a computer that does not have internet access. * I may want to send a tarball to a build server that does the building, but no install (so as not to corrupt my working system). Cheers, Ga?l From sklein at cpcug.org Wed Apr 9 22:00:18 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Wed, 9 Apr 2008 16:00:18 -0400 (EDT) Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409191948.GA11856@phare.normalesup.org> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <20080409191948.GA11856@phare.normalesup.org> Message-ID: <12119.207.188.248.157.1207771218.squirrel@www.cpcug.org> On Wed, April 9, 2008 3:19 pm, Gael Varoquaux wrote: > On Wed, Apr 09, 2008 at 02:26:31PM -0400, Stanley A. Klein wrote: >> The rpm and deb package managers (and their yum and other higher level >> dependency managers) do a lot of things: > >> 1. They install packages and maintain databases of what packages were >> installed >> 2. They manage dependencies >> 3. They support clean uninstalling of packages >> 4. They can query packages, both installed (via their databases) and >> not >> yet installed (e.g., as rpm or deb files), to determine attributes, such >> as files they install, dependencies, and other information defined at >> packaging time. >> 5. They build packages and (in some cases) can rebuild packages. >> 6. They can verify packages for integrity and security purposes. >> 7. They can download package files and maintain archives of installed >> package files for use as local repositories. > > You are collapsing three different functionalities in one: > > * Dealing with repositories and downloading: yum/apt > > * Installing + uninstalling packages, and dealing with system > consistency (thus checking the dependencies are available): rpm/dpkg > > * Building > > For me it is important that the 3 are separated: > > * I may want to download the dependencies of a package to burn to a CD > for a computer that does not have internet access. > > * I may want to send a tarball to a build server that does the building, > but no install (so as not to corrupt my working system). > > Cheers, > > Ga?l > Gael - The functionalities are combined in programs but are not necessarily required to be used all at the same time. I'm not that familiar with apt, but yum also installs, including downloading both a package and its dependencies. Yum also has a query capability (yum list, yum info). I think synaptic does the same thing yum does, and adds a GUI and search capabilities similar to yum info as well. The build capabilities of rpm were moved to rpmbuild, but the building remains part of the rpm system. IIRC, bdist_rpm actually calls rpmbuild as part of its processing. Also, IIRC, rpmbuild can build from a tarball if it contains an rpm spec. It does not install in the same process. That is a separate step. You would not corrupt your working system by building an rpm from a tarball on it. BTW, I would not want to do dependencies with rpm if yum is available. Doing dependencies with rpm is very difficult and it is easy to wind up in "dependency hell". Yum will find the dependencies and install them as long as they are in repositories that are registered in the yum configuration. I looked at "man yum" and couldn't find an option to download dependencies to the local repository without installing. However, if you did install a package and its dependencies, and if you have selected the option of retaining the cache and not cleaning it after installation, the rpms (e.g., for updates) are in /var/cache/yum/updates/packages/. They can be copied from there to a CD for a system without internet connectivity. Also, both Fedora and Ubuntu have software for building installable live CD's, although I don't know how they get their package files. Stan Klein From sklein at cpcug.org Wed Apr 9 22:43:32 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Wed, 9 Apr 2008 16:43:32 -0400 (EDT) Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <20080409194044.1A4E03A406A@sparrow.telecommunity.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> Message-ID: <12121.207.188.248.157.1207773812.squirrel@www.cpcug.org> On Wed, April 9, 2008 3:40 pm, Phillip J. Eby wrote: > At 11:52 AM 4/9/2008 -0400, Stanley A. Klein wrote: >>However, are you implying that the installation information for Python >> egg >>packages accesses and coordinates with the rpm database? > > Yes, when the information isn't stripped out. Try a more recent Fedora. > > >>IMHO, the main system without a package manager is Windows. > > You're ignoring shared environments and development > environments. (Not to mention Mac OS.) > I don't understand what you mean by "shared environments and development environments". I also don't know much about Mac OS, except that its underlying Darwin system is a version of BSD (that I assume would follow the Unix FHS). > >> A reasonable >>way to deal with Windows would be to create a package manager for it that >>could be used by Python and anyone else who wanted to use it. > > Let us know when you've finished it, along with the one for Mac OS. :) I have enough trouble with what I'm already doing. :-) > > Of course this still won't do anything for shared environments and > development environments. > > >>You are talking here about bdist_rpm and not about a tool that would take >>a Python package distributed as an egg file and convert the egg to an rpm >>or a deb. Unfortunately, some Python packagers are beginning to limit >>their focus only to egg distribution. That creates a problem for users >>who have native operating system package management. > > That is indeed a problem -- but it's a social one, not a technical > one. It's trivial for the publisher of an egg to change their > command line from "setup.py bdist_egg upload" to "setup.py sdist > bdist_egg upload", as soon as their users (politely) request that they do > so. > I agree that we are dealing with a combination of technical and social issues here. However, I think it takes a lot more understanding for a publisher to get everything straight. > >> > Applying LSB and FHS to the innards of Python packages makes as much >> > sense as applying them to the contents of Java .jar files -- i.e., >> > none. If it's unchanging data that's part of a program or library, >> > then it's a program or library, just like static data declared in a C >> > program or library. Whether the file extension is .py, .so, or even >> > .png is irrelevant. >> >>The FHS defines places to put specific kinds of files, such as command >>scripts (/bin, /usr/bin, /sbin, or /usr/sbin), documentation >>(/usr/share/doc/package-name), and configuration files (/etc). There are >>several kinds of files identified and places defined to put them. >>Distribution by eggs has a tendency to scoop up all of those files and >> put >>them in /usr/lib/python/site-packages, regardless of where they belong. > > Eggs don't include documentation or configuration files, and they > install scripts in script directories, so I don't get what you're > talking about here. For any other data that a package accesses at > runtime, my earlier comments apply. > But rpms and debs do include these files, plus manual pages, localization files and a lot of other ancillary stuff. IIRC, you once mentioned that you have a CENTOS system. Do an "rpm -qa |sort|less" to get an alphabetized list of your installed packages, and then an "rpm -qil" on some of the packages, and you will see the range of different kinds of files in there. > >>Having eggs support conformance to FHS would mean recognizing and tagging >>the relevant files. A tool for converting eggs to rpms or debs would >>essentially reformat the egg to rpm or deb and put files where they >>belong. > > No, because such files as you describe don't exist. If you think > they do, then either you have misunderstood the nature of the files > in question, or the developer has incorrectly placed non-runtime > files in their installation tree. > Most of the Python tarballs I have downloaded have all kinds of files in their installation trees. This is a major pain in the you-know-what for someone trying to use bdist_rpm and get proper, FHS-compliant rpms. If eggs are supposed to be strictly runtime files, I think very few developers actually understand that. Better yet, how do you define what should be included in an installation? It sounds like the egg concept doesn't include several kinds of files that rpm and deb would include in an installation. I think that may be an important issue here. Stan Klein From gael.varoquaux at normalesup.org Thu Apr 10 00:51:38 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 10 Apr 2008 00:51:38 +0200 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <79990c6b0804091023p51a0e343l73ac695944a4c9be@mail.gmail.com> <36773.207.188.248.157.1207765591.squirrel@www.cpcug.org> <79990c6b0804091546p5cf67f2ta800c675e7baa256@mail.gmail.com> Message-ID: <20080409225138.GH28763@phare.normalesup.org> On Wed, Apr 09, 2008 at 11:46:19PM +0100, Paul Moore wrote: > I find this whole discussion hugely confusing, because a lot of people > are stating opinions about environments which it seems they don't use, > or know much about. I don't know how to avoid this, but it does make > it highly unlikely that any practical progress will get made. I find that something that doesn't help at all the discussion move forward is that everybody has different usecases in mind, on different platforms, and is not interested in other people's usecases. Hopefuly I am wrong, Cheers, Ga?l From gael.varoquaux at normalesup.org Thu Apr 10 00:59:24 2008 From: gael.varoquaux at normalesup.org (Gael Varoquaux) Date: Thu, 10 Apr 2008 00:59:24 +0200 Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <79990c6b0804091552h4ee7f9f6t5512ed5f5aa70d2a@mail.gmail.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <87iqyqir2t.fsf@benfinney.id.au> <79990c6b0804091552h4ee7f9f6t5512ed5f5aa70d2a@mail.gmail.com> Message-ID: <20080409225924.GI28763@phare.normalesup.org> On Wed, Apr 09, 2008 at 11:52:08PM +0100, Paul Moore wrote: > And I would say that Windows doesn't have a problem. Are any Windows > users proposing building a package management system for Windows > (Python-specific or otherwise)? It's a genuine question - is this > something that Windows users are after, or is it just Linux users > trying to show Windows users what they are missing? Well, users don't phrase this that way, because they don't know what package management (or rather automatic dependency tracking) is, but yes, they are some usecases. It is nowadays really tedious to deploy Python applications making uses of many packages on Python. The scientific community is a domain in which this problem is crucial, as we are trying to ship desktop applications to non-computer-savy people, with many dependencies outside the standard library. Enthought is working on shipping a Python distribution with some sort of package management for this purpose ( see http://code.enthought.com/enstaller/ ), and finding it is not an easy problem. Cheers, Gael From adelagon at gmail.com Thu Apr 10 03:23:39 2008 From: adelagon at gmail.com (Alvin Delagon) Date: Thu, 10 Apr 2008 09:23:39 +0800 Subject: [Python-Dev] PyArg_ParseTuple and Py_BuildValue question Message-ID: <7a01f6c00804091823x74e29f75jaf4dd3ad53481b1a@mail.gmail.com> Hello fellow pythonistas, I'm currently writing a simple python SCTP module in C. So far it works sending and receiving strings from it. The C sctp function sctp_sendmsg() has been wrapped and my function looks like this: sendMessage(PyObject *self, PyObject *args) { const char *msg = ""; if (!PyArg_ParseTuple(args, "s", &msg)) return NULL; snprintf(buffer, 1025, msg); ret = sctp_sendmsg(connSock, (void *)buffer, (size_t)strlen(buffer), 0, 0, 0x03000000, 0, 0, 0, 0); return Py_BuildValue("b", ""); } I'm going to construct an SS7 packet in python using struct.pack(). Here's the question, how am I going to pass the packet I wrote in python to my module and back? I already asked this question in comp.lang.python but so far no responses yet. I hope anyone can point me to the right direction. Thanks in advance. --- Alvin Delagon -------------- next part -------------- An HTML attachment was scrubbed... URL: From sklein at cpcug.org Thu Apr 10 03:58:57 2008 From: sklein at cpcug.org (Stanley A. Klein) Date: Wed, 9 Apr 2008 21:58:57 -0400 (EDT) Subject: [Python-Dev] [Distutils] how to easily consume just the parts of eggs that are good for you In-Reply-To: <47FD4E9D.8060600@enthought.com> References: <20080409044125.AB5153A40D7@sparrow.telecommunity.com> <33116.207.188.248.157.1207756373.squirrel@www.cpcug.org> <20080409194044.1A4E03A406A@sparrow.telecommunity.com> <47FD4E9D.8060600@enthought.com> Message-ID: <43869.71.178.83.244.1207792737.squirrel@www.cpcug.org> On Wed, 2008-04-09 at 18:17 -0500, Dave Peterson wrote: > I think I can sum up any further points by simply asking: "Should it > be safe to assume I can distribute my application via eggs / > easy_install just because it is written in Python?" I think that based on this discussion the bottom line answer to this question is "No". Stan Klein -------------- next part -------------- An HTML attachment was scrubbed... URL: From lclark at 2bdx.net Tue Apr 15 22:32:12 2008 From: lclark at 2bdx.net (Leonard Clark) Date: Tue, 15 Apr 2008 13:32:12 -0700 Subject: [Python-Dev] Patch submitted for xmlrpclib Message-ID: I submitted a patch a few days ago (http://bugs.python.org/issue2623) to fix a datetime parameter formatting issue (See issue) I was wondering if this was adequate and whether it could be included in future releases. Thank you. -- Leonard Clark From ikshefem at gmail.com Wed Apr 16 15:08:38 2008 From: ikshefem at gmail.com (iks hefem) Date: Wed, 16 Apr 2008 15:08:38 +0200 Subject: [Python-Dev] SetType=set in types module ? Message-ID: <78e18a710804160608j68a07056y63fcd3a0ebab08ca@mail.gmail.com> Hi, the SetType is not available in the "types" module, so wouldn't it be needed here ? (in 2.6 by example) I guess the change is really simple and would be backward compatible : adding SetType = set From tom at vector-seven.com Thu Apr 17 01:23:53 2008 From: tom at vector-seven.com (Thomas Lee) Date: Thu, 17 Apr 2008 09:23:53 +1000 Subject: [Python-Dev] Global Python Sprint Weekends: May 10th-11th and June 21st-22nd. In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22F389FE@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E22F389FE@EXMBX04.exchhosting.com> Message-ID: <48068A89.9000306@vector-seven.com> Anyone in Melbourne, Australia keen for the first sprint? I'm not sure if I'll be available, but if I can it'd be great to work with some others. Failing that, it's red bull and pizza in my lounge room :) I've been working on some neat code for an AST optimizer. If I'm free that weekend, I'll probably continue my work on that. Cheers, T Trent Nelson wrote: > Following on from the success of previous sprint/bugfix weekends and > sprinting efforts at PyCon 2008, I'd like to propose the next two > Global Python Sprint Weekends take place on the following dates: > > * May 10th-11th (four days after 2.6a3 and 3.0a5 are released) > * June 21st-22nd (~week before 2.6b2 and 3.0b2 are released) > > It seems there are a few of the Python User Groups keen on meeting > up in person and sprinting collaboratively, akin to PyCon, which I > highly recommend. I'd like to nominate Saturday across the board > as the day for PUGs to meet up in person, with Sunday geared more > towards an online collaboration day via IRC, where we can take care > of all the little things that got in our way of coding on Saturday > (like finalising/preparing/reviewing patches, updating tracker and > documentation, writing tests ;-). > > For User Groups that are planning on meeting up to collaborate, > please reply to this thread on python-dev at python.org and let every- > one know your intentions! > > As is commonly the case, #python-dev on irc.freenode.net will be > the place to be over the course of each sprint weekend; a large > proportion of Python developers with commit access will be present, > increasing the amount of eyes available to review and apply patches. > > For those that have an idea on areas they'd like to sprint on and > want to look for other developers to rope in (or just to communicate > plans in advance), please also feel free to jump on this thread via > python-dev@ and indicate your intentions. > > For those that haven't the foggiest on what to work on, but would > like to contribute, the bugs tracker at http://bugs.python.org is > the best place to start. Register an account and start searching > for issues that you'd be able to lend a hand with. > > All contributors that submit code patches or documentation updates > will typically get listed in Misc/ACKS.txt; come September when the > final release of 2.6 and 3.0 come about, you'll be able to point at > the tarball or .msi and exclaim loudly ``I helped build that!'', > and actually back it up with hard evidence ;-) > > Bring on the pizza and Red Bull! > > Trent. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/krumms%40gmail.com > From robert.hoelzl at baltech.de Thu Apr 17 12:08:41 2008 From: robert.hoelzl at baltech.de (=?ISO-8859-15?Q?Robert_H=F6lzl?=) Date: Thu, 17 Apr 2008 12:08:41 +0200 Subject: [Python-Dev] socket recv on win32 can be extremly delayed (python bug?) Message-ID: <480721A9.1070603@baltech.de> hello, I tried to implement a simple python XMLRPC service on a win32 environment (client/server code inserted below). The profiler of the client told me, that a simple function call needs about 200ms (even if I run it in a loop, the time needed per call stays the same). After analysing the problem with etherreal I found out, that the XMLRPC request is transmitted via two TCP packets. One containing the HTTP header and one containting the data. But the acknowledge to the first TCP packet is delayed by 200ms. I tried around on the server side and found out that if the server reads exactly all bytes transfered in the first TCP frame (via socket.recv()), the next socket.recv(), even if reading only one byte, needs about 200 ms. But if I read one byte less than transfered in the first TCP frame and then reading 2 bytes (socket.recv(2)) there is no delay, although the same total amount of data was read. After some googling I found the website http://support.microsoft.com/?scid=kb%3Ben-us%3B823764&x=12&y=15, which proposed a workaround (modifing the registryentry for the tcp/ip driver) that did work. But modifing the clients registry settings is no option for us. Is there anybody who nows how to solve the problem? Or is it even a problem if the python socket implementation? By the way: I testet Win2000 SP4 and WinXP SP2 with Python 2.3.3 and Python 2.5.1 each. CLIENT: ---------- import xmlrpclib import profile server = xmlrpclib.ServerProxy("http://server:80") profile.run('server.test(1,2)') SERVER: ---------- import SimpleXMLRPCServer def test(a,b): return a+b server = SimpleXMLRPCServer.SimpleXMLRPCServer( ('', 80) ) server.register_function(test) server.serve_forever() -- Mit freundlichen Gr??en, Best Regards, Robert H?lzl BALTECH AG Firmensitz: Lilienthalstrasse 27, D-85399 Hallbergmoos Registergericht: Amtsgericht M?nchen, HRB 115215 Vorstand: J?rgen R?sch (Vorsitzender), Martina M. Schuster Aufsichtsratsvorsitzende: Eva Zeising -------------- next part -------------- A non-text attachment was scrubbed... Name: robert_hoelzl.vcf Type: text/x-vcard Size: 569 bytes Desc: not available URL: From guzarva at windowslive.com Thu Apr 17 23:22:17 2008 From: guzarva at windowslive.com (guzarva16) Date: Thu, 17 Apr 2008 21:22:17 -0000 Subject: [Python-Dev] Security Advisory for unicode repr() bug? Message-ID: An HTML attachment was scrubbed... URL: From alberto.casado.martin at gmail.com Tue Apr 22 09:43:02 2008 From: alberto.casado.martin at gmail.com (=?ISO-8859-1?Q?Alberto_Casado_Mart=EDn?=) Date: Tue, 22 Apr 2008 09:43:02 +0200 Subject: [Python-Dev] python hangs when parsing a bad-formed email Message-ID: Hi all, First of all, sorry if this isn't the list where I have to post this. And sorry for my english. As the subject says, I'm having problems with the attached email, when I try to get a email object reading the attached file, the python process gets hang and gets all cpu. I have debuged my code to find where it happens, and I found that is _parsegen method of the FeedParser class. I know that the email format is wrong but I don't know why python hangs. following paste the code showing where hangs. def _parsegen(self): # Create a new message and start by parsing headers. self._new_message() headers = [] # Collect the headers, searching for a line that doesn't match the RFC # 2822 header or continuation pattern (including an empty line). for line in self._input: if line is NeedMoreData: yield NeedMoreData continue if not headerRE.match(line): # If we saw the RFC defined header/body separator # (i.e. newline), just throw it away. Otherwise the line is # part of the body so push it back. if not NLCRE.match(line): self._input.unreadline(line) break headers.append(line) # Done with the headers, so parse them and figure out what we're # supposed to see in the body of the message. self._parse_headers(headers) # Headers-only parsing is a backwards compatibility hack, which was # necessary in the older parser, which could throw errors. All # remaining lines in the input are thrown into the message body. if self._headersonly: lines = [] while True: line = self._input.readline() if line is NeedMoreData: yield NeedMoreData continue if line == '': break lines.append(line) self._cur.set_payload(EMPTYSTRING.join(lines)) return if self._cur.get_content_type() == 'message/delivery-status': !!!!!! AT THIS POINT HANGS, AND STRAT TO GET ALL CPU FOR THE PROCESS # message/delivery-status contains blocks of headers separated by # a blank line. We'll represent each header block as a separate # nested message object, but the processing is a bit different # than standard message/* types because there is no body for the # nested messages. A blank line separates the subparts. ... ... ... I have workaround the problem adding this line in _parse_headers method def _parse_headers(self, lines): # Passed a list of lines that make up the headers for the current msg lastheader = '' lastvalue = [] for lineno, line in enumerate(lines): # Check for continuation if line[0] in ' \t': if not lastheader: # The first line of the headers was a continuation. This # is illegal, so let's note the defect, store the illegal # line, and ignore it for purposes of headers. defect = errors.FirstHeaderLineIsContinuationDefect(line) self._cur.defects.append(defect) continue if line.strip()!='': !!!!!!! IF THE CONTINUATION LINE IS NOT EMPTY ADD THE LINE TO THE HEADER. lastvalue.append(line) continue if lastheader: ... ... ... I don't know why it hangs and I'm not sure why with this line works...... I have tried to parse this email in python 2.3.3 SunOs, python 2.3.3 gcc python 2.5.1 SunOs,gcc, Windows Xp, and linux SUSE 10. And I have alway the same result. bash-3.00$ python Python 2.5.1 (r251:54863, Feb 28 2008, 07:48:25) [GCC 3.4.6] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> import email >>> fp = open('raro.txt') >>> mail = email.message_from_file(fp) never return............ I don't know if someone can tell me what is happening.... Best Regards. Alberto Casado. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: raro.txt URL: From janssen at parc.com Tue Apr 22 23:42:28 2008 From: janssen at parc.com (Bill Janssen) Date: Tue, 22 Apr 2008 14:42:28 PDT Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480E2918.7090006@v.loewis.de> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <91478133-664C-4F29-90CB-D0985BE01F24@cs.toronto.edu> <480D69E9.7020004@v.loewis.de> <091D7293-363C-4766-8931-CFE6A9E72331@cs.toronto.edu> <480E2918.7090006@v.loewis.de> Message-ID: <08Apr22.144234pdt."58696"@synergy1.parc.xerox.com> > > When a web browser POSTs data, there is no standard way of communicating > > which encoding it's using. > > That's just not true. Web browser should and do use the encoding of the > web page that originally contained the form. Since the site that receives the POST doesn't necessarily have access to the Web page that originally contained the form, that's not really helpful. However, POSTs can use the MIME type "multipart/form-data" for non-Latin-1 content, and should. That contains facilities for indicating the encoding and other things as well. Bill From musiccomposition at gmail.com Wed Apr 23 00:01:31 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Tue, 22 Apr 2008 17:01:31 -0500 Subject: [Python-Dev] PyArg_ParseTuple and Py_BuildValue question In-Reply-To: <7a01f6c00804091823x74e29f75jaf4dd3ad53481b1a@mail.gmail.com> References: <7a01f6c00804091823x74e29f75jaf4dd3ad53481b1a@mail.gmail.com> Message-ID: <1afaf6160804221501j1ca2f634kef0ab356bb2a25aa@mail.gmail.com> On Wed, Apr 9, 2008 at 8:23 PM, Alvin Delagon wrote: > > I'm going to construct an SS7 packet in python using struct.pack(). Here's > the question, how am I going to pass the packet I wrote in python to my > module and back? I already asked this question in comp.lang.python but so > far no responses yet. I hope anyone can point me to the right direction. > Thanks in advance. What exactly is your problem? -- Cheers, Benjamin Peterson From musiccomposition at gmail.com Wed Apr 23 00:04:57 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Tue, 22 Apr 2008 17:04:57 -0500 Subject: [Python-Dev] SetType=set in types module ? In-Reply-To: <78e18a710804160608j68a07056y63fcd3a0ebab08ca@mail.gmail.com> References: <78e18a710804160608j68a07056y63fcd3a0ebab08ca@mail.gmail.com> Message-ID: <1afaf6160804221504j6a83c84byc58329d66950fc96@mail.gmail.com> On Wed, Apr 16, 2008 at 8:08 AM, iks hefem wrote: > Hi, > > the SetType is not available in the "types" module, so wouldn't it be > needed here ? (in 2.6 by example) Nothing new is currently being added to the types module because we are trying to decide whether to remove it or not. Please file a bug report, though, to remind us if we decide to keep it. -- Cheers, Benjamin Peterson From janssen at parc.com Wed Apr 23 00:33:44 2008 From: janssen at parc.com (Bill Janssen) Date: Tue, 22 Apr 2008 15:33:44 PDT Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480E59DC.3010007@egenix.com> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <480DBE86.2000507@egenix.com> <08Apr22.081453pdt."58696"@synergy1.parc.xerox.com> <08Apr22.093328pdt."58696"@synergy1.parc.xerox.com> <480E59DC.3010007@egenix.com> Message-ID: <08Apr22.153354pdt."58696"@synergy1.parc.xerox.com> > Unless you're using a very broad scope, I don't think that > you'd need more than a few hundred LSEs for a typical > application - nothing you'd want to put in the Python stdlib, > though. I tend to agree with this (and I'm generally in favor of putting everything in the standard library!). For those of us doing document-processing applications (Martin, it's not just about Web browsers), this would be a very useful package to have up on PyPI. Bill From mike.klaas at gmail.com Wed Apr 23 01:10:17 2008 From: mike.klaas at gmail.com (Mike Klaas) Date: Tue, 22 Apr 2008 16:10:17 -0700 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480E5592.3050102@v.loewis.de> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <480DBE86.2000507@egenix.com> <1A9B6931-5D55-4442-BB50-0B888AA98086@gmail.com> <480E5592.3050102@v.loewis.de> Message-ID: On 22-Apr-08, at 2:16 PM, Martin v. L?wis wrote: > >> Any program that needs to examine the contents of >> documents/feeds/whatever on the web needs to deal with >> incorrectly-specified encodings > > That's not true. Most programs that need to examine the contents of > a web page don't need to guess the encoding. In most such programs, > the encoding can be hard-coded if the declared encoding is not > correct. Most such programs *know* what page they are webscraping, > or else they couldn't extract the information out of it that they > want to get at. I certainly agree that if the target set of documents is small enough it is possible to hand-code the encoding. There are many applications, however, that need to examine the content of an arbitrary, or at least non-small set of web documents. To name a few such applications: - web search engines - translation software - document/bookmark management systems - other kinds of document analysis (market research, seo, etc.) > As for feeds - can you give examples of incorrectly encoded one > (I don't ever use feeds, so I honestly don't know whether they > are typically encoded incorrectly. I've heard they are often XML, > in which case I strongly doubt they are incorrectly encoded) I also don't have much experience with feeds. My statement is based on the fact that chardet, the tool that has been cited most in this thread, was written specifically for use with the author's feed parsing package. > As for "whatever" - can you give specific examples? Not that I can substantiate. Documents & feeds covers a lot of what is on the web--I was only trying to make the point that on the web, whenever an encoding can be specified, it will be specified incorrectly for a significant chunk of exemplars. >> (which, sadly, is rather common). The >> set of programs of programs that need this functionality is >> probably the >> same set that needs BeautifulSoup--I think that set is larger than >> just >> browsers > > Again, can you give *specific* examples that are not web browsers? > Programs needing BeautifulSoup may still not need encoding guessing, > since they still might be able to hard-code the encoding of the web > page they want to process. Indeed, if it is only one site it is pretty easy to work around. My main use of python is processing and analyzing hundreds of millions of web documents, so it is pretty easy to see applications (which I have listed above). I think that libraries like Mark Pilgrim's FeedParser and BeautifulSoup are possible consumers of guessing as well. > In any case, I'm very skeptical that a general "guess encoding" > module would do a meaningful thing when applied to incorrectly > encoded HTML pages. Well, it does. I wish I could easily provide data on how often it is necessary over the whole web, but that would be somewhat difficult to generate. I can say that it is much more important to be able to parse all the different kinds of encoding _specification_ on the web (Content-Type/Content-Encoding/ References: <480721A9.1070603@baltech.de> Message-ID: Hi, This is not a python-specific problem. See http://en.wikipedia.org/wiki/Nagle's_algorithm -Mike On 17-Apr-08, at 3:08 AM, Robert H?lzl wrote: > hello, > > I tried to implement a simple python XMLRPC service on a win32 > environment (client/server code inserted below). > The profiler of the client told me, that a simple function call > needs about 200ms (even if I run it in a loop, the time needed per > call stays the same). > > After analysing the problem with etherreal I found out, that the > XMLRPC request is transmitted via two TCP packets. One containing > the HTTP header and one containting the data. But the acknowledge to > the first TCP packet is delayed by 200ms. > > I tried around on the server side and found out that if the server > reads exactly all bytes transfered in the first TCP frame (via > socket.recv()), the next socket.recv(), even if reading only one > byte, needs about 200 ms. But if I read one byte less than > transfered in the first TCP frame and then reading 2 bytes > (socket.recv(2)) there is no delay, although the same total amount > of data was read. > > After some googling I found the website http://support.microsoft.com/?scid=kb%3Ben-us%3B823764&x=12&y=15 > , which proposed a workaround (modifing the registryentry for the > tcp/ip driver) that did work. But modifing the clients registry > settings is no option for us. > > Is there anybody who nows how to solve the problem? Or is it even a > problem if the python socket implementation? > > By the way: I testet Win2000 SP4 and WinXP SP2 with Python 2.3.3 and > Python 2.5.1 each. > > CLIENT: > ---------- > import xmlrpclib > import profile > server = xmlrpclib.ServerProxy("http://server:80") > profile.run('server.test(1,2)') > > SERVER: > ---------- > import SimpleXMLRPCServer > def test(a,b): return a+b > server = SimpleXMLRPCServer.SimpleXMLRPCServer( ('', 80) ) > server.register_function(test) > server.serve_forever() > > -- > Mit freundlichen Gr??en, > Best Regards, > > Robert H?lzl > BALTECH AG > > Firmensitz: Lilienthalstrasse 27, D-85399 Hallbergmoos > Registergericht: Amtsgericht M?nchen, HRB 115215 > Vorstand: J?rgen R?sch (Vorsitzender), Martina M. Schuster > Aufsichtsratsvorsitzende: Eva Zeising > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/mike.klaas%40gmail.com From amauryfa at gmail.com Wed Apr 23 01:18:30 2008 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 23 Apr 2008 01:18:30 +0200 Subject: [Python-Dev] python hangs when parsing a bad-formed email In-Reply-To: References: Message-ID: Hello, Alberto Casado Mart?n wrote: > Hi all, > First of all, sorry if this isn't the list where I have to post this. > And sorry for my english. > > As the subject says, I'm having problems with the attached email, when > I try to get a email object reading the attached file, the python > process gets hang and gets all cpu. > > I have debuged my code to find where it happens, and I found that is > _parsegen method of the FeedParser class. I know that the email format > is wrong but I don't know why python hangs. > > following paste the code showing where hangs. [snip] > bash-3.00$ python > Python 2.5.1 (r251:54863, Feb 28 2008, 07:48:25) > [GCC 3.4.6] on sunos5 > Type "help", "copyright", "credits" or "license" for more information. > >>> import email > >>> fp = open('raro.txt') > >>> mail = email.message_from_file(fp) > never return............ When you think you found a problem with python, please submit an issue in the python issue tracker: http://bugs.python.org/ In your case, I suspect some regular expression trying to match all the empty lines of the message, one character at a time. -- Amaury Forgeot d'Arc From rbp at isnomore.net Wed Apr 23 01:35:22 2008 From: rbp at isnomore.net (Rodrigo Bernardo Pimentel) Date: Tue, 22 Apr 2008 20:35:22 -0300 Subject: [Python-Dev] GSoC student introduction and sandbox commit privileges request Message-ID: <20080422233518.GI16282@isnomore.net> Hi there, I've just been accepted into this year's Google Summer of Code, to work for the Python Software Foundation on 2to3. My project is to give 2to3 fixers the ability to rank how confident they are on each fix, and let users choose to intervene manually whenever that confidence level is below a certain threshold. Among other things, this might allow fixers for situations where the code translation is not always guaranteed to be correct (like % string formatting, which came up recently in another thread). The full proposal is at http://isnomore.net/2to3 . Collin Winter will be my mentor, and I'd like to thank him and Christian Heimes for all the help they gave me in designing the project. I'd also like to thank Martin L?wis, for discussing a project with me which ended up not turning into a proposal, but helped me write the 2to3 one. Finally, I'd like to request commit privileges to work on a sandbox branch, during the Summer of Code. If you have any further questions, please feel free to contact me. I'm really looking forward to working on this project! Cheers, rbp -- Rodrigo Bernardo Pimentel | GPG: <0x0DB14978> From tjreedy at udel.edu Wed Apr 23 01:35:29 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 22 Apr 2008 19:35:29 -0400 Subject: [Python-Dev] python hangs when parsing a bad-formed email References: Message-ID: "Amaury Forgeot d'Arc" wrote in message news:e27efe130804221618lf4efc93p685272eeca10796c at mail.gmail.com... | When you think you found a problem with python, please submit an issue | in the python issue tracker: | http://bugs.python.org/ Or post to comp.lang.python / python mailing list / gmane.comp.python.general From janssen at parc.com Wed Apr 23 01:50:40 2008 From: janssen at parc.com (Bill Janssen) Date: Tue, 22 Apr 2008 16:50:40 PDT Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <91478133-664C-4F29-90CB-D0985BE01F24@cs.toronto.edu> <480D69E9.7020004@v.loewis.de> <091D7293-363C-4766-8931-CFE6A9E72331@cs.toronto.edu> <480E2918.7090006@v.loewis.de> <08Apr22.144234pdt."58696"@synergy1.parc.xerox.com> Message-ID: <08Apr22.165049pdt."58696"@synergy1.parc.xerox.com> > Yup, but DrProject (the target application) also serves as a relay and > archive for email. We have no control over the agent used for > composition, and AFAIK there's no standard way to include encoding > information. Greg, Internet-compliant email actually has well-specified mechanisms for including encoding information; see RFCs 2047 and 2231. There's no need to guess; you can just look. Bill From brett at python.org Wed Apr 23 02:02:49 2008 From: brett at python.org (Brett Cannon) Date: Tue, 22 Apr 2008 17:02:49 -0700 Subject: [Python-Dev] GSoC student introduction and sandbox commit privileges request In-Reply-To: <20080422233518.GI16282@isnomore.net> References: <20080422233518.GI16282@isnomore.net> Message-ID: On Tue, Apr 22, 2008 at 4:35 PM, Rodrigo Bernardo Pimentel wrote: > Hi there, > > I've just been accepted into this year's Google Summer of Code, to work for > the Python Software Foundation on 2to3. My project is to give 2to3 fixers > the ability to rank how confident they are on each fix, and let users choose > to intervene manually whenever that confidence level is below a certain > threshold. Among other things, this might allow fixers for situations where > the code translation is not always guaranteed to be correct (like % string > formatting, which came up recently in another thread). The full proposal is > at http://isnomore.net/2to3 . > > Collin Winter will be my mentor, and I'd like to thank him and Christian > Heimes for all the help they gave me in designing the project. I'd also like > to thank Martin L?wis, for discussing a project with me which ended up not > turning into a proposal, but helped me write the 2to3 one. > > Finally, I'd like to request commit privileges to work on a sandbox branch, > during the Summer of Code. > Isn't this a chance for bzr to shine? With lib2to3 in the 3.0 bzr branch, can't Rodrigo and the other students who don't have some funky requirement just use bzr? > If you have any further questions, please feel free to contact me. I'm > really looking forward to working on this project! Thanks for contributing! -Brett From rbp at isnomore.net Wed Apr 23 02:18:05 2008 From: rbp at isnomore.net (Rodrigo Bernardo Pimentel) Date: Tue, 22 Apr 2008 21:18:05 -0300 Subject: [Python-Dev] GSoC student introduction and sandbox commit privileges request In-Reply-To: References: <20080422233518.GI16282@isnomore.net> Message-ID: <20080423001804.GK16282@isnomore.net> On Tue, Apr 22 2008 at 09:02:49PM BRT, Brett Cannon wrote: > On Tue, Apr 22, 2008 at 4:35 PM, Rodrigo Bernardo Pimentel > wrote: > > I've just been accepted into this year's Google Summer of Code (...) > > Finally, I'd like to request commit privileges to work on a sandbox > > branch, during the Summer of Code. > Isn't this a chance for bzr to shine? With lib2to3 in the 3.0 bzr > branch, can't Rodrigo and the other students who don't have some funky > requirement just use bzr? FWIW, +1 from me, I'm perfectly comfortable with bzr. > > If you have any further questions, please feel free to contact me. I'm > > really looking forward to working on this project! > > Thanks for contributing! My pleasure :) Cheers, rbp -- Rodrigo Bernardo Pimentel | GPG: <0x0DB14978> From lists at cheimes.de Wed Apr 23 02:35:31 2008 From: lists at cheimes.de (Christian Heimes) Date: Wed, 23 Apr 2008 02:35:31 +0200 Subject: [Python-Dev] SetType=set in types module ? In-Reply-To: <1afaf6160804221504j6a83c84byc58329d66950fc96@mail.gmail.com> References: <78e18a710804160608j68a07056y63fcd3a0ebab08ca@mail.gmail.com> <1afaf6160804221504j6a83c84byc58329d66950fc96@mail.gmail.com> Message-ID: <480E8453.2000005@cheimes.de> Benjamin Peterson schrieb: > On Wed, Apr 16, 2008 at 8:08 AM, iks hefem wrote: >> Hi, >> >> the SetType is not available in the "types" module, so wouldn't it be >> needed here ? (in 2.6 by example) > > Nothing new is currently being added to the types module because we > are trying to decide whether to remove it or not. Please file a bug > report, though, to remind us if we decide to keep it. Eventually the types module will go away or at least be stripped down in Python 3.0. New types like the set type weren't added to types deliberately. Please don't file a bug report. Christian From lists at cheimes.de Wed Apr 23 02:38:58 2008 From: lists at cheimes.de (Christian Heimes) Date: Wed, 23 Apr 2008 02:38:58 +0200 Subject: [Python-Dev] PyArg_ParseTuple and Py_BuildValue question In-Reply-To: <7a01f6c00804091823x74e29f75jaf4dd3ad53481b1a@mail.gmail.com> References: <7a01f6c00804091823x74e29f75jaf4dd3ad53481b1a@mail.gmail.com> Message-ID: <480E8522.9090008@cheimes.de> Alvin Delagon schrieb: > I'm going to construct an SS7 packet in python using struct.pack(). Here's > the question, how am I going to pass the packet I wrote in python to my > module and back? I already asked this question in comp.lang.python but so > far no responses yet. I hope anyone can point me to the right direction. > Thanks in advance. The Python developer list is meant for the development OF Python, not WITH Python. Please use the general Python user list to get help. Christian From nedds at uchicago.edu Wed Apr 23 02:42:32 2008 From: nedds at uchicago.edu (Nick Edds) Date: Tue, 22 Apr 2008 19:42:32 -0500 Subject: [Python-Dev] GSoC Student Introduction Message-ID: <1aff528e0804221742p2e6e9b21jfa0f7606fddf611e@mail.gmail.com> Hello, My name is Nick Edds. I am going to be working on the 2to3 tool with Collin Winter as my mentor. More specifically, I will be working on improving the performance of the 2to3 tool in general, and its use of patterns in particular. I would like to request commit privileges to work in a sandbox branch and although I don't have any familiarity with bzr, I would be comfortable using it. Regards, Nick -------------- next part -------------- An HTML attachment was scrubbed... URL: From collinw at gmail.com Wed Apr 23 02:56:00 2008 From: collinw at gmail.com (Collin Winter) Date: Tue, 22 Apr 2008 17:56:00 -0700 Subject: [Python-Dev] GSoC student introduction and sandbox commit privileges request In-Reply-To: <20080423001804.GK16282@isnomore.net> References: <20080422233518.GI16282@isnomore.net> <20080423001804.GK16282@isnomore.net> Message-ID: <43aa6ff70804221756k57626854x78bf2c45ccea3653@mail.gmail.com> On Tue, Apr 22, 2008 at 5:18 PM, Rodrigo Bernardo Pimentel wrote: > On Tue, Apr 22 2008 at 09:02:49PM BRT, Brett Cannon wrote: > > On Tue, Apr 22, 2008 at 4:35 PM, Rodrigo Bernardo Pimentel > > wrote: > > > > > I've just been accepted into this year's Google Summer of Code > (...) > > > > Finally, I'd like to request commit privileges to work on a sandbox > > > branch, during the Summer of Code. > > > Isn't this a chance for bzr to shine? With lib2to3 in the 3.0 bzr > > branch, can't Rodrigo and the other students who don't have some funky > > requirement just use bzr? > > FWIW, +1 from me, I'm perfectly comfortable with bzr. Fine by me; I don't care one way or the other. Collin > > > If you have any further questions, please feel free to contact me. I'm > > > really looking forward to working on this project! > > > > Thanks for contributing! > > My pleasure :) > > Cheers, > > > rbp > -- > Rodrigo Bernardo Pimentel | GPG: <0x0DB14978> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/collinw%40gmail.com > From musiccomposition at gmail.com Wed Apr 23 04:38:16 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Tue, 22 Apr 2008 21:38:16 -0500 Subject: [Python-Dev] GSoC Student Introduction In-Reply-To: <1aff528e0804221742p2e6e9b21jfa0f7606fddf611e@mail.gmail.com> References: <1aff528e0804221742p2e6e9b21jfa0f7606fddf611e@mail.gmail.com> Message-ID: <1afaf6160804221938j1a6672cdv938fc373e00068f@mail.gmail.com> On Tue, Apr 22, 2008 at 7:42 PM, Nick Edds wrote: > Hello, > > My name is Nick Edds. I am going to be working on the 2to3 tool with Collin > Winter as my mentor. More specifically, I will be working on improving the > performance of the 2to3 tool in general, and its use of patterns in > particular. > I would like to request commit privileges to work in a sandbox branch and > although I don't have any familiarity with bzr, I would be comfortable using > it. Luckily, Bazaar is really easy. Thanks for contributing! -- Cheers, Benjamin Peterson From jcea at argo.es Wed Apr 23 05:09:49 2008 From: jcea at argo.es (Jesus Cea) Date: Wed, 23 Apr 2008 05:09:49 +0200 Subject: [Python-Dev] BSDDB3 In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E243C7CA1@EXMBX04.exchhosting.com> References: <47C32D2D.1030206@free.fr> <47C4813F.2080403@v.loewis.de> <52dc1c820802262146n7b7c211dk973eb44390a4b8f7@mail.gmail.com> <20080227141322.GA5871@amk-desktop.matrixgroup.net> <47CDDF57.1020006@argo.es> <47CF0B7C.109@v.loewis.de> <480CABFB.2080302@argo.es> <87D3F9C72FBF214DB39FA4E3FE618CDC6E243C7CA1@EXMBX04.exchhosting.com> Message-ID: <480EA87D.5040107@argo.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Trent Nelson wrote: | I remember those rampant BSDDB crashes on Windows well. [...] | basically, the tests weren't cleaning up their environment in | the right order, so BSDDB was getting passed completely and | utterly bogus values. Next week I will (if nothing goes wrong) publish pybsddb 4.6.4. This release supports distributed transactions and replication, testsuite is way faster, and rewritten to be able to launch tests from multiple threads/processes if you wish, setuptools/pypi support, etc. I think this release would be appropiate to integrate in Python. I think most demands are solved and new features are interesting (replication, distributed transactions, do not crash when closing objects in the wrong order...). Also, I completed the documentation, with the full supported API, and ported it to Python 2.6 documentation system. The result: http://www.jcea.es/programacion/pybsddb.htm#bsddb3-4.6.4 http://www.jcea.es/programacion/pybsddb_doc/preview/ I'm very interested in integrating this release in Python 2.6 for the new features, the full documentation, and to get feedback from Buildbot and python-dev community. Also, I would like to avoid to integrate pybsddb late in the python 2.6 release cycle; I hope to be away of my computer in August! :). I'm a bit nervous about syncing, because I have the feeling that python-dev is committing changes to python private branch of pybsddb. I would rather prefer patches send to me and integrate "canonical" pybsddb releases in Python frequently. Somebody suggested to post patches in the tracker, but I think this is not going to work. The diff from current python bsddb and the official version is so huge that nobody could follow it. A more sensible approach, I think, is to "diff" current python pybsddb against the version I used as my root (January?), integrate the changes in current "canonical" pybsddb and, then, drop the entire updated package into python. Then, commits to python pybsddb should be avoided; patches should be send to me. I think this is the only way when integrating a project outside python SVN. Suggestions?. PS: I can't comment on Win64. It is an alien world to me :). - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ ~ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iQCVAwUBSA6oeJlgi5GaxT1NAQItswP+KR15vZWbnYZ23WQHoUozVOWvf+ghG2Q8 acVhCwJajzvxOEfozRMZRmQkPUBmWga1zbHjkHt5c196vku7+X0bDc7aO4T2jRHx 00PbPLGnYth972elTVFfSWpZVNkX/9A4EbtTHVCav105nW+u1/Kod/rY5fzgKcTn SxYkmk4Ax7U= =98uc -----END PGP SIGNATURE----- From brett at python.org Wed Apr 23 05:45:22 2008 From: brett at python.org (Brett Cannon) Date: Tue, 22 Apr 2008 20:45:22 -0700 Subject: [Python-Dev] GSoC Student Introduction In-Reply-To: <1afaf6160804221938j1a6672cdv938fc373e00068f@mail.gmail.com> References: <1aff528e0804221742p2e6e9b21jfa0f7606fddf611e@mail.gmail.com> <1afaf6160804221938j1a6672cdv938fc373e00068f@mail.gmail.com> Message-ID: On Tue, Apr 22, 2008 at 7:38 PM, Benjamin Peterson wrote: > > On Tue, Apr 22, 2008 at 7:42 PM, Nick Edds wrote: > > Hello, > > > > My name is Nick Edds. I am going to be working on the 2to3 tool with Collin > > Winter as my mentor. More specifically, I will be working on improving the > > performance of the 2to3 tool in general, and its use of patterns in > > particular. > > > I would like to request commit privileges to work in a sandbox branch and > > although I don't have any familiarity with bzr, I would be comfortable using > > it. > > Luckily, Bazaar is really easy. > See http://python.org/dev/bazaar/ for info. And if you have any other issues feel free to ask, Nick. -Brett From stephen at xemacs.org Wed Apr 23 06:44:20 2008 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 23 Apr 2008 13:44:20 +0900 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <08Apr22.165049pdt."58696"@synergy1.parc.xerox.com> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <91478133-664C-4F29-90CB-D0985BE01F24@cs.toronto.edu> <480D69E9.7020004@v.loewis.de> <091D7293-363C-4766-8931-CFE6A9E72331@cs.toronto.edu> <480E2918.7090006@v.loewis.de> <08Apr22.144234pdt."58696"@synergy1.parc.xerox.com> <08Apr22.165049pdt."58696"@synergy1.parc.xerox.com> Message-ID: <87ej8xp4d7.fsf@uwakimon.sk.tsukuba.ac.jp> Bill Janssen writes: > Internet-compliant email actually has well-specified mechanisms for > including encoding information; see RFCs 2047 and 2231. There's no > need to guess; you can just look. You must be very special to get only compliant email. About half my colleagues use RFC 2047 to encode Japanese file names in MIME attachments (a MUST NOT behavior according to RFC 2047), and a significant fraction of the rest end up with binary Shift JIS or EUC or MacRoman in there. And those are just the most widespread violations I can think of off the top of my head. Not to mention that I find this: =?X-UNKNOWN?Q?Martin_v=2E_L=F6wis?= , in the header I got from you. (I'm not ragging on you, I get Martin's name wrong a significant portion of the time myself. :-( ) From stephen at xemacs.org Wed Apr 23 06:56:47 2008 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 23 Apr 2008 13:56:47 +0900 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480E5592.3050102@v.loewis.de> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <480DBE86.2000507@egenix.com> <1A9B6931-5D55-4442-BB50-0B888AA98086@gmail.com> <480E5592.3050102@v.loewis.de> Message-ID: <87d4ohp3sg.fsf@uwakimon.sk.tsukuba.ac.jp> "Martin v. L?wis" writes: > In any case, I'm very skeptical that a general "guess encoding" > module would do a meaningful thing when applied to incorrectly > encoded HTML pages. That depends on whether you can get meaningful information about the language from the fact that you're looking at the page. In the browser context, for one, 99.44% of users are monolingual, so you only have to distinguish among the encodings for their language. In this context a two stage process of determining a category of encoding (eg, ISO 8859, ISO 2022 7-bit, ISO 2022 8-bit multibyte, UTF-8, etc), and then picking an encoding from the category according to a user-specified configuration has served Emacs/MULE users very well for about 20 years. It does *not* work in a context where multiple encodings from the same category are in use (eg, the email folder of a Polish Gastarbeiter in Berlin). Nonetheless it is pretty useful for user agents like mail clients, web browsers, and editors. From stephen at xemacs.org Wed Apr 23 06:59:50 2008 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 23 Apr 2008 13:59:50 +0900 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <480CC6D7.2020404@cheimes.de> Message-ID: <87bq41p3nd.fsf@uwakimon.sk.tsukuba.ac.jp> Guido van Rossum writes: > To the contrary, an encoding-guessing module is often needed, and > guessing can be done with a pretty high success rate. Other Unicode > libraries (e.g. ICU) contain guessing modules. I suppose the API could > return two values: the guessed encoding and a confidence indicator. > Note that the locale settings might figure in the guess. Not locale settings, but user configuration. A Bayesian detector (CodeBayes? hi, Skip!) might be a good way to go for servers, while a simple language preference might really up the probability for user agents. From martin at v.loewis.de Wed Apr 23 06:52:06 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Apr 2008 06:52:06 +0200 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <91478133-664C-4F29-90CB-D0985BE01F24@cs.toronto.edu> <480D69E9.7020004@v.loewis.de> <091D7293-363C-4766-8931-CFE6A9E72331@cs.toronto.edu> <480E2918.7090006@v.loewis.de> <08Apr22.144234pdt."58696"@synergy1.parc.xerox.com> Message-ID: <480EC076.4090207@v.loewis.de> > Yup, but DrProject (the target application) also serves as a relay and > archive for email. We have no control over the agent used for > composition, and AFAIK there's no standard way to include encoding > information. That's not at all the case. MIME defines that in full detail, since 1993. Regards, Martin From martin at v.loewis.de Wed Apr 23 07:04:54 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Apr 2008 07:04:54 +0200 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <480DBE86.2000507@egenix.com> <1A9B6931-5D55-4442-BB50-0B888AA98086@gmail.com> <480E5592.3050102@v.loewis.de> Message-ID: <480EC376.8070406@v.loewis.de> > I certainly agree that if the target set of documents is small enough it > is possible to hand-code the encoding. There are many applications, > however, that need to examine the content of an arbitrary, or at least > non-small set of web documents. To name a few such applications: > > - web search engines > - translation software I'll question whether these are "many" programs. Web search engines and translation software have many more challenges to master, and they are fairly special-cased, so I would expect they need to find their own answer to character set detection, anyway (see Bill Janssen's answer on machine translation, also). > - document/bookmark management systems > - other kinds of document analysis (market research, seo, etc.) Not sure what specifically you have in mind, however, I expect that these also have their own challenges. For example, I would expect that MS-Word documents are frequent. You don't need character set detection there (Word is all Unicode), but you need an API to look into the structure of .doc files. > Not that I can substantiate. Documents & feeds covers a lot of what is > on the web--I was only trying to make the point that on the web, > whenever an encoding can be specified, it will be specified incorrectly > for a significant chunk of exemplars. I firmly believe this assumption is false. If the encoding comes out of software (which it often does), it will be correct most of the time. It's incorrect only if the content editor has to type it. > Indeed, if it is only one site it is pretty easy to work around. My > main use of python is processing and analyzing hundreds of millions of > web documents, so it is pretty easy to see applications (which I have > listed above). Ok. What advantage would you (or somebody working on a similar project) gain if chardet was part of the standard library? What if it was not chardet, but some other algorithm? > I can also think of good arguments for excluding encoding detection for > maintenance reasons: is every case of the algorithm guessing wrong a bug > that needs to be fixed in the stdlib? That is an unbounded commitment. Indeed, that's what I meant with my initial remark. People will expect that it works correctly - both with the consequence of unknowingly proceeding with the incorrect response, and then complaining when they find out that it did produce an incorrect answer. For chardet specifically, my usual standard-library remark applies: it can't become part of the standard library unless the original author contributes it, anyway. I would then hope that he or a group of people would volunteer to maintain it, with the threat of removing it from the stdlib again if these volunteers go away and too many problems show up. Regards, Martin From tjreedy at udel.edu Wed Apr 23 07:26:22 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 23 Apr 2008 01:26:22 -0400 Subject: [Python-Dev] Encoding detection in the standard library? References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de><480DBE86.2000507@egenix.com><1A9B6931-5D55-4442-BB50-0B888AA98086@gmail.com><480E5592.3050102@v.loewis.de> <480EC376.8070406@v.loewis.de> Message-ID: ""Martin v. L?wis"" wrote in message news:480EC376.8070406 at v.loewis.de... |> I certainly agree that if the target set of documents is small enough it | | Ok. What advantage would you (or somebody working on a similar project) | gain if chardet was part of the standard library? What if it was not | chardet, but some other algorithm? It seems to me that since there is not a 'correct' algorithm but only competing heuristics, encoding detection modules should be made available via PyPI and only be considered for stdlib after a best of breed emerges with community support. From martin at v.loewis.de Wed Apr 23 07:49:13 2008 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Apr 2008 07:49:13 +0200 Subject: [Python-Dev] socket recv on win32 can be extremly delayed (python bug?) In-Reply-To: <480721A9.1070603@baltech.de> References: <480721A9.1070603@baltech.de> Message-ID: <480ECDD9.3070101@v.loewis.de> > Is there anybody who nows how to solve the problem? If it's really the problem described in the MSKB article, the article also suggests a solution: use non-blocking sockets. Regards, Martin From martin at v.loewis.de Wed Apr 23 08:03:13 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Apr 2008 08:03:13 +0200 Subject: [Python-Dev] GSoC Student Introduction In-Reply-To: References: <1aff528e0804221742p2e6e9b21jfa0f7606fddf611e@mail.gmail.com> <1afaf6160804221938j1a6672cdv938fc373e00068f@mail.gmail.com> Message-ID: <480ED121.7090005@v.loewis.de> > See http://python.org/dev/bazaar/ for info. And if you have any other > issues feel free to ask, Nick. I certainly can't speak for the respective mentors, but I feel that "use bazaar" really isn't the right answer to "can I get commit access?" One motivation for GSoC is also community bonding, and having the mentor (but not *only* the mentor) comment on the proposed changes, and monitor the progress of the project. That the development branch sits on the student's laptop doesn't really help in that process. Instead, the student would have to push the branch somewhere to a web-visible location. Now I question whether it's the student's obligation to find a server himself, or whether the mentoring org should provide the infrastructure (or, failing that, Google (*)). So I think an answer to the question above involving bazaar might be "yes, but please don't commit to subversion, but only to the bazaar repository". Regards, Martin (*) FWIW, Google does provide the infrastructure; students are encouraged (required?) to commit their work to code.google.com. From martin at v.loewis.de Wed Apr 23 08:12:45 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Apr 2008 08:12:45 +0200 Subject: [Python-Dev] BSDDB3 In-Reply-To: <480EA87D.5040107@argo.es> References: <47C32D2D.1030206@free.fr> <47C4813F.2080403@v.loewis.de> <52dc1c820802262146n7b7c211dk973eb44390a4b8f7@mail.gmail.com> <20080227141322.GA5871@amk-desktop.matrixgroup.net> <47CDDF57.1020006@argo.es> <47CF0B7C.109@v.loewis.de> <480CABFB.2080302@argo.es> <87D3F9C72FBF214DB39FA4E3FE618CDC6E243C7CA1@EXMBX04.exchhosting.com> <480EA87D.5040107@argo.es> Message-ID: <480ED35D.1040403@v.loewis.de> > I'm a bit nervous about syncing, because I have the feeling that > python-dev is committing changes to python private branch of pybsddb. I > would rather prefer patches send to me and integrate "canonical" pybsddb > releases in Python frequently. My understanding was that the pybsddb copy in Python *is* the official code base. I think it's unfortunate that the pybsddb project was revived, even though it was already dead. Perhaps we should remove bsddb from Python again, and refer people to pybsddb instead? > Somebody suggested to post patches in the tracker, but I think this is > not going to work. The diff from current python bsddb and the official > version is so huge that nobody could follow it. A more sensible > approach, I think, is to "diff" current python pybsddb against the > version I used as my root (January?), integrate the changes in current > "canonical" pybsddb and, then, drop the entire updated package into python. -1. Who is going to do the 3k porting? > I think this is the only way when integrating a project outside python > SVN. Suggestions?. The usual solution is to not integrate then, at all. Python doesn't really ship with any libraries that also have an active life outside of Python. Regards, Martin From hrvoje.niksic at avl.com Wed Apr 23 09:00:57 2008 From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=) Date: Wed, 23 Apr 2008 09:00:57 +0200 Subject: [Python-Dev] PyArg_ParseTuple and Py_BuildValue question In-Reply-To: <7a01f6c00804091823x74e29f75jaf4dd3ad53481b1a@mail.gmail.com> References: <7a01f6c00804091823x74e29f75jaf4dd3ad53481b1a@mail.gmail.com> Message-ID: <1208934057.11332.1.camel@localhost> On Thu, 2008-04-10 at 09:23 +0800, Alvin Delagon wrote: > I'm currently writing a simple python SCTP module in C. So far it > works sending and receiving strings from it. [...] > I'm going to construct an SS7 packet in python using struct.pack(). > Here's the question, how am I going to pass the packet I wrote in > python to my module and back? I already asked this question in > comp.lang.python but so far no responses yet. Have you tried posting to the C API mailing list? http://mail.python.org/mailman/listinfo/capi-sig From p.f.moore at gmail.com Wed Apr 23 10:34:44 2008 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 23 Apr 2008 09:34:44 +0100 Subject: [Python-Dev] GSoC Student Introduction In-Reply-To: <480ED121.7090005@v.loewis.de> References: <1aff528e0804221742p2e6e9b21jfa0f7606fddf611e@mail.gmail.com> <1afaf6160804221938j1a6672cdv938fc373e00068f@mail.gmail.com> <480ED121.7090005@v.loewis.de> Message-ID: <79990c6b0804230134v7238238dh57ad85e3852c752a@mail.gmail.com> On 23/04/2008, "Martin v. L?wis" wrote: > > See http://python.org/dev/bazaar/ for info. And if you have any other > > issues feel free to ask, Nick. > > I certainly can't speak for the respective mentors, but I feel that > "use bazaar" really isn't the right answer to "can I get commit access?" [...] > So I think an answer to the question above involving bazaar might > be "yes, but please don't commit to subversion, but only to the > bazaar repository". Agreed. Can I also point out that Bazaar isn't an official repository for Python (yet). I don't want to open up the DVCS discussion again, but I have recently finished looking at both Bazaar and Mercurial for my personal use (including managing Python contributions) and decided on Mercurial. A significant part of the decision was the fact that in certain configurations, Bazaar is really, really slow. It feels a little like Bazaar is becoming the "de facto" DVCS solution for Python, and I'd like to avoid that happening until a proper evaluation has been done. Paul. PS If the fact that Bazaar repos are set up is an important factor here, I'd be willing to set up the equivalent Mercurial repos - although I would need access to the relevant boxes and possibly some admin assistance. From mal at egenix.com Wed Apr 23 12:01:18 2008 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 23 Apr 2008 12:01:18 +0200 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de><480DBE86.2000507@egenix.com><1A9B6931-5D55-4442-BB50-0B888AA98086@gmail.com><480E5592.3050102@v.loewis.de> <480EC376.8070406@v.loewis.de> Message-ID: <480F08EE.1050207@egenix.com> On 2008-04-23 07:26, Terry Reedy wrote: > ""Martin v. L?wis"" wrote in message > news:480EC376.8070406 at v.loewis.de... > |> I certainly agree that if the target set of documents is small enough it > | > | Ok. What advantage would you (or somebody working on a similar project) > | gain if chardet was part of the standard library? What if it was not > | chardet, but some other algorithm? > > It seems to me that since there is not a 'correct' algorithm but only > competing heuristics, encoding detection modules should be made available > via PyPI and only be considered for stdlib after a best of breed emerges > with community support. +1 Though in practice, determining the "best of breed" often becomes a problem (see e.g. the JSON implementation discussion). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Apr 23 2008) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 From ggpolo at gmail.com Wed Apr 23 12:24:12 2008 From: ggpolo at gmail.com (Guilherme Polo) Date: Wed, 23 Apr 2008 07:24:12 -0300 Subject: [Python-Dev] GSoC Student Introduction In-Reply-To: <480ED121.7090005@v.loewis.de> References: <1aff528e0804221742p2e6e9b21jfa0f7606fddf611e@mail.gmail.com> <1afaf6160804221938j1a6672cdv938fc373e00068f@mail.gmail.com> <480ED121.7090005@v.loewis.de> Message-ID: 2008/4/23, "Martin v. L?wis" : > > See http://python.org/dev/bazaar/ for info. And if you have any other > > issues feel free to ask, Nick. > > > I certainly can't speak for the respective mentors, but I feel that > "use bazaar" really isn't the right answer to "can I get commit access?" > > One motivation for GSoC is also community bonding, and having the > mentor (but not *only* the mentor) comment on the proposed changes, > and monitor the progress of the project. That the development branch > sits on the student's laptop doesn't really help in that process. > > Instead, the student would have to push the branch somewhere to > a web-visible location. Now I question whether it's the student's > obligation to find a server himself, or whether the mentoring org > should provide the infrastructure (or, failing that, Google (*)). > > So I think an answer to the question above involving bazaar might > be "yes, but please don't commit to subversion, but only to the > bazaar repository". Hi Martin, What is the point on having a branch in the svn repo if you won't be able to commit ? Maybe I misunderstood what you said, so maybe could you clarify that answer ? > > Regards, > Martin > > (*) FWIW, Google does provide the infrastructure; students are > encouraged (required?) to commit their work to code.google.com. It is required to submit the "final and complete" work done during GSoC to a project under code.google.com. That project will be automatically created after gsoc ends. But yeh, we could create another project there to use as a repo for the summer. And thank you, James Tauber, effbot and everyone else that accepted me as a student this year too. I guess I should create a new thread to introduce myself and my project =) Thanks, > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ggpolo%40gmail.com > -- -- Guilherme H. Polo Goncalves From ndbecker2 at gmail.com Wed Apr 23 12:48:03 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Wed, 23 Apr 2008 06:48:03 -0400 Subject: [Python-Dev] pydoc works with eggs? (python-2.5.1) References: Message-ID: Neal Becker wrote: > pydoc blew up when I tried to view doc for pytools module, which is an > egg: > > pydoc -p 8082 > pydoc server ready at http://localhost:8082/ > ---------------------------------------- > Exception happened during processing of request from ('127.0.0.1', 52915) > Traceback (most recent call last): > File "/usr/lib64/python2.5/SocketServer.py", line 222, in handle_request > self.process_request(request, client_address) > File "/usr/lib64/python2.5/SocketServer.py", line 241, in > process_request > self.finish_request(request, client_address) > File "/usr/lib64/python2.5/SocketServer.py", line 254, in finish_request > self.RequestHandlerClass(request, client_address, self) > File "/usr/lib64/python2.5/SocketServer.py", line 522, in __init__ > self.handle() > File "/usr/lib64/python2.5/BaseHTTPServer.py", line 316, in handle > self.handle_one_request() > File "/usr/lib64/python2.5/BaseHTTPServer.py", line 310, in > handle_one_request > method() > File "/usr/lib64/python2.5/pydoc.py", line 1924, in do_GET > self.send_document(describe(obj), html.document(obj, path)) > File "/usr/lib64/python2.5/pydoc.py", line 321, in document > if inspect.ismodule(object): return self.docmodule(*args) > File "/usr/lib64/python2.5/pydoc.py", line 672, in docmodule > contents.append(self.document(value, key, name, fdict, cdict)) > File "/usr/lib64/python2.5/pydoc.py", line 322, in document > if inspect.isclass(object): return self.docclass(*args) > File "/usr/lib64/python2.5/pydoc.py", line 807, in docclass > lambda t: t[1] == 'method') > File "/usr/lib64/python2.5/pydoc.py", line 735, in spill > funcs, classes, mdict, object)) > File "/usr/lib64/python2.5/pydoc.py", line 323, in document > if inspect.isroutine(object): return self.docroutine(*args) > File "/usr/lib64/python2.5/pydoc.py", line 891, in docroutine > getdoc(object), self.preformat, funcs, classes, methods) > File "/usr/lib64/python2.5/pydoc.py", line 79, in getdoc > result = inspect.getdoc(object) or inspect.getcomments(object) > File "/usr/lib64/python2.5/inspect.py", line 521, in getcomments > lines, lnum = findsource(object) > File "/usr/lib64/python2.5/inspect.py", line 510, in findsource > if pat.match(lines[lnum]): break > IndexError: list index out of range > ---------------------------------------- > I see that installing the egg unzipped fixes this. It looks to me that pydoc doesn't work with zipped eggs. From lists at cheimes.de Wed Apr 23 12:56:44 2008 From: lists at cheimes.de (Christian Heimes) Date: Wed, 23 Apr 2008 12:56:44 +0200 Subject: [Python-Dev] GSoC Student Introduction In-Reply-To: <480ED121.7090005@v.loewis.de> References: <1aff528e0804221742p2e6e9b21jfa0f7606fddf611e@mail.gmail.com> <1afaf6160804221938j1a6672cdv938fc373e00068f@mail.gmail.com> <480ED121.7090005@v.loewis.de> Message-ID: <480F15EC.10905@cheimes.de> Martin v. L?wis schrieb: > I certainly can't speak for the respective mentors, but I feel that > "use bazaar" really isn't the right answer to "can I get commit access?" > > One motivation for GSoC is also community bonding, and having the > mentor (but not *only* the mentor) comment on the proposed changes, > and monitor the progress of the project. That the development branch > sits on the student's laptop doesn't really help in that process. Students have to sign the contributor agreement anyway or we can't use their work. I don't see a problem with giving them svn commit privileges as long as they restrict themselves to their sandbox. > (*) FWIW, Google does provide the infrastructure; students are > encouraged (required?) to commit their work to code.google.com. People doesn't require the students to use their infrastructure. Google just wants three evaluations from the mentors (after the community bounding phase, mid term and at the end). Google also requires the student to send in a tarball with their work - patches, docs, tests and so on. The tarball is just required for legal reasons. Leslie once explained it's a social and legal hack to make the contract work and the lawyers happy. Christian From musiccomposition at gmail.com Wed Apr 23 13:55:34 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Wed, 23 Apr 2008 06:55:34 -0500 Subject: [Python-Dev] PyArg_ParseTuple and Py_BuildValue question In-Reply-To: <1208934057.11332.1.camel@localhost> References: <7a01f6c00804091823x74e29f75jaf4dd3ad53481b1a@mail.gmail.com> <1208934057.11332.1.camel@localhost> Message-ID: <1afaf6160804230455r5a9e484dg8da870a444fefd81@mail.gmail.com> He emailed me back saying that he found the solution to his problem. -- Cheers, Benjamin Peterson From ncoghlan at gmail.com Wed Apr 23 15:41:02 2008 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 23 Apr 2008 23:41:02 +1000 Subject: [Python-Dev] Externally developed modules (was Re: BSDDB3) In-Reply-To: <480ED35D.1040403@v.loewis.de> References: <47C32D2D.1030206@free.fr> <47C4813F.2080403@v.loewis.de> <52dc1c820802262146n7b7c211dk973eb44390a4b8f7@mail.gmail.com> <20080227141322.GA5871@amk-desktop.matrixgroup.net> <47CDDF57.1020006@argo.es> <47CF0B7C.109@v.loewis.de> <480CABFB.2080302@argo.es> <87D3F9C72FBF214DB39FA4E3FE618CDC6E243C7CA1@EXMBX04.exchhosting.com> <480EA87D.5040107@argo.es> <480ED35D.1040403@v.loewis.de> Message-ID: <480F3C6E.8040103@gmail.com> Martin v. L?wis wrote: >> I think this is the only way when integrating a project outside python >> SVN. Suggestions?. > > The usual solution is to not integrate then, at all. Python doesn't > really ship with any libraries that also have an active life outside > of Python. Not entirely true - we have the ones listed in PEP 360. However, I think of those only Optik (aka optparse) is currently seeing active external updates. Of course, we also have a big warning at the top of the PEP saying *Never Ever Do This Again*. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From pje at telecommunity.com Wed Apr 23 16:03:43 2008 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 23 Apr 2008 10:03:43 -0400 Subject: [Python-Dev] pydoc works with eggs? (python-2.5.1) In-Reply-To: References: Message-ID: <20080423140329.97F503A40AC@sparrow.telecommunity.com> At 06:48 AM 4/23/2008 -0400, Neal Becker wrote: >Neal Becker wrote: > > > pydoc blew up when I tried to view doc for pytools module, which is an > > egg: > > > > pydoc -p 8082 > > pydoc server ready at http://localhost:8082/ > > ---------------------------------------- >... > > >I see that installing the egg unzipped fixes this. It looks to me that >pydoc doesn't work with zipped eggs. What's odd about this is that it *did* at one time. Or at least help() did. The changes I made in 2.5 were mainly so that help(package) would work on zipped eggs. From the traceback, it looks like the issue is that it's trying to parse comments out of the source for an object with no docstring. I didn't know it could do that, so I never tried testing it. From janssen at parc.com Wed Apr 23 18:16:55 2008 From: janssen at parc.com (Bill Janssen) Date: Wed, 23 Apr 2008 09:16:55 PDT Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <480E2918.7090006@v.loewis.de> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <91478133-664C-4F29-90CB-D0985BE01F24@cs.toronto.edu> <480D69E9.7020004@v.loewis.de> <091D7293-363C-4766-8931-CFE6A9E72331@cs.toronto.edu> <480E2918.7090006@v.loewis.de> Message-ID: <08Apr23.091700pdt."58696"@synergy1.parc.xerox.com> martin at v.loewis.de writes: > > When a web browser POSTs data, there is no standard way of communicating > > which encoding it's using. > > That's just not true. Web browser should and do use the encoding of the > web page that originally contained the form. I wonder if the discussion is confusing two different things. Take a look at . There are two prescribed ways of sending form data: application/x-www-form-urlencoded, which can only be used with ASCII data, and multipart/form-data. ``The content type "multipart/form-data" should be used for submitting forms that contain files, non-ASCII data, and binary data.'' It's true that the page containing the form may specify which of these two forms to use, but the character encodings are determined by the choice. > For web forms, I always encode the pages in UTF-8, and that always > works. Should work, if you use the "multipart/form-data" format. Bill From brett at python.org Wed Apr 23 19:52:20 2008 From: brett at python.org (Brett Cannon) Date: Wed, 23 Apr 2008 10:52:20 -0700 Subject: [Python-Dev] BSDDB3 In-Reply-To: <480ED35D.1040403@v.loewis.de> References: <52dc1c820802262146n7b7c211dk973eb44390a4b8f7@mail.gmail.com> <20080227141322.GA5871@amk-desktop.matrixgroup.net> <47CDDF57.1020006@argo.es> <47CF0B7C.109@v.loewis.de> <480CABFB.2080302@argo.es> <87D3F9C72FBF214DB39FA4E3FE618CDC6E243C7CA1@EXMBX04.exchhosting.com> <480EA87D.5040107@argo.es> <480ED35D.1040403@v.loewis.de> Message-ID: On Tue, Apr 22, 2008 at 11:12 PM, "Martin v. L?wis" wrote: > > I'm a bit nervous about syncing, because I have the feeling that > > python-dev is committing changes to python private branch of pybsddb. I > > would rather prefer patches send to me and integrate "canonical" pybsddb > > releases in Python frequently. > > My understanding was that the pybsddb copy in Python *is* the official > code base. I think it's unfortunate that the pybsddb project was > revived, even though it was already dead. > > Perhaps we should remove bsddb from Python again, and refer people to > pybsddb instead? > +1 from me! The amount of issues the bsddb module has caused over the years suggests to me it would do better as an external project again. -Brett From rbp at isnomore.net Wed Apr 23 20:02:40 2008 From: rbp at isnomore.net (Rodrigo Bernardo Pimentel) Date: Wed, 23 Apr 2008 15:02:40 -0300 Subject: [Python-Dev] GSoC Student Introduction In-Reply-To: <480F15EC.10905@cheimes.de> References: <1aff528e0804221742p2e6e9b21jfa0f7606fddf611e@mail.gmail.com> <1afaf6160804221938j1a6672cdv938fc373e00068f@mail.gmail.com> <480ED121.7090005@v.loewis.de> <480F15EC.10905@cheimes.de> Message-ID: <20080423180239.GU16282@isnomore.net> On Wed, Apr 23 2008 at 07:56:44AM BRT, Christian Heimes wrote: > Martin v. L?wis schrieb: > > I certainly can't speak for the respective mentors, but I feel that > > "use bazaar" really isn't the right answer to "can I get commit access?" > > > > One motivation for GSoC is also community bonding, and having the > > mentor (but not *only* the mentor) comment on the proposed changes, > > and monitor the progress of the project. That the development branch > > sits on the student's laptop doesn't really help in that process. Agreed. Using bazaar, at least in this context, involves pushing changes regularly to someplace visible, where the mentor and other developers may see and comment. > Students have to sign the contributor agreement anyway or we can't use > their work. I don't see a problem with giving them svn commit privileges > as long as they restrict themselves to their sandbox. With that, http://python.org/dev/bazaar/ says "anybody with write access to the Subversion repository can push their own branches to python.org", so GSoC students could use bzr and push their changes to python.org regularly, for mentor (and, more generally, core developers) comments. Cheers, rbp -- Rodrigo Bernardo Pimentel | GPG: <0x0DB14978> From greg at krypto.org Wed Apr 23 22:17:19 2008 From: greg at krypto.org (Gregory P. Smith) Date: Wed, 23 Apr 2008 13:17:19 -0700 Subject: [Python-Dev] BSDDB3 In-Reply-To: <480EA87D.5040107@argo.es> References: <52dc1c820802262146n7b7c211dk973eb44390a4b8f7@mail.gmail.com> <20080227141322.GA5871@amk-desktop.matrixgroup.net> <47CDDF57.1020006@argo.es> <47CF0B7C.109@v.loewis.de> <480CABFB.2080302@argo.es> <87D3F9C72FBF214DB39FA4E3FE618CDC6E243C7CA1@EXMBX04.exchhosting.com> <480EA87D.5040107@argo.es> Message-ID: <52dc1c820804231317h67954dcbmdb50b58da35b3b64@mail.gmail.com> On Tue, Apr 22, 2008 at 8:09 PM, Jesus Cea wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Trent Nelson wrote: > | I remember those rampant BSDDB crashes on Windows well. > [...] > | basically, the tests weren't cleaning up their environment in > | the right order, so BSDDB was getting passed completely and > | utterly bogus values. > > Next week I will (if nothing goes wrong) publish pybsddb 4.6.4. This > release supports distributed transactions and replication, testsuite is > way faster, and rewritten to be able to launch tests from multiple > threads/processes if you wish, setuptools/pypi support, etc. > > I think this release would be appropiate to integrate in Python. I think > most demands are solved and new features are interesting (replication, > distributed transactions, do not crash when closing objects in the wrong > order...). Also, I completed the documentation, with the full supported > API, and ported it to Python 2.6 documentation system. The result: > > http://www.jcea.es/programacion/pybsddb.htm#bsddb3-4.6.4 > http://www.jcea.es/programacion/pybsddb_doc/preview/ > > I'm very interested in integrating this release in Python 2.6 for the > new features, the full documentation, and to get feedback from Buildbot > and python-dev community. Also, I would like to avoid to integrate > pybsddb late in the python 2.6 release cycle; I hope to be away of my > computer in August! :). > > I'm a bit nervous about syncing, because I have the feeling that > python-dev is committing changes to python private branch of pybsddb. I > would rather prefer patches send to me and integrate "canonical" pybsddb > releases in Python frequently. > > Somebody suggested to post patches in the tracker, but I think this is > not going to work. The diff from current python bsddb and the official > version is so huge that nobody could follow it. A more sensible > approach, I think, is to "diff" current python pybsddb against the > version I used as my root (January?), integrate the changes in current > "canonical" pybsddb and, then, drop the entire updated package into > python. > Agreed. I "forked" jcea's version found in his svn repository from python trunk at r60209. The only change to any Modules/*bsddb* files made to python trunk since then was me manually applying one bugfix that jcea had made in his repository. My goal all along had been to merge his changes back into python trunk until he had commit access. Obviously I haven't had time to do that. The changes to Lib/bsddb/* since then have been test suite updates which I think jcea should go through and selectively merge as appropriate into his repository (he has done a lot of test suite cleanup). ( svn:// svn.argo.es/jcea/pybsddb/trunk ) After that, merging his changes into trunk will be relatively easy and I think we should give jcea commit access and let him do it and henceforth use python svn as the official repository for the pybsddb code and documentation. I do not see this as a daunting task assuming jcea is willing to be volunteered to do the merge and to handle any py3k merge issues that come up. Alternatively: (removal of bsddb from the standard library) If people want to go the other route a remove bsddb from Python all together letting it thrive as a standalone addon module, what will this mean to the old shelve and dbhash standard library that use bsddb? Someone should write a sqlite backend for shelve. Either way it should not be removed from 2.6 as it needs to be marked at deprecated for one release round only to be taken out in 2.7 and that does leave open the question of should it exist in 3.0 at all or just start out deprecated there? -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Apr 23 22:35:52 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Apr 2008 22:35:52 +0200 Subject: [Python-Dev] GSoC Student Introduction In-Reply-To: References: <1aff528e0804221742p2e6e9b21jfa0f7606fddf611e@mail.gmail.com> <1afaf6160804221938j1a6672cdv938fc373e00068f@mail.gmail.com> <480ED121.7090005@v.loewis.de> Message-ID: <480F9DA8.4030103@v.loewis.de> > What is the point on having a branch in the svn repo if you won't be > able to commit ? Maybe I misunderstood what you said, so maybe could > you clarify that answer ? What I meant is this: you need commit privileges regardless of whether you are going to use bazaar or subversion (i.e. even for bazaar, you still need commit privileges). So if mentors favor usage of bazaar over subversion, they still need to arrange their students to get commit privileges, and then ask them not to use these privileges for subversion, but only for bazaar (because we only have a single set of credentials that we manage). HTH, Martin From martin at v.loewis.de Wed Apr 23 22:38:17 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Apr 2008 22:38:17 +0200 Subject: [Python-Dev] Externally developed modules (was Re: BSDDB3) In-Reply-To: <480F3C6E.8040103@gmail.com> References: <47C32D2D.1030206@free.fr> <47C4813F.2080403@v.loewis.de> <52dc1c820802262146n7b7c211dk973eb44390a4b8f7@mail.gmail.com> <20080227141322.GA5871@amk-desktop.matrixgroup.net> <47CDDF57.1020006@argo.es> <47CF0B7C.109@v.loewis.de> <480CABFB.2080302@argo.es> <87D3F9C72FBF214DB39FA4E3FE618CDC6E243C7CA1@EXMBX04.exchhosting.com> <480EA87D.5040107@argo.es> <480ED35D.1040403@v.loewis.de> <480F3C6E.8040103@gmail.com> Message-ID: <480F9E39.60600@v.loewis.de> >> The usual solution is to not integrate then, at all. Python doesn't >> really ship with any libraries that also have an active life >> outside of Python. > > Not entirely true - we have the ones listed in PEP 360. However, I > think of those only Optik (aka optparse) is currently seeing active > external updates. Hence my classification "doesn't really", which I meant to be vague. > Of course, we also have a big warning at the top of the PEP saying > *Never Ever Do This Again*. I think that's what Guido once said. Regards, Martin From martin at v.loewis.de Wed Apr 23 22:43:29 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Apr 2008 22:43:29 +0200 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <08Apr23.091700pdt."58696"@synergy1.parc.xerox.com> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <91478133-664C-4F29-90CB-D0985BE01F24@cs.toronto.edu> <480D69E9.7020004@v.loewis.de> <091D7293-363C-4766-8931-CFE6A9E72331@cs.toronto.edu> <480E2918.7090006@v.loewis.de> <08Apr23.091700pdt."58696"@synergy1.parc.xerox.com> Message-ID: <480F9F71.7050103@v.loewis.de> >> For web forms, I always encode the pages in UTF-8, and that always >> works. > > Should work, if you use the "multipart/form-data" format. Right - I was implicitly assuming that. Regards, Martin From ggpolo at gmail.com Wed Apr 23 23:07:36 2008 From: ggpolo at gmail.com (Guilherme Polo) Date: Wed, 23 Apr 2008 18:07:36 -0300 Subject: [Python-Dev] GSoC Student Introduction In-Reply-To: <480F9DA8.4030103@v.loewis.de> References: <1aff528e0804221742p2e6e9b21jfa0f7606fddf611e@mail.gmail.com> <1afaf6160804221938j1a6672cdv938fc373e00068f@mail.gmail.com> <480ED121.7090005@v.loewis.de> <480F9DA8.4030103@v.loewis.de> Message-ID: 2008/4/23 "Martin v. L?wis" : > > What is the point on having a branch in the svn repo if you won't be > > able to commit ? Maybe I misunderstood what you said, so maybe could > > you clarify that answer ? > > What I meant is this: you need commit privileges regardless of whether > you are going to use bazaar or subversion (i.e. even for bazaar, you > still need commit privileges). So if mentors favor usage of bazaar > over subversion, they still need to arrange their students to get > commit privileges, and then ask them not to use these privileges for > subversion, but only for bazaar (because we only have a single set > of credentials that we manage). I see, thanks for clearing it up. But, what about giving students a branch in the svn and instructing them to commit only there ? (Chris mentioned this two emails ago). If for some reason svn is not the way to go, then I'm happy in using bazaar for commits anyway. Also, are you (PSF) planning to do this now or just at the "official" gsoc start ? And, is there some internal discussion going on to decide if students are going to get a branch or something at python repositories ? > > HTH, > Martin > -- -- Guilherme H. Polo Goncalves From martin at v.loewis.de Wed Apr 23 23:48:33 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 23 Apr 2008 23:48:33 +0200 Subject: [Python-Dev] GSoC Student Introduction In-Reply-To: References: <1aff528e0804221742p2e6e9b21jfa0f7606fddf611e@mail.gmail.com> <1afaf6160804221938j1a6672cdv938fc373e00068f@mail.gmail.com> <480ED121.7090005@v.loewis.de> <480F9DA8.4030103@v.loewis.de> Message-ID: <480FAEB1.2040500@v.loewis.de> > But, what about giving students a branch in the svn and instructing > them to commit only there ? (Chris mentioned this two emails ago). If > for some reason svn is not the way to go, then I'm happy in using > bazaar for commits anyway. I think it's for the mentor to decide. I see no problems with svn - it's just that Brett suggested to use bazaar. > Also, are you (PSF) planning to do this now or just at the "official" > gsoc start ? And, is there some internal discussion going on to decide > if students are going to get a branch or something at python > repositories ? No, no discussion needed. The students should just send me their SSH keys, and their preferred form of firstname.lastname (some people may have names more complex than that), and I'll add them. Regards, Martin From lists at cheimes.de Wed Apr 23 23:51:53 2008 From: lists at cheimes.de (Christian Heimes) Date: Wed, 23 Apr 2008 23:51:53 +0200 Subject: [Python-Dev] GSoC Student Introduction In-Reply-To: References: <1aff528e0804221742p2e6e9b21jfa0f7606fddf611e@mail.gmail.com> <1afaf6160804221938j1a6672cdv938fc373e00068f@mail.gmail.com> <480ED121.7090005@v.loewis.de> <480F9DA8.4030103@v.loewis.de> Message-ID: Guilherme Polo schrieb: > Also, are you (PSF) planning to do this now or just at the "official" > gsoc start ? And, is there some internal discussion going on to decide > if students are going to get a branch or something at python > repositories ? I suggest we use the bounding period to set up the infrastructure for the students. I've already written a private mail to my student with several small tasks like subscribing to mailing lists, creating accounts at bugs.python.org + wiki.python.org, mailing the contribution agreement etc. By the end of the bounding period everything should be in place - including the working environment on the students' computers. Christian From bvidinli at gmail.com Thu Apr 24 10:13:25 2008 From: bvidinli at gmail.com (bvidinli) Date: Thu, 24 Apr 2008 11:13:25 +0300 Subject: [Python-Dev] annoying dictionary problem, non-existing keys Message-ID: <36e8a7020804240113m7cbd10efibfe9f9b68f6135ac@mail.gmail.com> i use dictionaries to hold some config data, such as: conf={'key1':'value1','key2':'value2'} and so on... when i try to process conf, i have to code every time like: if conf.has_key('key1'): if conf['key1']<>'': other commands.... this is very annoying. in php, i was able to code only like: if conf['key1']=='someth' in python, this fails, because, if key1 does not exists, it raises an exception. MY question: is there a way to directly get value of an array/tuple/dict item by key, as in php above, even if key may not exist, i should not check if key exist, i should only use it, if it does not exist, it may return only empty, just as in php.... i hope you understand my question... -- ?.Bahattin Vidinli Elk-Elektronik M?h. ------------------- iletisim bilgileri (Tercih sirasina gore): skype: bvidinli (sesli gorusme icin, www.skype.com) msn: bvidinli at iyibirisi.com yahoo: bvidinli +90.532.7990607 +90.505.5667711 From tjreedy at udel.edu Thu Apr 24 10:17:53 2008 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 24 Apr 2008 04:17:53 -0400 Subject: [Python-Dev] annoying dictionary problem, non-existing keys References: <36e8a7020804240113m7cbd10efibfe9f9b68f6135ac@mail.gmail.com> Message-ID: Python-dev is for discussion of development of future Python. Use python-list / comp.lang.python / gmane.comp.python.general for usage questions. From martin at v.loewis.de Thu Apr 24 10:20:38 2008 From: martin at v.loewis.de (=?ISO-8859-9?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 24 Apr 2008 10:20:38 +0200 Subject: [Python-Dev] annoying dictionary problem, non-existing keys In-Reply-To: <36e8a7020804240113m7cbd10efibfe9f9b68f6135ac@mail.gmail.com> References: <36e8a7020804240113m7cbd10efibfe9f9b68f6135ac@mail.gmail.com> Message-ID: <481042D6.9000307@v.loewis.de> > this is very annoying. Posting to so many lists is even more annoying. You might not get a single helpful answer just because people are so frustrated by this behavior. Regards, Martin From bvidinli at gmail.com Thu Apr 24 10:36:52 2008 From: bvidinli at gmail.com (bvidinli) Date: Thu, 24 Apr 2008 11:36:52 +0300 Subject: [Python-Dev] annoying dictionary problem, non-existing keys In-Reply-To: References: <36e8a7020804240113m7cbd10efibfe9f9b68f6135ac@mail.gmail.com> Message-ID: <36e8a7020804240136r6e42f9fdv1f3d4dc3e947169a@mail.gmail.com> I posted to so many lists because, this issue is related to all lists, this is an idea for python, this is related to development of python... why are you so much defensive ? i think ideas all important for development of python, software.... i am sory anyway.... hope will be helpful. 2008/4/24, Terry Reedy : > Python-dev is for discussion of development of future Python. Use > python-list / comp.lang.python / gmane.comp.python.general for usage > questions. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/bvidinli%40gmail.com > -- ?.Bahattin Vidinli Elk-Elektronik M?h. ------------------- iletisim bilgileri (Tercih sirasina gore): skype: bvidinli (sesli gorusme icin, www.skype.com) msn: bvidinli at iyibirisi.com yahoo: bvidinli +90.532.7990607 +90.505.5667711 From martin at v.loewis.de Thu Apr 24 13:09:46 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 24 Apr 2008 13:09:46 +0200 Subject: [Python-Dev] BSDDB3 In-Reply-To: <52dc1c820804231317h67954dcbmdb50b58da35b3b64@mail.gmail.com> References: <52dc1c820802262146n7b7c211dk973eb44390a4b8f7@mail.gmail.com> <20080227141322.GA5871@amk-desktop.matrixgroup.net> <47CDDF57.1020006@argo.es> <47CF0B7C.109@v.loewis.de> <480CABFB.2080302@argo.es> <87D3F9C72FBF214DB39FA4E3FE618CDC6E243C7CA1@EXMBX04.exchhosting.com> <480EA87D.5040107@argo.es> <52dc1c820804231317h67954dcbmdb50b58da35b3b64@mail.gmail.com> Message-ID: <48106A7A.2090405@v.loewis.de> > After that, merging his changes into trunk will be relatively easy and I > think we should give jcea commit access and let him do it and henceforth > use python svn as the official repository for the pybsddb code and > documentation. Ok. Jesus, can you please send me your SSH key? > I do not see this as a daunting task assuming jcea is willing to be > volunteered to do the merge and to handle any py3k merge issues that > come up. Ideally, yes. Jesus, if you are uncertain of how exactly the merging works, don't hesitate to ask. Regards, Martin From tnelson at onresolve.com Thu Apr 24 15:34:53 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Thu, 24 Apr 2008 06:34:53 -0700 Subject: [Python-Dev] BSDDB3 In-Reply-To: <480EA87D.5040107@argo.es> References: <47C32D2D.1030206@free.fr> <47C4813F.2080403@v.loewis.de> <52dc1c820802262146n7b7c211dk973eb44390a4b8f7@mail.gmail.com> <20080227141322.GA5871@amk-desktop.matrixgroup.net> <47CDDF57.1020006@argo.es> <47CF0B7C.109@v.loewis.de> <480CABFB.2080302@argo.es> <87D3F9C72FBF214DB39FA4E3FE618CDC6E243C7CA1@EXMBX04.exchhosting.com> <480EA87D.5040107@argo.es> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E2456D05A@EXMBX04.exchhosting.com> > Next week I will (if nothing goes wrong) publish pybsddb > 4.6.4. This release supports distributed transactions and > replication, testsuite is way faster, and rewritten to be > able to launch tests from multiple threads/processes if you > wish, setuptools/pypi support, etc. Great! Once you've settled in with your changes let me know and I'll help with doing the necessary things on the Windows-side to set up the bsddb46.vcproj and switching the build to use that. Trent. From greg.ewing at canterbury.ac.nz Fri Apr 25 01:34:27 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 25 Apr 2008 11:34:27 +1200 Subject: [Python-Dev] BSDDB3 In-Reply-To: <48106A7A.2090405@v.loewis.de> References: <52dc1c820802262146n7b7c211dk973eb44390a4b8f7@mail.gmail.com> <20080227141322.GA5871@amk-desktop.matrixgroup.net> <47CDDF57.1020006@argo.es> <47CF0B7C.109@v.loewis.de> <480CABFB.2080302@argo.es> <87D3F9C72FBF214DB39FA4E3FE618CDC6E243C7CA1@EXMBX04.exchhosting.com> <480EA87D.5040107@argo.es> <52dc1c820804231317h67954dcbmdb50b58da35b3b64@mail.gmail.com> <48106A7A.2090405@v.loewis.de> Message-ID: <48111903.8020906@canterbury.ac.nz> Martin v. L?wis wrote: > Ok. Jesus, can you please send me your SSH key? Nice to see that Heaven is security conscious and accepting encrypted prayer nowadays. :-) -- Greg From nnorwitz at gmail.com Fri Apr 25 06:49:24 2008 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 24 Apr 2008 21:49:24 -0700 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: <4807D51A.2060104@voidspace.org.uk> References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> <4807D51A.2060104@voidspace.org.uk> Message-ID: [Michael working on cleaning up the unittest module] it seems like most of the good ideas have been captured already. I'll through two more (low priority) ideas out there. 1) Randomized test runner/option that runs tests in a random order (like regrtest.py -r, but for methods) 2) decorator to verify a test method is supposed to fail #2 is useful for getting test cases into the code sooner rather than later. I'm pretty sure I have a patch that implements this (http://bugs.python.org/issue1399935). It didn't fit in well with the old unittest structure, but seems closer to the direction you are headed. One other idea that probably ought not be done just yet: add a way of failing with the test continuing. We use this at work (not in Python though) and when used appropriately, it works quite well. It provides more information about the failure. It looks something like this: def testMethod(self): # setup self.assertTrue(precondition) self.expectTrue(value) self.expectEqual(expected_result, other_value) All the expect methods duplicate the assert methods. Asserts can the test to fail immediately, expects don't fail immediately allowing the test to continue. All the expect failures are collected and printed at the end of the method run. I was a little skeptical about assert vs expect at first, but it has proven useful in the long run. As I said, I don't think this should be done now, maybe later. n From ncoghlan at gmail.com Fri Apr 25 08:54:34 2008 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 25 Apr 2008 16:54:34 +1000 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> <4807D51A.2060104@voidspace.org.uk> Message-ID: <4811802A.3030802@gmail.com> Neal Norwitz wrote: > All the expect methods duplicate the assert methods. Asserts can the > test to fail immediately, expects don't fail immediately allowing the > test to continue. All the expect failures are collected and printed > at the end of the method run. I was a little skeptical about assert > vs expect at first, but it has proven useful in the long run. As I > said, I don't think this should be done now, maybe later. I'd agree these kinds of things can be very useful, particularly for tests which run the same test for multiple inputs - when those tests die on the first set of input values that do the wrong thing, it greatly slows down the test-fix-retest cycle, since you have to go through it multiple times. If the test is written to test all the input values, even if some of them fail, and then report at the end "test failed, here are the (input, expected_ouput, actual_output) that didn't work" it can make tests far more efficient. At the moment I do it manually by collected a list of failed triples, then reporting a test failure at the end if the list isn't empty, but something like "expect" methods would avoid the need to recreate that infrastructure every time I need it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From fuzzyman at voidspace.org.uk Fri Apr 25 11:36:14 2008 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 25 Apr 2008 10:36:14 +0100 Subject: [Python-Dev] Proposed unittest changes In-Reply-To: References: <4807555B.2070206@voidspace.org.uk> <480765B5.2010904@voidspace.org.uk> <4807D51A.2060104@voidspace.org.uk> Message-ID: <4811A60E.6010207@voidspace.org.uk> Neal Norwitz wrote: > [Michael working on cleaning up the unittest module] > > it seems like most of the good ideas have been captured already. I'll > through two more (low priority) ideas out there. > > 1) Randomized test runner/option that runs tests in a random order > (like regrtest.py -r, but for methods) > Should be easy enough. > 2) decorator to verify a test method is supposed to fail > > #2 is useful for getting test cases into the code sooner rather than > later. I'm pretty sure I have a patch that implements this > (http://bugs.python.org/issue1399935). It didn't fit in well with the > old unittest structure, but seems closer to the direction you are > headed. > We normally just prefix the test name with 'DONT' so that it isn't actually run... > One other idea that probably ought not be done just yet: add a way of > failing with the test continuing. We use this at work (not in Python > though) and when used appropriately, it works quite well. It provides > more information about the failure. It looks something like this: > > def testMethod(self): > # setup > self.assertTrue(precondition) > self.expectTrue(value) > self.expectEqual(expected_result, other_value) > > All the expect methods duplicate the assert methods. Asserts can the > test to fail immediately, expects don't fail immediately allowing the > test to continue. All the expect failures are collected and printed > at the end of the method run. I was a little skeptical about assert > vs expect at first, but it has proven useful in the long run. As I > said, I don't think this should be done now, maybe later. > > n > I like this pattern. Michael Foord From ndbecker2 at gmail.com Fri Apr 25 15:13:54 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 25 Apr 2008 09:13:54 -0400 Subject: [Python-Dev] segfault in 2.5.1 Message-ID: Attempt to write to a mmap which was opened mmap.PROT_READ causes python to segfault. From schmir at gmail.com Fri Apr 25 15:26:50 2008 From: schmir at gmail.com (Ralf Schmitt) Date: Fri, 25 Apr 2008 15:26:50 +0200 Subject: [Python-Dev] segfault in 2.5.1 In-Reply-To: References: Message-ID: <932f8baf0804250626i5924a1dcvb8ffadbb74a4850d@mail.gmail.com> On Fri, Apr 25, 2008 at 3:13 PM, Neal Becker wrote: > Attempt to write to a mmap which was opened mmap.PROT_READ causes python > to > segfault. http://bugs.python.org/issue2111 -------------- next part -------------- An HTML attachment was scrubbed... URL: From ndbecker2 at gmail.com Fri Apr 25 15:32:33 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 25 Apr 2008 09:32:33 -0400 Subject: [Python-Dev] mmap documentation error (2.5.1) Message-ID: mmap( fileno, length[, flags[, prot[, access]]]) (Unix version) Maps length bytes from the file specified by the file descriptor fileno, and returns a mmap object. If length is 0, the maximum length of the map will be the current size of the file when mmap() is called. flags specifies the nature of the mapping. MAP_PRIVATE creates a private copy-on-write mapping, so changes to the contents of the mmap object will be private to this process, and MAP_SHARED creates a mapping that's shared with all other processes mapping the same areas of the file. The default value is MAP_SHARED. Apparantly, this is wrong - the default is not MAP_SHARED, as I just found out the hard way. From ndbecker2 at gmail.com Fri Apr 25 15:41:12 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 25 Apr 2008 09:41:12 -0400 Subject: [Python-Dev] mmap documentation error (2.5.1) References: Message-ID: Sorry, my mistake. Acutally, I was trying to debug this: On linux, I don't understand why: f = open ('/dev/eos', 'rw') m = mmap.mmap(f.fileno(), 1000000, prot=mmap.PROT_READ|mmap.PROT_WRITE, flags=mmap.MAP_SHARED) gives 'permission denied', but this c++ code works: #include #include #include #include #include int main() { int fd = open ("/dev/eos", O_RDWR); void* m = mmap (NULL, 1000000, PROT_READ|PROT_WRITE, MAP_SHARED, fd, 0); std::cout << m << '\n'; } From aahz at pythoncraft.com Fri Apr 25 16:38:22 2008 From: aahz at pythoncraft.com (Aahz) Date: Fri, 25 Apr 2008 07:38:22 -0700 Subject: [Python-Dev] mmap documentation error (2.5.1) In-Reply-To: References: Message-ID: <20080425143822.GA25649@panix.com> On Fri, Apr 25, 2008, Neal Becker wrote: > > Sorry, my mistake. Acutally, I was trying to debug this: python-dev is probably not the right place for this -- please use c.l.py or the list capi-sig. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From status at bugs.python.org Fri Apr 25 18:06:36 2008 From: status at bugs.python.org (Tracker) Date: Fri, 25 Apr 2008 18:06:36 +0200 (CEST) Subject: [Python-Dev] Summary of Tracker Issues Message-ID: <20080425160636.37FEE78300@psf.upfronthosting.co.za> ACTIVITY SUMMARY (04/18/08 - 04/25/08) Tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue number. Do NOT respond to this message. 1850 open (+26) / 12667 closed (+11) / 14517 total (+37) Open issues with patches: 558 Average duration of open issues: 707 days. Median duration of open issues: 1333 days. Open Issues Breakdown open 1828 (+26) pending 22 ( +0) Issues Created Or Reopened (37) _______________________________ Create ctypes instances from buffer interface 04/18/08 CLOSED http://bugs.python.org/issue2655 created theller patch, patch Autodoc should skip inherited methods 04/18/08 http://bugs.python.org/issue2656 created jmillikin curses 04/18/08 http://bugs.python.org/issue2657 created fer_perez decode_header() fails on multiline headers 04/19/08 http://bugs.python.org/issue2658 created cschnee textwrap handling of hyphenation 04/19/08 http://bugs.python.org/issue2659 created fourmanoit patch Py3k fails to parse a file with an iso-8859-1 string 04/19/08 http://bugs.python.org/issue2660 created azverkan Mapping tests cannot be passed by user implementations 04/19/08 http://bugs.python.org/issue2661 created danderson Allows HUGE_VAL as extern variable 04/20/08 CLOSED http://bugs.python.org/issue2662 created ocean-city patch shutil.copytree glob-style filtering [patch] 04/20/08 http://bugs.python.org/issue2663 created tarek patch The position of special value tables (cmathmodule.c) 04/21/08 http://bugs.python.org/issue2664 created ocean-city patch unable to launch IDLE on windows XP 04/21/08 CLOSED http://bugs.python.org/issue2665 created Leo webbrowser.py doesn't properly handle BROWSER env var 04/21/08 http://bugs.python.org/issue2666 created phd patch Remove multiple inheritance in Python 3000 04/22/08 CLOSED http://bugs.python.org/issue2667 created gmarketer apply() documentation is slightly incorrect 04/22/08 http://bugs.python.org/issue2668 created belopolsky bsddb iterkeys sliently fails when database modified during iter 04/22/08 http://bugs.python.org/issue2669 created tebeka urllib2 build_opener() fails if two handlers use the same defaul 04/22/08 CLOSED http://bugs.python.org/issue2670 created tlynn PyErr_WarnPy3k 04/22/08 http://bugs.python.org/issue2671 created benjamin.peterson patch speed of set.update( 04/23/08 CLOSED http://bugs.python.org/issue2672 created jameinel error on optparse page 04/23/08 CLOSED http://bugs.python.org/issue2673 created fbe2 unittest.TestProgram uses sys.exit() 04/23/08 http://bugs.python.org/issue2674 created tuben Curses terminal resize problems when Python is in interactive mo 04/23/08 http://bugs.python.org/issue2675 created pbazant email/message.py [Message.get_content_type]: Trivial regex hangs 04/23/08 http://bugs.python.org/issue2676 created ajaksu2 patch Argument rules for callables do not apply when function implemen 04/24/08 http://bugs.python.org/issue2677 created garden hmac performance optimization 04/24/08 CLOSED http://bugs.python.org/issue2678 created fafhrd patch email.feedparser regex duplicate 04/24/08 http://bugs.python.org/issue2679 created jimjjewett gotcha: _fields_ is final but accepts lists 04/24/08 CLOSED http://bugs.python.org/issue2680 created cscheid octal literals beginning with 8 don't raise a SyntaxError 04/24/08 CLOSED http://bugs.python.org/issue2681 created lukas.meuser patch cyclic reference in ctypes CFunctionType objects 04/24/08 http://bugs.python.org/issue2682 created theller patch subprocess.Popen.communicate takes bytes, not str 04/24/08 http://bugs.python.org/issue2683 created kermode Logging Module still failing for %(filename)s, __init__ 04/24/08 http://bugs.python.org/issue2684 created CharlesMerriam Add -mieee to compile flags, when available 04/24/08 http://bugs.python.org/issue2685 created marketdickinson patch Any chance we could double the height of the 'Comment:' text are 04/24/08 CLOSED http://bugs.python.org/issue2686 created Trent.Nelson SSL example script fails mysteriously on MacOS 04/25/08 http://bugs.python.org/issue2687 created akuchling Error when nesting many while loops 04/25/08 http://bugs.python.org/issue2688 created rgreimel Fix indentation in range_item 04/25/08 http://bugs.python.org/issue2689 created belopolsky patch Precompute range length 04/25/08 http://bugs.python.org/issue2690 created belopolsky patch Document size_t related long object APIs 04/25/08 http://bugs.python.org/issue2691 created belopolsky Issues Now Closed (27) ______________________ Python 2.4+ spends too much time in PyEval_EvalFrame w/ xmlrpmcl 180 days http://bugs.python.org/issue1327 amaury.forgeotdarc cmath is numerically unsound 172 days http://bugs.python.org/issue1381 marketdickinson patch sqrt(-1) doesn't raise ValueError on OS X 92 days http://bugs.python.org/issue1879 marketdickinson patch xmlrpclib cannot send datetime objects with dates before 1900 75 days http://bugs.python.org/issue2014 schmir patch Restructure import.c into PEP 302 importer objects 64 days http://bugs.python.org/issue2135 brett.cannon patch % string formatting does not support %b 31 days http://bugs.python.org/issue2416 eric.smith Issues with getargs_n() and PyNumber_Index. 33 days http://bugs.python.org/issue2440 Trent.Nelson patch, patch, 64bit imp.get_magic() should return bytes, not bytearray 32 days http://bugs.python.org/issue2471 amaury.forgeotdarc patch IMPORT_NAME Documentation is incomplete 5 days http://bugs.python.org/issue2631 georg.brandl Improve subprocess.Popen() documentation ("env" parameter) 5 days http://bugs.python.org/issue2633 georg.brandl os.execvpe() docs need to be more specific 5 days http://bugs.python.org/issue2634 georg.brandl shutil.copyfile() documentation is vague 4 days http://bugs.python.org/issue2639 georg.brandl httplib throws ValueError 4 days http://bugs.python.org/issue2645 georg.brandl patch Typo at http://docs.python.org/dev/3.0/howto/doanddont.html 0 days http://bugs.python.org/issue2654 akuchling Create ctypes instances from buffer interface 7 days http://bugs.python.org/issue2655 theller patch, patch Allows HUGE_VAL as extern variable 0 days http://bugs.python.org/issue2662 marketdickinson patch unable to launch IDLE on windows XP 1 days http://bugs.python.org/issue2665 amaury.forgeotdarc Remove multiple inheritance in Python 3000 0 days http://bugs.python.org/issue2667 gmarketer urllib2 build_opener() fails if two handlers use the same defaul 0 days http://bugs.python.org/issue2670 amaury.forgeotdarc speed of set.update( 2 days http://bugs.python.org/issue2672 jameinel error on optparse page 1 days http://bugs.python.org/issue2673 benjamin.peterson hmac performance optimization 0 days http://bugs.python.org/issue2678 amaury.forgeotdarc patch gotcha: _fields_ is final but accepts lists 0 days http://bugs.python.org/issue2680 theller octal literals beginning with 8 don't raise a SyntaxError 0 days http://bugs.python.org/issue2681 amaury.forgeotdarc patch Any chance we could double the height of the 'Comment:' text are 0 days http://bugs.python.org/issue2686 Trent.Nelson tokenize module w/ coding cookie 1838 days http://bugs.python.org/issue719888 Trent.Nelson patch xmlrpclib.Binary doesn't say which base64 spec it implements 285 days http://bugs.python.org/issue1753732 benjamin.peterson patch, easy Top Issues Most Discussed (10) ______________________________ 25 Write user's version of the reference guide 869 days open http://bugs.python.org/issue1376292 7 Argument rules for callables do not apply when function impleme 1 days open http://bugs.python.org/issue2677 6 speed of set.update( 2 days closed http://bugs.python.org/issue2672 5 unable to launch IDLE on windows XP 1 days closed http://bugs.python.org/issue2665 4 cyclic reference in ctypes CFunctionType objects 1 days open http://bugs.python.org/issue2682 4 httplib throws ValueError 4 days closed http://bugs.python.org/issue2645 4 Regexp 2.6 (modifications to current re 2.2.2) 10 days open http://bugs.python.org/issue2636 4 Adapt pydoc to new doc system 95 days open http://bugs.python.org/issue1883 3 Any chance we could double the height of the 'Comment:' text ar 0 days closed http://bugs.python.org/issue2686 3 gotcha: _fields_ is final but accepts lists 0 days closed http://bugs.python.org/issue2680 From tnelson at onresolve.com Fri Apr 25 21:49:13 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Fri, 25 Apr 2008 12:49:13 -0700 Subject: [Python-Dev] Do we still need BaseAddress in .vcproj files? Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E246919B5@EXMBX04.exchhosting.com> I just did this locally to all my .vcproj files for .pyds: - BaseAddress="0x1e000000" + RandomizedBaseAddress="2" This was partly out of curiosity, and partly because it was quicker doing that than finding a new unique base address to use for a new module I added to my local tree. Rebuilt everything and ran a full regression test, and everything passed. What am I missing? Do we have parts of Python that rely on finding modules at explicit base addresses? Is BaseAddress a relic from the past? Christian or Martin? Trent. From lists at cheimes.de Fri Apr 25 22:16:11 2008 From: lists at cheimes.de (Christian Heimes) Date: Fri, 25 Apr 2008 22:16:11 +0200 Subject: [Python-Dev] Do we still need BaseAddress in .vcproj files? In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E246919B5@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E246919B5@EXMBX04.exchhosting.com> Message-ID: <48123C0B.10401@cheimes.de> Trent Nelson schrieb: > I just did this locally to all my .vcproj files for .pyds: > > - BaseAddress="0x1e000000" > + RandomizedBaseAddress="2" > > This was partly out of curiosity, and partly because it was quicker doing that than finding a new unique base address to use for a new module I added to my local tree. Rebuilt everything and ran a full regression test, and everything passed. What am I missing? Do we have parts of Python that rely on finding modules at explicit base addresses? Is BaseAddress a relic from the past? Christian or Martin? An explicit base address speeds up the loading of shared extension modules a bit. Checkout PC/dllbase_nt.txt Christian From martin at v.loewis.de Fri Apr 25 23:00:18 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 25 Apr 2008 23:00:18 +0200 Subject: [Python-Dev] Do we still need BaseAddress in .vcproj files? In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E246919B5@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E246919B5@EXMBX04.exchhosting.com> Message-ID: <48124662.2080502@v.loewis.de> > This was partly out of curiosity, and partly because it was quicker > doing that than finding a new unique base address to use for a new > module I added to my local tree. Rebuilt everything and ran a full > regression test, and everything passed. What am I missing? Do we > have parts of Python that rely on finding modules at explicit base > addresses? Is BaseAddress a relic from the past? Christian or > Martin? As Christian says: specifying the base address so that the addresses don't overlap avoids having Windows to perform relocations on startup. There is a Microsoft tool (editbin /rebase) to compute non-overlapping base addresses for a given set of DLLs. Regards, Martin From varmaa at gmail.com Fri Apr 25 23:52:31 2008 From: varmaa at gmail.com (Atul Varma) Date: Fri, 25 Apr 2008 14:52:31 -0700 Subject: [Python-Dev] Do we still need BaseAddress in .vcproj files? In-Reply-To: <48124662.2080502@v.loewis.de> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E246919B5@EXMBX04.exchhosting.com> <48124662.2080502@v.loewis.de> Message-ID: <361b27370804251452r3a17cb58p8fedf5c9461e65c8@mail.gmail.com> A while back I was reading an MSDN article that did some concrete research on the performance benefits of rebasing DLLs to avoid fixups at runtime, and it actually concluded that on modern systems, the benefits are really negligible. I tried finding the article on the net just now, though, and I couldn't, so don't take my word for it. This 1995 MSDN article provides an explanation and some raw numbers for systems at that time, though: http://msdn2.microsoft.com/en-us/library/ms810432.aspx - Atul On Fri, Apr 25, 2008 at 2:00 PM, "Martin v. L?wis" wrote: > > This was partly out of curiosity, and partly because it was quicker > > doing that than finding a new unique base address to use for a new > > module I added to my local tree. Rebuilt everything and ran a full > > regression test, and everything passed. What am I missing? Do we > > have parts of Python that rely on finding modules at explicit base > > addresses? Is BaseAddress a relic from the past? Christian or > > Martin? > > As Christian says: specifying the base address so that the addresses > don't overlap avoids having Windows to perform relocations on startup. > > There is a Microsoft tool (editbin /rebase) to compute non-overlapping > base addresses for a given set of DLLs. > > Regards, > Martin > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/varmaa%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From collinw at gmail.com Sat Apr 26 01:52:23 2008 From: collinw at gmail.com (Collin Winter) Date: Fri, 25 Apr 2008 16:52:23 -0700 Subject: [Python-Dev] annoying dictionary problem, non-existing keys In-Reply-To: <36e8a7020804240136r6e42f9fdv1f3d4dc3e947169a@mail.gmail.com> References: <36e8a7020804240113m7cbd10efibfe9f9b68f6135ac@mail.gmail.com> <36e8a7020804240136r6e42f9fdv1f3d4dc3e947169a@mail.gmail.com> Message-ID: <43aa6ff70804251652r4f0deb39he5b2e1c71e8aabee@mail.gmail.com> 2008/4/24 bvidinli : > I posted to so many lists because, > > this issue is related to all lists, > this is an idea for python, > this is related to development of python... > > why are you so much defensive ? > > i think ideas all important for development of python, software.... > i am sory anyway.... hope will be helpful. Please consult the documentation first: http://docs.python.org/lib/typesmapping.html . You're looking for the get() method. This attribute of PHP is hardly considered a feature, and is not something Python wishes to emulate. Collin Winter > 2008/4/24, Terry Reedy : > > > > Python-dev is for discussion of development of future Python. Use > > python-list / comp.lang.python / gmane.comp.python.general for usage > > questions. > > > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/bvidinli%40gmail.com > > > > > > -- > ?.Bahattin Vidinli > Elk-Elektronik M?h. > ------------------- > iletisim bilgileri (Tercih sirasina gore): > skype: bvidinli (sesli gorusme icin, www.skype.com) > msn: bvidinli at iyibirisi.com > yahoo: bvidinli > > +90.532.7990607 > +90.505.5667711 > _______________________________________________ > > > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/collinw%40gmail.com > From musiccomposition at gmail.com Sun Apr 27 02:33:08 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Sat, 26 Apr 2008 19:33:08 -0500 Subject: [Python-Dev] PyErr_Py3kWarn Message-ID: <1afaf6160804261733t41d8fde8pef16a749828ac766@mail.gmail.com> I have a patch to to add an easy way to give a Py3k warning (issue 2671). Does this sound useful? -- Cheers, Benjamin Peterson From brett at python.org Sun Apr 27 22:40:52 2008 From: brett at python.org (Brett Cannon) Date: Sun, 27 Apr 2008 13:40:52 -0700 Subject: [Python-Dev] Dealing with a desired change to warnings.showwarning() Message-ID: As part of my rewrite of warnings into C, I added a new, optional argument to showwarning(): line, which defaults to None. As http://bugs.python.org/issue2705 points out, though, since the function has been documented as being allowed to be overridden, this potentially breaks existing showwarning() implementations. That means something needs to change. One option is to introduce a new function. But what to name it? show_warning()? That small of a name change seems like asking for trouble, but showwarning_ex() just seems wrong to use in Python code. Another is that where showwarning() is called within the code, if a TypeError is raised or introspection proves the call can't work, then to try again without the 'line' argument, and raise a deprecation warning. Or try to re-architect _warnings.c to not use the 'line' argument (although it is handy for testing purposes). Anyway, people have any suggestions? -Brett From skip at pobox.com Mon Apr 28 00:16:12 2008 From: skip at pobox.com (skip at pobox.com) Date: Sun, 27 Apr 2008 17:16:12 -0500 Subject: [Python-Dev] Double specification of function signatures? Message-ID: <18452.64300.579813.77697@montanaro-dyndns-org.local> While cleaning up the documentation for the tempfile module I noticed that the docstrings for the mk*temp functions in the module itself list their signatures (incompletely) in the first line. I don't know if that was intentional, but it seems both redundant and error-prone to me. The help() function already displays the signatures of Python functions. There's no need to put them in docstrings and risk having them out-of-date. For example: >>> help(tempfile.mkdtemp) Help on function mkdtemp in module tempfile: mkdtemp(suffix='', prefix='tmp', dir=None) mkdtemp([suffix, [prefix, [dir]]]) User-callable function to create and return a unique temporary directory. The return value is the pathname of the directory. Am I way off-base here? Let me know, as I have a couple minor tweaks to check in besides these. Thx, Skip From fuzzyman at voidspace.org.uk Mon Apr 28 00:26:29 2008 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sun, 27 Apr 2008 23:26:29 +0100 Subject: [Python-Dev] [Doc-SIG] Double specification of function signatures? In-Reply-To: <18452.64300.579813.77697@montanaro-dyndns-org.local> References: <18452.64300.579813.77697@montanaro-dyndns-org.local> Message-ID: <4814FD95.2010206@voidspace.org.uk> skip at pobox.com wrote: > While cleaning up the documentation for the tempfile module I noticed that > the docstrings for the mk*temp functions in the module itself list their > signatures (incompletely) in the first line. I don't know if that was > intentional, but it seems both redundant and error-prone to me. The help() > function already displays the signatures of Python functions. There's no > need to put them in docstrings and risk having them out-of-date. For > example: > > >>> help(tempfile.mkdtemp) > Help on function mkdtemp in module tempfile: > > mkdtemp(suffix='', prefix='tmp', dir=None) > mkdtemp([suffix, [prefix, [dir]]]) > User-callable function to create and return a unique temporary > directory. The return value is the pathname of the directory. > > Am I way off-base here? Let me know, as I have a couple minor tweaks to > check in besides these. > It seems that any documentation or help tool worth its salt should fetch the parameters from the definition and so including them in the docstring should be redundant duplication. Michael Foord > Thx, > > Skip > _______________________________________________ > Doc-SIG maillist - Doc-SIG at python.org > http://mail.python.org/mailman/listinfo/doc-sig > From skip at pobox.com Mon Apr 28 00:29:33 2008 From: skip at pobox.com (skip at pobox.com) Date: Sun, 27 Apr 2008 17:29:33 -0500 Subject: [Python-Dev] [Doc-SIG] Double specification of function signatures? In-Reply-To: <4814FD95.2010206@voidspace.org.uk> References: <18452.64300.579813.77697@montanaro-dyndns-org.local> <4814FD95.2010206@voidspace.org.uk> Message-ID: <18452.65101.202393.473799@montanaro-dyndns-org.local> Michael> It seems that any documentation or help tool worth its salt Michael> should fetch the parameters from the definition and so Michael> including them in the docstring should be redundant Michael> duplication. That's my position as well. Currently we have no way to extract the function signatures from C code on-the-fly or in a preprocessing step (might be a good GSoC project), but for functions written in Python I'm of the opinion the docstrings should not include function signatures. Skip From brett at python.org Mon Apr 28 00:31:51 2008 From: brett at python.org (Brett Cannon) Date: Sun, 27 Apr 2008 15:31:51 -0700 Subject: [Python-Dev] [Doc-SIG] Double specification of function signatures? In-Reply-To: <18452.65101.202393.473799@montanaro-dyndns-org.local> References: <18452.64300.579813.77697@montanaro-dyndns-org.local> <4814FD95.2010206@voidspace.org.uk> <18452.65101.202393.473799@montanaro-dyndns-org.local> Message-ID: On Sun, Apr 27, 2008 at 3:29 PM, wrote: > > Michael> It seems that any documentation or help tool worth its salt > Michael> should fetch the parameters from the definition and so > Michael> including them in the docstring should be redundant > Michael> duplication. > > That's my position as well. Currently we have no way to extract the > function signatures from C code on-the-fly or in a preprocessing step (might > be a good GSoC project), but for functions written in Python I'm of the > opinion the docstrings should not include function signatures. They shouldn't. Maybe the tempfile module came from a third-party that had some internal doc rule of mentioning the call signature. Regardless, just rip it out. -Brett From skip at pobox.com Mon Apr 28 00:33:14 2008 From: skip at pobox.com (skip at pobox.com) Date: Sun, 27 Apr 2008 17:33:14 -0500 Subject: [Python-Dev] Warn about mktemp once again? Message-ID: <18452.65322.843226.299769@montanaro-dyndns-org.local> Back in r29829, Guido commented out the security hole warning for tempfile.mktemp: r29829 | gvanrossum | 2002-11-22 09:56:29 -0600 (Fri, 22 Nov 2002) | 3 lines Comment out the warnings about mktemp(). These are too annoying, and often unavoidable. Any thought about whether this warning should be restored? We're 5+ years later. Hopefully many uses of mktemp have been removed. If we're not going to restore the warning perhaps the commented code should just be deleted. Skip From stephen at xemacs.org Mon Apr 28 01:40:02 2008 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 28 Apr 2008 08:40:02 +0900 Subject: [Python-Dev] [Doc-SIG] Double specification of function signatures? In-Reply-To: <18452.65101.202393.473799@montanaro-dyndns-org.local> References: <18452.64300.579813.77697@montanaro-dyndns-org.local> <4814FD95.2010206@voidspace.org.uk> <18452.65101.202393.473799@montanaro-dyndns-org.local> Message-ID: <8763u2yii5.fsf@uwakimon.sk.tsukuba.ac.jp> skip at pobox.com writes: > Currently we have no way to extract the function signatures from C > code on-the-fly or in a preprocessing step (might be a good GSoC > project), +1 on GSoC, except for the bad timing (maybe somebody will start sponsoring a "Winter Coding Sprint" event?) From brett at python.org Mon Apr 28 02:31:37 2008 From: brett at python.org (Brett Cannon) Date: Sun, 27 Apr 2008 17:31:37 -0700 Subject: [Python-Dev] [Doc-SIG] Double specification of function signatures? In-Reply-To: <8763u2yii5.fsf@uwakimon.sk.tsukuba.ac.jp> References: <18452.64300.579813.77697@montanaro-dyndns-org.local> <4814FD95.2010206@voidspace.org.uk> <18452.65101.202393.473799@montanaro-dyndns-org.local> <8763u2yii5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Sun, Apr 27, 2008 at 4:40 PM, Stephen J. Turnbull wrote: > skip at pobox.com writes: > > > Currently we have no way to extract the function signatures from C > > code on-the-fly or in a preprocessing step (might be a good GSoC > > project), > > +1 on GSoC, except for the bad timing (maybe somebody will start > sponsoring a "Winter Coding Sprint" event?) > It can still go on the wiki, though, so we don't forget about it. -Brett From skip at pobox.com Mon Apr 28 05:05:11 2008 From: skip at pobox.com (skip at pobox.com) Date: Sun, 27 Apr 2008 22:05:11 -0500 Subject: [Python-Dev] [Doc-SIG] Double specification of function signatures? In-Reply-To: References: <18452.64300.579813.77697@montanaro-dyndns-org.local> <4814FD95.2010206@voidspace.org.uk> <18452.65101.202393.473799@montanaro-dyndns-org.local> <8763u2yii5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <18453.16103.592698.156270@montanaro-dyndns-org.local> >> > Currently we have no way to extract the function signatures from C >> > code on-the-fly or in a preprocessing step (might be a good GSoC >> > project), Brett> It can still go on the wiki, though, so we don't forget about it. Done. Skip From skip at pobox.com Mon Apr 28 05:06:06 2008 From: skip at pobox.com (skip at pobox.com) Date: Sun, 27 Apr 2008 22:06:06 -0500 Subject: [Python-Dev] [Doc-SIG] Double specification of function signatures? In-Reply-To: References: <18452.64300.579813.77697@montanaro-dyndns-org.local> <4814FD95.2010206@voidspace.org.uk> <18452.65101.202393.473799@montanaro-dyndns-org.local> Message-ID: <18453.16158.25027.879421@montanaro-dyndns-org.local> Brett> They shouldn't. Maybe the tempfile module came from a third-party Brett> that had some internal doc rule of mentioning the call signature. Brett> Regardless, just rip it out. Done. Skip From mhammond at skippinet.com.au Mon Apr 28 06:41:18 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Mon, 28 Apr 2008 14:41:18 +1000 Subject: [Python-Dev] Is Py_WIN_WIDE_FILENAMES still alive? In-Reply-To: <000901c8a384$e3965570$0300a8c0@whiterabc2znlh> References: <000901c8a384$e3965570$0300a8c0@whiterabc2znlh> Message-ID: <005d01c8a8ea$1c954f60$55bfee20$@com.au> > Hello. I noticed when I removes following line in trunk/PC/pyconfig.h > > #define Py_WIN_WIDE_FILENAMES > > _fileio.c and posixmodule.c (and maybe more) cannot be compiled on > Windows. > > When Py_WIN_WIDE_FILENAMES is not defined, how should python behave? > > - call posix functions like open(2) > > - call ANSI Win32 APIs like MoveFileA > > Or maybe this macro is not used anymore? I believe the macro should be removed, as Python currently assumes Unicode APIs are available in a number of places. This consistent with the versions of Windows Python currently supports. It is possible that patches would be accepted which enable Python to be built without this functionality - in which case the next-best behavior would probably be to convert Unicode to MBCS encoded strings and call the *A versions of the API. Looking at the history for posixmodule.c etc will offer some clues as to how this could best be done. Cheers, Mark From mhammond at skippinet.com.au Mon Apr 28 08:22:26 2008 From: mhammond at skippinet.com.au (Mark Hammond) Date: Mon, 28 Apr 2008 16:22:26 +1000 Subject: [Python-Dev] Python 2.6a2 execution times with various compilers In-Reply-To: <52dc1c820804121521u4c8f69d8q1bf0d0bb9b00e03a@mail.gmail.com> References: <20080411151024.GJ51167@nexus.in-nomine.org> <20080412180936.GP51167@nexus.in-nomine.org> <52dc1c820804121521u4c8f69d8q1bf0d0bb9b00e03a@mail.gmail.com> Message-ID: <006f01c8a8f8$3d651690$b82f43b0$@com.au> > Profile-guided optimization did not help much, as might be expected, it > pushed about the same kind of optimization as the mtune/march combination. > With gcc 4.1.3 i'm finding that profile guided optimization when trained > on pybench or regrtest does make a measurable difference (2-5% overall > time with 10-20% on some pybench tests).? I haven't run benchmarks enough > times to be confident in my results yet, I'll report back with data once > I have it.? I'm testing both pybench and regrtest as profiling training runs. It seems that profile guided optimization also offers some benefits on Windows (eg, http://mail.python.org/pipermail/python-dev/2007-May/072970.html), so it might be worth trying to coordinate such efforts between platforms (eg, a central document which holds results for all supported platforms sounds worthwhile, and maybe sharing the top-level script that generates the profile data, etc) Cheers, Mark From ocean at m2.ccsnet.ne.jp Mon Apr 28 08:32:59 2008 From: ocean at m2.ccsnet.ne.jp (ocean) Date: Mon, 28 Apr 2008 15:32:59 +0900 Subject: [Python-Dev] Is Py_WIN_WIDE_FILENAMES still alive? References: <000901c8a384$e3965570$0300a8c0@whiterabc2znlh> <005d01c8a8ea$1c954f60$55bfee20$@com.au> Message-ID: <000701c8a8f9$b4487670$0200a8c0@whiterabc2znlh> > I believe the macro should be removed, as Python currently assumes Unicode > APIs are available in a number of places. My +1 for removal. Even 2.5 cannot be compiled without this macro, probably no one is using this. > This consistent with the versions > of Windows Python currently supports. It is possible that patches would be > accepted which enable Python to be built without this functionality - in > which case the next-best behavior would probably be to convert Unicode to > MBCS encoded strings and call the *A versions of the API. Looking at the > history for posixmodule.c etc will offer some clues as to how this could > best be done. One problem with removal, PyArg_ParseTuple doesn't have option to convert to unicode (like "et" exists for unicode -> 8bit char buffer), so it's harder to report argument error. >>> os.rename(4, 2) Traceback (most recent call last): File "", line 1, in TypeError: rename() argument 1 must be string, not int /* After removal of *A win32 APIs */ if (PyArg_ParseTuple("OO:rename", &o1, &o2)) { convert_to_unicode(&o1); /* if these methods failed, should we report same error above convert_to_unicode(&o2); * by ourselves? */ From solipsis at pitrou.net Mon Apr 28 10:26:22 2008 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 28 Apr 2008 08:26:22 +0000 (UTC) Subject: [Python-Dev] Dealing with a desired change to warnings.showwarning() References: Message-ID: Brett Cannon python.org> writes: > As http://bugs.python.org/issue2705 points out, though, since the > function has been documented as being allowed to be overridden, this > potentially breaks existing showwarning() implementations. That means > something needs to change. > > One option is to introduce a new function. But what to name it? > show_warning()? That small of a name change seems like asking for > trouble, but showwarning_ex() just seems wrong to use in Python code. But then what happens if a library overrides one of the callbacks and another library overrides the other one? Fixing the original "problem" seems to introduce other hairier ones. As a Benjamin pointed out in the bug comments, the sane solution would be to keep an unique callback and mention the small change in the release notes. Regards Antoine. From ndbecker2 at gmail.com Mon Apr 28 14:02:57 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 28 Apr 2008 08:02:57 -0400 Subject: [Python-Dev] enhanced ioctl? Message-ID: IIUC, current ioctl is not capable of handling arbitrary argument types. This code will allow any arg type (such as structures with pointers to embedded structures). The code for _IOC is taken from linux and might not be portable. from ctypes import * libc = CDLL ('/lib/libc.so.6') #print libc.ioctl def set_ioctl_argtype (arg_type): libc.ioctl.argtypes = (c_int, c_int, arg_type) _IOC_WRITE = 0x1 _IOC_NRBITS= 8 _IOC_TYPEBITS= 8 _IOC_SIZEBITS= 14 _IOC_DIRBITS= 2 _IOC_NRSHIFT= 0 _IOC_TYPESHIFT= (_IOC_NRSHIFT+_IOC_NRBITS) _IOC_SIZESHIFT= (_IOC_TYPESHIFT+_IOC_TYPEBITS) _IOC_DIRSHIFT= (_IOC_SIZESHIFT+_IOC_SIZEBITS) def _IOC (dir, type, nr, size): return (((dir) << _IOC_DIRSHIFT) | \ ((type) << _IOC_TYPESHIFT) | \ ((nr) << _IOC_NRSHIFT) | \ ((size) << _IOC_SIZESHIFT)) def ioctl (fd, request, args): return libc.ioctl (fd, request, args) ------------------- Example usage: from ioctl import * set_ioctl_argtype (POINTER (eos_dl_args_t)) EOS_IOC_MAGIC = 0xF4 request = _IOC(_IOC_WRITE, EOS_IOC_MAGIC, 0x00, 0) # ignore size err = ioctl (eos_file.fileno(), request, args) From musiccomposition at gmail.com Mon Apr 28 22:41:12 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Mon, 28 Apr 2008 15:41:12 -0500 Subject: [Python-Dev] enhanced ioctl? In-Reply-To: References: Message-ID: <1afaf6160804281341w5fbd4b73j4b51487b7aff43a0@mail.gmail.com> On Mon, Apr 28, 2008 at 7:02 AM, Neal Becker wrote: > IIUC, current ioctl is not capable of handling arbitrary argument types. > This code will allow any arg type (such as structures with pointers to > embedded structures). Please submit this patch to the tracker. -- Cheers, Benjamin Peterson From ndbecker2 at gmail.com Mon Apr 28 23:05:11 2008 From: ndbecker2 at gmail.com (Neal Becker) Date: Mon, 28 Apr 2008 17:05:11 -0400 Subject: [Python-Dev] enhanced ioctl? References: <1afaf6160804281341w5fbd4b73j4b51487b7aff43a0@mail.gmail.com> Message-ID: Benjamin Peterson wrote: > On Mon, Apr 28, 2008 at 7:02 AM, Neal Becker wrote: >> IIUC, current ioctl is not capable of handling arbitrary argument types. >> This code will allow any arg type (such as structures with pointers to >> embedded structures). > > Please submit this patch to the tracker. > > issue 2712 From guido at python.org Tue Apr 29 01:01:09 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 28 Apr 2008 16:01:09 -0700 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: <18452.65322.843226.299769@montanaro-dyndns-org.local> References: <18452.65322.843226.299769@montanaro-dyndns-org.local> Message-ID: Have we documented the alternatives well enough? In most cases NamedTemporaryFile will work, but sometimes you will have to create a directory and pick names therein. Doing that so that it will always be cleaned up properly is a bit of a trick, involving an isdir() check and a shutil.rmtree() call. On Sun, Apr 27, 2008 at 3:33 PM, wrote: > Back in r29829, Guido commented out the security hole warning for > tempfile.mktemp: > > r29829 | gvanrossum | 2002-11-22 09:56:29 -0600 (Fri, 22 Nov 2002) | 3 lines > > Comment out the warnings about mktemp(). These are too annoying, and > often unavoidable. > > Any thought about whether this warning should be restored? We're 5+ years > later. Hopefully many uses of mktemp have been removed. If we're not going > to restore the warning perhaps the commented code should just be deleted. > > Skip > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Tue Apr 29 01:17:54 2008 From: skip at pobox.com (skip at pobox.com) Date: Mon, 28 Apr 2008 18:17:54 -0500 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: References: <18452.65322.843226.299769@montanaro-dyndns-org.local> Message-ID: <18454.23330.147475.830232@montanaro-dyndns-org.local> Guido> Have we documented the alternatives well enough? I suppose we could document explicitly how to use mkstemp() in place of mktemp(), but the difference in return value is fairly modest: >>> tempfile.mktemp() '/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/tmpV_5OLi' >>> tempfile.mkstemp() (3, '/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/tmpmS7K4T') and the argument list is quite similar as well: >>> help(tempfile.mktemp) Help on function mktemp in module tempfile: mktemp(suffix='', prefix='tmp', dir=None) ... >>> help(tempfile.mkstemp) Help on function mkstemp in module tempfile: mkstemp(suffix='', prefix='tmp', dir=None, text=False) ... Guido> In most cases NamedTemporaryFile will work, ... It's API seems to be a bit farther from the mktemp API than that of mkstemp. Skip From guido at python.org Tue Apr 29 01:22:42 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 28 Apr 2008 16:22:42 -0700 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: <18454.23330.147475.830232@montanaro-dyndns-org.local> References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> Message-ID: IMO mkstemp() is a major pain because you have to use raw file descriptors on the return value. I'd much rather recommend [Named]TemporaryFile which return streams. On Mon, Apr 28, 2008 at 4:17 PM, wrote: > > Guido> Have we documented the alternatives well enough? > > I suppose we could document explicitly how to use mkstemp() in place of > mktemp(), but the difference in return value is fairly modest: > > >>> tempfile.mktemp() > '/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/tmpV_5OLi' > >>> tempfile.mkstemp() > (3, '/var/folders/5q/5qTPn6xq2RaWqk+1Ytw3-U+++TI/-Tmp-/tmpmS7K4T') > > and the argument list is quite similar as well: > > >>> help(tempfile.mktemp) > Help on function mktemp in module tempfile: > > mktemp(suffix='', prefix='tmp', dir=None) > ... > >>> help(tempfile.mkstemp) > Help on function mkstemp in module tempfile: > > mkstemp(suffix='', prefix='tmp', dir=None, text=False) > ... > > Guido> In most cases NamedTemporaryFile will work, ... > > It's API seems to be a bit farther from the mktemp API than that of mkstemp. > > Skip > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Tue Apr 29 01:35:28 2008 From: skip at pobox.com (skip at pobox.com) Date: Mon, 28 Apr 2008 18:35:28 -0500 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> Message-ID: <18454.24384.41749.272069@montanaro-dyndns-org.local> Guido> IMO mkstemp() is a major pain because you have to use raw file Guido> descriptors on the return value. I'd much rather recommend Guido> [Named]TemporaryFile which return streams. Why not: fd, fname = tempfile.mkstemp() f = os.fdopen(fd) Seems fairly straightforward to me. Skip From guido at python.org Tue Apr 29 01:50:19 2008 From: guido at python.org (Guido van Rossum) Date: Mon, 28 Apr 2008 16:50:19 -0700 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: <18454.24384.41749.272069@montanaro-dyndns-org.local> References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <18454.24384.41749.272069@montanaro-dyndns-org.local> Message-ID: I'd be much happier if there was a standard API in tempfile.py that did that for you, so you wouldn't have to understand fdopen(). (Your example is wrong BTW, the open mode would have to be something like 'rb+' or 'wb+'.) On Mon, Apr 28, 2008 at 4:35 PM, wrote: > > Guido> IMO mkstemp() is a major pain because you have to use raw file > Guido> descriptors on the return value. I'd much rather recommend > Guido> [Named]TemporaryFile which return streams. > > Why not: > > fd, fname = tempfile.mkstemp() > f = os.fdopen(fd) > > Seems fairly straightforward to me. > > Skip > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Tue Apr 29 04:12:21 2008 From: skip at pobox.com (skip at pobox.com) Date: Mon, 28 Apr 2008 21:12:21 -0500 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <18454.24384.41749.272069@montanaro-dyndns-org.local> Message-ID: <18454.33797.918894.424380@montanaro-dyndns-org.local> Guido> I'd be much happier if there was a standard API in tempfile.py Guido> that did that for you, so you wouldn't have to understand Guido> fdopen(). That can be arranged: http://bugs.python.org/issue2717 Guido> (Your example is wrong BTW, the open mode would have to be Guido> something like 'rb+' or 'wb+'.) I'm not disagreeing with you, but it seems odd that os.fdopen doesn't simply obey the mode of the file descriptor it receives as its argument (e.g., normally opened with "w+b"). Skip From greg.ewing at canterbury.ac.nz Tue Apr 29 07:15:11 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 29 Apr 2008 17:15:11 +1200 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: <18454.23330.147475.830232@montanaro-dyndns-org.local> References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> Message-ID: <4816AEDF.1060905@canterbury.ac.nz> skip at pobox.com wrote: > Guido> Have we documented the alternatives well enough? > > I suppose we could document explicitly how to use mkstemp() in place of > mktemp(), but the difference in return value is fairly modest: I'd like to see a variation of mkstemp() that returns a file object instead of a file descriptor, since that's what you really want most of the time. At least I always end up calling fdopen on it. -- Greg From rasky at develer.com Tue Apr 29 07:43:34 2008 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 29 Apr 2008 05:43:34 +0000 (UTC) Subject: [Python-Dev] Warn about mktemp once again? References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <4816AEDF.1060905@canterbury.ac.nz> Message-ID: On Tue, 29 Apr 2008 17:15:11 +1200, Greg Ewing wrote: > skip at pobox.com wrote: >> Guido> Have we documented the alternatives well enough? >> >> I suppose we could document explicitly how to use mkstemp() in place of >> mktemp(), but the difference in return value is fairly modest: > > I'd like to see a variation of mkstemp() that returns a file object > instead of a file descriptor, since that's what you really want most of > the time. At least I always end up calling fdopen on it. Same here. In fact, is there a good reason to have mkstemp() return the fd (except backward compatibility)? -- Giovanni Bajo Develer S.r.l. http://www.develer.com From greg.ewing at canterbury.ac.nz Tue Apr 29 07:49:33 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 29 Apr 2008 17:49:33 +1200 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> Message-ID: <4816B6ED.6030501@canterbury.ac.nz> Guido van Rossum wrote: > IMO mkstemp() is a major pain because you have to use raw file > descriptors on the return value. I'd much rather recommend > [Named]TemporaryFile which return streams. The problem with NamedTemporaryFile is that it deletes the file as soon as you close it, which makes the named-ness of it rather useless, as far as I can see. If you don't want that to happen, you have to use mkstemp. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 29 07:50:33 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 29 Apr 2008 17:50:33 +1200 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: <18454.33797.918894.424380@montanaro-dyndns-org.local> References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <18454.24384.41749.272069@montanaro-dyndns-org.local> <18454.33797.918894.424380@montanaro-dyndns-org.local> Message-ID: <4816B729.9090108@canterbury.ac.nz> skip at pobox.com wrote: > I'm not disagreeing with you, but it seems odd that os.fdopen doesn't simply > obey the mode of the file descriptor it receives as its argument I'm not even sure if it's possible to find out the mode that a file descriptor was opened with -- is it? Certainly there's no way to tell whether you want it treated as binary, so at least that much of the mode needs to be specified. -- Greg From scott+python-dev at scottdial.com Tue Apr 29 08:05:49 2008 From: scott+python-dev at scottdial.com (Scott Dial) Date: Tue, 29 Apr 2008 02:05:49 -0400 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: <4816B729.9090108@canterbury.ac.nz> References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <18454.24384.41749.272069@montanaro-dyndns-org.local> <18454.33797.918894.424380@montanaro-dyndns-org.local> <4816B729.9090108@canterbury.ac.nz> Message-ID: <4816BABD.9080708@scottdial.com> Greg Ewing wrote: > skip at pobox.com wrote: >> I'm not disagreeing with you, but it seems odd that os.fdopen doesn't >> simply >> obey the mode of the file descriptor it receives as its argument > > I'm not even sure if it's possible to find out the mode > that a file descriptor was opened with -- is it? > int mode = fcntl(fd, F_GETFL); switch(mode) { case O_RDONLY: ... case O_WRONLY: ... case O_RDWR: ... } > Certainly there's no way to tell whether you want it > treated as binary, so at least that much of the mode > needs to be specified. Agreed. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From martin at v.loewis.de Tue Apr 29 08:35:00 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 29 Apr 2008 08:35:00 +0200 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <4816AEDF.1060905@canterbury.ac.nz> Message-ID: <4816C194.9060103@v.loewis.de> > Same here. In fact, is there a good reason to have mkstemp() return the > fd (except backward compatibility)? Except for backwards compatibility: is there a good reason to keep os.mkstemp at all? The tradition is that all APIs in os expose the "system" calls as-is, i.e. they don't try to adjust the result values at all. This has the advantage that the caller can use their system's man page to find out all details, and trust that the Python wrapper does exactly the same. For mkstemp, there are two special issues: the C API modifies the string parameter, which cannot be exposed to Python as-is (hence the two result values), and it isn't a system call (at least not on POSIX), so it doesn't really need to be exposed, except perhaps to also provide this on non-POSIX systems where a regular POSIX implementation (in Python) would fail. Regards, Martin From dave.l.harrison at gmail.com Tue Apr 29 11:58:39 2008 From: dave.l.harrison at gmail.com (David Harrison) Date: Tue, 29 Apr 2008 19:58:39 +1000 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: <4816C194.9060103@v.loewis.de> References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <4816AEDF.1060905@canterbury.ac.nz> <4816C194.9060103@v.loewis.de> Message-ID: 2008/4/29 "Martin v. L?wis" : > > Same here. In fact, is there a good reason to have mkstemp() return the > > fd (except backward compatibility)? > > Except for backwards compatibility: is there a good reason to keep > os.mkstemp at all? Greg Ewing's use-case is one I've also had at times - ie. as a convenience function for creating a "somewhat temporary" file that is randomly named, but persists beyond the closing of the file. If the function doesn't stay in os it doesn't make any difference to me though :-) From skip at pobox.com Tue Apr 29 12:55:10 2008 From: skip at pobox.com (skip at pobox.com) Date: Tue, 29 Apr 2008 05:55:10 -0500 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: <4816AEDF.1060905@canterbury.ac.nz> References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <4816AEDF.1060905@canterbury.ac.nz> Message-ID: <18454.65166.166747.638082@montanaro-dyndns-org.local> Greg> I'd like to see a variation of mkstemp() that returns a file Greg> object instead of a file descriptor... http://bugs.python.org/issue2717 Comments welcome. Skip From ncoghlan at gmail.com Tue Apr 29 14:15:00 2008 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 29 Apr 2008 22:15:00 +1000 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: <4816B6ED.6030501@canterbury.ac.nz> References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <4816B6ED.6030501@canterbury.ac.nz> Message-ID: <48171144.6050706@gmail.com> Greg Ewing wrote: > Guido van Rossum wrote: >> IMO mkstemp() is a major pain because you have to use raw file >> descriptors on the return value. I'd much rather recommend >> [Named]TemporaryFile which return streams. > > The problem with NamedTemporaryFile is that it deletes > the file as soon as you close it, which makes the > named-ness of it rather useless, as far as I can see. > If you don't want that to happen, you have to use > mkstemp. That much has been fixed in 2.6 - you can now just pass "delete=False" to the NamedTemporaryFile constructor to stop it doing that. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Tue Apr 29 14:18:16 2008 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 29 Apr 2008 22:18:16 +1000 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <4816AEDF.1060905@canterbury.ac.nz> <4816C194.9060103@v.loewis.de> Message-ID: <48171208.9030604@gmail.com> David Harrison wrote: > 2008/4/29 "Martin v. L?wis" : >>> Same here. In fact, is there a good reason to have mkstemp() return the >> > fd (except backward compatibility)? >> >> Except for backwards compatibility: is there a good reason to keep >> os.mkstemp at all? > > Greg Ewing's use-case is one I've also had at times - ie. as a > convenience function for creating a "somewhat temporary" file that is > randomly named, but persists beyond the closing of the file. If the > function doesn't stay in os it doesn't make any difference to me > though :-) As of 2.6, Greg's use case is addressed by the new 'delete' parameter on tempfile.NamedTemporaryFile. The implementation of the tempfile module uses os.mkstemp() though, so getting rid of the latter might cause a few problems :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From rasky at develer.com Tue Apr 29 15:34:24 2008 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 29 Apr 2008 15:34:24 +0200 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: <48171208.9030604@gmail.com> References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <4816AEDF.1060905@canterbury.ac.nz> <4816C194.9060103@v.loewis.de> <48171208.9030604@gmail.com> Message-ID: <481723E0.8020608@develer.com> On 4/29/2008 2:18 PM, Nick Coghlan wrote: >>>> Same here. In fact, is there a good reason to have mkstemp() return the >>> > fd (except backward compatibility)? >>> >>> Except for backwards compatibility: is there a good reason to keep >>> os.mkstemp at all? >> >> Greg Ewing's use-case is one I've also had at times - ie. as a >> convenience function for creating a "somewhat temporary" file that is >> randomly named, but persists beyond the closing of the file. If the >> function doesn't stay in os it doesn't make any difference to me >> though :-) > > As of 2.6, Greg's use case is addressed by the new 'delete' parameter on > tempfile.NamedTemporaryFile. Then I personally don't have any objection to the removal of os.mkstemp. Since we're at it, a common pattern I use is to create temporary file to atomically replace files: I create a named temporary file in the same directory of the file I want to replace; write data into it; close it; and move it (POSIX "move": rename with silent overwrite) to the destination file. AFAIK, this is allows an atomic file replacemente on most filesystems. I believe this is a common useful pattern that could be handled in module tmpfile (especially since the final "rename" step requires a little care to be truly multiplatform). -- Giovanni Bajo Develer S.r.l. http://www.develer.com From tnelson at onresolve.com Tue Apr 29 15:58:02 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Tue, 29 Apr 2008 06:58:02 -0700 Subject: [Python-Dev] socket.try_reuse_address() Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E2477ED1F@EXMBX04.exchhosting.com> Since the recent changes to networking-oriented tests (issue 2550, r62234 and r62237), I think it's safe to say stability of the test suite on all the non-Windows platforms has improved significantly in this area (I don't recall seeing any socket errors in *nix buildbot logs since those commits). However, Windows buildbots are still periodically failing. More specifically, my Windows buildbots are still failing. One thing that's different about my buildbots is that two are being run at the same time for both trunk and py3k -- one doing an x86 build, the other doing x64 build. Since the changes in the aforementioned revisions, the behaviour of my buildbots has definitely improved -- they no longer completely wedge on test_async(chat|core), mainly due to abolishing all use of SO_REUSEADDR as a socket option in any network-oriented tests. However, periodically, they're still dying/failing in a variety of ways -- see relevant log snippets at the end of this e-mail for examples. I attribute this to the fact that SO_REUSEADDR is still set as a socket option in asyncore.py and SocketServer.py. Basically, SO_REUSEADDR should *never* be set on Windows for TCP/IP sockets. Using asyncore.py as an example, here are two ways we could handle this: 1. Hard code the Windows opt-out: --- asyncore.py (revision 62509) +++ asyncore.py (working copy) @@ -267,6 +267,8 @@ def set_reuse_addr(self): # try to re-use a server port if possible + if os.name == 'nt' and self.socket.socket_type != socket.SOCK_DGRAM: + return try: self.socket.setsockopt( socket.SOL_SOCKET, socket.SO_REUSEADDR, 2. Introduce socket.try_reuse_address(): --- asyncore.py (revision 62509) +++ asyncore.py (working copy) @@ -266,15 +266,7 @@ self.add_channel(map) def set_reuse_addr(self): - # try to re-use a server port if possible - try: - self.socket.setsockopt( - socket.SOL_SOCKET, socket.SO_REUSEADDR, - self.socket.getsockopt(socket.SOL_SOCKET, - socket.SO_REUSEADDR) | 1 - ) - except socket.error: - pass + self.socket.try_reuse_address() With try_use_address implemented as follows: --- socket.py (revision 62509) +++ socket.py (working copy) @@ -197,6 +197,10 @@ Return a new socket object connected to the same system resource.""" return _socketobject(_sock=self._sock) + def try_reuse_address(self): + if not (os.name == 'nt' and self._sock.type != SOCK_DGRAM): + self._sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) + def makefile(self, mode='r', bufsize=-1): """makefile([mode[, bufsize]]) -> file object I prefer the latter as it's cleaner, easier to document and encapsulates what we're trying to do relatively well. The affected modules would be asyncore.py, SocketServer.py and idlelib/rpc.py. Thoughts? Regards, Trent. test_ftplib remoteFailed: [Failure instance: Traceback (failure with no frames): twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion. ] test_asynchat test test_asynchat failed -- errors occurred; run in verbose mode for details [snip to bottom of log where test_asynchat is re-run] 1 test failed: test_asynchat 33 tests skipped: test__locale test_aepack test_applesingle test_cProfile test_commands test_crypt test_curses test_dbm test_epoll test_fcntl test_fork1 test_gdbm test_grp test_ioctl test_kqueue test_macostools test_mhlib test_nis test_openpty test_ossaudiodev test_pipes test_poll test_posix test_pty test_pwd test_resource test_scriptpackages test_signal test_syslog test_threadsignals test_wait3 test_wait4 test_zipfile64 Those skips are all expected on win32. Re-running failed tests in verbose mode Re-running test 'test_asynchat' in verbose mode test_close_when_done (test.test_asynchat.TestAsynchat) ... ok test_empty_line (test.test_asynchat.TestAsynchat) ... ok test_line_terminator1 (test.test_asynchat.TestAsynchat) ... ok test_line_terminator2 (test.test_asynchat.TestAsynchat) ... ok test_line_terminator3 (test.test_asynchat.TestAsynchat) ... ok test_none_terminator (test.test_asynchat.TestAsynchat) ... ok test_numeric_terminator1 (test.test_asynchat.TestAsynchat) ... ok test_numeric_terminator2 (test.test_asynchat.TestAsynchat) ... ok test_simple_producer (test.test_asynchat.TestAsynchat) ... ok test_string_producer (test.test_asynchat.TestAsynchat) ... ok test_close_when_done (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_empty_line (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_line_terminator1 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_line_terminator2 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_line_terminator3 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_none_terminator (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_numeric_terminator1 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_numeric_terminator2 (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_simple_producer (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_string_producer (test.test_asynchat.TestAsynchat_WithPoll) ... ok test_find_prefix_at_end (test.test_asynchat.TestHelperFunctions) ... ok test_basic (test.test_asynchat.TestFifo) ... ok test_given_list (test.test_asynchat.TestFifo) ... ok ---------------------------------------------------------------------- Ran 23 tests in 11.812s OK (Note that re-running the test here didn't result in the test failing again.) 1 test failed: test_smtplib Traceback (most recent call last): File "S:\buildbots\python\3.0.nelson-windows\build\lib\threading.py", line 493, in _bootstrap_inner self.run() File "S:\buildbots\python\3.0.nelson-windows\build\lib\threading.py", line 449, in run self._target(*self._args, **self._kwargs) File "S:\buildbots\python\3.0.nelson-windows\build\lib\test\test_smtplib.py", line 106, in debugging_server poll_fun(0.01, asyncore.socket_map) File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 132, in poll read(obj) File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 72, in read obj.handle_error() File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 68, in read obj.handle_read_event() File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 390, in handle_read_event self.handle_read() File "S:\buildbots\python\3.0.nelson-windows\build\lib\test\test_ssl.py", line 524, in handle_read data = self.recv(1024) File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 342, in recv data = self.socket.recv(buffer_size) File "S:\buildbots\python\3.0.nelson-windows\build\lib\ssl.py", line 247, in recv return self.read(buflen) File "S:\buildbots\python\3.0.nelson-windows\build\lib\ssl.py", line 162, in read v = self._sslobj.read(len or 1024) socket.error: [Errno 10053] An established connection was aborted by the software in your host machine From hrvoje.niksic at avl.com Tue Apr 29 16:16:06 2008 From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=) Date: Tue, 29 Apr 2008 16:16:06 +0200 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: <481723E0.8020608@develer.com> References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <4816AEDF.1060905@canterbury.ac.nz> <4816C194.9060103@v.loewis.de> <48171208.9030604@gmail.com> <481723E0.8020608@develer.com> Message-ID: <1209478566.7635.4.camel@localhost> On Tue, 2008-04-29 at 15:34 +0200, Giovanni Bajo wrote: > > As of 2.6, Greg's use case is addressed by the new 'delete' parameter on > > tempfile.NamedTemporaryFile. > > Then I personally don't have any objection to the removal of os.mkstemp. os.mkstemp is still useful when you *don't* need the file object, but the actual file descriptor, for passing to C code or to the child processes, or to mmap.mmap and such. It is also familiar to the Unix/C hackers, and it should cost very little to keep it around. > Since we're at it, a common pattern I use is to create temporary file to > atomically replace files: I create a named temporary file in the same > directory of the file I want to replace; write data into it; close it; > and move it (POSIX "move": rename with silent overwrite) to the > destination file. AFAIK, this is allows an atomic file replacemente on > most filesystems. > > I believe this is a common useful pattern that could be handled in > module tmpfile (especially since the final "rename" step requires a > little care to be truly multiplatform). I agree, having a tempfile class with rename-on-close semantics would be very useful. From gnewsg at gmail.com Tue Apr 29 16:43:04 2008 From: gnewsg at gmail.com (Giampaolo Rodola') Date: Tue, 29 Apr 2008 07:43:04 -0700 (PDT) Subject: [Python-Dev] socket.try_reuse_address() In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E2477ED1F@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E2477ED1F@EXMBX04.exchhosting.com> Message-ID: Maybe it would be better considering Windows CE systems too: - if os.name == 'nt' + if os.name in ('nt', 'ce') Moreover if the method is called "try_reuse_address" I'd expect that "self._sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1)" would be placed inside a try/except block. On 29 Apr, 15:58, Trent Nelson wrote: > Since the recent changes to networking-oriented tests (issue 2550, r62234 and r62237), I think it's safe to say stability of the test suite on all the non-Windows platforms has improved significantly in this area (I don't recall seeing any socket errors in *nix buildbot logs since those commits). > > However, Windows buildbots are still periodically failing. ?More specifically, my Windows buildbots are still failing. ?One thing that's different about my buildbots is that two are being run at the same time for both trunk and py3k -- one doing an x86 build, the other doing x64 build. > > Since the changes in the aforementioned revisions, the behaviour of my buildbots has definitely improved -- they no longer completely wedge on test_async(chat|core), mainly due to abolishing all use of SO_REUSEADDR as a socket option in any network-oriented tests. > > However, periodically, they're still dying/failing in a variety of ways -- see relevant log snippets at the end of this e-mail for examples. ?I attribute this to the fact that SO_REUSEADDR is still set as a socket option in asyncore.py and SocketServer.py. ?Basically, SO_REUSEADDR should *never* be set on Windows for TCP/IP sockets. ?Using asyncore.py as an example, here are two ways we could handle this: > > 1. Hard code the Windows opt-out: > --- asyncore.py (revision 62509) > +++ asyncore.py (working copy) > @@ -267,6 +267,8 @@ > > ? ? ?def set_reuse_addr(self): > ? ? ? ? ?# try to re-use a server port if possible > + ? ? ? ?if os.name == 'nt' and self.socket.socket_type != socket.SOCK_DGRAM: > + ? ? ? ? ? ?return > ? ? ? ? ?try: > ? ? ? ? ? ? ?self.socket.setsockopt( > ? ? ? ? ? ? ? ? ?socket.SOL_SOCKET, socket.SO_REUSEADDR, > > 2. Introduce socket.try_reuse_address(): > --- asyncore.py (revision 62509) > +++ asyncore.py (working copy) > @@ -266,15 +266,7 @@ > ? ? ? ? ?self.add_channel(map) > > ? ? ?def set_reuse_addr(self): > - ? ? ? ?# try to re-use a server port if possible > - ? ? ? ?try: > - ? ? ? ? ? ?self.socket.setsockopt( > - ? ? ? ? ? ? ? ?socket.SOL_SOCKET, socket.SO_REUSEADDR, > - ? ? ? ? ? ? ? ?self.socket.getsockopt(socket.SOL_SOCKET, > - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? socket.SO_REUSEADDR) | 1 > - ? ? ? ? ? ? ? ?) > - ? ? ? ?except socket.error: > - ? ? ? ? ? ?pass > + ? ? ? ?self.socket.try_reuse_address() > > With try_use_address implemented as follows: > > --- socket.py ? (revision 62509) > +++ socket.py ? (working copy) > @@ -197,6 +197,10 @@ > ? ? ? ? ?Return a new socket object connected to the same system resource.""" > ? ? ? ? ?return _socketobject(_sock=self._sock) > > + ? ?def try_reuse_address(self): > + ? ? ? ?if not (os.name == 'nt' and self._sock.type != SOCK_DGRAM): > + ? ? ? ? ? ?self._sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) > + > ? ? ?def makefile(self, mode='r', bufsize=-1): > ? ? ? ? ?"""makefile([mode[, bufsize]]) -> file object > > I prefer the latter as it's cleaner, easier to document and encapsulates what we're trying to do relatively well. ?The affected modules would be asyncore.py, SocketServer.py and idlelib/rpc.py. ?Thoughts? > > Regards, > > ? ? ? ? Trent. > > > test_ftplib > > remoteFailed: [Failure instance: Traceback (failure with no frames): twisted.internet.error.ConnectionLost: Connection to the other side was lost in a non-clean fashion. > ] > > > > test_asynchat > test test_asynchat failed -- errors occurred; run in verbose mode for details > [snip to bottom of log where test_asynchat is re-run] > 1 test failed: > ? ? test_asynchat > 33 tests skipped: > ? ? test__locale test_aepack test_applesingle test_cProfile > ? ? test_commands test_crypt test_curses test_dbm test_epoll > ? ? test_fcntl test_fork1 test_gdbm test_grp test_ioctl test_kqueue > ? ? test_macostools test_mhlib test_nis test_openpty test_ossaudiodev > ? ? test_pipes test_poll test_posix test_pty test_pwd test_resource > ? ? test_scriptpackages test_signal test_syslog test_threadsignals > ? ? test_wait3 test_wait4 test_zipfile64 > Those skips are all expected on win32. > Re-running failed tests in verbose mode > Re-running test 'test_asynchat' in verbose mode > test_close_when_done (test.test_asynchat.TestAsynchat) ... ok > test_empty_line (test.test_asynchat.TestAsynchat) ... ok > test_line_terminator1 (test.test_asynchat.TestAsynchat) ... ok > test_line_terminator2 (test.test_asynchat.TestAsynchat) ... ok > test_line_terminator3 (test.test_asynchat.TestAsynchat) ... ok > test_none_terminator (test.test_asynchat.TestAsynchat) ... ok > test_numeric_terminator1 (test.test_asynchat.TestAsynchat) ... ok > test_numeric_terminator2 (test.test_asynchat.TestAsynchat) ... ok > test_simple_producer (test.test_asynchat.TestAsynchat) ... ok > test_string_producer (test.test_asynchat.TestAsynchat) ... ok > test_close_when_done (test.test_asynchat.TestAsynchat_WithPoll) ... ok > test_empty_line (test.test_asynchat.TestAsynchat_WithPoll) ... ok > test_line_terminator1 (test.test_asynchat.TestAsynchat_WithPoll) ... ok > test_line_terminator2 (test.test_asynchat.TestAsynchat_WithPoll) ... ok > test_line_terminator3 (test.test_asynchat.TestAsynchat_WithPoll) ... ok > test_none_terminator (test.test_asynchat.TestAsynchat_WithPoll) ... ok > test_numeric_terminator1 (test.test_asynchat.TestAsynchat_WithPoll) ... ok > test_numeric_terminator2 (test.test_asynchat.TestAsynchat_WithPoll) ... ok > test_simple_producer (test.test_asynchat.TestAsynchat_WithPoll) ... ok > test_string_producer (test.test_asynchat.TestAsynchat_WithPoll) ... ok > test_find_prefix_at_end (test.test_asynchat.TestHelperFunctions) ... ok > test_basic (test.test_asynchat.TestFifo) ... ok > test_given_list (test.test_asynchat.TestFifo) ... ok > > ---------------------------------------------------------------------- > Ran 23 tests in 11.812s > > OK > > (Note that re-running the test here didn't result in the test failing again.) > > > 1 test failed: > ? ? test_smtplib > > Traceback (most recent call last): > ? File "S:\buildbots\python\3.0.nelson-windows\build\lib\threading.py", line 493, in _bootstrap_inner > ? ? self.run() > ? File "S:\buildbots\python\3.0.nelson-windows\build\lib\threading.py", line 449, in run > ? ? self._target(*self._args, **self._kwargs) > ? File "S:\buildbots\python\3.0.nelson-windows\build\lib\test\test_smtplib.py", line 106, in debugging_server > ? ? poll_fun(0.01, asyncore.socket_map) > ? File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 132, in poll > ? ? read(obj) > ? File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 72, in read > ? ? obj.handle_error() > ? File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 68, in read > ? ? obj.handle_read_event() > ? File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 390, in handle_read_event > ? ? self.handle_read() > ? File "S:\buildbots\python\3.0.nelson-windows\build\lib\test\test_ssl.py", line 524, in handle_read > ? ? data = self.recv(1024) > ? File "S:\buildbots\python\3.0.nelson-windows\build\lib\asyncore.py", line 342, in recv > ? ? data = self.socket.recv(buffer_size) > ? File "S:\buildbots\python\3.0.nelson-windows\build\lib\ssl.py", line 247, in recv > ? ? return self.read(buflen) > ? File "S:\buildbots\python\3.0.nelson-windows\build\lib\ssl.py", line 162, in read > ? ? v = self._sslobj.read(len or 1024) > socket.error: [Errno 10053] An established connection was aborted by the software in your host machine > > > _______________________________________________ > Python-Dev mailing list > Python-... at python.orghttp://mail.python.org/mailman/listinfo/python-dev > Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv... From steve at holdenweb.com Tue Apr 29 17:20:19 2008 From: steve at holdenweb.com (Steve Holden) Date: Tue, 29 Apr 2008 11:20:19 -0400 Subject: [Python-Dev] socket.try_reuse_address() In-Reply-To: References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E2477ED1F@EXMBX04.exchhosting.com> Message-ID: Giampaolo Rodola' wrote: > Maybe it would be better considering Windows CE systems too: > > - if os.name == 'nt' > + if os.name in ('nt', 'ce') > Cygwin? I don't know how Unix-like it is. regards Steve -- Steve Holden +1 571 484 6266 +1 800 494 3119 Holden Web LLC http://www.holdenweb.com/ From tnelson at onresolve.com Tue Apr 29 18:27:43 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Tue, 29 Apr 2008 09:27:43 -0700 Subject: [Python-Dev] socket.try_reuse_address() In-Reply-To: References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E2477ED1F@EXMBX04.exchhosting.com> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E2477EF81@EXMBX04.exchhosting.com> > Giampaolo Rodola' wrote: > > Maybe it would be better considering Windows CE systems too: > > > > - if os.name == 'nt' > > + if os.name in ('nt', 'ce') > > > Cygwin? I don't know how Unix-like it is. Yeah, that's a fair point, it's the behaviour of the underlying Winsock API we're targeting, so it would apply to Cygwin as well. (And CE and anything else on Windows.) Trent. From guido at python.org Tue Apr 29 21:09:02 2008 From: guido at python.org (Guido van Rossum) Date: Tue, 29 Apr 2008 12:09:02 -0700 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: <4816B6ED.6030501@canterbury.ac.nz> References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <4816B6ED.6030501@canterbury.ac.nz> Message-ID: On Mon, Apr 28, 2008 at 10:49 PM, Greg Ewing wrote: > The problem with NamedTemporaryFile is that it deletes > the file as soon as you close it, which makes the > named-ness of it rather useless, as far as I can see. Why? You can flush it and then all the data is on the disk. The whole point of [Named]TemporaryFile is to automate the cleanup as well as the creation. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From armin.ronacher at active-4.com Tue Apr 29 21:30:07 2008 From: armin.ronacher at active-4.com (Armin Ronacher) Date: Tue, 29 Apr 2008 19:30:07 +0000 (UTC) Subject: [Python-Dev] Module Suggestion: ast Message-ID: Hi all, I would like to propose a new module for the stdlib for Python 2.6 and higher: "ast". The motivation for this module is the pending deprecation for compiler.ast which is widely used (debugging, template engines, code coverage etc.). _ast is a very solid module and is without a doubt easier to maintain then compiler.ast which was written in Python, it's lacking some features such as pretty printing the AST or traversing it. The idea of "ast" would be adding high level functionality for easier working with the AST. It currently provides these features: - pretty printing AST objects - a parse function as easier alias for compile() + flag - operator-node -> operator symbol mappings (eg: _ast.Add -> '+') - methods to modify lineno / col_offset (incrementing or copying the data over from existing nodes) - getting the fields of nodes as dicts - iterating over all child nodes - a function to get the docstring or an AST node - a node walker that yields all child-nodes recursively - a `NodeVistor` and `NodeTransformer` Additionally there is a `literate_eval` function in that module that can safely evaluate python literals in a string. Module and unittests are located in the Pocoo Sandbox HG repository: http://dev.pocoo.org/hg/sandbox/file/tip/ast/ast.py http://dev.pocoo.org/hg/sandbox/file/tip/ast/test_ast.py A slightly modified version of the ast.py module for Python 2.5 compatibility is currently in use for the Mako template engine to achieve support for Google's AppEngine. An example module for the NodeVisitor is in the repository which converts a Python AST back into Python source code: http://dev.pocoo.org/hg/sandbox/file/tip/ast/codegen.py Regards, Armin From martin at v.loewis.de Tue Apr 29 23:35:17 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 29 Apr 2008 23:35:17 +0200 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <4816AEDF.1060905@canterbury.ac.nz> <4816C194.9060103@v.loewis.de> Message-ID: <48179495.8080808@v.loewis.de> >>> Same here. In fact, is there a good reason to have mkstemp() return the >> > fd (except backward compatibility)? >> >> Except for backwards compatibility: is there a good reason to keep >> os.mkstemp at all? > > Greg Ewing's use-case is one I've also had at times - ie. as a > convenience function for creating a "somewhat temporary" file that is > randomly named, but persists beyond the closing of the file. If the > function doesn't stay in os it doesn't make any difference to me > though :-) I was talking about the specific implementation, which happens to be a wrapper around the C library's mkstemp. For the use case in question, I think passing delete=False to NamedTemporaryFile would work just as well. Regards, Martin From martin at v.loewis.de Tue Apr 29 23:38:50 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 29 Apr 2008 23:38:50 +0200 Subject: [Python-Dev] Module Suggestion: ast In-Reply-To: References: Message-ID: <4817956A.6090507@v.loewis.de> > An example module for the NodeVisitor is in the repository which > converts a Python AST back into Python source code: > > http://dev.pocoo.org/hg/sandbox/file/tip/ast/codegen.py And another example for the same functionality is in Demo/parser/unparse.py. Regards, Martin From tom at vector-seven.com Wed Apr 30 01:22:27 2008 From: tom at vector-seven.com (Thomas Lee) Date: Wed, 30 Apr 2008 09:22:27 +1000 Subject: [Python-Dev] Module Suggestion: ast In-Reply-To: References: Message-ID: <4817ADB3.5000003@vector-seven.com> Just a thought, but it would be great if this could be implemented over the top of a C layer that operates on real AST nodes (rather than the PyObject representation of those nodes). I'm working on stuff to perform code optimization at the AST level (see the tlee-ast-optimize branch), and the functionality you're describing may wind up being very useful to me. I've got more to say on the topic, but I'm at work right now. Just something to keep in mind. Cheers, T Armin Ronacher wrote: > Hi all, > > I would like to propose a new module for the stdlib for Python 2.6 > and higher: "ast". The motivation for this module is the pending > deprecation for compiler.ast which is widely used (debugging, > template engines, code coverage etc.). _ast is a very solid module > and is without a doubt easier to maintain then compiler.ast which > was written in Python, it's lacking some features such as pretty > printing the AST or traversing it. > > The idea of "ast" would be adding high level functionality for > easier working with the AST. It currently provides these features: > > - pretty printing AST objects > - a parse function as easier alias for compile() + flag > - operator-node -> operator symbol mappings (eg: _ast.Add -> '+') > - methods to modify lineno / col_offset (incrementing or copying > the data over from existing nodes) > - getting the fields of nodes as dicts > - iterating over all child nodes > - a function to get the docstring or an AST node > - a node walker that yields all child-nodes recursively > - a `NodeVistor` and `NodeTransformer` > > Additionally there is a `literate_eval` function in that module that > can safely evaluate python literals in a string. > > Module and unittests are located in the Pocoo Sandbox HG repository: > > http://dev.pocoo.org/hg/sandbox/file/tip/ast/ast.py > http://dev.pocoo.org/hg/sandbox/file/tip/ast/test_ast.py > > A slightly modified version of the ast.py module for Python 2.5 > compatibility is currently in use for the Mako template engine to > achieve support for Google's AppEngine. > > An example module for the NodeVisitor is in the repository which > converts a Python AST back into Python source code: > > http://dev.pocoo.org/hg/sandbox/file/tip/ast/codegen.py > > > Regards, > Armin > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/tom%40vector-seven.com > From greg.ewing at canterbury.ac.nz Wed Apr 30 01:42:54 2008 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 30 Apr 2008 11:42:54 +1200 Subject: [Python-Dev] Warn about mktemp once again? In-Reply-To: References: <18452.65322.843226.299769@montanaro-dyndns-org.local> <18454.23330.147475.830232@montanaro-dyndns-org.local> <4816B6ED.6030501@canterbury.ac.nz> Message-ID: <4817B27E.8060704@canterbury.ac.nz> Guido van Rossum wrote: > Why? You can flush it and then all the data is on the disk. That might be all right on Unix, but I would be worried that having the file open could prevent some other things being done with it on some platforms, such as renaming. You might also want to pass the file name on to some other process. > The whole point of [Named]TemporaryFile is to automate the cleanup as > well as the creation. The docs (at least up to 2.5) don't make it at all clear that this is the *whole* point of *both* these functions. The doc for NamedTemporaryFile seems to disavow any guarantee that you will be able to do anything with the name while the file is open. If you can't use the name while the file is open, and the file ceases to exist when it's closed, then what use is it to have the name? The obvious conclusion is that the point of NamedTemporaryFile is to give you a file that *doesn't* go away when you close it, and has a name so you can go on to do other things with it. So I plead ignorance due to misleading documentation. -- Greg From tom at vector-seven.com Wed Apr 30 12:15:46 2008 From: tom at vector-seven.com (Thomas Lee) Date: Wed, 30 Apr 2008 20:15:46 +1000 Subject: [Python-Dev] Optimization of Python ASTs: How should we deal with constant values? Message-ID: <481846D2.6090509@vector-seven.com> Hi all, I've been working on optimization of the AST, including the porting of the old bytecode-level optimizations to the AST level. A few questions have come up in the process of doing this, all of which are probably appropriate for discussion on this list. The code I'm referring to here can be found in the tlee-ast-optimize branch. Most of the relevant code is in Python/optimize.c and Python/peephole.c. Nearly all of the bytecode-level optimizations have been moved to the AST optimizer with a few exceptions. Most of those waiting to be ported are stuck in limbo due to the fact we can't yet inject arbitrary PyObject constants into the AST. Examples are tuples of constants and the optimization of "LOAD_GLOBAL/LOAD_NAME None" as "LOAD_CONST None". This leaves us with a few options: 1. A new AST expr node for constant values for types other than Str/Num I imagine this to be something like Const(PyObject* v), which is effectively translated to a "LOAD_CONST v" by the compiler. This trades the purity of the AST for a little practicality. A "Const" node has no real source representation, it would exist solely for the purpose of injecting PyObject constants into the AST. 2. Allow arbitrary annotations to be applied to the AST as compiler hints. For example, each AST node might have an optional dict that contains a set of annotation values. Then when traversing the AST, the compiler might do something along the lines of: if (expr->annotations) { PyObject* constvalue = PyDict_GetItemString(expr->annotations, "constantvalue"); if (constvalue) ADDOP_O(c, LOAD_CONST, constvalue, consts) else VISIT(c, expr, expr) } This is a more general solution if we want to keep other compiler hints down the track, but unless somebody can think of another use case this is probably overkill. 3. Keep these particular optimizations at the bytecode level. It would be great to be able to perform the optimizations at a higher level, but this would require no effort at all. This would mean two passes over the same code at two different levels. If anybody familiar with this stuff could weigh in on the matter, it would be much appreciated. I've got a list of other issues that I need to discuss here, but this would be a great start. Thanks, Tom From flashk at gmail.com Wed Apr 30 18:36:44 2008 From: flashk at gmail.com (Farshid Lashkari) Date: Wed, 30 Apr 2008 09:36:44 -0700 Subject: [Python-Dev] cStringIO buffer interface Message-ID: <978d1eac0804300936j707324afude1c18ce12dba71f@mail.gmail.com> Hello, I'm not sure if this is the right place to ask, so I apologize ahead of time if it is. I was just curious as to why cStringIO objects don't implement the buffer interface. cStringIO objects seem similar to string and array objects, and those support the buffer protocol. Is there a reason against allowing cStringIO to support at least the read buffer interface, or is just that nobody has considered it until now? Thanks, Farshid From guido at python.org Wed Apr 30 18:51:25 2008 From: guido at python.org (Guido van Rossum) Date: Wed, 30 Apr 2008 09:51:25 -0700 Subject: [Python-Dev] cStringIO buffer interface In-Reply-To: <978d1eac0804300936j707324afude1c18ce12dba71f@mail.gmail.com> References: <978d1eac0804300936j707324afude1c18ce12dba71f@mail.gmail.com> Message-ID: On Wed, Apr 30, 2008 at 9:36 AM, Farshid Lashkari wrote: > I was just curious as to why cStringIO objects don't implement the > buffer interface. cStringIO objects seem similar to string and array > objects, and those support the buffer protocol. Is there a reason > against allowing cStringIO to support at least the read buffer > interface, or is just that nobody has considered it until now? Well, for one, it would mean you could no longer exchange a StringIO instance for a cStringIO instance. Also, what's the compelling use case you're thinking of? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tnelson at onresolve.com Wed Apr 30 19:40:37 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 30 Apr 2008 10:40:37 -0700 Subject: [Python-Dev] socket.try_reuse_address() References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E2477ED1F@EXMBX04.exchhosting.com> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E248257A2@EXMBX04.exchhosting.com> > > Giampaolo Rodola' wrote: > > > Maybe it would be better considering Windows CE systems too: > > > > > > - if os.name == 'nt' > > > + if os.name in ('nt', 'ce') > > > > > Cygwin? I don't know how Unix-like it is. > > Yeah, that's a fair point, it's the behaviour of the > underlying Winsock API we're targeting, so it would apply to > Cygwin as well. (And CE and anything else on Windows.) ....including Jython and IronPython -- which all exhibit the same undesirable behaviour on Windows when SO_REUSEADDR is set against a TCP/IP socket. Updated patch below. Assuming there are no objections, I'd like to clean this up and commit over the weekend, once I've updated the various parts of the stdlib currently using SO_REUSEADDR, as well as affected documentation. Index: socket.py =================================================================== --- socket.py (revision 62509) +++ socket.py (working copy) @@ -143,8 +143,18 @@ 'sendall', 'setblocking', 'settimeout', 'gettimeout', 'shutdown') +# Attempt to determine if we're running on Windows, irrespective of our Python +# incarnation. We need to know this so that we *don't* set the SO_REUSEADDR +# against TCP/IP sockets in socket.try_reuse_addr(). Note that IronPython is +# covered below as it sets os.name to 'nt'. if os.name == "nt": _socketmethods = _socketmethods + ('ioctl',) + _is_windows = True +elif os.name == 'java': + from java.lang import System + _is_windows = 'windows' in System.getProperty('os.name').lower() +elif os.name == 'posix' and sys.platform == 'cygwin': + _is_windows = True if sys.platform == "riscos": _socketmethods = _socketmethods + ('sleeptaskw',) @@ -197,6 +207,13 @@ Return a new socket object connected to the same system resource.""" return _socketobject(_sock=self._sock) + def try_reuse_address(self): + if not (_is_windows and self._sock.type != SOCK_DGRAM): + try: + self._sock.setsockopt(SOL_SOCKET, SO_REUSEADDR, 1) + except socket.error: + pass + def makefile(self, mode='r', bufsize=-1): """makefile([mode[, bufsize]]) -> file object Trent. From tnelson at onresolve.com Wed Apr 30 19:42:48 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 30 Apr 2008 10:42:48 -0700 Subject: [Python-Dev] socket.try_reuse_address() References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E2477ED1F@EXMBX04.exchhosting.com> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E248257AA@EXMBX04.exchhosting.com> > if os.name == "nt": > _socketmethods = _socketmethods + ('ioctl',) > + _is_windows = True > +elif os.name == 'java': > + from java.lang import System > + _is_windows = 'windows' in System.getProperty('os.name').lower() > +elif os.name == 'posix' and sys.platform == 'cygwin': > + _is_windows = True Oops, that last line should have been: elif os.name == 'ce' or (os.name == 'posix' and sys.platform == 'cygwin'): _is_windows = True From brett at python.org Wed Apr 30 19:49:01 2008 From: brett at python.org (Brett Cannon) Date: Wed, 30 Apr 2008 10:49:01 -0700 Subject: [Python-Dev] Optimization of Python ASTs: How should we deal with constant values? In-Reply-To: <481846D2.6090509@vector-seven.com> References: <481846D2.6090509@vector-seven.com> Message-ID: On Wed, Apr 30, 2008 at 3:15 AM, Thomas Lee wrote: > Hi all, > > I've been working on optimization of the AST, including the porting of the > old bytecode-level optimizations to the AST level. A few questions have come > up in the process of doing this, all of which are probably appropriate for > discussion on this list. The code I'm referring to here can be found in the > tlee-ast-optimize branch. Most of the relevant code is in Python/optimize.c > and Python/peephole.c. > > Nearly all of the bytecode-level optimizations have been moved to the AST > optimizer with a few exceptions. Most of those waiting to be ported are > stuck in limbo due to the fact we can't yet inject arbitrary PyObject > constants into the AST. Examples are tuples of constants and the > optimization of "LOAD_GLOBAL/LOAD_NAME None" as "LOAD_CONST None". > > This leaves us with a few options: > > 1. A new AST expr node for constant values for types other than Str/Num > > I imagine this to be something like Const(PyObject* v), which is > effectively translated to a "LOAD_CONST v" by the compiler. This trades the > purity of the AST for a little practicality. A "Const" node has no real > source representation, it would exist solely for the purpose of injecting > PyObject constants into the AST. > Slight issue with this is that people like Jython are standardizing on our AST, so adding to it does up their burden if they have no use for it. > 2. Allow arbitrary annotations to be applied to the AST as compiler hints. > > For example, each AST node might have an optional dict that contains a set > of annotation values. Then when traversing the AST, the compiler might do > something along the lines of: > > if (expr->annotations) { > PyObject* constvalue = PyDict_GetItemString(expr->annotations, > "constantvalue"); > if (constvalue) > ADDOP_O(c, LOAD_CONST, constvalue, consts) > else > VISIT(c, expr, expr) > } > > This is a more general solution if we want to keep other compiler hints > down the track, but unless somebody can think of another use case this is > probably overkill. > Possibly, but the AST stuff is so new who knows if it truly will be overkill or not down the road. I personally prefer this approach. > 3. Keep these particular optimizations at the bytecode level. > > It would be great to be able to perform the optimizations at a higher > level, but this would require no effort at all. This would mean two passes > over the same code at two different levels. > This might end up being the case, but I would rather avoid the overhead if possible. The question becomes how many other possible optimizations might come up that would only work reasonably at the bytecode level. And you forgot option 4): ditching the optimizations that only work on the bytecode. I am not advocating this, but it is an option. -Brett From draghuram at gmail.com Wed Apr 30 20:01:35 2008 From: draghuram at gmail.com (Raghuram Devarakonda) Date: Wed, 30 Apr 2008 14:01:35 -0400 Subject: [Python-Dev] socket.try_reuse_address() In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E248257AA@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E2477ED1F@EXMBX04.exchhosting.com> <87D3F9C72FBF214DB39FA4E3FE618CDC6E248257AA@EXMBX04.exchhosting.com> Message-ID: <2c51ecee0804301101i7648951v78960bc539c54cde@mail.gmail.com> On Wed, Apr 30, 2008 at 1:42 PM, Trent Nelson wrote: > > if os.name == "nt": > > _socketmethods = _socketmethods + ('ioctl',) > > + _is_windows = True > > +elif os.name == 'java': > > + from java.lang import System > > + _is_windows = 'windows' in System.getProperty('os.name').lower() This one will not work. >>> 'windows' in System.getProperty('os.name').lower() Traceback (innermost last): File "", line 1, in ? TypeError: string member test needs char left operand >>> You may have to do something like System.getProperty('os.name').lower().find('windows'). From tnelson at onresolve.com Wed Apr 30 20:07:37 2008 From: tnelson at onresolve.com (Trent Nelson) Date: Wed, 30 Apr 2008 11:07:37 -0700 Subject: [Python-Dev] socket.try_reuse_address() In-Reply-To: <2c51ecee0804301101i7648951v78960bc539c54cde@mail.gmail.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E2477ED1F@EXMBX04.exchhosting.com> <87D3F9C72FBF214DB39FA4E3FE618CDC6E248257AA@EXMBX04.exchhosting.com> <2c51ecee0804301101i7648951v78960bc539c54cde@mail.gmail.com> Message-ID: <87D3F9C72FBF214DB39FA4E3FE618CDC6E24825805@EXMBX04.exchhosting.com> > This one will not work. > > >>> 'windows' in System.getProperty('os.name').lower() > Traceback (innermost last): > File "", line 1, in ? > TypeError: string member test needs char left operand > >>> Interesting, which version of Jython were you using? > You may have to do something like > System.getProperty('os.name').lower().find('windows'). That didn't work for me. I assume the following works for you: System.getProperty('os.name').lower().startswith('windows') From draghuram at gmail.com Wed Apr 30 20:22:49 2008 From: draghuram at gmail.com (Raghuram Devarakonda) Date: Wed, 30 Apr 2008 14:22:49 -0400 Subject: [Python-Dev] socket.try_reuse_address() In-Reply-To: <87D3F9C72FBF214DB39FA4E3FE618CDC6E24825805@EXMBX04.exchhosting.com> References: <87D3F9C72FBF214DB39FA4E3FE618CDC6E2477ED1F@EXMBX04.exchhosting.com> <87D3F9C72FBF214DB39FA4E3FE618CDC6E248257AA@EXMBX04.exchhosting.com> <2c51ecee0804301101i7648951v78960bc539c54cde@mail.gmail.com> <87D3F9C72FBF214DB39FA4E3FE618CDC6E24825805@EXMBX04.exchhosting.com> Message-ID: <2c51ecee0804301122s730bf86fy31f7cfa4883b6e98@mail.gmail.com> On Wed, Apr 30, 2008 at 2:07 PM, Trent Nelson wrote: > > > This one will not work. > > > > >>> 'windows' in System.getProperty('os.name').lower() > > Traceback (innermost last): > > File "", line 1, in ? > > TypeError: string member test needs char left operand > > >>> > > Interesting, which version of Jython were you using? 2.1. Now that you mentioned it, I tested with 2.2 and your code works there. > > You may have to do something like > > System.getProperty('os.name').lower().find('windows'). > > That didn't work for me. I assume the following works for you: > > System.getProperty('os.name').lower().startswith('windows') It does. From exarkun at divmod.com Wed Apr 30 20:38:04 2008 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Wed, 30 Apr 2008 14:38:04 -0400 Subject: [Python-Dev] cStringIO buffer interface In-Reply-To: Message-ID: <20080430183804.6859.262788219.divmod.quotient.57183@ohm> On Wed, 30 Apr 2008 09:51:25 -0700, Guido van Rossum wrote: >On Wed, Apr 30, 2008 at 9:36 AM, Farshid Lashkari wrote: >> I was just curious as to why cStringIO objects don't implement the >> buffer interface. cStringIO objects seem similar to string and array >> objects, and those support the buffer protocol. Is there a reason >> against allowing cStringIO to support at least the read buffer >> interface, or is just that nobody has considered it until now? > >Well, for one, it would mean you could no longer exchange a StringIO >instance for a cStringIO instance. It would probably only mean that there is one further incompatibility between cStringIO and StringIO - you already can't exchange them in a number of cases. They handle unicode differently, they have different methods, etc. Maybe making them diverge even further is a step in the wrong direction, though. >Also, what's the compelling use case you're thinking of? I'm not sure what use-case Farshid Lashkari had. For Twisted, it has been considered as a way to reduce peak memory usage (by reducing the need for memory copying, which also speeds things up). I'm not sure if anyone has benchmarked this yet, so I don't know if it's a real win or not. I think Thomas Herv? has a patch to cStringIO which implements the feature, though. For reference, . This isn't high on my priority list, but I thought I'd point out the potential use-case. Jean-Paul From martin at v.loewis.de Wed Apr 30 22:36:36 2008 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 30 Apr 2008 22:36:36 +0200 Subject: [Python-Dev] Optimization of Python ASTs: How should we deal with constant values? In-Reply-To: <481846D2.6090509@vector-seven.com> References: <481846D2.6090509@vector-seven.com> Message-ID: <4818D854.1060906@v.loewis.de> > This leaves us with a few options: 5. Reuse/Abuse Num(object) for arbitrary constants. AFAICT, this should work out of the box. > 1. A new AST expr node for constant values for types other than Str/Num > > I imagine this to be something like Const(PyObject* v), which is > effectively translated to a "LOAD_CONST v" by the compiler. This trades > the purity of the AST for a little practicality. A "Const" node has no > real source representation, it would exist solely for the purpose of > injecting PyObject constants into the AST. I think this is the way to go. It doesn't violate purity: it is an *abstract* syntax, meaning that there doesn't need to be a 1:1 relationship to source syntax. However, it is still possible to reproduce source code from this Const node. I also don't worry about Jython conflicts. The grammar has a version number precisely so that you can refer to a specific version if you need to. Regards, Martin From lists at cheimes.de Wed Apr 30 23:17:03 2008 From: lists at cheimes.de (Christian Heimes) Date: Wed, 30 Apr 2008 23:17:03 +0200 Subject: [Python-Dev] Module properties for C modules Message-ID: <4818E1CF.7030903@cheimes.de> Hello Python Dev! As you all know modules don't support properties. However several places and modules could use properties on module level. For example the sys.py3k_warnings flag could be implemented with a property. Other flags in the sys module could benefit from read only properties, too. How do you like the general idea of properties for builtin modules. That is modules written in C like the sys module. Christian From musiccomposition at gmail.com Wed Apr 30 23:22:51 2008 From: musiccomposition at gmail.com (Benjamin Peterson) Date: Wed, 30 Apr 2008 16:22:51 -0500 Subject: [Python-Dev] Module properties for C modules In-Reply-To: <4818E1CF.7030903@cheimes.de> References: <4818E1CF.7030903@cheimes.de> Message-ID: <1afaf6160804301422g6beff08aya616eebdda6ee874@mail.gmail.com> On Wed, Apr 30, 2008 at 4:17 PM, Christian Heimes wrote: > Hello Python Dev! > > As you all know modules don't support properties. However several places > and modules could use properties on module level. For example the > sys.py3k_warnings flag could be implemented with a property. Other flags > in the sys module could benefit from read only properties, too. Big +1. Frankly, the get/set methods of sys are quite ugly! > > How do you like the general idea of properties for builtin modules. That > is modules written in C like the sys module. Good idea. Perhaps eventually they could be extended to Python, but they are definitely useful in C now. How about passing a list of getsets to PyImport_InitModule(5)? > > Christian > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com > -- Cheers, Benjamin Peterson From dev.python.heiko at weinen.org Wed Apr 30 22:55:31 2008 From: dev.python.heiko at weinen.org (Heiko Weinen) Date: Wed, 30 Apr 2008 22:55:31 +0200 Subject: [Python-Dev] Another GSoC Student Introduction Message-ID: <20080430205531.GA15767@vortex.local.net> Hi! My name is Heiko N. Weinen and I was accepted as GSoC Student, too. Hoorray! The project i chose is about Filesystem Virtualisation for Python's Standard Library, hopefully something which will prove quite useful. :) Christian Heimes is my Mentor and I'd like to thank him right now for his help. Feel free to contact me, if you have any further questions or related ideas! I will set up a project site at http://heiko.weinen.org/gsoc08 in a few days. Regards, Heiko -- Lubarsky's Law of Cybernetic Entomology: prov. "There is always one more bug." From lists at cheimes.de Wed Apr 30 23:32:57 2008 From: lists at cheimes.de (Christian Heimes) Date: Wed, 30 Apr 2008 23:32:57 +0200 Subject: [Python-Dev] Module properties for C modules In-Reply-To: <1afaf6160804301422g6beff08aya616eebdda6ee874@mail.gmail.com> References: <4818E1CF.7030903@cheimes.de> <1afaf6160804301422g6beff08aya616eebdda6ee874@mail.gmail.com> Message-ID: <4818E589.1070400@cheimes.de> Benjamin Peterson schrieb: > Good idea. Perhaps eventually they could be extended to Python, but > they are definitely useful in C now. How about passing a list of > getsets to PyImport_InitModule(5)? Yeah, I've a similar idea with PyImport_InitModule5() and a list of structs containing name, getter, setter, deleter, docstring. The module struct has to gain an additional slot which may contain a dict of names -> propertyobjects. Christian From dickinsm at gmail.com Wed Apr 30 23:06:01 2008 From: dickinsm at gmail.com (Mark Dickinson) Date: Wed, 30 Apr 2008 17:06:01 -0400 Subject: [Python-Dev] Debian/alpha test_math failures Message-ID: <5c6f2a5d0804301406o1f20481fwc28ac7f46e211e93@mail.gmail.com> Hello all, test_math is currently failing on the Debian/alpha buildbots (trunk and py3k). I've been trying, unsuccessfully, to figure out what's going wrong, and I'm starting to run out of ideas, so I thought I'd ask the list for help to see if anyone has any useful suggestions. Details of the failure: running the following code: from math import log ar = 9.88e-324 x = log(ar) produces: ValueError: math domain error Somehow, it looks like errno is getting set to something nonzero in math_1_from_whatever in mathmodule.c, but I really can't figure out how. I've tried adding '-mieee' to the gcc compile flags, and I've added a bunch of autoconf tests to verify that log(9.88e-324) succeeds, produces roughly the right result, and doesn't set errno to anything nonzero. All the autoconf tests that should pass do. So I can't find anything wrong with the libm implementation of log. test_math is fine on Tru64/alpha. Does anyone have access to a Linux/alpha machine, and a few minutes to figure out exactly what's failing? Or any suggestions about what might be failing. I'm open to wild ideas at this point... :-) Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Apr 30 23:41:42 2008 From: guido at python.org (Guido van Rossum) Date: Wed, 30 Apr 2008 14:41:42 -0700 Subject: [Python-Dev] Module properties for C modules In-Reply-To: <4818E1CF.7030903@cheimes.de> References: <4818E1CF.7030903@cheimes.de> Message-ID: On Wed, Apr 30, 2008 at 2:17 PM, Christian Heimes wrote: > As you all know modules don't support properties. However several places > and modules could use properties on module level. For example the > sys.py3k_warnings flag could be implemented with a property. Other flags > in the sys module could benefit from read only properties, too. > > How do you like the general idea of properties for builtin modules. That > is modules written in C like the sys module. But wouldn't this mean that those properties would no longer be available in the module's __dict__? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From armin.ronacher at active-4.com Wed Apr 30 23:46:22 2008 From: armin.ronacher at active-4.com (Armin Ronacher) Date: Wed, 30 Apr 2008 21:46:22 +0000 (UTC) Subject: [Python-Dev] Problems with the new super() Message-ID: Hi all, I blogged about that topic today which turned out to be a very bad idea, so I summarize it for the mailinglist here to hopefully start a discussion about the topic, which I think is rather important. In the last weeks something remarkable happened in the Python3 sources: self kinda became implicit. Not in function definitions, but in super calls. But not only self: also the class passed to `super`. That's interesting because it means that the language shifts into a completely different direction. `super` was rarely used in the past, mainly because it was weird to use. In the most common use case the current class and the current instance where passed to it, and the super typed returned looked up the parent methods on the MRO for you. It was useful for multiple inheritance and mixin classes that don't know their parent but confusing for many. I can see that a replacement is a good idea, but I'm not so sure if the current implementation is the way to go. The main problem with replacing `super(Foo, self).bar()` with something like `super.bar()` is obviously that self is explicit and the class (in that case Foo) can't be determined by the caller. Furthermore the Python principle was always against functions doing stack introspection to find the caller. There are few examples in the stdlib or builtins that do some sort of caller introspection. Those are the special functions `vars`, `locals`, `gloabal`, `vars` and some functions in the inspect module. And all of them do nothing more than getting the current frame and accessing the dict of locals or globals. What super in current Python 3 builds does goes way beyond that. The implementation of the automatic super currently has two ugly details that I think violate the Python Zen: The bytecode generated is differently if the name "super" is used in the function. `__class__` is only added as cell to the code if `super` or `__class__` is referenced. That and the fact that `f_localsplus` is completely unavailable from within python makes the whole process appear magical. This is way more magical than anything we've had in Python in the past and just doesn't fit into the language in my opinion. We do have an explicit self in methods and methods are more or less just functions. Python's methods are functions, just that a descriptor puts a method object around it to pass the self as first arguments. That's an incredible cool thing to have and makes things very simple and non-magical. Breaking that principle by introducing an automatic super seems to harm the concept. Another odd thing is that Python 3 starts keeping information on the C layer we can't access from within Python. Super is one example, another good one are methods. They don't have a descriptor that wraps them if they are accessed via their classes. This as such is not a problem as you can call them the same (just that you can call them with completely different receivers now) but it becomes a problem if some of the functions are marked as staticmethods. Then they look completely the same when looking at them from a classes perspective: | >>> class C: | ... normal = lambda x: None | ... static = staticmethod(lambda x: None) | ... | >>> type(C.normal) is type(C.static) | True | >>> C.normal | at 0x4da150> As far as I can see a documentation tool has no chance to keep them apart even though they are completely different on an instance: | >>> type(C().normal) is type(C().static) | False | >>> C().normal | of <__main__.C object at 0x4dbcf0>> | >>> C().static | at 0x4da198> While I don't knwo about the method thing, I think an automatic super should at least be implementable from within Python. I could imagine that by adding __class__ and __self__ to scopes automatically a lot of that magic could be removed. Regards, Armin From guido at python.org Wed Apr 30 23:57:32 2008 From: guido at python.org (Guido van Rossum) Date: Wed, 30 Apr 2008 14:57:32 -0700 Subject: [Python-Dev] Problems with the new super() In-Reply-To: References: Message-ID: The staticmethod thing isn't new; that's also the case in 2.x. The super() thing is a case of practicality beats purity. Note that you pay a small but measurable cost for the implicit __class__ (it's implemented as a "cell variable", the same mechanism used for nested scopes) so we wouldn't want to introduce it unless it is used. On Wed, Apr 30, 2008 at 2:46 PM, Armin Ronacher wrote: > Hi all, > > I blogged about that topic today which turned out to be a very bad idea, > so I summarize it for the mailinglist here to hopefully start a discussion > about the topic, which I think is rather important. > > In the last weeks something remarkable happened in the Python3 sources: > self kinda became implicit. Not in function definitions, but in super > calls. But not only self: also the class passed to `super`. That's > interesting because it means that the language shifts into a completely > different direction. > > `super` was rarely used in the past, mainly because it was weird to use. > In the most common use case the current class and the current instance > where passed to it, and the super typed returned looked up the parent > methods on the MRO for you. It was useful for multiple inheritance and > mixin classes that don't know their parent but confusing for many. I can > see that a replacement is a good idea, but I'm not so sure if the current > implementation is the way to go. > > The main problem with replacing `super(Foo, self).bar()` with something > like `super.bar()` is obviously that self is explicit and the class (in > that case Foo) can't be determined by the caller. Furthermore the Python > principle was always against functions doing stack introspection to find > the caller. There are few examples in the stdlib or builtins that do > some sort of caller introspection. Those are the special functions > `vars`, `locals`, `gloabal`, `vars` and some functions in the inspect > module. And all of them do nothing more than getting the current frame > and accessing the dict of locals or globals. What super in current > Python 3 builds does goes way beyond that. > > The implementation of the automatic super currently has two ugly details > that I think violate the Python Zen: The bytecode generated is differently > if the name "super" is used in the function. `__class__` is only added as > cell to the code if `super` or `__class__` is referenced. That and the fact > that `f_localsplus` is completely unavailable from within python makes the > whole process appear magical. > > This is way more magical than anything we've had in Python in the past and > just doesn't fit into the language in my opinion. We do have an explicit > self in methods and methods are more or less just functions. Python's > methods are functions, just that a descriptor puts a method object around > it to pass the self as first arguments. That's an incredible cool thing > to have and makes things very simple and non-magical. Breaking that > principle by introducing an automatic super seems to harm the concept. > > Another odd thing is that Python 3 starts keeping information on the C > layer we can't access from within Python. Super is one example, another > good one are methods. They don't have a descriptor that wraps them if > they are accessed via their classes. This as such is not a problem as you > can call them the same (just that you can call them with completely > different receivers now) but it becomes a problem if some of the functions > are marked as staticmethods. Then they look completely the same when > looking at them from a classes perspective: > > | >>> class C: > | ... normal = lambda x: None > | ... static = staticmethod(lambda x: None) > | ... > | >>> type(C.normal) is type(C.static) > | True > | >>> C.normal > | at 0x4da150> > > As far as I can see a documentation tool has no chance to keep them apart > even though they are completely different on an instance: > > | >>> type(C().normal) is type(C().static) > | False > | >>> C().normal > | of <__main__.C object at 0x4dbcf0>> > | >>> C().static > | at 0x4da198> > > While I don't knwo about the method thing, I think an automatic super should > at least be implementable from within Python. I could imagine that by adding > __class__ and __self__ to scopes automatically a lot of that magic could be > removed. > > Regards, > Armin > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gvwilson at cs.utoronto.ca Wed Apr 23 01:08:12 2008 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Tue, 22 Apr 2008 23:08:12 -0000 Subject: [Python-Dev] Encoding detection in the standard library? In-Reply-To: <08Apr22.144234pdt."58696"@synergy1.parc.xerox.com> References: <79235B51-32E4-4E88-AE15-CB1A8A621943@cs.toronto.edu> <18444.50265.618909.751390@montanaro-dyndns-org.local> <91A34FC2-2DAA-45D7-8EEC-91BFF42BDC31@cs.toronto.edu> <480D079A.2020904@v.loewis.de> <91478133-664C-4F29-90CB-D0985BE01F24@cs.toronto.edu> <480D69E9.7020004@v.loewis.de> <091D7293-363C-4766-8931-CFE6A9E72331@cs.toronto.edu> <480E2918.7090006@v.loewis.de> <08Apr22.144234pdt."58696"@synergy1.parc.xerox.com> Message-ID: > Bill Janssen: > Since the site that receives the POST doesn't necessarily have access to > the Web page that originally contained the form, that's not really > helpful. However, POSTs can use the MIME type "multipart/form-data" for > non-Latin-1 content, and should. That contains facilities for > indicating the encoding and other things as well. Yup, but DrProject (the target application) also serves as a relay and archive for email. We have no control over the agent used for composition, and AFAIK there's no standard way to include encoding information. Thanks, Greg