From noreply at sourceforge.net Mon Aug 1 03:29:48 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 31 Jul 2005 18:29:48 -0700 Subject: [Patches] [ python-Patches-1121611 ] sha and md5 modules should use OpenSSL when possible Message-ID: Patches item #1121611, was opened at 2005-02-12 17:33 Message generated for change (Comment added) made by greg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121611&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Gregory P. Smith (greg) Assigned to: Gregory P. Smith (greg) Summary: sha and md5 modules should use OpenSSL when possible Initial Comment: The md5 and sha (sha1) modules should use OpenSSL for the algorithms when it is available as its implementations are much faster than pythons own. Attached is an initial patch to use OpenSSL for the sha module. Its not ready for committing as is yet, but it is setup to be a generic base for all OpenSSL hashes with a little bit of work in the future. Tossing this out there for people to see how trivial it is and enjoy the speedups. diff is against HEAD but it should apply to 2.4 just fine. ---------------------------------------------------------------------- >Comment By: Gregory P. Smith (greg) Date: 2005-07-31 18:29 Message: Logged In: YES user_id=413 per arigo's suggestion I have a version of _hashopenssl.c in my sandbox modified to use the more modern C type API. The constructor is slightly faster (~1-2%) and does seem like a better way to do things. i'll post it after polishing it up. ---------------------------------------------------------------------- Comment By: Terry J. Reedy (tjreedy) Date: 2005-06-12 13:35 Message: Logged In: YES user_id=593130 Re Doc page: As a somewhat naive (relative to the subject) reader, the title and first sentence implied that 'secure hash' and 'message digest' are two separate things, whereas, judging from the .digest() blurb, they both seem to be16-byte hashes. So I would prefer this equivalence and the actual meaning were made clear at the top. Something like "This module implements a common interface to several secure hash or message digest algorithms that produce 16-byte hashes." If, as I presume, xx.hexdigest() == binascii.hexlify(xx.digest()), then I would say so and reference binsacii for the interconversion functions one would need if one had the two versions to compare or needed to convert after the extraction. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2005-06-12 05:18 Message: Logged In: YES user_id=4771 On a side note, maybe it makes sense for a new module like this one to promote and use the modern (>=2.2) ways of defining C types. What I have in mind is using tp_methods instead of Py_FindMethod, and generally not reverting to strcmp(). In this case, the constants like 'digest_size' would be best stored as class attributes instead, if possible. Indeed, allowing expressions like "hashlib.md5.digest_size" conveys the idea that the result doesn't depend on a particular instance, unlike "hashlib.md5().digest_size". (Of course class attributes are also readable from the instance, as usual.) I can give it a try if you don't want to invest more time in this patch than you already did (for which we are grateful to you :-) ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-06-11 20:21 Message: Logged In: YES user_id=413 Ok, this patch is ready. documentation has been added. I'll bring it up on python-dev for discussion/approval with a link to the htmlified documentation. The speedups are great for any application hashing a lot of data when OpenSSL is used. It also adds a sha224, sha256, sha384 and sha512 support. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-12 17:13 Message: Logged In: YES user_id=413 I linked a _hashlib.so library statically against openssl and reran the speed test. no change. that means its not shared library overhead causing the higher startup time but just an artifact of the OpenSSL EVP interface. Next up, analyze what size things common heavy sha1 using applications regularly hash (BitTorrent and such). ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-10 00:09 Message: Logged In: YES user_id=413 The 007 patch improves the speed of the constructor. There is still a potential speed issue with the constructor/destructor to work on: greg at spiff src $ ./python Lib/test/test_hashlib_speed.py _sha testing speed of old _sha legacy interface 0.06 seconds [20000 creations] 0.24 seconds [20000 "" digests] 0.15 seconds 20 x 106201 bytes [huge data] 0.15 seconds 200 x 10620 bytes [large data] 0.17 seconds 2000 x 1062 bytes [medium data] 0.35 seconds 20020 x 106 bytes [small data] 1.37 seconds 106200 x 20 bytes [digest_size data] 2.75 seconds 212400 x 10 bytes [tiny data] greg at spiff src $ ./python Lib/test/test_hashlib_speed.py sha1 testing speed of hashlib.sha1 0.22 seconds [20000 creations] 0.57 seconds [20000 "" digests] 0.09 seconds 20 x 106201 bytes [huge data] 0.09 seconds 200 x 10620 bytes [large data] 0.15 seconds 2000 x 1062 bytes [medium data] 0.71 seconds 20020 x 106 bytes [small data] 3.39 seconds 106200 x 20 bytes [digest_size data] 6.70 seconds 212400 x 10 bytes [tiny data] I suspect the cause is either or both of the shared openssl library call overhead or the openssl EVP abstraction interface. The speed results are very similar to the above regardless of which digest is used (the above was a celeron 333mhz running linux). ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-03 13:15 Message: Logged In: YES user_id=413 hashlib-006.patch adds fast constructors and a speed test. documentation is the next step. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-01 01:14 Message: Logged In: YES user_id=413 hashlib-005.patch now passes its test suite and no problems appear in valgrind. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-02-28 10:11 Message: Logged In: YES user_id=413 a much updated patch (hashlib-patch-004.patch). it incorporates some suggestions as well as including sf patch 935454's sha256/224 and sha512/384 implementations. still not complete but shows the direction its going in (i see a segfault part way thru the test suite after running the sha512 tests). as for the private modules being under another package, i see no reason to do that since there aren't very many (how does that work for binary modules anyways?). ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2005-02-18 11:21 Message: Logged In: YES user_id=764593 Should the private modules (such as _sha) be placed in a crypto package, instead of directly in the parent/everything library? ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-02-16 22:46 Message: Logged In: YES user_id=413 hashes-openssl-002.patch replaces the sha and md5 modules with a general hashes module that wraps all hashes that OpenSSL supports. note that OpenSSLs implementations are much faster than the previous python versions as it choses versions optimized for your particular hardware. Incase python is compiled without openssl the hashes wrapper falls back on the old python sha and md5 module implementations. side note: This may be sufficient for the Debian folks to work around their random odd licensing issue. just have debian python depend on openssl; use this and remove the old md5 module/code that wouldn't get used anyways. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121611&group_id=5470 From noreply at sourceforge.net Mon Aug 1 12:10:29 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 01 Aug 2005 03:10:29 -0700 Subject: [Patches] [ python-Patches-1223381 ] PEP 342 Generator enhancements Message-ID: Patches item #1223381, was opened at 2005-06-19 00:08 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1223381&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Phillip J. Eby (pje) Assigned to: Phillip J. Eby (pje) Summary: PEP 342 Generator enhancements Initial Comment: The attached patch implements code and test changes for: * yield expressions * bare yield (short for yield None) * yield in try/finally * generator.send(value) (send value into generator; substituted for PEP 342's next(arg)) * generator.throw(typ[,val[,tb]]) (raise error in generator) * generator.close() (per PEP 343) * GeneratorExit built-in exception type * generator.__del__ (well, the C equivalent) * All necessary mods to the compiler, parser module, and Python 'compiler' package to support these changes. It was necessary to change a small part of the eval loop (well, the initialization, not the loop) and the gc module's has_finalizer() logic in order to support a C equivalent to __del__. Specialists in these areas should probably scrutinize this patch! There is one additional implementation detail that was not contemplated in either PEP. in order to prevent used-up generators from retaining unnecessary references to their frame's contents, I set the generator's gi_frame member to None whenever the generator finishes normally or with an error. Thus, an exhausted generator cannot be part of a cycle, and it releases its frame object sooner than in previous Python versions. For generators used only in a direct "for" loop, this makes no difference, but for generators used with the iterator protocol (i.e. "gen.next()") from Python, this avoids stranding the generator's frame in a traceback cycle. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2005-08-01 11:10 Message: Logged In: YES user_id=6656 I think so, yes :) ---------------------------------------------------------------------- Comment By: Phillip J. Eby (pje) Date: 2005-07-31 22:10 Message: Logged In: YES user_id=56214 mwh, is the only difference in your patch the addition of the incref? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-07-07 15:14 Message: Logged In: YES user_id=6656 You missed an incref in gen_throw (symptom was Fatal Python error: ../Objects/tupleobject.c:169 object at 0x4013807c has negative ref count -606348326 on exit after running test_generators). New patch attached. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2005-06-28 08:10 Message: Logged In: YES user_id=6380 Since I've accepted PEP 342+343 last night in my Europython keynote, you can check this in now. ---------------------------------------------------------------------- Comment By: Phillip J. Eby (pje) Date: 2005-06-27 16:40 Message: Logged In: YES user_id=56214 I had hoped to be able to avoid dealing with string exceptions, but I guess it makes sense that if somebody has thrown one, it should be able to propagate. So I guess I'll take care of that along with the rest. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-06-27 16:31 Message: Logged In: YES user_id=6656 Heh heh heh. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2005-06-27 16:22 Message: Logged In: YES user_id=6380 (Removing 343 from the subject since all this is now in PEP 342.) Playing with the code a bit, I notice that throw() doesn't like string exceptions. That could cause some problems. Also, I think the tb argument should be allowed to be None. Otherwise, great! In a few hours you can check it in. :-) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-06-24 00:20 Message: Logged In: YES user_id=6656 Fair enough. I want to use with statements in a presentation at EuroPython next week, so I've hacked together a truly horrible implementation that will do for now and will see about a real one some time after... ---------------------------------------------------------------------- Comment By: Phillip J. Eby (pje) Date: 2005-06-24 00:07 Message: Logged In: YES user_id=56214 No, I haven't, and I never really planned to be the person implementing "with". Doesn't mean I might not end up doing it eventually, but it's not currently in my plans. I'm going to have my hands full just updating PEP 342 (Guido has asked me to take it over), and getting docs written to go with this patch. Between all that and my distutils-related work, my pro-bono Python project time is booked up solid for maybe the next 2 months or so. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-06-22 21:59 Message: Logged In: YES user_id=6656 Argh, I'd hoped you'd done with, too :) Have you done any work on that yet? ---------------------------------------------------------------------- Comment By: Reinhold Birkenfeld (birkenfeld) Date: 2005-06-19 08:37 Message: Logged In: YES user_id=1188172 Minor nit: on line 642, 'yield_stmt' should be changed to 'yield_expr'. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1223381&group_id=5470 From noreply at sourceforge.net Mon Aug 1 14:41:22 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 01 Aug 2005 05:41:22 -0700 Subject: [Patches] [ python-Patches-1038256 ] Provide tuple of "special" exceptions Message-ID: Patches item #1038256, was opened at 2004-10-01 15:41 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1038256&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.5 >Status: Closed Resolution: None Priority: 5 Submitted By: Nick Coghlan (ncoghlan) Assigned to: Nobody/Anonymous (nobody) Summary: Provide tuple of "special" exceptions Initial Comment: Add a "special_exceptions" tuple to builtins and exceptions module which lists the exceptions that users should be careful to avoid accidentally suppressing. The current list is SystemExit, MemoryError, KeyboardInterrupt and StopIteration A 'no builtin' version of the patch is supplied which only provides 'exceptions.special_exception', and leaves builtins alone. ---------------------------------------------------------------------- >Comment By: Nick Coghlan (ncoghlan) Date: 2005-08-01 22:41 Message: Logged In: YES user_id=1038590 Closing as this will be addressed properly by the exception heirarchy reorg. The relevant exceptions will be grouped under a common ancestor (hopefully for 2.5). ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-11 12:25 Message: Logged In: YES user_id=1038590 Not important enough to try and squeeze into the 2.4 release cycle - reconsider for 2.5 ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-03 07:15 Message: Logged In: YES user_id=1038590 Hmm, how about "terminal_exceptions" and only include SystemExit and KeyboardInterrupt? MWH gave a decent example on py-dev (x = range(sys.maxint)) as to why MemoryError *shouldn't* be included - there are too many cases where a particular excessive request will fail, but subsequent, more sensible, memory requests will succeed. (x = list(somefile.readlines()) would be another example) Looking at StopIteration more closely, I'm inclined to agree with MvL. I can't see any way to trigger it except by calling an iterator's .next() method directly or indirectly. And that generally means one is either in a for loop, or iterating manually looking for StopIteration explicitly. (For what it's worth, the original list of 'special' exceptions was based on my reading of Tim Peter's message to Py-Dev regarding possible restructuring of the exception heirarchy) ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-02 17:16 Message: Logged In: YES user_id=21627 I don't like the name of the tuple. "special_exceptions" does not indicate in what way they are special; the name should more directly indicate what this is or why one would use it. Also, I'm uncertain about the rationale for picking precisely these exceptions. I agree that one would normally not suppress SystemExit, and usually not KeyboardInterrupt. I'm uncertain about MemoryError, and I can't see why StopIteration would ever occur in a context where exceptions are suppressed. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-01 16:56 Message: Logged In: YES user_id=1038590 Yes, we are. At the moment, the best way to spell that is: try: myfunc() except special_exceptions: raise except: dealwithit() I think it would be neat to spell that as: try: myfunc() except not special_exceptions: dealwithit() but that would be a helluva lot more work, and needs this tuple anyway :) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-01 15:55 Message: Logged In: YES user_id=80475 Aren't we looking for the inverse of that tuple, some way to write: try: myfunc() except non-special exceptions: altfunc() ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1038256&group_id=5470 From noreply at sourceforge.net Tue Aug 2 00:50:54 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 01 Aug 2005 15:50:54 -0700 Subject: [Patches] [ python-Patches-1000267 ] A BSD-style wait4 implementation Message-ID: Patches item #1000267, was opened at 2004-07-29 13:52 Message generated for change (Comment added) made by cjschr You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1000267&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: chads (cjschr) Assigned to: Nobody/Anonymous (nobody) Summary: A BSD-style wait4 implementation Initial Comment: A BSD-style wait4 implementation. Using existing code from the posixmodule.c and resource.c files, I threw together a patch that implements the wait4 function. This function is similar to waitpid, but it also returns usage information about the child process. wait4(pid, options) -> (pid, status, rusage) It works for me on RedHat Linux 9 and FreeBSD 4.5 boxes using Python-2.3.4. The patch may need some fine tuning. Thanks Chad ---------------------------------------------------------------------- >Comment By: chads (cjschr) Date: 2005-08-01 17:50 Message: Logged In: YES user_id=1093928 Fix compilation error (gcc-2.95). Has anyone had a chance to look at this for inclusion? ---------------------------------------------------------------------- Comment By: chads (cjschr) Date: 2005-06-01 17:18 Message: Logged In: YES user_id=1093928 Per Martin's suggestions - thanks again. Documentation additions, configure test, import when called, single diff. LMK how this one fairs. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-05-24 16:15 Message: Logged In: YES user_id=21627 The code looks fine. The only problem I have with it that it seems to assume that any POSIX system implements wait4. However, I don't see it in Single Unix V3, so I think there should be some configure test to determine whether the function is present. Also, it would be nice if the resource module was only imported if wait4 was actually called. I can't see documentation changes; please provide a patch for Doc/lib/libos.tex as well. If you resubmit the patch, it would be easiest if the patch was a single file, eg. generated with "cvs diff -u". There is no need to include configure changes; configure.in would be sufficient. ---------------------------------------------------------------------- Comment By: chads (cjschr) Date: 2005-05-18 15:32 Message: Logged In: YES user_id=1093928 Finally took the time to implement os.wait4 based the comments from loewis. Added a resourcemodule.h header and modified resource.c and posixmodule.c appropriately. Take a gander and let me know what you think. Thanks. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-08-22 11:23 Message: Logged In: YES user_id=21627 struct rusage is already defined in Modules/resourcemodule.c. It would be good if there was only a single definition of the type object. To achieve this, resourcemodule.c should expose a C API (e.g. PyObject *PyResource_FromResource(struct rusage*) ). This should be put into a CObject, which should be published through the module; then posixmodule should import resource when wait4 is called for the first time. Alternative, wait4 could be added to resourcemodule.c entirely. Yet another alternative, on top of this approach, posixmodule and/or os.py could provide their own definition of wait4 which delegates to resource.wait4 on first usage. ---------------------------------------------------------------------- Comment By: chads (cjschr) Date: 2004-08-13 14:00 Message: Logged In: YES user_id=1093928 Added a test suite and an example of using wait4. Feedback is welcome. Thanks. ---------------------------------------------------------------------- Comment By: Jeff Epler (jepler) Date: 2004-08-04 16:53 Message: Logged In: YES user_id=2772 You should also add code to the test suite to test wait4 when it is available. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1000267&group_id=5470 From noreply at sourceforge.net Tue Aug 2 02:48:55 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 01 Aug 2005 17:48:55 -0700 Subject: [Patches] [ python-Patches-1223381 ] PEP 342 Generator enhancements Message-ID: Patches item #1223381, was opened at 2005-06-18 23:08 Message generated for change (Comment added) made by pje You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1223381&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.5 >Status: Closed Resolution: None Priority: 5 Submitted By: Phillip J. Eby (pje) Assigned to: Phillip J. Eby (pje) Summary: PEP 342 Generator enhancements Initial Comment: The attached patch implements code and test changes for: * yield expressions * bare yield (short for yield None) * yield in try/finally * generator.send(value) (send value into generator; substituted for PEP 342's next(arg)) * generator.throw(typ[,val[,tb]]) (raise error in generator) * generator.close() (per PEP 343) * GeneratorExit built-in exception type * generator.__del__ (well, the C equivalent) * All necessary mods to the compiler, parser module, and Python 'compiler' package to support these changes. It was necessary to change a small part of the eval loop (well, the initialization, not the loop) and the gc module's has_finalizer() logic in order to support a C equivalent to __del__. Specialists in these areas should probably scrutinize this patch! There is one additional implementation detail that was not contemplated in either PEP. in order to prevent used-up generators from retaining unnecessary references to their frame's contents, I set the generator's gi_frame member to None whenever the generator finishes normally or with an error. Thus, an exhausted generator cannot be part of a cycle, and it releases its frame object sooner than in previous Python versions. For generators used only in a direct "for" loop, this makes no difference, but for generators used with the iterator protocol (i.e. "gen.next()") from Python, this avoids stranding the generator's frame in a traceback cycle. ---------------------------------------------------------------------- >Comment By: Phillip J. Eby (pje) Date: 2005-08-02 00:48 Message: Logged In: YES user_id=56214 Checked in to CVS. throw() still needs string support and to allow None as the third argument, but I figure it's better to get this thing into mainline CVS and address those items separately. I did include mwh's Py_INCREF fix, however. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-08-01 10:10 Message: Logged In: YES user_id=6656 I think so, yes :) ---------------------------------------------------------------------- Comment By: Phillip J. Eby (pje) Date: 2005-07-31 21:10 Message: Logged In: YES user_id=56214 mwh, is the only difference in your patch the addition of the incref? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-07-07 14:14 Message: Logged In: YES user_id=6656 You missed an incref in gen_throw (symptom was Fatal Python error: ../Objects/tupleobject.c:169 object at 0x4013807c has negative ref count -606348326 on exit after running test_generators). New patch attached. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2005-06-28 07:10 Message: Logged In: YES user_id=6380 Since I've accepted PEP 342+343 last night in my Europython keynote, you can check this in now. ---------------------------------------------------------------------- Comment By: Phillip J. Eby (pje) Date: 2005-06-27 15:40 Message: Logged In: YES user_id=56214 I had hoped to be able to avoid dealing with string exceptions, but I guess it makes sense that if somebody has thrown one, it should be able to propagate. So I guess I'll take care of that along with the rest. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-06-27 15:31 Message: Logged In: YES user_id=6656 Heh heh heh. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2005-06-27 15:22 Message: Logged In: YES user_id=6380 (Removing 343 from the subject since all this is now in PEP 342.) Playing with the code a bit, I notice that throw() doesn't like string exceptions. That could cause some problems. Also, I think the tb argument should be allowed to be None. Otherwise, great! In a few hours you can check it in. :-) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-06-23 23:20 Message: Logged In: YES user_id=6656 Fair enough. I want to use with statements in a presentation at EuroPython next week, so I've hacked together a truly horrible implementation that will do for now and will see about a real one some time after... ---------------------------------------------------------------------- Comment By: Phillip J. Eby (pje) Date: 2005-06-23 23:07 Message: Logged In: YES user_id=56214 No, I haven't, and I never really planned to be the person implementing "with". Doesn't mean I might not end up doing it eventually, but it's not currently in my plans. I'm going to have my hands full just updating PEP 342 (Guido has asked me to take it over), and getting docs written to go with this patch. Between all that and my distutils-related work, my pro-bono Python project time is booked up solid for maybe the next 2 months or so. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-06-22 20:59 Message: Logged In: YES user_id=6656 Argh, I'd hoped you'd done with, too :) Have you done any work on that yet? ---------------------------------------------------------------------- Comment By: Reinhold Birkenfeld (birkenfeld) Date: 2005-06-19 07:37 Message: Logged In: YES user_id=1188172 Minor nit: on line 642, 'yield_stmt' should be changed to 'yield_expr'. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1223381&group_id=5470 From noreply at sourceforge.net Tue Aug 2 04:56:16 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 01 Aug 2005 19:56:16 -0700 Subject: [Patches] [ python-Patches-1243910 ] Patch for (Doc) #1243553 Message-ID: Patches item #1243910, was opened at 2005-07-24 07:54 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1243910&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Documentation Group: None >Status: Closed Resolution: None Priority: 5 Submitted By: Peter van Kampen (pterk) Assigned to: Nobody/Anonymous (nobody) Summary: Patch for (Doc) #1243553 Initial Comment: Add documentation to doc-string (copy/paste from Docs/lib) for 'quote' argument in cgi.escape. ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2005-08-01 21:56 Message: Logged In: YES user_id=44345 I checked in a change to cgi.py before seeing the reference to your patch. My change was pretty much identical. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1243910&group_id=5470 From eurosemi at eurosemi.eu.com Tue Aug 2 11:37:59 2005 From: eurosemi at eurosemi.eu.com (eurosemi@eurosemi.eu.com) Date: Tue, 2 Aug 2005 05:37:59 -0400 (EDT) Subject: [Patches] MEMS Manufacturing - July/August 2005 issue Message-ID: <20050802093759.0E70D550BB5@sulux.angelbc.co.uk> charset="iso-8859-1" Content-Transfer-Encoding:8bit If you receive this message your email system is not set up to receive this html mailshot. http://www.eurosemi.eu.com/mems-magazine/index.htm MEMS Manufacturing TeaM -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20050802/362e9bcd/attachment.htm From noreply at sourceforge.net Tue Aug 2 15:46:23 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 02 Aug 2005 06:46:23 -0700 Subject: [Patches] [ python-Patches-1235943 ] PEP 343 implementation Message-ID: Patches item #1235943, was opened at 2005-07-11 11:09 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1235943&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: PEP 343 implementation Initial Comment: This large patch (nearly 90 K, affecting 25 files!) implements PEP 343. Work done: Changes Grammar/Grammar, Python/compile.c and Python/ceval.c to support new statement (a new opcode, LOAD_EXIT_ARGS, is possibly not strictly needed, but the stack yoga to avoid it would be very tiring). Implements a new __future__ statement, "with_statement", by cribbing the '#if 0'ed out code from when generators were optional. Implements support for the with statement in Lib/compiler (but Lib/ compiler doesn't support future statements at all?). Updates Lib/opcode.py, Lib/symbol.py. Updates the parser module and it's tests. Lib/test/test_with.py contains a couple of doctests, one a reasonably comprehensive tour of with statement functionality and the other is intended to contain the examples from the PEP, but most of these depend on PEP 342, the implementation of which hasn't been checked in yet. The code isn't the prettiest in places (esp search for the two XXXs). ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2005-08-02 14:46 Message: Logged In: YES user_id=6656 Update post the checking in of PEP 342 stuff. There were a couple of small conflicts, and I added a lot more tests from the PEP (they used PEP 342 dependent generators). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1235943&group_id=5470 From noreply at sourceforge.net Wed Aug 3 09:31:35 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 03 Aug 2005 00:31:35 -0700 Subject: [Patches] [ python-Patches-1105730 ] Faster commonprefix in macpath, ntpath, etc. Message-ID: Patches item #1105730, was opened at 2005-01-20 06:34 Message generated for change (Comment added) made by birkenfeld You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1105730&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.4 >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: Jimmy Retzlaff (jretz) >Assigned to: Reinhold Birkenfeld (birkenfeld) Summary: Faster commonprefix in macpath, ntpath, etc. Initial Comment: Patch 681780 resulted in a faster version of commonprefix for posixpath.py. The same implementation should work fine for macpath.py, ntpath.py, os2emxpath.py, and plat-riscos/riscospath.py but it didn't make it into those. This is simply the posixpath.commonprefix implementation copied into these files. ---------------------------------------------------------------------- >Comment By: Reinhold Birkenfeld (birkenfeld) Date: 2005-08-03 09:31 Message: Logged In: YES user_id=1188172 Committed! Log follows: Checking in Misc/NEWS; new revision: 1.1329; previous revision: 1.1328 Checking in Lib/macpath.py; new revision: 1.51; previous revision: 1.50 Checking in Lib/ntpath.py; new revision: 1.62; previous revision: 1.61 Checking in Lib/os2emxpath.py; new revision: 1.14; previous revision: 1.13 Checking in Lib/plat-riscos/riscospath.py; new revision: 1.12; previous revision: 1.11 ---------------------------------------------------------------------- Comment By: Jimmy Retzlaff (jretz) Date: 2005-01-24 23:57 Message: Logged In: YES user_id=101588 I had assumed there were historical reasons that the platform specific implementations of os.path weren?t importing from each other. If this is not the case then I?m all for eliminating the duplication of code. I?m attaching an alternate patch (called importcommonprefix.diff) for this approach. I've tested it on Windows, but I don't have a means of testing the other versions. ---------------------------------------------------------------------- Comment By: Jeff Epler (jepler) Date: 2005-01-24 17:34 Message: Logged In: YES user_id=2772 Why not write 'from posixpath import commonprefix' in each of the other XXXpath.py modules, to benefit from any future improvements? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1105730&group_id=5470 From noreply at sourceforge.net Thu Aug 4 12:28:57 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 04 Aug 2005 03:28:57 -0700 Subject: [Patches] [ python-Patches-1251748 ] compiler package: "global a; a=5" Message-ID: Patches item #1251748, was opened at 2005-08-04 10:28 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1251748&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Parser/Compiler Group: None Status: Open Resolution: None Priority: 5 Submitted By: Armin Rigo (arigo) Assigned to: Nobody/Anonymous (nobody) Summary: compiler package: "global a; a=5" Initial Comment: The stdlib compiler package produces wrong code for "global a; a=5". It produces a STORE_NAME instead of a STORE_GLOBAL. Quick patch from Michael attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1251748&group_id=5470 From eurosemi at eurosemi.eu.com Thu Aug 4 14:19:01 2005 From: eurosemi at eurosemi.eu.com (eurosemi@eurosemi.eu.com) Date: Thu, 4 Aug 2005 08:19:01 -0400 (EDT) Subject: [Patches] Eurosemi E-News Message-ID: <20050804121901.923E0552A02@sulux.angelbc.co.uk> charset="iso-8859-1" Content-Transfer-Encoding:8bit ------------------- EUROSEMI E-NEWS ------------------- This is a HTML based email. To view it please go to: --------------------- Should you wish to unsubscribe please go to: http://www.eurosemi.eu.com/front-end/unsubscribe2.php?email=patches at python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20050804/2ac1fe21/attachment-0001.htm From noreply at sourceforge.net Thu Aug 4 23:51:49 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 04 Aug 2005 14:51:49 -0700 Subject: [Patches] [ python-Patches-1252236 ] Simplying Tkinter's event loop Message-ID: Patches item #1252236, was opened at 2005-08-05 06:51 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252236&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Tkinter Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Martin v. L?wis (loewis) Summary: Simplying Tkinter's event loop Initial Comment: This patch is a simplification of Tkinter's event loop by moving the event loop to Python itself. Tkinter needs an event loop to be able to handle both Tcl/Tk events and keyboard input by the user. Usually, an event loop looks like this: while true: wait for an event (Tcl/Tk events or keyboard input) handle the event Tkinter's loop is set up differently. If Tkinter is imported, it sets the PyOS_InputHook pointer to the EventHook function in _tkinter.c. The EventHook function handles Tcl/Tk events only. In essence, Tkinter's event loop looks like this: while true: handle all Tcl/Tk events that need to be handled if keyboard input is waiting: break sleep for 20 milliseconds Note that the event loop exits only if the user presses a key. This causes the following problems: 1) This event loop is available only for Tkinter. There is no way to handle events other than the Tcl/Tk events. 2) If another extension module needs events to be handled, it can do so by using a similar event loop. However, it is not possible to handle both Tcl/Tk events and other events, (except for keyboard input). 3) Chaining the two event loops will fail. As soon as Tkinter's event loop starts running, it will break out of this loop only in the case of keyboard input. So any other events will not be processed until a key is pressed. 4) Even if Tkinter's event loop is the only event loop that is needed, it can fail. If a Python thread is waiting for a mutex variable to change, it cannot run Tkinter's event loop while waiting, since Tkinter's event loop will only return after a key is pressed (it doesn't pay attention to the mutex variable). This is the reason that running Tkinter in an IDLE shell will fail. 5) On some platforms (e.g. Cygwin), the event loop is unable to check for keyboard input. Hence, one needs to call mainloop() in order to see Tkinter's widgets. The reason that the event loop was set up in this peculiar way may be due to a bug with PyOS_InputHook in Python. If Python is compiled with readline support, PyOS_InputHook will be called ten times per second, while Python is waiting for keyboard input. (As shown below, this allows us to fix this event loop problem.) However, if Python is compiled without readline support (notably on Windows), PyOS_InputHook is called only once (just before a user starts entering a new command on the keyboard). This necessitates Tkinter to run an event loop, and not return from it until keyboard input is ready. If PyOS_InputHook is called ten times per second, we can set up the event loop as follows: Set PyOS_InputHook to Tkinter's EventHook while true: call PyOS_InputHook handle keyboard input, if present sleep for 100 milliseconds Here, the key point is that Tkinter's EventHook returns immediately if there are no more Tcl/Tk events to be handled. Effectively, we have moved the event loop from Tkinter to Python itself. Hence, there are two problems to be solved: 1) PyOS_InputHook should be called ten times per second, regardless of whether Python is compiled with or without readline support. 2) Tkinter's EventHook should return immediately after handling all Tcl/Tk events. Problem 1) is solved in patch #1049855; problem 2) is solved in this patch. This patch is a considerable simplication of _tkinter.c, as it removes all lines that are no longer needed (and adds almost no code). This patch changes the nature of the function to which PyOS_InputHook points. Before, PyOS_InputHook is used to start a Tcl/Tk event loop. Now, PyOS_InputHook is called from an existing event loop to handle Tcl/Tk events. With these two patches, Python has a functioning PyOS_InputHook, which can be used both by Tkinter and by other extension modules. We can handle Tcl/Tk events by calling Tkinter's EventHook function if we're waiting for something else other than keyboard input, like a mutex variable. We can chain event-handling functions from different extension modules (e.g. Tkinter and some other module), by first calling Tkinter's EventHook and then the other extension module's event hook function. Finally, on Cygwin, we can now create Tkinter widgets without having to call mainloop. So this will now work on Cygwin: >>> from Tkinter import * >>> Label() >>> # label appears, without having to call mainloop. This patch has been tested on WIndows and Cygwin. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252236&group_id=5470 From noreply at sourceforge.net Fri Aug 5 14:32:26 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 05 Aug 2005 05:32:26 -0700 Subject: [Patches] [ python-Patches-1252550 ] modulefinder misses modules Message-ID: Patches item #1252550, was opened at 2005-08-05 14:32 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252550&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Thomas Heller (theller) Assigned to: Thomas Heller (theller) Summary: modulefinder misses modules Initial Comment: Failing top-level imports hide modules with the same name in packages. The attached zipfile contains a script main.py and a simple contrived package demonstrating the problem. If the zip-archive is unpacked to some directory, and modulefinder is run in this directory, it prints this: c:\sf\test>py24 -m modulefinder main.py Name File ---- ---- m __main__ main.py P a a\__init__.py P a.c a\c\__init__.py Missing modules: ? spam imported from __main__, a c:\sf\test> With the attached patch, the output is like this: c:\sf\test>py24 -m modulefinder main.py Name File ---- ---- m __main__ main.py P a a\__init__.py P a.c a\c\__init__.py m a.c.spam a\c\spam.py m a.spam a\spam.py Missing modules: ? spam imported from __main__ c:\sf\test> ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252550&group_id=5470 From noreply at sourceforge.net Fri Aug 5 14:34:41 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 05 Aug 2005 05:34:41 -0700 Subject: [Patches] [ python-Patches-1252550 ] modulefinder misses modules Message-ID: Patches item #1252550, was opened at 2005-08-05 14:32 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252550&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Thomas Heller (theller) Assigned to: Thomas Heller (theller) Summary: modulefinder misses modules Initial Comment: Failing top-level imports hide modules with the same name in packages. The attached zipfile contains a script main.py and a simple contrived package demonstrating the problem. If the zip-archive is unpacked to some directory, and modulefinder is run in this directory, it prints this: c:\sf\test>py24 -m modulefinder main.py Name File ---- ---- m __main__ main.py P a a\__init__.py P a.c a\c\__init__.py Missing modules: ? spam imported from __main__, a c:\sf\test> With the attached patch, the output is like this: c:\sf\test>py24 -m modulefinder main.py Name File ---- ---- m __main__ main.py P a a\__init__.py P a.c a\c\__init__.py m a.c.spam a\c\spam.py m a.spam a\spam.py Missing modules: ? spam imported from __main__ c:\sf\test> ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2005-08-05 14:34 Message: Logged In: YES user_id=11105 Patch attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252550&group_id=5470 From noreply at sourceforge.net Fri Aug 5 18:57:05 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 05 Aug 2005 09:57:05 -0700 Subject: [Patches] [ python-Patches-1252706 ] poplib list() docstring fix Message-ID: Patches item #1252706, was opened at 2005-08-05 11:57 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252706&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Steve Greenland (vmole) Assigned to: Nobody/Anonymous (nobody) Summary: poplib list() docstring fix Initial Comment: The POP list() function returns 3 items, not 2. Attached is the fairly obvious patch against poplib.py in CVS at HEAD. Steve ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252706&group_id=5470 From noreply at sourceforge.net Fri Aug 5 23:03:04 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 05 Aug 2005 14:03:04 -0700 Subject: [Patches] [ python-Patches-1252706 ] poplib list() docstring fix Message-ID: Patches item #1252706, was opened at 2005-08-05 18:57 Message generated for change (Comment added) made by birkenfeld You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252706&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Documentation Group: None >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: Steve Greenland (vmole) >Assigned to: Reinhold Birkenfeld (birkenfeld) Summary: poplib list() docstring fix Initial Comment: The POP list() function returns 3 items, not 2. Attached is the fairly obvious patch against poplib.py in CVS at HEAD. Steve ---------------------------------------------------------------------- >Comment By: Reinhold Birkenfeld (birkenfeld) Date: 2005-08-05 23:03 Message: Logged In: YES user_id=1188172 Thanks for the report, I corrected the docs too. Committed as Lib/poplib.py r1.24, r1.23.4.1, Doc/lib/libpoplib.py r1.18, 1.17.4.1. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252706&group_id=5470 From noreply at sourceforge.net Fri Aug 5 23:07:18 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 05 Aug 2005 14:07:18 -0700 Subject: [Patches] [ python-Patches-1049855 ] PyOS_InputHook inconsistency on Windows Message-ID: Patches item #1049855, was opened at 2004-10-19 18:03 Message generated for change (Comment added) made by mdehoon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1049855&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Nobody/Anonymous (nobody) Summary: PyOS_InputHook inconsistency on Windows Initial Comment: PyOS_InputHook is a pointer to a function to be called periodically when Python is idle. It is used, for example, to get messages delivered to graphics windows. If I compile Python from source (which uses Modules/readline.c), PyOS_InputHook is called ten times per second. However, with the Windows-version of Python, PyOS_InputHook is called only once for each command typed by the user. It seems that the problem resides in Parser/myreadline.c (which I presume is used for the Windows-version of Python): if (PyOS_InputHook != NULL) (void)(PyOS_InputHook)(); ... p = fgets(buf, len, fp); Python idles at "fgets", but PyOS_InputHook is not being called while Python is idle. The attached patch solves this problem by using the "select" function, basically in the same way as what is in Module/readline.c. I don't know how to compile Python for Windows, so I wasn't able to test this patch. ---------------------------------------------------------------------- >Comment By: Michiel de Hoon (mdehoon) Date: 2005-08-06 06:07 Message: Logged In: YES user_id=488897 One more fix: I noticed that on Windows, the patch causes the CPU usage to go up to 100%. The reason for this is that if the users presses a control key (e.g. shift), the WaitForSingleObject function returns, but _kbhit() returns false since no character input is waiting in the buffer. Hence, the loop continues to run, and WaitForSingleObject will again return because the control key stroke has not been flushed from the buffer. The fix is in the patch dated 20050803, which is to flush the buffer with FlushConsoleInputBuffer if a key was hit but no character input is available. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-07-29 11:40 Message: Logged In: YES user_id=488897 I have found a solution for the problem described below ("python -i < inputfile.py" behaves differently). The solution was to use _isatty to check if stdin is redirected, and check for keyboard input only if stdin is not redirected. With the latest version of this patch (20050728), there are no changes in Python's behaviour on Windows or Unix (except that PyOS_InputHook is called ten times per second, which is what this patch intends to solve). I apologize for the fact that multiple iterations were needed to converge to the right (I hope) solution for this bug. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-07-16 11:14 Message: Logged In: YES user_id=488897 I have now recompiled Python on Windows to test this patch. When compiling this patch, I found out that _get_osfhandle needs to be cast to a HANDLE*. The latest version of this patch (dated 20050715) includes this cast. I have tested this patch by running the test suite. One note: Whereas "python -i inputfile.py" and "python < inputfile.py" work as expected, "python -i < inputfile.py" behaves differently with this patch (python waits for the user to hit enter before proceeding). I am not sure if this is significant, as I wouldn't know what a user might try to achieve with "python -i < inputfile.py". I'd be happy to send a binary for Windows to anybody who would like to test this patch. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-06-11 05:46 Message: Logged In: YES user_id=488897 Thomas Heller sent me this comment: > The PyOS_StdioReadline function calls my_fgets with a file > pointer argument. The my_fgets function in the patch assumes > that STD_INPUT_HANDLE is the handle to use - is this > assumption always correct? He is right, this assumption is not necessarily correct. I have made a new patch (labeled 20050610) to solve this issue. This latest version has been tested on Cygwin, but not yet on Windows -- I need to dig up a compiler for Windows first. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-05-13 12:57 Message: Logged In: YES user_id=488897 I have rewritten the patch to include Windows support. I compiled Python on Windows with VC98, and on Linux and Mac OS X and found no problems with it. The new patch is attached. ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2005-05-02 12:40 Message: Logged In: YES user_id=149084 Yeah, that's why I didn't want to check it in, I can't build on Windows ithe VC 5. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-05-02 11:52 Message: Logged In: YES user_id=488897 Thanks for looking at this patch. Today I found out that it is possible to compile Python 2.4.1 with the older Microsoft Visual Studio 6.0, which I happen to have in my office. It turned out that the patch does not work correctly on Windows, due to the fact that the "select" function doesn't work with stdin on Windows. I will fix the patch so it'll work on Windows too. ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2005-05-01 03:21 Message: Logged In: YES user_id=149084 The revised patch looks ok to me. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-03-26 22:33 Message: Logged In: YES user_id=488897 I have now tested this patch. After fixing one error (I had forgotton to declare one variable), the patch works correctly. I have uploaded a fixed patch. Note that this bug occurs not only on Windows, but on any Python compiled without readline. (which allowed me to test this patch by compiling Python without readline support). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1049855&group_id=5470 From noreply at sourceforge.net Sat Aug 6 21:14:18 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 06 Aug 2005 12:14:18 -0700 Subject: [Patches] [ python-Patches-1232343 ] PyOS_Readline Message-ID: Patches item #1232343, was opened at 2005-07-04 20:03 Message generated for change (Comment added) made by greglielens You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1232343&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Lisandro Dalcin (dalcinl) Assigned to: Nobody/Anonymous (nobody) Summary: PyOS_Readline Initial Comment: Greg Lielens submitted some time ago a patch [id 955928] about 'PyOS_Readline()' behavior with non-interactive tty's. Basically, there is no way to properly override 'PyOS_ReadlineFunctionPointer' as 'PyOS_Readline()' calls 'PyOS_StdioReadline()' when 'stdin' or 'stdout' are not tty's. A snippet of "Parser/myreadline.c": ... if (!isatty (fileno (sys_stdin)) || !isatty (fileno (sys_stdout))) rv = PyOS_StdioReadline (sys_stdin, sys_stdout, prompt); else rv = (*PyOS_ReadlineFunctionPointer)(sys_stdin, sys_stdout, prompt); ... Greg Lielens is completely right about the problem, but his patch is perhaps a bit crude, it also modifies "Python/bltinmodule.c" to solve the same issue with 'raw_input'. I think I have found a not so crude solution, and completely backward compatible. Basically, I moved 'isatty()' test from 'PyOS_Readline()' in file "Parser/myreadline.c" to 'call_readline()' in file "Modules/readline.c". In order to do that, I believe 'PyOS_StdioReadline' have to be added to file "Include/pythonrun.h". All those changes will not affect in any way the behavior in interactive sessions. Now 'PyOS_ReadlineFunctionPointer' can be properly overrode and users of 'readline' module will not see any change: in non-interactive tty's 'PyOS_StdioReadline()' will be called anyway. The problem in 'input' and 'raw_input' builtins remains, but its solution is not so clear to me, I think this part should not be touched right now. This patch passes 'Lib/test/regrtest.py' (of course, all those tests are non-interactive!). I use Python interpreter every day in the standard way and also in MPI-parallelized interactive sessions with this patch applied, and I never have any problem. I send my patch obtained with 'cvs diff -c' from current sources. ---------------------------------------------------------------------- Comment By: Gregory Lielens (greglielens) Date: 2005-08-06 21:14 Message: Logged In: YES user_id=1044428 Hi Lisandro, sorry that I did not follow this matter, we had other short term focus in our company and I had trouble finding time for it. Good that you subject this though, I have checked it fast and from what I understand it would have the same effect as my older patch... However, I used some features of my patch that are not present in yours: in Parser/myreadline.c, I removed the test about PyOS_ReadlineFunctionPointer being null to instead initialising it at creation to the correct value depending to system: This was important because it allows me in my specific MPI Readline function to use this defautl functio for the process 0...In general, it allows to re-use the default in the new Readline function, incrementing functionalities instead of re-implementing full functionality. You will have the same mechanism with the inclusion of PyOS_StdioReadline in the pythonrun API, but I personally find this solution less appealing: it increase the size of the API, and you do not have the correct readline on VMS...That's why I have done it like this, even if it may seems hackish ;) Of sourse manual has to be updated to say that PyOS_ReadlineFunctionPointer has a meaningfull defautl value...but it would also have to be updated in your approach, to tell about PyOS_StdioReadline... Regarding your modif of the Modules/readline.c, I have the feeling that your approach is cleaner...because It seems nicer to check for tty inside the new readline function, instead of installing the new readline function only if stdin/stdout are tty....You have to use PyOS_StdioReadline though, that would not be accessible if one does not add it to the pythonrun.h API, and that is not correct on vms system anyway (at least that's how I understand it...but I have no vms system to check ;) ). Maybe a better approach would be to cache the original PyOS_ReadlineFunctionPointer in the readline module, and re-use it if stdin/stdout are non-tty? This would be close to the approach I used for implementing my MPI readline: cache PyOS_ReadlineFunctionPointer and re-use it for modif... If this is considerd unclean, the approach putting PyOS_StdioReadline in the API can be kept, but It would have to be renamed something like PyOS_ReadlineFunction and be defined as PyOS_StdioReadline/vms__StdioReadline depending on the environment... Now for my patch on "Python/bltinmodule.c" to solve the same issue with 'raw_input', I think it was necessary to be able to use ipython in an interractive MPI session...Our embedded python interpreter had the possibility to use this (it gives a far nicier shell than the standard one), but I think It uses raw_input as entry mechanism...Could you test your patch to see if it work with this? The only drawback I see with this is that it could slow down a bit the parsing when using this function on a non-interractive file...but I doubt this is significative and could be optimized by keeping the call to PyOS_Readline but stripping out the prompt treatment in this case... We now are moving from an embbedded python interpreter to using a standard python...which make the acceptance of the patch more interresting: it would allows us to uses stock python interpreters > 2.x, no need to distribute our own interpretter anymore!!! :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1232343&group_id=5470 From noreply at sourceforge.net Sat Aug 6 21:35:18 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 06 Aug 2005 12:35:18 -0700 Subject: [Patches] [ python-Patches-1232343 ] PyOS_Readline Message-ID: Patches item #1232343, was opened at 2005-07-04 20:03 Message generated for change (Comment added) made by greglielens You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1232343&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Lisandro Dalcin (dalcinl) Assigned to: Nobody/Anonymous (nobody) Summary: PyOS_Readline Initial Comment: Greg Lielens submitted some time ago a patch [id 955928] about 'PyOS_Readline()' behavior with non-interactive tty's. Basically, there is no way to properly override 'PyOS_ReadlineFunctionPointer' as 'PyOS_Readline()' calls 'PyOS_StdioReadline()' when 'stdin' or 'stdout' are not tty's. A snippet of "Parser/myreadline.c": ... if (!isatty (fileno (sys_stdin)) || !isatty (fileno (sys_stdout))) rv = PyOS_StdioReadline (sys_stdin, sys_stdout, prompt); else rv = (*PyOS_ReadlineFunctionPointer)(sys_stdin, sys_stdout, prompt); ... Greg Lielens is completely right about the problem, but his patch is perhaps a bit crude, it also modifies "Python/bltinmodule.c" to solve the same issue with 'raw_input'. I think I have found a not so crude solution, and completely backward compatible. Basically, I moved 'isatty()' test from 'PyOS_Readline()' in file "Parser/myreadline.c" to 'call_readline()' in file "Modules/readline.c". In order to do that, I believe 'PyOS_StdioReadline' have to be added to file "Include/pythonrun.h". All those changes will not affect in any way the behavior in interactive sessions. Now 'PyOS_ReadlineFunctionPointer' can be properly overrode and users of 'readline' module will not see any change: in non-interactive tty's 'PyOS_StdioReadline()' will be called anyway. The problem in 'input' and 'raw_input' builtins remains, but its solution is not so clear to me, I think this part should not be touched right now. This patch passes 'Lib/test/regrtest.py' (of course, all those tests are non-interactive!). I use Python interpreter every day in the standard way and also in MPI-parallelized interactive sessions with this patch applied, and I never have any problem. I send my patch obtained with 'cvs diff -c' from current sources. ---------------------------------------------------------------------- Comment By: Gregory Lielens (greglielens) Date: 2005-08-06 21:35 Message: Logged In: YES user_id=1044428 Oups: please forget my previous message, I clicked on submit too fast and there is no correction/unsubmit option on sourceforge... Hi Lisandro, sorry that I did not follow this matter, we had other short term focus in our company and I had trouble finding time for it. Good that you subject this though, I have checked it fast and from what I understand it would have the same effect as my older patch... However, I still prefer some aspects of my patch over your?s ;) in Parser/myreadline.c, I removed the test about PyOS_ReadlineFunctionPointer being null: instead I initialize it at creation (to the correct value depending to system, vms or not): This initialization was important because it allows me to use this default function in my parallel Readline function... In general, ithe idea is to allow re-using the default in the new Readline function, incrementing functionalities instead of re-implementing full functionality. You will have the same mechanism with the inclusion of PyOS_StdioReadline in the pythonrun API, but I personally find this solution less appealing: it increase the size of the API, and you do not have the correct readline on VMS...That's why I have done it like this, even if it may seems hackish ;) Of course manual has to be updated to say that PyOS_ReadlineFunctionPointer has a meaningful default value...but it would also have to be updated in your approach, to tell about PyOS_StdioReadline... Well, a matter of taste I guess, but if the general opinion is that this initialization is unclean, I would then advice for renaming of PyOS_StdioReadline in the API to something like PyOS_ReadlineFunction, that would defined as PyOS_StdioReadline/vms__StdioReadline depending on the environment... Regarding your modify of the Modules/readline.c, I have the feeling that your approach is cleaner...because It seems nicer to check for tty inside the new readline function, instead of installing the new readline function only if stdin/stdout are tty....You have to use PyOS_StdioReadline though, again that would not be accessible if one does not add it to the pythonrun.h API (again that is not correct on vms system, but could be corrected taking my previous remark into account). An alternative approach would be to cache the original PyOS_ReadlineFunctionPointer in the readline module, and re-use it if stdin/stdout are non-tty? This would be close to the approach I used for implementing my MPI readline: cache PyOS_ReadlineFunctionPointer and re-use it for modif... Again, a matter of choice, I guess, more experienced python developers may have something to say about it. Now for my patch on "Python/bltinmodule.c" to solve the same issue with 'raw_input', I think it was necessary to be able to use ipython in an interactive MPI session...Our embedded python interpreter had the possibility to use this (it gives a far nicer shell than the standard one), but I think It uses raw_input as entry mechanism... Could you test your patch to see if it work also with ipython? If not then this modify has to stay I think... The only drawback I see with this is that it could slow down a bit the parsing when using this function on a non-interactive file...but I doubt this is significative and could be optimized by keeping the call to PyOS_Readline but stripping out the prompt treatment in this case... We now are moving from an embedded python interpreter to using a standard python...which make the acceptance of the patch all the more interesting for us: it would allows to use stock python interpreters > 2.x, no need to distribute our own interpreter anymore!!! :) ---------------------------------------------------------------------- Comment By: Gregory Lielens (greglielens) Date: 2005-08-06 21:14 Message: Logged In: YES user_id=1044428 Hi Lisandro, sorry that I did not follow this matter, we had other short term focus in our company and I had trouble finding time for it. Good that you subject this though, I have checked it fast and from what I understand it would have the same effect as my older patch... However, I used some features of my patch that are not present in yours: in Parser/myreadline.c, I removed the test about PyOS_ReadlineFunctionPointer being null to instead initialising it at creation to the correct value depending to system: This was important because it allows me in my specific MPI Readline function to use this defautl functio for the process 0...In general, it allows to re-use the default in the new Readline function, incrementing functionalities instead of re-implementing full functionality. You will have the same mechanism with the inclusion of PyOS_StdioReadline in the pythonrun API, but I personally find this solution less appealing: it increase the size of the API, and you do not have the correct readline on VMS...That's why I have done it like this, even if it may seems hackish ;) Of sourse manual has to be updated to say that PyOS_ReadlineFunctionPointer has a meaningfull defautl value...but it would also have to be updated in your approach, to tell about PyOS_StdioReadline... Regarding your modif of the Modules/readline.c, I have the feeling that your approach is cleaner...because It seems nicer to check for tty inside the new readline function, instead of installing the new readline function only if stdin/stdout are tty....You have to use PyOS_StdioReadline though, that would not be accessible if one does not add it to the pythonrun.h API, and that is not correct on vms system anyway (at least that's how I understand it...but I have no vms system to check ;) ). Maybe a better approach would be to cache the original PyOS_ReadlineFunctionPointer in the readline module, and re-use it if stdin/stdout are non-tty? This would be close to the approach I used for implementing my MPI readline: cache PyOS_ReadlineFunctionPointer and re-use it for modif... If this is considerd unclean, the approach putting PyOS_StdioReadline in the API can be kept, but It would have to be renamed something like PyOS_ReadlineFunction and be defined as PyOS_StdioReadline/vms__StdioReadline depending on the environment... Now for my patch on "Python/bltinmodule.c" to solve the same issue with 'raw_input', I think it was necessary to be able to use ipython in an interractive MPI session...Our embedded python interpreter had the possibility to use this (it gives a far nicier shell than the standard one), but I think It uses raw_input as entry mechanism...Could you test your patch to see if it work with this? The only drawback I see with this is that it could slow down a bit the parsing when using this function on a non-interractive file...but I doubt this is significative and could be optimized by keeping the call to PyOS_Readline but stripping out the prompt treatment in this case... We now are moving from an embbedded python interpreter to using a standard python...which make the acceptance of the patch more interresting: it would allows us to uses stock python interpreters > 2.x, no need to distribute our own interpretter anymore!!! :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1232343&group_id=5470 From noreply at sourceforge.net Sun Aug 7 20:25:28 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 07 Aug 2005 11:25:28 -0700 Subject: [Patches] [ python-Patches-1049855 ] PyOS_InputHook inconsistency on Windows Message-ID: Patches item #1049855, was opened at 2004-10-19 11:03 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1049855&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Nobody/Anonymous (nobody) Summary: PyOS_InputHook inconsistency on Windows Initial Comment: PyOS_InputHook is a pointer to a function to be called periodically when Python is idle. It is used, for example, to get messages delivered to graphics windows. If I compile Python from source (which uses Modules/readline.c), PyOS_InputHook is called ten times per second. However, with the Windows-version of Python, PyOS_InputHook is called only once for each command typed by the user. It seems that the problem resides in Parser/myreadline.c (which I presume is used for the Windows-version of Python): if (PyOS_InputHook != NULL) (void)(PyOS_InputHook)(); ... p = fgets(buf, len, fp); Python idles at "fgets", but PyOS_InputHook is not being called while Python is idle. The attached patch solves this problem by using the "select" function, basically in the same way as what is in Module/readline.c. I don't know how to compile Python for Windows, so I wasn't able to test this patch. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2005-08-07 20:25 Message: Logged In: YES user_id=21627 I think this code is way too fragile to be applied to 2.4. As for specific errors I could find: - if the system is not windows, it invokes select unconditionally; this should be conditioned with HAVE_SELECT - the Windows version uses kbhit; this may or may not operate on the same handle as hStdIn. In fact, it likely is a different handle. See the source code of the crt: kbhit uses _coninpfh, which is created from opening CONIN$. Try using GetNumberOfConsoleInputEvents/PeekConsoleInput instead, checking for a KEY_EVENT event. - as stdin may be buffered, it may be the case that there is already input available, even though the file descriptor is not ready. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-08-05 23:07 Message: Logged In: YES user_id=488897 One more fix: I noticed that on Windows, the patch causes the CPU usage to go up to 100%. The reason for this is that if the users presses a control key (e.g. shift), the WaitForSingleObject function returns, but _kbhit() returns false since no character input is waiting in the buffer. Hence, the loop continues to run, and WaitForSingleObject will again return because the control key stroke has not been flushed from the buffer. The fix is in the patch dated 20050803, which is to flush the buffer with FlushConsoleInputBuffer if a key was hit but no character input is available. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-07-29 04:40 Message: Logged In: YES user_id=488897 I have found a solution for the problem described below ("python -i < inputfile.py" behaves differently). The solution was to use _isatty to check if stdin is redirected, and check for keyboard input only if stdin is not redirected. With the latest version of this patch (20050728), there are no changes in Python's behaviour on Windows or Unix (except that PyOS_InputHook is called ten times per second, which is what this patch intends to solve). I apologize for the fact that multiple iterations were needed to converge to the right (I hope) solution for this bug. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-07-16 04:14 Message: Logged In: YES user_id=488897 I have now recompiled Python on Windows to test this patch. When compiling this patch, I found out that _get_osfhandle needs to be cast to a HANDLE*. The latest version of this patch (dated 20050715) includes this cast. I have tested this patch by running the test suite. One note: Whereas "python -i inputfile.py" and "python < inputfile.py" work as expected, "python -i < inputfile.py" behaves differently with this patch (python waits for the user to hit enter before proceeding). I am not sure if this is significant, as I wouldn't know what a user might try to achieve with "python -i < inputfile.py". I'd be happy to send a binary for Windows to anybody who would like to test this patch. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-06-10 22:46 Message: Logged In: YES user_id=488897 Thomas Heller sent me this comment: > The PyOS_StdioReadline function calls my_fgets with a file > pointer argument. The my_fgets function in the patch assumes > that STD_INPUT_HANDLE is the handle to use - is this > assumption always correct? He is right, this assumption is not necessarily correct. I have made a new patch (labeled 20050610) to solve this issue. This latest version has been tested on Cygwin, but not yet on Windows -- I need to dig up a compiler for Windows first. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-05-13 05:57 Message: Logged In: YES user_id=488897 I have rewritten the patch to include Windows support. I compiled Python on Windows with VC98, and on Linux and Mac OS X and found no problems with it. The new patch is attached. ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2005-05-02 05:40 Message: Logged In: YES user_id=149084 Yeah, that's why I didn't want to check it in, I can't build on Windows ithe VC 5. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-05-02 04:52 Message: Logged In: YES user_id=488897 Thanks for looking at this patch. Today I found out that it is possible to compile Python 2.4.1 with the older Microsoft Visual Studio 6.0, which I happen to have in my office. It turned out that the patch does not work correctly on Windows, due to the fact that the "select" function doesn't work with stdin on Windows. I will fix the patch so it'll work on Windows too. ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2005-04-30 20:21 Message: Logged In: YES user_id=149084 The revised patch looks ok to me. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-03-26 14:33 Message: Logged In: YES user_id=488897 I have now tested this patch. After fixing one error (I had forgotton to declare one variable), the patch works correctly. I have uploaded a fixed patch. Note that this bug occurs not only on Windows, but on any Python compiled without readline. (which allowed me to test this patch by compiling Python without readline support). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1049855&group_id=5470 From noreply at sourceforge.net Sun Aug 7 22:52:02 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 07 Aug 2005 13:52:02 -0700 Subject: [Patches] [ python-Patches-827386 ] absolute paths cause problems for MSVC Message-ID: Patches item #827386, was opened at 2003-10-21 11:11 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=827386&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Distutils and setup.py Group: None >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: eric jones (eaj) Assigned to: Nobody/Anonymous (nobody) Summary: absolute paths cause problems for MSVC Initial Comment: When source files are given in the form: c:\foo\bar.c the object file is written to: c:\foo\bar.o instead of the build\temp.win32-2.3 directory. This is because the object_filenames method doesn't chop off the drive name from source file name base before trying to concatenate the build directory with the base. This results in the build directory being ignored. The ccompiler.py object_filenames does it right, so I just copied its behavior. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2005-08-07 22:52 Message: Logged In: YES user_id=21627 The original problem was apparently #668662, which caused the introduction of support for absolute paths in ccompiler.py. It's not clear in that report why anybody would use absolute paths in setup.py, but apparently, f2py would generate setup files that use absolute paths. Integrating the entire path into the Release directory is intentional, since the same source file name may occur multiple times for the same project (in different directories). Thanks for the patch and the review, committed as msvccompiler.py 1.64.2.4 NEWS 1.1193.2.69 msvccompiler.py 1.69 NEWS 1.1330 ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-05-01 11:20 Message: Logged In: YES user_id=488897 I have tested your patch and found that it works as intended. I am a bit surprised about the approach taken in ccompiler.py: 1) Using absolute paths is not portable, so I'm not sure why somebody would want to do that. 2) With this patch, If the source file is in c:\foo\bar.c, then the object file is written to build\temp.win32-2.3\Release\foo\bar.o. However, if the source file is written with a relative patch ("bar.c"), the object file is written to build\temp.win32-2.3\Release\bar.o (so without "foo"). Hence, even though the source file bar.c is in the exact same location in these two cases, the object file ends up in a different location depending on whether the path is specified as absolute or relative. Is that really what we want? But I agree that the behavior of msvccompiler.py and ccompiler.py should be consistent. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=827386&group_id=5470 From noreply at sourceforge.net Sun Aug 7 23:10:45 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 07 Aug 2005 14:10:45 -0700 Subject: [Patches] [ python-Patches-1239112 ] Fix LINKCC (Bug #1189330) Message-ID: Patches item #1239112, was opened at 2005-07-15 21:26 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1239112&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Build Group: Python 2.4 >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: Christoph Ludwig (cludwig) Assigned to: Nobody/Anonymous (nobody) Summary: Fix LINKCC (Bug #1189330) Initial Comment: A build of Python 2.4.1 on a i686-pc-linux-gnu system with GCC 4.0 failed. configure set `LINKCC=$(PURIFY) $(CC) ' even though main() was compiled with g++ and you cannot lin C++ object files with the C compiler on this platform (Bug #1189330). There was some discussion on python-dev that on some platforms (notably Linux/ELF) the C++ compiler is unnecessarily used to build Python which makes the python executable depend on this particular compiler version's C++ runtime library. This patch does *not* address this issue; it keeps the criticized behaviour but fixes configure.in such that make does not fail anymore on above mentioned platform. (I will try to come up in the next weeks with a patch for the main trunk that addresses the - on some platforms - needless dependency on the C++ runtime.) In 2.4.1, configure compiled a single source file with the C++ compiler and tries to link it with the C compiler. Apparently the improved optimizer of GCC 4.0 realized that in this simple test program all symbols from the C++ runtime could be omitted whence linking with the C compiler succeeded. The new test generates two source files that more closely resemble the situation with ccpython.cc: main() calls a C function in another translation unit. main() is compiled with the C++ compiler, the C function with the C compiler. Only if the resulting object files can be linked into an executable by the C compiler then configure sets 'LINKCC=$(PURIFY) $(CC)'. I tested the patch with GCC 4.0 and GCC 2.95.2 on i686-pc-linux-gnu. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2005-08-07 23:10 Message: Logged In: YES user_id=21627 Thanks for the patch. Committed as configure 1.473 configure.in 1.486 NEWS 1.1331 configure 1.462.2.9 configure.in 1.475.2.9 NEWS 1.1193.2.70 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1239112&group_id=5470 From noreply at sourceforge.net Sun Aug 7 23:38:45 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 07 Aug 2005 14:38:45 -0700 Subject: [Patches] [ python-Patches-1214879 ] Support non-file source/dest in marshal Message-ID: Patches item #1214879, was opened at 2005-06-04 19:20 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1214879&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Skip Montanaro (montanaro) Assigned to: Nobody/Anonymous (nobody) Summary: Support non-file source/dest in marshal Initial Comment: I own bug #780354. The problem is that socket.makefile() no longer returns a true file. This breaks code that had assumed it did. The marshal module is one such instance. In the above bug's corresponce Martin stated: Instead, I think the C modules that expect file objects need to be fixed. Later on Martin stated that he thought it would be fine to simply document the limitation for the marshal module. I ignored that and fixed it in the attached patch. It includes modified docs and an updated test suite. It's obviously somewhat slower than the current version when reading/writing true files. That's to be expected. I wanted to get it right first. I imagine something can be done to speed things up in the common case (run PyFile_Check() and make calls to stdio functions if it is a true file). Even if this patch is rejected, the modified test_marshal.py should probably be kept (with suitable deletions), as it refactors the individual tests into helper functions, thus avoiding a lot of code duplication and potential future cut-n-paste errors. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2005-08-07 23:38 Message: Logged In: YES user_id=21627 I think I would prefer if WFILE continued to support FILE* as a special case, for performance reasons. Also, since write() will be called really often (once per byte), it might be worth to speed up the call somewhat: - look up file.write initially, and store that in WFILE, instead of storing the object itself. Make sure to DECREF .write when you are done. - Call the function through a tuple, to avoid the argument parsing. You know in advance that there is a single only a single argument, and that this is a string. Also, I'm worried about the error handling. If .write() raises an exception, marshalling should abort, IMO. The right way would be to add an error code to all w_* functions, and immediately return when there was an error. However, just putting the WFILE into an error state (by setting p->error to 3), and then refusing to perform further .write calls might be sufficient. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1214879&group_id=5470 From noreply at sourceforge.net Sun Aug 7 23:49:03 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 07 Aug 2005 14:49:03 -0700 Subject: [Patches] [ python-Patches-1192789 ] Use GCC4 ELF symbol visibility Message-ID: Patches item #1192789, was opened at 2005-04-30 01:46 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1192789&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: James Henstridge (jhenstridge) Assigned to: Nobody/Anonymous (nobody) Summary: Use GCC4 ELF symbol visibility Initial Comment: GCC4 includes some new features that make it easier to control what symbols are exported from a library for linking. The attached patch makes Python use these features, limiting the exported symbols to the public Python C API (i.e. the same symbols as are exported under Windows). The patch sets the ELF visibility of the public Python APIs to "protected", which means that the symbol is exported but internal calls to those symbols are done as direct function calls. In my local tests on i386 Linux, pystone and parrotbench run about 5% faster on a "--enable-shared" build of Python with the patch applied. For a non shared library build of Python, performance is pretty much identical (I'll probably need to do some more tests). However it still has the benefit of not exporting private Python API. The patch also touches the init routines of a number of extension modules, since they weren't marked with PyMODINIT_FUNC, which caused them to build incorrectly with the patch (the init function ended up being private, so Python couldn't load the module). I've only tested this patch against 2.4.1, but it probably applies without too much trouble to the 2.5 development line. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2005-08-07 23:49 Message: Logged In: YES user_id=21627 Doesn't this cause backwards compatibility problems with extension modules which fail to declare their entry point with PyMODINIT_FUNC? Such modules would stop working when recompiled with the patch, right? Also, shouldn't the -fvisibility argument only be used if the __attribute__ syntax is supported? ---------------------------------------------------------------------- Comment By: James Henstridge (jhenstridge) Date: 2005-05-10 04:47 Message: Logged In: YES user_id=146903 I just tried out the patch on an AMD64 Linux machine. There is some improvement in the --enable-shared case, but it doesn't seem to be as big as on i386. I only tested pystone though, since parrotbench gives an assertion error on this box. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1192789&group_id=5470 From noreply at sourceforge.net Sun Aug 7 23:51:41 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 07 Aug 2005 14:51:41 -0700 Subject: [Patches] [ python-Patches-1177307 ] UTF-8-Sig codec Message-ID: Patches item #1177307, was opened at 2005-04-05 21:26 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1177307&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Walter D?rwald (doerwalter) Assigned to: Nobody/Anonymous (nobody) Summary: UTF-8-Sig codec Initial Comment: This patch implements a UTF-8-Sig codec. This codec works like UTF-8 but adds a BOM on writing and skips (at most) one BOM on reading. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2005-08-07 23:51 Message: Logged In: YES user_id=21627 The patch looks fine, but lacks documentation changes. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2005-04-05 22:28 Message: Logged In: YES user_id=89016 This second version of the patch will return starting bytes immediately, if they don't look like a BOM. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1177307&group_id=5470 From noreply at sourceforge.net Mon Aug 8 08:49:50 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 07 Aug 2005 23:49:50 -0700 Subject: [Patches] [ python-Patches-1214889 ] file.encoding support for file.write and file.writelines Message-ID: Patches item #1214889, was opened at 2005-06-04 19:45 Message generated for change (Comment added) made by birkenfeld You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1214889&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.5 >Status: Closed >Resolution: Rejected Priority: 5 Submitted By: Reinhold Birkenfeld (birkenfeld) Assigned to: Nobody/Anonymous (nobody) Summary: file.encoding support for file.write and file.writelines Initial Comment: Here is a patch that allows Unicode strings written to a file being automatically encoded. It enables Python code to set file.encoding and obeys this encoding when writing Unicode strings with write() or writelines(). It is my first core hackery, so forgive me one leaked ref or the other. I hope I got the error handling right; it is kind of confusing... (btw: Bug #967986 will be fixed with this) ---------------------------------------------------------------------- >Comment By: Reinhold Birkenfeld (birkenfeld) Date: 2005-08-08 08:49 Message: Logged In: YES user_id=1188172 Rejecting. This is incomplete and will be addressed more properly in Py3k. ---------------------------------------------------------------------- Comment By: Reinhold Birkenfeld (birkenfeld) Date: 2005-07-14 10:34 Message: Logged In: YES user_id=1188172 I agree with you that writing Unicode objects to a binary file should raise an exception, but with the 'et#' format string, 8-bit string objects should pass through file.write unrecoded. About your second comment: Yes, codecs is one way to do it, but then I think that the encoding handling for print should be ripped out, too. After all, that's what many people complain about: "print unistr" works, while "sys.stdout.write(unistr)" does not. As the comment below about bug 1099364 shows, this shows up in various locations. If this is rejected, file.write() shouldn't accept Unicode anymore, and print should behave the same way. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2005-07-14 10:19 Message: Logged In: YES user_id=38388 I've thought about this some more: I'm not sure whether it is such a good idea to try to move code from the codecs into the standard file object - after all, the codecs already support all this and do a much better job at handling error cases and the like. Furthermore, codecs support both directions: reading and writing. Your patch only does one way. The encoding support you currently find in the file object is only needed for printing Unicode objects - it is not used anywhere else. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2005-07-14 09:52 Message: Logged In: YES user_id=38388 This doesn't quite work (yet): you've broken the support for writing binary data to the file via file.write(). Encodings should only be used for non-binary files. Also note that you are not freeing the memory allocated by the "et#" parser for s. Please add some test cases where you open a binary file and write: a) binary strings b) contents of a buffer object c) Unicode objects to it. Case c) should raise an exception. a) and b) should result in the data being written as-is - without doing any recoding. ---------------------------------------------------------------------- Comment By: Petr Prikryl (prikryl) Date: 2005-07-12 11:59 Message: Logged In: YES user_id=771873 The title and the comments do not say so, but the patch was created by Reinhold Birkenfeld to solve the bug [ 1099364 ] raw_input() displays wrong unicode prompt As the bug was closed and Reinhold claims to be his "first core hackery", I'd like to ask someone else to revise, whether the patch is the correct solution to the reported bug. The bug seems to be very visible (hence serious) in non-English speaking countries where Unicode promisses to solve many problems. Because of that I ask whether the bug should be closed before accepting the patch. I am adding this text also to link this patch to the original problem. Thanks, Petr ---------------------------------------------------------------------- Comment By: Reinhold Birkenfeld (birkenfeld) Date: 2005-06-05 18:31 Message: Logged In: YES user_id=1188172 Okay, put on #5. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2005-06-05 17:20 Message: Logged In: YES user_id=55188 Yet another thing to fix: You can't put local namespace declarations after non-declaration statements. Because Python uses C89 as a C source code standard, you should all declarations in the top of functions only. ---------------------------------------------------------------------- Comment By: Reinhold Birkenfeld (birkenfeld) Date: 2005-06-05 14:17 Message: Logged In: YES user_id=1188172 Thanks! Corrected in patch #4. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2005-06-05 14:09 Message: Logged In: YES user_id=671362 Reinhold, libstdtypes.tex needs two fixes. \versionadded{2.3} +\versionchanged[The encoding attribute is now writable and is used +for encoding Unicode strings given to \method{write()} and +\method{writelines()}.]{ ~~~ First, versionchanged tag does not have a trailing brace and it resuls in compile error. Second(really trivial), versionchanged macro automatically appends a period at the end of the sentence(see the link [*]), so you don't need to put it by hand. Then the above line would become: +\method{writelines()}]{2.5} [*]: http://docs.python.org/doc/inline-markup.html ---------------------------------------------------------------------- Comment By: Reinhold Birkenfeld (birkenfeld) Date: 2005-06-05 09:56 Message: Logged In: YES user_id=1188172 Third revision; adds new documentation and allows Python code to set the encoding to Py_None. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2005-06-05 04:26 Message: Logged In: YES user_id=55188 The idea looks good to me. I attached a revised patch fixed code style, C99-style local variable declaration and added a regrtest. I think some documentation update will be needed also. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1214889&group_id=5470 From noreply at sourceforge.net Tue Aug 9 06:03:55 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 08 Aug 2005 21:03:55 -0700 Subject: [Patches] [ python-Patches-1254695 ] QuickTime API needs corrected object types Message-ID: Patches item #1254695, was opened at 2005-08-09 00:03 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1254695&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Macintosh Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Christopher K Davis (ckdavis) Assigned to: Jack Jansen (jackjansen) Summary: QuickTime API needs corrected object types Initial Comment: Many of the objects in the Carbon.Qt module are typed as Component (CmpObj) when the QuickTime API documents them as ComponentInstance (CmpInstObj). This makes it impossible to use GetGraphicsImporterForFile and then GraphicsImportGetAsPicture, because the latter is incorrectly expecting a CmpObj while the former is correctly returning a CmpInstObj. Attached is a patch to qtsupport.py correcting the incorrect CmpObj typing, checked by reference to the QuickTime API docs. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1254695&group_id=5470 From noreply at sourceforge.net Tue Aug 9 09:08:48 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 09 Aug 2005 00:08:48 -0700 Subject: [Patches] [ python-Patches-1254718 ] GCC detection for runtime_library_dirs when ccache is used Message-ID: Patches item #1254718, was opened at 2005-08-09 13:53 Message generated for change (Comment added) made by sanxiyn You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1254718&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Distutils and setup.py Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Seo Sanghyeon (sanxiyn) Assigned to: Nobody/Anonymous (nobody) Summary: GCC detection for runtime_library_dirs when ccache is used Initial Comment: Recently I discovered ccache and am very happily using it. However, some Python extensions (python-ldap, to be precise) didn't link correctly, ignoring runtime_library_dirs, only if I used ccache. (CC='ccache gcc' python setup.py...) Attached patch fixes this by treating any compiler with string gcc or g++ in it as GCC, not only those start with gcc or g++. This feels like hack over hack, but oh well. ---------------------------------------------------------------------- >Comment By: Seo Sanghyeon (sanxiyn) Date: 2005-08-09 16:08 Message: Logged In: YES user_id=837148 Hye-Shik Chang pointed out that Intel C Compiler(icc) needs -Wl,-R as well, not -R. Perhaps this should check for icc too? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1254718&group_id=5470 From noreply at sourceforge.net Tue Aug 9 17:03:43 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 09 Aug 2005 08:03:43 -0700 Subject: [Patches] [ python-Patches-1245379 ] Encoding alias "unicode-1-1-utf-7" Message-ID: Patches item #1245379, was opened at 2005-07-26 18:37 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1245379&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Oren Tirosh (orenti) Assigned to: Nobody/Anonymous (nobody) Summary: Encoding alias "unicode-1-1-utf-7" Initial Comment: The utf7 encoding seems to be used quite a lot in Europe for emails (probably because it's more compact and readable than utf8+base64 or utf8+quopri). Some systems use the encoding name "unicode-1-1-utf-7" for 'utf7': http://www.google.com/search?q=unicode-1-1-utf-7 This patch adds this name as an alias for utf7. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-08-09 17:03 Message: Logged In: YES user_id=21627 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. In addition, even if you *did* check this checkbox, a bug in SourceForge prevents attaching a file when *creating* an issue. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1245379&group_id=5470 From noreply at sourceforge.net Tue Aug 9 17:08:08 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 09 Aug 2005 08:08:08 -0700 Subject: [Patches] [ python-Patches-1231336 ] Add unicode for sys.argv, os.environ, os.system Message-ID: Patches item #1231336, was opened at 2005-07-02 03:55 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1231336&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Neil Hodgson (nyamatongwe) Assigned to: Nobody/Anonymous (nobody) Summary: Add unicode for sys.argv, os.environ, os.system Initial Comment: Most installations of Windows (2000, XP) are unicode native with narrow character APIs only providing a distorted view of the system. Python does not currently provide access to some basic features through wide character calls and so may see distorted values. This patch adds unicode compatibility for sys.argv, os.environ, and os.system. os.system accepts a unicode argument in the same way as described in PEP 277 for file APIs. For sys.argv and os.environ, new parallel unicode attributes sys.argvu and os.environu are added as it would cause too many problems to use unicode values for the existing attributes or to use unicode only for non-ASCII values. The features are only enabled on unicode native versions of Windows. The three features are demonstrated at http://www.scintilla.org/pyunicode.png The patch contains some documentation additions for sys.argvu and os.environu. There are no test cases as test cases involving running extra processes can be messy and fail for uninteresting reasons. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2005-08-09 17:08 Message: Logged In: YES user_id=21627 I think the discussion came to the following conclusion: environu should not be added, instead, os.environ should have Unicode where necessary (i.e. non-ASCII), I guess this applies both to keys and to values. Are you interested in revising the patch in this direction? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-07-15 07:17 Message: Logged In: YES user_id=21627 Sorry, I missed the discussion; reopening. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-15 02:17 Message: Logged In: YES user_id=12579 I thought that posixmodule.c was creating os.environ but now see the code in os.py. "As you are not willing to discuss the issues on python-dev". Eh? I thought that was what I was doing in the "Adding the 'path' module" thread. You can reject the patch due to the discussion on python-dev but I don't think the given reason is valid. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2005-07-14 20:33 Message: Logged In: YES user_id=38388 Just as data point: the idea of using the type of a dictionary key to determine the resulting return type is a really bad design idea - just like the idea to let functions determine their return type based on the types of their input parameters. These things should always be made explicit, e.g. os.environ.get_unicode(), sys.argv.get_unicode() etc. However, as the discussion on python-dev shows, we may not need this kind of approach at all. Cheers, Marc-Andre. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-07-14 20:10 Message: Logged In: YES user_id=21627 os.environ is not a dictionary, it is a UserDict.IterableUserDict. Discerning strings and Unicode object would well be possible. As you are not willing to discuss the issues on python-dev, I'm rejecting the patch. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-14 12:35 Message: Logged In: YES user_id=12579 os.environ is a dictionary and unicode keys can not be discerned from string keys. For sys.argv it appears that there is no support for the "parallel universe" approach with sys.argvu and I expect one of the "promotion" models will be chosen. The patch should be rejected (or parked?) until consensus emerges. os.system was only included to allow testing but I saw difficulties in writing robust unit tests for these features so didn't include any. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-07-11 18:42 Message: Logged In: YES user_id=21627 For os.environ, I think I would prefer a solution where Unicode keys result in Unicode values and string keys result in string values, with the canonical conversion through "mbcs" in place. For argv, I agree something should be done, but I'm not certain that the introduction of argvu is the best thing to do; this should be dicsussed on python-dev, and with all people originally involved in PEP 277. The change to system() is not mentioned at all in your message. It doesn't seem to belong into this patch, either, so please submit it as a separate patch. If system() is changed to support Unicode commands, I think spawn*() should be changed as well. These seem less debatable, as they come as natural extensions to PEP 277 (i.e. pass Unicode through to the system). ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-05 06:27 Message: Logged In: YES user_id=12579 Added a description to diff file. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-05 06:25 Message: Logged In: YES user_id=12579 There are problems in sys.argvu as the current argument processing code removes some option arguments where these are processed by python. This can be almost fixed by storing the argc last elements into sys.argvu. However, when using [-c command], the command is removed from sys.argv as this allows the Python code to determine that it is either running with a command line command ("-c") or the name of the file. Attached patch fixes these problems. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1231336&group_id=5470 From noreply at sourceforge.net Tue Aug 9 15:41:12 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 09 Aug 2005 06:41:12 -0700 Subject: [Patches] [ python-Patches-1177307 ] UTF-8-Sig codec Message-ID: Patches item #1177307, was opened at 2005-04-05 21:26 Message generated for change (Comment added) made by doerwalter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1177307&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Walter D?rwald (doerwalter) Assigned to: Nobody/Anonymous (nobody) Summary: UTF-8-Sig codec Initial Comment: This patch implements a UTF-8-Sig codec. This codec works like UTF-8 but adds a BOM on writing and skips (at most) one BOM on reading. ---------------------------------------------------------------------- >Comment By: Walter D?rwald (doerwalter) Date: 2005-08-09 15:41 Message: Logged In: YES user_id=89016 This version (diff3.txt) of the patch adds a note to Misc/NEWS and a section to Doc/lib/libcodecs.tex. Is this the correct place to add the documentation? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-08-07 23:51 Message: Logged In: YES user_id=21627 The patch looks fine, but lacks documentation changes. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2005-04-05 22:28 Message: Logged In: YES user_id=89016 This second version of the patch will return starting bytes immediately, if they don't look like a BOM. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1177307&group_id=5470 From noreply at sourceforge.net Tue Aug 9 17:03:03 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 09 Aug 2005 08:03:03 -0700 Subject: [Patches] [ python-Patches-1180695 ] st_gen and st_birthtime support for FreeBSD Message-ID: Patches item #1180695, was opened at 2005-04-11 15:04 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1180695&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.4 >Status: Closed >Resolution: Accepted Priority: 7 Submitted By: Antti Louko (alouko) Assigned to: Martin v. L?wis (loewis) Summary: st_gen and st_birthtime support for FreeBSD Initial Comment: This patch implements nanosecond resolution in stat results for FreeBSD systems. It also adds st_birthtime and st_gen (only seen by root) fields. Tested on FreeBSD 5.3 ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2005-08-09 17:03 Message: Logged In: YES user_id=21627 Thanks for the patch. Committed as configure 1.474 configure.in 1.487 pyconfig.h.in 1.108 libos.tex 1.165 NEWS 1.1332 posixmodule.c 2.338 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1180695&group_id=5470 From noreply at sourceforge.net Tue Aug 9 20:53:44 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 09 Aug 2005 11:53:44 -0700 Subject: [Patches] [ python-Patches-1177307 ] UTF-8-Sig codec Message-ID: Patches item #1177307, was opened at 2005-04-05 21:26 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1177307&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Walter D?rwald (doerwalter) Assigned to: Nobody/Anonymous (nobody) Summary: UTF-8-Sig codec Initial Comment: This patch implements a UTF-8-Sig codec. This codec works like UTF-8 but adds a BOM on writing and skips (at most) one BOM on reading. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2005-08-09 20:53 Message: Logged In: YES user_id=21627 The place is right, but I feel this documentation is incomplete still. The library reference should explain somewhere what the difference between utf-8 and utf-8-sig is. Perhaps a footnote could be added. I think I would prefer a separate subsection on the BOM, explaining byte order in UTF-{16,32}, and how the BOM can be used as a magic signature for UTF-8. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2005-08-09 15:41 Message: Logged In: YES user_id=89016 This version (diff3.txt) of the patch adds a note to Misc/NEWS and a section to Doc/lib/libcodecs.tex. Is this the correct place to add the documentation? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-08-07 23:51 Message: Logged In: YES user_id=21627 The patch looks fine, but lacks documentation changes. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2005-04-05 22:28 Message: Logged In: YES user_id=89016 This second version of the patch will return starting bytes immediately, if they don't look like a BOM. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1177307&group_id=5470 From noreply at sourceforge.net Tue Aug 9 06:53:25 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 08 Aug 2005 21:53:25 -0700 Subject: [Patches] [ python-Patches-1254718 ] GCC detection for runtime_library_dirs when ccache is used Message-ID: Patches item #1254718, was opened at 2005-08-09 13:53 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1254718&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Distutils and setup.py Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Seo Sanghyeon (sanxiyn) Assigned to: Nobody/Anonymous (nobody) Summary: GCC detection for runtime_library_dirs when ccache is used Initial Comment: Recently I discovered ccache and am very happily using it. However, some Python extensions (python-ldap, to be precise) didn't link correctly, ignoring runtime_library_dirs, only if I used ccache. (CC='ccache gcc' python setup.py...) Attached patch fixes this by treating any compiler with string gcc or g++ in it as GCC, not only those start with gcc or g++. This feels like hack over hack, but oh well. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1254718&group_id=5470 From noreply at sourceforge.net Wed Aug 10 01:23:46 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 09 Aug 2005 16:23:46 -0700 Subject: [Patches] [ python-Patches-1231336 ] Add unicode for sys.argv, os.environ, os.system Message-ID: Patches item #1231336, was opened at 2005-07-02 11:55 Message generated for change (Comment added) made by nyamatongwe You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1231336&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Neil Hodgson (nyamatongwe) Assigned to: Nobody/Anonymous (nobody) Summary: Add unicode for sys.argv, os.environ, os.system Initial Comment: Most installations of Windows (2000, XP) are unicode native with narrow character APIs only providing a distorted view of the system. Python does not currently provide access to some basic features through wide character calls and so may see distorted values. This patch adds unicode compatibility for sys.argv, os.environ, and os.system. os.system accepts a unicode argument in the same way as described in PEP 277 for file APIs. For sys.argv and os.environ, new parallel unicode attributes sys.argvu and os.environu are added as it would cause too many problems to use unicode values for the existing attributes or to use unicode only for non-ASCII values. The features are only enabled on unicode native versions of Windows. The three features are demonstrated at http://www.scintilla.org/pyunicode.png The patch contains some documentation additions for sys.argvu and os.environu. There are no test cases as test cases involving running extra processes can be messy and fail for uninteresting reasons. ---------------------------------------------------------------------- >Comment By: Neil Hodgson (nyamatongwe) Date: 2005-08-10 09:23 Message: Logged In: YES user_id=12579 Marc-Andre Lemburg's point of view that os.environ use unicode when the string is outside Python's default encoding attracted most support. For the reasons given in the discussion, I feel this will cause problems for users. It is more difficult to code than a CP_ACP or non-ASCII test and there would be flow-on work for other calls such as open that would need to convert from the default encoding to Unicode. Due to the size of these changes and my doubts about this being the correct design, I don't want to work on its implementation. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-08-10 01:08 Message: Logged In: YES user_id=21627 I think the discussion came to the following conclusion: environu should not be added, instead, os.environ should have Unicode where necessary (i.e. non-ASCII), I guess this applies both to keys and to values. Are you interested in revising the patch in this direction? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-07-15 15:17 Message: Logged In: YES user_id=21627 Sorry, I missed the discussion; reopening. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-15 10:17 Message: Logged In: YES user_id=12579 I thought that posixmodule.c was creating os.environ but now see the code in os.py. "As you are not willing to discuss the issues on python-dev". Eh? I thought that was what I was doing in the "Adding the 'path' module" thread. You can reject the patch due to the discussion on python-dev but I don't think the given reason is valid. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2005-07-15 04:33 Message: Logged In: YES user_id=38388 Just as data point: the idea of using the type of a dictionary key to determine the resulting return type is a really bad design idea - just like the idea to let functions determine their return type based on the types of their input parameters. These things should always be made explicit, e.g. os.environ.get_unicode(), sys.argv.get_unicode() etc. However, as the discussion on python-dev shows, we may not need this kind of approach at all. Cheers, Marc-Andre. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-07-15 04:10 Message: Logged In: YES user_id=21627 os.environ is not a dictionary, it is a UserDict.IterableUserDict. Discerning strings and Unicode object would well be possible. As you are not willing to discuss the issues on python-dev, I'm rejecting the patch. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-14 20:35 Message: Logged In: YES user_id=12579 os.environ is a dictionary and unicode keys can not be discerned from string keys. For sys.argv it appears that there is no support for the "parallel universe" approach with sys.argvu and I expect one of the "promotion" models will be chosen. The patch should be rejected (or parked?) until consensus emerges. os.system was only included to allow testing but I saw difficulties in writing robust unit tests for these features so didn't include any. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-07-12 02:42 Message: Logged In: YES user_id=21627 For os.environ, I think I would prefer a solution where Unicode keys result in Unicode values and string keys result in string values, with the canonical conversion through "mbcs" in place. For argv, I agree something should be done, but I'm not certain that the introduction of argvu is the best thing to do; this should be dicsussed on python-dev, and with all people originally involved in PEP 277. The change to system() is not mentioned at all in your message. It doesn't seem to belong into this patch, either, so please submit it as a separate patch. If system() is changed to support Unicode commands, I think spawn*() should be changed as well. These seem less debatable, as they come as natural extensions to PEP 277 (i.e. pass Unicode through to the system). ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-05 14:27 Message: Logged In: YES user_id=12579 Added a description to diff file. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-05 14:25 Message: Logged In: YES user_id=12579 There are problems in sys.argvu as the current argument processing code removes some option arguments where these are processed by python. This can be almost fixed by storing the argc last elements into sys.argvu. However, when using [-c command], the command is removed from sys.argv as this allows the Python code to determine that it is either running with a command line command ("-c") or the name of the file. Attached patch fixes these problems. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1231336&group_id=5470 From noreply at sourceforge.net Wed Aug 10 09:17:13 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 10 Aug 2005 00:17:13 -0700 Subject: [Patches] [ python-Patches-1231336 ] Add unicode for sys.argv, os.environ, os.system Message-ID: Patches item #1231336, was opened at 2005-07-02 03:55 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1231336&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.5 >Status: Closed >Resolution: Rejected Priority: 5 Submitted By: Neil Hodgson (nyamatongwe) Assigned to: Nobody/Anonymous (nobody) Summary: Add unicode for sys.argv, os.environ, os.system Initial Comment: Most installations of Windows (2000, XP) are unicode native with narrow character APIs only providing a distorted view of the system. Python does not currently provide access to some basic features through wide character calls and so may see distorted values. This patch adds unicode compatibility for sys.argv, os.environ, and os.system. os.system accepts a unicode argument in the same way as described in PEP 277 for file APIs. For sys.argv and os.environ, new parallel unicode attributes sys.argvu and os.environu are added as it would cause too many problems to use unicode values for the existing attributes or to use unicode only for non-ASCII values. The features are only enabled on unicode native versions of Windows. The three features are demonstrated at http://www.scintilla.org/pyunicode.png The patch contains some documentation additions for sys.argvu and os.environu. There are no test cases as test cases involving running extra processes can be messy and fail for uninteresting reasons. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2005-08-10 09:17 Message: Logged In: YES user_id=21627 Ok, I think we have to reject this patch, then, and wait for somebody to write a PEP. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-08-10 01:23 Message: Logged In: YES user_id=12579 Marc-Andre Lemburg's point of view that os.environ use unicode when the string is outside Python's default encoding attracted most support. For the reasons given in the discussion, I feel this will cause problems for users. It is more difficult to code than a CP_ACP or non-ASCII test and there would be flow-on work for other calls such as open that would need to convert from the default encoding to Unicode. Due to the size of these changes and my doubts about this being the correct design, I don't want to work on its implementation. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-08-09 17:08 Message: Logged In: YES user_id=21627 I think the discussion came to the following conclusion: environu should not be added, instead, os.environ should have Unicode where necessary (i.e. non-ASCII), I guess this applies both to keys and to values. Are you interested in revising the patch in this direction? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-07-15 07:17 Message: Logged In: YES user_id=21627 Sorry, I missed the discussion; reopening. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-15 02:17 Message: Logged In: YES user_id=12579 I thought that posixmodule.c was creating os.environ but now see the code in os.py. "As you are not willing to discuss the issues on python-dev". Eh? I thought that was what I was doing in the "Adding the 'path' module" thread. You can reject the patch due to the discussion on python-dev but I don't think the given reason is valid. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2005-07-14 20:33 Message: Logged In: YES user_id=38388 Just as data point: the idea of using the type of a dictionary key to determine the resulting return type is a really bad design idea - just like the idea to let functions determine their return type based on the types of their input parameters. These things should always be made explicit, e.g. os.environ.get_unicode(), sys.argv.get_unicode() etc. However, as the discussion on python-dev shows, we may not need this kind of approach at all. Cheers, Marc-Andre. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-07-14 20:10 Message: Logged In: YES user_id=21627 os.environ is not a dictionary, it is a UserDict.IterableUserDict. Discerning strings and Unicode object would well be possible. As you are not willing to discuss the issues on python-dev, I'm rejecting the patch. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-14 12:35 Message: Logged In: YES user_id=12579 os.environ is a dictionary and unicode keys can not be discerned from string keys. For sys.argv it appears that there is no support for the "parallel universe" approach with sys.argvu and I expect one of the "promotion" models will be chosen. The patch should be rejected (or parked?) until consensus emerges. os.system was only included to allow testing but I saw difficulties in writing robust unit tests for these features so didn't include any. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-07-11 18:42 Message: Logged In: YES user_id=21627 For os.environ, I think I would prefer a solution where Unicode keys result in Unicode values and string keys result in string values, with the canonical conversion through "mbcs" in place. For argv, I agree something should be done, but I'm not certain that the introduction of argvu is the best thing to do; this should be dicsussed on python-dev, and with all people originally involved in PEP 277. The change to system() is not mentioned at all in your message. It doesn't seem to belong into this patch, either, so please submit it as a separate patch. If system() is changed to support Unicode commands, I think spawn*() should be changed as well. These seem less debatable, as they come as natural extensions to PEP 277 (i.e. pass Unicode through to the system). ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-05 06:27 Message: Logged In: YES user_id=12579 Added a description to diff file. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-05 06:25 Message: Logged In: YES user_id=12579 There are problems in sys.argvu as the current argument processing code removes some option arguments where these are processed by python. This can be almost fixed by storing the argc last elements into sys.argvu. However, when using [-c command], the command is removed from sys.argv as this allows the Python code to determine that it is either running with a command line command ("-c") or the name of the file. Attached patch fixes these problems. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1231336&group_id=5470 From noreply at sourceforge.net Wed Aug 10 09:24:39 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 10 Aug 2005 00:24:39 -0700 Subject: [Patches] [ python-Patches-1231336 ] Add unicode for sys.argv, os.environ, os.system Message-ID: Patches item #1231336, was opened at 2005-07-02 11:55 Message generated for change (Comment added) made by nyamatongwe You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1231336&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.5 Status: Closed Resolution: Rejected Priority: 5 Submitted By: Neil Hodgson (nyamatongwe) Assigned to: Nobody/Anonymous (nobody) Summary: Add unicode for sys.argv, os.environ, os.system Initial Comment: Most installations of Windows (2000, XP) are unicode native with narrow character APIs only providing a distorted view of the system. Python does not currently provide access to some basic features through wide character calls and so may see distorted values. This patch adds unicode compatibility for sys.argv, os.environ, and os.system. os.system accepts a unicode argument in the same way as described in PEP 277 for file APIs. For sys.argv and os.environ, new parallel unicode attributes sys.argvu and os.environu are added as it would cause too many problems to use unicode values for the existing attributes or to use unicode only for non-ASCII values. The features are only enabled on unicode native versions of Windows. The three features are demonstrated at http://www.scintilla.org/pyunicode.png The patch contains some documentation additions for sys.argvu and os.environu. There are no test cases as test cases involving running extra processes can be messy and fail for uninteresting reasons. ---------------------------------------------------------------------- >Comment By: Neil Hodgson (nyamatongwe) Date: 2005-08-10 17:24 Message: Logged In: YES user_id=12579 Yes, the scope of the changes needed requires a PEP and transition plan and needs to make sense in moving towards the all-unicode string future. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-08-10 17:17 Message: Logged In: YES user_id=21627 Ok, I think we have to reject this patch, then, and wait for somebody to write a PEP. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-08-10 09:23 Message: Logged In: YES user_id=12579 Marc-Andre Lemburg's point of view that os.environ use unicode when the string is outside Python's default encoding attracted most support. For the reasons given in the discussion, I feel this will cause problems for users. It is more difficult to code than a CP_ACP or non-ASCII test and there would be flow-on work for other calls such as open that would need to convert from the default encoding to Unicode. Due to the size of these changes and my doubts about this being the correct design, I don't want to work on its implementation. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-08-10 01:08 Message: Logged In: YES user_id=21627 I think the discussion came to the following conclusion: environu should not be added, instead, os.environ should have Unicode where necessary (i.e. non-ASCII), I guess this applies both to keys and to values. Are you interested in revising the patch in this direction? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-07-15 15:17 Message: Logged In: YES user_id=21627 Sorry, I missed the discussion; reopening. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-15 10:17 Message: Logged In: YES user_id=12579 I thought that posixmodule.c was creating os.environ but now see the code in os.py. "As you are not willing to discuss the issues on python-dev". Eh? I thought that was what I was doing in the "Adding the 'path' module" thread. You can reject the patch due to the discussion on python-dev but I don't think the given reason is valid. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2005-07-15 04:33 Message: Logged In: YES user_id=38388 Just as data point: the idea of using the type of a dictionary key to determine the resulting return type is a really bad design idea - just like the idea to let functions determine their return type based on the types of their input parameters. These things should always be made explicit, e.g. os.environ.get_unicode(), sys.argv.get_unicode() etc. However, as the discussion on python-dev shows, we may not need this kind of approach at all. Cheers, Marc-Andre. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-07-15 04:10 Message: Logged In: YES user_id=21627 os.environ is not a dictionary, it is a UserDict.IterableUserDict. Discerning strings and Unicode object would well be possible. As you are not willing to discuss the issues on python-dev, I'm rejecting the patch. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-14 20:35 Message: Logged In: YES user_id=12579 os.environ is a dictionary and unicode keys can not be discerned from string keys. For sys.argv it appears that there is no support for the "parallel universe" approach with sys.argvu and I expect one of the "promotion" models will be chosen. The patch should be rejected (or parked?) until consensus emerges. os.system was only included to allow testing but I saw difficulties in writing robust unit tests for these features so didn't include any. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-07-12 02:42 Message: Logged In: YES user_id=21627 For os.environ, I think I would prefer a solution where Unicode keys result in Unicode values and string keys result in string values, with the canonical conversion through "mbcs" in place. For argv, I agree something should be done, but I'm not certain that the introduction of argvu is the best thing to do; this should be dicsussed on python-dev, and with all people originally involved in PEP 277. The change to system() is not mentioned at all in your message. It doesn't seem to belong into this patch, either, so please submit it as a separate patch. If system() is changed to support Unicode commands, I think spawn*() should be changed as well. These seem less debatable, as they come as natural extensions to PEP 277 (i.e. pass Unicode through to the system). ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-05 14:27 Message: Logged In: YES user_id=12579 Added a description to diff file. ---------------------------------------------------------------------- Comment By: Neil Hodgson (nyamatongwe) Date: 2005-07-05 14:25 Message: Logged In: YES user_id=12579 There are problems in sys.argvu as the current argument processing code removes some option arguments where these are processed by python. This can be almost fixed by storing the argc last elements into sys.argvu. However, when using [-c command], the command is removed from sys.argv as this allows the Python code to determine that it is either running with a command line command ("-c") or the name of the file. Attached patch fixes these problems. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1231336&group_id=5470 From noreply at sourceforge.net Wed Aug 10 13:05:31 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 10 Aug 2005 04:05:31 -0700 Subject: [Patches] [ python-Patches-1093253 ] Refactoring Python/import.c Message-ID: Patches item #1093253, was opened at 2004-12-30 13:50 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1093253&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.5 >Status: Closed >Resolution: Rejected Priority: 5 Submitted By: Thomas Heller (theller) Assigned to: Nobody/Anonymous (nobody) Summary: Refactoring Python/import.c Initial Comment: This patch refactores Python/import.c. find_module() was changed to return an PyObject* pointer which contains the module's pathname, instead of filling out a char* buffer. load_module() accepts the PyObject* pathname instead of a char*. The patch is probably missing some error checking, and the 8 character hack for loading extensions on OS2 is not implemented, but the test case runs without errors on Windows XP pro. If a change in spirit of this patch is accepted, I'm willing to further work on it so that eventually unicode entries on sys.path, which can not be encoded with the default file system encodings, will work as expected (currently they don't). See also: http://mail.python.org/pipermail/python-list/2004-December/256969.html ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2005-08-10 13:05 Message: Logged In: YES user_id=11105 Self rejecting. The patch is out of date now, and there seems zero interest in it. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2005-05-04 20:43 Message: Logged In: YES user_id=11105 Unassigning from MvL. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2005-01-06 21:20 Message: Logged In: YES user_id=11105 Martin, this is the first step for making unicode entries on sys.path work the way that is expected on Windows. You requested patches, so please take a look, if you have time. Otherwise, unassign yourself. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2005-01-06 21:17 Message: Logged In: YES user_id=11105 For easier reading, I've attached the complete, new Python/import.c file. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2005-01-04 20:20 Message: Logged In: YES user_id=11105 New patch attached with multiple implementations of case_ok, and more error checking: import.c.patch2 Slightly tested on OSX, Linux, Windows. The case_ok function still needs to be fixed for RISCOS (which I cannot test). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-12-31 15:17 Message: Logged In: YES user_id=6656 Perhaps there should be multiple implementations of case_ok ... i.e. #if PLAT1 int case_ok(...) { ... } #elif PLAT2 int case_ok(...) { ... } #endif the current spaghetti is confusing, even by the standards of import.c... ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-12-31 15:11 Message: Logged In: YES user_id=11105 Yes, I overlooked that the initialization of the variables is inside an #if defined(MS_WINDOWS) block. Probably it would be better to leave the signature of case_ok() as before and call it through a wrapper which converts the arguments. I will prepare a new patch in a few days. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-12-31 14:57 Message: Logged In: YES user_id=6656 Applied the patch and built on OS X. This was the result: $ ./python.exe 'import site' failed; use -v for traceback ../Python/import.c:1496: failed assertion `dirlen <= MAXPATHLEN' Abort trap dirlen is 796092779, which seems fishy :) An uninitialized variable, maybe? Haven't looked, really... ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1093253&group_id=5470 From noreply at sourceforge.net Fri Aug 12 17:29:07 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 12 Aug 2005 08:29:07 -0700 Subject: [Patches] [ python-Patches-1252236 ] Simplying Tkinter's event loop Message-ID: Patches item #1252236, was opened at 2005-08-04 17:51 Message generated for change (Comment added) made by jimjjewett You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252236&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Tkinter Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Martin v. L?wis (loewis) Summary: Simplying Tkinter's event loop Initial Comment: This patch is a simplification of Tkinter's event loop by moving the event loop to Python itself. Tkinter needs an event loop to be able to handle both Tcl/Tk events and keyboard input by the user. Usually, an event loop looks like this: while true: wait for an event (Tcl/Tk events or keyboard input) handle the event Tkinter's loop is set up differently. If Tkinter is imported, it sets the PyOS_InputHook pointer to the EventHook function in _tkinter.c. The EventHook function handles Tcl/Tk events only. In essence, Tkinter's event loop looks like this: while true: handle all Tcl/Tk events that need to be handled if keyboard input is waiting: break sleep for 20 milliseconds Note that the event loop exits only if the user presses a key. This causes the following problems: 1) This event loop is available only for Tkinter. There is no way to handle events other than the Tcl/Tk events. 2) If another extension module needs events to be handled, it can do so by using a similar event loop. However, it is not possible to handle both Tcl/Tk events and other events, (except for keyboard input). 3) Chaining the two event loops will fail. As soon as Tkinter's event loop starts running, it will break out of this loop only in the case of keyboard input. So any other events will not be processed until a key is pressed. 4) Even if Tkinter's event loop is the only event loop that is needed, it can fail. If a Python thread is waiting for a mutex variable to change, it cannot run Tkinter's event loop while waiting, since Tkinter's event loop will only return after a key is pressed (it doesn't pay attention to the mutex variable). This is the reason that running Tkinter in an IDLE shell will fail. 5) On some platforms (e.g. Cygwin), the event loop is unable to check for keyboard input. Hence, one needs to call mainloop() in order to see Tkinter's widgets. The reason that the event loop was set up in this peculiar way may be due to a bug with PyOS_InputHook in Python. If Python is compiled with readline support, PyOS_InputHook will be called ten times per second, while Python is waiting for keyboard input. (As shown below, this allows us to fix this event loop problem.) However, if Python is compiled without readline support (notably on Windows), PyOS_InputHook is called only once (just before a user starts entering a new command on the keyboard). This necessitates Tkinter to run an event loop, and not return from it until keyboard input is ready. If PyOS_InputHook is called ten times per second, we can set up the event loop as follows: Set PyOS_InputHook to Tkinter's EventHook while true: call PyOS_InputHook handle keyboard input, if present sleep for 100 milliseconds Here, the key point is that Tkinter's EventHook returns immediately if there are no more Tcl/Tk events to be handled. Effectively, we have moved the event loop from Tkinter to Python itself. Hence, there are two problems to be solved: 1) PyOS_InputHook should be called ten times per second, regardless of whether Python is compiled with or without readline support. 2) Tkinter's EventHook should return immediately after handling all Tcl/Tk events. Problem 1) is solved in patch #1049855; problem 2) is solved in this patch. This patch is a considerable simplication of _tkinter.c, as it removes all lines that are no longer needed (and adds almost no code). This patch changes the nature of the function to which PyOS_InputHook points. Before, PyOS_InputHook is used to start a Tcl/Tk event loop. Now, PyOS_InputHook is called from an existing event loop to handle Tcl/Tk events. With these two patches, Python has a functioning PyOS_InputHook, which can be used both by Tkinter and by other extension modules. We can handle Tcl/Tk events by calling Tkinter's EventHook function if we're waiting for something else other than keyboard input, like a mutex variable. We can chain event-handling functions from different extension modules (e.g. Tkinter and some other module), by first calling Tkinter's EventHook and then the other extension module's event hook function. Finally, on Cygwin, we can now create Tkinter widgets without having to call mainloop. So this will now work on Cygwin: >>> from Tkinter import * >>> Label() >>> # label appears, without having to call mainloop. This patch has been tested on WIndows and Cygwin. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2005-08-12 11:29 Message: Logged In: YES user_id=764593 (1) Why was the behavior different before? Is 10 times per second not responsive enough? Does a busy-wait of 10 times per second cause too much thrashing? (2) It seems like the problem isn't really about Tkinter so much as it is about event loops vs threading. The event loop is infinite, so nothing else *in that thread* will happen after it. This isn't solvable with a single-threaded python. (On the other hand, single-threaded python should never have the mutex problem you mentioned.) (3) With multi-threaded python, is there any reason not to start the event loop in a fresh thread? (And let that new thread block waiting for events.) This would also reduce contention with other frameworks that want to treat the "main" thread differently. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252236&group_id=5470 From noreply at sourceforge.net Fri Aug 12 22:18:58 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 12 Aug 2005 13:18:58 -0700 Subject: [Patches] [ python-Patches-1257988 ] fix smtplib when local host isn't resolvable in dns Message-ID: Patches item #1257988, was opened at 2005-08-12 22:18 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1257988&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Arkadiusz Miskiewicz (arekm) Assigned to: Nobody/Anonymous (nobody) Summary: fix smtplib when local host isn't resolvable in dns Initial Comment: Suppose that hostname gives hostX. hostX doesn't exists in dns nor in / etc/hosts (it's unresolvable). Now when trying to connect to smtp service: >>> import smtplib >>> smtplib.SMTP('akcyza.pld-linux.org') Traceback (most recent call last): File "", line 1, in ? File "/usr/share/python2.4/smtplib.py", line 255, in __init__ socket.gaierror: (-2, 'Name or service not known') The problem is gethostbyname() here: # We can't find an fqdn hostname, so use a domain literal addr = socket.gethostbyname(socket.gethostname()) self.local_hostname = '[%s]' % addr It's much easier and better to just use getsockname() for local socket. Attached patch fixes this problem and doesn't require for the local system to be resolvable in dns/hosts. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1257988&group_id=5470 From noreply at sourceforge.net Fri Aug 12 23:52:03 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 12 Aug 2005 14:52:03 -0700 Subject: [Patches] [ python-Patches-1231053 ] audioop - alaw encoding/decoding added, code updated Message-ID: Patches item #1231053, was opened at 2005-07-02 02:04 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1231053&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Lars Immisch (larsimmisch) >Assigned to: Anthony Baxter (anthonybaxter) Summary: audioop - alaw encoding/decoding added, code updated Initial Comment: This patch adds a-LAW encoding to audioop and replaces[1] the old u-LAW encoding/decoding code with the current code from sox. If audioop has an ulaw codec, it should also have an alaw codec. Besides, shtoom wants/needs it. Tests and documentation are updated with this patch too. The patch is a unified diff against CVS that can be applied in one piece frrom the root of the python tree. Beyond the cursory tests in the unittests, I have done successful tests with real-life data. Possible issues: the code from sox uses int16_t. [1] Rationale for the replacement of existing code: the alaw code would have been different in style from the ulaw code and attribution of the code would have been less clear. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2005-08-13 07:52 Message: Logged In: YES user_id=29957 I'll look at this shortly. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1231053&group_id=5470 From noreply at sourceforge.net Sun Aug 14 21:06:22 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 14 Aug 2005 12:06:22 -0700 Subject: [Patches] [ python-Patches-1180695 ] st_gen and st_birthtime support for FreeBSD Message-ID: Patches item #1180695, was opened at 2005-04-11 14:04 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1180695&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.4 Status: Closed Resolution: Accepted Priority: 7 Submitted By: Antti Louko (alouko) Assigned to: Martin v. L?wis (loewis) Summary: st_gen and st_birthtime support for FreeBSD Initial Comment: This patch implements nanosecond resolution in stat results for FreeBSD systems. It also adds st_birthtime and st_gen (only seen by root) fields. Tested on FreeBSD 5.3 ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2005-08-14 20:06 Message: Logged In: YES user_id=6656 I'm fairly sure it's this patch that has broken Python CVS HEAD, at least on OS X. Some kind of refcounting snafu. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-08-09 16:03 Message: Logged In: YES user_id=21627 Thanks for the patch. Committed as configure 1.474 configure.in 1.487 pyconfig.h.in 1.108 libos.tex 1.165 NEWS 1.1332 posixmodule.c 2.338 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1180695&group_id=5470 From noreply at sourceforge.net Mon Aug 15 00:03:28 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 14 Aug 2005 15:03:28 -0700 Subject: [Patches] [ python-Patches-1180695 ] st_gen and st_birthtime support for FreeBSD Message-ID: Patches item #1180695, was opened at 2005-04-11 15:04 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1180695&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.4 Status: Closed Resolution: Accepted Priority: 7 Submitted By: Antti Louko (alouko) Assigned to: Martin v. L?wis (loewis) Summary: st_gen and st_birthtime support for FreeBSD Initial Comment: This patch implements nanosecond resolution in stat results for FreeBSD systems. It also adds st_birthtime and st_gen (only seen by root) fields. Tested on FreeBSD 5.3 ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2005-08-15 00:03 Message: Logged In: YES user_id=21627 Please check again - the numbering of fields was wrong. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-08-14 21:06 Message: Logged In: YES user_id=6656 I'm fairly sure it's this patch that has broken Python CVS HEAD, at least on OS X. Some kind of refcounting snafu. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-08-09 17:03 Message: Logged In: YES user_id=21627 Thanks for the patch. Committed as configure 1.474 configure.in 1.487 pyconfig.h.in 1.108 libos.tex 1.165 NEWS 1.1332 posixmodule.c 2.338 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1180695&group_id=5470 From noreply at sourceforge.net Mon Aug 15 05:28:13 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 14 Aug 2005 20:28:13 -0700 Subject: [Patches] [ python-Patches-1121611 ] sha and md5 modules should use OpenSSL when possible Message-ID: Patches item #1121611, was opened at 2005-02-12 17:33 Message generated for change (Comment added) made by greg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121611&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Gregory P. Smith (greg) Assigned to: Gregory P. Smith (greg) Summary: sha and md5 modules should use OpenSSL when possible Initial Comment: The md5 and sha (sha1) modules should use OpenSSL for the algorithms when it is available as its implementations are much faster than pythons own. Attached is an initial patch to use OpenSSL for the sha module. Its not ready for committing as is yet, but it is setup to be a generic base for all OpenSSL hashes with a little bit of work in the future. Tossing this out there for people to see how trivial it is and enjoy the speedups. diff is against HEAD but it should apply to 2.4 just fine. ---------------------------------------------------------------------- >Comment By: Gregory P. Smith (greg) Date: 2005-08-14 20:28 Message: Logged In: YES user_id=413 tjreedy and arigo's comments have been taken into consideration. An updated patch (009) has been attached. it uses the python >= 2.2 interface for defining methods and member variables rather than the getattr function with manual strcmp's. I was unable to make digest_size and such class attributes because the hashes are not classes. The hashlib.md5 function for instance is a constructor function that returns an appropriate internal HASH object. The goal of those constructors is to be as fast as possible; wrapping them with python in order to make them actual classes would be too slow and I did not see an obvious way to do it from C. I believe this patch is ready to commit. Further improvements or refinements can be made to it in CVS. the documentation in html for easy viewing has been updated at http://electricrain.com/greg/hashlib-py25-doc/ ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-07-31 18:29 Message: Logged In: YES user_id=413 per arigo's suggestion I have a version of _hashopenssl.c in my sandbox modified to use the more modern C type API. The constructor is slightly faster (~1-2%) and does seem like a better way to do things. i'll post it after polishing it up. ---------------------------------------------------------------------- Comment By: Terry J. Reedy (tjreedy) Date: 2005-06-12 13:35 Message: Logged In: YES user_id=593130 Re Doc page: As a somewhat naive (relative to the subject) reader, the title and first sentence implied that 'secure hash' and 'message digest' are two separate things, whereas, judging from the .digest() blurb, they both seem to be16-byte hashes. So I would prefer this equivalence and the actual meaning were made clear at the top. Something like "This module implements a common interface to several secure hash or message digest algorithms that produce 16-byte hashes." If, as I presume, xx.hexdigest() == binascii.hexlify(xx.digest()), then I would say so and reference binsacii for the interconversion functions one would need if one had the two versions to compare or needed to convert after the extraction. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2005-06-12 05:18 Message: Logged In: YES user_id=4771 On a side note, maybe it makes sense for a new module like this one to promote and use the modern (>=2.2) ways of defining C types. What I have in mind is using tp_methods instead of Py_FindMethod, and generally not reverting to strcmp(). In this case, the constants like 'digest_size' would be best stored as class attributes instead, if possible. Indeed, allowing expressions like "hashlib.md5.digest_size" conveys the idea that the result doesn't depend on a particular instance, unlike "hashlib.md5().digest_size". (Of course class attributes are also readable from the instance, as usual.) I can give it a try if you don't want to invest more time in this patch than you already did (for which we are grateful to you :-) ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-06-11 20:21 Message: Logged In: YES user_id=413 Ok, this patch is ready. documentation has been added. I'll bring it up on python-dev for discussion/approval with a link to the htmlified documentation. The speedups are great for any application hashing a lot of data when OpenSSL is used. It also adds a sha224, sha256, sha384 and sha512 support. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-12 17:13 Message: Logged In: YES user_id=413 I linked a _hashlib.so library statically against openssl and reran the speed test. no change. that means its not shared library overhead causing the higher startup time but just an artifact of the OpenSSL EVP interface. Next up, analyze what size things common heavy sha1 using applications regularly hash (BitTorrent and such). ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-10 00:09 Message: Logged In: YES user_id=413 The 007 patch improves the speed of the constructor. There is still a potential speed issue with the constructor/destructor to work on: greg at spiff src $ ./python Lib/test/test_hashlib_speed.py _sha testing speed of old _sha legacy interface 0.06 seconds [20000 creations] 0.24 seconds [20000 "" digests] 0.15 seconds 20 x 106201 bytes [huge data] 0.15 seconds 200 x 10620 bytes [large data] 0.17 seconds 2000 x 1062 bytes [medium data] 0.35 seconds 20020 x 106 bytes [small data] 1.37 seconds 106200 x 20 bytes [digest_size data] 2.75 seconds 212400 x 10 bytes [tiny data] greg at spiff src $ ./python Lib/test/test_hashlib_speed.py sha1 testing speed of hashlib.sha1 0.22 seconds [20000 creations] 0.57 seconds [20000 "" digests] 0.09 seconds 20 x 106201 bytes [huge data] 0.09 seconds 200 x 10620 bytes [large data] 0.15 seconds 2000 x 1062 bytes [medium data] 0.71 seconds 20020 x 106 bytes [small data] 3.39 seconds 106200 x 20 bytes [digest_size data] 6.70 seconds 212400 x 10 bytes [tiny data] I suspect the cause is either or both of the shared openssl library call overhead or the openssl EVP abstraction interface. The speed results are very similar to the above regardless of which digest is used (the above was a celeron 333mhz running linux). ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-03 13:15 Message: Logged In: YES user_id=413 hashlib-006.patch adds fast constructors and a speed test. documentation is the next step. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-01 01:14 Message: Logged In: YES user_id=413 hashlib-005.patch now passes its test suite and no problems appear in valgrind. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-02-28 10:11 Message: Logged In: YES user_id=413 a much updated patch (hashlib-patch-004.patch). it incorporates some suggestions as well as including sf patch 935454's sha256/224 and sha512/384 implementations. still not complete but shows the direction its going in (i see a segfault part way thru the test suite after running the sha512 tests). as for the private modules being under another package, i see no reason to do that since there aren't very many (how does that work for binary modules anyways?). ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2005-02-18 11:21 Message: Logged In: YES user_id=764593 Should the private modules (such as _sha) be placed in a crypto package, instead of directly in the parent/everything library? ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-02-16 22:46 Message: Logged In: YES user_id=413 hashes-openssl-002.patch replaces the sha and md5 modules with a general hashes module that wraps all hashes that OpenSSL supports. note that OpenSSLs implementations are much faster than the previous python versions as it choses versions optimized for your particular hardware. Incase python is compiled without openssl the hashes wrapper falls back on the old python sha and md5 module implementations. side note: This may be sufficient for the Debian folks to work around their random odd licensing issue. just have debian python depend on openssl; use this and remove the old md5 module/code that wouldn't get used anyways. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121611&group_id=5470 From noreply at sourceforge.net Mon Aug 15 06:46:53 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 14 Aug 2005 21:46:53 -0700 Subject: [Patches] [ python-Patches-1252236 ] Simplying Tkinter's event loop Message-ID: Patches item #1252236, was opened at 2005-08-05 06:51 Message generated for change (Comment added) made by mdehoon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252236&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Tkinter Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Martin v. L?wis (loewis) Summary: Simplying Tkinter's event loop Initial Comment: This patch is a simplification of Tkinter's event loop by moving the event loop to Python itself. Tkinter needs an event loop to be able to handle both Tcl/Tk events and keyboard input by the user. Usually, an event loop looks like this: while true: wait for an event (Tcl/Tk events or keyboard input) handle the event Tkinter's loop is set up differently. If Tkinter is imported, it sets the PyOS_InputHook pointer to the EventHook function in _tkinter.c. The EventHook function handles Tcl/Tk events only. In essence, Tkinter's event loop looks like this: while true: handle all Tcl/Tk events that need to be handled if keyboard input is waiting: break sleep for 20 milliseconds Note that the event loop exits only if the user presses a key. This causes the following problems: 1) This event loop is available only for Tkinter. There is no way to handle events other than the Tcl/Tk events. 2) If another extension module needs events to be handled, it can do so by using a similar event loop. However, it is not possible to handle both Tcl/Tk events and other events, (except for keyboard input). 3) Chaining the two event loops will fail. As soon as Tkinter's event loop starts running, it will break out of this loop only in the case of keyboard input. So any other events will not be processed until a key is pressed. 4) Even if Tkinter's event loop is the only event loop that is needed, it can fail. If a Python thread is waiting for a mutex variable to change, it cannot run Tkinter's event loop while waiting, since Tkinter's event loop will only return after a key is pressed (it doesn't pay attention to the mutex variable). This is the reason that running Tkinter in an IDLE shell will fail. 5) On some platforms (e.g. Cygwin), the event loop is unable to check for keyboard input. Hence, one needs to call mainloop() in order to see Tkinter's widgets. The reason that the event loop was set up in this peculiar way may be due to a bug with PyOS_InputHook in Python. If Python is compiled with readline support, PyOS_InputHook will be called ten times per second, while Python is waiting for keyboard input. (As shown below, this allows us to fix this event loop problem.) However, if Python is compiled without readline support (notably on Windows), PyOS_InputHook is called only once (just before a user starts entering a new command on the keyboard). This necessitates Tkinter to run an event loop, and not return from it until keyboard input is ready. If PyOS_InputHook is called ten times per second, we can set up the event loop as follows: Set PyOS_InputHook to Tkinter's EventHook while true: call PyOS_InputHook handle keyboard input, if present sleep for 100 milliseconds Here, the key point is that Tkinter's EventHook returns immediately if there are no more Tcl/Tk events to be handled. Effectively, we have moved the event loop from Tkinter to Python itself. Hence, there are two problems to be solved: 1) PyOS_InputHook should be called ten times per second, regardless of whether Python is compiled with or without readline support. 2) Tkinter's EventHook should return immediately after handling all Tcl/Tk events. Problem 1) is solved in patch #1049855; problem 2) is solved in this patch. This patch is a considerable simplication of _tkinter.c, as it removes all lines that are no longer needed (and adds almost no code). This patch changes the nature of the function to which PyOS_InputHook points. Before, PyOS_InputHook is used to start a Tcl/Tk event loop. Now, PyOS_InputHook is called from an existing event loop to handle Tcl/Tk events. With these two patches, Python has a functioning PyOS_InputHook, which can be used both by Tkinter and by other extension modules. We can handle Tcl/Tk events by calling Tkinter's EventHook function if we're waiting for something else other than keyboard input, like a mutex variable. We can chain event-handling functions from different extension modules (e.g. Tkinter and some other module), by first calling Tkinter's EventHook and then the other extension module's event hook function. Finally, on Cygwin, we can now create Tkinter widgets without having to call mainloop. So this will now work on Cygwin: >>> from Tkinter import * >>> Label() >>> # label appears, without having to call mainloop. This patch has been tested on WIndows and Cygwin. ---------------------------------------------------------------------- >Comment By: Michiel de Hoon (mdehoon) Date: 2005-08-15 13:46 Message: Logged In: YES user_id=488897 > (1) Why was the behavior different before? Actually, the behavior is not much different from before; this patch together with patch #1049855 essentially move the event loop from Tkinter to Python core, so that it is available to other extension modules also. But effectively the program goes through the same steps as before. If you want to know why the design of Tkinter's event loop is the way it is: I am not quite sure, but it may just be a quick solution to get Tkinter working (and it does work fine as long as you're interested in Tkinter only). Since Tcl/Tk already has an event loop, it is easy to run that event loop and let it check for keyboard input (via Tcl_CreateFileHandler in EventHook in _tkinter.c). Writing such a loop for Python is not extremely difficult but also not straightforward (see my woes with patch #1049855). > Is 10 times per second not responsive enough? With the current Python, the loop sleeps every 20 ms before checking for keyboard input and handling Tcl/Tk events. With the patch, Tcl/Tk events are handled every 100 ms; keyboard input is handled much faster (depending on how quickly "select" on Unix or "WaitForSingleObject" on Windows respond to keyboard input). Of course, these delays can be set to some other value in the code. > Does a busy-wait of 10 times per second cause too much thrashing? I am not sure if I understand this question correctly. The "select" function on Unix and "WaitForSingleObject" function on Windows do not do a busy-wait. > (2) It seems like the problem isn't really about Tkinter so > much as it is about event loops vs threading. The event > loop is infinite, so nothing else *in that thread* will happen > after it. This isn't solvable with a single-threaded python. Sure it's solvable. Even the current implementation of Tkinter can handle Tcl/Tk events as well as listen for keyboard input. If you import Tkinter and create a Tk widget, you can still issue Python commands and move the Tk widget around, even though they are running in the same thread. The problem with the current implementation is that it works for Tkinter only, and secondly, that it doesn't allow chaining of hook functions. Patch #1049855 solves this by calling select on Unix (WaitForSingleObject on Windows) to find out if keyboard input is available, and if not, to handle Tk/Tcl events. No need for a separate thread for that. > (On the other hand, single-threaded python should never > have the mutex problem you mentioned.) > (3) With multi-threaded python, is there any reason not to > start the event loop in a fresh thread? (And let that new > thread block waiting for events.) This would also reduce > contention with other frameworks that want to treat the > "main" thread differently. Yes, this is possible on multi-threaded python. However, an extension module (such as Tkinter) cannot rely on the assumption that Python is multi-threaded. Personally, I am interested in PyOS_InputHook for scientific visualization software, which is likely to be used on all kinds of outlandish systems. I don't know if I can expect a multi-threaded Python to be available on all those systems. Secondly, given that Python has the PyOS_InputHook functionality, why not make sure it is implemented correctly? Meaning that its behavior should not depend on whether readline is installed or not, and that its usage in Tkinter does not block other extension modules from using it. I am not sure if I interpreted all of your questions correctly. Please let me know if I didn't. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2005-08-13 00:29 Message: Logged In: YES user_id=764593 (1) Why was the behavior different before? Is 10 times per second not responsive enough? Does a busy-wait of 10 times per second cause too much thrashing? (2) It seems like the problem isn't really about Tkinter so much as it is about event loops vs threading. The event loop is infinite, so nothing else *in that thread* will happen after it. This isn't solvable with a single-threaded python. (On the other hand, single-threaded python should never have the mutex problem you mentioned.) (3) With multi-threaded python, is there any reason not to start the event loop in a fresh thread? (And let that new thread block waiting for events.) This would also reduce contention with other frameworks that want to treat the "main" thread differently. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252236&group_id=5470 From noreply at sourceforge.net Mon Aug 15 10:47:17 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 15 Aug 2005 01:47:17 -0700 Subject: [Patches] [ python-Patches-1121611 ] sha and md5 modules should use OpenSSL when possible Message-ID: Patches item #1121611, was opened at 2005-02-13 01:33 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121611&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Gregory P. Smith (greg) Assigned to: Gregory P. Smith (greg) Summary: sha and md5 modules should use OpenSSL when possible Initial Comment: The md5 and sha (sha1) modules should use OpenSSL for the algorithms when it is available as its implementations are much faster than pythons own. Attached is an initial patch to use OpenSSL for the sha module. Its not ready for committing as is yet, but it is setup to be a generic base for all OpenSSL hashes with a little bit of work in the future. Tossing this out there for people to see how trivial it is and enjoy the speedups. diff is against HEAD but it should apply to 2.4 just fine. ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2005-08-15 08:47 Message: Logged In: YES user_id=4771 I see that it would indeed be messy to have 'md5' be a type and 'digest_size' a class attribute given that 'md5' can come from various places depending on what is installed; moreover in the hashopenssl.c file unless I'm mistaken all hashes use the same Python type. Fine by me. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-08-15 03:28 Message: Logged In: YES user_id=413 tjreedy and arigo's comments have been taken into consideration. An updated patch (009) has been attached. it uses the python >= 2.2 interface for defining methods and member variables rather than the getattr function with manual strcmp's. I was unable to make digest_size and such class attributes because the hashes are not classes. The hashlib.md5 function for instance is a constructor function that returns an appropriate internal HASH object. The goal of those constructors is to be as fast as possible; wrapping them with python in order to make them actual classes would be too slow and I did not see an obvious way to do it from C. I believe this patch is ready to commit. Further improvements or refinements can be made to it in CVS. the documentation in html for easy viewing has been updated at http://electricrain.com/greg/hashlib-py25-doc/ ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-08-01 01:29 Message: Logged In: YES user_id=413 per arigo's suggestion I have a version of _hashopenssl.c in my sandbox modified to use the more modern C type API. The constructor is slightly faster (~1-2%) and does seem like a better way to do things. i'll post it after polishing it up. ---------------------------------------------------------------------- Comment By: Terry J. Reedy (tjreedy) Date: 2005-06-12 20:35 Message: Logged In: YES user_id=593130 Re Doc page: As a somewhat naive (relative to the subject) reader, the title and first sentence implied that 'secure hash' and 'message digest' are two separate things, whereas, judging from the .digest() blurb, they both seem to be16-byte hashes. So I would prefer this equivalence and the actual meaning were made clear at the top. Something like "This module implements a common interface to several secure hash or message digest algorithms that produce 16-byte hashes." If, as I presume, xx.hexdigest() == binascii.hexlify(xx.digest()), then I would say so and reference binsacii for the interconversion functions one would need if one had the two versions to compare or needed to convert after the extraction. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2005-06-12 12:18 Message: Logged In: YES user_id=4771 On a side note, maybe it makes sense for a new module like this one to promote and use the modern (>=2.2) ways of defining C types. What I have in mind is using tp_methods instead of Py_FindMethod, and generally not reverting to strcmp(). In this case, the constants like 'digest_size' would be best stored as class attributes instead, if possible. Indeed, allowing expressions like "hashlib.md5.digest_size" conveys the idea that the result doesn't depend on a particular instance, unlike "hashlib.md5().digest_size". (Of course class attributes are also readable from the instance, as usual.) I can give it a try if you don't want to invest more time in this patch than you already did (for which we are grateful to you :-) ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-06-12 03:21 Message: Logged In: YES user_id=413 Ok, this patch is ready. documentation has been added. I'll bring it up on python-dev for discussion/approval with a link to the htmlified documentation. The speedups are great for any application hashing a lot of data when OpenSSL is used. It also adds a sha224, sha256, sha384 and sha512 support. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-13 01:13 Message: Logged In: YES user_id=413 I linked a _hashlib.so library statically against openssl and reran the speed test. no change. that means its not shared library overhead causing the higher startup time but just an artifact of the OpenSSL EVP interface. Next up, analyze what size things common heavy sha1 using applications regularly hash (BitTorrent and such). ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-10 08:09 Message: Logged In: YES user_id=413 The 007 patch improves the speed of the constructor. There is still a potential speed issue with the constructor/destructor to work on: greg at spiff src $ ./python Lib/test/test_hashlib_speed.py _sha testing speed of old _sha legacy interface 0.06 seconds [20000 creations] 0.24 seconds [20000 "" digests] 0.15 seconds 20 x 106201 bytes [huge data] 0.15 seconds 200 x 10620 bytes [large data] 0.17 seconds 2000 x 1062 bytes [medium data] 0.35 seconds 20020 x 106 bytes [small data] 1.37 seconds 106200 x 20 bytes [digest_size data] 2.75 seconds 212400 x 10 bytes [tiny data] greg at spiff src $ ./python Lib/test/test_hashlib_speed.py sha1 testing speed of hashlib.sha1 0.22 seconds [20000 creations] 0.57 seconds [20000 "" digests] 0.09 seconds 20 x 106201 bytes [huge data] 0.09 seconds 200 x 10620 bytes [large data] 0.15 seconds 2000 x 1062 bytes [medium data] 0.71 seconds 20020 x 106 bytes [small data] 3.39 seconds 106200 x 20 bytes [digest_size data] 6.70 seconds 212400 x 10 bytes [tiny data] I suspect the cause is either or both of the shared openssl library call overhead or the openssl EVP abstraction interface. The speed results are very similar to the above regardless of which digest is used (the above was a celeron 333mhz running linux). ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-03 21:15 Message: Logged In: YES user_id=413 hashlib-006.patch adds fast constructors and a speed test. documentation is the next step. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-01 09:14 Message: Logged In: YES user_id=413 hashlib-005.patch now passes its test suite and no problems appear in valgrind. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-02-28 18:11 Message: Logged In: YES user_id=413 a much updated patch (hashlib-patch-004.patch). it incorporates some suggestions as well as including sf patch 935454's sha256/224 and sha512/384 implementations. still not complete but shows the direction its going in (i see a segfault part way thru the test suite after running the sha512 tests). as for the private modules being under another package, i see no reason to do that since there aren't very many (how does that work for binary modules anyways?). ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2005-02-18 19:21 Message: Logged In: YES user_id=764593 Should the private modules (such as _sha) be placed in a crypto package, instead of directly in the parent/everything library? ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-02-17 06:46 Message: Logged In: YES user_id=413 hashes-openssl-002.patch replaces the sha and md5 modules with a general hashes module that wraps all hashes that OpenSSL supports. note that OpenSSLs implementations are much faster than the previous python versions as it choses versions optimized for your particular hardware. Incase python is compiled without openssl the hashes wrapper falls back on the old python sha and md5 module implementations. side note: This may be sufficient for the Debian folks to work around their random odd licensing issue. just have debian python depend on openssl; use this and remove the old md5 module/code that wouldn't get used anyways. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121611&group_id=5470 From noreply at sourceforge.net Mon Aug 15 13:07:44 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 15 Aug 2005 04:07:44 -0700 Subject: [Patches] [ python-Patches-1192789 ] Use GCC4 ELF symbol visibility Message-ID: Patches item #1192789, was opened at 2005-04-30 07:46 Message generated for change (Comment added) made by jhenstridge You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1192789&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: James Henstridge (jhenstridge) Assigned to: Nobody/Anonymous (nobody) Summary: Use GCC4 ELF symbol visibility Initial Comment: GCC4 includes some new features that make it easier to control what symbols are exported from a library for linking. The attached patch makes Python use these features, limiting the exported symbols to the public Python C API (i.e. the same symbols as are exported under Windows). The patch sets the ELF visibility of the public Python APIs to "protected", which means that the symbol is exported but internal calls to those symbols are done as direct function calls. In my local tests on i386 Linux, pystone and parrotbench run about 5% faster on a "--enable-shared" build of Python with the patch applied. For a non shared library build of Python, performance is pretty much identical (I'll probably need to do some more tests). However it still has the benefit of not exporting private Python API. The patch also touches the init routines of a number of extension modules, since they weren't marked with PyMODINIT_FUNC, which caused them to build incorrectly with the patch (the init function ended up being private, so Python couldn't load the module). I've only tested this patch against 2.4.1, but it probably applies without too much trouble to the 2.5 development line. ---------------------------------------------------------------------- >Comment By: James Henstridge (jhenstridge) Date: 2005-08-15 19:07 Message: Logged In: YES user_id=146903 The PyMODINIT_FUNC() problem could be fixed by filtering out the -fvisibility=hidden flag from Makefile.pre.in (this would require some more extensive build changes). You are right about the patch relying on -fvisibility only being implemented by compilers that implement the visibility attribute. I think this is currently true, but it would be better not to rely on it. In the time since I made the patch though, I have been told about a problem with the "protected" visibility modifier: on Linux systems at least, it slows down the symbol lookup process. Since code inside the same library as the protected symbol will be using a direct reference and code outside will be going through the PLT, the dynamic linker needs special code to return different addresses for the symbol depending on what code is calling it (this is slower than the code path for "default" visibility symbols). This is required in order to satisfy the C standard's requirements that comparisons of pointers to functions be simple and fast. The way around this is to use two functions: one is a "hidden" visibility function that is used by the code inside libpython, and the other is an alias of the hidden function with normal visibility and the function's advertised name. Unfortunately, this is more invasive to implement (but not overly so -- libxml2 and GTK contain such code). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-08-08 05:49 Message: Logged In: YES user_id=21627 Doesn't this cause backwards compatibility problems with extension modules which fail to declare their entry point with PyMODINIT_FUNC? Such modules would stop working when recompiled with the patch, right? Also, shouldn't the -fvisibility argument only be used if the __attribute__ syntax is supported? ---------------------------------------------------------------------- Comment By: James Henstridge (jhenstridge) Date: 2005-05-10 10:47 Message: Logged In: YES user_id=146903 I just tried out the patch on an AMD64 Linux machine. There is some improvement in the --enable-shared case, but it doesn't seem to be as big as on i386. I only tested pystone though, since parrotbench gives an assertion error on this box. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1192789&group_id=5470 From noreply at sourceforge.net Mon Aug 15 15:35:11 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 15 Aug 2005 06:35:11 -0700 Subject: [Patches] [ python-Patches-1252236 ] Simplying Tkinter's event loop Message-ID: Patches item #1252236, was opened at 2005-08-04 17:51 Message generated for change (Comment added) made by jimjjewett You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252236&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Tkinter Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Martin v. L?wis (loewis) Summary: Simplying Tkinter's event loop Initial Comment: This patch is a simplification of Tkinter's event loop by moving the event loop to Python itself. Tkinter needs an event loop to be able to handle both Tcl/Tk events and keyboard input by the user. Usually, an event loop looks like this: while true: wait for an event (Tcl/Tk events or keyboard input) handle the event Tkinter's loop is set up differently. If Tkinter is imported, it sets the PyOS_InputHook pointer to the EventHook function in _tkinter.c. The EventHook function handles Tcl/Tk events only. In essence, Tkinter's event loop looks like this: while true: handle all Tcl/Tk events that need to be handled if keyboard input is waiting: break sleep for 20 milliseconds Note that the event loop exits only if the user presses a key. This causes the following problems: 1) This event loop is available only for Tkinter. There is no way to handle events other than the Tcl/Tk events. 2) If another extension module needs events to be handled, it can do so by using a similar event loop. However, it is not possible to handle both Tcl/Tk events and other events, (except for keyboard input). 3) Chaining the two event loops will fail. As soon as Tkinter's event loop starts running, it will break out of this loop only in the case of keyboard input. So any other events will not be processed until a key is pressed. 4) Even if Tkinter's event loop is the only event loop that is needed, it can fail. If a Python thread is waiting for a mutex variable to change, it cannot run Tkinter's event loop while waiting, since Tkinter's event loop will only return after a key is pressed (it doesn't pay attention to the mutex variable). This is the reason that running Tkinter in an IDLE shell will fail. 5) On some platforms (e.g. Cygwin), the event loop is unable to check for keyboard input. Hence, one needs to call mainloop() in order to see Tkinter's widgets. The reason that the event loop was set up in this peculiar way may be due to a bug with PyOS_InputHook in Python. If Python is compiled with readline support, PyOS_InputHook will be called ten times per second, while Python is waiting for keyboard input. (As shown below, this allows us to fix this event loop problem.) However, if Python is compiled without readline support (notably on Windows), PyOS_InputHook is called only once (just before a user starts entering a new command on the keyboard). This necessitates Tkinter to run an event loop, and not return from it until keyboard input is ready. If PyOS_InputHook is called ten times per second, we can set up the event loop as follows: Set PyOS_InputHook to Tkinter's EventHook while true: call PyOS_InputHook handle keyboard input, if present sleep for 100 milliseconds Here, the key point is that Tkinter's EventHook returns immediately if there are no more Tcl/Tk events to be handled. Effectively, we have moved the event loop from Tkinter to Python itself. Hence, there are two problems to be solved: 1) PyOS_InputHook should be called ten times per second, regardless of whether Python is compiled with or without readline support. 2) Tkinter's EventHook should return immediately after handling all Tcl/Tk events. Problem 1) is solved in patch #1049855; problem 2) is solved in this patch. This patch is a considerable simplication of _tkinter.c, as it removes all lines that are no longer needed (and adds almost no code). This patch changes the nature of the function to which PyOS_InputHook points. Before, PyOS_InputHook is used to start a Tcl/Tk event loop. Now, PyOS_InputHook is called from an existing event loop to handle Tcl/Tk events. With these two patches, Python has a functioning PyOS_InputHook, which can be used both by Tkinter and by other extension modules. We can handle Tcl/Tk events by calling Tkinter's EventHook function if we're waiting for something else other than keyboard input, like a mutex variable. We can chain event-handling functions from different extension modules (e.g. Tkinter and some other module), by first calling Tkinter's EventHook and then the other extension module's event hook function. Finally, on Cygwin, we can now create Tkinter widgets without having to call mainloop. So this will now work on Cygwin: >>> from Tkinter import * >>> Label() >>> # label appears, without having to call mainloop. This patch has been tested on WIndows and Cygwin. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2005-08-15 09:35 Message: Logged In: YES user_id=764593 Clarifying (1a) -- Why (pre-patch) were Windows and Unix intentionally set to act differently? Is there something in the default runtimes or libraries which makes (expected?) performance very different on the platforms? Clarifying (1b) -- Is ten times per second enough? 0.1 seconds is long enough that people can notice it. If the pre-patch version cycles 50 times/second, then going to only 10 times/second might make the interface seem sluggish. I'm not sure I'm qualified to make this judgement myself, but it is a concern. Clarifying (1c) -- My (possibly wrong) understanding is that before this pair of patches, unix effectively did an active check for input, but the windows version waited for notification from the OS that keyboard input was available -- and after this, both would actively check N times/ second. If both are just passively waiting, then I'm not sure what the 20ms/100ms timeout actually does. If python is running as a batch process, then forcing it back into the "is there input" section several times a second (even though there is never any keyboard input) will cause the program to take more clocktime. Clarifying (2) -- The pre-patch version can certainly take events (including keyboard events) during the event loop. What you can't do is: """ (define/run a bunch of stuff) ... start the event loop ... (define/run a bunch more stuff) """ You need to set up all the definitions and event handlers before the loop starts, or else do them as a result of events. Roughly, calling mainloop has to be the *last* thing you do in a given thread. Which leads to (3) Clarifying (3) -- Why not just assume threads? The problem you are trying to solve can't exist without threads. Assuming threads won't make anything fail any harder than it does now. If you default to dummy-threads and ensure that the event-loop the *last* pseudo-thread, you'll even clear up some bugs in carelessly written single- threaded code. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-08-15 00:46 Message: Logged In: YES user_id=488897 > (1) Why was the behavior different before? Actually, the behavior is not much different from before; this patch together with patch #1049855 essentially move the event loop from Tkinter to Python core, so that it is available to other extension modules also. But effectively the program goes through the same steps as before. If you want to know why the design of Tkinter's event loop is the way it is: I am not quite sure, but it may just be a quick solution to get Tkinter working (and it does work fine as long as you're interested in Tkinter only). Since Tcl/Tk already has an event loop, it is easy to run that event loop and let it check for keyboard input (via Tcl_CreateFileHandler in EventHook in _tkinter.c). Writing such a loop for Python is not extremely difficult but also not straightforward (see my woes with patch #1049855). > Is 10 times per second not responsive enough? With the current Python, the loop sleeps every 20 ms before checking for keyboard input and handling Tcl/Tk events. With the patch, Tcl/Tk events are handled every 100 ms; keyboard input is handled much faster (depending on how quickly "select" on Unix or "WaitForSingleObject" on Windows respond to keyboard input). Of course, these delays can be set to some other value in the code. > Does a busy-wait of 10 times per second cause too much thrashing? I am not sure if I understand this question correctly. The "select" function on Unix and "WaitForSingleObject" function on Windows do not do a busy-wait. > (2) It seems like the problem isn't really about Tkinter so > much as it is about event loops vs threading. The event > loop is infinite, so nothing else *in that thread* will happen > after it. This isn't solvable with a single-threaded python. Sure it's solvable. Even the current implementation of Tkinter can handle Tcl/Tk events as well as listen for keyboard input. If you import Tkinter and create a Tk widget, you can still issue Python commands and move the Tk widget around, even though they are running in the same thread. The problem with the current implementation is that it works for Tkinter only, and secondly, that it doesn't allow chaining of hook functions. Patch #1049855 solves this by calling select on Unix (WaitForSingleObject on Windows) to find out if keyboard input is available, and if not, to handle Tk/Tcl events. No need for a separate thread for that. > (On the other hand, single-threaded python should never > have the mutex problem you mentioned.) > (3) With multi-threaded python, is there any reason not to > start the event loop in a fresh thread? (And let that new > thread block waiting for events.) This would also reduce > contention with other frameworks that want to treat the > "main" thread differently. Yes, this is possible on multi-threaded python. However, an extension module (such as Tkinter) cannot rely on the assumption that Python is multi-threaded. Personally, I am interested in PyOS_InputHook for scientific visualization software, which is likely to be used on all kinds of outlandish systems. I don't know if I can expect a multi-threaded Python to be available on all those systems. Secondly, given that Python has the PyOS_InputHook functionality, why not make sure it is implemented correctly? Meaning that its behavior should not depend on whether readline is installed or not, and that its usage in Tkinter does not block other extension modules from using it. I am not sure if I interpreted all of your questions correctly. Please let me know if I didn't. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2005-08-12 11:29 Message: Logged In: YES user_id=764593 (1) Why was the behavior different before? Is 10 times per second not responsive enough? Does a busy-wait of 10 times per second cause too much thrashing? (2) It seems like the problem isn't really about Tkinter so much as it is about event loops vs threading. The event loop is infinite, so nothing else *in that thread* will happen after it. This isn't solvable with a single-threaded python. (On the other hand, single-threaded python should never have the mutex problem you mentioned.) (3) With multi-threaded python, is there any reason not to start the event loop in a fresh thread? (And let that new thread block waiting for events.) This would also reduce contention with other frameworks that want to treat the "main" thread differently. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252236&group_id=5470 From noreply at sourceforge.net Mon Aug 15 22:38:17 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 15 Aug 2005 13:38:17 -0700 Subject: [Patches] [ python-Patches-1192789 ] Use GCC4 ELF symbol visibility Message-ID: Patches item #1192789, was opened at 2005-04-30 01:46 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1192789&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: James Henstridge (jhenstridge) Assigned to: Nobody/Anonymous (nobody) Summary: Use GCC4 ELF symbol visibility Initial Comment: GCC4 includes some new features that make it easier to control what symbols are exported from a library for linking. The attached patch makes Python use these features, limiting the exported symbols to the public Python C API (i.e. the same symbols as are exported under Windows). The patch sets the ELF visibility of the public Python APIs to "protected", which means that the symbol is exported but internal calls to those symbols are done as direct function calls. In my local tests on i386 Linux, pystone and parrotbench run about 5% faster on a "--enable-shared" build of Python with the patch applied. For a non shared library build of Python, performance is pretty much identical (I'll probably need to do some more tests). However it still has the benefit of not exporting private Python API. The patch also touches the init routines of a number of extension modules, since they weren't marked with PyMODINIT_FUNC, which caused them to build incorrectly with the patch (the init function ended up being private, so Python couldn't load the module). I've only tested this patch against 2.4.1, but it probably applies without too much trouble to the 2.5 development line. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2005-08-15 22:38 Message: Logged In: YES user_id=21627 So would you agree that the patch should be rejected, then? I would classify it as "nice to have" - it doesn't fix an actual bug, nor does it add an important new feature. ---------------------------------------------------------------------- Comment By: James Henstridge (jhenstridge) Date: 2005-08-15 13:07 Message: Logged In: YES user_id=146903 The PyMODINIT_FUNC() problem could be fixed by filtering out the -fvisibility=hidden flag from Makefile.pre.in (this would require some more extensive build changes). You are right about the patch relying on -fvisibility only being implemented by compilers that implement the visibility attribute. I think this is currently true, but it would be better not to rely on it. In the time since I made the patch though, I have been told about a problem with the "protected" visibility modifier: on Linux systems at least, it slows down the symbol lookup process. Since code inside the same library as the protected symbol will be using a direct reference and code outside will be going through the PLT, the dynamic linker needs special code to return different addresses for the symbol depending on what code is calling it (this is slower than the code path for "default" visibility symbols). This is required in order to satisfy the C standard's requirements that comparisons of pointers to functions be simple and fast. The way around this is to use two functions: one is a "hidden" visibility function that is used by the code inside libpython, and the other is an alias of the hidden function with normal visibility and the function's advertised name. Unfortunately, this is more invasive to implement (but not overly so -- libxml2 and GTK contain such code). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-08-07 23:49 Message: Logged In: YES user_id=21627 Doesn't this cause backwards compatibility problems with extension modules which fail to declare their entry point with PyMODINIT_FUNC? Such modules would stop working when recompiled with the patch, right? Also, shouldn't the -fvisibility argument only be used if the __attribute__ syntax is supported? ---------------------------------------------------------------------- Comment By: James Henstridge (jhenstridge) Date: 2005-05-10 04:47 Message: Logged In: YES user_id=146903 I just tried out the patch on an AMD64 Linux machine. There is some improvement in the --enable-shared case, but it doesn't seem to be as big as on i386. I only tested pystone though, since parrotbench gives an assertion error on this box. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1192789&group_id=5470 From noreply at sourceforge.net Mon Aug 15 22:52:43 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 15 Aug 2005 13:52:43 -0700 Subject: [Patches] [ python-Patches-1175984 ] Make subprocess.Popen support file-like objects (win) Message-ID: Patches item #1175984, was opened at 2005-04-03 15:46 Message generated for change (Comment added) made by ianbicking You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1175984&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Nicolas Fleury (nidoizo) Assigned to: Peter ?strand (astrand) Summary: Make subprocess.Popen support file-like objects (win) Initial Comment: This patch make subprocess.Popen support file-like objects without fileno for stdout and stderr on Windows (intend to do Unix once patch accepted). It makes subprocess.Popen able to redirect stdout and stderr to objects like StringIO or other objects with file objects interface. ---------------------------------------------------------------------- Comment By: Ian Bicking (ianbicking) Date: 2005-08-15 15:52 Message: Logged In: YES user_id=210337 Bug/feature request 1260171 addresses the same basic issue, but with the idea of a method like Popen.communicate() that writes and reads directly to/from file-like objects, instead of reading and returning strings. http://sourceforge.net/tracker/index.php?func=detail&aid=1260171&group_id=5470&atid=105470 But having submitted that other request, I have no real opinion of one technique over the other, except that one of them would be really nice to have. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1175984&group_id=5470 From noreply at sourceforge.net Tue Aug 16 18:30:16 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 16 Aug 2005 09:30:16 -0700 Subject: [Patches] [ python-Patches-1252236 ] Simplying Tkinter's event loop Message-ID: Patches item #1252236, was opened at 2005-08-05 06:51 Message generated for change (Comment added) made by mdehoon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252236&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Tkinter Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Martin v. L?wis (loewis) Summary: Simplying Tkinter's event loop Initial Comment: This patch is a simplification of Tkinter's event loop by moving the event loop to Python itself. Tkinter needs an event loop to be able to handle both Tcl/Tk events and keyboard input by the user. Usually, an event loop looks like this: while true: wait for an event (Tcl/Tk events or keyboard input) handle the event Tkinter's loop is set up differently. If Tkinter is imported, it sets the PyOS_InputHook pointer to the EventHook function in _tkinter.c. The EventHook function handles Tcl/Tk events only. In essence, Tkinter's event loop looks like this: while true: handle all Tcl/Tk events that need to be handled if keyboard input is waiting: break sleep for 20 milliseconds Note that the event loop exits only if the user presses a key. This causes the following problems: 1) This event loop is available only for Tkinter. There is no way to handle events other than the Tcl/Tk events. 2) If another extension module needs events to be handled, it can do so by using a similar event loop. However, it is not possible to handle both Tcl/Tk events and other events, (except for keyboard input). 3) Chaining the two event loops will fail. As soon as Tkinter's event loop starts running, it will break out of this loop only in the case of keyboard input. So any other events will not be processed until a key is pressed. 4) Even if Tkinter's event loop is the only event loop that is needed, it can fail. If a Python thread is waiting for a mutex variable to change, it cannot run Tkinter's event loop while waiting, since Tkinter's event loop will only return after a key is pressed (it doesn't pay attention to the mutex variable). This is the reason that running Tkinter in an IDLE shell will fail. 5) On some platforms (e.g. Cygwin), the event loop is unable to check for keyboard input. Hence, one needs to call mainloop() in order to see Tkinter's widgets. The reason that the event loop was set up in this peculiar way may be due to a bug with PyOS_InputHook in Python. If Python is compiled with readline support, PyOS_InputHook will be called ten times per second, while Python is waiting for keyboard input. (As shown below, this allows us to fix this event loop problem.) However, if Python is compiled without readline support (notably on Windows), PyOS_InputHook is called only once (just before a user starts entering a new command on the keyboard). This necessitates Tkinter to run an event loop, and not return from it until keyboard input is ready. If PyOS_InputHook is called ten times per second, we can set up the event loop as follows: Set PyOS_InputHook to Tkinter's EventHook while true: call PyOS_InputHook handle keyboard input, if present sleep for 100 milliseconds Here, the key point is that Tkinter's EventHook returns immediately if there are no more Tcl/Tk events to be handled. Effectively, we have moved the event loop from Tkinter to Python itself. Hence, there are two problems to be solved: 1) PyOS_InputHook should be called ten times per second, regardless of whether Python is compiled with or without readline support. 2) Tkinter's EventHook should return immediately after handling all Tcl/Tk events. Problem 1) is solved in patch #1049855; problem 2) is solved in this patch. This patch is a considerable simplication of _tkinter.c, as it removes all lines that are no longer needed (and adds almost no code). This patch changes the nature of the function to which PyOS_InputHook points. Before, PyOS_InputHook is used to start a Tcl/Tk event loop. Now, PyOS_InputHook is called from an existing event loop to handle Tcl/Tk events. With these two patches, Python has a functioning PyOS_InputHook, which can be used both by Tkinter and by other extension modules. We can handle Tcl/Tk events by calling Tkinter's EventHook function if we're waiting for something else other than keyboard input, like a mutex variable. We can chain event-handling functions from different extension modules (e.g. Tkinter and some other module), by first calling Tkinter's EventHook and then the other extension module's event hook function. Finally, on Cygwin, we can now create Tkinter widgets without having to call mainloop. So this will now work on Cygwin: >>> from Tkinter import * >>> Label() >>> # label appears, without having to call mainloop. This patch has been tested on WIndows and Cygwin. ---------------------------------------------------------------------- >Comment By: Michiel de Hoon (mdehoon) Date: 2005-08-17 01:30 Message: Logged In: YES user_id=488897 > Clarifying (1a) -- Why (pre-patch) were Windows and > Unix intentionally set to act differently? Is there something > in the default runtimes or libraries which makes > (expected?) performance very different on the platforms? The difference is not Windows versus Unix, but Unix-with-readline versus Unix-without-readline and Windows-without-readline. I wasn't there when PyOS_InputHook was first added to Python, but I doubt that the difference in behavior was intentional. Essentially, I think that the behavior of Tkinter (on Python-without-readline) is a design flaw. > Clarifying (1b) -- Is ten times per second enough? 0.1 > seconds is long enough that people can notice it. If the > pre-patch version cycles 50 times/second, then going to > only 10 times/second might make the interface seem > sluggish. I'm not sure I'm qualified to make this > judgement myself, but it is a concern. Good point. The 10 times/second is the default used in readline, and hence this is the frequency at which PyOS_InputHook is called in Python-with-readline. I can modify the patch to increase this to 50 times/second. > Clarifying (1c) -- My (possibly wrong) understanding is > that before this pair of patches, unix effectively did an > active check for input, but the windows version waited for > notification from the OS that keyboard input was available > -- and after this, both would actively check N times/ > second. If both are just passively waiting, then I'm not > sure what the 20ms/100ms timeout actually does. That is correct. Pre-patch, both Unix and Windows sit in a loop inside Tkinter's EventHook function that calls Tcl_DoOneEvent and check for keyboard input, and then sleep for 20 ms. The check for keyboard input is done via a call to _kbhit (on Windows) or select (on Unix, hiding inside Tcl/Tk's Tcl_DoOneEvent function). Post-patch, this loop is located inside the my_fgets function. It works essentially the same, except that I'm using WaitForSingleObject (on Windows) / select (on Unix) for the timeout. > If python is running as a batch process, then forcing it > back into the "is there input" section several times a > second (even though there is never any keyboard input) > will cause the program to take more clocktime. Also a good point. I will modify the patch such that it will skip to fgets immediately if PyOS_InputHook==NULL. In a batch process, there is no reason to load a GUI extension module such as Tkinter, so PyOS_InputHook should be NULL. > Clarifying (2) -- The pre-patch version can certainly take > events (including keyboard events) during the event loop. > What you can't do is: > """ > (define/run a bunch of stuff) > ... > start the event loop > ... > (define/run a bunch more stuff) > """ Sure you can. I do it all the time. Try this, for example: >>> from Tkinter import * >>> Label(text="Label1").pack() # ... Label1 appears >>> print "more stuff I want to do" more stuff I want to do Here, Label1 appears and responds to events, even though I did not run mainloop. Note that pre-patch, this does not work on Cygwin; post-patch, it works on Cygwin too. > You need to set up all the definitions and event handlers > before the loop starts, or else do them as a result of > events. Roughly, calling mainloop has to be the *last* > thing you do in a given thread. Which leads to (3) You don't need to call mainloop at all. This is quite useful for scientific visualization: >>> do_some_calculation() >>> plot_the_result() >>> if dont_like_the_plot: do_some_more_calculations() >>> plot_the_new_result() Here, you never need to start or stop the event loop. It continues running as long as PyOS_InputHook is set appropriately by the extension module that does the plotting. > Clarifying (3) -- Why not just assume threads? The > problem you are trying to solve can't exist without threads. > Assuming threads won't make anything fail any harder > than it does now. If you default to dummy-threads and > ensure that the event-loop the *last* pseudo-thread, you'll > even clear up some bugs in carelessly written single- > threaded code. Whether or not Tkinter should have been coded with threads is a separate question. Tkinter is written for a single thread, using an event loop to handle window events and keyboard input. While it is possible to rewrite Tkinter to use threads, that would constitute a separate patch (and would probably not have a good chance of getting accepted, since Tkinter already works well as a single thread). ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2005-08-15 22:35 Message: Logged In: YES user_id=764593 Clarifying (1a) -- Why (pre-patch) were Windows and Unix intentionally set to act differently? Is there something in the default runtimes or libraries which makes (expected?) performance very different on the platforms? Clarifying (1b) -- Is ten times per second enough? 0.1 seconds is long enough that people can notice it. If the pre-patch version cycles 50 times/second, then going to only 10 times/second might make the interface seem sluggish. I'm not sure I'm qualified to make this judgement myself, but it is a concern. Clarifying (1c) -- My (possibly wrong) understanding is that before this pair of patches, unix effectively did an active check for input, but the windows version waited for notification from the OS that keyboard input was available -- and after this, both would actively check N times/ second. If both are just passively waiting, then I'm not sure what the 20ms/100ms timeout actually does. If python is running as a batch process, then forcing it back into the "is there input" section several times a second (even though there is never any keyboard input) will cause the program to take more clocktime. Clarifying (2) -- The pre-patch version can certainly take events (including keyboard events) during the event loop. What you can't do is: """ (define/run a bunch of stuff) ... start the event loop ... (define/run a bunch more stuff) """ You need to set up all the definitions and event handlers before the loop starts, or else do them as a result of events. Roughly, calling mainloop has to be the *last* thing you do in a given thread. Which leads to (3) Clarifying (3) -- Why not just assume threads? The problem you are trying to solve can't exist without threads. Assuming threads won't make anything fail any harder than it does now. If you default to dummy-threads and ensure that the event-loop the *last* pseudo-thread, you'll even clear up some bugs in carelessly written single- threaded code. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2005-08-15 13:46 Message: Logged In: YES user_id=488897 > (1) Why was the behavior different before? Actually, the behavior is not much different from before; this patch together with patch #1049855 essentially move the event loop from Tkinter to Python core, so that it is available to other extension modules also. But effectively the program goes through the same steps as before. If you want to know why the design of Tkinter's event loop is the way it is: I am not quite sure, but it may just be a quick solution to get Tkinter working (and it does work fine as long as you're interested in Tkinter only). Since Tcl/Tk already has an event loop, it is easy to run that event loop and let it check for keyboard input (via Tcl_CreateFileHandler in EventHook in _tkinter.c). Writing such a loop for Python is not extremely difficult but also not straightforward (see my woes with patch #1049855). > Is 10 times per second not responsive enough? With the current Python, the loop sleeps every 20 ms before checking for keyboard input and handling Tcl/Tk events. With the patch, Tcl/Tk events are handled every 100 ms; keyboard input is handled much faster (depending on how quickly "select" on Unix or "WaitForSingleObject" on Windows respond to keyboard input). Of course, these delays can be set to some other value in the code. > Does a busy-wait of 10 times per second cause too much thrashing? I am not sure if I understand this question correctly. The "select" function on Unix and "WaitForSingleObject" function on Windows do not do a busy-wait. > (2) It seems like the problem isn't really about Tkinter so > much as it is about event loops vs threading. The event > loop is infinite, so nothing else *in that thread* will happen > after it. This isn't solvable with a single-threaded python. Sure it's solvable. Even the current implementation of Tkinter can handle Tcl/Tk events as well as listen for keyboard input. If you import Tkinter and create a Tk widget, you can still issue Python commands and move the Tk widget around, even though they are running in the same thread. The problem with the current implementation is that it works for Tkinter only, and secondly, that it doesn't allow chaining of hook functions. Patch #1049855 solves this by calling select on Unix (WaitForSingleObject on Windows) to find out if keyboard input is available, and if not, to handle Tk/Tcl events. No need for a separate thread for that. > (On the other hand, single-threaded python should never > have the mutex problem you mentioned.) > (3) With multi-threaded python, is there any reason not to > start the event loop in a fresh thread? (And let that new > thread block waiting for events.) This would also reduce > contention with other frameworks that want to treat the > "main" thread differently. Yes, this is possible on multi-threaded python. However, an extension module (such as Tkinter) cannot rely on the assumption that Python is multi-threaded. Personally, I am interested in PyOS_InputHook for scientific visualization software, which is likely to be used on all kinds of outlandish systems. I don't know if I can expect a multi-threaded Python to be available on all those systems. Secondly, given that Python has the PyOS_InputHook functionality, why not make sure it is implemented correctly? Meaning that its behavior should not depend on whether readline is installed or not, and that its usage in Tkinter does not block other extension modules from using it. I am not sure if I interpreted all of your questions correctly. Please let me know if I didn't. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2005-08-13 00:29 Message: Logged In: YES user_id=764593 (1) Why was the behavior different before? Is 10 times per second not responsive enough? Does a busy-wait of 10 times per second cause too much thrashing? (2) It seems like the problem isn't really about Tkinter so much as it is about event loops vs threading. The event loop is infinite, so nothing else *in that thread* will happen after it. This isn't solvable with a single-threaded python. (On the other hand, single-threaded python should never have the mutex problem you mentioned.) (3) With multi-threaded python, is there any reason not to start the event loop in a fresh thread? (And let that new thread block waiting for events.) This would also reduce contention with other frameworks that want to treat the "main" thread differently. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1252236&group_id=5470 From noreply at sourceforge.net Wed Aug 17 14:33:14 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 17 Aug 2005 05:33:14 -0700 Subject: [Patches] [ python-Patches-1262036 ] tarfile: fix for bug #1257255 Message-ID: Patches item #1262036, was opened at 2005-08-17 14:33 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1262036&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Lars Gust?bel (gustaebel) Assigned to: Nobody/Anonymous (nobody) Summary: tarfile: fix for bug #1257255 Initial Comment: (See bug #1257255 for a detailed description of the problem.) While the problem of the OP is that tarfile won't work when the current working dir has been removed from the filesystem, the more important one for me is that the test in the add() method, that should prevent the archive from being added to itself, will only succeed by accident. So, I decided to save the TarFile's name as an absolute path from the beginning to ensure that the archive cannot be added to itself even if the cwd changed during the operation. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1262036&group_id=5470 From eurosemi at eurosemi.eu.com Thu Aug 18 15:31:01 2005 From: eurosemi at eurosemi.eu.com (eurosemi@eurosemi.eu.com) Date: Thu, 18 Aug 2005 14:31:01 +0100 (BST) Subject: [Patches] Eurosemi E-News Message-ID: <20050818133101.B1DF9D6D04@silux.angelbc.co.uk> charset="iso-8859-1" Content-Transfer-Encoding:8bit +---------------------------------------------------------+ | EUROSEMI NEWSLETTER - PLAIN TEXT VERSION | | if you would like to see this in HTML goto: | | http://www.eurosemi.eu.com | +---------------------------------------------------------+ | 16th August 2005 +---------------------------------------------------------+ | Contents | | ~~~~~~~~~ | | - Global semi sales fall 0.5% to $18 billion in June | - Infineon hit by the DRAM wars | - Sematech develops pore-sealing for 45nm low-k | - NOR market forecast to decline by 15% in 2005 | - EDA industry suffers dissapointing start to 2005 +---------------------------------------------------------+ +---------------------------------------------------------+ Global semi sales fall 0.5% to $18 billion in June Sales of semiconductors (three-month running average) declined slightly in June to $18.0 billion, a sequential fall of 0.5% from the $18.1 billion reported in May, according to the latest World Semiconductor Trade Statistics (WSTS). To view the whole story goto:http://www.eurosemi.eu.com/cgi-bin/viewnew22.pl?id=6485 +---------------------------------------------------------+ Infineon hit by the DRAM wars German chip maker Infineon has fallen three positions in market analyst IC Insight's ranking of leading semiconductor companies for the first half of 2005. Intel remains by far the largest company while Freescale, which ranked 11th in the fullyear 2004 listing, has entered the top ten, moving into ninth position in 1H05. To view the whole story goto:http://www.eurosemi.eu.com/cgi-bin/viewnew22.pl?id=6486 +---------------------------------------------------------+ Sematech develops pore-sealing for 45nm low-k Engineers from US research consortium Sematech have developed an innovative poresealing technique that appears to prevent metal and precursor penetration into low-k dielectric materials, easing the introduction of low-k at the 45nm technology node. To view the whole story goto:http://www.eurosemi.eu.com/cgi-bin/viewnew22.pl?id=6487 +---------------------------------------------------------+ NOR market forecast to decline by 15% in 2005 Sales of NOR flash memory could fall below NAND flash sales for the first time in 2005, according to the latest forecast from Semico Research. To view the whole story goto:http://www.eurosemi.eu.com/cgi-bin/viewnew22.pl?id=6488 +---------------------------------------------------------+ EDA industry suffers dissapointing start to 2005 The electronic design automation (EDA) industry recorded revenues of $989 million for Q1 of 2005 compared with $995 million for the same period in 2004. To view the whole story goto:http://www.eurosemi.eu.com/cgi-bin/viewnew22.pl?id=6489 +--------------------------------------------------------+ | To Unsubscribe Goto | Should you wish to unsubscribe please go to: http://www.eurosemi.eu.com/front-end/unsubscribe2.php?email=patches at python.org +-------------------------------------------------------- -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20050818/57b5f2a0/attachment-0001.htm From noreply at sourceforge.net Thu Aug 18 17:28:00 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 18 Aug 2005 08:28:00 -0700 Subject: [Patches] [ python-Patches-936169 ] CodeContext - an extension to show you where you are Message-ID: Patches item #936169, was opened at 2004-04-16 11:40 Message generated for change (Comment added) made by taleinat You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=936169&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: None Status: Open Resolution: Accepted Priority: 5 Submitted By: Noam Raphael (noamr) Assigned to: Kurt B. Kaiser (kbk) Summary: CodeContext - an extension to show you where you are Initial Comment: Have you ever edited a code, and did not know exactly which block you were closing? Or didn't know exactly even in which block you were? This extension solves this problem, by adding a few lines (3 by default) above the IDLE edit box, which show you exactly where you are in your code. For example, you may be editing a code with three indentation levels (has three tabs on its left), but to find the reason for this - to find what is the meaning of your block - you have to scroll upwards a long distance. CodeContext will add three lines of text above your edit box, so your editing window will look like this: ---------------------------------- Class MyClass: def myMethod(self, arg1, arg2): <- auto-updated if something_happens: ---------------------------------- i += 1 print "hello" <- You edit here ... ---------------------------------- So you can know that i is incremented by 1, in the method myMethod of class MyClass, and only if something_happens. To make this extension work, you have to put the attached file, CodeContext.py, in your IDLE directory, and add these lines to config-extensions.def: ---------------------------------- [CodeContext] enable=1 numlines=3 bgcolor=LightGray fgcolor=Black ---------------------------------- A note to KBK: I think this extension can go into the Python distribution. If you don't like it, or if you want a testing period, you can put "enable=0" instead of "enable=1". In this way, most users will not even know CodeContext exists, but "power users" will be able to put "enable=1" and have CodeContext if they like it (- I like it). Best wishes, Noam Raphael ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2005-08-18 18:28 Message: Logged In: YES user_id=1330769 This fixes the text alignment: (just replace te appropriate code in toggle_code_context_event with this) if not self.label: self.padding_frame = Tkinter.Frame(self.editwin.top, bg=self.bgcolor, border=2, relief="sunken", ) self.label = Tkinter.Label(self.padding_frame, text="\n" * (self.numlines - 1), anchor="w", justify="left", font=self.textfont, bg=self.bgcolor, fg=self.fgcolor, border=0, width=1, # Don't request more than we get ) self.label.pack(side="top", fill="x", expand=1, padx=4, pady=0) self.padding_frame.pack(side="top", fill="x", expand=0, padx=0, pady=0, after=self.editwin.status_bar) else: self.label.destroy() self.padding_frame.destroy() self.label = None Sorry it's not a diff... ---------------------------------------------------------------------- Comment By: Noam Raphael (noamr) Date: 2005-02-04 02:17 Message: Logged In: YES user_id=679426 Thanks for your comments. What do you think of the new patch? ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2005-02-03 00:17 Message: Logged In: YES user_id=149084 1. Please improve the doc string for interesting_lines(), it's no longer a generator. 2. You removed a comment from update_label() which was helpful when studying the code. What does it do now? 3. The variable 'l' should not be used because it's hard to discriminate from '1'. See PEP 8, Naming Conventions. I suggest 'ln' if it must be short, otherwise 'line' ---------------------------------------------------------------------- Comment By: Noam Raphael (noamr) Date: 2004-09-08 01:42 Message: Logged In: YES user_id=679426 I improved the algorithm for finding the context lines, so it now scans the minimum number of lines it needs in order to update the panel - it uses the information it already has in a better way. This makes the scrolling of long modules more "streaming", on my not-that-new computer. I tried it, and it seems to work just as good as the old algorithm - it produces the same results. ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2004-06-06 04:33 Message: Logged In: YES user_id=149084 Checked in Noam Raphael's improvements. CodeContext.py 1.4 et. al. ---------------------------------------------------------------------- Comment By: Noam Raphael (noamr) Date: 2004-04-28 00:24 Message: Logged In: YES user_id=679426 Hello, I think that showing / not showing the Code Context panel should be a persistent setting - if the user wants it, it will appear, and if he doesn't, it won't. The concept of "it has an initial state, and you can change it afterwards if you like" is confusing. Using a persistent setting makes a keyboard shortcut unnecessary - I think that usually the user will be happy with what he likes, and won't hide/show the panel many times. So I made the <> event save the new setting in the user's configuration file. The new setting will be applied also to new windows opened in the same session. This required me to add a SetOption method to configHandler.IdleConf. Also, the initialization of the CodeContext instance required the menu item to be already installed, but EditorWindow.load_extension installs the menu items after it initializes the extension's instance. It turns out that the menu items can be installed before the instance initialization, so I changed it. Another problem was that the Code Context menu item was installed on shell windows, but wasn't functioning, which was quite confusing. So I added new optional configurations for extensions: enable_shell and enable_editor, which allow you to write extensions only for the editor/shell window, without using the "isinstance(editwin, PyShell)" trick. About using a Text widget instead of a Label - it may work, I don't know, but making the Label behave as I wanted was hard enough, and it looks fine to me as it is, so I leave it for now. Bye Bye, Noam ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2004-04-27 01:28 Message: Logged In: YES user_id=149084 Yeah, I noticed that problem and wasn't sure if I had caused it. Thanks, fixed. CodeContext.py 1.3. Sorry I took so long getting back, for some reason I'm not getting mail from SF when a Tracker item changes (though I don't have a problem sending mail to myself via users.sourceforge.net). I added an entry for Code Context on the Options menu, it toggles the context pane. Any suggestion for a keybinding? A remaining problem is the text in the Label doesn't quite line up with the text in the main pane. Adding a pad of one space makes it almost perfect. But maybe using a Text widget instead of a Label would fix it 100%? ---------------------------------------------------------------------- Comment By: Noam Raphael (noamr) Date: 2004-04-22 23:11 Message: Logged In: YES user_id=679426 I'm glad you like it! Thanks for the changes - your English is certainly better than mine. A buglet you made - in line 91 you added "elif" as a block opener which should show the previous block opener of the same indentation, which is a good idea, but you wrote: if opener == "else" or "elif": which is an expression which always yields True. It caused all block openers to be displayed, not only the relevant ones. Change it to if opener in ("else", "elif"): and it will work fine. Happy Israel Independence Day! (I know you probably don't celebrate it, but it is on 27 April this year) Noam ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2004-04-21 23:09 Message: Logged In: YES user_id=149084 Cool Extension! Thanks! CodeContext.py 1.1 config-extensions.def 1.13 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=936169&group_id=5470 From noreply at sourceforge.net Sun Aug 21 20:50:52 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 21 Aug 2005 11:50:52 -0700 Subject: [Patches] [ python-Patches-1121611 ] sha and md5 modules should use OpenSSL when possible Message-ID: Patches item #1121611, was opened at 2005-02-12 17:33 Message generated for change (Comment added) made by greg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121611&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Gregory P. Smith (greg) Assigned to: Gregory P. Smith (greg) Summary: sha and md5 modules should use OpenSSL when possible Initial Comment: The md5 and sha (sha1) modules should use OpenSSL for the algorithms when it is available as its implementations are much faster than pythons own. Attached is an initial patch to use OpenSSL for the sha module. Its not ready for committing as is yet, but it is setup to be a generic base for all OpenSSL hashes with a little bit of work in the future. Tossing this out there for people to see how trivial it is and enjoy the speedups. diff is against HEAD but it should apply to 2.4 just fine. ---------------------------------------------------------------------- >Comment By: Gregory P. Smith (greg) Date: 2005-08-21 11:50 Message: Logged In: YES user_id=413 hashlib has been committed to HEAD for inclusion in python 2.5. I've attached a hashlib-010.patch that is the exact cvs diff of what i committed after further testing. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2005-08-15 01:47 Message: Logged In: YES user_id=4771 I see that it would indeed be messy to have 'md5' be a type and 'digest_size' a class attribute given that 'md5' can come from various places depending on what is installed; moreover in the hashopenssl.c file unless I'm mistaken all hashes use the same Python type. Fine by me. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-08-14 20:28 Message: Logged In: YES user_id=413 tjreedy and arigo's comments have been taken into consideration. An updated patch (009) has been attached. it uses the python >= 2.2 interface for defining methods and member variables rather than the getattr function with manual strcmp's. I was unable to make digest_size and such class attributes because the hashes are not classes. The hashlib.md5 function for instance is a constructor function that returns an appropriate internal HASH object. The goal of those constructors is to be as fast as possible; wrapping them with python in order to make them actual classes would be too slow and I did not see an obvious way to do it from C. I believe this patch is ready to commit. Further improvements or refinements can be made to it in CVS. the documentation in html for easy viewing has been updated at http://electricrain.com/greg/hashlib-py25-doc/ ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-07-31 18:29 Message: Logged In: YES user_id=413 per arigo's suggestion I have a version of _hashopenssl.c in my sandbox modified to use the more modern C type API. The constructor is slightly faster (~1-2%) and does seem like a better way to do things. i'll post it after polishing it up. ---------------------------------------------------------------------- Comment By: Terry J. Reedy (tjreedy) Date: 2005-06-12 13:35 Message: Logged In: YES user_id=593130 Re Doc page: As a somewhat naive (relative to the subject) reader, the title and first sentence implied that 'secure hash' and 'message digest' are two separate things, whereas, judging from the .digest() blurb, they both seem to be16-byte hashes. So I would prefer this equivalence and the actual meaning were made clear at the top. Something like "This module implements a common interface to several secure hash or message digest algorithms that produce 16-byte hashes." If, as I presume, xx.hexdigest() == binascii.hexlify(xx.digest()), then I would say so and reference binsacii for the interconversion functions one would need if one had the two versions to compare or needed to convert after the extraction. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2005-06-12 05:18 Message: Logged In: YES user_id=4771 On a side note, maybe it makes sense for a new module like this one to promote and use the modern (>=2.2) ways of defining C types. What I have in mind is using tp_methods instead of Py_FindMethod, and generally not reverting to strcmp(). In this case, the constants like 'digest_size' would be best stored as class attributes instead, if possible. Indeed, allowing expressions like "hashlib.md5.digest_size" conveys the idea that the result doesn't depend on a particular instance, unlike "hashlib.md5().digest_size". (Of course class attributes are also readable from the instance, as usual.) I can give it a try if you don't want to invest more time in this patch than you already did (for which we are grateful to you :-) ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-06-11 20:21 Message: Logged In: YES user_id=413 Ok, this patch is ready. documentation has been added. I'll bring it up on python-dev for discussion/approval with a link to the htmlified documentation. The speedups are great for any application hashing a lot of data when OpenSSL is used. It also adds a sha224, sha256, sha384 and sha512 support. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-12 17:13 Message: Logged In: YES user_id=413 I linked a _hashlib.so library statically against openssl and reran the speed test. no change. that means its not shared library overhead causing the higher startup time but just an artifact of the OpenSSL EVP interface. Next up, analyze what size things common heavy sha1 using applications regularly hash (BitTorrent and such). ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-10 00:09 Message: Logged In: YES user_id=413 The 007 patch improves the speed of the constructor. There is still a potential speed issue with the constructor/destructor to work on: greg at spiff src $ ./python Lib/test/test_hashlib_speed.py _sha testing speed of old _sha legacy interface 0.06 seconds [20000 creations] 0.24 seconds [20000 "" digests] 0.15 seconds 20 x 106201 bytes [huge data] 0.15 seconds 200 x 10620 bytes [large data] 0.17 seconds 2000 x 1062 bytes [medium data] 0.35 seconds 20020 x 106 bytes [small data] 1.37 seconds 106200 x 20 bytes [digest_size data] 2.75 seconds 212400 x 10 bytes [tiny data] greg at spiff src $ ./python Lib/test/test_hashlib_speed.py sha1 testing speed of hashlib.sha1 0.22 seconds [20000 creations] 0.57 seconds [20000 "" digests] 0.09 seconds 20 x 106201 bytes [huge data] 0.09 seconds 200 x 10620 bytes [large data] 0.15 seconds 2000 x 1062 bytes [medium data] 0.71 seconds 20020 x 106 bytes [small data] 3.39 seconds 106200 x 20 bytes [digest_size data] 6.70 seconds 212400 x 10 bytes [tiny data] I suspect the cause is either or both of the shared openssl library call overhead or the openssl EVP abstraction interface. The speed results are very similar to the above regardless of which digest is used (the above was a celeron 333mhz running linux). ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-03 13:15 Message: Logged In: YES user_id=413 hashlib-006.patch adds fast constructors and a speed test. documentation is the next step. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-01 01:14 Message: Logged In: YES user_id=413 hashlib-005.patch now passes its test suite and no problems appear in valgrind. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-02-28 10:11 Message: Logged In: YES user_id=413 a much updated patch (hashlib-patch-004.patch). it incorporates some suggestions as well as including sf patch 935454's sha256/224 and sha512/384 implementations. still not complete but shows the direction its going in (i see a segfault part way thru the test suite after running the sha512 tests). as for the private modules being under another package, i see no reason to do that since there aren't very many (how does that work for binary modules anyways?). ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2005-02-18 11:21 Message: Logged In: YES user_id=764593 Should the private modules (such as _sha) be placed in a crypto package, instead of directly in the parent/everything library? ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-02-16 22:46 Message: Logged In: YES user_id=413 hashes-openssl-002.patch replaces the sha and md5 modules with a general hashes module that wraps all hashes that OpenSSL supports. note that OpenSSLs implementations are much faster than the previous python versions as it choses versions optimized for your particular hardware. Incase python is compiled without openssl the hashes wrapper falls back on the old python sha and md5 module implementations. side note: This may be sufficient for the Debian folks to work around their random odd licensing issue. just have debian python depend on openssl; use this and remove the old md5 module/code that wouldn't get used anyways. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121611&group_id=5470 From noreply at sourceforge.net Sun Aug 21 20:51:58 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 21 Aug 2005 11:51:58 -0700 Subject: [Patches] [ python-Patches-1121611 ] sha and md5 modules should use OpenSSL when possible Message-ID: Patches item #1121611, was opened at 2005-02-12 17:33 Message generated for change (Settings changed) made by greg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121611&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: None >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: Gregory P. Smith (greg) Assigned to: Gregory P. Smith (greg) Summary: sha and md5 modules should use OpenSSL when possible Initial Comment: The md5 and sha (sha1) modules should use OpenSSL for the algorithms when it is available as its implementations are much faster than pythons own. Attached is an initial patch to use OpenSSL for the sha module. Its not ready for committing as is yet, but it is setup to be a generic base for all OpenSSL hashes with a little bit of work in the future. Tossing this out there for people to see how trivial it is and enjoy the speedups. diff is against HEAD but it should apply to 2.4 just fine. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-08-21 11:50 Message: Logged In: YES user_id=413 hashlib has been committed to HEAD for inclusion in python 2.5. I've attached a hashlib-010.patch that is the exact cvs diff of what i committed after further testing. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2005-08-15 01:47 Message: Logged In: YES user_id=4771 I see that it would indeed be messy to have 'md5' be a type and 'digest_size' a class attribute given that 'md5' can come from various places depending on what is installed; moreover in the hashopenssl.c file unless I'm mistaken all hashes use the same Python type. Fine by me. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-08-14 20:28 Message: Logged In: YES user_id=413 tjreedy and arigo's comments have been taken into consideration. An updated patch (009) has been attached. it uses the python >= 2.2 interface for defining methods and member variables rather than the getattr function with manual strcmp's. I was unable to make digest_size and such class attributes because the hashes are not classes. The hashlib.md5 function for instance is a constructor function that returns an appropriate internal HASH object. The goal of those constructors is to be as fast as possible; wrapping them with python in order to make them actual classes would be too slow and I did not see an obvious way to do it from C. I believe this patch is ready to commit. Further improvements or refinements can be made to it in CVS. the documentation in html for easy viewing has been updated at http://electricrain.com/greg/hashlib-py25-doc/ ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-07-31 18:29 Message: Logged In: YES user_id=413 per arigo's suggestion I have a version of _hashopenssl.c in my sandbox modified to use the more modern C type API. The constructor is slightly faster (~1-2%) and does seem like a better way to do things. i'll post it after polishing it up. ---------------------------------------------------------------------- Comment By: Terry J. Reedy (tjreedy) Date: 2005-06-12 13:35 Message: Logged In: YES user_id=593130 Re Doc page: As a somewhat naive (relative to the subject) reader, the title and first sentence implied that 'secure hash' and 'message digest' are two separate things, whereas, judging from the .digest() blurb, they both seem to be16-byte hashes. So I would prefer this equivalence and the actual meaning were made clear at the top. Something like "This module implements a common interface to several secure hash or message digest algorithms that produce 16-byte hashes." If, as I presume, xx.hexdigest() == binascii.hexlify(xx.digest()), then I would say so and reference binsacii for the interconversion functions one would need if one had the two versions to compare or needed to convert after the extraction. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2005-06-12 05:18 Message: Logged In: YES user_id=4771 On a side note, maybe it makes sense for a new module like this one to promote and use the modern (>=2.2) ways of defining C types. What I have in mind is using tp_methods instead of Py_FindMethod, and generally not reverting to strcmp(). In this case, the constants like 'digest_size' would be best stored as class attributes instead, if possible. Indeed, allowing expressions like "hashlib.md5.digest_size" conveys the idea that the result doesn't depend on a particular instance, unlike "hashlib.md5().digest_size". (Of course class attributes are also readable from the instance, as usual.) I can give it a try if you don't want to invest more time in this patch than you already did (for which we are grateful to you :-) ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-06-11 20:21 Message: Logged In: YES user_id=413 Ok, this patch is ready. documentation has been added. I'll bring it up on python-dev for discussion/approval with a link to the htmlified documentation. The speedups are great for any application hashing a lot of data when OpenSSL is used. It also adds a sha224, sha256, sha384 and sha512 support. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-12 17:13 Message: Logged In: YES user_id=413 I linked a _hashlib.so library statically against openssl and reran the speed test. no change. that means its not shared library overhead causing the higher startup time but just an artifact of the OpenSSL EVP interface. Next up, analyze what size things common heavy sha1 using applications regularly hash (BitTorrent and such). ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-10 00:09 Message: Logged In: YES user_id=413 The 007 patch improves the speed of the constructor. There is still a potential speed issue with the constructor/destructor to work on: greg at spiff src $ ./python Lib/test/test_hashlib_speed.py _sha testing speed of old _sha legacy interface 0.06 seconds [20000 creations] 0.24 seconds [20000 "" digests] 0.15 seconds 20 x 106201 bytes [huge data] 0.15 seconds 200 x 10620 bytes [large data] 0.17 seconds 2000 x 1062 bytes [medium data] 0.35 seconds 20020 x 106 bytes [small data] 1.37 seconds 106200 x 20 bytes [digest_size data] 2.75 seconds 212400 x 10 bytes [tiny data] greg at spiff src $ ./python Lib/test/test_hashlib_speed.py sha1 testing speed of hashlib.sha1 0.22 seconds [20000 creations] 0.57 seconds [20000 "" digests] 0.09 seconds 20 x 106201 bytes [huge data] 0.09 seconds 200 x 10620 bytes [large data] 0.15 seconds 2000 x 1062 bytes [medium data] 0.71 seconds 20020 x 106 bytes [small data] 3.39 seconds 106200 x 20 bytes [digest_size data] 6.70 seconds 212400 x 10 bytes [tiny data] I suspect the cause is either or both of the shared openssl library call overhead or the openssl EVP abstraction interface. The speed results are very similar to the above regardless of which digest is used (the above was a celeron 333mhz running linux). ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-03 13:15 Message: Logged In: YES user_id=413 hashlib-006.patch adds fast constructors and a speed test. documentation is the next step. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-03-01 01:14 Message: Logged In: YES user_id=413 hashlib-005.patch now passes its test suite and no problems appear in valgrind. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-02-28 10:11 Message: Logged In: YES user_id=413 a much updated patch (hashlib-patch-004.patch). it incorporates some suggestions as well as including sf patch 935454's sha256/224 and sha512/384 implementations. still not complete but shows the direction its going in (i see a segfault part way thru the test suite after running the sha512 tests). as for the private modules being under another package, i see no reason to do that since there aren't very many (how does that work for binary modules anyways?). ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2005-02-18 11:21 Message: Logged In: YES user_id=764593 Should the private modules (such as _sha) be placed in a crypto package, instead of directly in the parent/everything library? ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-02-16 22:46 Message: Logged In: YES user_id=413 hashes-openssl-002.patch replaces the sha and md5 modules with a general hashes module that wraps all hashes that OpenSSL supports. note that OpenSSLs implementations are much faster than the previous python versions as it choses versions optimized for your particular hardware. Incase python is compiled without openssl the hashes wrapper falls back on the old python sha and md5 module implementations. side note: This may be sufficient for the Debian folks to work around their random odd licensing issue. just have debian python depend on openssl; use this and remove the old md5 module/code that wouldn't get used anyways. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121611&group_id=5470 From noreply at sourceforge.net Sun Aug 21 20:54:27 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 21 Aug 2005 11:54:27 -0700 Subject: [Patches] [ python-Patches-935454 ] sha256 module Message-ID: Patches item #935454, was opened at 2004-04-14 23:57 Message generated for change (Comment added) made by greg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=935454&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.4 >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: Trevor Perrin (trevp) Assigned to: Gregory P. Smith (greg) Summary: sha256 module Initial Comment: This module is a copy of shamodule.c, with the SHA-1 compression function replaced with the SHA-256 compression function (copied from the LibTomCrypt public-domain crypto library). SHA-256 is similar to SHA-1: it's a US Federal Standard hash algorithm (FIPS 180-2). The difference is that it produces a 256 bit hash value, instead of a 160 bit hash value. SHA-256 thus has 128 bits of resistance against birthday attacks, which makes it secure in certain protocols where SHA-1 is questionable (e.g. digital signatures; or RNGs or Key-Derivation Functions where you want to produce keys for 256-bit ciphers). There's other flavors of SHA, but they're not as useful: SHA-384 and SHA-512 are defined on 64-bit values, so are slow on 32-bit architectures. SHA-224 is just silly (it saves 32 bits over SHA-256; that's its sole rationale). ---------------------------------------------------------------------- >Comment By: Gregory P. Smith (greg) Date: 2005-08-21 11:54 Message: Logged In: YES user_id=413 sha224, 256, 384 and 512 support derived from this code for use when the available openssl library does not support them has been committed to python cvs HEAD as part of SF patch 1121611. thanks! ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2005-02-21 08:05 Message: Logged In: YES user_id=413 i'll take care of this patch along with patch 1121611 using these sha 256/512 implementations when openssl doesn't include its own implementations (current common versions of openssl don't, future versions will). ---------------------------------------------------------------------- Comment By: Trevor Perrin (trevp) Date: 2004-07-14 00:04 Message: Logged In: YES user_id=973611 New set of files - full source, patches, and docs for all new SHA versions. Cleans things up a bit: - sha256.c, sha512.c - sha224.py, sha384.py (wrappers) - test_sha*.py - libsha*.tex ---------------------------------------------------------------------- Comment By: Trevor Perrin (trevp) Date: 2004-07-13 01:10 Message: Logged In: YES user_id=973611 2nd version of documentation for sha256 and sha512 ---------------------------------------------------------------------- Comment By: Trevor Perrin (trevp) Date: 2004-07-02 01:00 Message: Logged In: YES user_id=973611 Uploading a new sha256 module with a sha224 function, and a sha512 module with a sha384 function. ---------------------------------------------------------------------- Comment By: Trevor Perrin (trevp) Date: 2004-06-30 23:48 Message: Logged In: YES user_id=973611 I'm uploading an attempt at documentation for this module. I had trouble installing the tex tools so I'm not sure if it's correct, but it should be easy to fix if not. I'll try to straighten myself out with the tools so I can make sure. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-06-29 07:23 Message: Logged In: YES user_id=11375 I can't rule on whether the module should be added or not, so I'll bring it up on python-dev. Feel free to join the resulting thread. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=935454&group_id=5470 From noreply at sourceforge.net Mon Aug 22 22:54:30 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 22 Aug 2005 13:54:30 -0700 Subject: [Patches] [ python-Patches-1266570 ] PEP 349: allow str() to return unicode Message-ID: Patches item #1266570, was opened at 2005-08-22 20:54 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1266570&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Neil Schemenauer (nascheme) Assigned to: Nobody/Anonymous (nobody) Summary: PEP 349: allow str() to return unicode Initial Comment: The patch implements PEP 349: "Allow str() to return unicode strings". The backwards compatibility problems appear to be rare. Hopefully more people can test their applications. I'm not very happy with the new function named PyString_New since it really belongs in Python/object.c, not stringobject.c. However, I can't think of a better name. I think it should be made available to extension modules somehow so _PyObject_Str is no good. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1266570&group_id=5470 From noreply at sourceforge.net Mon Aug 22 22:55:26 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 22 Aug 2005 13:55:26 -0700 Subject: [Patches] [ python-Patches-1266570 ] PEP 349: allow str() to return unicode Message-ID: Patches item #1266570, was opened at 2005-08-22 20:54 Message generated for change (Comment added) made by nascheme You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1266570&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Neil Schemenauer (nascheme) Assigned to: Nobody/Anonymous (nobody) Summary: PEP 349: allow str() to return unicode Initial Comment: The patch implements PEP 349: "Allow str() to return unicode strings". The backwards compatibility problems appear to be rare. Hopefully more people can test their applications. I'm not very happy with the new function named PyString_New since it really belongs in Python/object.c, not stringobject.c. However, I can't think of a better name. I think it should be made available to extension modules somehow so _PyObject_Str is no good. ---------------------------------------------------------------------- >Comment By: Neil Schemenauer (nascheme) Date: 2005-08-22 20:55 Message: Logged In: YES user_id=35752 Attaching patch (forgot to click on stupid checkbox). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1266570&group_id=5470 From noreply at sourceforge.net Mon Aug 22 22:57:36 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 22 Aug 2005 13:57:36 -0700 Subject: [Patches] [ python-Patches-1159501 ] Improve %s support for unicode Message-ID: Patches item #1159501, was opened at 2005-03-09 01:43 Message generated for change (Comment added) made by nascheme You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1159501&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None >Status: Closed Resolution: None Priority: 5 Submitted By: Neil Schemenauer (nascheme) Assigned to: Fredrik Lundh (effbot) Summary: Improve %s support for unicode Initial Comment: "'%s' % unicode_string" produces a unicode result. I think the following code should also return a unicode string: class Wrapper: ....def __str__(self): ........return unicode_string '%s' % Wrapper() That behavior would make it easier to write library code that can work with either str objects or unicode objects. The fix is pretty simple (see that attached patch). Perhaps the PyObject_Text function should be called _PyObject_Text instead. Alternatively, if the function is make public then we should document it and perhaps also provide a builtin function called 'text' that uses it. ---------------------------------------------------------------------- >Comment By: Neil Schemenauer (nascheme) Date: 2005-08-22 20:57 Message: Logged In: YES user_id=35752 Closing in favor of patch 1266570. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2005-04-20 21:46 Message: Logged In: YES user_id=35752 Here's a quote from him: > I'm beginning to think that we need an extra method (__text__), that > can return any kind of string that's compatible with Python's text model. > > (in today's CPython, that's an 8-bit string with ASCII only, or a Uni- > code string. future Python's may support more string types, at least at > the C implementation level). > > I'm not sure we can change __str__ or __unicode__ without breaking > code in really obscure ways (but I'd be happy to be proven wrong). My idea is that we can change __str__ without breaking code. The reason is that no one should be calling tp_str directly. Instead they use PyObject_Str. I don't know what he meant by "string that's compatible with Python's text model". With my change, Python can only deal with str or unicode instances. I have no idea how we could support other string implementations. I don't want to introduce a text() builtin that calls __str__ and then later realize that __text__ would be a useful. Perhaps this change is big enough to require a PEP. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2005-04-20 21:27 Message: Logged In: YES user_id=38388 Looks OK to me; not sure what you mean with __text__ - __str__ already has taken that role long ago. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2005-04-20 21:00 Message: Logged In: YES user_id=35752 Assigning to effbot for review. He had mentioned something about __text__ at one point. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2005-03-10 21:13 Message: Logged In: YES user_id=35752 attempt to attach patch again ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2005-03-10 21:12 Message: Logged In: YES user_id=35752 Attaching a better patch. Add a builtin function called "text". Change PyObject_Text to check the return types as suggested by Mark. Update the documentation and the tests. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2005-03-09 10:10 Message: Logged In: YES user_id=38388 Nice patch. Only nit: PyObject_Text() should check that the result of tp_str() is indeed either a string or unicode instance (possibly from a subclass). Otherwise, the function wouldn't be able to guarantee this feature - which is what it's all about. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1159501&group_id=5470 From noreply at sourceforge.net Tue Aug 23 05:59:28 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 22 Aug 2005 20:59:28 -0700 Subject: [Patches] [ python-Patches-1173998 ] Python crashes in pyexpat.c if malformed XML is parsed Message-ID: Patches item #1173998, was opened at 2005-03-31 05:59 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1173998&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.3 Status: Open Resolution: Works For Me Priority: 7 Submitted By: pdecat (pdecat) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: Python crashes in pyexpat.c if malformed XML is parsed Initial Comment: If a malformed XML file (containing non unicode characters) is parsed with pyexpat, python crashes. Most details on request. ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2005-08-22 23:59 Message: Logged In: YES user_id=3066 I realize this has sat too long; sorry. Can you send an example XML file for which this crashes for you? Do you let Expat determine and handle the encoding itself, or do you override the detected encoding when you create the parser? ---------------------------------------------------------------------- Comment By: pdecat (pdecat) Date: 2005-03-31 08:00 Message: Logged In: YES user_id=1210681 STRING_CONV_FUNC returns NULL if the string is contains non-ascii and non-unicode characters. ---------------------------------------------------------------------- Comment By: pdecat (pdecat) Date: 2005-03-31 07:18 Message: Logged In: YES user_id=1210681 Maybe security related, as it can lead to denial of service: it crashes a Zope server using the ParsedXML product simply by uploading a malformed XML file. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1173998&group_id=5470 From noreply at sourceforge.net Tue Aug 23 06:09:37 2005 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 22 Aug 2005 21:09:37 -0700 Subject: [Patches] [ python-Patches-901369 ] markupbase misses comments (bug 736659) Message-ID: Patches item #901369, was opened at 2004-02-20 15:58 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=901369&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None >Status: Closed >Resolution: Rejected Priority: 5 Submitted By: Jim Jewett (jimjjewett) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: markupbase misses comments (bug 736659) Initial Comment: markupbase.ParserBase().parse_declaration calls parse_comment if the text starts with "