From noreply at sourceforge.net Sat May 1 19:59:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 1 19:59:46 2004 Subject: [Patches] [ python-Patches-946153 ] askstring => grab fail Message-ID: Patches item #946153, was opened at 2004-05-01 16:59 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=946153&group_id=5470 Category: Tkinter Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: wes (wweston) Assigned to: Martin v. L?wis (loewis) Summary: askstring => grab fail Initial Comment: Calling askstring results in a "grab fail" exception about 2/3rds of the time on redhat 9/Python 2.3.3. By inserting one line in: /usr/local/Python/lib/python2.3/lib-tk/Tkinter.py //line 521 the problem seems to go away. def grab_set(self): """Set grab for this widget. A grab directs all events to this and descendant widgets in the application.""" self.wait_visibility()############added##################### self.tk.call('grab', 'set', self._w) wes ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=946153&group_id=5470 From noreply at sourceforge.net Sat May 1 22:20:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 1 22:20:41 2004 Subject: [Patches] [ python-Patches-934711 ] platform-specific entropy Message-ID: Patches item #934711, was opened at 2004-04-14 00:05 Message generated for change (Comment added) made by nickm You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=934711&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Trevor Perrin (trevp) Assigned to: Nobody/Anonymous (nobody) Summary: platform-specific entropy Initial Comment: This is a very simple module for accessing platform- specific entropy sources. On Win32, it uses CryptGenRandom. Otherwise, it tries to use /dev/urandom. If it can't find that, it raises NotImplementedError. >>> import entropy >>> entropy.entropy(10) '\x99/\xbd\x8e\xb0\xfbz/\xf6\xe9' To compile for Win32, you have to link with "advapi32". The /dev/urandom code has not been tested. Issues: - I believe this works with all versions of Windows later than Win95. But I'm not sure. - there's overhead in opening/closing the source. On Win32 at least, opening it is slower than reading from it - it seems to take around 1/5th of a millisecond (i.e. I can make 5000 calls per second). I think this is okay; you can make fewer calls and buffer the results, or seed your own PRNG. However, we could make it more object-based, where you open an entropy-source, and then repeatedly read from it. - It would be nice if there was some attribute that told you what source you were using, so you could display it, and in case you trust /dev/urandom but not Win32's CryptoAPI, for example. ---------------------------------------------------------------------- Comment By: Nick Mathewson (nickm) Date: 2004-05-01 22:20 Message: Logged In: YES user_id=499 This version looks exactly like what I need. I really hope it gets accepted for 2.4. (I have no strong opinion on whether the fd/crypto context should be held open.) ---------------------------------------------------------------------- Comment By: Trevor Perrin (trevp) Date: 2004-04-27 03:21 Message: Logged In: YES user_id=973611 I'm uploading a 4th version that does this as an "os.urandom()" function. The win32 code is in posixmodule.c, and the /dev/urandom code is in os.py. I think the name "urandom" is better than "entropy" - to some people, "entropy" sounds more like /dev/random. "urandom" makes it clear that we're trying to provide /dev/urandom-like functionality across platforms (i.e.: try to use real entropy, but if you have to stretch it with a cryptographic PRNG that's okay). Also, I cleaned up error-handling, so it will always return OSError (or an OSError subclass, like WindowsError). Finally I changed it to open the random source once, instead of re-opening it every call. At least on windows this is a big speedup - from being able to make 5K calls per second it can now make 250K. It's probable that people will write crypto code using this as their sole RNG source, so this is probably worthwhile. This introduces a small risk of leaking file descriptors/handles if the os module is reloaded, but I think that's okay. ---------------------------------------------------------------------- Comment By: Nick Mathewson (nickm) Date: 2004-04-26 13:11 Message: Logged In: YES user_id=499 Your lastest version works fine for me (on Redhat Linux 9), but I still think it would be a better idea to write the /dev/urandom-ish code in Python, using os.open. That way, you could report better error messages if /dev/urandom is inaccessable, instead of simply claiming that it "wasn't found." (Maybe it was found, but we don't have permission to access it.) ---------------------------------------------------------------------- Comment By: Trevor Perrin (trevp) Date: 2004-04-25 23:53 Message: Logged In: YES user_id=973611 > - I can't find a statement on the page you link about using > CRYPT_VERIFYCONTEXT that way, Note that retrying on NTE_BAD_KEYSET is only described under the heading "Private Key Operations Are Performed", in which case you need to open/create a private key container. But if you use CRYPT_VERIFYCONTEXT it just creates a temporary context in memory. More corroboration: http://tinyurl.com/2ct2o For awhile I was distributing code without CRYPT_VERIFYCONTEXT, and a user ran into the NTE_BAD_KEYSET error. But CRYPT_VERIFYCONTEXT fixed it, which is how I stumbled on this... > - One more important issue: It is a bad idea to use stdio > (C's 'fopen', Python's builtin 'open') to read from > /dev/urandom. Good point. I've tried to update the code to use syscalls. Is there any chance you could test this out, and see whether the #includes look correct and portable? I don't have a UNIX box available. If it needs fixes, feel free to upload a new version. ---------------------------------------------------------------------- Comment By: Nick Mathewson (nickm) Date: 2004-04-25 20:49 Message: Logged In: YES user_id=499 Thanks for the reply! - As for /dev/*random -- yes, I believe you are right, and /dev/urandom is almost always what you want. I haven't been able to find a platform that has one of the others, but lacks /dev/urandom. - I can't find a statement on the page you link about using CRYPT_VERIFYCONTEXT that way, but you may well be right anway. - One more important issue: It is a bad idea to use stdio (C's 'fopen', Python's builtin 'open') to read from /dev/urandom. Most stdio implementation buffer data; on my GNU/Linux box, when I call open('/dev/urandom').read(10), my underlying fread() function sucks 4096 bytes into memory. (It does other weird stuff too, including calls to stat64, mmap, and others.) This has proved to be a problem in the past, especially when running on systems with heavy user process limits. Instead, it is a better idea to use the open syscall directly (open in C, os.open in Python). ---------------------------------------------------------------------- Comment By: Trevor Perrin (trevp) Date: 2004-04-25 20:08 Message: Logged In: YES user_id=973611 Thanks for the comments! - > - According to MSDN, CryptGenRandom exists on Win98 and > later, and on Win95 OSR2, and on Win95 with IE 3.something > or later. I'm uploading a new version that fails gracefully on old Win95s (that's the only change). > - It's necessary on some platforms, and for some > applications, to use a device other than /dev/urandom. > (Some high-security code demands /dev/random; some > OpenBSD people swear by /dev/arandom; and so on.) My understanding is that /dev/random should only be used in exceptionally rare cases, if at all. You can turn up several posts by David Wagner, a respected cryptographer, about this. For example: http://tinyurl.com/2z2fx In any case, if you really want /dev/random, or one of the OpenBSD variants (arandom, srandom, prandom, etc.), it's easy to do it yourself: open("/dev/random").read(). So I think we should ignore these and stick with /dev/urandom, since it's the easiest-to-use (non-blocking) and most portable (unless there are systems that don't offer it?). > - Maybe it would be a good idea to only implement the > windows CryptGenRandom part in C, and implement the Unix > part in Python. That's not a bad idea - I sorta think this function should be placed in the 'os' module, instead of its own module. So we could put the /dev/urandom code in 'os.py', and allow more specific code in, e.g., posixmodule.c to override it. We could also add a variable 'os.entropySource' which would return '/dev/urandom', or 'CryptoAPI', or whatever. > - According to the MSDN documentation for > CryptAcquireContext, if your first call fails, you're > supposed to retry with a final argument of > CRYPT_NEWKEYSET before you report an error. I'm pretty sure using CRYPT_VERIFYCONTEXT eliminates the need for that: http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q238187&ID=KB;EN-US;Q238187 ---------------------------------------------------------------------- Comment By: Nick Mathewson (nickm) Date: 2004-04-25 14:55 Message: Logged In: YES user_id=499 This patch would be tremendously valuable to me. I've had to maintain similar code for a couple of projects, and having this functionality in Python would make my life one step similar. A few comments: - According to MSDN, CryptGenRandom exists on Win98 and later, and on Win95 OSR2, and on Win95 with IE 3.something or later. - It's necessary on some platforms, and for some applications, to use a device other than /dev/urandom. (Some high-security code demands /dev/random; some OpenBSD people swear by /dev/arandom; and so on.) - Maybe it would be a good idea to only implement the windows CryptGenRandom part in C, and implement the Unix part in Python. Then you could expose the windows code as (say) "entopy.win_entropy(nBytes)", the unix part as (say) "entropy.unix_entropy(nBytes, file='/dev/urandom')", and have "entropy.entropy(nBytes)" be a cross-platform wrapper. This would cut down on the amount of C you need to add; make it easier to switch entropy devices; provide better errors when /dev/urandom is unreadable; and provide users the option to trust only certain entropy sources. - I believe that you're right not to worry too much about the performance here. - According to the MSDN documentation for CryptAcquireContext, if your first call fails, you're supposed to retry with a final argument of CRYPT_NEWKEYSET before you report an error. I don't know when/if this is necessary, or whether there are hidden gotchas. Once again, this patch is a great idea, and I heartily hope that it gets in! ---------------------------------------------------------------------- Comment By: Trevor Perrin (trevp) Date: 2004-04-14 02:19 Message: Logged In: YES user_id=973611 Just a thought - this might make sense as a function within the 'os' module, instead of its own module. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=934711&group_id=5470 From noreply at sourceforge.net Sat May 1 22:45:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 1 22:45:26 2004 Subject: [Patches] [ python-Patches-946207 ] Non-blocking Socket Server Message-ID: Patches item #946207, was opened at 2004-05-01 19:45 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=946207&group_id=5470 Category: Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Jesper ?JJ? Jurcenoks (jesperjurcenoks) Assigned to: Nobody/Anonymous (nobody) Summary: Non-blocking Socket Server Initial Comment: Changed the included Python 2.3.3 SocketServer.py to be non-blocking. I have made a web-server on BaseHTTPServer, but the server could get shot down externally by sending malformed request. Making it very Easy to Perform a Denial of Service. The problem also exists in "Module Docs" web-server To experience problem run Module Docs, and verify that web-server is working telnet to webserver (telnet 127.0.0.1 7464#) type "GET /\r" access any web-page on web-server and notice how it is hung. Replace existing SocketServer.py with included SocketServer.py and re-test noticing that the problem is gone. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=946207&group_id=5470 From noreply at sourceforge.net Sat May 1 22:46:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 1 22:46:27 2004 Subject: [Patches] [ python-Patches-945642 ] nonblocking i/o with ssl socket not working at all Message-ID: Patches item #945642, was opened at 2004-04-30 15:14 Message generated for change (Comment added) made by jesperjurcenoks You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Tino Lange (tinolange) Assigned to: Nobody/Anonymous (nobody) Summary: nonblocking i/o with ssl socket not working at all Initial Comment: Hi! Unfortunately the applied patches #675750 and #676472 (End of January by G. Talvola / B. Laurie / G. v. Rossum) for timeout-sockets in version 1.09 and 1.10 to Modules/_ssl.c broke the nonblocking SSL I/O. In fact sslobj.read() became *always* blocking - whether the underlying socket is blocking or nonblocking doesn't matter at all. In Python < 2.3 this has worked correctly, starting with 2.3, 2.3.x up to the current CVS it doesn't work anymore. Attached is a sample: 0) [ Preparation ] Please run stunnel on your machine to have a SSL service: /usr/sbin/stunnel -p /path/to/some/cert -f -d 4711 -r 4712 1) [ Server ] Please run the easy server.py 2) [Client showing the bug] Please run the client - and see how it hangs when you use Python2.3. Also attached is a diff against the current _ssl.c which shows how to correct that behaviour while also having the possibilities from the above timeout patches. I carefully tested it, please review and test yourself and finally check-in. If you have questions, please don't hesitate to ask me. I really would like to see that not only in 2.4, but also (if possible together with patch #909007 from S. Nicolary) in a 2.3.4 release. Otherwise 2.3.x wasn't at all able to handle nonblocking SSL connections properly. Thanks and cheers, Tino ---------------------------------------------------------------------- Comment By: Jesper ?JJ? Jurcenoks (jesperjurcenoks) Date: 2004-05-01 19:46 Message: Logged In: YES user_id=1033524 Hi Tino. I just submitted a nonblocking SocketServer.py, it might just solve your problem. JJ ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 From noreply at sourceforge.net Sun May 2 04:40:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 2 04:41:02 2004 Subject: [Patches] [ python-Patches-945642 ] nonblocking i/o with ssl socket not working at all Message-ID: Patches item #945642, was opened at 2004-05-01 00:14 Message generated for change (Comment added) made by tinolange You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Tino Lange (tinolange) Assigned to: Nobody/Anonymous (nobody) Summary: nonblocking i/o with ssl socket not working at all Initial Comment: Hi! Unfortunately the applied patches #675750 and #676472 (End of January by G. Talvola / B. Laurie / G. v. Rossum) for timeout-sockets in version 1.09 and 1.10 to Modules/_ssl.c broke the nonblocking SSL I/O. In fact sslobj.read() became *always* blocking - whether the underlying socket is blocking or nonblocking doesn't matter at all. In Python < 2.3 this has worked correctly, starting with 2.3, 2.3.x up to the current CVS it doesn't work anymore. Attached is a sample: 0) [ Preparation ] Please run stunnel on your machine to have a SSL service: /usr/sbin/stunnel -p /path/to/some/cert -f -d 4711 -r 4712 1) [ Server ] Please run the easy server.py 2) [Client showing the bug] Please run the client - and see how it hangs when you use Python2.3. Also attached is a diff against the current _ssl.c which shows how to correct that behaviour while also having the possibilities from the above timeout patches. I carefully tested it, please review and test yourself and finally check-in. If you have questions, please don't hesitate to ask me. I really would like to see that not only in 2.4, but also (if possible together with patch #909007 from S. Nicolary) in a 2.3.4 release. Otherwise 2.3.x wasn't at all able to handle nonblocking SSL connections properly. Thanks and cheers, Tino ---------------------------------------------------------------------- >Comment By: Tino Lange (tinolange) Date: 2004-05-02 10:40 Message: Logged In: YES user_id=212920 Hi JJ, I just scanned your SocketServer.py - and I really don't see how this class might help? You don't offer a better sslob.read() there, don't you? So how could it help (besides maybe making my example "server.py nicer). If I put a sslobj on top of a socket object it is blocking in 2.3, 2.3.x and higher. As proven in my example: Making the underlying socket nonblocking has no effect - because SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE from the SSL_xxx() functions in the C-Library do not break out of the while loop that was introduced in the above mentioned patches. Cheers, Tino ---------------------------------------------------------------------- Comment By: Jesper ?JJ? Jurcenoks (jesperjurcenoks) Date: 2004-05-02 04:46 Message: Logged In: YES user_id=1033524 Hi Tino. I just submitted a nonblocking SocketServer.py, it might just solve your problem. JJ ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 From noreply at sourceforge.net Sun May 2 08:10:13 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 2 08:10:17 2004 Subject: [Patches] [ python-Patches-944110 ] urllib2 authentication mishandles empty pass bugfix 944082 Message-ID: Patches item #944110, was opened at 2004-04-29 02:52 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=944110&group_id=5470 Category: Library (Lib) Group: None Status: Open Resolution: None >Priority: 7 Submitted By: sc0rp (yangir) >Assigned to: Martin v. L?wis (loewis) Summary: urllib2 authentication mishandles empty pass bugfix 944082 Initial Comment: If example.org requires authentication, then following code: host = 'example.org' user = 'testuser' password = '' url = 'http://%s/' % host authInfo = urllib2.HTTPPasswordMgrWithDefaultRealm() authInfo.add_password( None, host, user, password ) authHandler = urllib2.HTTPBasicAuthHandler( authInfo ) opener = urllib2.build_opener( authHandler ) urlFile = opener.open( url ) print urlFile.read() will die by throwing HTTPError 401: File "/usr/lib/python2.3/urllib2.py", line 419, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 401: Authorization Required even if authenticating with 'testuser' and empty password is valid. Empty password is mishandled (i.e. authentication with empty password string is ignored) in AbstractBasicAuthHandler.retry_http_basic_auth def retry_http_basic_auth(self, host, req, realm): user,pw = self.passwd.find_user_password(realm, host) if pw: [...] It can be fixed by changing: if pw: to if pw is not None: Patch attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=944110&group_id=5470 From noreply at sourceforge.net Sun May 2 08:51:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 2 08:51:21 2004 Subject: [Patches] [ python-Patches-946373 ] do not add directory of sys.argv[0] into sys.path Message-ID: Patches item #946373, was opened at 2004-05-02 14:51 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=946373&group_id=5470 Category: Core (C code) Group: None Status: Open Resolution: None Priority: 5 Submitted By: wrobell (wrobell) Assigned to: Nobody/Anonymous (nobody) Summary: do not add directory of sys.argv[0] into sys.path Initial Comment: Python adds magically directory of sys.argv[0] into sys.path, i.e. >>> import sys >>> sys.path ['', '/usr/lib/python23.zip', '/usr/share/python2.3', '/usr/share/python2.3/plat-linux2', '/usr/share/python2.3/lib-tk', '/usr/lib/python2.3/lib-dynload', '/usr/lib/python2.3/site-packages', '/usr/lib/python2.3/site-packages/gtk-2.0', '/usr/share/python2.3/site-packages'] where '' (or /usr/bin when executed script is in /usr/bin directory, etc.) is added automatically. It is useful in many circumstances but fails when name conflict occurs. For example, create getpass.py or pdb.py scripts which import getpass and pdb modules. Script names conflict with modules names and modules are not going to be imported because path to the scripts is appended into sys.path, so a script is imported instead of a module. The solutions: 1. User of script with conflicting name (i.e. getpass.py or timeit.py) can set PYTHONPATH to system library path, i.e. /usr/lib/python2.3. 2. User can modify the script to delete site.path[0]. 3. User can rename the script. 4. Python can be modified to not add magically directory of sys.argv[0]. The 1. is a tricky and not intuitive and quite funny: set PYTHONPATH to system library path to import system module (and only in specific circumstances). ;-P The 2. is a dirty hack: hey, we gonna import system module, ain't it? The 3. is, IMHO, not acceptable because there is more than 200 python system modules, more in the future and user cannot be forced to maintain script names blacklist. The 4. is only acceptable, IMHO. It makes python more inconvenient but gives no trouble when names conflict occurs. Moreover, fourth solution makes python more standard with other languages behaviour, i.e. one has to set CLASSPATH to load Java classes. Maybe there is another solution, but... Patch attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=946373&group_id=5470 From blush at you-plonker.com Mon May 3 00:04:24 2004 From: blush at you-plonker.com (Bate G. Biretta) Date: Mon May 3 07:48:07 2004 Subject: [Patches] Dont spend more on software, Patches. Message-ID: <100101c430c3$92ac3b42$27477fa2@you-plonker.com> Thinking of driving in that condition, sir? Allow motion to equal emotion. Low rates on Software Looking for not expensive high-quality software? Our site might be just what you need. http://inequilobate.sales-outlets.net And more! We ship worldwide. We know that the nature of genius is to provide idiots with ideas twenty years later. Productivity is being able to do things that you were never able to do before. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040502/63b2d876/attachment.html From noreply at sourceforge.net Mon May 3 10:40:26 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 3 10:40:38 2004 Subject: [Patches] [ python-Patches-876130 ] add C API to datetime module Message-ID: Patches item #876130, was opened at 2004-01-13 08:20 Message generated for change (Comment added) made by atuining You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 Category: Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Anthony Tuininga (atuining) Assigned to: M.-A. Lemburg (lemburg) Summary: add C API to datetime module Initial Comment: The datetime module does not have a C API which means that modules written in C cannot take advantage of the datetime module without incurring significant overhead. These patches supply this lack and are based on the C API exported by the mxDateTime module and the cStringIO module and enhanced as suggested by Guido. ---------------------------------------------------------------------- >Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 08:40 Message: Logged In: YES user_id=619560 I spent some time getting the two include files merged together, only to discover that Py_BUILD_CORE is __NOT__ defined when building modules, only when building stuff linked directly into the core interpreter. Perhaps this ought to be defined when building the modules with setup.py? At any rate, I cannot proceed along the route suggested without this problem being resolved. On the DateFromTicks(), TimestampFromTicks(), TimeFromTicks() issue, I'm not sure that there is much benefit to including a C API for this since if the datetime module becomes the standard method for managing datetime objects, there is no point in having the DB API modules provide something that the datetime module already provides, right? During the transition period the few lines of code needed can be posted. If you disagree I can certainly provide the few lines as part of the patch but I suspect you will find resistance to having this included by Tim Peters, the maintainer (I believe) of the datetime module. Feel free to correct me... :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-04-26 10:19 Message: Logged In: YES user_id=38388 Sorry for not getting back to you earlier: I didn't see you last message. 1-4) Python defines a macro during the Python build process that is not defined when compiling extensions: Py_BUILD_CORE You can use that macro to compile things differently for core things vs. extensions. 5) Fair enough; I don't see a point in creating a new numbering for each and every module. Please just use 1, 2, 3, 4, ... 6) Hope this makes sense :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-26 09:35 Message: Logged In: YES user_id=619560 Any chance to look at this? Any hope of this being included in Python 2.4? I know a number of people (including Guido) who would appreciate that.... :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-05 16:04 Message: Logged In: YES user_id=619560 I am back at work after finishing my move so I think I can finally devote a little more time to this issue. 1-4) The main reason for datetimeapi.h as opposed to simply modifying datetime.h was that the internal build needs to define things __differently__ from an external build of a module and there doesn't appear to be any way of fixing that -- so if you know a way I think I can manage to create a reasonable patch; otherwise, I'm out of my depth! 5) Guido was suggesting that a new PyIMPORT macro be made available which would verify that the magic number defined in the header file actually matches the magic number exported by the built module; this is simply a developer protection scheme which should eliminate really stupid uses; you can ask him about this or I can e-mail you part of the thread that brought this up 6) Makes sense. I'd like to get the questions above resolved first before proceeding any further, though. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 11:23 Message: Logged In: YES user_id=38388 Thanks for the comments... 1) Please put *all* the code for the C API into datetimeapi.h. I don't like your current mix of putting some things into datetime.h and others into datetimeapi.h (e.g. the PyDateTime_CAPI definition). 2) Since it's the only API missing if you want to use datetime objects in database interfacing, please add it. 3) dito. 4) Yes, the header file is the right place. Have a look at unicodeobject.h for what I have in mind (or mxDateTime.h for that matter). 5) Why add yet another magic number ? Isn't the Python API number enough ? 6) Since you don't provide a doc patch, please put comments into the header file. There can never be enough documentation and developers tend to look at the code rather than the docs ;-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-03-25 11:08 Message: Logged In: YES user_id=619560 Sorry for the delay. Things here have been quite busy recently both at work and at home. Let me see if I can answer your questions. 1) are you suggesting comments that include the macro names in datetimeapi.h similar to what I put in the comments below? Or something else? See modified datetimeapi.h for more info. 2) TimeFromTicks() is a very DB API specific routine. In chatting with others about the datetime module, they were not keen on adding such a beast. I guess there will have to be some discussion about whether datetimeapi.h is also a sort of half implementation for a C module implementing the DB API. I was intending to do this in my cx_Oracle module but not in datetimeapi.h. Could you clarify? 3) I could add C APIS for the three DB API ticks interfaces but I was simply intending to do that in cx_Oracle. Again, see #2. I can do it either way. :-) 4) I have added limited documentation in datetimeapi.h but is that really the right place for it? Is it even remotely close to what is needed? I'm not sure what is needed here exactly. 5) the API magic number is simply an arbitrary number which is stored in datetime.h and need not be understood by the implementor. For your information I simply took the name "datetime", converted each letter to its ordinal value in the alphabet, modded by 16 and used that hex character -- does that make any sense? This can be changed to whatever makes sense, though. 6) Whether or not datetimeapi.h should be sufficient for a developer or whether he should look at documentation in the usual locations (html pages, for example) I'll leave up to the Python development team. Let me know how I can assist. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 10:35 Message: Logged In: YES user_id=38388 Anthony, have you had a chance to look at the comments ? ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 08:06 Message: Logged In: YES user_id=619560 Oops! It looks like my own misunderstanding of SourceForge and how it works is to blame. :-( I'll take a look at this and get back to you as soon as possible -- ignore my previous comment. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 08:03 Message: Logged In: YES user_id=619560 You haven't yet responded to my previous comments -- it looks like SourceForge put them __after__ your comments so it is quite understandable why you didn't notice them. If you can give me the answers to those questions or provide additional questions that I need to answer, then I think we can make progress again. I was waiting for __you__ :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-02-25 16:08 Message: Logged In: YES user_id=38388 Any progress on this ? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-28 04:49 Message: Logged In: YES user_id=38388 Thanks for the clarification. I had forgotten about the macros -- I'd suggest that you document them in the datetimeapi.h include file to not cause the same confusion on the developer side (the trick to achieve adoption is to make things easy on the developer). As for TimeFromTicks(): This should be rather straight forward to implement: you take the time part of the ticks timestamp and create a time object from it. This is the way the DB API constructor works. Could you add C APIs for the three DB API ticks interface methods ? Other things that are missing: * documentation of the way the CAPI object can be used in datetimeapi.h * documentation of the various CAPI entries * documentation of the API magic number (there should be some scheme for this) I think that datetimeapi.h should be enough for a developer to read in order to use the interface. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 14:02 Message: Logged In: YES user_id=619560 I didn't think any additional API would be necessary for extracting date time information since the following macros are available: PyDateTime_GET_YEAR(o) PyDateTime_GET_MONTH(o) PyDateTime_GET_DAY(o) PyDateTime_DATE_GET_HOUR(o) PyDateTime_DATE_GET_MINUTE(o) PyDateTime_DATE_GET_SECOND(o) PyDateTime_DATE_GET_MICROSECOND(o) PyDateTime_TIME_GET_HOUR(o) PyDateTime_TIME_GET_MINUTE(o) PyDateTime_TIME_GET_SECOND(o) PyDateTime_TIME_GET_MICROSECOND(o) Were you thinking of something else that is needed? As for interfacing at the C level for converting from Unix ticks, the following method is already available and could simply be used as a drop in replacement for DateFromTicks() and TimestampFromTicks() from the DB API document. DateFromTicks() --> datetime.date.fromtimestamp() TimestampFromTicks() --> datetime.datetime.fromtimestamp() TimeFromTicks() is not already exposed but discussions with Tim Peters already suggested that would not happen since the concept is rather strange. You'll have to discuss that with him if you disagree! :-) Any comments? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-26 13:46 Message: Logged In: YES user_id=38388 This is a good start. However, you should add more APIs for extracting date/time information to the C API. mxDateTime.h from egenix-mx-base can provide a good basis for this. If you want to make the datetime module usable for database interfacing at C level, you'll also have to add interfaces for Unix ticks conversion, since these are often used by database C level interfaces. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 11:36 Message: Logged In: YES user_id=619560 I have no objection to providing patches for doc and test. A quick look at _testcapimodule.c doesn't provide any obvious ideas as to how to patch it. I looked for cStringIO which has a C API already and found nothing in there at all. The documentation also simply says "see the source for the information you require". Any suggestions as to how to proceed? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-01-25 20:17 Message: Logged In: YES user_id=31435 Marc-Andre, can you review this? I think you have more experience building C APIs for modules than anyone else (while I don't have any). There's certainly no objection to giving datetime a C API, and Guido already said he doesn't want to rely on that Martin changed datetime to be built as part of the core (in 2.3 on Windows, datetime.pyd was a distinct DLL; in CVS, datetime is compiled into the core DLL now). Anthony, you probably need to add doc and test patches. _testcapimodule.c holds tests of things you can only get at from C. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 From noreply at sourceforge.net Mon May 3 11:26:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 3 11:26:37 2004 Subject: [Patches] [ python-Patches-876130 ] add C API to datetime module Message-ID: Patches item #876130, was opened at 2004-01-13 16:20 Message generated for change (Comment added) made by lemburg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 Category: Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Anthony Tuininga (atuining) Assigned to: M.-A. Lemburg (lemburg) Summary: add C API to datetime module Initial Comment: The datetime module does not have a C API which means that modules written in C cannot take advantage of the datetime module without incurring significant overhead. These patches supply this lack and are based on the C API exported by the mxDateTime module and the cStringIO module and enhanced as suggested by Guido. ---------------------------------------------------------------------- >Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 17:26 Message: Logged In: YES user_id=38388 About the Py_BUILD_CORE issue: Sorry, I didn't know that Python does not define this macro for core modules. Nevertheless, looking at the datetime module code I don't really understand why you would need such a macro in the first place. Since the datetime module is the only code including the datetime.h header file it should be easy to add a macro definition just before importing datetime.h. On the TimeFromTicks() issue: The whole point of this discussion is to make the datetime module easily usable for authors of database modules. Since the DB API specifies that the module will need to export the three functions, two of which are available through the datetime module already, I don't really see your point in not wanting to add it. BTW, you are wrong in the assumption that the DB API spec will move away from a generic API for date/time objects. The DB API has always been defined in terms of interfaces and does not force a certain set of tools onto the developer. The datetime module will become one possible choice for DB API authors, others will want to stick to mxDateTime, use strings, ticks integers or more specific objects for interfacing. That won't change, so there is no "transition period". ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 16:40 Message: Logged In: YES user_id=619560 I spent some time getting the two include files merged together, only to discover that Py_BUILD_CORE is __NOT__ defined when building modules, only when building stuff linked directly into the core interpreter. Perhaps this ought to be defined when building the modules with setup.py? At any rate, I cannot proceed along the route suggested without this problem being resolved. On the DateFromTicks(), TimestampFromTicks(), TimeFromTicks() issue, I'm not sure that there is much benefit to including a C API for this since if the datetime module becomes the standard method for managing datetime objects, there is no point in having the DB API modules provide something that the datetime module already provides, right? During the transition period the few lines of code needed can be posted. If you disagree I can certainly provide the few lines as part of the patch but I suspect you will find resistance to having this included by Tim Peters, the maintainer (I believe) of the datetime module. Feel free to correct me... :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-04-26 18:19 Message: Logged In: YES user_id=38388 Sorry for not getting back to you earlier: I didn't see you last message. 1-4) Python defines a macro during the Python build process that is not defined when compiling extensions: Py_BUILD_CORE You can use that macro to compile things differently for core things vs. extensions. 5) Fair enough; I don't see a point in creating a new numbering for each and every module. Please just use 1, 2, 3, 4, ... 6) Hope this makes sense :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-26 17:35 Message: Logged In: YES user_id=619560 Any chance to look at this? Any hope of this being included in Python 2.4? I know a number of people (including Guido) who would appreciate that.... :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-06 00:04 Message: Logged In: YES user_id=619560 I am back at work after finishing my move so I think I can finally devote a little more time to this issue. 1-4) The main reason for datetimeapi.h as opposed to simply modifying datetime.h was that the internal build needs to define things __differently__ from an external build of a module and there doesn't appear to be any way of fixing that -- so if you know a way I think I can manage to create a reasonable patch; otherwise, I'm out of my depth! 5) Guido was suggesting that a new PyIMPORT macro be made available which would verify that the magic number defined in the header file actually matches the magic number exported by the built module; this is simply a developer protection scheme which should eliminate really stupid uses; you can ask him about this or I can e-mail you part of the thread that brought this up 6) Makes sense. I'd like to get the questions above resolved first before proceeding any further, though. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 19:23 Message: Logged In: YES user_id=38388 Thanks for the comments... 1) Please put *all* the code for the C API into datetimeapi.h. I don't like your current mix of putting some things into datetime.h and others into datetimeapi.h (e.g. the PyDateTime_CAPI definition). 2) Since it's the only API missing if you want to use datetime objects in database interfacing, please add it. 3) dito. 4) Yes, the header file is the right place. Have a look at unicodeobject.h for what I have in mind (or mxDateTime.h for that matter). 5) Why add yet another magic number ? Isn't the Python API number enough ? 6) Since you don't provide a doc patch, please put comments into the header file. There can never be enough documentation and developers tend to look at the code rather than the docs ;-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-03-25 19:08 Message: Logged In: YES user_id=619560 Sorry for the delay. Things here have been quite busy recently both at work and at home. Let me see if I can answer your questions. 1) are you suggesting comments that include the macro names in datetimeapi.h similar to what I put in the comments below? Or something else? See modified datetimeapi.h for more info. 2) TimeFromTicks() is a very DB API specific routine. In chatting with others about the datetime module, they were not keen on adding such a beast. I guess there will have to be some discussion about whether datetimeapi.h is also a sort of half implementation for a C module implementing the DB API. I was intending to do this in my cx_Oracle module but not in datetimeapi.h. Could you clarify? 3) I could add C APIS for the three DB API ticks interfaces but I was simply intending to do that in cx_Oracle. Again, see #2. I can do it either way. :-) 4) I have added limited documentation in datetimeapi.h but is that really the right place for it? Is it even remotely close to what is needed? I'm not sure what is needed here exactly. 5) the API magic number is simply an arbitrary number which is stored in datetime.h and need not be understood by the implementor. For your information I simply took the name "datetime", converted each letter to its ordinal value in the alphabet, modded by 16 and used that hex character -- does that make any sense? This can be changed to whatever makes sense, though. 6) Whether or not datetimeapi.h should be sufficient for a developer or whether he should look at documentation in the usual locations (html pages, for example) I'll leave up to the Python development team. Let me know how I can assist. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 18:35 Message: Logged In: YES user_id=38388 Anthony, have you had a chance to look at the comments ? ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 16:06 Message: Logged In: YES user_id=619560 Oops! It looks like my own misunderstanding of SourceForge and how it works is to blame. :-( I'll take a look at this and get back to you as soon as possible -- ignore my previous comment. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 16:03 Message: Logged In: YES user_id=619560 You haven't yet responded to my previous comments -- it looks like SourceForge put them __after__ your comments so it is quite understandable why you didn't notice them. If you can give me the answers to those questions or provide additional questions that I need to answer, then I think we can make progress again. I was waiting for __you__ :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-02-26 00:08 Message: Logged In: YES user_id=38388 Any progress on this ? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-28 12:49 Message: Logged In: YES user_id=38388 Thanks for the clarification. I had forgotten about the macros -- I'd suggest that you document them in the datetimeapi.h include file to not cause the same confusion on the developer side (the trick to achieve adoption is to make things easy on the developer). As for TimeFromTicks(): This should be rather straight forward to implement: you take the time part of the ticks timestamp and create a time object from it. This is the way the DB API constructor works. Could you add C APIs for the three DB API ticks interface methods ? Other things that are missing: * documentation of the way the CAPI object can be used in datetimeapi.h * documentation of the various CAPI entries * documentation of the API magic number (there should be some scheme for this) I think that datetimeapi.h should be enough for a developer to read in order to use the interface. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 22:02 Message: Logged In: YES user_id=619560 I didn't think any additional API would be necessary for extracting date time information since the following macros are available: PyDateTime_GET_YEAR(o) PyDateTime_GET_MONTH(o) PyDateTime_GET_DAY(o) PyDateTime_DATE_GET_HOUR(o) PyDateTime_DATE_GET_MINUTE(o) PyDateTime_DATE_GET_SECOND(o) PyDateTime_DATE_GET_MICROSECOND(o) PyDateTime_TIME_GET_HOUR(o) PyDateTime_TIME_GET_MINUTE(o) PyDateTime_TIME_GET_SECOND(o) PyDateTime_TIME_GET_MICROSECOND(o) Were you thinking of something else that is needed? As for interfacing at the C level for converting from Unix ticks, the following method is already available and could simply be used as a drop in replacement for DateFromTicks() and TimestampFromTicks() from the DB API document. DateFromTicks() --> datetime.date.fromtimestamp() TimestampFromTicks() --> datetime.datetime.fromtimestamp() TimeFromTicks() is not already exposed but discussions with Tim Peters already suggested that would not happen since the concept is rather strange. You'll have to discuss that with him if you disagree! :-) Any comments? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-26 21:46 Message: Logged In: YES user_id=38388 This is a good start. However, you should add more APIs for extracting date/time information to the C API. mxDateTime.h from egenix-mx-base can provide a good basis for this. If you want to make the datetime module usable for database interfacing at C level, you'll also have to add interfaces for Unix ticks conversion, since these are often used by database C level interfaces. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 19:36 Message: Logged In: YES user_id=619560 I have no objection to providing patches for doc and test. A quick look at _testcapimodule.c doesn't provide any obvious ideas as to how to patch it. I looked for cStringIO which has a C API already and found nothing in there at all. The documentation also simply says "see the source for the information you require". Any suggestions as to how to proceed? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-01-26 04:17 Message: Logged In: YES user_id=31435 Marc-Andre, can you review this? I think you have more experience building C APIs for modules than anyone else (while I don't have any). There's certainly no objection to giving datetime a C API, and Guido already said he doesn't want to rely on that Martin changed datetime to be built as part of the core (in 2.3 on Windows, datetime.pyd was a distinct DLL; in CVS, datetime is compiled into the core DLL now). Anthony, you probably need to add doc and test patches. _testcapimodule.c holds tests of things you can only get at from C. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 From noreply at sourceforge.net Mon May 3 12:48:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 3 12:48:54 2004 Subject: [Patches] [ python-Patches-876130 ] add C API to datetime module Message-ID: Patches item #876130, was opened at 2004-01-13 08:20 Message generated for change (Comment added) made by atuining You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 Category: Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Anthony Tuininga (atuining) Assigned to: M.-A. Lemburg (lemburg) Summary: add C API to datetime module Initial Comment: The datetime module does not have a C API which means that modules written in C cannot take advantage of the datetime module without incurring significant overhead. These patches supply this lack and are based on the C API exported by the mxDateTime module and the cStringIO module and enhanced as suggested by Guido. ---------------------------------------------------------------------- >Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 10:48 Message: Logged In: YES user_id=619560 About the Py_BUILD_CORE issue: You are suggesting that I define this in datetimemodule.c just prior to the include of datetime.h? Or are you suggesting some other macro? I am not sure about the comment about "not needing a macro in the first place" since there are two groups that will be importing datetime.h: the first being the datetime module itself and the second being the DB API modules. The macro is definitely required to distinguish between the two -- unless I am missing something? On the TimeFromTicks() issue: You obviously consider it important that the three methods be available at the C level for DB API module authors to use directly. My objection is simply a reiteration of an objection raised by Tim Peters. I think you'll have to argue that one with him. :-) I'll provide my implementations for cx_Oracle as part of the patches and then you can fight it out and I'll stay out of it. :-) As for the transition period, I assumed that with the introduction of a standard datetime module, that it would become the suggested implementation for database interfaace modules. That said, I don't really have any authority to say anything on this front so I'll leave that to you and others to decide. :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 09:26 Message: Logged In: YES user_id=38388 About the Py_BUILD_CORE issue: Sorry, I didn't know that Python does not define this macro for core modules. Nevertheless, looking at the datetime module code I don't really understand why you would need such a macro in the first place. Since the datetime module is the only code including the datetime.h header file it should be easy to add a macro definition just before importing datetime.h. On the TimeFromTicks() issue: The whole point of this discussion is to make the datetime module easily usable for authors of database modules. Since the DB API specifies that the module will need to export the three functions, two of which are available through the datetime module already, I don't really see your point in not wanting to add it. BTW, you are wrong in the assumption that the DB API spec will move away from a generic API for date/time objects. The DB API has always been defined in terms of interfaces and does not force a certain set of tools onto the developer. The datetime module will become one possible choice for DB API authors, others will want to stick to mxDateTime, use strings, ticks integers or more specific objects for interfacing. That won't change, so there is no "transition period". ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 08:40 Message: Logged In: YES user_id=619560 I spent some time getting the two include files merged together, only to discover that Py_BUILD_CORE is __NOT__ defined when building modules, only when building stuff linked directly into the core interpreter. Perhaps this ought to be defined when building the modules with setup.py? At any rate, I cannot proceed along the route suggested without this problem being resolved. On the DateFromTicks(), TimestampFromTicks(), TimeFromTicks() issue, I'm not sure that there is much benefit to including a C API for this since if the datetime module becomes the standard method for managing datetime objects, there is no point in having the DB API modules provide something that the datetime module already provides, right? During the transition period the few lines of code needed can be posted. If you disagree I can certainly provide the few lines as part of the patch but I suspect you will find resistance to having this included by Tim Peters, the maintainer (I believe) of the datetime module. Feel free to correct me... :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-04-26 10:19 Message: Logged In: YES user_id=38388 Sorry for not getting back to you earlier: I didn't see you last message. 1-4) Python defines a macro during the Python build process that is not defined when compiling extensions: Py_BUILD_CORE You can use that macro to compile things differently for core things vs. extensions. 5) Fair enough; I don't see a point in creating a new numbering for each and every module. Please just use 1, 2, 3, 4, ... 6) Hope this makes sense :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-26 09:35 Message: Logged In: YES user_id=619560 Any chance to look at this? Any hope of this being included in Python 2.4? I know a number of people (including Guido) who would appreciate that.... :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-05 16:04 Message: Logged In: YES user_id=619560 I am back at work after finishing my move so I think I can finally devote a little more time to this issue. 1-4) The main reason for datetimeapi.h as opposed to simply modifying datetime.h was that the internal build needs to define things __differently__ from an external build of a module and there doesn't appear to be any way of fixing that -- so if you know a way I think I can manage to create a reasonable patch; otherwise, I'm out of my depth! 5) Guido was suggesting that a new PyIMPORT macro be made available which would verify that the magic number defined in the header file actually matches the magic number exported by the built module; this is simply a developer protection scheme which should eliminate really stupid uses; you can ask him about this or I can e-mail you part of the thread that brought this up 6) Makes sense. I'd like to get the questions above resolved first before proceeding any further, though. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 11:23 Message: Logged In: YES user_id=38388 Thanks for the comments... 1) Please put *all* the code for the C API into datetimeapi.h. I don't like your current mix of putting some things into datetime.h and others into datetimeapi.h (e.g. the PyDateTime_CAPI definition). 2) Since it's the only API missing if you want to use datetime objects in database interfacing, please add it. 3) dito. 4) Yes, the header file is the right place. Have a look at unicodeobject.h for what I have in mind (or mxDateTime.h for that matter). 5) Why add yet another magic number ? Isn't the Python API number enough ? 6) Since you don't provide a doc patch, please put comments into the header file. There can never be enough documentation and developers tend to look at the code rather than the docs ;-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-03-25 11:08 Message: Logged In: YES user_id=619560 Sorry for the delay. Things here have been quite busy recently both at work and at home. Let me see if I can answer your questions. 1) are you suggesting comments that include the macro names in datetimeapi.h similar to what I put in the comments below? Or something else? See modified datetimeapi.h for more info. 2) TimeFromTicks() is a very DB API specific routine. In chatting with others about the datetime module, they were not keen on adding such a beast. I guess there will have to be some discussion about whether datetimeapi.h is also a sort of half implementation for a C module implementing the DB API. I was intending to do this in my cx_Oracle module but not in datetimeapi.h. Could you clarify? 3) I could add C APIS for the three DB API ticks interfaces but I was simply intending to do that in cx_Oracle. Again, see #2. I can do it either way. :-) 4) I have added limited documentation in datetimeapi.h but is that really the right place for it? Is it even remotely close to what is needed? I'm not sure what is needed here exactly. 5) the API magic number is simply an arbitrary number which is stored in datetime.h and need not be understood by the implementor. For your information I simply took the name "datetime", converted each letter to its ordinal value in the alphabet, modded by 16 and used that hex character -- does that make any sense? This can be changed to whatever makes sense, though. 6) Whether or not datetimeapi.h should be sufficient for a developer or whether he should look at documentation in the usual locations (html pages, for example) I'll leave up to the Python development team. Let me know how I can assist. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 10:35 Message: Logged In: YES user_id=38388 Anthony, have you had a chance to look at the comments ? ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 08:06 Message: Logged In: YES user_id=619560 Oops! It looks like my own misunderstanding of SourceForge and how it works is to blame. :-( I'll take a look at this and get back to you as soon as possible -- ignore my previous comment. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 08:03 Message: Logged In: YES user_id=619560 You haven't yet responded to my previous comments -- it looks like SourceForge put them __after__ your comments so it is quite understandable why you didn't notice them. If you can give me the answers to those questions or provide additional questions that I need to answer, then I think we can make progress again. I was waiting for __you__ :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-02-25 16:08 Message: Logged In: YES user_id=38388 Any progress on this ? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-28 04:49 Message: Logged In: YES user_id=38388 Thanks for the clarification. I had forgotten about the macros -- I'd suggest that you document them in the datetimeapi.h include file to not cause the same confusion on the developer side (the trick to achieve adoption is to make things easy on the developer). As for TimeFromTicks(): This should be rather straight forward to implement: you take the time part of the ticks timestamp and create a time object from it. This is the way the DB API constructor works. Could you add C APIs for the three DB API ticks interface methods ? Other things that are missing: * documentation of the way the CAPI object can be used in datetimeapi.h * documentation of the various CAPI entries * documentation of the API magic number (there should be some scheme for this) I think that datetimeapi.h should be enough for a developer to read in order to use the interface. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 14:02 Message: Logged In: YES user_id=619560 I didn't think any additional API would be necessary for extracting date time information since the following macros are available: PyDateTime_GET_YEAR(o) PyDateTime_GET_MONTH(o) PyDateTime_GET_DAY(o) PyDateTime_DATE_GET_HOUR(o) PyDateTime_DATE_GET_MINUTE(o) PyDateTime_DATE_GET_SECOND(o) PyDateTime_DATE_GET_MICROSECOND(o) PyDateTime_TIME_GET_HOUR(o) PyDateTime_TIME_GET_MINUTE(o) PyDateTime_TIME_GET_SECOND(o) PyDateTime_TIME_GET_MICROSECOND(o) Were you thinking of something else that is needed? As for interfacing at the C level for converting from Unix ticks, the following method is already available and could simply be used as a drop in replacement for DateFromTicks() and TimestampFromTicks() from the DB API document. DateFromTicks() --> datetime.date.fromtimestamp() TimestampFromTicks() --> datetime.datetime.fromtimestamp() TimeFromTicks() is not already exposed but discussions with Tim Peters already suggested that would not happen since the concept is rather strange. You'll have to discuss that with him if you disagree! :-) Any comments? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-26 13:46 Message: Logged In: YES user_id=38388 This is a good start. However, you should add more APIs for extracting date/time information to the C API. mxDateTime.h from egenix-mx-base can provide a good basis for this. If you want to make the datetime module usable for database interfacing at C level, you'll also have to add interfaces for Unix ticks conversion, since these are often used by database C level interfaces. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 11:36 Message: Logged In: YES user_id=619560 I have no objection to providing patches for doc and test. A quick look at _testcapimodule.c doesn't provide any obvious ideas as to how to patch it. I looked for cStringIO which has a C API already and found nothing in there at all. The documentation also simply says "see the source for the information you require". Any suggestions as to how to proceed? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-01-25 20:17 Message: Logged In: YES user_id=31435 Marc-Andre, can you review this? I think you have more experience building C APIs for modules than anyone else (while I don't have any). There's certainly no objection to giving datetime a C API, and Guido already said he doesn't want to rely on that Martin changed datetime to be built as part of the core (in 2.3 on Windows, datetime.pyd was a distinct DLL; in CVS, datetime is compiled into the core DLL now). Anthony, you probably need to add doc and test patches. _testcapimodule.c holds tests of things you can only get at from C. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 From noreply at sourceforge.net Mon May 3 13:24:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 3 13:24:07 2004 Subject: [Patches] [ python-Patches-876130 ] add C API to datetime module Message-ID: Patches item #876130, was opened at 2004-01-13 16:20 Message generated for change (Comment added) made by lemburg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 Category: Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Anthony Tuininga (atuining) Assigned to: M.-A. Lemburg (lemburg) Summary: add C API to datetime module Initial Comment: The datetime module does not have a C API which means that modules written in C cannot take advantage of the datetime module without incurring significant overhead. These patches supply this lack and are based on the C API exported by the mxDateTime module and the cStringIO module and enhanced as suggested by Guido. ---------------------------------------------------------------------- >Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 19:24 Message: Logged In: YES user_id=38388 Macro: I don't know whether you need the macro or not (you probably do for the type checks). In any case, you can do without relying on Python providing you this information via the build process, since only datetimemodule.c is using the header file while Python is being built. TimeFromTicks(): I' ve stated my point. I don't think there's anything to add - I'm tired of fighting for these things... one of the reasons I stopped actively discussing things on python-dev. IMHO, a product that's aimed at developers should make things easy for the developer and try to avoid code duplication if possible. The API is needed in order to support database interfaces that use Unix ticks as basis for date/time, so if you don't add it to the module, the ten or twenty module authors out there will have to duplicate that piece of work in their implementation. Transition: This was already discussed on the db-api list. We love freedom of choice. Nothing to add. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 18:48 Message: Logged In: YES user_id=619560 About the Py_BUILD_CORE issue: You are suggesting that I define this in datetimemodule.c just prior to the include of datetime.h? Or are you suggesting some other macro? I am not sure about the comment about "not needing a macro in the first place" since there are two groups that will be importing datetime.h: the first being the datetime module itself and the second being the DB API modules. The macro is definitely required to distinguish between the two -- unless I am missing something? On the TimeFromTicks() issue: You obviously consider it important that the three methods be available at the C level for DB API module authors to use directly. My objection is simply a reiteration of an objection raised by Tim Peters. I think you'll have to argue that one with him. :-) I'll provide my implementations for cx_Oracle as part of the patches and then you can fight it out and I'll stay out of it. :-) As for the transition period, I assumed that with the introduction of a standard datetime module, that it would become the suggested implementation for database interfaace modules. That said, I don't really have any authority to say anything on this front so I'll leave that to you and others to decide. :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 17:26 Message: Logged In: YES user_id=38388 About the Py_BUILD_CORE issue: Sorry, I didn't know that Python does not define this macro for core modules. Nevertheless, looking at the datetime module code I don't really understand why you would need such a macro in the first place. Since the datetime module is the only code including the datetime.h header file it should be easy to add a macro definition just before importing datetime.h. On the TimeFromTicks() issue: The whole point of this discussion is to make the datetime module easily usable for authors of database modules. Since the DB API specifies that the module will need to export the three functions, two of which are available through the datetime module already, I don't really see your point in not wanting to add it. BTW, you are wrong in the assumption that the DB API spec will move away from a generic API for date/time objects. The DB API has always been defined in terms of interfaces and does not force a certain set of tools onto the developer. The datetime module will become one possible choice for DB API authors, others will want to stick to mxDateTime, use strings, ticks integers or more specific objects for interfacing. That won't change, so there is no "transition period". ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 16:40 Message: Logged In: YES user_id=619560 I spent some time getting the two include files merged together, only to discover that Py_BUILD_CORE is __NOT__ defined when building modules, only when building stuff linked directly into the core interpreter. Perhaps this ought to be defined when building the modules with setup.py? At any rate, I cannot proceed along the route suggested without this problem being resolved. On the DateFromTicks(), TimestampFromTicks(), TimeFromTicks() issue, I'm not sure that there is much benefit to including a C API for this since if the datetime module becomes the standard method for managing datetime objects, there is no point in having the DB API modules provide something that the datetime module already provides, right? During the transition period the few lines of code needed can be posted. If you disagree I can certainly provide the few lines as part of the patch but I suspect you will find resistance to having this included by Tim Peters, the maintainer (I believe) of the datetime module. Feel free to correct me... :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-04-26 18:19 Message: Logged In: YES user_id=38388 Sorry for not getting back to you earlier: I didn't see you last message. 1-4) Python defines a macro during the Python build process that is not defined when compiling extensions: Py_BUILD_CORE You can use that macro to compile things differently for core things vs. extensions. 5) Fair enough; I don't see a point in creating a new numbering for each and every module. Please just use 1, 2, 3, 4, ... 6) Hope this makes sense :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-26 17:35 Message: Logged In: YES user_id=619560 Any chance to look at this? Any hope of this being included in Python 2.4? I know a number of people (including Guido) who would appreciate that.... :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-06 00:04 Message: Logged In: YES user_id=619560 I am back at work after finishing my move so I think I can finally devote a little more time to this issue. 1-4) The main reason for datetimeapi.h as opposed to simply modifying datetime.h was that the internal build needs to define things __differently__ from an external build of a module and there doesn't appear to be any way of fixing that -- so if you know a way I think I can manage to create a reasonable patch; otherwise, I'm out of my depth! 5) Guido was suggesting that a new PyIMPORT macro be made available which would verify that the magic number defined in the header file actually matches the magic number exported by the built module; this is simply a developer protection scheme which should eliminate really stupid uses; you can ask him about this or I can e-mail you part of the thread that brought this up 6) Makes sense. I'd like to get the questions above resolved first before proceeding any further, though. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 19:23 Message: Logged In: YES user_id=38388 Thanks for the comments... 1) Please put *all* the code for the C API into datetimeapi.h. I don't like your current mix of putting some things into datetime.h and others into datetimeapi.h (e.g. the PyDateTime_CAPI definition). 2) Since it's the only API missing if you want to use datetime objects in database interfacing, please add it. 3) dito. 4) Yes, the header file is the right place. Have a look at unicodeobject.h for what I have in mind (or mxDateTime.h for that matter). 5) Why add yet another magic number ? Isn't the Python API number enough ? 6) Since you don't provide a doc patch, please put comments into the header file. There can never be enough documentation and developers tend to look at the code rather than the docs ;-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-03-25 19:08 Message: Logged In: YES user_id=619560 Sorry for the delay. Things here have been quite busy recently both at work and at home. Let me see if I can answer your questions. 1) are you suggesting comments that include the macro names in datetimeapi.h similar to what I put in the comments below? Or something else? See modified datetimeapi.h for more info. 2) TimeFromTicks() is a very DB API specific routine. In chatting with others about the datetime module, they were not keen on adding such a beast. I guess there will have to be some discussion about whether datetimeapi.h is also a sort of half implementation for a C module implementing the DB API. I was intending to do this in my cx_Oracle module but not in datetimeapi.h. Could you clarify? 3) I could add C APIS for the three DB API ticks interfaces but I was simply intending to do that in cx_Oracle. Again, see #2. I can do it either way. :-) 4) I have added limited documentation in datetimeapi.h but is that really the right place for it? Is it even remotely close to what is needed? I'm not sure what is needed here exactly. 5) the API magic number is simply an arbitrary number which is stored in datetime.h and need not be understood by the implementor. For your information I simply took the name "datetime", converted each letter to its ordinal value in the alphabet, modded by 16 and used that hex character -- does that make any sense? This can be changed to whatever makes sense, though. 6) Whether or not datetimeapi.h should be sufficient for a developer or whether he should look at documentation in the usual locations (html pages, for example) I'll leave up to the Python development team. Let me know how I can assist. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 18:35 Message: Logged In: YES user_id=38388 Anthony, have you had a chance to look at the comments ? ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 16:06 Message: Logged In: YES user_id=619560 Oops! It looks like my own misunderstanding of SourceForge and how it works is to blame. :-( I'll take a look at this and get back to you as soon as possible -- ignore my previous comment. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 16:03 Message: Logged In: YES user_id=619560 You haven't yet responded to my previous comments -- it looks like SourceForge put them __after__ your comments so it is quite understandable why you didn't notice them. If you can give me the answers to those questions or provide additional questions that I need to answer, then I think we can make progress again. I was waiting for __you__ :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-02-26 00:08 Message: Logged In: YES user_id=38388 Any progress on this ? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-28 12:49 Message: Logged In: YES user_id=38388 Thanks for the clarification. I had forgotten about the macros -- I'd suggest that you document them in the datetimeapi.h include file to not cause the same confusion on the developer side (the trick to achieve adoption is to make things easy on the developer). As for TimeFromTicks(): This should be rather straight forward to implement: you take the time part of the ticks timestamp and create a time object from it. This is the way the DB API constructor works. Could you add C APIs for the three DB API ticks interface methods ? Other things that are missing: * documentation of the way the CAPI object can be used in datetimeapi.h * documentation of the various CAPI entries * documentation of the API magic number (there should be some scheme for this) I think that datetimeapi.h should be enough for a developer to read in order to use the interface. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 22:02 Message: Logged In: YES user_id=619560 I didn't think any additional API would be necessary for extracting date time information since the following macros are available: PyDateTime_GET_YEAR(o) PyDateTime_GET_MONTH(o) PyDateTime_GET_DAY(o) PyDateTime_DATE_GET_HOUR(o) PyDateTime_DATE_GET_MINUTE(o) PyDateTime_DATE_GET_SECOND(o) PyDateTime_DATE_GET_MICROSECOND(o) PyDateTime_TIME_GET_HOUR(o) PyDateTime_TIME_GET_MINUTE(o) PyDateTime_TIME_GET_SECOND(o) PyDateTime_TIME_GET_MICROSECOND(o) Were you thinking of something else that is needed? As for interfacing at the C level for converting from Unix ticks, the following method is already available and could simply be used as a drop in replacement for DateFromTicks() and TimestampFromTicks() from the DB API document. DateFromTicks() --> datetime.date.fromtimestamp() TimestampFromTicks() --> datetime.datetime.fromtimestamp() TimeFromTicks() is not already exposed but discussions with Tim Peters already suggested that would not happen since the concept is rather strange. You'll have to discuss that with him if you disagree! :-) Any comments? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-26 21:46 Message: Logged In: YES user_id=38388 This is a good start. However, you should add more APIs for extracting date/time information to the C API. mxDateTime.h from egenix-mx-base can provide a good basis for this. If you want to make the datetime module usable for database interfacing at C level, you'll also have to add interfaces for Unix ticks conversion, since these are often used by database C level interfaces. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 19:36 Message: Logged In: YES user_id=619560 I have no objection to providing patches for doc and test. A quick look at _testcapimodule.c doesn't provide any obvious ideas as to how to patch it. I looked for cStringIO which has a C API already and found nothing in there at all. The documentation also simply says "see the source for the information you require". Any suggestions as to how to proceed? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-01-26 04:17 Message: Logged In: YES user_id=31435 Marc-Andre, can you review this? I think you have more experience building C APIs for modules than anyone else (while I don't have any). There's certainly no objection to giving datetime a C API, and Guido already said he doesn't want to rely on that Martin changed datetime to be built as part of the core (in 2.3 on Windows, datetime.pyd was a distinct DLL; in CVS, datetime is compiled into the core DLL now). Anthony, you probably need to add doc and test patches. _testcapimodule.c holds tests of things you can only get at from C. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 From noreply at sourceforge.net Mon May 3 13:46:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 3 13:46:50 2004 Subject: [Patches] [ python-Patches-876130 ] add C API to datetime module Message-ID: Patches item #876130, was opened at 2004-01-13 10:20 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 Category: Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Anthony Tuininga (atuining) Assigned to: M.-A. Lemburg (lemburg) Summary: add C API to datetime module Initial Comment: The datetime module does not have a C API which means that modules written in C cannot take advantage of the datetime module without incurring significant overhead. These patches supply this lack and are based on the C API exported by the mxDateTime module and the cStringIO module and enhanced as suggested by Guido. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-03 13:46 Message: Logged In: YES user_id=31435 I don't understand what TimeFromTicks() is supposed to do. Seconds-from-the-epoch is a point in time, not a time-of- day. If some DB API requires guessing some transformation from seconds-from-the-epoch to time-of-day, that's fine, but it doesn't belong in Python's datetime module. The datetime module should certainly have a method to construct a datetime.datetime from a seconds-from-the- epoch argument -- seconds-from-the-epoch is a way of specifying a datetime.datetime. Given that, if the intent is to throw away the datetime.date portion of the datetime.datetime object, retaining only the datetime.time portion, then that's trivially accomplished by invoking dt.time () (where dt is the datetime.datetime object). Do any databases really store time-of-day as a seconds-from- the-epoch value? Google's top hit on TimeFromTicks() today happens to be a msg from Anthony saying that Oracle does not. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 13:24 Message: Logged In: YES user_id=38388 Macro: I don't know whether you need the macro or not (you probably do for the type checks). In any case, you can do without relying on Python providing you this information via the build process, since only datetimemodule.c is using the header file while Python is being built. TimeFromTicks(): I' ve stated my point. I don't think there's anything to add - I'm tired of fighting for these things... one of the reasons I stopped actively discussing things on python-dev. IMHO, a product that's aimed at developers should make things easy for the developer and try to avoid code duplication if possible. The API is needed in order to support database interfaces that use Unix ticks as basis for date/time, so if you don't add it to the module, the ten or twenty module authors out there will have to duplicate that piece of work in their implementation. Transition: This was already discussed on the db-api list. We love freedom of choice. Nothing to add. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 12:48 Message: Logged In: YES user_id=619560 About the Py_BUILD_CORE issue: You are suggesting that I define this in datetimemodule.c just prior to the include of datetime.h? Or are you suggesting some other macro? I am not sure about the comment about "not needing a macro in the first place" since there are two groups that will be importing datetime.h: the first being the datetime module itself and the second being the DB API modules. The macro is definitely required to distinguish between the two -- unless I am missing something? On the TimeFromTicks() issue: You obviously consider it important that the three methods be available at the C level for DB API module authors to use directly. My objection is simply a reiteration of an objection raised by Tim Peters. I think you'll have to argue that one with him. :-) I'll provide my implementations for cx_Oracle as part of the patches and then you can fight it out and I'll stay out of it. :-) As for the transition period, I assumed that with the introduction of a standard datetime module, that it would become the suggested implementation for database interfaace modules. That said, I don't really have any authority to say anything on this front so I'll leave that to you and others to decide. :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 11:26 Message: Logged In: YES user_id=38388 About the Py_BUILD_CORE issue: Sorry, I didn't know that Python does not define this macro for core modules. Nevertheless, looking at the datetime module code I don't really understand why you would need such a macro in the first place. Since the datetime module is the only code including the datetime.h header file it should be easy to add a macro definition just before importing datetime.h. On the TimeFromTicks() issue: The whole point of this discussion is to make the datetime module easily usable for authors of database modules. Since the DB API specifies that the module will need to export the three functions, two of which are available through the datetime module already, I don't really see your point in not wanting to add it. BTW, you are wrong in the assumption that the DB API spec will move away from a generic API for date/time objects. The DB API has always been defined in terms of interfaces and does not force a certain set of tools onto the developer. The datetime module will become one possible choice for DB API authors, others will want to stick to mxDateTime, use strings, ticks integers or more specific objects for interfacing. That won't change, so there is no "transition period". ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 10:40 Message: Logged In: YES user_id=619560 I spent some time getting the two include files merged together, only to discover that Py_BUILD_CORE is __NOT__ defined when building modules, only when building stuff linked directly into the core interpreter. Perhaps this ought to be defined when building the modules with setup.py? At any rate, I cannot proceed along the route suggested without this problem being resolved. On the DateFromTicks(), TimestampFromTicks(), TimeFromTicks() issue, I'm not sure that there is much benefit to including a C API for this since if the datetime module becomes the standard method for managing datetime objects, there is no point in having the DB API modules provide something that the datetime module already provides, right? During the transition period the few lines of code needed can be posted. If you disagree I can certainly provide the few lines as part of the patch but I suspect you will find resistance to having this included by Tim Peters, the maintainer (I believe) of the datetime module. Feel free to correct me... :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-04-26 12:19 Message: Logged In: YES user_id=38388 Sorry for not getting back to you earlier: I didn't see you last message. 1-4) Python defines a macro during the Python build process that is not defined when compiling extensions: Py_BUILD_CORE You can use that macro to compile things differently for core things vs. extensions. 5) Fair enough; I don't see a point in creating a new numbering for each and every module. Please just use 1, 2, 3, 4, ... 6) Hope this makes sense :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-26 11:35 Message: Logged In: YES user_id=619560 Any chance to look at this? Any hope of this being included in Python 2.4? I know a number of people (including Guido) who would appreciate that.... :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-05 18:04 Message: Logged In: YES user_id=619560 I am back at work after finishing my move so I think I can finally devote a little more time to this issue. 1-4) The main reason for datetimeapi.h as opposed to simply modifying datetime.h was that the internal build needs to define things __differently__ from an external build of a module and there doesn't appear to be any way of fixing that -- so if you know a way I think I can manage to create a reasonable patch; otherwise, I'm out of my depth! 5) Guido was suggesting that a new PyIMPORT macro be made available which would verify that the magic number defined in the header file actually matches the magic number exported by the built module; this is simply a developer protection scheme which should eliminate really stupid uses; you can ask him about this or I can e-mail you part of the thread that brought this up 6) Makes sense. I'd like to get the questions above resolved first before proceeding any further, though. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 13:23 Message: Logged In: YES user_id=38388 Thanks for the comments... 1) Please put *all* the code for the C API into datetimeapi.h. I don't like your current mix of putting some things into datetime.h and others into datetimeapi.h (e.g. the PyDateTime_CAPI definition). 2) Since it's the only API missing if you want to use datetime objects in database interfacing, please add it. 3) dito. 4) Yes, the header file is the right place. Have a look at unicodeobject.h for what I have in mind (or mxDateTime.h for that matter). 5) Why add yet another magic number ? Isn't the Python API number enough ? 6) Since you don't provide a doc patch, please put comments into the header file. There can never be enough documentation and developers tend to look at the code rather than the docs ;-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-03-25 13:08 Message: Logged In: YES user_id=619560 Sorry for the delay. Things here have been quite busy recently both at work and at home. Let me see if I can answer your questions. 1) are you suggesting comments that include the macro names in datetimeapi.h similar to what I put in the comments below? Or something else? See modified datetimeapi.h for more info. 2) TimeFromTicks() is a very DB API specific routine. In chatting with others about the datetime module, they were not keen on adding such a beast. I guess there will have to be some discussion about whether datetimeapi.h is also a sort of half implementation for a C module implementing the DB API. I was intending to do this in my cx_Oracle module but not in datetimeapi.h. Could you clarify? 3) I could add C APIS for the three DB API ticks interfaces but I was simply intending to do that in cx_Oracle. Again, see #2. I can do it either way. :-) 4) I have added limited documentation in datetimeapi.h but is that really the right place for it? Is it even remotely close to what is needed? I'm not sure what is needed here exactly. 5) the API magic number is simply an arbitrary number which is stored in datetime.h and need not be understood by the implementor. For your information I simply took the name "datetime", converted each letter to its ordinal value in the alphabet, modded by 16 and used that hex character -- does that make any sense? This can be changed to whatever makes sense, though. 6) Whether or not datetimeapi.h should be sufficient for a developer or whether he should look at documentation in the usual locations (html pages, for example) I'll leave up to the Python development team. Let me know how I can assist. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 12:35 Message: Logged In: YES user_id=38388 Anthony, have you had a chance to look at the comments ? ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 10:06 Message: Logged In: YES user_id=619560 Oops! It looks like my own misunderstanding of SourceForge and how it works is to blame. :-( I'll take a look at this and get back to you as soon as possible -- ignore my previous comment. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 10:03 Message: Logged In: YES user_id=619560 You haven't yet responded to my previous comments -- it looks like SourceForge put them __after__ your comments so it is quite understandable why you didn't notice them. If you can give me the answers to those questions or provide additional questions that I need to answer, then I think we can make progress again. I was waiting for __you__ :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-02-25 18:08 Message: Logged In: YES user_id=38388 Any progress on this ? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-28 06:49 Message: Logged In: YES user_id=38388 Thanks for the clarification. I had forgotten about the macros -- I'd suggest that you document them in the datetimeapi.h include file to not cause the same confusion on the developer side (the trick to achieve adoption is to make things easy on the developer). As for TimeFromTicks(): This should be rather straight forward to implement: you take the time part of the ticks timestamp and create a time object from it. This is the way the DB API constructor works. Could you add C APIs for the three DB API ticks interface methods ? Other things that are missing: * documentation of the way the CAPI object can be used in datetimeapi.h * documentation of the various CAPI entries * documentation of the API magic number (there should be some scheme for this) I think that datetimeapi.h should be enough for a developer to read in order to use the interface. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 16:02 Message: Logged In: YES user_id=619560 I didn't think any additional API would be necessary for extracting date time information since the following macros are available: PyDateTime_GET_YEAR(o) PyDateTime_GET_MONTH(o) PyDateTime_GET_DAY(o) PyDateTime_DATE_GET_HOUR(o) PyDateTime_DATE_GET_MINUTE(o) PyDateTime_DATE_GET_SECOND(o) PyDateTime_DATE_GET_MICROSECOND(o) PyDateTime_TIME_GET_HOUR(o) PyDateTime_TIME_GET_MINUTE(o) PyDateTime_TIME_GET_SECOND(o) PyDateTime_TIME_GET_MICROSECOND(o) Were you thinking of something else that is needed? As for interfacing at the C level for converting from Unix ticks, the following method is already available and could simply be used as a drop in replacement for DateFromTicks() and TimestampFromTicks() from the DB API document. DateFromTicks() --> datetime.date.fromtimestamp() TimestampFromTicks() --> datetime.datetime.fromtimestamp() TimeFromTicks() is not already exposed but discussions with Tim Peters already suggested that would not happen since the concept is rather strange. You'll have to discuss that with him if you disagree! :-) Any comments? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-26 15:46 Message: Logged In: YES user_id=38388 This is a good start. However, you should add more APIs for extracting date/time information to the C API. mxDateTime.h from egenix-mx-base can provide a good basis for this. If you want to make the datetime module usable for database interfacing at C level, you'll also have to add interfaces for Unix ticks conversion, since these are often used by database C level interfaces. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 13:36 Message: Logged In: YES user_id=619560 I have no objection to providing patches for doc and test. A quick look at _testcapimodule.c doesn't provide any obvious ideas as to how to patch it. I looked for cStringIO which has a C API already and found nothing in there at all. The documentation also simply says "see the source for the information you require". Any suggestions as to how to proceed? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-01-25 22:17 Message: Logged In: YES user_id=31435 Marc-Andre, can you review this? I think you have more experience building C APIs for modules than anyone else (while I don't have any). There's certainly no objection to giving datetime a C API, and Guido already said he doesn't want to rely on that Martin changed datetime to be built as part of the core (in 2.3 on Windows, datetime.pyd was a distinct DLL; in CVS, datetime is compiled into the core DLL now). Anthony, you probably need to add doc and test patches. _testcapimodule.c holds tests of things you can only get at from C. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 From noreply at sourceforge.net Mon May 3 14:14:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 3 14:14:25 2004 Subject: [Patches] [ python-Patches-945642 ] nonblocking i/o with ssl socket not working at all Message-ID: Patches item #945642, was opened at 2004-04-30 15:14 Message generated for change (Comment added) made by jesperjurcenoks You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Tino Lange (tinolange) Assigned to: Nobody/Anonymous (nobody) Summary: nonblocking i/o with ssl socket not working at all Initial Comment: Hi! Unfortunately the applied patches #675750 and #676472 (End of January by G. Talvola / B. Laurie / G. v. Rossum) for timeout-sockets in version 1.09 and 1.10 to Modules/_ssl.c broke the nonblocking SSL I/O. In fact sslobj.read() became *always* blocking - whether the underlying socket is blocking or nonblocking doesn't matter at all. In Python < 2.3 this has worked correctly, starting with 2.3, 2.3.x up to the current CVS it doesn't work anymore. Attached is a sample: 0) [ Preparation ] Please run stunnel on your machine to have a SSL service: /usr/sbin/stunnel -p /path/to/some/cert -f -d 4711 -r 4712 1) [ Server ] Please run the easy server.py 2) [Client showing the bug] Please run the client - and see how it hangs when you use Python2.3. Also attached is a diff against the current _ssl.c which shows how to correct that behaviour while also having the possibilities from the above timeout patches. I carefully tested it, please review and test yourself and finally check-in. If you have questions, please don't hesitate to ask me. I really would like to see that not only in 2.4, but also (if possible together with patch #909007 from S. Nicolary) in a 2.3.4 release. Otherwise 2.3.x wasn't at all able to handle nonblocking SSL connections properly. Thanks and cheers, Tino ---------------------------------------------------------------------- Comment By: Jesper ?JJ? Jurcenoks (jesperjurcenoks) Date: 2004-05-03 11:14 Message: Logged In: YES user_id=1033524 Hi Tino. I'll take your word for it. I though your ssl was based on the included SocketServer.py Cheers JJ ---------------------------------------------------------------------- Comment By: Tino Lange (tinolange) Date: 2004-05-02 01:40 Message: Logged In: YES user_id=212920 Hi JJ, I just scanned your SocketServer.py - and I really don't see how this class might help? You don't offer a better sslob.read() there, don't you? So how could it help (besides maybe making my example "server.py nicer). If I put a sslobj on top of a socket object it is blocking in 2.3, 2.3.x and higher. As proven in my example: Making the underlying socket nonblocking has no effect - because SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE from the SSL_xxx() functions in the C-Library do not break out of the while loop that was introduced in the above mentioned patches. Cheers, Tino ---------------------------------------------------------------------- Comment By: Jesper ?JJ? Jurcenoks (jesperjurcenoks) Date: 2004-05-01 19:46 Message: Logged In: YES user_id=1033524 Hi Tino. I just submitted a nonblocking SocketServer.py, it might just solve your problem. JJ ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 From noreply at sourceforge.net Mon May 3 19:36:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 3 19:36:14 2004 Subject: [Patches] [ python-Patches-947352 ] AF_PACKET Hardware address support in socket module Message-ID: Patches item #947352, was opened at 2004-05-03 19:36 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947352&group_id=5470 Category: Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jason Andryuk (thedeuce) Assigned to: Nobody/Anonymous (nobody) Summary: AF_PACKET Hardware address support in socket module Initial Comment: The current implementation of AF_PACKET only uses the device name and protocol options even when a 5-tuple of Device, Protocol, Packet Type, Hardware Type, and Hardware Address are supplied. I needed socket.sendto() to support sending to a Hardware Address, so this patch supports such functionality. The length check on the hardware address is hard coded to 8 since that is the value used in the sockaddr_ll struct. I'm not that familiar with the Python internals, so s->errorhandler() may not be the best call to make for an invalid hardware address length. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947352&group_id=5470 From noreply at sourceforge.net Mon May 3 20:56:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 3 20:56:30 2004 Subject: [Patches] [ python-Patches-947386 ] Fix for #947380 - sys.path is wrong on windows sometimes Message-ID: Patches item #947386, was opened at 2004-05-04 03:56 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947386&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Ori Berger (orib) Assigned to: Nobody/Anonymous (nobody) Summary: Fix for #947380 - sys.path is wrong on windows sometimes Initial Comment: Fixes http://python.org/sf/947380 - see details there. One line of code, one line of comment. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947386&group_id=5470 From noreply at sourceforge.net Tue May 4 01:42:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 01:43:07 2004 Subject: [Patches] [ python-Patches-941071 ] Replace if/elif chain with dispatch pattern in sre_compile Message-ID: Patches item #941071, was opened at 2004-04-23 19:07 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=941071&group_id=5470 Category: None Group: None >Status: Closed >Resolution: Rejected Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Gustavo Niemeyer (niemeyer) Summary: Replace if/elif chain with dispatch pattern in sre_compile Initial Comment: Use a dictionary to implement switch/case sematics instead of a long if/elif chain. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 00:42 Message: Logged In: YES user_id=80475 I can't see any way to fix the long if/elif chain without introducing switch/case. The dispatch idiom is too expensive in terms of function call overhead and data sharing costs. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 17:39 Message: Logged In: YES user_id=80475 It needs to be timed on very large patterns -- it will always do worse on very small patterns (due to the time to setup the dispatch table). I'm not satisfied with the patch the way it stands. Ideally, the dispatch table needs to be pre-computed at compile time. The trick is how to view the enclosing variables (passing them as arguments is slow; using globals is not especially clean or fast). If that were resolved, then this approach would just about always be faster (a dictioary lookup and function call versus a long chain of elifs). Any suggestions are welcome. ---------------------------------------------------------------------- Comment By: Gustavo Niemeyer (niemeyer) Date: 2004-04-28 16:36 Message: Logged In: YES user_id=7887 Raymond, have you done any performance tests showing that the proposed scheme is better? I've created a small test bed for the two patterns here and it looks like the current pattern is faster for small sets. Given the fact that this is a recursive pattern, and that I'm unsure about the kind of patterns which are most used, I'm a little bit uncomfortable in applying this. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-04-28 15:35 Message: Logged In: YES user_id=21627 Gustavo, what do you think? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=941071&group_id=5470 From noreply at sourceforge.net Tue May 4 01:47:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 01:47:20 2004 Subject: [Patches] [ python-Patches-947476 ] Apply freelist technique to empty lists Message-ID: Patches item #947476, was opened at 2004-05-04 00:47 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Armin Rigo (arigo) Summary: Apply freelist technique to empty lists Initial Comment: Saves malloc/free and reduces fragmentation for the ubiquitous operation of instantiating and freeing empty lists. Improves the timing of timeit.py "[]" by about a quarter. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 From noreply at sourceforge.net Tue May 4 10:34:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 10:34:27 2004 Subject: [Patches] [ python-Patches-947476 ] Apply freelist technique to empty lists Message-ID: Patches item #947476, was opened at 2004-05-04 05:47 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Armin Rigo (arigo) Summary: Apply freelist technique to empty lists Initial Comment: Saves malloc/free and reduces fragmentation for the ubiquitous operation of instantiating and freeing empty lists. Improves the timing of timeit.py "[]" by about a quarter. ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-04 14:34 Message: Logged In: YES user_id=4771 Why is the idea restricted to empty lists? A priori I don't see why empty lists are so common. On the other hand we should be able to save and restore the PyListObject part of any list using the same mecanism. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 From noreply at sourceforge.net Tue May 4 11:38:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 11:39:06 2004 Subject: [Patches] [ python-Patches-947476 ] Apply freelist technique to empty lists Message-ID: Patches item #947476, was opened at 2004-05-04 00:47 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Armin Rigo (arigo) Summary: Apply freelist technique to empty lists Initial Comment: Saves malloc/free and reduces fragmentation for the ubiquitous operation of instantiating and freeing empty lists. Improves the timing of timeit.py "[]" by about a quarter. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 10:38 Message: Logged In: YES user_id=80475 Attaching a revised patch that incorporates your suggestion about always being able to save/restore the PyListObject part. The a priori reason empty lists are so common is that most lists are built up from zero. Also, there is a lot a testing for a==[] and default instantiation in expressions like mydict.setdefault(k, []).append(v). A posteriori, it is easy to get a quick visual on the this, add a printf("*") on one side of the "if" and printf(".") on the other. Then run the test suite. The pattern of PyList_New(0) calls completely dominate the output. It looks like about 90% of the PyList_New calls are for size 0 (this pattern holds is not unique -- it holds across the whole test suite). The patch captures the benefit for the most common cases and avoids the code complexity and overhead of tracking multiple blocks of different sized memory blocks (that is what PyMalloc is supposed to be good at). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 09:34 Message: Logged In: YES user_id=4771 Why is the idea restricted to empty lists? A priori I don't see why empty lists are so common. On the other hand we should be able to save and restore the PyListObject part of any list using the same mecanism. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 From noreply at sourceforge.net Tue May 4 11:53:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 11:54:05 2004 Subject: [Patches] [ python-Patches-947476 ] Apply freelist technique to empty lists Message-ID: Patches item #947476, was opened at 2004-05-04 05:47 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Armin Rigo (arigo) Summary: Apply freelist technique to empty lists Initial Comment: Saves malloc/free and reduces fragmentation for the ubiquitous operation of instantiating and freeing empty lists. Improves the timing of timeit.py "[]" by about a quarter. ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-04 15:53 Message: Logged In: YES user_id=4771 Ok. I meant that lists that are still empty when they are destructed are probably not too common, although most lists are created empty. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 15:38 Message: Logged In: YES user_id=80475 Attaching a revised patch that incorporates your suggestion about always being able to save/restore the PyListObject part. The a priori reason empty lists are so common is that most lists are built up from zero. Also, there is a lot a testing for a==[] and default instantiation in expressions like mydict.setdefault(k, []).append(v). A posteriori, it is easy to get a quick visual on the this, add a printf("*") on one side of the "if" and printf(".") on the other. Then run the test suite. The pattern of PyList_New(0) calls completely dominate the output. It looks like about 90% of the PyList_New calls are for size 0 (this pattern holds is not unique -- it holds across the whole test suite). The patch captures the benefit for the most common cases and avoids the code complexity and overhead of tracking multiple blocks of different sized memory blocks (that is what PyMalloc is supposed to be good at). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 14:34 Message: Logged In: YES user_id=4771 Why is the idea restricted to empty lists? A priori I don't see why empty lists are so common. On the other hand we should be able to save and restore the PyListObject part of any list using the same mecanism. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 From noreply at sourceforge.net Tue May 4 11:58:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 11:59:08 2004 Subject: [Patches] [ python-Patches-947476 ] Apply freelist technique to empty lists Message-ID: Patches item #947476, was opened at 2004-05-04 01:47 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Armin Rigo (arigo) Summary: Apply freelist technique to empty lists Initial Comment: Saves malloc/free and reduces fragmentation for the ubiquitous operation of instantiating and freeing empty lists. Improves the timing of timeit.py "[]" by about a quarter. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-04 11:58 Message: Logged In: YES user_id=31435 FWIW, I like this. The list header could also be reclaimed from the freelist regardless of list size, a la if (free list isn't empty) { op = pop off the free list; do object initialization; if (size == 0) { finish & return; } else { op = PyObject_GC_New(...); if (op == NULL) error; } Note that PyMalloc isn't involved in allocating list *guts* regardless of list size. They're allocated with PyMem_MALLOC, which is the system malloc(). That's a murkier tradoff (if the guts get "too big" for PyMalloc later, the PyMalloc realloc function is a bit slower than calling the system realloc, since the former has the additional overhead of deducing that it needs to call the latter; OTOH, we no longer call a realloc function on every stinkin' append anymore). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 11:53 Message: Logged In: YES user_id=4771 Ok. I meant that lists that are still empty when they are destructed are probably not too common, although most lists are created empty. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 11:38 Message: Logged In: YES user_id=80475 Attaching a revised patch that incorporates your suggestion about always being able to save/restore the PyListObject part. The a priori reason empty lists are so common is that most lists are built up from zero. Also, there is a lot a testing for a==[] and default instantiation in expressions like mydict.setdefault(k, []).append(v). A posteriori, it is easy to get a quick visual on the this, add a printf("*") on one side of the "if" and printf(".") on the other. Then run the test suite. The pattern of PyList_New(0) calls completely dominate the output. It looks like about 90% of the PyList_New calls are for size 0 (this pattern holds is not unique -- it holds across the whole test suite). The patch captures the benefit for the most common cases and avoids the code complexity and overhead of tracking multiple blocks of different sized memory blocks (that is what PyMalloc is supposed to be good at). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 10:34 Message: Logged In: YES user_id=4771 Why is the idea restricted to empty lists? A priori I don't see why empty lists are so common. On the other hand we should be able to save and restore the PyListObject part of any list using the same mecanism. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 From noreply at sourceforge.net Tue May 4 12:02:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 12:02:20 2004 Subject: [Patches] [ python-Patches-947476 ] Apply freelist technique to empty lists Message-ID: Patches item #947476, was opened at 2004-05-04 00:47 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Armin Rigo (arigo) Summary: Apply freelist technique to empty lists Initial Comment: Saves malloc/free and reduces fragmentation for the ubiquitous operation of instantiating and freeing empty lists. Improves the timing of timeit.py "[]" by about a quarter. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 11:02 Message: Logged In: YES user_id=80475 Okay, I see. If that was an issue, it was fixed by incorporating your suggestion which means that the freelist will almost never be starved. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:58 Message: Logged In: YES user_id=31435 FWIW, I like this. The list header could also be reclaimed from the freelist regardless of list size, a la if (free list isn't empty) { op = pop off the free list; do object initialization; if (size == 0) { finish & return; } else { op = PyObject_GC_New(...); if (op == NULL) error; } Note that PyMalloc isn't involved in allocating list *guts* regardless of list size. They're allocated with PyMem_MALLOC, which is the system malloc(). That's a murkier tradoff (if the guts get "too big" for PyMalloc later, the PyMalloc realloc function is a bit slower than calling the system realloc, since the former has the additional overhead of deducing that it needs to call the latter; OTOH, we no longer call a realloc function on every stinkin' append anymore). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 10:53 Message: Logged In: YES user_id=4771 Ok. I meant that lists that are still empty when they are destructed are probably not too common, although most lists are created empty. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 10:38 Message: Logged In: YES user_id=80475 Attaching a revised patch that incorporates your suggestion about always being able to save/restore the PyListObject part. The a priori reason empty lists are so common is that most lists are built up from zero. Also, there is a lot a testing for a==[] and default instantiation in expressions like mydict.setdefault(k, []).append(v). A posteriori, it is easy to get a quick visual on the this, add a printf("*") on one side of the "if" and printf(".") on the other. Then run the test suite. The pattern of PyList_New(0) calls completely dominate the output. It looks like about 90% of the PyList_New calls are for size 0 (this pattern holds is not unique -- it holds across the whole test suite). The patch captures the benefit for the most common cases and avoids the code complexity and overhead of tracking multiple blocks of different sized memory blocks (that is what PyMalloc is supposed to be good at). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 09:34 Message: Logged In: YES user_id=4771 Why is the idea restricted to empty lists? A priori I don't see why empty lists are so common. On the other hand we should be able to save and restore the PyListObject part of any list using the same mecanism. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 From noreply at sourceforge.net Tue May 4 12:40:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 12:40:13 2004 Subject: [Patches] [ python-Patches-947476 ] Apply freelist technique to empty lists Message-ID: Patches item #947476, was opened at 2004-05-04 05:47 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Armin Rigo (arigo) Summary: Apply freelist technique to empty lists Initial Comment: Saves malloc/free and reduces fragmentation for the ubiquitous operation of instantiating and freeing empty lists. Improves the timing of timeit.py "[]" by about a quarter. ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-04 16:40 Message: Logged In: YES user_id=4771 Wouldn't it be nice to have a set of macros for free lists, something that could easily be dropped in any object implementation and that would do the right thing, e.g. cache the ob_type? When looking at integers, I can't help noticing that they reinitialize the ob_type field at each object creation. Would it pay to try to remove that? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 16:02 Message: Logged In: YES user_id=80475 Okay, I see. If that was an issue, it was fixed by incorporating your suggestion which means that the freelist will almost never be starved. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 15:58 Message: Logged In: YES user_id=31435 FWIW, I like this. The list header could also be reclaimed from the freelist regardless of list size, a la if (free list isn't empty) { op = pop off the free list; do object initialization; if (size == 0) { finish & return; } else { op = PyObject_GC_New(...); if (op == NULL) error; } Note that PyMalloc isn't involved in allocating list *guts* regardless of list size. They're allocated with PyMem_MALLOC, which is the system malloc(). That's a murkier tradoff (if the guts get "too big" for PyMalloc later, the PyMalloc realloc function is a bit slower than calling the system realloc, since the former has the additional overhead of deducing that it needs to call the latter; OTOH, we no longer call a realloc function on every stinkin' append anymore). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 15:53 Message: Logged In: YES user_id=4771 Ok. I meant that lists that are still empty when they are destructed are probably not too common, although most lists are created empty. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 15:38 Message: Logged In: YES user_id=80475 Attaching a revised patch that incorporates your suggestion about always being able to save/restore the PyListObject part. The a priori reason empty lists are so common is that most lists are built up from zero. Also, there is a lot a testing for a==[] and default instantiation in expressions like mydict.setdefault(k, []).append(v). A posteriori, it is easy to get a quick visual on the this, add a printf("*") on one side of the "if" and printf(".") on the other. Then run the test suite. The pattern of PyList_New(0) calls completely dominate the output. It looks like about 90% of the PyList_New calls are for size 0 (this pattern holds is not unique -- it holds across the whole test suite). The patch captures the benefit for the most common cases and avoids the code complexity and overhead of tracking multiple blocks of different sized memory blocks (that is what PyMalloc is supposed to be good at). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 14:34 Message: Logged In: YES user_id=4771 Why is the idea restricted to empty lists? A priori I don't see why empty lists are so common. On the other hand we should be able to save and restore the PyListObject part of any list using the same mecanism. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 From noreply at sourceforge.net Tue May 4 12:48:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 12:49:20 2004 Subject: [Patches] [ python-Patches-947476 ] Apply freelist technique to empty lists Message-ID: Patches item #947476, was opened at 2004-05-04 00:47 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Armin Rigo (arigo) Summary: Apply freelist technique to empty lists Initial Comment: Saves malloc/free and reduces fragmentation for the ubiquitous operation of instantiating and freeing empty lists. Improves the timing of timeit.py "[]" by about a quarter. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 11:48 Message: Logged In: YES user_id=80475 The needs of the different types vary enough to make a single macro set unlikely. Also, macros tend to hide what is going on under the hood making it difficult to discern genuine optimizations. Sound like there may be room for improvement for integers. Sticking with small steps though, I'll take lists today and leave integers for another patch on another day. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 11:40 Message: Logged In: YES user_id=4771 Wouldn't it be nice to have a set of macros for free lists, something that could easily be dropped in any object implementation and that would do the right thing, e.g. cache the ob_type? When looking at integers, I can't help noticing that they reinitialize the ob_type field at each object creation. Would it pay to try to remove that? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 11:02 Message: Logged In: YES user_id=80475 Okay, I see. If that was an issue, it was fixed by incorporating your suggestion which means that the freelist will almost never be starved. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:58 Message: Logged In: YES user_id=31435 FWIW, I like this. The list header could also be reclaimed from the freelist regardless of list size, a la if (free list isn't empty) { op = pop off the free list; do object initialization; if (size == 0) { finish & return; } else { op = PyObject_GC_New(...); if (op == NULL) error; } Note that PyMalloc isn't involved in allocating list *guts* regardless of list size. They're allocated with PyMem_MALLOC, which is the system malloc(). That's a murkier tradoff (if the guts get "too big" for PyMalloc later, the PyMalloc realloc function is a bit slower than calling the system realloc, since the former has the additional overhead of deducing that it needs to call the latter; OTOH, we no longer call a realloc function on every stinkin' append anymore). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 10:53 Message: Logged In: YES user_id=4771 Ok. I meant that lists that are still empty when they are destructed are probably not too common, although most lists are created empty. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 10:38 Message: Logged In: YES user_id=80475 Attaching a revised patch that incorporates your suggestion about always being able to save/restore the PyListObject part. The a priori reason empty lists are so common is that most lists are built up from zero. Also, there is a lot a testing for a==[] and default instantiation in expressions like mydict.setdefault(k, []).append(v). A posteriori, it is easy to get a quick visual on the this, add a printf("*") on one side of the "if" and printf(".") on the other. Then run the test suite. The pattern of PyList_New(0) calls completely dominate the output. It looks like about 90% of the PyList_New calls are for size 0 (this pattern holds is not unique -- it holds across the whole test suite). The patch captures the benefit for the most common cases and avoids the code complexity and overhead of tracking multiple blocks of different sized memory blocks (that is what PyMalloc is supposed to be good at). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 09:34 Message: Logged In: YES user_id=4771 Why is the idea restricted to empty lists? A priori I don't see why empty lists are so common. On the other hand we should be able to save and restore the PyListObject part of any list using the same mecanism. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 From noreply at sourceforge.net Tue May 4 13:08:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 13:08:52 2004 Subject: [Patches] [ python-Patches-947476 ] Apply freelist technique to empty lists Message-ID: Patches item #947476, was opened at 2004-05-04 01:47 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Armin Rigo (arigo) Summary: Apply freelist technique to empty lists Initial Comment: Saves malloc/free and reduces fragmentation for the ubiquitous operation of instantiating and freeing empty lists. Improves the timing of timeit.py "[]" by about a quarter. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-04 13:08 Message: Logged In: YES user_id=31435 WRT ints, the free list is chained together via abusing the ob_type field, and there's no other field in a PyIntObject guaranteed large enough to hold a pointer (or guaranteed to be pointer-aligned). ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 12:48 Message: Logged In: YES user_id=80475 The needs of the different types vary enough to make a single macro set unlikely. Also, macros tend to hide what is going on under the hood making it difficult to discern genuine optimizations. Sound like there may be room for improvement for integers. Sticking with small steps though, I'll take lists today and leave integers for another patch on another day. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 12:40 Message: Logged In: YES user_id=4771 Wouldn't it be nice to have a set of macros for free lists, something that could easily be dropped in any object implementation and that would do the right thing, e.g. cache the ob_type? When looking at integers, I can't help noticing that they reinitialize the ob_type field at each object creation. Would it pay to try to remove that? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 12:02 Message: Logged In: YES user_id=80475 Okay, I see. If that was an issue, it was fixed by incorporating your suggestion which means that the freelist will almost never be starved. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 11:58 Message: Logged In: YES user_id=31435 FWIW, I like this. The list header could also be reclaimed from the freelist regardless of list size, a la if (free list isn't empty) { op = pop off the free list; do object initialization; if (size == 0) { finish & return; } else { op = PyObject_GC_New(...); if (op == NULL) error; } Note that PyMalloc isn't involved in allocating list *guts* regardless of list size. They're allocated with PyMem_MALLOC, which is the system malloc(). That's a murkier tradoff (if the guts get "too big" for PyMalloc later, the PyMalloc realloc function is a bit slower than calling the system realloc, since the former has the additional overhead of deducing that it needs to call the latter; OTOH, we no longer call a realloc function on every stinkin' append anymore). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 11:53 Message: Logged In: YES user_id=4771 Ok. I meant that lists that are still empty when they are destructed are probably not too common, although most lists are created empty. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 11:38 Message: Logged In: YES user_id=80475 Attaching a revised patch that incorporates your suggestion about always being able to save/restore the PyListObject part. The a priori reason empty lists are so common is that most lists are built up from zero. Also, there is a lot a testing for a==[] and default instantiation in expressions like mydict.setdefault(k, []).append(v). A posteriori, it is easy to get a quick visual on the this, add a printf("*") on one side of the "if" and printf(".") on the other. Then run the test suite. The pattern of PyList_New(0) calls completely dominate the output. It looks like about 90% of the PyList_New calls are for size 0 (this pattern holds is not unique -- it holds across the whole test suite). The patch captures the benefit for the most common cases and avoids the code complexity and overhead of tracking multiple blocks of different sized memory blocks (that is what PyMalloc is supposed to be good at). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 10:34 Message: Logged In: YES user_id=4771 Why is the idea restricted to empty lists? A priori I don't see why empty lists are so common. On the other hand we should be able to save and restore the PyListObject part of any list using the same mecanism. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 From noreply at sourceforge.net Tue May 4 13:41:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 13:41:46 2004 Subject: [Patches] [ python-Patches-947476 ] Apply freelist technique to empty lists Message-ID: Patches item #947476, was opened at 2004-05-04 00:47 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Armin Rigo (arigo) Summary: Apply freelist technique to empty lists Initial Comment: Saves malloc/free and reduces fragmentation for the ubiquitous operation of instantiating and freeing empty lists. Improves the timing of timeit.py "[]" by about a quarter. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 12:41 Message: Logged In: YES user_id=80475 Attaching a new patch that incorporates both Tim and Armin's suggestions. The PyListObject portion is always reclaimed whenever there is room in the free list. The header is always re-used when available. This provides some benefit to lists of all sizes and it simplifies the patch a bit. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 12:08 Message: Logged In: YES user_id=31435 WRT ints, the free list is chained together via abusing the ob_type field, and there's no other field in a PyIntObject guaranteed large enough to hold a pointer (or guaranteed to be pointer-aligned). ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 11:48 Message: Logged In: YES user_id=80475 The needs of the different types vary enough to make a single macro set unlikely. Also, macros tend to hide what is going on under the hood making it difficult to discern genuine optimizations. Sound like there may be room for improvement for integers. Sticking with small steps though, I'll take lists today and leave integers for another patch on another day. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 11:40 Message: Logged In: YES user_id=4771 Wouldn't it be nice to have a set of macros for free lists, something that could easily be dropped in any object implementation and that would do the right thing, e.g. cache the ob_type? When looking at integers, I can't help noticing that they reinitialize the ob_type field at each object creation. Would it pay to try to remove that? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 11:02 Message: Logged In: YES user_id=80475 Okay, I see. If that was an issue, it was fixed by incorporating your suggestion which means that the freelist will almost never be starved. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:58 Message: Logged In: YES user_id=31435 FWIW, I like this. The list header could also be reclaimed from the freelist regardless of list size, a la if (free list isn't empty) { op = pop off the free list; do object initialization; if (size == 0) { finish & return; } else { op = PyObject_GC_New(...); if (op == NULL) error; } Note that PyMalloc isn't involved in allocating list *guts* regardless of list size. They're allocated with PyMem_MALLOC, which is the system malloc(). That's a murkier tradoff (if the guts get "too big" for PyMalloc later, the PyMalloc realloc function is a bit slower than calling the system realloc, since the former has the additional overhead of deducing that it needs to call the latter; OTOH, we no longer call a realloc function on every stinkin' append anymore). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 10:53 Message: Logged In: YES user_id=4771 Ok. I meant that lists that are still empty when they are destructed are probably not too common, although most lists are created empty. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 10:38 Message: Logged In: YES user_id=80475 Attaching a revised patch that incorporates your suggestion about always being able to save/restore the PyListObject part. The a priori reason empty lists are so common is that most lists are built up from zero. Also, there is a lot a testing for a==[] and default instantiation in expressions like mydict.setdefault(k, []).append(v). A posteriori, it is easy to get a quick visual on the this, add a printf("*") on one side of the "if" and printf(".") on the other. Then run the test suite. The pattern of PyList_New(0) calls completely dominate the output. It looks like about 90% of the PyList_New calls are for size 0 (this pattern holds is not unique -- it holds across the whole test suite). The patch captures the benefit for the most common cases and avoids the code complexity and overhead of tracking multiple blocks of different sized memory blocks (that is what PyMalloc is supposed to be good at). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 09:34 Message: Logged In: YES user_id=4771 Why is the idea restricted to empty lists? A priori I don't see why empty lists are so common. On the other hand we should be able to save and restore the PyListObject part of any list using the same mecanism. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 From ERXGFXTK at sina.com.cn Tue May 4 13:50:42 2004 From: ERXGFXTK at sina.com.cn (Percy Brooks) Date: Tue May 4 16:13:33 2004 Subject: [Patches] no more lying in applications - buy a degree from an accredited university here Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040504/f30e663d/attachment.html From noreply at sourceforge.net Tue May 4 18:13:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 18:15:59 2004 Subject: [Patches] [ python-Patches-851736 ] urllib2 CacheFTPHandler doesn't work on multiple dirs Message-ID: Patches item #851736, was opened at 2003-11-30 21:51 Message generated for change (Comment added) made by jjlee You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Submitted By: John J Lee (jjlee) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2 CacheFTPHandler doesn't work on multiple dirs Initial Comment: This is a fix for bug 738973. Where are functional tests supposed to go? Is the __name__ == "__main__" block the official place? The urllib2 functional tests are quite out of date, thanks to shifting URLs, so I'd like to fix them, but need to know where to do it. ---------------------------------------------------------------------- >Comment By: John J Lee (jjlee) Date: 2004-05-04 23:13 Message: Logged In: YES user_id=261020 For now at least, should probably just stick the test in the if __name__ == "__main__" block of urllib2 without its unittest wrapper. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2003-11-30 21:53 Message: Logged In: YES user_id=261020 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 From noreply at sourceforge.net Tue May 4 20:28:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 20:28:48 2004 Subject: [Patches] [ python-Patches-947476 ] Apply freelist technique to empty lists Message-ID: Patches item #947476, was opened at 2004-05-04 00:47 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Armin Rigo (arigo) Summary: Apply freelist technique to empty lists Initial Comment: Saves malloc/free and reduces fragmentation for the ubiquitous operation of instantiating and freeing empty lists. Improves the timing of timeit.py "[]" by about a quarter. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 19:28 Message: Logged In: YES user_id=80475 Any further ideas or should I go ahead and put this one in? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 12:41 Message: Logged In: YES user_id=80475 Attaching a new patch that incorporates both Tim and Armin's suggestions. The PyListObject portion is always reclaimed whenever there is room in the free list. The header is always re-used when available. This provides some benefit to lists of all sizes and it simplifies the patch a bit. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 12:08 Message: Logged In: YES user_id=31435 WRT ints, the free list is chained together via abusing the ob_type field, and there's no other field in a PyIntObject guaranteed large enough to hold a pointer (or guaranteed to be pointer-aligned). ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 11:48 Message: Logged In: YES user_id=80475 The needs of the different types vary enough to make a single macro set unlikely. Also, macros tend to hide what is going on under the hood making it difficult to discern genuine optimizations. Sound like there may be room for improvement for integers. Sticking with small steps though, I'll take lists today and leave integers for another patch on another day. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 11:40 Message: Logged In: YES user_id=4771 Wouldn't it be nice to have a set of macros for free lists, something that could easily be dropped in any object implementation and that would do the right thing, e.g. cache the ob_type? When looking at integers, I can't help noticing that they reinitialize the ob_type field at each object creation. Would it pay to try to remove that? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 11:02 Message: Logged In: YES user_id=80475 Okay, I see. If that was an issue, it was fixed by incorporating your suggestion which means that the freelist will almost never be starved. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:58 Message: Logged In: YES user_id=31435 FWIW, I like this. The list header could also be reclaimed from the freelist regardless of list size, a la if (free list isn't empty) { op = pop off the free list; do object initialization; if (size == 0) { finish & return; } else { op = PyObject_GC_New(...); if (op == NULL) error; } Note that PyMalloc isn't involved in allocating list *guts* regardless of list size. They're allocated with PyMem_MALLOC, which is the system malloc(). That's a murkier tradoff (if the guts get "too big" for PyMalloc later, the PyMalloc realloc function is a bit slower than calling the system realloc, since the former has the additional overhead of deducing that it needs to call the latter; OTOH, we no longer call a realloc function on every stinkin' append anymore). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 10:53 Message: Logged In: YES user_id=4771 Ok. I meant that lists that are still empty when they are destructed are probably not too common, although most lists are created empty. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 10:38 Message: Logged In: YES user_id=80475 Attaching a revised patch that incorporates your suggestion about always being able to save/restore the PyListObject part. The a priori reason empty lists are so common is that most lists are built up from zero. Also, there is a lot a testing for a==[] and default instantiation in expressions like mydict.setdefault(k, []).append(v). A posteriori, it is easy to get a quick visual on the this, add a printf("*") on one side of the "if" and printf(".") on the other. Then run the test suite. The pattern of PyList_New(0) calls completely dominate the output. It looks like about 90% of the PyList_New calls are for size 0 (this pattern holds is not unique -- it holds across the whole test suite). The patch captures the benefit for the most common cases and avoids the code complexity and overhead of tracking multiple blocks of different sized memory blocks (that is what PyMalloc is supposed to be good at). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 09:34 Message: Logged In: YES user_id=4771 Why is the idea restricted to empty lists? A priori I don't see why empty lists are so common. On the other hand we should be able to save and restore the PyListObject part of any list using the same mecanism. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 From noreply at sourceforge.net Tue May 4 21:13:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 21:14:10 2004 Subject: [Patches] [ python-Patches-947476 ] Apply freelist technique to empty lists Message-ID: Patches item #947476, was opened at 2004-05-04 01:47 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open >Resolution: Accepted Priority: 5 Submitted By: Raymond Hettinger (rhettinger) >Assigned to: Raymond Hettinger (rhettinger) Summary: Apply freelist technique to empty lists Initial Comment: Saves malloc/free and reduces fragmentation for the ubiquitous operation of instantiating and freeing empty lists. Improves the timing of timeit.py "[]" by about a quarter. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-04 21:13 Message: Logged In: YES user_id=31435 Marked Accepted & assigned back to you. If Armin wants more changes, he (or you, or I, or ...) can fiddle it later. What's here is fine! ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 20:28 Message: Logged In: YES user_id=80475 Any further ideas or should I go ahead and put this one in? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 13:41 Message: Logged In: YES user_id=80475 Attaching a new patch that incorporates both Tim and Armin's suggestions. The PyListObject portion is always reclaimed whenever there is room in the free list. The header is always re-used when available. This provides some benefit to lists of all sizes and it simplifies the patch a bit. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 13:08 Message: Logged In: YES user_id=31435 WRT ints, the free list is chained together via abusing the ob_type field, and there's no other field in a PyIntObject guaranteed large enough to hold a pointer (or guaranteed to be pointer-aligned). ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 12:48 Message: Logged In: YES user_id=80475 The needs of the different types vary enough to make a single macro set unlikely. Also, macros tend to hide what is going on under the hood making it difficult to discern genuine optimizations. Sound like there may be room for improvement for integers. Sticking with small steps though, I'll take lists today and leave integers for another patch on another day. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 12:40 Message: Logged In: YES user_id=4771 Wouldn't it be nice to have a set of macros for free lists, something that could easily be dropped in any object implementation and that would do the right thing, e.g. cache the ob_type? When looking at integers, I can't help noticing that they reinitialize the ob_type field at each object creation. Would it pay to try to remove that? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 12:02 Message: Logged In: YES user_id=80475 Okay, I see. If that was an issue, it was fixed by incorporating your suggestion which means that the freelist will almost never be starved. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 11:58 Message: Logged In: YES user_id=31435 FWIW, I like this. The list header could also be reclaimed from the freelist regardless of list size, a la if (free list isn't empty) { op = pop off the free list; do object initialization; if (size == 0) { finish & return; } else { op = PyObject_GC_New(...); if (op == NULL) error; } Note that PyMalloc isn't involved in allocating list *guts* regardless of list size. They're allocated with PyMem_MALLOC, which is the system malloc(). That's a murkier tradoff (if the guts get "too big" for PyMalloc later, the PyMalloc realloc function is a bit slower than calling the system realloc, since the former has the additional overhead of deducing that it needs to call the latter; OTOH, we no longer call a realloc function on every stinkin' append anymore). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 11:53 Message: Logged In: YES user_id=4771 Ok. I meant that lists that are still empty when they are destructed are probably not too common, although most lists are created empty. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 11:38 Message: Logged In: YES user_id=80475 Attaching a revised patch that incorporates your suggestion about always being able to save/restore the PyListObject part. The a priori reason empty lists are so common is that most lists are built up from zero. Also, there is a lot a testing for a==[] and default instantiation in expressions like mydict.setdefault(k, []).append(v). A posteriori, it is easy to get a quick visual on the this, add a printf("*") on one side of the "if" and printf(".") on the other. Then run the test suite. The pattern of PyList_New(0) calls completely dominate the output. It looks like about 90% of the PyList_New calls are for size 0 (this pattern holds is not unique -- it holds across the whole test suite). The patch captures the benefit for the most common cases and avoids the code complexity and overhead of tracking multiple blocks of different sized memory blocks (that is what PyMalloc is supposed to be good at). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 10:34 Message: Logged In: YES user_id=4771 Why is the idea restricted to empty lists? A priori I don't see why empty lists are so common. On the other hand we should be able to save and restore the PyListObject part of any list using the same mecanism. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 From KTVFFVQEPEIU at msn.com Tue May 4 23:30:34 2004 From: KTVFFVQEPEIU at msn.com (Clarence Conner) Date: Tue May 4 21:47:15 2004 Subject: [Patches] Front Page 2004, Office 2004 XP and More $60 Fre.e Shipping choral ducat dissociate Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040504/21b6dcfb/attachment-0001.html From euiaqhhspazrf at hotmail.com Tue May 4 23:49:57 2004 From: euiaqhhspazrf at hotmail.com (Maude Tackett) Date: Tue May 4 22:12:34 2004 Subject: [Patches] Front Page 2004, Office 2004 XP and More $60 Fre.e Shipping stromberg danbury suspensor Message-ID: <885543587047.i3LGcKC57670291@peoria9.edgewise29atavistic.com> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040505/2d7d1df4/attachment.html From euiaqhhspazrf at hotmail.com Tue May 4 23:49:57 2004 From: euiaqhhspazrf at hotmail.com (Maude Tackett) Date: Tue May 4 22:35:49 2004 Subject: [Patches] Front Page 2004, Office 2004 XP and More $60 Fre.e Shipping stromberg danbury suspensor Message-ID: <885543587047.i3LGcKC57670291@peoria9.edgewise29atavistic.com> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040505/2d7d1df4/attachment-0003.html From Patrick at meetingalove.com Wed May 5 17:09:29 2004 From: Patrick at meetingalove.com (Mindy Nielsen) Date: Tue May 4 23:55:33 2004 Subject: [Patches] YOU you could be john holmes Message-ID: Interested in a blind-date that has been pre-arranged by a mutual friend? Click here to accept the invitation: http://thesitefordating.com/confirm/?oc=50795989 Click here if you do not wish to be invited again: http://lovingbetteronline.com/remove/?oc=50795989 danorthitenikko hatteras canoga simulcast posterior penis newcomer protege almond tact paramedic bucharest scutum ernie sauce flammable differ miltonic alliance advisory bater herr beecham gymnastic crewmen drastic differentiable firepower duress asiatic xparatroopdatabase cutaneous brewery irrelevant cactus dewar bellatrix susan dissonant dallas polygon placeable photometry conic conklin nib bien From noreply at sourceforge.net Wed May 5 01:38:57 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 01:39:02 2004 Subject: [Patches] [ python-Patches-947476 ] Apply freelist technique to empty lists Message-ID: Patches item #947476, was opened at 2004-05-04 00:47 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 Category: Core (C code) Group: Python 2.4 >Status: Closed Resolution: Accepted Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Raymond Hettinger (rhettinger) Summary: Apply freelist technique to empty lists Initial Comment: Saves malloc/free and reduces fragmentation for the ubiquitous operation of instantiating and freeing empty lists. Improves the timing of timeit.py "[]" by about a quarter. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-05 00:38 Message: Logged In: YES user_id=80475 Thanks. Checked-in as Objects/llistobject.c 2.204 ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 20:13 Message: Logged In: YES user_id=31435 Marked Accepted & assigned back to you. If Armin wants more changes, he (or you, or I, or ...) can fiddle it later. What's here is fine! ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 19:28 Message: Logged In: YES user_id=80475 Any further ideas or should I go ahead and put this one in? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 12:41 Message: Logged In: YES user_id=80475 Attaching a new patch that incorporates both Tim and Armin's suggestions. The PyListObject portion is always reclaimed whenever there is room in the free list. The header is always re-used when available. This provides some benefit to lists of all sizes and it simplifies the patch a bit. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 12:08 Message: Logged In: YES user_id=31435 WRT ints, the free list is chained together via abusing the ob_type field, and there's no other field in a PyIntObject guaranteed large enough to hold a pointer (or guaranteed to be pointer-aligned). ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 11:48 Message: Logged In: YES user_id=80475 The needs of the different types vary enough to make a single macro set unlikely. Also, macros tend to hide what is going on under the hood making it difficult to discern genuine optimizations. Sound like there may be room for improvement for integers. Sticking with small steps though, I'll take lists today and leave integers for another patch on another day. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 11:40 Message: Logged In: YES user_id=4771 Wouldn't it be nice to have a set of macros for free lists, something that could easily be dropped in any object implementation and that would do the right thing, e.g. cache the ob_type? When looking at integers, I can't help noticing that they reinitialize the ob_type field at each object creation. Would it pay to try to remove that? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 11:02 Message: Logged In: YES user_id=80475 Okay, I see. If that was an issue, it was fixed by incorporating your suggestion which means that the freelist will almost never be starved. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:58 Message: Logged In: YES user_id=31435 FWIW, I like this. The list header could also be reclaimed from the freelist regardless of list size, a la if (free list isn't empty) { op = pop off the free list; do object initialization; if (size == 0) { finish & return; } else { op = PyObject_GC_New(...); if (op == NULL) error; } Note that PyMalloc isn't involved in allocating list *guts* regardless of list size. They're allocated with PyMem_MALLOC, which is the system malloc(). That's a murkier tradoff (if the guts get "too big" for PyMalloc later, the PyMalloc realloc function is a bit slower than calling the system realloc, since the former has the additional overhead of deducing that it needs to call the latter; OTOH, we no longer call a realloc function on every stinkin' append anymore). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 10:53 Message: Logged In: YES user_id=4771 Ok. I meant that lists that are still empty when they are destructed are probably not too common, although most lists are created empty. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-04 10:38 Message: Logged In: YES user_id=80475 Attaching a revised patch that incorporates your suggestion about always being able to save/restore the PyListObject part. The a priori reason empty lists are so common is that most lists are built up from zero. Also, there is a lot a testing for a==[] and default instantiation in expressions like mydict.setdefault(k, []).append(v). A posteriori, it is easy to get a quick visual on the this, add a printf("*") on one side of the "if" and printf(".") on the other. Then run the test suite. The pattern of PyList_New(0) calls completely dominate the output. It looks like about 90% of the PyList_New calls are for size 0 (this pattern holds is not unique -- it holds across the whole test suite). The patch captures the benefit for the most common cases and avoids the code complexity and overhead of tracking multiple blocks of different sized memory blocks (that is what PyMalloc is supposed to be good at). ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-04 09:34 Message: Logged In: YES user_id=4771 Why is the idea restricted to empty lists? A priori I don't see why empty lists are so common. On the other hand we should be able to save and restore the PyListObject part of any list using the same mecanism. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947476&group_id=5470 From noreply at sourceforge.net Wed May 5 08:23:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 08:24:17 2004 Subject: [Patches] [ python-Patches-851736 ] urllib2 CacheFTPHandler doesn't work on multiple dirs Message-ID: Patches item #851736, was opened at 2003-12-01 08:51 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 Category: Library (Lib) Group: None Status: Open Resolution: None >Priority: 6 Submitted By: John J Lee (jjlee) >Assigned to: Mark Hammond (mhammond) Summary: urllib2 CacheFTPHandler doesn't work on multiple dirs Initial Comment: This is a fix for bug 738973. Where are functional tests supposed to go? Is the __name__ == "__main__" block the official place? The urllib2 functional tests are quite out of date, thanks to shifting URLs, so I'd like to fix them, but need to know where to do it. ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-05-05 22:23 Message: Logged In: YES user_id=14198 The password is being dropped from the cache - I can't see how this could possibly hurt though. This all looks reasonable to me, and fails (then works) as described. I will check this in in a couple of days to make sure noone objects. I won't check the test code in though - ideally, it should go in the main test directory, but only if the 'network' resource is enabled - but I don't see that as a blocking issue. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-05 08:13 Message: Logged In: YES user_id=261020 For now at least, should probably just stick the test in the if __name__ == "__main__" block of urllib2 without its unittest wrapper. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2003-12-01 08:53 Message: Logged In: YES user_id=261020 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 From noreply at sourceforge.net Wed May 5 13:13:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 13:13:45 2004 Subject: [Patches] [ python-Patches-948614 ] Linux signals during threads Message-ID: Patches item #948614, was opened at 2004-05-05 13:13 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=948614&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: Linux signals during threads Initial Comment: The Linux implementation of pthreads in the 2.0 through 2.4 (called LinuxThreads ) has one notable place where it diverges from the pthreads standard. A signal caused by a thread will only trigger the signal handler on that thread, and it is not seen or able to be handled by any other thread. The LinuxThreads package was first created in 1996 and is only now starting to be replaced by the new Linux 2.6 threading facility (called NPTL.) The non-standard LinuxThreads behavior falls afoul with Python's thread semantics, since it assumes that only the main thread can set or act on signal handlers. For the pthreads environment, it enforces this by blocking signals on all threads except the main thread. On LinuxThreads, this causes these signals to become ignored and lost, sometimes causing deadlock. The enclosed patch tests for LinuxThread's specific peculiarities in the configure script and adjusts the handling of thread creation and signal handling in order to give threads the documented Python thread and signal behavior. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=948614&group_id=5470 From noreply at sourceforge.net Wed May 5 13:43:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 13:43:52 2004 Subject: [Patches] [ python-Patches-851736 ] urllib2 CacheFTPHandler doesn't work on multiple dirs Message-ID: Patches item #851736, was opened at 2003-11-30 21:51 Message generated for change (Comment added) made by jjlee You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 6 Submitted By: John J Lee (jjlee) Assigned to: Mark Hammond (mhammond) Summary: urllib2 CacheFTPHandler doesn't work on multiple dirs Initial Comment: This is a fix for bug 738973. Where are functional tests supposed to go? Is the __name__ == "__main__" block the official place? The urllib2 functional tests are quite out of date, thanks to shifting URLs, so I'd like to fix them, but need to know where to do it. ---------------------------------------------------------------------- >Comment By: John J Lee (jjlee) Date: 2004-05-05 18:43 Message: Logged In: YES user_id=261020 Thanks Mark FWIW, the idea of having the test in __main__ is that it's a functional test, not a unit test. But when I raised this issue on python-dev (thread ends at message whose URL is below), I left more confused than when I arrived, so feel free to put them where you like ;-) http://mail.python.org/pipermail/python-dev/2003-December/040861.html ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-05 13:23 Message: Logged In: YES user_id=14198 The password is being dropped from the cache - I can't see how this could possibly hurt though. This all looks reasonable to me, and fails (then works) as described. I will check this in in a couple of days to make sure noone objects. I won't check the test code in though - ideally, it should go in the main test directory, but only if the 'network' resource is enabled - but I don't see that as a blocking issue. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-04 23:13 Message: Logged In: YES user_id=261020 For now at least, should probably just stick the test in the if __name__ == "__main__" block of urllib2 without its unittest wrapper. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2003-11-30 21:53 Message: Logged In: YES user_id=261020 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 From noreply at sourceforge.net Wed May 5 14:09:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 14:10:30 2004 Subject: [Patches] [ python-Patches-851736 ] urllib2 CacheFTPHandler doesn't work on multiple dirs Message-ID: Patches item #851736, was opened at 2003-11-30 22:51 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 6 Submitted By: John J Lee (jjlee) Assigned to: Mark Hammond (mhammond) Summary: urllib2 CacheFTPHandler doesn't work on multiple dirs Initial Comment: This is a fix for bug 738973. Where are functional tests supposed to go? Is the __name__ == "__main__" block the official place? The urllib2 functional tests are quite out of date, thanks to shifting URLs, so I'd like to fix them, but need to know where to do it. ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-05-05 20:09 Message: Logged In: YES user_id=11105 The problem with test code in the __main__ block is that py2exe, for example, concludes that the unittest module is needed. If that is done, imo it would be better to write the main block in this way, so that at least this is compiled away with -O or -OO: if __name__ == '__main__': if __debug__: import unittest # test code... ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-05 19:43 Message: Logged In: YES user_id=261020 Thanks Mark FWIW, the idea of having the test in __main__ is that it's a functional test, not a unit test. But when I raised this issue on python-dev (thread ends at message whose URL is below), I left more confused than when I arrived, so feel free to put them where you like ;-) http://mail.python.org/pipermail/python-dev/2003-December/040861.html ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-05 14:23 Message: Logged In: YES user_id=14198 The password is being dropped from the cache - I can't see how this could possibly hurt though. This all looks reasonable to me, and fails (then works) as described. I will check this in in a couple of days to make sure noone objects. I won't check the test code in though - ideally, it should go in the main test directory, but only if the 'network' resource is enabled - but I don't see that as a blocking issue. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-05 00:13 Message: Logged In: YES user_id=261020 For now at least, should probably just stick the test in the if __name__ == "__main__" block of urllib2 without its unittest wrapper. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2003-11-30 22:53 Message: Logged In: YES user_id=261020 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 From tkvdvqph at msn.com Thu May 6 19:56:40 2004 From: tkvdvqph at msn.com (Bert Hackett) Date: Wed May 5 17:39:37 2004 Subject: [Patches] International Open Dance Grand Prix Festival Awards Competition June 2004 in Italy ADVCHANGE OF E-MAIL ADDRESS Message-ID: We would like to Invite you in Italy on 17th to 23th July 2004 to participate to Dance Awards Europe 2004 the greatest dance meeting competition in europe this season! International Event Competition for Ballet-Contemporary-Modern-Hip Hop-Latin-Swing-Folk- Tap Schools/ Companies and Dance-Theatrical Educational Institutes...like no other! all info on our web site < http://web.tiscali.it/worldancefestival/ > - Attention there is change of e-mail address, organizer new address please send us your info bye FAX fax. 0039054751422 Just in a few days you will receive some advertising materials from us, the new brochures and posters of the Dance Awards Prize 2004 Italy 2004 Full Name:_____________________________________________ School/Company Name: _________________________________ Telephone:_____________________________________________ Fax:___________________________________________________ Email Address:_________________________________________* REQUIRED FIELD Shipping Address:______________________________________ City:____________________ State/Province:______________ Country:_________________________ZIP/Postal:____________ web info in lingua italiana FAX fax. 0039054751422 e.mail Richiedete via il Pacchetto gratuito con le Informazioni e la video anche per via fax al: 054751422 MODULO DA COMPILARE Nome :____________________________________________________ Scuola/Compagnia Nome:____________________________________ Tel e Fax:________________________________________________ indirizzo E.mail :_________________________________________ indirizzo postale:_________________________________________ Citta:____________________ Stato/Provincia:________________ Codice Postale:____________ to remove please send a blank message to From noreply at sourceforge.net Wed May 5 21:36:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 21:36:26 2004 Subject: [Patches] [ python-Patches-940026 ] Explain 'in' keyword as it is first used Message-ID: Patches item #940026, was opened at 2004-04-22 15:35 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=940026&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Accepted Priority: 7 Submitted By: Gerrit Holl (gerrit) Assigned to: Martin v. L?wis (loewis) Summary: Explain 'in' keyword as it is first used Initial Comment: Because of a question on a mailing list, I tried to find where the 'in' keyword, for containment testing, was first introduced in the Python tutorial. It turned out not to be, but to be silently introduced in section 4.7.1 (default argument values), and very briefly explained in 5.7 (more on conditions). This patch adds a (very) brief note at the place where the 'in' keyword is first used for containment testing (4.7.1) explaining what it means. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-06 03:36 Message: Logged In: YES user_id=21627 Thanks for the patch. Applied as tut.tex 1.196.8.19 and 1.227. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=940026&group_id=5470 From RFKNDZCS at msn.com Tue May 4 23:07:20 2004 From: RFKNDZCS at msn.com (Salvador Leslie) Date: Wed May 5 21:42:39 2004 Subject: [Patches] Adobe Photoshop Illustrator and More $60 w/Fre.e Shipping adjectival chad sibley Message-ID: <855690r2t690$44141186$qq0c7246@Nancyc92z9ccv4v> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040505/5eb2a1a7/attachment-0001.html From noreply at sourceforge.net Wed May 5 21:42:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 21:42:59 2004 Subject: [Patches] [ python-Patches-944110 ] urllib2 authentication mishandles empty pass bugfix 944082 Message-ID: Patches item #944110, was opened at 2004-04-29 02:52 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=944110&group_id=5470 Category: Library (Lib) Group: None >Status: Closed >Resolution: Accepted Priority: 7 Submitted By: sc0rp (yangir) Assigned to: Martin v. L?wis (loewis) Summary: urllib2 authentication mishandles empty pass bugfix 944082 Initial Comment: If example.org requires authentication, then following code: host = 'example.org' user = 'testuser' password = '' url = 'http://%s/' % host authInfo = urllib2.HTTPPasswordMgrWithDefaultRealm() authInfo.add_password( None, host, user, password ) authHandler = urllib2.HTTPBasicAuthHandler( authInfo ) opener = urllib2.build_opener( authHandler ) urlFile = opener.open( url ) print urlFile.read() will die by throwing HTTPError 401: File "/usr/lib/python2.3/urllib2.py", line 419, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 401: Authorization Required even if authenticating with 'testuser' and empty password is valid. Empty password is mishandled (i.e. authentication with empty password string is ignored) in AbstractBasicAuthHandler.retry_http_basic_auth def retry_http_basic_auth(self, host, req, realm): user,pw = self.passwd.find_user_password(realm, host) if pw: [...] It can be fixed by changing: if pw: to if pw is not None: Patch attached. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-06 03:42 Message: Logged In: YES user_id=21627 Thanks for the patch. Applied as urllib2.py 1.53.6.5, 1.65 NEWS 1.831.4.103 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=944110&group_id=5470 From jodie.mcNeilkc at challenger.ca Thu May 6 13:33:06 2004 From: jodie.mcNeilkc at challenger.ca (Jodie McNeil) Date: Thu May 6 07:29:01 2004 Subject: [Patches] Gotham Alerts: Monthly stock tips Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040506/7210a145/attachment.html From noreply at sourceforge.net Thu May 6 13:07:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 13:07:32 2004 Subject: [Patches] [ python-Patches-949332 ] synchronous signals blocked in pthreads Message-ID: Patches item #949332, was opened at 2004-05-06 13:07 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: synchronous signals blocked in pthreads Initial Comment: In a pthreads environment, signals sent asynchronously from things like kill() or the terminal driver are sent to the process and can be handled by any thread. Signals that are sent synchronously from the instruction stream (signals like SIGSEGV, SIGFPE, or SIGILL) are only delivered to the thread that executed the instruction. The current Python threads, when implemented via pthreads, blocks these synchronous signals and so thread that generates one can hang the entire process. The attached patch will remove some signals the list of signals to be blocked. The list of signals to be removed correspond to signals that can be generated synchronously. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 From noreply at sourceforge.net Thu May 6 14:11:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 14:11:49 2004 Subject: [Patches] [ python-Patches-949332 ] synchronous signals blocked in pthreads Message-ID: Patches item #949332, was opened at 2004-05-06 13:07 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None >Priority: 7 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: synchronous signals blocked in pthreads Initial Comment: In a pthreads environment, signals sent asynchronously from things like kill() or the terminal driver are sent to the process and can be handled by any thread. Signals that are sent synchronously from the instruction stream (signals like SIGSEGV, SIGFPE, or SIGILL) are only delivered to the thread that executed the instruction. The current Python threads, when implemented via pthreads, blocks these synchronous signals and so thread that generates one can hang the entire process. The attached patch will remove some signals the list of signals to be blocked. The list of signals to be removed correspond to signals that can be generated synchronously. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-06 14:11 Message: Logged In: YES user_id=31435 Boosted the priority in the hopes this will get looked at before 2.3.4. Unfortunately, I'm not in a position to test a pthreads patch. I'll note that the good comments about *why* this change is being entertained would be far more useful in the code than in the patch comment here. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 From VDVKUOHPH at mail.uni-greifswald.de Fri May 7 02:38:36 2004 From: VDVKUOHPH at mail.uni-greifswald.de (Traci Pina) Date: Fri May 7 01:39:40 2004 Subject: [Patches] 56( this is the real deal Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040507/e0f20725/attachment.html From architect at canyoufeeltheforce.com Fri May 7 11:43:02 2004 From: architect at canyoufeeltheforce.com (Distinguishes P. Kebob) Date: Fri May 7 11:53:54 2004 Subject: [Patches] Save on cds, Patches! Message-ID: <100001c43449$bf253896$6272c84e@canyoufeeltheforce.com> Hiya! :) Live your life, do your work, then take your hat. Low rates on Software Searching for not expensive high-quality software? Our site might be just what you need. http://blossomhead.shnoem.biz/OE017/?affiliate_id=233712&campaign_id=601 And more! We can ship worldwide. Hell is the highest reward that the devil can offer you for being a servant of his. Hatred is the vice of narrow souls they feed it with all their littleness, and make it the pretext of base tyrannies. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040507/a563a09d/attachment.html From noreply at sourceforge.net Fri May 7 19:45:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 19:45:45 2004 Subject: [Patches] [ python-Patches-946373 ] do not add directory of sys.argv[0] into sys.path Message-ID: Patches item #946373, was opened at 2004-05-02 05:51 Message generated for change (Comment added) made by isandler You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=946373&group_id=5470 Category: Core (C code) Group: None Status: Open Resolution: None Priority: 5 Submitted By: wrobell (wrobell) Assigned to: Nobody/Anonymous (nobody) Summary: do not add directory of sys.argv[0] into sys.path Initial Comment: Python adds magically directory of sys.argv[0] into sys.path, i.e. >>> import sys >>> sys.path ['', '/usr/lib/python23.zip', '/usr/share/python2.3', '/usr/share/python2.3/plat-linux2', '/usr/share/python2.3/lib-tk', '/usr/lib/python2.3/lib-dynload', '/usr/lib/python2.3/site-packages', '/usr/lib/python2.3/site-packages/gtk-2.0', '/usr/share/python2.3/site-packages'] where '' (or /usr/bin when executed script is in /usr/bin directory, etc.) is added automatically. It is useful in many circumstances but fails when name conflict occurs. For example, create getpass.py or pdb.py scripts which import getpass and pdb modules. Script names conflict with modules names and modules are not going to be imported because path to the scripts is appended into sys.path, so a script is imported instead of a module. The solutions: 1. User of script with conflicting name (i.e. getpass.py or timeit.py) can set PYTHONPATH to system library path, i.e. /usr/lib/python2.3. 2. User can modify the script to delete site.path[0]. 3. User can rename the script. 4. Python can be modified to not add magically directory of sys.argv[0]. The 1. is a tricky and not intuitive and quite funny: set PYTHONPATH to system library path to import system module (and only in specific circumstances). ;-P The 2. is a dirty hack: hey, we gonna import system module, ain't it? The 3. is, IMHO, not acceptable because there is more than 200 python system modules, more in the future and user cannot be forced to maintain script names blacklist. The 4. is only acceptable, IMHO. It makes python more inconvenient but gives no trouble when names conflict occurs. Moreover, fourth solution makes python more standard with other languages behaviour, i.e. one has to set CLASSPATH to load Java classes. Maybe there is another solution, but... Patch attached. ---------------------------------------------------------------------- Comment By: Ilya Sandler (isandler) Date: 2004-05-07 16:45 Message: Logged In: YES user_id=971153 Would not this cause serious backward compatibility problems?? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=946373&group_id=5470 From edematous at pieceofpiss.com Sat May 8 23:16:53 2004 From: edematous at pieceofpiss.com (Whitewashes M. Video) Date: Sun May 9 00:57:55 2004 Subject: [Patches] Software cheap, Patches ! Message-ID: <111101c43574$f21b89a4$f7f37bbb@pieceofpiss.com> Of whom do I have the honour? :) Forgive your enemies, but never forget their names. Low rates on Software Looking for cheap high-quality software? We might be just what you need. http://bridgeman.oevier.biz/OE017/?affiliate_id=233712&campaign_id=601 And more! We can ship worldwide. Any sufficiently advanced bureaucracy is indistinguishable from molasses. Throw a lucky man into the sea, and he will come up with a fish in his mouth. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040508/cf867d8f/attachment.html From noreply at sourceforge.net Sun May 9 04:58:44 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 9 04:59:09 2004 Subject: [Patches] [ python-Patches-851736 ] urllib2 CacheFTPHandler doesn't work on multiple dirs Message-ID: Patches item #851736, was opened at 2003-12-01 08:51 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 6 Submitted By: John J Lee (jjlee) Assigned to: Mark Hammond (mhammond) Summary: urllib2 CacheFTPHandler doesn't work on multiple dirs Initial Comment: This is a fix for bug 738973. Where are functional tests supposed to go? Is the __name__ == "__main__" block the official place? The urllib2 functional tests are quite out of date, thanks to shifting URLs, so I'd like to fix them, but need to know where to do it. ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-05-09 18:58 Message: Logged In: YES user_id=14198 Actually, wouldn't it be better to store 'dirs' directly in the dict? ie, instead of: + key = user, host, port, '/'.join(dirs) just do: + key = user, host, port, dirs Then the only assumption we make is that it is hashable. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-06 04:09 Message: Logged In: YES user_id=11105 The problem with test code in the __main__ block is that py2exe, for example, concludes that the unittest module is needed. If that is done, imo it would be better to write the main block in this way, so that at least this is compiled away with -O or -OO: if __name__ == '__main__': if __debug__: import unittest # test code... ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-06 03:43 Message: Logged In: YES user_id=261020 Thanks Mark FWIW, the idea of having the test in __main__ is that it's a functional test, not a unit test. But when I raised this issue on python-dev (thread ends at message whose URL is below), I left more confused than when I arrived, so feel free to put them where you like ;-) http://mail.python.org/pipermail/python-dev/2003-December/040861.html ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-05 22:23 Message: Logged In: YES user_id=14198 The password is being dropped from the cache - I can't see how this could possibly hurt though. This all looks reasonable to me, and fails (then works) as described. I will check this in in a couple of days to make sure noone objects. I won't check the test code in though - ideally, it should go in the main test directory, but only if the 'network' resource is enabled - but I don't see that as a blocking issue. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-05 08:13 Message: Logged In: YES user_id=261020 For now at least, should probably just stick the test in the if __name__ == "__main__" block of urllib2 without its unittest wrapper. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2003-12-01 08:53 Message: Logged In: YES user_id=261020 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 From abstention at no-matter-what.com Fri May 7 10:22:01 2004 From: abstention at no-matter-what.com (Fanatics D. Socialize) Date: Sun May 9 07:56:21 2004 Subject: [Patches] Special offers on software, Patches. Message-ID: <011101c4343e$cad3fb73$0d0687d7@no-matter-what.com> Hello I am sure that since I have had the full use of my reason, nobody has heard me laugh. Low rates on Software Looking for not expensive high-quality software? We might be just what you need. http://sagaie.fosraw.biz/OE017/?affiliate_id=233712&campaign_id=601 And more! We can ship worldwide. For me, hard work represents the supreme luxury of life. Courage is the capacity to confirm what can be imagined. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040507/9f192eaa/attachment.html From noreply at sourceforge.net Sun May 9 09:07:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 9 09:07:31 2004 Subject: [Patches] [ python-Patches-851736 ] urllib2 CacheFTPHandler doesn't work on multiple dirs Message-ID: Patches item #851736, was opened at 2003-11-30 21:51 Message generated for change (Comment added) made by jjlee You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 6 Submitted By: John J Lee (jjlee) Assigned to: Mark Hammond (mhammond) Summary: urllib2 CacheFTPHandler doesn't work on multiple dirs Initial Comment: This is a fix for bug 738973. Where are functional tests supposed to go? Is the __name__ == "__main__" block the official place? The urllib2 functional tests are quite out of date, thanks to shifting URLs, so I'd like to fix them, but need to know where to do it. ---------------------------------------------------------------------- >Comment By: John J Lee (jjlee) Date: 2004-05-09 14:07 Message: Logged In: YES user_id=261020 OK. dirs is a list, though, so it should be: + key = user, host, port, tuple(dirs) ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-09 09:58 Message: Logged In: YES user_id=14198 Actually, wouldn't it be better to store 'dirs' directly in the dict? ie, instead of: + key = user, host, port, '/'.join(dirs) just do: + key = user, host, port, dirs Then the only assumption we make is that it is hashable. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-05 19:09 Message: Logged In: YES user_id=11105 The problem with test code in the __main__ block is that py2exe, for example, concludes that the unittest module is needed. If that is done, imo it would be better to write the main block in this way, so that at least this is compiled away with -O or -OO: if __name__ == '__main__': if __debug__: import unittest # test code... ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-05 18:43 Message: Logged In: YES user_id=261020 Thanks Mark FWIW, the idea of having the test in __main__ is that it's a functional test, not a unit test. But when I raised this issue on python-dev (thread ends at message whose URL is below), I left more confused than when I arrived, so feel free to put them where you like ;-) http://mail.python.org/pipermail/python-dev/2003-December/040861.html ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-05 13:23 Message: Logged In: YES user_id=14198 The password is being dropped from the cache - I can't see how this could possibly hurt though. This all looks reasonable to me, and fails (then works) as described. I will check this in in a couple of days to make sure noone objects. I won't check the test code in though - ideally, it should go in the main test directory, but only if the 'network' resource is enabled - but I don't see that as a blocking issue. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-04 23:13 Message: Logged In: YES user_id=261020 For now at least, should probably just stick the test in the if __name__ == "__main__" block of urllib2 without its unittest wrapper. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2003-11-30 21:53 Message: Logged In: YES user_id=261020 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 From noreply at sourceforge.net Mon May 10 03:38:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 03:38:30 2004 Subject: [Patches] [ python-Patches-851736 ] urllib2 CacheFTPHandler doesn't work on multiple dirs Message-ID: Patches item #851736, was opened at 2003-12-01 08:51 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 Category: Library (Lib) Group: None >Status: Closed >Resolution: Fixed Priority: 6 Submitted By: John J Lee (jjlee) Assigned to: Mark Hammond (mhammond) Summary: urllib2 CacheFTPHandler doesn't work on multiple dirs Initial Comment: This is a fix for bug 738973. Where are functional tests supposed to go? Is the __name__ == "__main__" block the official place? The urllib2 functional tests are quite out of date, thanks to shifting URLs, so I'd like to fix them, but need to know where to do it. ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-05-10 17:38 Message: Logged In: YES user_id=14198 Re the tuple() - I think your original patch is better. I haven't done anything with the tests, but I encourage you to pursue a patch to the test suite when the network resource is enabled, and basically ignore other implications until they happen :) These tests would only go in 2.4. Fixed on 2.3 branch: Checking in urllib2.py; new revision: 1.53.6.6; previous revision: 1.53.6.5 Fixed on trunk: new revision: 1.66; previous revision: 1.65 ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-09 23:07 Message: Logged In: YES user_id=261020 OK. dirs is a list, though, so it should be: + key = user, host, port, tuple(dirs) ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-09 18:58 Message: Logged In: YES user_id=14198 Actually, wouldn't it be better to store 'dirs' directly in the dict? ie, instead of: + key = user, host, port, '/'.join(dirs) just do: + key = user, host, port, dirs Then the only assumption we make is that it is hashable. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-06 04:09 Message: Logged In: YES user_id=11105 The problem with test code in the __main__ block is that py2exe, for example, concludes that the unittest module is needed. If that is done, imo it would be better to write the main block in this way, so that at least this is compiled away with -O or -OO: if __name__ == '__main__': if __debug__: import unittest # test code... ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-06 03:43 Message: Logged In: YES user_id=261020 Thanks Mark FWIW, the idea of having the test in __main__ is that it's a functional test, not a unit test. But when I raised this issue on python-dev (thread ends at message whose URL is below), I left more confused than when I arrived, so feel free to put them where you like ;-) http://mail.python.org/pipermail/python-dev/2003-December/040861.html ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-05 22:23 Message: Logged In: YES user_id=14198 The password is being dropped from the cache - I can't see how this could possibly hurt though. This all looks reasonable to me, and fails (then works) as described. I will check this in in a couple of days to make sure noone objects. I won't check the test code in though - ideally, it should go in the main test directory, but only if the 'network' resource is enabled - but I don't see that as a blocking issue. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-05 08:13 Message: Logged In: YES user_id=261020 For now at least, should probably just stick the test in the if __name__ == "__main__" block of urllib2 without its unittest wrapper. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2003-12-01 08:53 Message: Logged In: YES user_id=261020 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=851736&group_id=5470 From noreply at sourceforge.net Mon May 10 06:48:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 06:48:49 2004 Subject: [Patches] [ python-Patches-943898 ] A simple 3-4% speed-up for PCs Message-ID: Patches item #943898, was opened at 2004-04-28 18:33 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=943898&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Armin Rigo (arigo) Assigned to: Nobody/Anonymous (nobody) Summary: A simple 3-4% speed-up for PCs Initial Comment: The result of a few experiments looking at the assembler produced by gcc for eval_frame(): * on PCs, reading the arguments as an unsigned short instead of two bytes is a good win. * oparg is more "local" with this patch: its value doesn't need to be saved across an iteration of the main loop, allowing it to live in a register only. * added an explicit "case STOP_CODE:" so that the switch starts at 0 instead of 1 -- that's one instruction less with gcc. * it seems not to pay off to move reading the argument at the start of each case of an operation that expects one, even though it removes the unpredictable branch "if (HAS_ARG(op))". This patch should be timed on other platforms to make sure that it doesn't slow things down. If it does, then only reading the arg as an unsigned short could be checked in -- it is compilation-conditional over the fact that shorts are 2 bytes in little endian order. By the way, anyone knows why 'stack_pointer' isn't a 'register' local? I bet it would make a difference on PowerPC, for example, with compilers that care about this keyword. ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-10 10:48 Message: Logged In: YES user_id=4771 The short trick might be a bit fragile. For example, the current patch would incorrectly use it on machines where unaligned accesses are forbidden. I isolated the other issue I talked about (making stack_pointer a register variable) in a separate patch. This patch alone is clearly safe. It should give a bit of speed-up on any machine but it definitely gives 5% on PCs with gcc by forcing the two most important local variables into specific registers. (If someone knows the corresponding syntax for other compilers, it can be added in the #if.) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 23:45 Message: Logged In: YES user_id=80475 With MSVC++ 6.0 under WinME on a Pentium III, there is no change in timing (measurements accurate within 0.25%): I wonder if the speedup from retrieving the unsigned short is offset by alignment penalties when the starting address is odd. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-04-28 21:02 Message: Logged In: YES user_id=4771 stack_pointer isn't a register because its address is taken at two places. This is a really bad idea for optimization. Instead of &stack_pointer, we should do: PyObject **sp = stack_pointer; ... use &sp ... stack_pointer = sp; I'm pretty sure this simple change along with a 'register' declaration of stack_pointer gives a good speed-up on all architectures with plenty of registers. For PCs I've experimented with forcing one or two locals into specific registers, with the gcc syntax asm("esi"), asm("ebx"), etc. Forcing stack_pointer and next_instr gives another 3-4% of improvement. Next step is to see if this can be done with #if's for common compilers beside gcc. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=943898&group_id=5470 From noreply at sourceforge.net Mon May 10 10:26:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 10:26:35 2004 Subject: [Patches] [ python-Patches-943898 ] A simple 3-4% speed-up for PCs Message-ID: Patches item #943898, was opened at 2004-04-28 18:33 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=943898&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Armin Rigo (arigo) Assigned to: Nobody/Anonymous (nobody) Summary: A simple 3-4% speed-up for PCs Initial Comment: The result of a few experiments looking at the assembler produced by gcc for eval_frame(): * on PCs, reading the arguments as an unsigned short instead of two bytes is a good win. * oparg is more "local" with this patch: its value doesn't need to be saved across an iteration of the main loop, allowing it to live in a register only. * added an explicit "case STOP_CODE:" so that the switch starts at 0 instead of 1 -- that's one instruction less with gcc. * it seems not to pay off to move reading the argument at the start of each case of an operation that expects one, even though it removes the unpredictable branch "if (HAS_ARG(op))". This patch should be timed on other platforms to make sure that it doesn't slow things down. If it does, then only reading the arg as an unsigned short could be checked in -- it is compilation-conditional over the fact that shorts are 2 bytes in little endian order. By the way, anyone knows why 'stack_pointer' isn't a 'register' local? I bet it would make a difference on PowerPC, for example, with compilers that care about this keyword. ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-10 14:26 Message: Logged In: YES user_id=4771 Tested on a MacOSX box, the patch also gives a 5% speed-up there. Allowing stack_pointer to be in a register is a very good idea. (all tests with Pystone) ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-10 10:48 Message: Logged In: YES user_id=4771 The short trick might be a bit fragile. For example, the current patch would incorrectly use it on machines where unaligned accesses are forbidden. I isolated the other issue I talked about (making stack_pointer a register variable) in a separate patch. This patch alone is clearly safe. It should give a bit of speed-up on any machine but it definitely gives 5% on PCs with gcc by forcing the two most important local variables into specific registers. (If someone knows the corresponding syntax for other compilers, it can be added in the #if.) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 23:45 Message: Logged In: YES user_id=80475 With MSVC++ 6.0 under WinME on a Pentium III, there is no change in timing (measurements accurate within 0.25%): I wonder if the speedup from retrieving the unsigned short is offset by alignment penalties when the starting address is odd. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-04-28 21:02 Message: Logged In: YES user_id=4771 stack_pointer isn't a register because its address is taken at two places. This is a really bad idea for optimization. Instead of &stack_pointer, we should do: PyObject **sp = stack_pointer; ... use &sp ... stack_pointer = sp; I'm pretty sure this simple change along with a 'register' declaration of stack_pointer gives a good speed-up on all architectures with plenty of registers. For PCs I've experimented with forcing one or two locals into specific registers, with the gcc syntax asm("esi"), asm("ebx"), etc. Forcing stack_pointer and next_instr gives another 3-4% of improvement. Next step is to see if this can be done with #if's for common compilers beside gcc. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=943898&group_id=5470 From noreply at sourceforge.net Mon May 10 21:10:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 21:10:47 2004 Subject: [Patches] [ python-Patches-945642 ] nonblocking i/o with ssl socket not working at all Message-ID: Patches item #945642, was opened at 2004-04-30 15:14 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Tino Lange (tinolange) Assigned to: Nobody/Anonymous (nobody) Summary: nonblocking i/o with ssl socket not working at all Initial Comment: Hi! Unfortunately the applied patches #675750 and #676472 (End of January by G. Talvola / B. Laurie / G. v. Rossum) for timeout-sockets in version 1.09 and 1.10 to Modules/_ssl.c broke the nonblocking SSL I/O. In fact sslobj.read() became *always* blocking - whether the underlying socket is blocking or nonblocking doesn't matter at all. In Python < 2.3 this has worked correctly, starting with 2.3, 2.3.x up to the current CVS it doesn't work anymore. Attached is a sample: 0) [ Preparation ] Please run stunnel on your machine to have a SSL service: /usr/sbin/stunnel -p /path/to/some/cert -f -d 4711 -r 4712 1) [ Server ] Please run the easy server.py 2) [Client showing the bug] Please run the client - and see how it hangs when you use Python2.3. Also attached is a diff against the current _ssl.c which shows how to correct that behaviour while also having the possibilities from the above timeout patches. I carefully tested it, please review and test yourself and finally check-in. If you have questions, please don't hesitate to ask me. I really would like to see that not only in 2.4, but also (if possible together with patch #909007 from S. Nicolary) in a 2.3.4 release. Otherwise 2.3.x wasn't at all able to handle nonblocking SSL connections properly. Thanks and cheers, Tino ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-05-10 18:10 Message: Logged In: YES user_id=357491 A style nit: put 'else' statements on their own line. See PEP 7 (http:// www.python.org/peps/pep-0007.html) on C code guidelines. But I am no network programmer so I have no clue if the solution is good. ---------------------------------------------------------------------- Comment By: Jesper ?JJ? Jurcenoks (jesperjurcenoks) Date: 2004-05-03 11:14 Message: Logged In: YES user_id=1033524 Hi Tino. I'll take your word for it. I though your ssl was based on the included SocketServer.py Cheers JJ ---------------------------------------------------------------------- Comment By: Tino Lange (tinolange) Date: 2004-05-02 01:40 Message: Logged In: YES user_id=212920 Hi JJ, I just scanned your SocketServer.py - and I really don't see how this class might help? You don't offer a better sslob.read() there, don't you? So how could it help (besides maybe making my example "server.py nicer). If I put a sslobj on top of a socket object it is blocking in 2.3, 2.3.x and higher. As proven in my example: Making the underlying socket nonblocking has no effect - because SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE from the SSL_xxx() functions in the C-Library do not break out of the while loop that was introduced in the above mentioned patches. Cheers, Tino ---------------------------------------------------------------------- Comment By: Jesper ?JJ? Jurcenoks (jesperjurcenoks) Date: 2004-05-01 19:46 Message: Logged In: YES user_id=1033524 Hi Tino. I just submitted a nonblocking SocketServer.py, it might just solve your problem. JJ ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 From amarillo at postaldigital.com Mon May 10 15:40:59 2004 From: amarillo at postaldigital.com (Descry O. Cusp) Date: Mon May 10 22:36:13 2004 Subject: [Patches] Unbeleivable prices on software, Patches. Message-ID: <010001c436c6$a22a224d$2e10e88a@postaldigital.com> Which side of the bed did you get out of this morning, then? :)))) The optimist proclaims that we live in the best of all possible worlds, and the pessimist fears this is true. Low rates on Software Looking for not expensive high-quality software? We might be just what you need. http://siderographist.livere.biz/OE017/?affiliate_id=233712&campaign_id=601 And more! We ship worldwide. It is not human nature we should accuse but the despicable conventions that pervert it. The test of tolerance comes when we are in a majority the test of courage comes when we are in a minority. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040510/7d0a1eba/attachment.html From noreply at sourceforge.net Tue May 11 03:53:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 03:53:36 2004 Subject: [Patches] [ python-Patches-945642 ] nonblocking i/o with ssl socket not working at all Message-ID: Patches item #945642, was opened at 2004-05-01 00:14 Message generated for change (Comment added) made by tinolange You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Tino Lange (tinolange) Assigned to: Nobody/Anonymous (nobody) Summary: nonblocking i/o with ssl socket not working at all Initial Comment: Hi! Unfortunately the applied patches #675750 and #676472 (End of January by G. Talvola / B. Laurie / G. v. Rossum) for timeout-sockets in version 1.09 and 1.10 to Modules/_ssl.c broke the nonblocking SSL I/O. In fact sslobj.read() became *always* blocking - whether the underlying socket is blocking or nonblocking doesn't matter at all. In Python < 2.3 this has worked correctly, starting with 2.3, 2.3.x up to the current CVS it doesn't work anymore. Attached is a sample: 0) [ Preparation ] Please run stunnel on your machine to have a SSL service: /usr/sbin/stunnel -p /path/to/some/cert -f -d 4711 -r 4712 1) [ Server ] Please run the easy server.py 2) [Client showing the bug] Please run the client - and see how it hangs when you use Python2.3. Also attached is a diff against the current _ssl.c which shows how to correct that behaviour while also having the possibilities from the above timeout patches. I carefully tested it, please review and test yourself and finally check-in. If you have questions, please don't hesitate to ask me. I really would like to see that not only in 2.4, but also (if possible together with patch #909007 from S. Nicolary) in a 2.3.4 release. Otherwise 2.3.x wasn't at all able to handle nonblocking SSL connections properly. Thanks and cheers, Tino ---------------------------------------------------------------------- >Comment By: Tino Lange (tinolange) Date: 2004-05-11 09:53 Message: Logged In: YES user_id=212920 Thanks for the hint. I guess --in the context of _ssl.c -- you only complain about the lines: } else sockstate = SOCKET_OPERATION_OK; or? Is that right? I think all the other else's are consistent - at least with the rest of the file - and some others I looked into (for example _bsddb.c). Or do you think that I shall reformat all the other else's in the file? Also those that are not from me? Really following the PEP it should look like: if (foo) { ... } else if (otherfoo) { ... } else { ... } instead of if (foo) { ... } else if (otherfoo) { ... } else { ... } everywhere, or? ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-05-11 03:10 Message: Logged In: YES user_id=357491 A style nit: put 'else' statements on their own line. See PEP 7 (http:// www.python.org/peps/pep-0007.html) on C code guidelines. But I am no network programmer so I have no clue if the solution is good. ---------------------------------------------------------------------- Comment By: Jesper ?JJ? Jurcenoks (jesperjurcenoks) Date: 2004-05-03 20:14 Message: Logged In: YES user_id=1033524 Hi Tino. I'll take your word for it. I though your ssl was based on the included SocketServer.py Cheers JJ ---------------------------------------------------------------------- Comment By: Tino Lange (tinolange) Date: 2004-05-02 10:40 Message: Logged In: YES user_id=212920 Hi JJ, I just scanned your SocketServer.py - and I really don't see how this class might help? You don't offer a better sslob.read() there, don't you? So how could it help (besides maybe making my example "server.py nicer). If I put a sslobj on top of a socket object it is blocking in 2.3, 2.3.x and higher. As proven in my example: Making the underlying socket nonblocking has no effect - because SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE from the SSL_xxx() functions in the C-Library do not break out of the while loop that was introduced in the above mentioned patches. Cheers, Tino ---------------------------------------------------------------------- Comment By: Jesper ?JJ? Jurcenoks (jesperjurcenoks) Date: 2004-05-02 04:46 Message: Logged In: YES user_id=1033524 Hi Tino. I just submitted a nonblocking SocketServer.py, it might just solve your problem. JJ ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 From noreply at sourceforge.net Tue May 11 06:24:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 06:24:44 2004 Subject: [Patches] [ python-Patches-700858 ] Replacing and deleting files in a zipfile archive. Message-ID: Patches item #700858, was opened at 2003-03-10 17:08 Message generated for change (Comment added) made by davidfraser You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=700858&group_id=5470 Category: Library (Lib) Group: Python 2.2.x Status: Open Resolution: None Priority: 5 Submitted By: Nev Delap (nevdelap) Assigned to: Nobody/Anonymous (nobody) Summary: Replacing and deleting files in a zipfile archive. Initial Comment: Addition of replace, replacestr and delete methods into zipfile.py. ---------------------------------------------------------------------- Comment By: David Fraser (davidfraser) Date: 2004-05-11 12:24 Message: Logged In: YES user_id=221678 This patch works fine on Python 2.3, if you just add the line del self.NameToInfo[name] before the return in delete (and the first HUNK with author info isn't applied) Unfortunately I can't add a file for some reason... nevdelap, could you update the bug for Python 2.3? ---------------------------------------------------------------------- Comment By: Nev Delap (nevdelap) Date: 2003-03-10 17:19 Message: Logged In: YES user_id=730416 OK, so after refreshing it finally decided to show the files I'd added. ---------------------------------------------------------------------- Comment By: Nev Delap (nevdelap) Date: 2003-03-10 17:16 Message: Logged In: YES user_id=730416 . ---------------------------------------------------------------------- Comment By: Nev Delap (nevdelap) Date: 2003-03-10 17:15 Message: Logged In: YES user_id=730416 The file upload say "Successful" but the file isn't listed!? I've tried it several times and yes I've checked the checkbox. ---------------------------------------------------------------------- Comment By: Nev Delap (nevdelap) Date: 2003-03-10 17:14 Message: Logged In: YES user_id=730416 The file upload say "Successful" but the file isn't listed!? I've tried it several times and yes I've checked the checkbox. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=700858&group_id=5470 From ufuhtlvq at uni-koeln.de Tue May 11 07:29:43 2004 From: ufuhtlvq at uni-koeln.de (Collin Swanson) Date: Tue May 11 06:32:39 2004 Subject: [Patches] Pick of the year beats revenue forecast Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040511/438dad0f/attachment.html From noreply at sourceforge.net Tue May 11 09:47:57 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 09:48:08 2004 Subject: [Patches] [ python-Patches-949332 ] synchronous signals blocked in pthreads Message-ID: Patches item #949332, was opened at 2004-05-06 13:07 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: synchronous signals blocked in pthreads Initial Comment: In a pthreads environment, signals sent asynchronously from things like kill() or the terminal driver are sent to the process and can be handled by any thread. Signals that are sent synchronously from the instruction stream (signals like SIGSEGV, SIGFPE, or SIGILL) are only delivered to the thread that executed the instruction. The current Python threads, when implemented via pthreads, blocks these synchronous signals and so thread that generates one can hang the entire process. The attached patch will remove some signals the list of signals to be blocked. The list of signals to be removed correspond to signals that can be generated synchronously. ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 09:47 Message: Logged In: YES user_id=6380 As Tim mentioned, this is a possible stopgap fix for bug 949332. While many objections have been raised, I still think this is a safer approach than some of the more advanced suggestions. (Too bad it doesn't solve my own, related problem, which is that SIGTERM is ignored by default in a process started from a Python thread.) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 14:11 Message: Logged In: YES user_id=31435 Boosted the priority in the hopes this will get looked at before 2.3.4. Unfortunately, I'm not in a position to test a pthreads patch. I'll note that the good comments about *why* this change is being entertained would be far more useful in the code than in the patch comment here. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 From noreply at sourceforge.net Tue May 11 10:01:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 10:01:43 2004 Subject: [Patches] [ python-Patches-468347 ] mask signals for non-main pthreads Message-ID: Patches item #468347, was opened at 2001-10-05 12:58 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=468347&group_id=5470 Category: Modules Group: None Status: Closed Resolution: Accepted Priority: 5 Submitted By: Jason Lowe (jasonlowe) Assigned to: Guido van Rossum (gvanrossum) Summary: mask signals for non-main pthreads Initial Comment: This patch updates Python/thread_pthread.h to mask all signals for any thread created. This will keep all signals masked for any thread that isn't the initial thread. For Solaris and Linux, the two platforms I was able to test it on, it solves bug #465673 (pthreads need signal protection) and probably will solve bug #219772 (Interactive InterPreter+ Thread -> core dump at exit). I'd be great if this could get some testing on other platforms, especially HP-UX pre 11.00 and post 11.00, as I had to make some guesses for the DCE thread case. AIX is also a concern as I saw some mention of using sigthreadmask() as a pthread_sigmask() equivalent, but this patch doesn't use sigthreadmask(). I don't have access to AIX. Note that thread_pthread.h.orig in this patch is the unmodified version from the Python2.1 release. ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 10:01 Message: Logged In: YES user_id=6380 This patch has caused more problems than it solves, IMO! Have a look at bug 756924: http://sourceforge.net/tracker/index.php?func=detail&aid=756924&group_id=5470&atid=105470 Jason, if you're still around, would you mind looking into an alternative fix for 465673 as suggested there? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2001-10-12 17:49 Message: Logged In: YES user_id=6380 As you said, the patch was out of date. Normally that would be enough to ask you to resubmit it relative to the current CVS, but it was too easy. I've checked this in as thread_pthread.h rev 2.33 now, so that it gets maximal exposure when the 2.2b1 release goes out next week. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=468347&group_id=5470 From noreply at sourceforge.net Tue May 11 10:07:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 10:07:51 2004 Subject: [Patches] [ python-Patches-951915 ] fix bug in StringIO.truncate - length not changed Message-ID: Patches item #951915, was opened at 2004-05-11 16:07 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=951915&group_id=5470 Category: Library (Lib) Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: David Fraser (davidfraser) Assigned to: Nobody/Anonymous (nobody) Summary: fix bug in StringIO.truncate - length not changed Initial Comment: If truncate() is called on a StringIO object, the length is not changed, so that seek(0, 2) calls will go beyond the correct end of the file. This patch adds a line to update the length, and a test to the test method that checks that it works. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=951915&group_id=5470 From noreply at sourceforge.net Tue May 11 10:57:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 10:57:57 2004 Subject: [Patches] [ python-Patches-700858 ] Replacing and deleting files in a zipfile archive. Message-ID: Patches item #700858, was opened at 2003-03-10 17:08 Message generated for change (Comment added) made by davidfraser You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=700858&group_id=5470 Category: Library (Lib) Group: Python 2.2.x Status: Open Resolution: None Priority: 5 Submitted By: Nev Delap (nevdelap) Assigned to: Nobody/Anonymous (nobody) Summary: Replacing and deleting files in a zipfile archive. Initial Comment: Addition of replace, replacestr and delete methods into zipfile.py. ---------------------------------------------------------------------- Comment By: David Fraser (davidfraser) Date: 2004-05-11 16:57 Message: Logged In: YES user_id=221678 Note that there is a problem with using StringIO files as zipfiles with this as they fail to truncate properly - see bug 951915 ---------------------------------------------------------------------- Comment By: David Fraser (davidfraser) Date: 2004-05-11 12:24 Message: Logged In: YES user_id=221678 This patch works fine on Python 2.3, if you just add the line del self.NameToInfo[name] before the return in delete (and the first HUNK with author info isn't applied) Unfortunately I can't add a file for some reason... nevdelap, could you update the bug for Python 2.3? ---------------------------------------------------------------------- Comment By: Nev Delap (nevdelap) Date: 2003-03-10 17:19 Message: Logged In: YES user_id=730416 OK, so after refreshing it finally decided to show the files I'd added. ---------------------------------------------------------------------- Comment By: Nev Delap (nevdelap) Date: 2003-03-10 17:16 Message: Logged In: YES user_id=730416 . ---------------------------------------------------------------------- Comment By: Nev Delap (nevdelap) Date: 2003-03-10 17:15 Message: Logged In: YES user_id=730416 The file upload say "Successful" but the file isn't listed!? I've tried it several times and yes I've checked the checkbox. ---------------------------------------------------------------------- Comment By: Nev Delap (nevdelap) Date: 2003-03-10 17:14 Message: Logged In: YES user_id=730416 The file upload say "Successful" but the file isn't listed!? I've tried it several times and yes I've checked the checkbox. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=700858&group_id=5470 From intuitive at deep-shit.com Tue May 11 11:14:08 2004 From: intuitive at deep-shit.com (Atmosphere V. Catalina) Date: Tue May 11 11:26:29 2004 Subject: [Patches] Dont spend more on software, Patches. Message-ID: <000001c4376a$c23fab31$57016249@deep-shit.com> Good morning, campers :) Things forbidden have a secret charm. Low rates on Software Searching for cheap high-quality software? Our site might be just what you need. http://alcoholimeter.livere.biz/OE017/?affiliate_id=233712&campaign_id=601 And more! We can ship worldwide. When anything goes, it's women who lose. The best blood will at some time get into a fool or a mosquito. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040511/3ca7d287/attachment.html From noreply at sourceforge.net Tue May 11 13:17:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 13:17:48 2004 Subject: [Patches] [ python-Patches-945642 ] nonblocking i/o with ssl socket not working at all Message-ID: Patches item #945642, was opened at 2004-04-30 15:14 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Tino Lange (tinolange) Assigned to: Nobody/Anonymous (nobody) Summary: nonblocking i/o with ssl socket not working at all Initial Comment: Hi! Unfortunately the applied patches #675750 and #676472 (End of January by G. Talvola / B. Laurie / G. v. Rossum) for timeout-sockets in version 1.09 and 1.10 to Modules/_ssl.c broke the nonblocking SSL I/O. In fact sslobj.read() became *always* blocking - whether the underlying socket is blocking or nonblocking doesn't matter at all. In Python < 2.3 this has worked correctly, starting with 2.3, 2.3.x up to the current CVS it doesn't work anymore. Attached is a sample: 0) [ Preparation ] Please run stunnel on your machine to have a SSL service: /usr/sbin/stunnel -p /path/to/some/cert -f -d 4711 -r 4712 1) [ Server ] Please run the easy server.py 2) [Client showing the bug] Please run the client - and see how it hangs when you use Python2.3. Also attached is a diff against the current _ssl.c which shows how to correct that behaviour while also having the possibilities from the above timeout patches. I carefully tested it, please review and test yourself and finally check-in. If you have questions, please don't hesitate to ask me. I really would like to see that not only in 2.4, but also (if possible together with patch #909007 from S. Nicolary) in a 2.3.4 release. Otherwise 2.3.x wasn't at all able to handle nonblocking SSL connections properly. Thanks and cheers, Tino ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-05-11 10:17 Message: Logged In: YES user_id=357491 The way you showed it, Tino, is correct. And I didn't realize the file was not following the spec. At this point, though, since we are trying to get someone to check this in before 2.3.4 goes out the door I wouldn't worry about it too much. Going in and fixing the syntax afterwards is not a huge deal. So if you have the time, great. Otherwise, because we have this possible time constraint and whom ever wrote the file in the first place didn't follow it, I wouldn't not worry about it too much. ---------------------------------------------------------------------- Comment By: Tino Lange (tinolange) Date: 2004-05-11 00:53 Message: Logged In: YES user_id=212920 Thanks for the hint. I guess --in the context of _ssl.c -- you only complain about the lines: } else sockstate = SOCKET_OPERATION_OK; or? Is that right? I think all the other else's are consistent - at least with the rest of the file - and some others I looked into (for example _bsddb.c). Or do you think that I shall reformat all the other else's in the file? Also those that are not from me? Really following the PEP it should look like: if (foo) { ... } else if (otherfoo) { ... } else { ... } instead of if (foo) { ... } else if (otherfoo) { ... } else { ... } everywhere, or? ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-05-10 18:10 Message: Logged In: YES user_id=357491 A style nit: put 'else' statements on their own line. See PEP 7 (http:// www.python.org/peps/pep-0007.html) on C code guidelines. But I am no network programmer so I have no clue if the solution is good. ---------------------------------------------------------------------- Comment By: Jesper ?JJ? Jurcenoks (jesperjurcenoks) Date: 2004-05-03 11:14 Message: Logged In: YES user_id=1033524 Hi Tino. I'll take your word for it. I though your ssl was based on the included SocketServer.py Cheers JJ ---------------------------------------------------------------------- Comment By: Tino Lange (tinolange) Date: 2004-05-02 01:40 Message: Logged In: YES user_id=212920 Hi JJ, I just scanned your SocketServer.py - and I really don't see how this class might help? You don't offer a better sslob.read() there, don't you? So how could it help (besides maybe making my example "server.py nicer). If I put a sslobj on top of a socket object it is blocking in 2.3, 2.3.x and higher. As proven in my example: Making the underlying socket nonblocking has no effect - because SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE from the SSL_xxx() functions in the C-Library do not break out of the while loop that was introduced in the above mentioned patches. Cheers, Tino ---------------------------------------------------------------------- Comment By: Jesper ?JJ? Jurcenoks (jesperjurcenoks) Date: 2004-05-01 19:46 Message: Logged In: YES user_id=1033524 Hi Tino. I just submitted a nonblocking SocketServer.py, it might just solve your problem. JJ ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 From daughters at halfsoaked.com Tue May 11 13:22:39 2004 From: daughters at halfsoaked.com (Potion E. Gilberto) Date: Tue May 11 13:27:43 2004 Subject: [Patches] Software cheap, Patches ! Message-ID: <111001c4377c$f10d51bb$30a6287e@halfsoaked.com> Well well! Always be ready to speak your mind, and a base man will avoid you. Low rates on Software Looking for not expensive high-quality software? We might be just what you need. http://noncolloid.fosraw.biz/OE017/?affiliate_id=233712&campaign_id=601 And more! We are able to ship worldwide. Of all tasks of government the most basic is to protect its citizens against violence. If you stand straight, do not fear a crooked shadow. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040511/d8701033/attachment.html From noreply at sourceforge.net Tue May 11 13:46:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 13:46:05 2004 Subject: [Patches] [ python-Patches-952047 ] fix typos Message-ID: Patches item #952047, was opened at 2004-05-12 02:45 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=952047&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Nobody/Anonymous (nobody) Summary: fix typos Initial Comment: Fix a bunch of typos before Python 2.3.4 gets released. All of them are in the src/Doc/lib directory. diff is against CVS HEAD. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=952047&group_id=5470 From noreply at sourceforge.net Tue May 11 13:48:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 13:48:07 2004 Subject: [Patches] [ python-Patches-952047 ] fix typos Message-ID: Patches item #952047, was opened at 2004-05-12 02:45 Message generated for change (Settings changed) made by quiver You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=952047&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: George Yoshida (quiver) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: fix typos Initial Comment: Fix a bunch of typos before Python 2.3.4 gets released. All of them are in the src/Doc/lib directory. diff is against CVS HEAD. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=952047&group_id=5470 From noreply at sourceforge.net Tue May 11 15:59:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 15:59:38 2004 Subject: [Patches] [ python-Patches-468347 ] mask signals for non-main pthreads Message-ID: Patches item #468347, was opened at 2001-10-05 11:58 Message generated for change (Comment added) made by jasonlowe You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=468347&group_id=5470 Category: Modules Group: None Status: Closed Resolution: Accepted Priority: 5 Submitted By: Jason Lowe (jasonlowe) Assigned to: Guido van Rossum (gvanrossum) Summary: mask signals for non-main pthreads Initial Comment: This patch updates Python/thread_pthread.h to mask all signals for any thread created. This will keep all signals masked for any thread that isn't the initial thread. For Solaris and Linux, the two platforms I was able to test it on, it solves bug #465673 (pthreads need signal protection) and probably will solve bug #219772 (Interactive InterPreter+ Thread -> core dump at exit). I'd be great if this could get some testing on other platforms, especially HP-UX pre 11.00 and post 11.00, as I had to make some guesses for the DCE thread case. AIX is also a concern as I saw some mention of using sigthreadmask() as a pthread_sigmask() equivalent, but this patch doesn't use sigthreadmask(). I don't have access to AIX. Note that thread_pthread.h.orig in this patch is the unmodified version from the Python2.1 release. ---------------------------------------------------------------------- >Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 14:59 Message: Logged In: YES user_id=56897 I agree, the patch is very problematic, especially for synchronous signals like SIGSEGV. I've added more details to the discussion in bug 756924. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 09:01 Message: Logged In: YES user_id=6380 This patch has caused more problems than it solves, IMO! Have a look at bug 756924: http://sourceforge.net/tracker/index.php?func=detail&aid=756924&group_id=5470&atid=105470 Jason, if you're still around, would you mind looking into an alternative fix for 465673 as suggested there? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2001-10-12 16:49 Message: Logged In: YES user_id=6380 As you said, the patch was out of date. Normally that would be enough to ask you to resubmit it relative to the current CVS, but it was too easy. I've checked this in as thread_pthread.h rev 2.33 now, so that it gets maximal exposure when the 2.2b1 release goes out next week. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=468347&group_id=5470 From noreply at sourceforge.net Tue May 11 18:20:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 18:20:27 2004 Subject: [Patches] [ python-Patches-945642 ] nonblocking i/o with ssl socket not working at all Message-ID: Patches item #945642, was opened at 2004-05-01 00:14 Message generated for change (Comment added) made by tinolange You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Tino Lange (tinolange) Assigned to: Nobody/Anonymous (nobody) Summary: nonblocking i/o with ssl socket not working at all Initial Comment: Hi! Unfortunately the applied patches #675750 and #676472 (End of January by G. Talvola / B. Laurie / G. v. Rossum) for timeout-sockets in version 1.09 and 1.10 to Modules/_ssl.c broke the nonblocking SSL I/O. In fact sslobj.read() became *always* blocking - whether the underlying socket is blocking or nonblocking doesn't matter at all. In Python < 2.3 this has worked correctly, starting with 2.3, 2.3.x up to the current CVS it doesn't work anymore. Attached is a sample: 0) [ Preparation ] Please run stunnel on your machine to have a SSL service: /usr/sbin/stunnel -p /path/to/some/cert -f -d 4711 -r 4712 1) [ Server ] Please run the easy server.py 2) [Client showing the bug] Please run the client - and see how it hangs when you use Python2.3. Also attached is a diff against the current _ssl.c which shows how to correct that behaviour while also having the possibilities from the above timeout patches. I carefully tested it, please review and test yourself and finally check-in. If you have questions, please don't hesitate to ask me. I really would like to see that not only in 2.4, but also (if possible together with patch #909007 from S. Nicolary) in a 2.3.4 release. Otherwise 2.3.x wasn't at all able to handle nonblocking SSL connections properly. Thanks and cheers, Tino ---------------------------------------------------------------------- >Comment By: Tino Lange (tinolange) Date: 2004-05-12 00:20 Message: Logged In: YES user_id=212920 OK. Fine :-) Please find below in file "patch_honoring_the_style_guide.diff" an slightly changed version of my original patch that follows the PEP7 better (for the single else statements containg only one commad without parantheses) - but still honors the rest of the file also. Only the formatting has changed a little bit, no content. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-05-11 19:17 Message: Logged In: YES user_id=357491 The way you showed it, Tino, is correct. And I didn't realize the file was not following the spec. At this point, though, since we are trying to get someone to check this in before 2.3.4 goes out the door I wouldn't worry about it too much. Going in and fixing the syntax afterwards is not a huge deal. So if you have the time, great. Otherwise, because we have this possible time constraint and whom ever wrote the file in the first place didn't follow it, I wouldn't not worry about it too much. ---------------------------------------------------------------------- Comment By: Tino Lange (tinolange) Date: 2004-05-11 09:53 Message: Logged In: YES user_id=212920 Thanks for the hint. I guess --in the context of _ssl.c -- you only complain about the lines: } else sockstate = SOCKET_OPERATION_OK; or? Is that right? I think all the other else's are consistent - at least with the rest of the file - and some others I looked into (for example _bsddb.c). Or do you think that I shall reformat all the other else's in the file? Also those that are not from me? Really following the PEP it should look like: if (foo) { ... } else if (otherfoo) { ... } else { ... } instead of if (foo) { ... } else if (otherfoo) { ... } else { ... } everywhere, or? ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-05-11 03:10 Message: Logged In: YES user_id=357491 A style nit: put 'else' statements on their own line. See PEP 7 (http:// www.python.org/peps/pep-0007.html) on C code guidelines. But I am no network programmer so I have no clue if the solution is good. ---------------------------------------------------------------------- Comment By: Jesper ?JJ? Jurcenoks (jesperjurcenoks) Date: 2004-05-03 20:14 Message: Logged In: YES user_id=1033524 Hi Tino. I'll take your word for it. I though your ssl was based on the included SocketServer.py Cheers JJ ---------------------------------------------------------------------- Comment By: Tino Lange (tinolange) Date: 2004-05-02 10:40 Message: Logged In: YES user_id=212920 Hi JJ, I just scanned your SocketServer.py - and I really don't see how this class might help? You don't offer a better sslob.read() there, don't you? So how could it help (besides maybe making my example "server.py nicer). If I put a sslobj on top of a socket object it is blocking in 2.3, 2.3.x and higher. As proven in my example: Making the underlying socket nonblocking has no effect - because SSL_ERROR_WANT_READ and SSL_ERROR_WANT_WRITE from the SSL_xxx() functions in the C-Library do not break out of the while loop that was introduced in the above mentioned patches. Cheers, Tino ---------------------------------------------------------------------- Comment By: Jesper ?JJ? Jurcenoks (jesperjurcenoks) Date: 2004-05-02 04:46 Message: Logged In: YES user_id=1033524 Hi Tino. I just submitted a nonblocking SocketServer.py, it might just solve your problem. JJ ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 From noreply at sourceforge.net Tue May 11 23:08:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 23:08:27 2004 Subject: [Patches] [ python-Patches-952047 ] fix typos Message-ID: Patches item #952047, was opened at 2004-05-11 13:45 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=952047&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: fix typos Initial Comment: Fix a bunch of typos before Python 2.3.4 gets released. All of them are in the src/Doc/lib directory. diff is against CVS HEAD. ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-11 23:08 Message: Logged In: YES user_id=3066 Committed as: Doc/lib/libcode.tex 1.15, 1.14.18.1 Doc/lib/libinspect.tex 1.16, 1.13.10.3 Doc/lib/libregex.tex 1.36, 1.33.42.3 Doc/lib/liburllib2.tex 1.17, 1.13.8.2 Doc/lib/libxmlrpclib.tex 1.16, 1.13.12.1 Doc/lib/xmlsaxutils.tex 1.6, 1.5.12.1 Thanks! ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=952047&group_id=5470 From noreply at sourceforge.net Tue May 11 23:16:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 23:16:25 2004 Subject: [Patches] [ python-Patches-932796 ] Error caused by patch #852334 Message-ID: Patches item #932796, was opened at 2004-04-10 09:52 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=932796&group_id=5470 Category: Demos and tools Group: None >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: George Yoshida (quiver) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: Error caused by patch #852334 Initial Comment: Looks like patch #852334``Replace backticks with repr ()'' had a problem. In Demo/threads/sync.py there is a statement with an extra parenthesis and the script raises a SyntaxError. # error message File "sync.py", line 421 'initial value %r' % (maxcount,)) ^ SyntaxError: invalid syntax My patch removes the extra parenthesis. ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-11 23:16 Message: Logged In: YES user_id=3066 It's also should be self.maxcount rather than maxcount; all better in Demo/threads/sync.py 1.8 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=932796&group_id=5470 From noreply at sourceforge.net Tue May 11 23:17:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 23:17:58 2004 Subject: [Patches] [ python-Patches-927232 ] Option to propagate errors from pkgutil Message-ID: Patches item #927232, was opened at 2004-03-31 22:38 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=927232&group_id=5470 Category: Library (Lib) Group: Python 2.4 >Status: Pending Resolution: None Priority: 5 Submitted By: Jp Calderone (kuran) Assigned to: Nobody/Anonymous (nobody) Summary: Option to propagate errors from pkgutil Initial Comment: pkgutil currently catches IOErrors when opening files. Instead of letting the error be handled, it logs something to stdout. The attached patch adds an optional parameter to extend_path which allows this behavior to be changed to simply raise the exception. ---------------------------------------------------------------------- Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-11 23:17 Message: Logged In: YES user_id=3066 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. In addition, even if you *did* check this checkbox, a bug in SourceForge prevents attaching a file when *creating* an issue. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=927232&group_id=5470 From noreply at sourceforge.net Tue May 11 23:23:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 23:24:07 2004 Subject: [Patches] [ python-Patches-919299 ] API Ref: PyErr_SetInterrupt not Obsolete Message-ID: Patches item #919299, was opened at 2004-03-19 01:21 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=919299&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed Resolution: Fixed Priority: 5 Submitted By: Kurt B. Kaiser (kbk) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: API Ref: PyErr_SetInterrupt not Obsolete Initial Comment: The API document Section 4. claims that this function is obsolete, but it is used in thread.interrupt_main(), which in turn is used in IDLE. Attached patch removes the 'obsolete' statement. If there is some reason that it is going to disappear, then we have to figure something out for thread.interrupt_main. ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-11 23:23 Message: Logged In: YES user_id=3066 Fixed wording on trunk in Doc/api/exceptions.tex 1.18, backported the final fix for Python 2.3.4 as revision 1.15.10.2. ---------------------------------------------------------------------- Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-03-25 09:37 Message: Logged In: YES user_id=3066 Fixed in Doc/api/exceptions.tex 1.17. Someone should backport the change to Python 2.3.x and close this bug report. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=919299&group_id=5470 From noreply at sourceforge.net Tue May 11 23:25:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 23:25:27 2004 Subject: [Patches] [ python-Patches-920037 ] email/__init__.py: Parse headersonly Message-ID: Patches item #920037, was opened at 2004-03-20 07:18 Message generated for change (Settings changed) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=920037&group_id=5470 Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Chris Mayo (cjmayo) >Assigned to: Barry A. Warsaw (bwarsaw) Summary: email/__init__.py: Parse headersonly Initial Comment: The attached simple patch brings the useful headersonly parameter up to the message_from_string function to optionally parse the headers of messages without bodies. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=920037&group_id=5470 From noreply at sourceforge.net Tue May 11 23:27:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 23:27:46 2004 Subject: [Patches] [ python-Patches-912410 ] HTMLParser should support entities in attributes Message-ID: Patches item #912410, was opened at 2004-03-08 20:20 Message generated for change (Settings changed) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=912410&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Aaron Swartz (aaronsw) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: HTMLParser should support entities in attributes Initial Comment: HTMLParser doesn't currently support entities in attributes, like this: foo This patch fixes that. Simply replace the unescape in HTMLParser.py with: import htmlentitydefs def unescape(self, s): def replaceEntities(s): s = s.groups()[0] if s[0] == "#": s = s[1:] if s[0] in ['x','X']: c = int(s[1:], 16) else: c = int(s) return unichr(c) else: return unichr(htmlentitydefs.name2codepoint[c]) return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));", replaceEntities, s) ---------------------------------------------------------------------- Comment By: Aaron Swartz (aaronsw) Date: 2004-03-08 20:21 Message: Logged In: YES user_id=122141 Argh. Hopefully now. ---------------------------------------------------------------------- Comment By: Aaron Swartz (aaronsw) Date: 2004-03-08 20:21 Message: Logged In: YES user_id=122141 Oops. The replacement function is attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=912410&group_id=5470 From noreply at sourceforge.net Tue May 11 23:27:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 23:27:48 2004 Subject: [Patches] [ python-Patches-865455 ] ConfigParser: fixes mixed-case interpolations (bug 857881) Message-ID: Patches item #865455, was opened at 2003-12-24 12:25 Message generated for change (Settings changed) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=865455&group_id=5470 Category: Library (Lib) Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Jordan R McCoy (jrm) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: ConfigParser: fixes mixed-case interpolations (bug 857881) Initial Comment: (Python CVS on FreeBSD 4.7 release) ConfigParser translates option names into full lowercase before associating them with the section dictionary (in function RawConfigParser->optionxform, used multiple locations); this seems to indicate that option names should be interpreted in a case-insensitive manner. However, the interpolation function in ConfigParser (_interpolate), uses the standard % string substitution construct, which is of course case-sensitive...thus, interpolations like '%(logFilename)' generate an exception. (See bug report 857881 for test case.) This patch translates interpolations to full lowercase using a regular expression substitution before they are interpolated. Since interpolations may run multiple times depending on their depth, this isn't the optimal solution, but is probably faster then the tokenizer interpolation used by SafeConfigParser. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=865455&group_id=5470 From noreply at sourceforge.net Tue May 11 23:27:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 23:27:50 2004 Subject: [Patches] [ python-Patches-854484 ] ConfigParser should raise an error for duplicate options Message-ID: Patches item #854484, was opened at 2003-12-04 20:24 Message generated for change (Settings changed) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=854484&group_id=5470 Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Mark McEahern (mceahm) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: ConfigParser should raise an error for duplicate options Initial Comment: ConfigParser should raise an error for duplicate options within a section; e.g., [test] key=value1 key=value2 but currently, it silently adds only one of the duplicate options to the section (since options are stored in a dictionary, keyed by option name). Suggested checkin comments: * Added an Error for DuplicateOptionError. * Modified _readfp() to raise DuplicateOptionError when a duplicate option is found within a section. * Added a test case to verify that the error is raised. ---------------------------------------------------------------------- Comment By: Mark McEahern (mceahm) Date: 2003-12-14 11:25 Message: Logged In: YES user_id=150272 I hadn't considered multiple files being read into the ConfigParser instance. It seems like there's three potential uses: * Duplicate options should override previous settings. * Duplicate options should result in an array of values for that setting. * Duplicate options should be considered an error. The scope of the problem is larger than I thought. Thanks for your comments. Cheers, // m ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2003-12-08 21:16 Message: Logged In: YES user_id=552329 Definately -1 here. Firstly, a duplicate option isn't fatal, so if there is an error at all, it should be treated like other non-fatal (i.e. parsing) errors and be raised only at the end. Secondly, there's no reason why a duplicate option is invalid. In particular, when working with multiple config files, it's desireable to be able to have later files override earlier ones. This might not apply so much to single files, but to be consistent, should. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=854484&group_id=5470 From OFUOUJUNDEIFI at mail.hf.ah.cn Tue May 11 19:56:51 2004 From: OFUOUJUNDEIFI at mail.hf.ah.cn (Leonard Messer) Date: Tue May 11 23:28:12 2004 Subject: [Patches] get the job you deserve with a university degree - no need to go to school Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040511/1154858e/attachment.html From noreply at sourceforge.net Tue May 11 23:52:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 23:52:26 2004 Subject: [Patches] [ python-Patches-870287 ] Fix docstring for posixpath.getctime Message-ID: Patches item #870287, was opened at 2004-01-04 04:22 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=870287&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: Gerrit Holl (gerrit) Assigned to: Nobody/Anonymous (nobody) Summary: Fix docstring for posixpath.getctime Initial Comment: (not sure whether this is "Library" or "Documentation") The docstring of posixpath.getctime calls ctime the "creation time". This is incorrect, because it is the "change time". The library documentation is correct here. This patch fixes the docstring. ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-11 23:52 Message: Logged In: YES user_id=3066 I've committed a variation of the patch ("metadata" instead of "mode", since that's closer to what's described in the man pages on my machine), and also updated the descriptions for os.stat() and stat.ST_CTIME. Doc/lib/libos.tex 1.135, 1.127.10.4 Doc/lib/libstat.tex 1.24, 1.23.10.1 Lib/posixpath.py 1.65, 1.62.6.1 ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-01-18 02:12 Message: Logged In: YES user_id=671362 Yes, your're right. The library document of os.path.ctime was fixed, but the docstring of posixpath.py has been left unchanged. So I'm +1 for applying this patch. I've noticed that there are still several places where the definition of ctime is not fixed. For example, stat.st_ctime and os.stat. http://www.python.org/doc/current/lib/module-stat.html > ST_CTIME : Time of last status change (see manual pages for details). http://www.python.org/doc/current/lib/os-file-dir.html > st_ctime (time of most recent content modification or metadata change). In my opinion, these two also need to be corrected. ---------------------------------------------------------------------- Comment By: Gerrit Holl (gerrit) Date: 2004-01-17 09:55 Message: Logged In: YES user_id=13298 The docstring is a different issue: The standard library needs to be accurate for all platforms, whereas each platform has it's own implementation of os.path.getctime. Hence, it's possible to give each platform it's own docstring for os.path.getctime, so that posixpath.getctime.__doc__ != ntpath.getctime.__doc__. Is this it a good thing to have different docstrings? If so, I'd say that each platform documents for it's own getctime what ctime means. If not, I'd say we copy the standard library documentation. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-01-11 23:11 Message: Logged In: YES user_id=671362 > IIRC, it is creation time on some platforms and is the > change time on others. This was discussed before. [ 827902 ] ctime is not creation time http://www.python.org/sf/827902 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-01-11 18:32 Message: Logged In: YES user_id=80475 IIRC, it is creation time on some platforms and is the change time on others. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=870287&group_id=5470 From noreply at sourceforge.net Tue May 11 23:53:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 23:53:53 2004 Subject: [Patches] [ python-Patches-901369 ] markupbase misses comments (bug 736659) Message-ID: Patches item #901369, was opened at 2004-02-20 15:58 Message generated for change (Settings changed) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=901369&group_id=5470 Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jim Jewett (jimjjewett) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: markupbase misses comments (bug 736659) Initial Comment: markupbase.ParserBase().parse_declaration calls parse_comment if the text starts with " datetime.date.fromtimestamp() TimestampFromTicks() --> datetime.datetime.fromtimestamp() TimeFromTicks() is not already exposed but discussions with Tim Peters already suggested that would not happen since the concept is rather strange. You'll have to discuss that with him if you disagree! :-) Any comments? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-26 21:46 Message: Logged In: YES user_id=38388 This is a good start. However, you should add more APIs for extracting date/time information to the C API. mxDateTime.h from egenix-mx-base can provide a good basis for this. If you want to make the datetime module usable for database interfacing at C level, you'll also have to add interfaces for Unix ticks conversion, since these are often used by database C level interfaces. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 19:36 Message: Logged In: YES user_id=619560 I have no objection to providing patches for doc and test. A quick look at _testcapimodule.c doesn't provide any obvious ideas as to how to patch it. I looked for cStringIO which has a C API already and found nothing in there at all. The documentation also simply says "see the source for the information you require". Any suggestions as to how to proceed? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-01-26 04:17 Message: Logged In: YES user_id=31435 Marc-Andre, can you review this? I think you have more experience building C APIs for modules than anyone else (while I don't have any). There's certainly no objection to giving datetime a C API, and Guido already said he doesn't want to rely on that Martin changed datetime to be built as part of the core (in 2.3 on Windows, datetime.pyd was a distinct DLL; in CVS, datetime is compiled into the core DLL now). Anthony, you probably need to add doc and test patches. _testcapimodule.c holds tests of things you can only get at from C. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 From noreply at sourceforge.net Wed May 19 05:55:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 05:55:43 2004 Subject: [Patches] [ python-Patches-956394 ] imaplib.IMAP4_SSL: changed quadratic read() code to linear Message-ID: Patches item #956394, was opened at 2004-05-19 02:15 Message generated for change (Settings changed) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=956394&group_id=5470 Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Carl Howells (c_wraith) >Assigned to: Piers Lauder (pierslauder) Summary: imaplib.IMAP4_SSL: changed quadratic read() code to linear Initial Comment: read() in imaplib.IMAP4_SSL used string concatenation with += in a loop. This lead to exceptionally poor performance retreiving large email messages. Changed both read() and readline() in IMAP4_SSL to aggregate string fragments in a list, then join the elements of the list before returning them. This replaces an O(n^2) algorithm with an O(n) algorithm, for a significant speed increase when retreiving large email messages. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=956394&group_id=5470 From Julius at abandonedlove.com Mon May 17 13:28:07 2004 From: Julius at abandonedlove.com (Jordan Xiong) Date: Wed May 19 09:01:18 2004 Subject: [Patches] RE: Soul Mate References: Message-ID: <0EU40Z940PT339FVB$fx320SX20otl50$OXR919EJJ7@PYX6> Is love just around the corner? It is, if you’re looking in the right places. Click on the link below and discover where thousands of people have fallen in love… and it’s free http://abandonedlove.com/confirm/?oc=50793336 Don't want these notifications?: http://abandonedlove.com/remove/?oc=50793336 ybaggysqueaky trepidation arrowroot butch belove euler don't davy indemnity luger brass rasp rickettsia christie casbah becky allergy blouse cytology builtin tuck syringa o's debate pent awe odorous prejudicial backwood lament faustralisamicable bub megavolt dainty sigma liturgic ellipsoidal milch ifni krypton veldt trichloroethane doltish morphemic lykes longhorn dinghy repulsion deletion allegation delinquent dowling loosen advantage avid murk begrudge hung lullaby From noreply at sourceforge.net Wed May 19 16:37:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 16:37:51 2004 Subject: [Patches] [ python-Patches-957033 ] Allow compilation of C/C++ Python extensions without MSVC Message-ID: Patches item #957033, was opened at 2004-05-19 20:37 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957033&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Connelly (connelly) Assigned to: Nobody/Anonymous (nobody) Summary: Allow compilation of C/C++ Python extensions without MSVC Initial Comment: This page describes how to compile C/C++ Python extensions with Mingw: http://sebsauvage.net/python/mingw.html This process works. However, you need to patch Python to get the 'python setup.py install' step to work correctly. This is the second step in installing a Python extension. The first step: 'python setup.py build -cmingw32' works fine. The second step: 'python setup.py install' fails with the following error: "error: Python was built with version 6 of Visual Studio, and extensions need to be built with the same version of the compiler, but it isn't installed." This statement is factually untrue. Python uses .pyd files to run Python extensions, which are actually .dll files, and are compatible across different compilers and compiler versions. So open \Python\Lib\distutils\msvccompiler.py and remove the warning (it's around line 211). Or apply the patch: --- msvccompiler.bak Thu Dec 4 09:38:22 2003 +++ msvccompiler.py Wed May 19 13:15:34 2004 @@ -208,11 +208,10 @@ self.__root = r"Software\Microsoft\Devstudio" self.__paths = self.get_msvc_paths("path") - if len (self.__paths) == 0: - raise DistutilsPlatformError, - ("Python was built with version %s of Visual Studio, " - "and extensions need to be built with the same " - "version of the compiler, but it isn't installed." % self.__version) + # Note: Extensions CAN be built with Mingw32. + # See http://sebsauvage.net/python/mingw.html. + # The .pyd files created are DLLs, and these are compatible across + # different compilers and compiler versions. self.cc = self.find_exe("cl.exe") self.linker = self.find_exe("link.exe") ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957033&group_id=5470 From noreply at sourceforge.net Wed May 19 16:40:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 16:40:50 2004 Subject: [Patches] [ python-Patches-957033 ] Allow compilation of C/C++ Python extensions without MSVC Message-ID: Patches item #957033, was opened at 2004-05-19 20:37 Message generated for change (Comment added) made by connelly You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957033&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Connelly (connelly) Assigned to: Nobody/Anonymous (nobody) Summary: Allow compilation of C/C++ Python extensions without MSVC Initial Comment: This page describes how to compile C/C++ Python extensions with Mingw: http://sebsauvage.net/python/mingw.html This process works. However, you need to patch Python to get the 'python setup.py install' step to work correctly. This is the second step in installing a Python extension. The first step: 'python setup.py build -cmingw32' works fine. The second step: 'python setup.py install' fails with the following error: "error: Python was built with version 6 of Visual Studio, and extensions need to be built with the same version of the compiler, but it isn't installed." This statement is factually untrue. Python uses .pyd files to run Python extensions, which are actually .dll files, and are compatible across different compilers and compiler versions. So open \Python\Lib\distutils\msvccompiler.py and remove the warning (it's around line 211). Or apply the patch: --- msvccompiler.bak Thu Dec 4 09:38:22 2003 +++ msvccompiler.py Wed May 19 13:15:34 2004 @@ -208,11 +208,10 @@ self.__root = r"Software\Microsoft\Devstudio" self.__paths = self.get_msvc_paths("path") - if len (self.__paths) == 0: - raise DistutilsPlatformError, - ("Python was built with version %s of Visual Studio, " - "and extensions need to be built with the same " - "version of the compiler, but it isn't installed." % self.__version) + # Note: Extensions CAN be built with Mingw32. + # See http://sebsauvage.net/python/mingw.html. + # The .pyd files created are DLLs, and these are compatible across + # different compilers and compiler versions. self.cc = self.find_exe("cl.exe") self.linker = self.find_exe("link.exe") ---------------------------------------------------------------------- >Comment By: Connelly (connelly) Date: 2004-05-19 20:40 Message: Logged In: YES user_id=1039782 BTW: You can test this out with the extension module myspell-python, available at: http://www.zgoda.biz/dnld/myspell-python-1.0- minimal.tar.gz Follow the directions at http://sebsauvage.net/python/mingw.html. You'll encounter an error. Patch msvccompiler.py and try again; now it works. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957033&group_id=5470 From noreply at sourceforge.net Wed May 19 16:41:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 16:41:58 2004 Subject: [Patches] [ python-Patches-957033 ] Allow compilation of C/C++ Python extensions without MSVC Message-ID: Patches item #957033, was opened at 2004-05-19 20:37 Message generated for change (Comment added) made by connelly You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957033&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Connelly (connelly) Assigned to: Nobody/Anonymous (nobody) Summary: Allow compilation of C/C++ Python extensions without MSVC Initial Comment: This page describes how to compile C/C++ Python extensions with Mingw: http://sebsauvage.net/python/mingw.html This process works. However, you need to patch Python to get the 'python setup.py install' step to work correctly. This is the second step in installing a Python extension. The first step: 'python setup.py build -cmingw32' works fine. The second step: 'python setup.py install' fails with the following error: "error: Python was built with version 6 of Visual Studio, and extensions need to be built with the same version of the compiler, but it isn't installed." This statement is factually untrue. Python uses .pyd files to run Python extensions, which are actually .dll files, and are compatible across different compilers and compiler versions. So open \Python\Lib\distutils\msvccompiler.py and remove the warning (it's around line 211). Or apply the patch: --- msvccompiler.bak Thu Dec 4 09:38:22 2003 +++ msvccompiler.py Wed May 19 13:15:34 2004 @@ -208,11 +208,10 @@ self.__root = r"Software\Microsoft\Devstudio" self.__paths = self.get_msvc_paths("path") - if len (self.__paths) == 0: - raise DistutilsPlatformError, - ("Python was built with version %s of Visual Studio, " - "and extensions need to be built with the same " - "version of the compiler, but it isn't installed." % self.__version) + # Note: Extensions CAN be built with Mingw32. + # See http://sebsauvage.net/python/mingw.html. + # The .pyd files created are DLLs, and these are compatible across + # different compilers and compiler versions. self.cc = self.find_exe("cl.exe") self.linker = self.find_exe("link.exe") ---------------------------------------------------------------------- >Comment By: Connelly (connelly) Date: 2004-05-19 20:41 Message: Logged In: YES user_id=1039782 Here's the upload. ---------------------------------------------------------------------- Comment By: Connelly (connelly) Date: 2004-05-19 20:40 Message: Logged In: YES user_id=1039782 BTW: You can test this out with the extension module myspell-python, available at: http://www.zgoda.biz/dnld/myspell-python-1.0- minimal.tar.gz Follow the directions at http://sebsauvage.net/python/mingw.html. You'll encounter an error. Patch msvccompiler.py and try again; now it works. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957033&group_id=5470 From noreply at sourceforge.net Wed May 19 19:06:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 19:06:50 2004 Subject: [Patches] [ python-Patches-957122 ] Improved hash for tuples Message-ID: Patches item #957122, was opened at 2004-05-20 02:06 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957122&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Yitz Gale (ygale) Assigned to: Nobody/Anonymous (nobody) Summary: Improved hash for tuples Initial Comment: See bug #942952. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957122&group_id=5470 From noreply at sourceforge.net Wed May 19 20:50:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 20:50:44 2004 Subject: [Patches] [ python-Patches-956394 ] imaplib.IMAP4_SSL: changed quadratic read() code to linear Message-ID: Patches item #956394, was opened at 2004-05-19 11:15 Message generated for change (Comment added) made by pierslauder You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=956394&group_id=5470 Category: Library (Lib) Group: None Status: Open >Resolution: Accepted Priority: 5 Submitted By: Carl Howells (c_wraith) Assigned to: Piers Lauder (pierslauder) Summary: imaplib.IMAP4_SSL: changed quadratic read() code to linear Initial Comment: read() in imaplib.IMAP4_SSL used string concatenation with += in a loop. This lead to exceptionally poor performance retreiving large email messages. Changed both read() and readline() in IMAP4_SSL to aggregate string fragments in a list, then join the elements of the list before returning them. This replaces an O(n^2) algorithm with an O(n) algorithm, for a significant speed increase when retreiving large email messages. ---------------------------------------------------------------------- >Comment By: Piers Lauder (pierslauder) Date: 2004-05-20 10:50 Message: Logged In: YES user_id=196212 Thanks for your changes - will check them in! ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=956394&group_id=5470 From noreply at sourceforge.net Wed May 19 21:17:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 21:17:28 2004 Subject: [Patches] [ python-Patches-956394 ] imaplib.IMAP4_SSL: changed quadratic read() code to linear Message-ID: Patches item #956394, was opened at 2004-05-19 11:15 Message generated for change (Comment added) made by pierslauder You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=956394&group_id=5470 Category: Library (Lib) Group: None >Status: Closed Resolution: Accepted Priority: 5 Submitted By: Carl Howells (c_wraith) Assigned to: Piers Lauder (pierslauder) Summary: imaplib.IMAP4_SSL: changed quadratic read() code to linear Initial Comment: read() in imaplib.IMAP4_SSL used string concatenation with += in a loop. This lead to exceptionally poor performance retreiving large email messages. Changed both read() and readline() in IMAP4_SSL to aggregate string fragments in a list, then join the elements of the list before returning them. This replaces an O(n^2) algorithm with an O(n) algorithm, for a significant speed increase when retreiving large email messages. ---------------------------------------------------------------------- >Comment By: Piers Lauder (pierslauder) Date: 2004-05-20 11:17 Message: Logged In: YES user_id=196212 changes checked in. ---------------------------------------------------------------------- Comment By: Piers Lauder (pierslauder) Date: 2004-05-20 10:50 Message: Logged In: YES user_id=196212 Thanks for your changes - will check them in! ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=956394&group_id=5470 From noreply at sourceforge.net Thu May 20 02:23:34 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 02:23:40 2004 Subject: [Patches] [ python-Patches-957240 ] Patch for feature request 491033 Message-ID: Patches item #957240, was opened at 2004-05-19 23:23 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957240&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Josiah Carlson (josiahcarlson) Assigned to: Nobody/Anonymous (nobody) Summary: Patch for feature request 491033 Initial Comment: The attached patch is to offer the functionality desired by feature request: python.org/sf/491033 On machines that use standard IEEE floating point doubles, the 'infinity' needs only to be 1e16 due to the limited precision of floating point. 1e309 is used for convenience sake, and will continue to be a precision infinity until 1024 bits of precision in floating point values is available. At such point in time where 1e309 is available in hardware, it will still sufficient due to the practical limitations of counting. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957240&group_id=5470 From noreply at sourceforge.net Thu May 20 02:26:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 02:26:29 2004 Subject: [Patches] [ python-Patches-957240 ] Patch for feature request 491033 Message-ID: Patches item #957240, was opened at 2004-05-19 23:23 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957240&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Josiah Carlson (josiahcarlson) Assigned to: Nobody/Anonymous (nobody) Summary: Patch for feature request 491033 Initial Comment: The attached patch is to offer the functionality desired by feature request: python.org/sf/491033 On machines that use standard IEEE floating point doubles, the 'infinity' needs only to be 1e16 due to the limited precision of floating point. 1e309 is used for convenience sake, and will continue to be a precision infinity until 1024 bits of precision in floating point values is available. At such point in time where 1e309 is available in hardware, it will still sufficient due to the practical limitations of counting. ---------------------------------------------------------------------- >Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-19 23:26 Message: Logged In: YES user_id=341410 Sorry about that, I generated the patch by hand and got the ordering messed up. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957240&group_id=5470 From noreply at sourceforge.net Tue May 18 09:41:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 04:46:42 2004 Subject: [Patches] [ python-Patches-955928 ] use PyOS_ReadlineFunctionPointer in non-interractive input Message-ID: Patches item #955928, was opened at 2004-05-18 15:41 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=955928&group_id=5470 Category: Parser/Compiler Group: None Status: Open Resolution: None Priority: 5 Submitted By: Gregory Lielens (greglielens) Assigned to: Nobody/Anonymous (nobody) Summary: use PyOS_ReadlineFunctionPointer in non-interractive input Initial Comment: Extend the control provided by PyOS_ReadlineFunctionPointer to non-interractive inputs. In addition, initialize PyOS_ReadlineFunctionPointer to the default input function, so that it can be retrieved by extern code (allowing for embedding the old input function and modifying it). Readline Module changed accordingly (only modify PyOS_ReadlineFunctionPointer to call_readline if input is interractive). This was neccessary to modify the input behavior of PyRun_InteractiveLoop, in case input is not interractive... The application is a python driven parallel framework using MPI, where the interractive process broadcast the input lines to the non-interractive processes, who listen...but other applications may benefit also. patched: Modules/readline.c Parser/myreadline.c ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=955928&group_id=5470 From noreply at sourceforge.net Thu May 20 09:59:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 09:59:36 2004 Subject: [Patches] [ python-Patches-957398 ] Generator Objects Message-ID: Patches item #957398, was opened at 2004-05-20 06:59 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957398&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: James William Pye (jwpye) Assigned to: Nobody/Anonymous (nobody) Summary: Generator Objects Initial Comment: Here the patches/files for public generator object/type. removal patch for ceval.c ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957398&group_id=5470 From noreply at sourceforge.net Thu May 20 11:43:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 11:43:19 2004 Subject: [Patches] [ python-Patches-957398 ] Public Generator Object/Type Message-ID: Patches item #957398, was opened at 2004-05-20 06:59 Message generated for change (Settings changed) made by jwpye You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957398&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: James William Pye (jwpye) Assigned to: Nobody/Anonymous (nobody) >Summary: Public Generator Object/Type Initial Comment: Here the patches/files for public generator object/type. removal patch for ceval.c ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957398&group_id=5470 From noreply at sourceforge.net Thu May 20 12:03:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 12:04:03 2004 Subject: [Patches] [ python-Patches-736962 ] Port tests to unittest (Part 2) Message-ID: Patches item #736962, was opened at 2003-05-13 05:45 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 Category: Tests Group: None Status: Open Resolution: Accepted Priority: 5 Submitted By: Walter D?rwald (doerwalter) >Assigned to: Walter D?rwald (doerwalter) Summary: Port tests to unittest (Part 2) Initial Comment: Here are the next test scripts ported to PyUnit: test_winsound and test_array. For test_array many additional tests have been added (code coverage is at 91%) ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 11:03 Message: Logged In: YES user_id=80475 test_dict is fine. Consider beefing up the tests for update(). As of Py2.4, it now takes all the same argument possibilites as dict(): d.update(a=1, b=2) d.update([('a', 1), ('b', 2)]) Also, try to link in: from test_userdict import TestMappingProtocol This was supposed to provide tests common to all mappings. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-04-03 06:06 Message: Logged In: YES user_id=89016 The attached test_dict.py moves the dict tests from test_types.py to a new PyUnit test script. Code coverage for dictobject.c is at 90.76% (up from 89.51%) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-03-15 07:17 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_binascii delete Lib/test/test_binascii.py 1.17 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 15:26 Message: Logged In: YES user_id=80475 Okay, this one is fine. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-18 18:49 Message: Logged In: YES user_id=89016 Here's the next one: test_binascii.py. Code coverage in binascii.c is at 92% (could be improved by enhancing test_binhex.py). ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 17:46 Message: Logged In: YES user_id=33168 Looked good to me, test_future_pyunit.diff checked in as: * Lib/test/badsyntax_future3.py 1.2 * Lib/test/badsyntax_future4.py 1.2 * Lib/test/badsyntax_future5.py 1.2 * Lib/test/badsyntax_future6.py 1.2 * Lib/test/badsyntax_future7.py 1.2 * Lib/test/badsyntax_future8.py 1.1 * Lib/test/badsyntax_future9.py 1.1 * Lib/test/test_future.py 1.7 * Lib/test/test_future1.py 1.3 * Lib/test/test_future2.py 1.2 * Lib/test/output/test_future 1.4 ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 16:47 Message: Logged In: YES user_id=33168 Walter, I didn't get a mail from SF either. After you do a cvs add, you need to use -N to cvs diff. For example, cvs diff -N new_file_added_to_cvs_but_not_committed. If you look carefully, you can see this in the patch on the diff line for each file. I'll take a look at the patch. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-13 15:18 Message: Logged In: YES user_id=89016 Here is a version of test_future ported to PyUnit (test_future_pyunit.diff). If this makes sense to you Neal, please check it in. (BTW, how did you convince CVS to include badsyntax_future8.py and badsyntax_future9.py in the diff? Doing a "cvs add" only results in " Lib/test/badsyntax_future8.py is a new entry, no comparison available") ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-11 07:36 Message: Logged In: YES user_id=89016 md5/weakref patch checked in as: Lib/test/test_weakref.py 1.33 Lib/test/test_md5.py 1.5 Lib/test/output/test_md5 remove ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 01:05 Message: Logged In: YES user_id=33168 Improve coverage for test_future too. I'm not sure if test future can be ported to PyUnit. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 00:52 Message: Logged In: YES user_id=33168 Attached are two tests (in one file, I'm being lazy, sorry) to improve test coverage for md5c.c and _weakref.c. test_md5 was ported to unittest, weakref, just adds two little tests to get coverage up to 100%. If you like, just go ahead and check in. You will need to remove Lib/test/output/test_md5 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-08 06:42 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/list_tests.py 1.1 Lib/test/seq_tests.py 1.1 Lib/test/test_list.py 1.1 Lib/test/test_tuple.py 1.1 Lib/test/test_types.py 1.56 Lib/test/test_userlist.py 1.11 Lib/test/output/test_types 1.3 Next will be test_binascii.py before I continue with test_types.py. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-12-07 18:49 Message: Logged In: YES user_id=80475 Looks good. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-11-27 14:48 Message: Logged In: YES user_id=89016 Here the first part of the test_types port: list and tuple tests have been moved to their own scripts: test_tuple.py and test_list.py. Common tests for tuple, list and UserList are shared (in seq_test.py and list_test.py) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-09-01 20:53 Message: Logged In: YES user_id=80475 Applied as: Lib/test/test_slice.py 1.5. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-31 18:52 Message: Logged In: YES user_id=80475 Looks good. I added a couple of minor tests. Revised patch attached. Okay to apply. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-31 14:49 Message: Logged In: YES user_id=89016 Thanks! Here's another one: test_slice.py ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-30 18:04 Message: Logged In: YES user_id=80475 Done. See: Lib/test/test_longexp.py 1.9; previous revision: 1.8 Lib/test/test_pep263.py 1.3; previous revision: 1.2 Lib/test/test_sets.py 1.27; previous revision: 1.26 Lib/test/test_structseq.py 1.5; previous revision: 1.4 Lib/test/output/test_longexp delete; previous revision: 1.3 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-30 12:10 Message: Logged In: YES user_id=80475 Will do! ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-30 12:06 Message: Logged In: YES user_id=89016 Here's test_structse.py converted with a few additional tests. If all three scripts are OK, could you check them in Raymond, as I'll be on vacation for three weeks? ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-09 10:47 Message: Logged In: YES user_id=89016 Here are two simple ones: test_pep263 and test_longexp. I can't think of any additional tests to add. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-23 08:38 Message: Logged In: YES user_id=80475 Fixed Walter's review comments and committed as Lib/test/test_compile.py 1.19 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-23 06:49 Message: Logged In: YES user_id=89016 Are you sure, that the code path is the same in the new test_argument_handling() as in the old test? I.e. is "eval('lambda a,a: 0')" the same as "exec 'def f(a, a): pass'"? The print statement in test_float_literals should be changed to a comment. Should the imports in test_unary_minus() be moved to the start of the script? Otherwise the test looks (and runs) OK. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-20 14:26 Message: Logged In: YES user_id=80475 Here's one for you: test_compile.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-19 04:46 Message: Logged In: YES user_id=89016 Maybe test_types.py should be split into several scripts: test_dict, test_tuple, test_list, test_int, test_long etc. Some of them already exist (like test_long), some don't. The constructor tests from test_builtin should probably be moved to the new test scripts as well. Furthermore we should try to share as much testing functionality as possible (e.g. between test_int and test_long, or between test_list and test_userlist) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-18 14:48 Message: Logged In: YES user_id=80475 Great! Can I suggest that test_types.py be next. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-18 09:27 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_builtin.py 1.21 Lib/test/test_complex.py 1.10 (I've moved the constructor tests from test_builtin to test_complex) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-17 18:49 Message: Logged In: YES user_id=80475 I added a few tests. If they are fine with you, go ahead and commit. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 16:36 Message: Logged In: YES user_id=89016 Here's the next one: text_complex.py. There one test that's currently commented out, because it crashes Python on Alpha (see http://www.python.org/sf/756093. The old test scripts states that tests for the constructor are in test_builtin, but I've added many tests to this script, so this is no longer true. We could move the tests to test_builtin, but IMHO that doesn't make sense, we'd better move the rest of the constructor tests from test_builtin to test_complex. I'd like to have a version of assertAlmostEqual() in unittest.py that can cope with complex numbers, but this would have to be coordinated with the standalone version of PyUnit (and it would probably have to wait until the 2.4 cycle starts) (I noticed that there is no assertAlmostEqual in the code on pyunit.sf.net anyway.) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 07:06 Message: Logged In: YES user_id=89016 I'd like to keep this patch open, as it is an ongoing task (the next test scripts to be converted will be test_complex and then maybe test_marshal) ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 16:55 Message: Logged In: YES user_id=357491 OK, all tests pass cleanly. Applied as revision 1.6. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 14:51 Message: Logged In: YES user_id=89016 The joys of unittesting: Breaking code to make it better! ;) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 14:41 Message: Logged In: YES user_id=80475 Okay, it runs fine here. Once Brett confirms that it runs on the Mac, go ahead and load it. P.S. Your improved test_mimetools.py helped detect a latent error in mimetools.py when it was run under Windows. Tim made the fix this weekend. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 14:33 Message: Logged In: YES user_id=89016 Strange, I didn't get this failure here, but I got it on my laptop at home. I've removed the comparison with the getatime() value from test_time(). I hope this fixes it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 14:09 Message: Logged In: YES user_id=80475 Walter, there is one failure left: ====================================== ================================ FAIL: test_time (__main__.PosixPathTest) ------------------------------------------------------------------- --- Traceback (most recent call last): File "test_posixpath.py", line 125, in test_time self.assert_( File "C:\PY23\lib\unittest.py", line 268, in failUnless if not expr: raise self.failureException, msg AssertionError Brett, after Walter revises the patch, just load the patch and make sure the test runs on the Mac. Between the three of us, we can validate the suite on three different platforms. Cheers. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 13:20 Message: Logged In: YES user_id=357491 Just wanting to work me like a dog, huh, Raymond? =) And to clarify for my and Walter's benefit, when you say guards, you mean that the tests don't crap out and say they failed on Windows, right? I thought posixpath was not meant to work under Windows. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 13:09 Message: Logged In: YES user_id=89016 I didn't realize that test_posixpath must work on Windows too. Here's a new version. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 11:04 Message: Logged In: YES user_id=80475 The previous comment applied to another patch. It should have said: Assigning to Brett to make sure the patch runs on the Mac. Don't accept this one until it has guards that allow the tests to run on Windows. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 10:59 Message: Logged In: YES user_id=80475 Assigning to Brett to give experience doing a detail review on this type of change. * examine every line of the diff and consider whether there is any semantic change (exceptions raised, etc). * apply the diff and run the test suite * in the interactive mode, call-up each function and make sure it behaves as expected (this is necessary because the test coverage is very low). * verify that the whitespace has been cleaned up. * look for missing changes (such as use of +=) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 10:44 Message: Logged In: YES user_id=80475 The test file now has dependencies that do not apply to windows. The failure messages are attached. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 07:48 Message: Logged In: YES user_id=89016 Here's the next one: test_posixpath.py with many additional tests. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 12:33 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_mimetools delete Lib/test/test_mimetools.py 1.4 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-22 11:18 Message: Logged In: YES user_id=80475 test_mimetools.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 10:05 Message: Logged In: YES user_id=89016 I've attached a third version of test_mimetools.py that does some checks for the mimetools.Message class. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-21 08:04 Message: Logged In: YES user_id=80475 Attaching a slightly modified test_mimetools which covers more encodings and has a stronger set test. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 18:46 Message: Logged In: YES user_id=89016 Agreed, this is too much magic for too little gain. Back to business: Here is test_mimetools ported to PyUnit. Tests for mimetools.Message are still missing. If you can think of any tests please add them. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 22:18 Message: Logged In: YES user_id=80475 Get the module with sys.modules: tests = test_support.findtestclasses(sys.modules [__name__]) test_support.unittest(*tests) Yeah, the inheritance thing is a problem. I was trying to avoid having to modify unittest.TestCase to have a metaclass. The control of the module is kept in a separate SF project and one of its goals is to be backward compatible through 1.5.2 (meaning no metaclasses). A possible workaround is to define a modified testcase in test_support so that people don't import unittest directly anymore: test_support.py ------------------------- import unittest class SmartTestCase(unittest.TestCase): __metaclass__ = autotracktests pass test_sets.py ------------------ class TestBasicOps(test_support.SmartTestCase): run = False . . . class TestBasicOpsEmpty(TestBasicOps): def setUp(self): . . . Still, this is starting to seem a bit magical and tricky. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-17 21:52 Message: Logged In: YES user_id=89016 But how do I pass the module object from inside the module? And skipping abstract classes seems to be more work in this version: If skipping is done via a class attribute, derived classes have to explicitely reset this flag because of interitance. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 20:59 Message: Logged In: YES user_id=80475 Good call. Instead of using metaclasses, perhaps add a module introspector function to test_support: def findtestclasses(mod): tests = [] for elem in dir(mod): member = getattr(mod, elem) if type(member) != type: continue if issubclass(member, unittest.TestCase): tests.append(member) return tests ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-17 20:45 Message: Logged In: YES user_id=89016 But this can be solved with a special non-inheritable class attribute: class BaseTest(unittest.TestCase): run = False Then the metaclass can do the following: def __new__(cls, name, bases, dict): if "run" not in dict: dict["run"] = True cls = type.__new__(cls, name, bases, dict) if cls.run: tests.append(cls) return cls ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 20:04 Message: Logged In: YES user_id=80475 I don't think metaclasses or module introspection would help whenever there are classes that derive from TestCase but are not meant to be run directly (their subclasses have the setup/teardown/or class data). test_sets.py has examples of that kind of thing. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-17 19:50 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_array.py 1.20 Lib/test/test_winsound.py 1.5 Lib/test/output/test_winsound delete > The approach of using tests.append() is elegant and > makes it easier to verify that no tests are being omitted. The most elegant approach would probably be a metaclass that collects all TestCase subclasses that get defined. Classes that only serve as a base class could be skipped by specifying a certain class attribute. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 18:35 Message: Logged In: YES user_id=80475 The approach of using tests.append() is elegant and makes it easier to verify that no tests are being omitted. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 From noreply at sourceforge.net Thu May 20 12:09:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 12:09:36 2004 Subject: [Patches] [ python-Patches-957398 ] Public Generator Object/Type Message-ID: Patches item #957398, was opened at 2004-05-20 06:59 Message generated for change (Comment added) made by jwpye You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957398&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: James William Pye (jwpye) Assigned to: Nobody/Anonymous (nobody) Summary: Public Generator Object/Type Initial Comment: Here the patches/files for public generator object/type. removal patch for ceval.c ---------------------------------------------------------------------- >Comment By: James William Pye (jwpye) Date: 2004-05-20 09:09 Message: Logged In: YES user_id=1044177 Ok, got it to compile, and tried out a simple generator; seems to be fine.. Probably should have asked first, but I'm assuming(yay) that it was desired to draw it inline with *object.[ch] organization protocol.. Only gotcha appears to be that genobject.c, of course, references eval_frame, so I just wrapped a call to it in eval.c with a non-static function(PyEval_EvaluateFrame()), this is probably not be what python-dev would like.. I mostly did it like this to get it to compile.(probably a bad idea, and should probably reference it directly. no need for an extra layer of function call overhead...) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957398&group_id=5470 From noreply at sourceforge.net Thu May 20 12:51:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 12:51:14 2004 Subject: [Patches] [ python-Patches-872326 ] generator expression implementation Message-ID: Patches item #872326, was opened at 2004-01-07 21:10 Message generated for change (Comment added) made by jiwon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470 Category: Parser/Compiler Group: None Status: Open Resolution: Accepted Priority: 5 Submitted By: Jiwon Seo (jiwon) Assigned to: Nobody/Anonymous (nobody) Summary: generator expression implementation Initial Comment: Since I was interested in pep 289(generator expression), I dabbled with it, and implemented a working version of it. I'm not sure if I did it right, but maybe someone who knows better can fix it right. 1. Grammar has two changes, which is a. atom: '(' [testlist] ')' | '[' [listmaker] ']' | ... changes to atom: '(' [testgenexpr] ')' | '[' [listmaker] ']' | ... where testgenexpr defines like this. testgenexpr: test ( gen_for | (',' test)* [','] ) b. argument: [test '='] test changes to argument: [test '='] test [gen_for] (gen_for, gen_if, gen_iter is similar to list_for, list_if, list_iter respectively.) 2. Instead of changing rule of arglist in Grammar to accept generator expression, I changed argument rule like 1.b. This means Grammar accepts generator expression without parenthesis in function call even there are several arguments, like reduce(operator.add, (x for x in range(10))) This is against what pep requires, so I made parsermodule.c and compile.c to catch that and throw error message when there's more than one argument in a function. The reason I did it that way is to go around a problem of ambiguity in the grammar by adding generator expression to arglist. 3. I implemented generator expression as equivalent to a call to a function which has variable capturing code and generator code. For example, x = 1 g = (x for i in range(10)) is equivalent to x = 1 def __gen_wrapper(): _[x] = x # variable capture here def __gen(): for i in range(10): yield _[x] return __gen() g = __gen_wrapper() 4. Since I implemented generator expression equivalent to a function call, symbol table should enter new scope when it meets generator expression. So I ended up with adding following code to symtable.c PyObject * PySymtableEntry_New(struct symtable *st, char *name, int type, int lineno) { ... switch (type) { case funcdef: case lambdef: ! case testgenexpr: /* generator expression */ ! case argument: /* generator expression */ ste->ste_type = TYPE_FUNCTION; break; ... so that generator expression can be dealt as function type. This is kind of stupid, but I couldn't find other easy for it, so I just left it like that. 5. this patch does not include diff of src/Lib/symbol.py, so you must run python Lib/symbol.py to get it right. ---------------------------------------------------------------------- >Comment By: Jiwon Seo (jiwon) Date: 2004-05-21 01:51 Message: Logged In: YES user_id=595483 Thank you, and I'm happy to contribute. =) I updated documentation so that it deals with the precomputation issue. But I don't know how much I should write or where, so please review/revise it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-19 17:20 Message: Logged In: YES user_id=80475 Thank you for the superb contribution. It passes all of my tests without exception (see the new file, test_genexps). I do not see any issues with the coding. Am accepting and applying the patch except for the documentation. Please update the documentation part of the patch and I will go through and clarify where necessary. Also, I will update the PEP to reflect Guido's rationale for the design decisions on binding behavior. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-16 02:24 Message: Logged In: YES user_id=595483 OK. I added outmost-iterable-precomputation feature to lazy-binding version of the patch, and also added related tests to test_grammar.py. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-15 15:08 Message: Logged In: YES user_id=595483 Ok. I see. I've almost finished adding it(outmost iterable precomputation) to CPython, and now trying to do it in python compiler package. Probably I'll commit patch till tomorrow. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-15 09:31 Message: Logged In: YES user_id=80475 Jiwon, do you see how to proceed for here? Note that given, (for x in exp1 if exp2 for y in exp3 if exp4), only exp1 is evaluated immediately. exp2 is deferred. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-13 12:58 Message: Logged In: YES user_id=6380 On why the first expr should be evaluated immediately: Consider sum(x for x in foo()). Now suppose there's a bug in foo() that raises an exception, and a bug in sum() that raises an exception before it starts iterating over its argument. Which exception would you expect to see? I'd be surprised if the one in sum() was raised rather the one in foo(), since the call to foo() is part of the argument to sum(), and I expect arguments to be processed before the function is called. OTOH, in sum(bar(x) for x in foo()), where sum() and foo() are bugfree, but bar() raises an exception, we have no choice but to delay the call to bar() until sum() starts iterating -- that's part of the contract of generators. (They do nothing until their next() method is first called.) Ditto for subsequent iterables if nested for loops are used: in sum(x*y for x in range(10) for y in bar(x)), there are 10 calls to bar(), with arguments supplied by the first (outer) for loop, and we have no choice but to delay even the first call to bar() until sum() calls next() on its argument. I know this isn't the easiest thing to implement, but I still think it's the right thing. The generator function for the last example would be something like def G(arg): for x in arg: for y in bar(x): yield x*y and the call to sum() would be sum(G(range(10))) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-13 03:54 Message: Logged In: YES user_id=80475 Jeremy: I reviewed the lazy binding version of the patch as much as I could. Can you take a brief look at just the changes to the grammar and at the new functions in compile.c. The functions were clearly written and neatly paralleled list comps but I wondered whether it was possible and desirable to factor-out the commonalities. Jiwon: the graminit.h renumbering is fine (all of the automatically generated files should be left as-as). Guido: I was clear on your rationale for preferring late binding but did not find wording for the reason behind the precompution of the innermost iterable. I speculate that the semantics reflect that this is an expression and that all expressions (except and/or) are immediate. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-13 02:36 Message: Logged In: YES user_id=6380 Read back what I wrote on python-dev about capturing free variables as a new concept with which I am uncomfortable. There should not be such "advanced" usages of genexprs -- advanced usages are better served by explicit generator functions. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-13 00:26 Message: Logged In: YES user_id=4771 Which is exactly why I cannot understand why you are so keen on one binding model rather than the other, if it doesn't matter. It looks like we are going to add is a feature that doesn't scale to advanced usages. Moreover it seems really obvious that non-experts *will* get surprizes, because they *will* try to replace all their listcomps with genexprs -- they shouldn't do that, but of course they will do it anyway, because it appears to work -- until the occasional strange bug kreeps in, and one that is difficult to actually relate to the genexpr in the first place. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 23:03 Message: Logged In: YES user_id=6380 You're just showing that saving genexprs up for later use is a bad idea. We should place clear warnings in the docs that the genexpr should be used up right away, and otherwise to write an explicit generator function. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-12 17:30 Message: Logged In: YES user_id=595483 I don't think that "lazy-binding + leftmost iterable precomputation" makes sense. Just adding another for-loop to the example shows the reason. >>> x = 10 >>> g = ((i,j) for i in range(x) for j in range(x)) >>> x = 5 >>> list(g) Here, if j is iterated only to 5 when i is iterated to 10, I think it would make users confused. I think "early-binding + leftmost iterable precomputation" makes sense, but "lazy-binding + leftmost iterable precomputation" does not make sense IMHO. About renumbering testlist_gexp in graminit.h, it's not what I did but it's auto-generated by pgen(Parser/pgenmain.c). Although it makes another file (symbol.py) need to be changed (and thus makes cvs log a bit dirty), it's tool- generated file, so I did it like that. If you think it's better to do it without re-numbering, let me know or you could do it yourself. ;^) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-12 16:32 Message: Logged In: YES user_id=80475 The lazy_bind version has survived over of a month of my testing and multiple read throughs of the code :-) I'm not expert with graminit.h but did wonder if testlist_gexp could have a high numbered code so that other codes would not have to be re-numbered. When you get back from military camp, please load the version that precomputes the outermost iterable. I believe that is Guido's currently favored approach. Currently, it fails this test: >>> x = 10 >>> g = (i*i for i in range(x)) >>> x = 5 >>> list(g) # should be ten items long [0, 1, 4, 9, 16] Tim, summarized it as: """ ... Guido favored mostly-late binding -- which is late binding, except that the iterable in the leftmost for-clause is evaluated at once. So ... (e1 for stuff in e2 for morestuff in e3 ...) ... is effectively replaced by def __g(primary): for stuff in primary: for morestuff in e3: ... yield e1 ... __g(e2) ... """ ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-28 00:56 Message: Logged In: YES user_id=595483 I'm submitting two versions now. One is variable-capture version(but without any iterable pre-computation), and the other is lazy-binding version. I'll be in military camp for about 1 month(4/6~5/2) so I will not be able to fix/patch anything for the period, thus I'm submitting this in now. :) If I have time, and necessity arises (before 4/6), I'll also add outmost-iterable-precomputation version. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-03-23 08:00 Message: Logged In: YES user_id=6380 Sorry, I meant the first, not the last. I was confused about how the 'for' clauses are nested, but the outermost one is the first. So the nesting remark is probably just confusing and wrong; ignore it. What I meant is: (f(x,y) for x in A() for y in B(x)) should IMO precompute A(), but not B(x). I guess the equivalent generator function would be: def __gen(__outer__=A(), f=f, B=B): for x in __outer__: for y in B(x): yield f(x,y) In general the value of every free variable used anywhere except in the outer expression should be captured; the *value* of the outer expression should be captured. This should give the least surprising semantics in a variaty of use cases. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-03-23 04:39 Message: Logged In: YES user_id=6380 Only the outermost (last) iterable should be precomputed. While (f(x,y) for x in A(y) for y in B) is not the same as ((f(x,y) for x in A(y)) for y in B) I think the scoping should be done similarly. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-22 10:57 Message: Logged In: YES user_id=595483 Ah... Maybe we need to drop precomputation of iterables. I'll post it on the mailing list so that people can consider about that. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-03-22 08:40 Message: Logged In: YES user_id=4771 Oh ho. Am I right in fearing that the idea of precomputing the iterables was broken in the first place? We just cannot do this. The things we are looping over may depend on other stuff from the generator expression itself. Example: >>> [x for l in [[1,2],[3,4]] for x in l] [1, 2, 3, 4] >>> (y for m in [[1,2],[3,4]] for y in m) NameError: name 'm' is not defined ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-22 02:55 Message: Logged In: YES user_id=55188 This inconsistency may not be fixed by list(iter()) workaround, either. >>> def counter(): ... counter.count += 1 ... return range(counter.count) ... >>> counter.count = 0 >>> [y for x in 'abc' for y in counter()] [0, 0, 1, 0, 1, 2] >>> counter.count = 0 >>> list(y for x in 'abc' for y in counter()) [0, 0, 0] ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-22 02:42 Message: Logged In: YES user_id=55188 Aah. But I'm not insisting on that because it's incompatible even with plain nested for-loops. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-22 02:38 Message: Logged In: YES user_id=55188 Agreed. I'd prefer the current implementation's behavor rather than defined in PEP289. Yes, It's incompatible with list comprehensions but generator expressions are already quite different enough from it. :) How about to change PEP289's genexpr semantics to this? g = (x**2 for x in range(10)) print g.next() is equivalent to: def __gen(_i1): for x in _i1: yield x**2 g = __gen(range(10)) print g.next() ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-22 02:28 Message: Logged In: YES user_id=80475 You need a solution other than list(). A key to the success of genexps is to never to eat memory by expanding an iterator into a container. For behavior questions, your reference point is the list comprehension. list(genexp) should always produce the same result as a list comprehension. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-22 01:59 Message: Logged In: YES user_id=595483 Hi, quiver. I don't think we can easily go around this problem if we have to capture iterators in generator expression. If you run following, you'll know what I mean. >>> a = iter("abcd") >>> b = iter("abcd") >>> [(x, y) for x in a for y in b] [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] I think one possible solution could be, instead of passing evaluations of iterators in generator expression, make a list from iterator and, then pass it as argument. For instance, g = (x for x in iter("abcd")) will be equivalent to, def __gen(_[1]): for x in _[1]: yield x g = __gen(list(iter("abcd"))) # see 'list' - instead of g = __gen(iter("abcd")) . I'm not sure if I'm in a position to decide to do that way or not. If the current reviewer (rhettinger) approves it, I'll do that way. Or else, I think I will post it on the mailing list. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-03-19 22:37 Message: Logged In: YES user_id=671362 Hi, Jiwon. This is another bug candidate. If you use genexp with iterators, it doesn't work well. # test Perky's previous post using iterators >>> list((x,y) for x in iter('abcd') for y in iter('abcd')) [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] Thanks. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-17 17:45 Message: Logged In: YES user_id=595483 To fix the bug that perky picked out, I hand-made ast.py instead of using auto-generated code. But, still I don't understand why Tools/compiler/regrtest didn't tell me about it. (Maybe I didn't look at the output carefully enough.) Ah, and it would be nice if somebody tell me how to run test_grammar.py(only) with python-compiler package. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-17 15:28 Message: Logged In: YES user_id=55188 The compiler package patch has some problems in its compiler/ast.py currently. jiwon said he regenerated it using Tools/compiler/astgen.py. But it made some incompatibilities against ast.py on current CVS. For example, class Expression. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-17 15:21 Message: Logged In: YES user_id=80475 I'll give it a second review. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-17 14:37 Message: Logged In: YES user_id=55188 Okay. Here's my draft for documentation. Any review/fix/enhance is very welcome! I think it's ready to putting it into CVS now. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-17 01:24 Message: Logged In: YES user_id=595483 ah, the following bug was due to the patch I uploaded 2004-02-04 15:13. In order to get an error instantly from an expression like "g=(x for x in None)", I made it equivalent to, def __gen(_[1]): for x in _[1]: yield x g=__gen(iter(None)) ^^^^ But when there is two or more iterator expression of same symbol(or string), that patch produces error, because currently, "((x, y) for x in 'abcd' for y in 'abcd') " is equivalent to, def __gen(_[1]): for x in _[1]: for y in _[1]: yield (x,y) g = __gen(iter("abcd")) # passing only one parameter I could make it pass every iterator expressions respectively even when they are same symbol(or string, ...), but for now, I just dropped that patch (patch of 2004-02-04) since that's the simplest thing now. If somebody says getting instance error for cases like "(x for x in None)" is important, I'll make another patch for it. Btw, added more test case that covers what perky mentioned. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-16 13:16 Message: Logged In: YES user_id=55188 Another bug in the implementation. >>> list((x, y) for x in 'abcd' for y in 'abcd') [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] Expected behavior may be: >>> [(x, y) for x in 'abcd' for y in 'abcd'] [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd'), ('b', 'a'), ('b', 'b'), ('b', 'c'), ('b', 'd'), ('c', 'a'), ('c', 'b'), ('c', 'c'), ('c', 'd'), ('d', 'a'), ('d', 'b'), ('d', 'c'), ('d', 'd')] ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-14 11:59 Message: Logged In: YES user_id=55188 I'm writing docs for tutorial and language reference. It'll be completed in a day or two. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-14 06:17 Message: Logged In: YES user_id=80475 Any recent progress? ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-24 01:34 Message: Logged In: YES user_id=595483 Whoa! I finally patched compiler package in pure python. (I was a bit busy recently :) I've run regression test with 'Tools/compiler/regrtest.py' before I submit this patch, and there was one failure (which is test_dis.py). I'm not sure if that's a problem, so I'm just submitting this in. Now, I think I'll refactor things a bit! ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-02-04 22:53 Message: Logged In: YES user_id=55188 As it mentioned in PEP, even listcomp's leaking of loop value will be dropped in Python 3. I'm sorry but I don't see that the usage is solid in the future. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-04 19:45 Message: Logged In: YES user_id=671362 What about this code? In the currently implementation, loop variables inside a list comprehension is not visible outside if you use it inside a generator expression. For example: >>> (a*b for a in [b for b in range(5)]) Traceback (most recent call last): File "", line 1, in ? NameError: name 'b' is not defined Its list comprehension counterpart is: >>> [a*b for a in [b for b in range(5)]] [0, 4, 8, 12, 16] ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-04 15:13 Message: Logged In: YES user_id=595483 Again, I fixed the patch so that it can get error from '(x for x in None)' immediately. (upto now, error does not occur until generator expression is evaluated) ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-02-03 20:46 Message: Logged In: YES user_id=4771 After thinking a bit more on the issue, note that the generator version of your example is equivalent to: def g(): for y in range(3): yield lambda x: x+y which means that it can suffer from the same problem as the first example, but more subtly (and even more confusingly): for f in g(): print f(0) # 0, 1, 2 for f in list(g()): print f(0) # 2, 2, 2 This is because due to Python's nested scope rules the above generator is equivalent to: def g(): result = lambda x: x+y for y in range(3): yield result at which point I think it gets pretty confusing... ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-03 18:07 Message: Logged In: YES user_id=671362 Thanks, Arigo and Perky. Hmm, maybe I should read PEP and the thread about namespace more carefully, not just skim the surface. Anyway, PEP has not been updated since last October, so I think it's good time to merge recent progress of genexpr and update PEP-289. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-02-02 21:58 Message: Logged In: YES user_id=4771 The behavior is indeed the one described by the PEP but it is still surprising. As far as I'm concerned it is yet another reason why free variable bindings ("nested scopes") are done wrong in Python currently :-( If you use the older trick "lambda x,y=y:x+y" instead, then both cases will agree (with the sensible result in my opinion). A few more issues and I'll start a campain for definition-time bindings of free variables :-( ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-02-02 21:37 Message: Logged In: YES user_id=55188 I think it's right for namespace difference between listcomp and genexpr. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-02 20:47 Message: Logged In: YES user_id=671362 I am not sure if I should call this a bug, but here it goes: >>> lst = [lambda x:x+y for y in range(3)] >>> for f in lst:print f(0) ... 2 2 2 >>> gen = (lambda x:x+y for y in range(3)) >>> for f in gen:print f(0) ... 0 1 2 Is this the intended behavior? ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-02 17:29 Message: Logged In: YES user_id=595483 ok. I fixed another bug reported by perky, and added related test to test_grammar.py. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-30 14:09 Message: Logged In: YES user_id=55188 Yet another Fatal Python error; >>> (a for a in (b for b in (a for a in 'xxx' if True) if False) if True) lookup 'True' in ? 3 -1 freevars of : ('True',) Fatal Python error: com_make_closure() zsh: abort (core dumped) ./python # applied patch as of 2004-01-30 ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-30 12:01 Message: Logged In: YES user_id=595483 ok, I've fixed the bug quiver commented, and added related test code to test_grammar.py. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-01-29 20:48 Message: Logged In: YES user_id=671362 yet another Fatal Python error; >>> (a for a in (b for b in (0,1))) >>> (a for a in (b for b in range(2))) Fatal Python error: unknown scope for range in ?(0) in symbols: {'range': 8} locals: {} globals: {} Aborted # applied patch as of 2004-01-27 ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-27 16:44 Message: Logged In: YES user_id=595483 I fixed the patch for the bug that arigo mentioned, and for what perky mentioned - PyList_GetSlice error handling - . now, I'll tackle python compiler package :) ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-01-27 00:15 Message: Logged In: YES user_id=4771 >>> (a for d in a) Fatal Python error: unknown scope for a in (1) in symbols: {'a': 2048, '_[1]': 4, 'd': 2} locals: {'_[1]': 0, 'd': 1} globals: {} ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-26 15:17 Message: Logged In: YES user_id=55188 Please ignore the previous comment. It was a result from an old revision of patches. It works on the recent. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-26 08:44 Message: Logged In: YES user_id=55188 BTW, does unavaliability of yield (genexpr) and return (genexpr) an intended behavior? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-01-22 03:13 Message: Logged In: YES user_id=6656 Documentation would be nice! ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-21 23:14 Message: Logged In: YES user_id=55188 Okay. My review is done. The revised patch "gexp.diff" looks fine for me. There're few minor tweaks still needed to get into CVS. - Some error exit cases are ignored. You need to provide safe exit paths for MemoryError. (eg. PyList_GetSlice usage of Python/ compiler.c) - A patch for 'compiler' pure python package. (there's an old implementation on SF #795947) ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-19 20:10 Message: Logged In: YES user_id=595483 ok. I modified the patch so that it evaluates iterator expr in generator expression creation time. That means, g = (x for x in range(10)) is equivalent to def __gen(iter1): for x in iter1: yield x g = __gen(range(10)) so, evalution of range(10) is not deferred anymore. I also changed testgenexpr to testlist_gexp in Grammar/Grammar, since Hye-Shik Chang advised as such. I'm willing to take any advice about this patch, so please do. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-12 12:26 Message: Logged In: YES user_id=595483 ok. I've implemented capturing variables part as arigo suggested. File genexpr-capture-vars-in-args.diff is that. If you look at the code, you'll find that the code for generator expression is much shorter, and there's lesser modification in the existing code. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-09 10:49 Message: Logged In: YES user_id=595483 What arigo wrote sounds reasonable, and not very difficult to implement. I'll try to do that way. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-01-09 03:50 Message: Logged In: YES user_id=4771 We may not need two levels of nested anonymous functions. It seems to me that the following equivalent code would do, because it captures the variable in an argument instead of via nested scopes: x = 1 def __gen(x): for i in range(10): yield x g = __gen(x) I don't know though if this is easy to implement in compile.c. Alternatively: x = 1 def __gen(x=x): for i in range(10): yield x g = __gen() ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-08 14:44 Message: Logged In: YES user_id=55188 Okay. I verified that basic characteristics mentioned on PEP are working. I started to review the implementation. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-08 13:50 Message: Logged In: YES user_id=595483 I added diff of Lib/symbol.py, so no need to run python Lib/symbol.py now. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-07 21:25 Message: Logged In: YES user_id=55188 Assigned to me. The originator is my friend and I have much interest on this. :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470 From noreply at sourceforge.net Thu May 20 13:08:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 13:08:08 2004 Subject: [Patches] [ python-Patches-872326 ] generator expression implementation Message-ID: Patches item #872326, was opened at 2004-01-07 07:10 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470 Category: Parser/Compiler Group: None Status: Open Resolution: Accepted Priority: 5 Submitted By: Jiwon Seo (jiwon) Assigned to: Nobody/Anonymous (nobody) Summary: generator expression implementation Initial Comment: Since I was interested in pep 289(generator expression), I dabbled with it, and implemented a working version of it. I'm not sure if I did it right, but maybe someone who knows better can fix it right. 1. Grammar has two changes, which is a. atom: '(' [testlist] ')' | '[' [listmaker] ']' | ... changes to atom: '(' [testgenexpr] ')' | '[' [listmaker] ']' | ... where testgenexpr defines like this. testgenexpr: test ( gen_for | (',' test)* [','] ) b. argument: [test '='] test changes to argument: [test '='] test [gen_for] (gen_for, gen_if, gen_iter is similar to list_for, list_if, list_iter respectively.) 2. Instead of changing rule of arglist in Grammar to accept generator expression, I changed argument rule like 1.b. This means Grammar accepts generator expression without parenthesis in function call even there are several arguments, like reduce(operator.add, (x for x in range(10))) This is against what pep requires, so I made parsermodule.c and compile.c to catch that and throw error message when there's more than one argument in a function. The reason I did it that way is to go around a problem of ambiguity in the grammar by adding generator expression to arglist. 3. I implemented generator expression as equivalent to a call to a function which has variable capturing code and generator code. For example, x = 1 g = (x for i in range(10)) is equivalent to x = 1 def __gen_wrapper(): _[x] = x # variable capture here def __gen(): for i in range(10): yield _[x] return __gen() g = __gen_wrapper() 4. Since I implemented generator expression equivalent to a function call, symbol table should enter new scope when it meets generator expression. So I ended up with adding following code to symtable.c PyObject * PySymtableEntry_New(struct symtable *st, char *name, int type, int lineno) { ... switch (type) { case funcdef: case lambdef: ! case testgenexpr: /* generator expression */ ! case argument: /* generator expression */ ste->ste_type = TYPE_FUNCTION; break; ... so that generator expression can be dealt as function type. This is kind of stupid, but I couldn't find other easy for it, so I just left it like that. 5. this patch does not include diff of src/Lib/symbol.py, so you must run python Lib/symbol.py to get it right. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 12:08 Message: Logged In: YES user_id=80475 Jiwon, while working through the doc patch, I noticed a code generation nit. The code g=(i for i in range(10)) gets translated to: def __gen(exp): for i in exp: yield i g = __gen(iter(range(10)) del __gen On the next to last line, the call to iter() may be superfluous. Do you see a need for it? If so, I need to include that in the spec. If not, then it should be removed (line 1822 in compile.c). In terms of byte codes, the difference shows up in a comparison to an equivalent generator: >>> dis(compile('g(range(10))', '', 'eval')) 0 0 LOAD_NAME 0 (g) 3 LOAD_NAME 1 (range) 6 LOAD_CONST 0 (10) 9 CALL_FUNCTION 1 12 CALL_FUNCTION 1 15 RETURN_VALUE >>> dis(compile('(i for i in range(10))', '', 'eval')) 0 0 LOAD_CONST 0 ( at 00A4B220, file "", line 1>) 3 MAKE_FUNCTION 0 6 LOAD_NAME 0 (range) 9 LOAD_CONST 1 (10) 12 CALL_FUNCTION 1 15 GET_ITER 16 CALL_FUNCTION 1 19 RETURN_VALUE The get_iter at code position 15 is what is in question. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-20 11:51 Message: Logged In: YES user_id=595483 Thank you, and I'm happy to contribute. =) I updated documentation so that it deals with the precomputation issue. But I don't know how much I should write or where, so please review/revise it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-19 03:20 Message: Logged In: YES user_id=80475 Thank you for the superb contribution. It passes all of my tests without exception (see the new file, test_genexps). I do not see any issues with the coding. Am accepting and applying the patch except for the documentation. Please update the documentation part of the patch and I will go through and clarify where necessary. Also, I will update the PEP to reflect Guido's rationale for the design decisions on binding behavior. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-15 12:24 Message: Logged In: YES user_id=595483 OK. I added outmost-iterable-precomputation feature to lazy-binding version of the patch, and also added related tests to test_grammar.py. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-15 01:08 Message: Logged In: YES user_id=595483 Ok. I see. I've almost finished adding it(outmost iterable precomputation) to CPython, and now trying to do it in python compiler package. Probably I'll commit patch till tomorrow. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-14 19:31 Message: Logged In: YES user_id=80475 Jiwon, do you see how to proceed for here? Note that given, (for x in exp1 if exp2 for y in exp3 if exp4), only exp1 is evaluated immediately. exp2 is deferred. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 22:58 Message: Logged In: YES user_id=6380 On why the first expr should be evaluated immediately: Consider sum(x for x in foo()). Now suppose there's a bug in foo() that raises an exception, and a bug in sum() that raises an exception before it starts iterating over its argument. Which exception would you expect to see? I'd be surprised if the one in sum() was raised rather the one in foo(), since the call to foo() is part of the argument to sum(), and I expect arguments to be processed before the function is called. OTOH, in sum(bar(x) for x in foo()), where sum() and foo() are bugfree, but bar() raises an exception, we have no choice but to delay the call to bar() until sum() starts iterating -- that's part of the contract of generators. (They do nothing until their next() method is first called.) Ditto for subsequent iterables if nested for loops are used: in sum(x*y for x in range(10) for y in bar(x)), there are 10 calls to bar(), with arguments supplied by the first (outer) for loop, and we have no choice but to delay even the first call to bar() until sum() calls next() on its argument. I know this isn't the easiest thing to implement, but I still think it's the right thing. The generator function for the last example would be something like def G(arg): for x in arg: for y in bar(x): yield x*y and the call to sum() would be sum(G(range(10))) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-12 13:54 Message: Logged In: YES user_id=80475 Jeremy: I reviewed the lazy binding version of the patch as much as I could. Can you take a brief look at just the changes to the grammar and at the new functions in compile.c. The functions were clearly written and neatly paralleled list comps but I wondered whether it was possible and desirable to factor-out the commonalities. Jiwon: the graminit.h renumbering is fine (all of the automatically generated files should be left as-as). Guido: I was clear on your rationale for preferring late binding but did not find wording for the reason behind the precompution of the innermost iterable. I speculate that the semantics reflect that this is an expression and that all expressions (except and/or) are immediate. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 12:36 Message: Logged In: YES user_id=6380 Read back what I wrote on python-dev about capturing free variables as a new concept with which I am uncomfortable. There should not be such "advanced" usages of genexprs -- advanced usages are better served by explicit generator functions. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-12 10:26 Message: Logged In: YES user_id=4771 Which is exactly why I cannot understand why you are so keen on one binding model rather than the other, if it doesn't matter. It looks like we are going to add is a feature that doesn't scale to advanced usages. Moreover it seems really obvious that non-experts *will* get surprizes, because they *will* try to replace all their listcomps with genexprs -- they shouldn't do that, but of course they will do it anyway, because it appears to work -- until the occasional strange bug kreeps in, and one that is difficult to actually relate to the genexpr in the first place. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 09:03 Message: Logged In: YES user_id=6380 You're just showing that saving genexprs up for later use is a bad idea. We should place clear warnings in the docs that the genexpr should be used up right away, and otherwise to write an explicit generator function. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-12 03:30 Message: Logged In: YES user_id=595483 I don't think that "lazy-binding + leftmost iterable precomputation" makes sense. Just adding another for-loop to the example shows the reason. >>> x = 10 >>> g = ((i,j) for i in range(x) for j in range(x)) >>> x = 5 >>> list(g) Here, if j is iterated only to 5 when i is iterated to 10, I think it would make users confused. I think "early-binding + leftmost iterable precomputation" makes sense, but "lazy-binding + leftmost iterable precomputation" does not make sense IMHO. About renumbering testlist_gexp in graminit.h, it's not what I did but it's auto-generated by pgen(Parser/pgenmain.c). Although it makes another file (symbol.py) need to be changed (and thus makes cvs log a bit dirty), it's tool- generated file, so I did it like that. If you think it's better to do it without re-numbering, let me know or you could do it yourself. ;^) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-12 02:32 Message: Logged In: YES user_id=80475 The lazy_bind version has survived over of a month of my testing and multiple read throughs of the code :-) I'm not expert with graminit.h but did wonder if testlist_gexp could have a high numbered code so that other codes would not have to be re-numbered. When you get back from military camp, please load the version that precomputes the outermost iterable. I believe that is Guido's currently favored approach. Currently, it fails this test: >>> x = 10 >>> g = (i*i for i in range(x)) >>> x = 5 >>> list(g) # should be ten items long [0, 1, 4, 9, 16] Tim, summarized it as: """ ... Guido favored mostly-late binding -- which is late binding, except that the iterable in the leftmost for-clause is evaluated at once. So ... (e1 for stuff in e2 for morestuff in e3 ...) ... is effectively replaced by def __g(primary): for stuff in primary: for morestuff in e3: ... yield e1 ... __g(e2) ... """ ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-27 10:56 Message: Logged In: YES user_id=595483 I'm submitting two versions now. One is variable-capture version(but without any iterable pre-computation), and the other is lazy-binding version. I'll be in military camp for about 1 month(4/6~5/2) so I will not be able to fix/patch anything for the period, thus I'm submitting this in now. :) If I have time, and necessity arises (before 4/6), I'll also add outmost-iterable-precomputation version. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-03-22 18:00 Message: Logged In: YES user_id=6380 Sorry, I meant the first, not the last. I was confused about how the 'for' clauses are nested, but the outermost one is the first. So the nesting remark is probably just confusing and wrong; ignore it. What I meant is: (f(x,y) for x in A() for y in B(x)) should IMO precompute A(), but not B(x). I guess the equivalent generator function would be: def __gen(__outer__=A(), f=f, B=B): for x in __outer__: for y in B(x): yield f(x,y) In general the value of every free variable used anywhere except in the outer expression should be captured; the *value* of the outer expression should be captured. This should give the least surprising semantics in a variaty of use cases. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-03-22 14:39 Message: Logged In: YES user_id=6380 Only the outermost (last) iterable should be precomputed. While (f(x,y) for x in A(y) for y in B) is not the same as ((f(x,y) for x in A(y)) for y in B) I think the scoping should be done similarly. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-21 20:57 Message: Logged In: YES user_id=595483 Ah... Maybe we need to drop precomputation of iterables. I'll post it on the mailing list so that people can consider about that. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-03-21 18:40 Message: Logged In: YES user_id=4771 Oh ho. Am I right in fearing that the idea of precomputing the iterables was broken in the first place? We just cannot do this. The things we are looping over may depend on other stuff from the generator expression itself. Example: >>> [x for l in [[1,2],[3,4]] for x in l] [1, 2, 3, 4] >>> (y for m in [[1,2],[3,4]] for y in m) NameError: name 'm' is not defined ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-21 12:55 Message: Logged In: YES user_id=55188 This inconsistency may not be fixed by list(iter()) workaround, either. >>> def counter(): ... counter.count += 1 ... return range(counter.count) ... >>> counter.count = 0 >>> [y for x in 'abc' for y in counter()] [0, 0, 1, 0, 1, 2] >>> counter.count = 0 >>> list(y for x in 'abc' for y in counter()) [0, 0, 0] ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-21 12:42 Message: Logged In: YES user_id=55188 Aah. But I'm not insisting on that because it's incompatible even with plain nested for-loops. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-21 12:38 Message: Logged In: YES user_id=55188 Agreed. I'd prefer the current implementation's behavor rather than defined in PEP289. Yes, It's incompatible with list comprehensions but generator expressions are already quite different enough from it. :) How about to change PEP289's genexpr semantics to this? g = (x**2 for x in range(10)) print g.next() is equivalent to: def __gen(_i1): for x in _i1: yield x**2 g = __gen(range(10)) print g.next() ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-21 12:28 Message: Logged In: YES user_id=80475 You need a solution other than list(). A key to the success of genexps is to never to eat memory by expanding an iterator into a container. For behavior questions, your reference point is the list comprehension. list(genexp) should always produce the same result as a list comprehension. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-21 11:59 Message: Logged In: YES user_id=595483 Hi, quiver. I don't think we can easily go around this problem if we have to capture iterators in generator expression. If you run following, you'll know what I mean. >>> a = iter("abcd") >>> b = iter("abcd") >>> [(x, y) for x in a for y in b] [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] I think one possible solution could be, instead of passing evaluations of iterators in generator expression, make a list from iterator and, then pass it as argument. For instance, g = (x for x in iter("abcd")) will be equivalent to, def __gen(_[1]): for x in _[1]: yield x g = __gen(list(iter("abcd"))) # see 'list' - instead of g = __gen(iter("abcd")) . I'm not sure if I'm in a position to decide to do that way or not. If the current reviewer (rhettinger) approves it, I'll do that way. Or else, I think I will post it on the mailing list. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-03-19 08:37 Message: Logged In: YES user_id=671362 Hi, Jiwon. This is another bug candidate. If you use genexp with iterators, it doesn't work well. # test Perky's previous post using iterators >>> list((x,y) for x in iter('abcd') for y in iter('abcd')) [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] Thanks. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-17 03:45 Message: Logged In: YES user_id=595483 To fix the bug that perky picked out, I hand-made ast.py instead of using auto-generated code. But, still I don't understand why Tools/compiler/regrtest didn't tell me about it. (Maybe I didn't look at the output carefully enough.) Ah, and it would be nice if somebody tell me how to run test_grammar.py(only) with python-compiler package. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-17 01:28 Message: Logged In: YES user_id=55188 The compiler package patch has some problems in its compiler/ast.py currently. jiwon said he regenerated it using Tools/compiler/astgen.py. But it made some incompatibilities against ast.py on current CVS. For example, class Expression. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-17 01:21 Message: Logged In: YES user_id=80475 I'll give it a second review. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-17 00:37 Message: Logged In: YES user_id=55188 Okay. Here's my draft for documentation. Any review/fix/enhance is very welcome! I think it's ready to putting it into CVS now. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-16 11:24 Message: Logged In: YES user_id=595483 ah, the following bug was due to the patch I uploaded 2004-02-04 15:13. In order to get an error instantly from an expression like "g=(x for x in None)", I made it equivalent to, def __gen(_[1]): for x in _[1]: yield x g=__gen(iter(None)) ^^^^ But when there is two or more iterator expression of same symbol(or string), that patch produces error, because currently, "((x, y) for x in 'abcd' for y in 'abcd') " is equivalent to, def __gen(_[1]): for x in _[1]: for y in _[1]: yield (x,y) g = __gen(iter("abcd")) # passing only one parameter I could make it pass every iterator expressions respectively even when they are same symbol(or string, ...), but for now, I just dropped that patch (patch of 2004-02-04) since that's the simplest thing now. If somebody says getting instance error for cases like "(x for x in None)" is important, I'll make another patch for it. Btw, added more test case that covers what perky mentioned. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-15 23:16 Message: Logged In: YES user_id=55188 Another bug in the implementation. >>> list((x, y) for x in 'abcd' for y in 'abcd') [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] Expected behavior may be: >>> [(x, y) for x in 'abcd' for y in 'abcd'] [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd'), ('b', 'a'), ('b', 'b'), ('b', 'c'), ('b', 'd'), ('c', 'a'), ('c', 'b'), ('c', 'c'), ('c', 'd'), ('d', 'a'), ('d', 'b'), ('d', 'c'), ('d', 'd')] ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-13 21:59 Message: Logged In: YES user_id=55188 I'm writing docs for tutorial and language reference. It'll be completed in a day or two. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 16:17 Message: Logged In: YES user_id=80475 Any recent progress? ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-23 11:34 Message: Logged In: YES user_id=595483 Whoa! I finally patched compiler package in pure python. (I was a bit busy recently :) I've run regression test with 'Tools/compiler/regrtest.py' before I submit this patch, and there was one failure (which is test_dis.py). I'm not sure if that's a problem, so I'm just submitting this in. Now, I think I'll refactor things a bit! ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-02-04 08:53 Message: Logged In: YES user_id=55188 As it mentioned in PEP, even listcomp's leaking of loop value will be dropped in Python 3. I'm sorry but I don't see that the usage is solid in the future. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-04 05:45 Message: Logged In: YES user_id=671362 What about this code? In the currently implementation, loop variables inside a list comprehension is not visible outside if you use it inside a generator expression. For example: >>> (a*b for a in [b for b in range(5)]) Traceback (most recent call last): File "", line 1, in ? NameError: name 'b' is not defined Its list comprehension counterpart is: >>> [a*b for a in [b for b in range(5)]] [0, 4, 8, 12, 16] ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-04 01:13 Message: Logged In: YES user_id=595483 Again, I fixed the patch so that it can get error from '(x for x in None)' immediately. (upto now, error does not occur until generator expression is evaluated) ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-02-03 06:46 Message: Logged In: YES user_id=4771 After thinking a bit more on the issue, note that the generator version of your example is equivalent to: def g(): for y in range(3): yield lambda x: x+y which means that it can suffer from the same problem as the first example, but more subtly (and even more confusingly): for f in g(): print f(0) # 0, 1, 2 for f in list(g()): print f(0) # 2, 2, 2 This is because due to Python's nested scope rules the above generator is equivalent to: def g(): result = lambda x: x+y for y in range(3): yield result at which point I think it gets pretty confusing... ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-03 04:07 Message: Logged In: YES user_id=671362 Thanks, Arigo and Perky. Hmm, maybe I should read PEP and the thread about namespace more carefully, not just skim the surface. Anyway, PEP has not been updated since last October, so I think it's good time to merge recent progress of genexpr and update PEP-289. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-02-02 07:58 Message: Logged In: YES user_id=4771 The behavior is indeed the one described by the PEP but it is still surprising. As far as I'm concerned it is yet another reason why free variable bindings ("nested scopes") are done wrong in Python currently :-( If you use the older trick "lambda x,y=y:x+y" instead, then both cases will agree (with the sensible result in my opinion). A few more issues and I'll start a campain for definition-time bindings of free variables :-( ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-02-02 07:37 Message: Logged In: YES user_id=55188 I think it's right for namespace difference between listcomp and genexpr. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-02 06:47 Message: Logged In: YES user_id=671362 I am not sure if I should call this a bug, but here it goes: >>> lst = [lambda x:x+y for y in range(3)] >>> for f in lst:print f(0) ... 2 2 2 >>> gen = (lambda x:x+y for y in range(3)) >>> for f in gen:print f(0) ... 0 1 2 Is this the intended behavior? ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-02 03:29 Message: Logged In: YES user_id=595483 ok. I fixed another bug reported by perky, and added related test to test_grammar.py. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-30 00:09 Message: Logged In: YES user_id=55188 Yet another Fatal Python error; >>> (a for a in (b for b in (a for a in 'xxx' if True) if False) if True) lookup 'True' in ? 3 -1 freevars of : ('True',) Fatal Python error: com_make_closure() zsh: abort (core dumped) ./python # applied patch as of 2004-01-30 ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-29 22:01 Message: Logged In: YES user_id=595483 ok, I've fixed the bug quiver commented, and added related test code to test_grammar.py. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-01-29 06:48 Message: Logged In: YES user_id=671362 yet another Fatal Python error; >>> (a for a in (b for b in (0,1))) >>> (a for a in (b for b in range(2))) Fatal Python error: unknown scope for range in ?(0) in symbols: {'range': 8} locals: {} globals: {} Aborted # applied patch as of 2004-01-27 ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-27 02:44 Message: Logged In: YES user_id=595483 I fixed the patch for the bug that arigo mentioned, and for what perky mentioned - PyList_GetSlice error handling - . now, I'll tackle python compiler package :) ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-01-26 10:15 Message: Logged In: YES user_id=4771 >>> (a for d in a) Fatal Python error: unknown scope for a in (1) in symbols: {'a': 2048, '_[1]': 4, 'd': 2} locals: {'_[1]': 0, 'd': 1} globals: {} ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-26 01:17 Message: Logged In: YES user_id=55188 Please ignore the previous comment. It was a result from an old revision of patches. It works on the recent. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-25 18:44 Message: Logged In: YES user_id=55188 BTW, does unavaliability of yield (genexpr) and return (genexpr) an intended behavior? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-01-21 13:13 Message: Logged In: YES user_id=6656 Documentation would be nice! ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-21 09:14 Message: Logged In: YES user_id=55188 Okay. My review is done. The revised patch "gexp.diff" looks fine for me. There're few minor tweaks still needed to get into CVS. - Some error exit cases are ignored. You need to provide safe exit paths for MemoryError. (eg. PyList_GetSlice usage of Python/ compiler.c) - A patch for 'compiler' pure python package. (there's an old implementation on SF #795947) ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-19 06:10 Message: Logged In: YES user_id=595483 ok. I modified the patch so that it evaluates iterator expr in generator expression creation time. That means, g = (x for x in range(10)) is equivalent to def __gen(iter1): for x in iter1: yield x g = __gen(range(10)) so, evalution of range(10) is not deferred anymore. I also changed testgenexpr to testlist_gexp in Grammar/Grammar, since Hye-Shik Chang advised as such. I'm willing to take any advice about this patch, so please do. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-11 22:26 Message: Logged In: YES user_id=595483 ok. I've implemented capturing variables part as arigo suggested. File genexpr-capture-vars-in-args.diff is that. If you look at the code, you'll find that the code for generator expression is much shorter, and there's lesser modification in the existing code. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-08 20:49 Message: Logged In: YES user_id=595483 What arigo wrote sounds reasonable, and not very difficult to implement. I'll try to do that way. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-01-08 13:50 Message: Logged In: YES user_id=4771 We may not need two levels of nested anonymous functions. It seems to me that the following equivalent code would do, because it captures the variable in an argument instead of via nested scopes: x = 1 def __gen(x): for i in range(10): yield x g = __gen(x) I don't know though if this is easy to implement in compile.c. Alternatively: x = 1 def __gen(x=x): for i in range(10): yield x g = __gen() ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-08 00:44 Message: Logged In: YES user_id=55188 Okay. I verified that basic characteristics mentioned on PEP are working. I started to review the implementation. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-07 23:50 Message: Logged In: YES user_id=595483 I added diff of Lib/symbol.py, so no need to run python Lib/symbol.py now. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-07 07:25 Message: Logged In: YES user_id=55188 Assigned to me. The originator is my friend and I have much interest on this. :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470 From noreply at sourceforge.net Thu May 20 13:32:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 13:32:10 2004 Subject: [Patches] [ python-Patches-872326 ] generator expression implementation Message-ID: Patches item #872326, was opened at 2004-01-07 07:10 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470 Category: Parser/Compiler Group: None Status: Open Resolution: Accepted Priority: 5 Submitted By: Jiwon Seo (jiwon) Assigned to: Nobody/Anonymous (nobody) Summary: generator expression implementation Initial Comment: Since I was interested in pep 289(generator expression), I dabbled with it, and implemented a working version of it. I'm not sure if I did it right, but maybe someone who knows better can fix it right. 1. Grammar has two changes, which is a. atom: '(' [testlist] ')' | '[' [listmaker] ']' | ... changes to atom: '(' [testgenexpr] ')' | '[' [listmaker] ']' | ... where testgenexpr defines like this. testgenexpr: test ( gen_for | (',' test)* [','] ) b. argument: [test '='] test changes to argument: [test '='] test [gen_for] (gen_for, gen_if, gen_iter is similar to list_for, list_if, list_iter respectively.) 2. Instead of changing rule of arglist in Grammar to accept generator expression, I changed argument rule like 1.b. This means Grammar accepts generator expression without parenthesis in function call even there are several arguments, like reduce(operator.add, (x for x in range(10))) This is against what pep requires, so I made parsermodule.c and compile.c to catch that and throw error message when there's more than one argument in a function. The reason I did it that way is to go around a problem of ambiguity in the grammar by adding generator expression to arglist. 3. I implemented generator expression as equivalent to a call to a function which has variable capturing code and generator code. For example, x = 1 g = (x for i in range(10)) is equivalent to x = 1 def __gen_wrapper(): _[x] = x # variable capture here def __gen(): for i in range(10): yield _[x] return __gen() g = __gen_wrapper() 4. Since I implemented generator expression equivalent to a function call, symbol table should enter new scope when it meets generator expression. So I ended up with adding following code to symtable.c PyObject * PySymtableEntry_New(struct symtable *st, char *name, int type, int lineno) { ... switch (type) { case funcdef: case lambdef: ! case testgenexpr: /* generator expression */ ! case argument: /* generator expression */ ste->ste_type = TYPE_FUNCTION; break; ... so that generator expression can be dealt as function type. This is kind of stupid, but I couldn't find other easy for it, so I just left it like that. 5. this patch does not include diff of src/Lib/symbol.py, so you must run python Lib/symbol.py to get it right. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 12:32 Message: Logged In: YES user_id=80475 FWIW, here is a patch that moves the GET_ITER back inside the generator so that the implementation will more closely match the spec and match the operation of regular generators. *** compile.c 19 May 2004 08:20:15 -0000 2.302 --- compile.c 20 May 2004 17:29:35 -0000 *************** *** 1628,1633 **** --- 1628,1634 ---- if (is_outmost) { com_addop_varname(c, VAR_LOAD, "[outmost-iterable]"); + com_addbyte(c, GET_ITER); com_push(c, 1); } else { *************** *** 1819,1825 **** com_addoparg(c, MAKE_FUNCTION, 0); com_test(c, CHILD(CHILD(n, 1), 3)); - com_addbyte(c, GET_ITER); com_addoparg(c, CALL_FUNCTION, 1); com_pop(c, 1); --- 1820,1825 ---- ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 12:08 Message: Logged In: YES user_id=80475 Jiwon, while working through the doc patch, I noticed a code generation nit. The code g=(i for i in range(10)) gets translated to: def __gen(exp): for i in exp: yield i g = __gen(iter(range(10)) del __gen On the next to last line, the call to iter() may be superfluous. Do you see a need for it? If so, I need to include that in the spec. If not, then it should be removed (line 1822 in compile.c). In terms of byte codes, the difference shows up in a comparison to an equivalent generator: >>> dis(compile('g(range(10))', '', 'eval')) 0 0 LOAD_NAME 0 (g) 3 LOAD_NAME 1 (range) 6 LOAD_CONST 0 (10) 9 CALL_FUNCTION 1 12 CALL_FUNCTION 1 15 RETURN_VALUE >>> dis(compile('(i for i in range(10))', '', 'eval')) 0 0 LOAD_CONST 0 ( at 00A4B220, file "", line 1>) 3 MAKE_FUNCTION 0 6 LOAD_NAME 0 (range) 9 LOAD_CONST 1 (10) 12 CALL_FUNCTION 1 15 GET_ITER 16 CALL_FUNCTION 1 19 RETURN_VALUE The get_iter at code position 15 is what is in question. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-20 11:51 Message: Logged In: YES user_id=595483 Thank you, and I'm happy to contribute. =) I updated documentation so that it deals with the precomputation issue. But I don't know how much I should write or where, so please review/revise it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-19 03:20 Message: Logged In: YES user_id=80475 Thank you for the superb contribution. It passes all of my tests without exception (see the new file, test_genexps). I do not see any issues with the coding. Am accepting and applying the patch except for the documentation. Please update the documentation part of the patch and I will go through and clarify where necessary. Also, I will update the PEP to reflect Guido's rationale for the design decisions on binding behavior. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-15 12:24 Message: Logged In: YES user_id=595483 OK. I added outmost-iterable-precomputation feature to lazy-binding version of the patch, and also added related tests to test_grammar.py. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-15 01:08 Message: Logged In: YES user_id=595483 Ok. I see. I've almost finished adding it(outmost iterable precomputation) to CPython, and now trying to do it in python compiler package. Probably I'll commit patch till tomorrow. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-14 19:31 Message: Logged In: YES user_id=80475 Jiwon, do you see how to proceed for here? Note that given, (for x in exp1 if exp2 for y in exp3 if exp4), only exp1 is evaluated immediately. exp2 is deferred. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 22:58 Message: Logged In: YES user_id=6380 On why the first expr should be evaluated immediately: Consider sum(x for x in foo()). Now suppose there's a bug in foo() that raises an exception, and a bug in sum() that raises an exception before it starts iterating over its argument. Which exception would you expect to see? I'd be surprised if the one in sum() was raised rather the one in foo(), since the call to foo() is part of the argument to sum(), and I expect arguments to be processed before the function is called. OTOH, in sum(bar(x) for x in foo()), where sum() and foo() are bugfree, but bar() raises an exception, we have no choice but to delay the call to bar() until sum() starts iterating -- that's part of the contract of generators. (They do nothing until their next() method is first called.) Ditto for subsequent iterables if nested for loops are used: in sum(x*y for x in range(10) for y in bar(x)), there are 10 calls to bar(), with arguments supplied by the first (outer) for loop, and we have no choice but to delay even the first call to bar() until sum() calls next() on its argument. I know this isn't the easiest thing to implement, but I still think it's the right thing. The generator function for the last example would be something like def G(arg): for x in arg: for y in bar(x): yield x*y and the call to sum() would be sum(G(range(10))) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-12 13:54 Message: Logged In: YES user_id=80475 Jeremy: I reviewed the lazy binding version of the patch as much as I could. Can you take a brief look at just the changes to the grammar and at the new functions in compile.c. The functions were clearly written and neatly paralleled list comps but I wondered whether it was possible and desirable to factor-out the commonalities. Jiwon: the graminit.h renumbering is fine (all of the automatically generated files should be left as-as). Guido: I was clear on your rationale for preferring late binding but did not find wording for the reason behind the precompution of the innermost iterable. I speculate that the semantics reflect that this is an expression and that all expressions (except and/or) are immediate. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 12:36 Message: Logged In: YES user_id=6380 Read back what I wrote on python-dev about capturing free variables as a new concept with which I am uncomfortable. There should not be such "advanced" usages of genexprs -- advanced usages are better served by explicit generator functions. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-12 10:26 Message: Logged In: YES user_id=4771 Which is exactly why I cannot understand why you are so keen on one binding model rather than the other, if it doesn't matter. It looks like we are going to add is a feature that doesn't scale to advanced usages. Moreover it seems really obvious that non-experts *will* get surprizes, because they *will* try to replace all their listcomps with genexprs -- they shouldn't do that, but of course they will do it anyway, because it appears to work -- until the occasional strange bug kreeps in, and one that is difficult to actually relate to the genexpr in the first place. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 09:03 Message: Logged In: YES user_id=6380 You're just showing that saving genexprs up for later use is a bad idea. We should place clear warnings in the docs that the genexpr should be used up right away, and otherwise to write an explicit generator function. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-12 03:30 Message: Logged In: YES user_id=595483 I don't think that "lazy-binding + leftmost iterable precomputation" makes sense. Just adding another for-loop to the example shows the reason. >>> x = 10 >>> g = ((i,j) for i in range(x) for j in range(x)) >>> x = 5 >>> list(g) Here, if j is iterated only to 5 when i is iterated to 10, I think it would make users confused. I think "early-binding + leftmost iterable precomputation" makes sense, but "lazy-binding + leftmost iterable precomputation" does not make sense IMHO. About renumbering testlist_gexp in graminit.h, it's not what I did but it's auto-generated by pgen(Parser/pgenmain.c). Although it makes another file (symbol.py) need to be changed (and thus makes cvs log a bit dirty), it's tool- generated file, so I did it like that. If you think it's better to do it without re-numbering, let me know or you could do it yourself. ;^) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-12 02:32 Message: Logged In: YES user_id=80475 The lazy_bind version has survived over of a month of my testing and multiple read throughs of the code :-) I'm not expert with graminit.h but did wonder if testlist_gexp could have a high numbered code so that other codes would not have to be re-numbered. When you get back from military camp, please load the version that precomputes the outermost iterable. I believe that is Guido's currently favored approach. Currently, it fails this test: >>> x = 10 >>> g = (i*i for i in range(x)) >>> x = 5 >>> list(g) # should be ten items long [0, 1, 4, 9, 16] Tim, summarized it as: """ ... Guido favored mostly-late binding -- which is late binding, except that the iterable in the leftmost for-clause is evaluated at once. So ... (e1 for stuff in e2 for morestuff in e3 ...) ... is effectively replaced by def __g(primary): for stuff in primary: for morestuff in e3: ... yield e1 ... __g(e2) ... """ ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-27 10:56 Message: Logged In: YES user_id=595483 I'm submitting two versions now. One is variable-capture version(but without any iterable pre-computation), and the other is lazy-binding version. I'll be in military camp for about 1 month(4/6~5/2) so I will not be able to fix/patch anything for the period, thus I'm submitting this in now. :) If I have time, and necessity arises (before 4/6), I'll also add outmost-iterable-precomputation version. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-03-22 18:00 Message: Logged In: YES user_id=6380 Sorry, I meant the first, not the last. I was confused about how the 'for' clauses are nested, but the outermost one is the first. So the nesting remark is probably just confusing and wrong; ignore it. What I meant is: (f(x,y) for x in A() for y in B(x)) should IMO precompute A(), but not B(x). I guess the equivalent generator function would be: def __gen(__outer__=A(), f=f, B=B): for x in __outer__: for y in B(x): yield f(x,y) In general the value of every free variable used anywhere except in the outer expression should be captured; the *value* of the outer expression should be captured. This should give the least surprising semantics in a variaty of use cases. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-03-22 14:39 Message: Logged In: YES user_id=6380 Only the outermost (last) iterable should be precomputed. While (f(x,y) for x in A(y) for y in B) is not the same as ((f(x,y) for x in A(y)) for y in B) I think the scoping should be done similarly. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-21 20:57 Message: Logged In: YES user_id=595483 Ah... Maybe we need to drop precomputation of iterables. I'll post it on the mailing list so that people can consider about that. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-03-21 18:40 Message: Logged In: YES user_id=4771 Oh ho. Am I right in fearing that the idea of precomputing the iterables was broken in the first place? We just cannot do this. The things we are looping over may depend on other stuff from the generator expression itself. Example: >>> [x for l in [[1,2],[3,4]] for x in l] [1, 2, 3, 4] >>> (y for m in [[1,2],[3,4]] for y in m) NameError: name 'm' is not defined ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-21 12:55 Message: Logged In: YES user_id=55188 This inconsistency may not be fixed by list(iter()) workaround, either. >>> def counter(): ... counter.count += 1 ... return range(counter.count) ... >>> counter.count = 0 >>> [y for x in 'abc' for y in counter()] [0, 0, 1, 0, 1, 2] >>> counter.count = 0 >>> list(y for x in 'abc' for y in counter()) [0, 0, 0] ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-21 12:42 Message: Logged In: YES user_id=55188 Aah. But I'm not insisting on that because it's incompatible even with plain nested for-loops. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-21 12:38 Message: Logged In: YES user_id=55188 Agreed. I'd prefer the current implementation's behavor rather than defined in PEP289. Yes, It's incompatible with list comprehensions but generator expressions are already quite different enough from it. :) How about to change PEP289's genexpr semantics to this? g = (x**2 for x in range(10)) print g.next() is equivalent to: def __gen(_i1): for x in _i1: yield x**2 g = __gen(range(10)) print g.next() ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-21 12:28 Message: Logged In: YES user_id=80475 You need a solution other than list(). A key to the success of genexps is to never to eat memory by expanding an iterator into a container. For behavior questions, your reference point is the list comprehension. list(genexp) should always produce the same result as a list comprehension. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-21 11:59 Message: Logged In: YES user_id=595483 Hi, quiver. I don't think we can easily go around this problem if we have to capture iterators in generator expression. If you run following, you'll know what I mean. >>> a = iter("abcd") >>> b = iter("abcd") >>> [(x, y) for x in a for y in b] [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] I think one possible solution could be, instead of passing evaluations of iterators in generator expression, make a list from iterator and, then pass it as argument. For instance, g = (x for x in iter("abcd")) will be equivalent to, def __gen(_[1]): for x in _[1]: yield x g = __gen(list(iter("abcd"))) # see 'list' - instead of g = __gen(iter("abcd")) . I'm not sure if I'm in a position to decide to do that way or not. If the current reviewer (rhettinger) approves it, I'll do that way. Or else, I think I will post it on the mailing list. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-03-19 08:37 Message: Logged In: YES user_id=671362 Hi, Jiwon. This is another bug candidate. If you use genexp with iterators, it doesn't work well. # test Perky's previous post using iterators >>> list((x,y) for x in iter('abcd') for y in iter('abcd')) [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] Thanks. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-17 03:45 Message: Logged In: YES user_id=595483 To fix the bug that perky picked out, I hand-made ast.py instead of using auto-generated code. But, still I don't understand why Tools/compiler/regrtest didn't tell me about it. (Maybe I didn't look at the output carefully enough.) Ah, and it would be nice if somebody tell me how to run test_grammar.py(only) with python-compiler package. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-17 01:28 Message: Logged In: YES user_id=55188 The compiler package patch has some problems in its compiler/ast.py currently. jiwon said he regenerated it using Tools/compiler/astgen.py. But it made some incompatibilities against ast.py on current CVS. For example, class Expression. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-17 01:21 Message: Logged In: YES user_id=80475 I'll give it a second review. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-17 00:37 Message: Logged In: YES user_id=55188 Okay. Here's my draft for documentation. Any review/fix/enhance is very welcome! I think it's ready to putting it into CVS now. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-16 11:24 Message: Logged In: YES user_id=595483 ah, the following bug was due to the patch I uploaded 2004-02-04 15:13. In order to get an error instantly from an expression like "g=(x for x in None)", I made it equivalent to, def __gen(_[1]): for x in _[1]: yield x g=__gen(iter(None)) ^^^^ But when there is two or more iterator expression of same symbol(or string), that patch produces error, because currently, "((x, y) for x in 'abcd' for y in 'abcd') " is equivalent to, def __gen(_[1]): for x in _[1]: for y in _[1]: yield (x,y) g = __gen(iter("abcd")) # passing only one parameter I could make it pass every iterator expressions respectively even when they are same symbol(or string, ...), but for now, I just dropped that patch (patch of 2004-02-04) since that's the simplest thing now. If somebody says getting instance error for cases like "(x for x in None)" is important, I'll make another patch for it. Btw, added more test case that covers what perky mentioned. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-15 23:16 Message: Logged In: YES user_id=55188 Another bug in the implementation. >>> list((x, y) for x in 'abcd' for y in 'abcd') [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] Expected behavior may be: >>> [(x, y) for x in 'abcd' for y in 'abcd'] [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd'), ('b', 'a'), ('b', 'b'), ('b', 'c'), ('b', 'd'), ('c', 'a'), ('c', 'b'), ('c', 'c'), ('c', 'd'), ('d', 'a'), ('d', 'b'), ('d', 'c'), ('d', 'd')] ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-13 21:59 Message: Logged In: YES user_id=55188 I'm writing docs for tutorial and language reference. It'll be completed in a day or two. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 16:17 Message: Logged In: YES user_id=80475 Any recent progress? ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-23 11:34 Message: Logged In: YES user_id=595483 Whoa! I finally patched compiler package in pure python. (I was a bit busy recently :) I've run regression test with 'Tools/compiler/regrtest.py' before I submit this patch, and there was one failure (which is test_dis.py). I'm not sure if that's a problem, so I'm just submitting this in. Now, I think I'll refactor things a bit! ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-02-04 08:53 Message: Logged In: YES user_id=55188 As it mentioned in PEP, even listcomp's leaking of loop value will be dropped in Python 3. I'm sorry but I don't see that the usage is solid in the future. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-04 05:45 Message: Logged In: YES user_id=671362 What about this code? In the currently implementation, loop variables inside a list comprehension is not visible outside if you use it inside a generator expression. For example: >>> (a*b for a in [b for b in range(5)]) Traceback (most recent call last): File "", line 1, in ? NameError: name 'b' is not defined Its list comprehension counterpart is: >>> [a*b for a in [b for b in range(5)]] [0, 4, 8, 12, 16] ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-04 01:13 Message: Logged In: YES user_id=595483 Again, I fixed the patch so that it can get error from '(x for x in None)' immediately. (upto now, error does not occur until generator expression is evaluated) ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-02-03 06:46 Message: Logged In: YES user_id=4771 After thinking a bit more on the issue, note that the generator version of your example is equivalent to: def g(): for y in range(3): yield lambda x: x+y which means that it can suffer from the same problem as the first example, but more subtly (and even more confusingly): for f in g(): print f(0) # 0, 1, 2 for f in list(g()): print f(0) # 2, 2, 2 This is because due to Python's nested scope rules the above generator is equivalent to: def g(): result = lambda x: x+y for y in range(3): yield result at which point I think it gets pretty confusing... ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-03 04:07 Message: Logged In: YES user_id=671362 Thanks, Arigo and Perky. Hmm, maybe I should read PEP and the thread about namespace more carefully, not just skim the surface. Anyway, PEP has not been updated since last October, so I think it's good time to merge recent progress of genexpr and update PEP-289. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-02-02 07:58 Message: Logged In: YES user_id=4771 The behavior is indeed the one described by the PEP but it is still surprising. As far as I'm concerned it is yet another reason why free variable bindings ("nested scopes") are done wrong in Python currently :-( If you use the older trick "lambda x,y=y:x+y" instead, then both cases will agree (with the sensible result in my opinion). A few more issues and I'll start a campain for definition-time bindings of free variables :-( ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-02-02 07:37 Message: Logged In: YES user_id=55188 I think it's right for namespace difference between listcomp and genexpr. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-02 06:47 Message: Logged In: YES user_id=671362 I am not sure if I should call this a bug, but here it goes: >>> lst = [lambda x:x+y for y in range(3)] >>> for f in lst:print f(0) ... 2 2 2 >>> gen = (lambda x:x+y for y in range(3)) >>> for f in gen:print f(0) ... 0 1 2 Is this the intended behavior? ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-02 03:29 Message: Logged In: YES user_id=595483 ok. I fixed another bug reported by perky, and added related test to test_grammar.py. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-30 00:09 Message: Logged In: YES user_id=55188 Yet another Fatal Python error; >>> (a for a in (b for b in (a for a in 'xxx' if True) if False) if True) lookup 'True' in ? 3 -1 freevars of : ('True',) Fatal Python error: com_make_closure() zsh: abort (core dumped) ./python # applied patch as of 2004-01-30 ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-29 22:01 Message: Logged In: YES user_id=595483 ok, I've fixed the bug quiver commented, and added related test code to test_grammar.py. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-01-29 06:48 Message: Logged In: YES user_id=671362 yet another Fatal Python error; >>> (a for a in (b for b in (0,1))) >>> (a for a in (b for b in range(2))) Fatal Python error: unknown scope for range in ?(0) in symbols: {'range': 8} locals: {} globals: {} Aborted # applied patch as of 2004-01-27 ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-27 02:44 Message: Logged In: YES user_id=595483 I fixed the patch for the bug that arigo mentioned, and for what perky mentioned - PyList_GetSlice error handling - . now, I'll tackle python compiler package :) ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-01-26 10:15 Message: Logged In: YES user_id=4771 >>> (a for d in a) Fatal Python error: unknown scope for a in (1) in symbols: {'a': 2048, '_[1]': 4, 'd': 2} locals: {'_[1]': 0, 'd': 1} globals: {} ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-26 01:17 Message: Logged In: YES user_id=55188 Please ignore the previous comment. It was a result from an old revision of patches. It works on the recent. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-25 18:44 Message: Logged In: YES user_id=55188 BTW, does unavaliability of yield (genexpr) and return (genexpr) an intended behavior? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-01-21 13:13 Message: Logged In: YES user_id=6656 Documentation would be nice! ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-21 09:14 Message: Logged In: YES user_id=55188 Okay. My review is done. The revised patch "gexp.diff" looks fine for me. There're few minor tweaks still needed to get into CVS. - Some error exit cases are ignored. You need to provide safe exit paths for MemoryError. (eg. PyList_GetSlice usage of Python/ compiler.c) - A patch for 'compiler' pure python package. (there's an old implementation on SF #795947) ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-19 06:10 Message: Logged In: YES user_id=595483 ok. I modified the patch so that it evaluates iterator expr in generator expression creation time. That means, g = (x for x in range(10)) is equivalent to def __gen(iter1): for x in iter1: yield x g = __gen(range(10)) so, evalution of range(10) is not deferred anymore. I also changed testgenexpr to testlist_gexp in Grammar/Grammar, since Hye-Shik Chang advised as such. I'm willing to take any advice about this patch, so please do. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-11 22:26 Message: Logged In: YES user_id=595483 ok. I've implemented capturing variables part as arigo suggested. File genexpr-capture-vars-in-args.diff is that. If you look at the code, you'll find that the code for generator expression is much shorter, and there's lesser modification in the existing code. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-08 20:49 Message: Logged In: YES user_id=595483 What arigo wrote sounds reasonable, and not very difficult to implement. I'll try to do that way. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-01-08 13:50 Message: Logged In: YES user_id=4771 We may not need two levels of nested anonymous functions. It seems to me that the following equivalent code would do, because it captures the variable in an argument instead of via nested scopes: x = 1 def __gen(x): for i in range(10): yield x g = __gen(x) I don't know though if this is easy to implement in compile.c. Alternatively: x = 1 def __gen(x=x): for i in range(10): yield x g = __gen() ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-08 00:44 Message: Logged In: YES user_id=55188 Okay. I verified that basic characteristics mentioned on PEP are working. I started to review the implementation. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-07 23:50 Message: Logged In: YES user_id=595483 I added diff of Lib/symbol.py, so no need to run python Lib/symbol.py now. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-07 07:25 Message: Logged In: YES user_id=55188 Assigned to me. The originator is my friend and I have much interest on this. :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470 From noreply at sourceforge.net Thu May 20 16:35:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 16:35:47 2004 Subject: [Patches] [ python-Patches-957650 ] Fix for bugs relating to ntpath.expanduser() Message-ID: Patches item #957650, was opened at 2004-05-20 13:35 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957650&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Josiah Carlson (josiahcarlson) Assigned to: Nobody/Anonymous (nobody) Summary: Fix for bugs relating to ntpath.expanduser() Initial Comment: Attached is a patch for sf bug #796219 that fixes ntpath.expanduser() for paths that embed other environment variables, and also includes functionality that mirrors *nix-style ~user\extra expansions. I will comment with output from both the unpatched and patched version of ntpath. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957650&group_id=5470 From noreply at sourceforge.net Thu May 20 16:36:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 16:36:29 2004 Subject: [Patches] [ python-Patches-957650 ] Fix for bugs relating to ntpath.expanduser() Message-ID: Patches item #957650, was opened at 2004-05-20 13:35 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957650&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Josiah Carlson (josiahcarlson) Assigned to: Nobody/Anonymous (nobody) Summary: Fix for bugs relating to ntpath.expanduser() Initial Comment: Attached is a patch for sf bug #796219 that fixes ntpath.expanduser() for paths that embed other environment variables, and also includes functionality that mirrors *nix-style ~user\extra expansions. I will comment with output from both the unpatched and patched version of ntpath. ---------------------------------------------------------------------- >Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-20 13:36 Message: Logged In: YES user_id=341410 #test setup: >>> import os >>> os.environ['TESTING'] = '%TESTING1%' >>> os.environ['TESTING1'] = '%TESTING2%' >>> os.environ['TESTING2'] = 'Final\Path' #test standard ntpath >>> import ntpath >>> ntpath.expanduser('~') 'C:\Documents and Settings\jcarlson' >>> ntpath.expanduser('~billy') '~billy' >>> ntpath.expanduser('~billy\bob') '~billy\bob' >>> ntpath.expanduser('~\bob') 'C:\Documents and Settings\jcarlson\bob' >>> ntpath.expanduser('~billy\%TESTING%\%TESTING1%\%TESTING2%') '~billy\%TESTING%\%TESTING1%\%TESTING2%' #test patched ntpath >>> import ntpath_patched >>> ntpath_patched.expanduser('~') 'C:Documents and Settings\jcarlson' >>> ntpath_patched.expanduser('~billy') 'C:Documents and Settings\billy' >>> ntpath_patched.expanduser('~billy\bob') 'C:Documents and Settings\billy\bob' >>> ntpath_patched.expanduser('~\bob') 'C:Documents and Settings\jcarlson\bob' >>> ntpath_patched.expanduser('~billy\%TESTING%\%TESTING1%\%TESTING2%') 'C:Documents and Settings\billy\Final\Path\Final\Path\Final\Path' ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957650&group_id=5470 From noreply at sourceforge.net Thu May 20 16:38:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 16:38:20 2004 Subject: [Patches] [ python-Patches-957650 ] Fix for bugs relating to ntpath.expanduser() Message-ID: Patches item #957650, was opened at 2004-05-20 13:35 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957650&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Josiah Carlson (josiahcarlson) Assigned to: Nobody/Anonymous (nobody) Summary: Fix for bugs relating to ntpath.expanduser() Initial Comment: Attached is a patch for sf bug #796219 that fixes ntpath.expanduser() for paths that embed other environment variables, and also includes functionality that mirrors *nix-style ~user\extra expansions. I will comment with output from both the unpatched and patched version of ntpath. ---------------------------------------------------------------------- >Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-20 13:38 Message: Logged In: YES user_id=341410 I uploaded the testing as text to alleviate text wrapping issues that could confuse. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-20 13:36 Message: Logged In: YES user_id=341410 #test setup: >>> import os >>> os.environ['TESTING'] = '%TESTING1%' >>> os.environ['TESTING1'] = '%TESTING2%' >>> os.environ['TESTING2'] = 'Final\Path' #test standard ntpath >>> import ntpath >>> ntpath.expanduser('~') 'C:\Documents and Settings\jcarlson' >>> ntpath.expanduser('~billy') '~billy' >>> ntpath.expanduser('~billy\bob') '~billy\bob' >>> ntpath.expanduser('~\bob') 'C:\Documents and Settings\jcarlson\bob' >>> ntpath.expanduser('~billy\%TESTING%\%TESTING1%\%TESTING2%') '~billy\%TESTING%\%TESTING1%\%TESTING2%' #test patched ntpath >>> import ntpath_patched >>> ntpath_patched.expanduser('~') 'C:Documents and Settings\jcarlson' >>> ntpath_patched.expanduser('~billy') 'C:Documents and Settings\billy' >>> ntpath_patched.expanduser('~billy\bob') 'C:Documents and Settings\billy\bob' >>> ntpath_patched.expanduser('~\bob') 'C:Documents and Settings\jcarlson\bob' >>> ntpath_patched.expanduser('~billy\%TESTING%\%TESTING1%\%TESTING2%') 'C:Documents and Settings\billy\Final\Path\Final\Path\Final\Path' ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957650&group_id=5470 From noreply at sourceforge.net Thu May 20 19:07:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 19:07:31 2004 Subject: [Patches] [ python-Patches-872326 ] generator expression implementation Message-ID: Patches item #872326, was opened at 2004-01-07 07:10 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470 Category: Parser/Compiler Group: None >Status: Closed Resolution: Accepted Priority: 5 Submitted By: Jiwon Seo (jiwon) Assigned to: Nobody/Anonymous (nobody) Summary: generator expression implementation Initial Comment: Since I was interested in pep 289(generator expression), I dabbled with it, and implemented a working version of it. I'm not sure if I did it right, but maybe someone who knows better can fix it right. 1. Grammar has two changes, which is a. atom: '(' [testlist] ')' | '[' [listmaker] ']' | ... changes to atom: '(' [testgenexpr] ')' | '[' [listmaker] ']' | ... where testgenexpr defines like this. testgenexpr: test ( gen_for | (',' test)* [','] ) b. argument: [test '='] test changes to argument: [test '='] test [gen_for] (gen_for, gen_if, gen_iter is similar to list_for, list_if, list_iter respectively.) 2. Instead of changing rule of arglist in Grammar to accept generator expression, I changed argument rule like 1.b. This means Grammar accepts generator expression without parenthesis in function call even there are several arguments, like reduce(operator.add, (x for x in range(10))) This is against what pep requires, so I made parsermodule.c and compile.c to catch that and throw error message when there's more than one argument in a function. The reason I did it that way is to go around a problem of ambiguity in the grammar by adding generator expression to arglist. 3. I implemented generator expression as equivalent to a call to a function which has variable capturing code and generator code. For example, x = 1 g = (x for i in range(10)) is equivalent to x = 1 def __gen_wrapper(): _[x] = x # variable capture here def __gen(): for i in range(10): yield _[x] return __gen() g = __gen_wrapper() 4. Since I implemented generator expression equivalent to a function call, symbol table should enter new scope when it meets generator expression. So I ended up with adding following code to symtable.c PyObject * PySymtableEntry_New(struct symtable *st, char *name, int type, int lineno) { ... switch (type) { case funcdef: case lambdef: ! case testgenexpr: /* generator expression */ ! case argument: /* generator expression */ ste->ste_type = TYPE_FUNCTION; break; ... so that generator expression can be dealt as function type. This is kind of stupid, but I couldn't find other easy for it, so I just left it like that. 5. this patch does not include diff of src/Lib/symbol.py, so you must run python Lib/symbol.py to get it right. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 18:07 Message: Logged In: YES user_id=80475 Please disregard the last two posts. Guido and I agreed that the current behavior is correct. I've updated the docs accordingly and added a test. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 12:32 Message: Logged In: YES user_id=80475 FWIW, here is a patch that moves the GET_ITER back inside the generator so that the implementation will more closely match the spec and match the operation of regular generators. *** compile.c 19 May 2004 08:20:15 -0000 2.302 --- compile.c 20 May 2004 17:29:35 -0000 *************** *** 1628,1633 **** --- 1628,1634 ---- if (is_outmost) { com_addop_varname(c, VAR_LOAD, "[outmost-iterable]"); + com_addbyte(c, GET_ITER); com_push(c, 1); } else { *************** *** 1819,1825 **** com_addoparg(c, MAKE_FUNCTION, 0); com_test(c, CHILD(CHILD(n, 1), 3)); - com_addbyte(c, GET_ITER); com_addoparg(c, CALL_FUNCTION, 1); com_pop(c, 1); --- 1820,1825 ---- ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 12:08 Message: Logged In: YES user_id=80475 Jiwon, while working through the doc patch, I noticed a code generation nit. The code g=(i for i in range(10)) gets translated to: def __gen(exp): for i in exp: yield i g = __gen(iter(range(10)) del __gen On the next to last line, the call to iter() may be superfluous. Do you see a need for it? If so, I need to include that in the spec. If not, then it should be removed (line 1822 in compile.c). In terms of byte codes, the difference shows up in a comparison to an equivalent generator: >>> dis(compile('g(range(10))', '', 'eval')) 0 0 LOAD_NAME 0 (g) 3 LOAD_NAME 1 (range) 6 LOAD_CONST 0 (10) 9 CALL_FUNCTION 1 12 CALL_FUNCTION 1 15 RETURN_VALUE >>> dis(compile('(i for i in range(10))', '', 'eval')) 0 0 LOAD_CONST 0 ( at 00A4B220, file "", line 1>) 3 MAKE_FUNCTION 0 6 LOAD_NAME 0 (range) 9 LOAD_CONST 1 (10) 12 CALL_FUNCTION 1 15 GET_ITER 16 CALL_FUNCTION 1 19 RETURN_VALUE The get_iter at code position 15 is what is in question. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-20 11:51 Message: Logged In: YES user_id=595483 Thank you, and I'm happy to contribute. =) I updated documentation so that it deals with the precomputation issue. But I don't know how much I should write or where, so please review/revise it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-19 03:20 Message: Logged In: YES user_id=80475 Thank you for the superb contribution. It passes all of my tests without exception (see the new file, test_genexps). I do not see any issues with the coding. Am accepting and applying the patch except for the documentation. Please update the documentation part of the patch and I will go through and clarify where necessary. Also, I will update the PEP to reflect Guido's rationale for the design decisions on binding behavior. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-15 12:24 Message: Logged In: YES user_id=595483 OK. I added outmost-iterable-precomputation feature to lazy-binding version of the patch, and also added related tests to test_grammar.py. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-15 01:08 Message: Logged In: YES user_id=595483 Ok. I see. I've almost finished adding it(outmost iterable precomputation) to CPython, and now trying to do it in python compiler package. Probably I'll commit patch till tomorrow. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-14 19:31 Message: Logged In: YES user_id=80475 Jiwon, do you see how to proceed for here? Note that given, (for x in exp1 if exp2 for y in exp3 if exp4), only exp1 is evaluated immediately. exp2 is deferred. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 22:58 Message: Logged In: YES user_id=6380 On why the first expr should be evaluated immediately: Consider sum(x for x in foo()). Now suppose there's a bug in foo() that raises an exception, and a bug in sum() that raises an exception before it starts iterating over its argument. Which exception would you expect to see? I'd be surprised if the one in sum() was raised rather the one in foo(), since the call to foo() is part of the argument to sum(), and I expect arguments to be processed before the function is called. OTOH, in sum(bar(x) for x in foo()), where sum() and foo() are bugfree, but bar() raises an exception, we have no choice but to delay the call to bar() until sum() starts iterating -- that's part of the contract of generators. (They do nothing until their next() method is first called.) Ditto for subsequent iterables if nested for loops are used: in sum(x*y for x in range(10) for y in bar(x)), there are 10 calls to bar(), with arguments supplied by the first (outer) for loop, and we have no choice but to delay even the first call to bar() until sum() calls next() on its argument. I know this isn't the easiest thing to implement, but I still think it's the right thing. The generator function for the last example would be something like def G(arg): for x in arg: for y in bar(x): yield x*y and the call to sum() would be sum(G(range(10))) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-12 13:54 Message: Logged In: YES user_id=80475 Jeremy: I reviewed the lazy binding version of the patch as much as I could. Can you take a brief look at just the changes to the grammar and at the new functions in compile.c. The functions were clearly written and neatly paralleled list comps but I wondered whether it was possible and desirable to factor-out the commonalities. Jiwon: the graminit.h renumbering is fine (all of the automatically generated files should be left as-as). Guido: I was clear on your rationale for preferring late binding but did not find wording for the reason behind the precompution of the innermost iterable. I speculate that the semantics reflect that this is an expression and that all expressions (except and/or) are immediate. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 12:36 Message: Logged In: YES user_id=6380 Read back what I wrote on python-dev about capturing free variables as a new concept with which I am uncomfortable. There should not be such "advanced" usages of genexprs -- advanced usages are better served by explicit generator functions. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-12 10:26 Message: Logged In: YES user_id=4771 Which is exactly why I cannot understand why you are so keen on one binding model rather than the other, if it doesn't matter. It looks like we are going to add is a feature that doesn't scale to advanced usages. Moreover it seems really obvious that non-experts *will* get surprizes, because they *will* try to replace all their listcomps with genexprs -- they shouldn't do that, but of course they will do it anyway, because it appears to work -- until the occasional strange bug kreeps in, and one that is difficult to actually relate to the genexpr in the first place. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 09:03 Message: Logged In: YES user_id=6380 You're just showing that saving genexprs up for later use is a bad idea. We should place clear warnings in the docs that the genexpr should be used up right away, and otherwise to write an explicit generator function. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-12 03:30 Message: Logged In: YES user_id=595483 I don't think that "lazy-binding + leftmost iterable precomputation" makes sense. Just adding another for-loop to the example shows the reason. >>> x = 10 >>> g = ((i,j) for i in range(x) for j in range(x)) >>> x = 5 >>> list(g) Here, if j is iterated only to 5 when i is iterated to 10, I think it would make users confused. I think "early-binding + leftmost iterable precomputation" makes sense, but "lazy-binding + leftmost iterable precomputation" does not make sense IMHO. About renumbering testlist_gexp in graminit.h, it's not what I did but it's auto-generated by pgen(Parser/pgenmain.c). Although it makes another file (symbol.py) need to be changed (and thus makes cvs log a bit dirty), it's tool- generated file, so I did it like that. If you think it's better to do it without re-numbering, let me know or you could do it yourself. ;^) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-12 02:32 Message: Logged In: YES user_id=80475 The lazy_bind version has survived over of a month of my testing and multiple read throughs of the code :-) I'm not expert with graminit.h but did wonder if testlist_gexp could have a high numbered code so that other codes would not have to be re-numbered. When you get back from military camp, please load the version that precomputes the outermost iterable. I believe that is Guido's currently favored approach. Currently, it fails this test: >>> x = 10 >>> g = (i*i for i in range(x)) >>> x = 5 >>> list(g) # should be ten items long [0, 1, 4, 9, 16] Tim, summarized it as: """ ... Guido favored mostly-late binding -- which is late binding, except that the iterable in the leftmost for-clause is evaluated at once. So ... (e1 for stuff in e2 for morestuff in e3 ...) ... is effectively replaced by def __g(primary): for stuff in primary: for morestuff in e3: ... yield e1 ... __g(e2) ... """ ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-27 10:56 Message: Logged In: YES user_id=595483 I'm submitting two versions now. One is variable-capture version(but without any iterable pre-computation), and the other is lazy-binding version. I'll be in military camp for about 1 month(4/6~5/2) so I will not be able to fix/patch anything for the period, thus I'm submitting this in now. :) If I have time, and necessity arises (before 4/6), I'll also add outmost-iterable-precomputation version. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-03-22 18:00 Message: Logged In: YES user_id=6380 Sorry, I meant the first, not the last. I was confused about how the 'for' clauses are nested, but the outermost one is the first. So the nesting remark is probably just confusing and wrong; ignore it. What I meant is: (f(x,y) for x in A() for y in B(x)) should IMO precompute A(), but not B(x). I guess the equivalent generator function would be: def __gen(__outer__=A(), f=f, B=B): for x in __outer__: for y in B(x): yield f(x,y) In general the value of every free variable used anywhere except in the outer expression should be captured; the *value* of the outer expression should be captured. This should give the least surprising semantics in a variaty of use cases. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-03-22 14:39 Message: Logged In: YES user_id=6380 Only the outermost (last) iterable should be precomputed. While (f(x,y) for x in A(y) for y in B) is not the same as ((f(x,y) for x in A(y)) for y in B) I think the scoping should be done similarly. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-21 20:57 Message: Logged In: YES user_id=595483 Ah... Maybe we need to drop precomputation of iterables. I'll post it on the mailing list so that people can consider about that. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-03-21 18:40 Message: Logged In: YES user_id=4771 Oh ho. Am I right in fearing that the idea of precomputing the iterables was broken in the first place? We just cannot do this. The things we are looping over may depend on other stuff from the generator expression itself. Example: >>> [x for l in [[1,2],[3,4]] for x in l] [1, 2, 3, 4] >>> (y for m in [[1,2],[3,4]] for y in m) NameError: name 'm' is not defined ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-21 12:55 Message: Logged In: YES user_id=55188 This inconsistency may not be fixed by list(iter()) workaround, either. >>> def counter(): ... counter.count += 1 ... return range(counter.count) ... >>> counter.count = 0 >>> [y for x in 'abc' for y in counter()] [0, 0, 1, 0, 1, 2] >>> counter.count = 0 >>> list(y for x in 'abc' for y in counter()) [0, 0, 0] ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-21 12:42 Message: Logged In: YES user_id=55188 Aah. But I'm not insisting on that because it's incompatible even with plain nested for-loops. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-21 12:38 Message: Logged In: YES user_id=55188 Agreed. I'd prefer the current implementation's behavor rather than defined in PEP289. Yes, It's incompatible with list comprehensions but generator expressions are already quite different enough from it. :) How about to change PEP289's genexpr semantics to this? g = (x**2 for x in range(10)) print g.next() is equivalent to: def __gen(_i1): for x in _i1: yield x**2 g = __gen(range(10)) print g.next() ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-21 12:28 Message: Logged In: YES user_id=80475 You need a solution other than list(). A key to the success of genexps is to never to eat memory by expanding an iterator into a container. For behavior questions, your reference point is the list comprehension. list(genexp) should always produce the same result as a list comprehension. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-21 11:59 Message: Logged In: YES user_id=595483 Hi, quiver. I don't think we can easily go around this problem if we have to capture iterators in generator expression. If you run following, you'll know what I mean. >>> a = iter("abcd") >>> b = iter("abcd") >>> [(x, y) for x in a for y in b] [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] I think one possible solution could be, instead of passing evaluations of iterators in generator expression, make a list from iterator and, then pass it as argument. For instance, g = (x for x in iter("abcd")) will be equivalent to, def __gen(_[1]): for x in _[1]: yield x g = __gen(list(iter("abcd"))) # see 'list' - instead of g = __gen(iter("abcd")) . I'm not sure if I'm in a position to decide to do that way or not. If the current reviewer (rhettinger) approves it, I'll do that way. Or else, I think I will post it on the mailing list. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-03-19 08:37 Message: Logged In: YES user_id=671362 Hi, Jiwon. This is another bug candidate. If you use genexp with iterators, it doesn't work well. # test Perky's previous post using iterators >>> list((x,y) for x in iter('abcd') for y in iter('abcd')) [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] Thanks. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-17 03:45 Message: Logged In: YES user_id=595483 To fix the bug that perky picked out, I hand-made ast.py instead of using auto-generated code. But, still I don't understand why Tools/compiler/regrtest didn't tell me about it. (Maybe I didn't look at the output carefully enough.) Ah, and it would be nice if somebody tell me how to run test_grammar.py(only) with python-compiler package. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-17 01:28 Message: Logged In: YES user_id=55188 The compiler package patch has some problems in its compiler/ast.py currently. jiwon said he regenerated it using Tools/compiler/astgen.py. But it made some incompatibilities against ast.py on current CVS. For example, class Expression. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-17 01:21 Message: Logged In: YES user_id=80475 I'll give it a second review. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-17 00:37 Message: Logged In: YES user_id=55188 Okay. Here's my draft for documentation. Any review/fix/enhance is very welcome! I think it's ready to putting it into CVS now. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-16 11:24 Message: Logged In: YES user_id=595483 ah, the following bug was due to the patch I uploaded 2004-02-04 15:13. In order to get an error instantly from an expression like "g=(x for x in None)", I made it equivalent to, def __gen(_[1]): for x in _[1]: yield x g=__gen(iter(None)) ^^^^ But when there is two or more iterator expression of same symbol(or string), that patch produces error, because currently, "((x, y) for x in 'abcd' for y in 'abcd') " is equivalent to, def __gen(_[1]): for x in _[1]: for y in _[1]: yield (x,y) g = __gen(iter("abcd")) # passing only one parameter I could make it pass every iterator expressions respectively even when they are same symbol(or string, ...), but for now, I just dropped that patch (patch of 2004-02-04) since that's the simplest thing now. If somebody says getting instance error for cases like "(x for x in None)" is important, I'll make another patch for it. Btw, added more test case that covers what perky mentioned. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-15 23:16 Message: Logged In: YES user_id=55188 Another bug in the implementation. >>> list((x, y) for x in 'abcd' for y in 'abcd') [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] Expected behavior may be: >>> [(x, y) for x in 'abcd' for y in 'abcd'] [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd'), ('b', 'a'), ('b', 'b'), ('b', 'c'), ('b', 'd'), ('c', 'a'), ('c', 'b'), ('c', 'c'), ('c', 'd'), ('d', 'a'), ('d', 'b'), ('d', 'c'), ('d', 'd')] ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-13 21:59 Message: Logged In: YES user_id=55188 I'm writing docs for tutorial and language reference. It'll be completed in a day or two. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 16:17 Message: Logged In: YES user_id=80475 Any recent progress? ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-23 11:34 Message: Logged In: YES user_id=595483 Whoa! I finally patched compiler package in pure python. (I was a bit busy recently :) I've run regression test with 'Tools/compiler/regrtest.py' before I submit this patch, and there was one failure (which is test_dis.py). I'm not sure if that's a problem, so I'm just submitting this in. Now, I think I'll refactor things a bit! ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-02-04 08:53 Message: Logged In: YES user_id=55188 As it mentioned in PEP, even listcomp's leaking of loop value will be dropped in Python 3. I'm sorry but I don't see that the usage is solid in the future. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-04 05:45 Message: Logged In: YES user_id=671362 What about this code? In the currently implementation, loop variables inside a list comprehension is not visible outside if you use it inside a generator expression. For example: >>> (a*b for a in [b for b in range(5)]) Traceback (most recent call last): File "", line 1, in ? NameError: name 'b' is not defined Its list comprehension counterpart is: >>> [a*b for a in [b for b in range(5)]] [0, 4, 8, 12, 16] ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-04 01:13 Message: Logged In: YES user_id=595483 Again, I fixed the patch so that it can get error from '(x for x in None)' immediately. (upto now, error does not occur until generator expression is evaluated) ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-02-03 06:46 Message: Logged In: YES user_id=4771 After thinking a bit more on the issue, note that the generator version of your example is equivalent to: def g(): for y in range(3): yield lambda x: x+y which means that it can suffer from the same problem as the first example, but more subtly (and even more confusingly): for f in g(): print f(0) # 0, 1, 2 for f in list(g()): print f(0) # 2, 2, 2 This is because due to Python's nested scope rules the above generator is equivalent to: def g(): result = lambda x: x+y for y in range(3): yield result at which point I think it gets pretty confusing... ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-03 04:07 Message: Logged In: YES user_id=671362 Thanks, Arigo and Perky. Hmm, maybe I should read PEP and the thread about namespace more carefully, not just skim the surface. Anyway, PEP has not been updated since last October, so I think it's good time to merge recent progress of genexpr and update PEP-289. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-02-02 07:58 Message: Logged In: YES user_id=4771 The behavior is indeed the one described by the PEP but it is still surprising. As far as I'm concerned it is yet another reason why free variable bindings ("nested scopes") are done wrong in Python currently :-( If you use the older trick "lambda x,y=y:x+y" instead, then both cases will agree (with the sensible result in my opinion). A few more issues and I'll start a campain for definition-time bindings of free variables :-( ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-02-02 07:37 Message: Logged In: YES user_id=55188 I think it's right for namespace difference between listcomp and genexpr. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-02 06:47 Message: Logged In: YES user_id=671362 I am not sure if I should call this a bug, but here it goes: >>> lst = [lambda x:x+y for y in range(3)] >>> for f in lst:print f(0) ... 2 2 2 >>> gen = (lambda x:x+y for y in range(3)) >>> for f in gen:print f(0) ... 0 1 2 Is this the intended behavior? ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-02 03:29 Message: Logged In: YES user_id=595483 ok. I fixed another bug reported by perky, and added related test to test_grammar.py. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-30 00:09 Message: Logged In: YES user_id=55188 Yet another Fatal Python error; >>> (a for a in (b for b in (a for a in 'xxx' if True) if False) if True) lookup 'True' in ? 3 -1 freevars of : ('True',) Fatal Python error: com_make_closure() zsh: abort (core dumped) ./python # applied patch as of 2004-01-30 ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-29 22:01 Message: Logged In: YES user_id=595483 ok, I've fixed the bug quiver commented, and added related test code to test_grammar.py. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-01-29 06:48 Message: Logged In: YES user_id=671362 yet another Fatal Python error; >>> (a for a in (b for b in (0,1))) >>> (a for a in (b for b in range(2))) Fatal Python error: unknown scope for range in ?(0) in symbols: {'range': 8} locals: {} globals: {} Aborted # applied patch as of 2004-01-27 ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-27 02:44 Message: Logged In: YES user_id=595483 I fixed the patch for the bug that arigo mentioned, and for what perky mentioned - PyList_GetSlice error handling - . now, I'll tackle python compiler package :) ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-01-26 10:15 Message: Logged In: YES user_id=4771 >>> (a for d in a) Fatal Python error: unknown scope for a in (1) in symbols: {'a': 2048, '_[1]': 4, 'd': 2} locals: {'_[1]': 0, 'd': 1} globals: {} ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-26 01:17 Message: Logged In: YES user_id=55188 Please ignore the previous comment. It was a result from an old revision of patches. It works on the recent. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-25 18:44 Message: Logged In: YES user_id=55188 BTW, does unavaliability of yield (genexpr) and return (genexpr) an intended behavior? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-01-21 13:13 Message: Logged In: YES user_id=6656 Documentation would be nice! ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-21 09:14 Message: Logged In: YES user_id=55188 Okay. My review is done. The revised patch "gexp.diff" looks fine for me. There're few minor tweaks still needed to get into CVS. - Some error exit cases are ignored. You need to provide safe exit paths for MemoryError. (eg. PyList_GetSlice usage of Python/ compiler.c) - A patch for 'compiler' pure python package. (there's an old implementation on SF #795947) ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-19 06:10 Message: Logged In: YES user_id=595483 ok. I modified the patch so that it evaluates iterator expr in generator expression creation time. That means, g = (x for x in range(10)) is equivalent to def __gen(iter1): for x in iter1: yield x g = __gen(range(10)) so, evalution of range(10) is not deferred anymore. I also changed testgenexpr to testlist_gexp in Grammar/Grammar, since Hye-Shik Chang advised as such. I'm willing to take any advice about this patch, so please do. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-11 22:26 Message: Logged In: YES user_id=595483 ok. I've implemented capturing variables part as arigo suggested. File genexpr-capture-vars-in-args.diff is that. If you look at the code, you'll find that the code for generator expression is much shorter, and there's lesser modification in the existing code. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-08 20:49 Message: Logged In: YES user_id=595483 What arigo wrote sounds reasonable, and not very difficult to implement. I'll try to do that way. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-01-08 13:50 Message: Logged In: YES user_id=4771 We may not need two levels of nested anonymous functions. It seems to me that the following equivalent code would do, because it captures the variable in an argument instead of via nested scopes: x = 1 def __gen(x): for i in range(10): yield x g = __gen(x) I don't know though if this is easy to implement in compile.c. Alternatively: x = 1 def __gen(x=x): for i in range(10): yield x g = __gen() ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-08 00:44 Message: Logged In: YES user_id=55188 Okay. I verified that basic characteristics mentioned on PEP are working. I started to review the implementation. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-07 23:50 Message: Logged In: YES user_id=595483 I added diff of Lib/symbol.py, so no need to run python Lib/symbol.py now. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-07 07:25 Message: Logged In: YES user_id=55188 Assigned to me. The originator is my friend and I have much interest on this. :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470 From noreply at sourceforge.net Thu May 20 19:55:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 19:55:47 2004 Subject: [Patches] [ python-Patches-946373 ] do not add directory of sys.argv[0] into sys.path Message-ID: Patches item #946373, was opened at 2004-05-02 05:51 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=946373&group_id=5470 Category: Core (C code) Group: None Status: Open Resolution: None Priority: 5 Submitted By: wrobell (wrobell) Assigned to: Nobody/Anonymous (nobody) Summary: do not add directory of sys.argv[0] into sys.path Initial Comment: Python adds magically directory of sys.argv[0] into sys.path, i.e. >>> import sys >>> sys.path ['', '/usr/lib/python23.zip', '/usr/share/python2.3', '/usr/share/python2.3/plat-linux2', '/usr/share/python2.3/lib-tk', '/usr/lib/python2.3/lib-dynload', '/usr/lib/python2.3/site-packages', '/usr/lib/python2.3/site-packages/gtk-2.0', '/usr/share/python2.3/site-packages'] where '' (or /usr/bin when executed script is in /usr/bin directory, etc.) is added automatically. It is useful in many circumstances but fails when name conflict occurs. For example, create getpass.py or pdb.py scripts which import getpass and pdb modules. Script names conflict with modules names and modules are not going to be imported because path to the scripts is appended into sys.path, so a script is imported instead of a module. The solutions: 1. User of script with conflicting name (i.e. getpass.py or timeit.py) can set PYTHONPATH to system library path, i.e. /usr/lib/python2.3. 2. User can modify the script to delete site.path[0]. 3. User can rename the script. 4. Python can be modified to not add magically directory of sys.argv[0]. The 1. is a tricky and not intuitive and quite funny: set PYTHONPATH to system library path to import system module (and only in specific circumstances). ;-P The 2. is a dirty hack: hey, we gonna import system module, ain't it? The 3. is, IMHO, not acceptable because there is more than 200 python system modules, more in the future and user cannot be forced to maintain script names blacklist. The 4. is only acceptable, IMHO. It makes python more inconvenient but gives no trouble when names conflict occurs. Moreover, fourth solution makes python more standard with other languages behaviour, i.e. one has to set CLASSPATH to load Java classes. Maybe there is another solution, but... Patch attached. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-20 16:55 Message: Logged In: YES user_id=341410 This "problem" will be fixed in Python 2.4 with the introduction of absolute and relative import semantics as given in PEP 328: http://www.python.org/peps/pep-0328.html As stated in the PEP, to use the obviously backwards incompatible semantics, the future import will be used for 2.4 and 2.5, where in 2.6 it will become the default. from __future__ import absolute_import ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-12 08:34 Message: Logged In: YES user_id=29957 I've been bitten by this before. See e.g. the shtoom.py script clashing with the shtoom package. I used the hacky approach of moving '' to the end of sys.path. While it would be nice if this wasn't needed, I can't see this being anything other than a backwards compatibility nightmare. It will absolutely break a lot of things to change it. ---------------------------------------------------------------------- Comment By: Ilya Sandler (isandler) Date: 2004-05-07 16:45 Message: Logged In: YES user_id=971153 Would not this cause serious backward compatibility problems?? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=946373&group_id=5470 From noreply at sourceforge.net Thu May 20 20:07:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 20:07:49 2004 Subject: [Patches] [ python-Patches-947386 ] Fix for #947380 - sys.path is wrong on windows sometimes Message-ID: Patches item #947386, was opened at 2004-05-03 17:56 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947386&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Ori Berger (orib) Assigned to: Nobody/Anonymous (nobody) Summary: Fix for #947380 - sys.path is wrong on windows sometimes Initial Comment: Fixes http://python.org/sf/947380 - see details there. One line of code, one line of comment. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-20 17:07 Message: Logged In: YES user_id=341410 See my comment in #947380 as to why this will go away in future versions of Python, without your patch. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=947386&group_id=5470 From noreply at sourceforge.net Fri May 21 11:44:09 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 21 11:44:21 2004 Subject: [Patches] [ python-Patches-943898 ] A simple 3-4% speed-up for PCs Message-ID: Patches item #943898, was opened at 2004-04-28 18:33 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=943898&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Armin Rigo (arigo) Assigned to: Tim Peters (tim_one) Summary: A simple 3-4% speed-up for PCs Initial Comment: The result of a few experiments looking at the assembler produced by gcc for eval_frame(): * on PCs, reading the arguments as an unsigned short instead of two bytes is a good win. * oparg is more "local" with this patch: its value doesn't need to be saved across an iteration of the main loop, allowing it to live in a register only. * added an explicit "case STOP_CODE:" so that the switch starts at 0 instead of 1 -- that's one instruction less with gcc. * it seems not to pay off to move reading the argument at the start of each case of an operation that expects one, even though it removes the unpredictable branch "if (HAS_ARG(op))". This patch should be timed on other platforms to make sure that it doesn't slow things down. If it does, then only reading the arg as an unsigned short could be checked in -- it is compilation-conditional over the fact that shorts are 2 bytes in little endian order. By the way, anyone knows why 'stack_pointer' isn't a 'register' local? I bet it would make a difference on PowerPC, for example, with compilers that care about this keyword. ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-21 15:44 Message: Logged In: YES user_id=4771 The bit with gcc-specific keywords is useful but arguably scary, but the other part of the patch -- stack_pointer not being assignable to a register -- solves a definite performance bug in my opinion. I'd even suggest back-porting this one to 2.3. Apple is more likely to ship its next MacOSX release with the latest 2.3 than with 2.4, as far as I can tell. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-12 15:27 Message: Logged In: YES user_id=80475 Tim, I remember you having some options about these sort of optimizations. Will you take a brief look at Armin's latest patch. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-10 14:26 Message: Logged In: YES user_id=4771 Tested on a MacOSX box, the patch also gives a 5% speed-up there. Allowing stack_pointer to be in a register is a very good idea. (all tests with Pystone) ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-10 10:48 Message: Logged In: YES user_id=4771 The short trick might be a bit fragile. For example, the current patch would incorrectly use it on machines where unaligned accesses are forbidden. I isolated the other issue I talked about (making stack_pointer a register variable) in a separate patch. This patch alone is clearly safe. It should give a bit of speed-up on any machine but it definitely gives 5% on PCs with gcc by forcing the two most important local variables into specific registers. (If someone knows the corresponding syntax for other compilers, it can be added in the #if.) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 23:45 Message: Logged In: YES user_id=80475 With MSVC++ 6.0 under WinME on a Pentium III, there is no change in timing (measurements accurate within 0.25%): I wonder if the speedup from retrieving the unsigned short is offset by alignment penalties when the starting address is odd. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-04-28 21:02 Message: Logged In: YES user_id=4771 stack_pointer isn't a register because its address is taken at two places. This is a really bad idea for optimization. Instead of &stack_pointer, we should do: PyObject **sp = stack_pointer; ... use &sp ... stack_pointer = sp; I'm pretty sure this simple change along with a 'register' declaration of stack_pointer gives a good speed-up on all architectures with plenty of registers. For PCs I've experimented with forcing one or two locals into specific registers, with the gcc syntax asm("esi"), asm("ebx"), etc. Forcing stack_pointer and next_instr gives another 3-4% of improvement. Next step is to see if this can be done with #if's for common compilers beside gcc. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=943898&group_id=5470 From vctltmh at hotmail.com Wed May 19 19:26:41 2004 From: vctltmh at hotmail.com (Judy Fox) Date: Fri May 21 15:33:25 2004 Subject: [Patches] Home is where the heart is Message-ID: <14159191RND_LC_CHAR[1-5]128q$78b4$524hpq8ir51@borrow> Greetings, This is an email to notify you that you have been accepted into our "Lowest_Mortgage_Rate" program. Please visit the following link to complete your application which has been pre-approved. http://ezwayhomeloan.com/?partid=wh6 Thank You Lupe Senior Mortgage Consultant to opt out of this program click here...it may take up to 48 hours to complete thank you http://ezwayhomeloan.com/st.html From gabood at powerup.com.au Fri May 21 09:02:08 2004 From: gabood at powerup.com.au (Calling T. Doily) Date: Fri May 21 20:49:19 2004 Subject: [Patches] Read:_800 world best software - 90 % savings Message-ID: <0717089904.20040521080208@powerup.com.au> >Ben Clinton Albert Marina Luke Harold Cora Jeanine Sensation!! We opened a NEW site with unbeatable prices and products. 800 WORLD BEST software with 90% discount - that is a really BEST offer just imagine, you can buy ALL software that you ever need and pay price of just one of it! Office 2003 for 50$ - nice deal right ? ;) retail price is 700$ - great savings, huh? Please spend few moments of yours precious time to check our offer - it is more than worth it! http://Dee.oem-licensed-soft.biz/?Kim >Taken in any context, except seriously, life on earth is palatable and positively enjoyable. >Democracy means simply the bludgeoning of the people by the people for the people. >I feel as if I were a piece in a game of chess, when my opponent says of it: That piece cannot be moved. >Discovery consists of seeing what everybody has seen and thinking what nobody has thought. >Listening well is as powerful a means of communication and influence as to talk well. >The man who says he has exhausted life generally means that life has exhausted him. >Good fellowship and friendship are lasting, rational and manly pleasures. >Harold, like the rest of us, had many impressions which saved him the trouble of distinct ideas. >Never let the other fellow set the agenda. >Those who are believed to be most abject and humble are usually most ambitious and envious. >The busier we are the more leisure we have. >What we feel and think and are is to a great extent determined by the state of our ductless glands and viscera. >It is a miserable state of mind to have few things to desire and many things to fear. >It's motive alone which gives character to the actions of men. >Nothing makes one so vain as being told that one is a sinner. >Fortune sides with him who dares. >Experience comprises illusions lost, rather than wisdom gained. >In our own hearts, we mold the whole world's hereafters and in our own hearts we fashion our own gods. >It is far more important to me to preserve an unblemished conscience than to compass any object however great. From noreply at sourceforge.net Fri May 21 04:05:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 22 07:26:24 2004 Subject: [Patches] [ python-Patches-872326 ] generator expression implementation Message-ID: Patches item #872326, was opened at 2004-01-07 12:10 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470 Category: Parser/Compiler Group: None Status: Closed Resolution: Accepted Priority: 5 Submitted By: Jiwon Seo (jiwon) Assigned to: Nobody/Anonymous (nobody) Summary: generator expression implementation Initial Comment: Since I was interested in pep 289(generator expression), I dabbled with it, and implemented a working version of it. I'm not sure if I did it right, but maybe someone who knows better can fix it right. 1. Grammar has two changes, which is a. atom: '(' [testlist] ')' | '[' [listmaker] ']' | ... changes to atom: '(' [testgenexpr] ')' | '[' [listmaker] ']' | ... where testgenexpr defines like this. testgenexpr: test ( gen_for | (',' test)* [','] ) b. argument: [test '='] test changes to argument: [test '='] test [gen_for] (gen_for, gen_if, gen_iter is similar to list_for, list_if, list_iter respectively.) 2. Instead of changing rule of arglist in Grammar to accept generator expression, I changed argument rule like 1.b. This means Grammar accepts generator expression without parenthesis in function call even there are several arguments, like reduce(operator.add, (x for x in range(10))) This is against what pep requires, so I made parsermodule.c and compile.c to catch that and throw error message when there's more than one argument in a function. The reason I did it that way is to go around a problem of ambiguity in the grammar by adding generator expression to arglist. 3. I implemented generator expression as equivalent to a call to a function which has variable capturing code and generator code. For example, x = 1 g = (x for i in range(10)) is equivalent to x = 1 def __gen_wrapper(): _[x] = x # variable capture here def __gen(): for i in range(10): yield _[x] return __gen() g = __gen_wrapper() 4. Since I implemented generator expression equivalent to a function call, symbol table should enter new scope when it meets generator expression. So I ended up with adding following code to symtable.c PyObject * PySymtableEntry_New(struct symtable *st, char *name, int type, int lineno) { ... switch (type) { case funcdef: case lambdef: ! case testgenexpr: /* generator expression */ ! case argument: /* generator expression */ ste->ste_type = TYPE_FUNCTION; break; ... so that generator expression can be dealt as function type. This is kind of stupid, but I couldn't find other easy for it, so I just left it like that. 5. this patch does not include diff of src/Lib/symbol.py, so you must run python Lib/symbol.py to get it right. ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-21 08:05 Message: Logged In: YES user_id=4771 Note that FOR_ITER segfaults if it is passed a non-iterable. This gives us a dangerous code object without invoking the 'new' module. A compiler-produced code object should probably not allow us to do the following: >>> def f(): (a for b in c) >>> f.func_code.co_consts (None, at xxx>, ...) >>> co = f.func_code.co_consts[1] >>> dis.dis(co) 1 0 SETUP_LOOP 17 (to 20) 3 LOAD_FAST 0 ([outmost-iterable]) >> 6 FOR_ITER 10 (to 20) ... >>> f.func_code = co >>> f(5).next() Segmentation fault ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 23:07 Message: Logged In: YES user_id=80475 Please disregard the last two posts. Guido and I agreed that the current behavior is correct. I've updated the docs accordingly and added a test. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 17:32 Message: Logged In: YES user_id=80475 FWIW, here is a patch that moves the GET_ITER back inside the generator so that the implementation will more closely match the spec and match the operation of regular generators. *** compile.c 19 May 2004 08:20:15 -0000 2.302 --- compile.c 20 May 2004 17:29:35 -0000 *************** *** 1628,1633 **** --- 1628,1634 ---- if (is_outmost) { com_addop_varname(c, VAR_LOAD, "[outmost-iterable]"); + com_addbyte(c, GET_ITER); com_push(c, 1); } else { *************** *** 1819,1825 **** com_addoparg(c, MAKE_FUNCTION, 0); com_test(c, CHILD(CHILD(n, 1), 3)); - com_addbyte(c, GET_ITER); com_addoparg(c, CALL_FUNCTION, 1); com_pop(c, 1); --- 1820,1825 ---- ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 17:08 Message: Logged In: YES user_id=80475 Jiwon, while working through the doc patch, I noticed a code generation nit. The code g=(i for i in range(10)) gets translated to: def __gen(exp): for i in exp: yield i g = __gen(iter(range(10)) del __gen On the next to last line, the call to iter() may be superfluous. Do you see a need for it? If so, I need to include that in the spec. If not, then it should be removed (line 1822 in compile.c). In terms of byte codes, the difference shows up in a comparison to an equivalent generator: >>> dis(compile('g(range(10))', '', 'eval')) 0 0 LOAD_NAME 0 (g) 3 LOAD_NAME 1 (range) 6 LOAD_CONST 0 (10) 9 CALL_FUNCTION 1 12 CALL_FUNCTION 1 15 RETURN_VALUE >>> dis(compile('(i for i in range(10))', '', 'eval')) 0 0 LOAD_CONST 0 ( at 00A4B220, file "", line 1>) 3 MAKE_FUNCTION 0 6 LOAD_NAME 0 (range) 9 LOAD_CONST 1 (10) 12 CALL_FUNCTION 1 15 GET_ITER 16 CALL_FUNCTION 1 19 RETURN_VALUE The get_iter at code position 15 is what is in question. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-20 16:51 Message: Logged In: YES user_id=595483 Thank you, and I'm happy to contribute. =) I updated documentation so that it deals with the precomputation issue. But I don't know how much I should write or where, so please review/revise it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-19 08:20 Message: Logged In: YES user_id=80475 Thank you for the superb contribution. It passes all of my tests without exception (see the new file, test_genexps). I do not see any issues with the coding. Am accepting and applying the patch except for the documentation. Please update the documentation part of the patch and I will go through and clarify where necessary. Also, I will update the PEP to reflect Guido's rationale for the design decisions on binding behavior. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-15 17:24 Message: Logged In: YES user_id=595483 OK. I added outmost-iterable-precomputation feature to lazy-binding version of the patch, and also added related tests to test_grammar.py. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-15 06:08 Message: Logged In: YES user_id=595483 Ok. I see. I've almost finished adding it(outmost iterable precomputation) to CPython, and now trying to do it in python compiler package. Probably I'll commit patch till tomorrow. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-15 00:31 Message: Logged In: YES user_id=80475 Jiwon, do you see how to proceed for here? Note that given, (for x in exp1 if exp2 for y in exp3 if exp4), only exp1 is evaluated immediately. exp2 is deferred. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-13 03:58 Message: Logged In: YES user_id=6380 On why the first expr should be evaluated immediately: Consider sum(x for x in foo()). Now suppose there's a bug in foo() that raises an exception, and a bug in sum() that raises an exception before it starts iterating over its argument. Which exception would you expect to see? I'd be surprised if the one in sum() was raised rather the one in foo(), since the call to foo() is part of the argument to sum(), and I expect arguments to be processed before the function is called. OTOH, in sum(bar(x) for x in foo()), where sum() and foo() are bugfree, but bar() raises an exception, we have no choice but to delay the call to bar() until sum() starts iterating -- that's part of the contract of generators. (They do nothing until their next() method is first called.) Ditto for subsequent iterables if nested for loops are used: in sum(x*y for x in range(10) for y in bar(x)), there are 10 calls to bar(), with arguments supplied by the first (outer) for loop, and we have no choice but to delay even the first call to bar() until sum() calls next() on its argument. I know this isn't the easiest thing to implement, but I still think it's the right thing. The generator function for the last example would be something like def G(arg): for x in arg: for y in bar(x): yield x*y and the call to sum() would be sum(G(range(10))) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-12 18:54 Message: Logged In: YES user_id=80475 Jeremy: I reviewed the lazy binding version of the patch as much as I could. Can you take a brief look at just the changes to the grammar and at the new functions in compile.c. The functions were clearly written and neatly paralleled list comps but I wondered whether it was possible and desirable to factor-out the commonalities. Jiwon: the graminit.h renumbering is fine (all of the automatically generated files should be left as-as). Guido: I was clear on your rationale for preferring late binding but did not find wording for the reason behind the precompution of the innermost iterable. I speculate that the semantics reflect that this is an expression and that all expressions (except and/or) are immediate. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 17:36 Message: Logged In: YES user_id=6380 Read back what I wrote on python-dev about capturing free variables as a new concept with which I am uncomfortable. There should not be such "advanced" usages of genexprs -- advanced usages are better served by explicit generator functions. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-12 15:26 Message: Logged In: YES user_id=4771 Which is exactly why I cannot understand why you are so keen on one binding model rather than the other, if it doesn't matter. It looks like we are going to add is a feature that doesn't scale to advanced usages. Moreover it seems really obvious that non-experts *will* get surprizes, because they *will* try to replace all their listcomps with genexprs -- they shouldn't do that, but of course they will do it anyway, because it appears to work -- until the occasional strange bug kreeps in, and one that is difficult to actually relate to the genexpr in the first place. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 14:03 Message: Logged In: YES user_id=6380 You're just showing that saving genexprs up for later use is a bad idea. We should place clear warnings in the docs that the genexpr should be used up right away, and otherwise to write an explicit generator function. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-05-12 08:30 Message: Logged In: YES user_id=595483 I don't think that "lazy-binding + leftmost iterable precomputation" makes sense. Just adding another for-loop to the example shows the reason. >>> x = 10 >>> g = ((i,j) for i in range(x) for j in range(x)) >>> x = 5 >>> list(g) Here, if j is iterated only to 5 when i is iterated to 10, I think it would make users confused. I think "early-binding + leftmost iterable precomputation" makes sense, but "lazy-binding + leftmost iterable precomputation" does not make sense IMHO. About renumbering testlist_gexp in graminit.h, it's not what I did but it's auto-generated by pgen(Parser/pgenmain.c). Although it makes another file (symbol.py) need to be changed (and thus makes cvs log a bit dirty), it's tool- generated file, so I did it like that. If you think it's better to do it without re-numbering, let me know or you could do it yourself. ;^) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-12 07:32 Message: Logged In: YES user_id=80475 The lazy_bind version has survived over of a month of my testing and multiple read throughs of the code :-) I'm not expert with graminit.h but did wonder if testlist_gexp could have a high numbered code so that other codes would not have to be re-numbered. When you get back from military camp, please load the version that precomputes the outermost iterable. I believe that is Guido's currently favored approach. Currently, it fails this test: >>> x = 10 >>> g = (i*i for i in range(x)) >>> x = 5 >>> list(g) # should be ten items long [0, 1, 4, 9, 16] Tim, summarized it as: """ ... Guido favored mostly-late binding -- which is late binding, except that the iterable in the leftmost for-clause is evaluated at once. So ... (e1 for stuff in e2 for morestuff in e3 ...) ... is effectively replaced by def __g(primary): for stuff in primary: for morestuff in e3: ... yield e1 ... __g(e2) ... """ ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-27 15:56 Message: Logged In: YES user_id=595483 I'm submitting two versions now. One is variable-capture version(but without any iterable pre-computation), and the other is lazy-binding version. I'll be in military camp for about 1 month(4/6~5/2) so I will not be able to fix/patch anything for the period, thus I'm submitting this in now. :) If I have time, and necessity arises (before 4/6), I'll also add outmost-iterable-precomputation version. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-03-22 23:00 Message: Logged In: YES user_id=6380 Sorry, I meant the first, not the last. I was confused about how the 'for' clauses are nested, but the outermost one is the first. So the nesting remark is probably just confusing and wrong; ignore it. What I meant is: (f(x,y) for x in A() for y in B(x)) should IMO precompute A(), but not B(x). I guess the equivalent generator function would be: def __gen(__outer__=A(), f=f, B=B): for x in __outer__: for y in B(x): yield f(x,y) In general the value of every free variable used anywhere except in the outer expression should be captured; the *value* of the outer expression should be captured. This should give the least surprising semantics in a variaty of use cases. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-03-22 19:39 Message: Logged In: YES user_id=6380 Only the outermost (last) iterable should be precomputed. While (f(x,y) for x in A(y) for y in B) is not the same as ((f(x,y) for x in A(y)) for y in B) I think the scoping should be done similarly. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-22 01:57 Message: Logged In: YES user_id=595483 Ah... Maybe we need to drop precomputation of iterables. I'll post it on the mailing list so that people can consider about that. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-03-21 23:40 Message: Logged In: YES user_id=4771 Oh ho. Am I right in fearing that the idea of precomputing the iterables was broken in the first place? We just cannot do this. The things we are looping over may depend on other stuff from the generator expression itself. Example: >>> [x for l in [[1,2],[3,4]] for x in l] [1, 2, 3, 4] >>> (y for m in [[1,2],[3,4]] for y in m) NameError: name 'm' is not defined ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-21 17:55 Message: Logged In: YES user_id=55188 This inconsistency may not be fixed by list(iter()) workaround, either. >>> def counter(): ... counter.count += 1 ... return range(counter.count) ... >>> counter.count = 0 >>> [y for x in 'abc' for y in counter()] [0, 0, 1, 0, 1, 2] >>> counter.count = 0 >>> list(y for x in 'abc' for y in counter()) [0, 0, 0] ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-21 17:42 Message: Logged In: YES user_id=55188 Aah. But I'm not insisting on that because it's incompatible even with plain nested for-loops. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-21 17:38 Message: Logged In: YES user_id=55188 Agreed. I'd prefer the current implementation's behavor rather than defined in PEP289. Yes, It's incompatible with list comprehensions but generator expressions are already quite different enough from it. :) How about to change PEP289's genexpr semantics to this? g = (x**2 for x in range(10)) print g.next() is equivalent to: def __gen(_i1): for x in _i1: yield x**2 g = __gen(range(10)) print g.next() ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-21 17:28 Message: Logged In: YES user_id=80475 You need a solution other than list(). A key to the success of genexps is to never to eat memory by expanding an iterator into a container. For behavior questions, your reference point is the list comprehension. list(genexp) should always produce the same result as a list comprehension. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-21 16:59 Message: Logged In: YES user_id=595483 Hi, quiver. I don't think we can easily go around this problem if we have to capture iterators in generator expression. If you run following, you'll know what I mean. >>> a = iter("abcd") >>> b = iter("abcd") >>> [(x, y) for x in a for y in b] [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] I think one possible solution could be, instead of passing evaluations of iterators in generator expression, make a list from iterator and, then pass it as argument. For instance, g = (x for x in iter("abcd")) will be equivalent to, def __gen(_[1]): for x in _[1]: yield x g = __gen(list(iter("abcd"))) # see 'list' - instead of g = __gen(iter("abcd")) . I'm not sure if I'm in a position to decide to do that way or not. If the current reviewer (rhettinger) approves it, I'll do that way. Or else, I think I will post it on the mailing list. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-03-19 13:37 Message: Logged In: YES user_id=671362 Hi, Jiwon. This is another bug candidate. If you use genexp with iterators, it doesn't work well. # test Perky's previous post using iterators >>> list((x,y) for x in iter('abcd') for y in iter('abcd')) [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] Thanks. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-17 08:45 Message: Logged In: YES user_id=595483 To fix the bug that perky picked out, I hand-made ast.py instead of using auto-generated code. But, still I don't understand why Tools/compiler/regrtest didn't tell me about it. (Maybe I didn't look at the output carefully enough.) Ah, and it would be nice if somebody tell me how to run test_grammar.py(only) with python-compiler package. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-17 06:28 Message: Logged In: YES user_id=55188 The compiler package patch has some problems in its compiler/ast.py currently. jiwon said he regenerated it using Tools/compiler/astgen.py. But it made some incompatibilities against ast.py on current CVS. For example, class Expression. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-17 06:21 Message: Logged In: YES user_id=80475 I'll give it a second review. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-17 05:37 Message: Logged In: YES user_id=55188 Okay. Here's my draft for documentation. Any review/fix/enhance is very welcome! I think it's ready to putting it into CVS now. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-03-16 16:24 Message: Logged In: YES user_id=595483 ah, the following bug was due to the patch I uploaded 2004-02-04 15:13. In order to get an error instantly from an expression like "g=(x for x in None)", I made it equivalent to, def __gen(_[1]): for x in _[1]: yield x g=__gen(iter(None)) ^^^^ But when there is two or more iterator expression of same symbol(or string), that patch produces error, because currently, "((x, y) for x in 'abcd' for y in 'abcd') " is equivalent to, def __gen(_[1]): for x in _[1]: for y in _[1]: yield (x,y) g = __gen(iter("abcd")) # passing only one parameter I could make it pass every iterator expressions respectively even when they are same symbol(or string, ...), but for now, I just dropped that patch (patch of 2004-02-04) since that's the simplest thing now. If somebody says getting instance error for cases like "(x for x in None)" is important, I'll make another patch for it. Btw, added more test case that covers what perky mentioned. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-16 04:16 Message: Logged In: YES user_id=55188 Another bug in the implementation. >>> list((x, y) for x in 'abcd' for y in 'abcd') [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd')] Expected behavior may be: >>> [(x, y) for x in 'abcd' for y in 'abcd'] [('a', 'a'), ('a', 'b'), ('a', 'c'), ('a', 'd'), ('b', 'a'), ('b', 'b'), ('b', 'c'), ('b', 'd'), ('c', 'a'), ('c', 'b'), ('c', 'c'), ('c', 'd'), ('d', 'a'), ('d', 'b'), ('d', 'c'), ('d', 'd')] ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-03-14 02:59 Message: Logged In: YES user_id=55188 I'm writing docs for tutorial and language reference. It'll be completed in a day or two. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 21:17 Message: Logged In: YES user_id=80475 Any recent progress? ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-23 16:34 Message: Logged In: YES user_id=595483 Whoa! I finally patched compiler package in pure python. (I was a bit busy recently :) I've run regression test with 'Tools/compiler/regrtest.py' before I submit this patch, and there was one failure (which is test_dis.py). I'm not sure if that's a problem, so I'm just submitting this in. Now, I think I'll refactor things a bit! ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-02-04 13:53 Message: Logged In: YES user_id=55188 As it mentioned in PEP, even listcomp's leaking of loop value will be dropped in Python 3. I'm sorry but I don't see that the usage is solid in the future. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-04 10:45 Message: Logged In: YES user_id=671362 What about this code? In the currently implementation, loop variables inside a list comprehension is not visible outside if you use it inside a generator expression. For example: >>> (a*b for a in [b for b in range(5)]) Traceback (most recent call last): File "", line 1, in ? NameError: name 'b' is not defined Its list comprehension counterpart is: >>> [a*b for a in [b for b in range(5)]] [0, 4, 8, 12, 16] ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-04 06:13 Message: Logged In: YES user_id=595483 Again, I fixed the patch so that it can get error from '(x for x in None)' immediately. (upto now, error does not occur until generator expression is evaluated) ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-02-03 11:46 Message: Logged In: YES user_id=4771 After thinking a bit more on the issue, note that the generator version of your example is equivalent to: def g(): for y in range(3): yield lambda x: x+y which means that it can suffer from the same problem as the first example, but more subtly (and even more confusingly): for f in g(): print f(0) # 0, 1, 2 for f in list(g()): print f(0) # 2, 2, 2 This is because due to Python's nested scope rules the above generator is equivalent to: def g(): result = lambda x: x+y for y in range(3): yield result at which point I think it gets pretty confusing... ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-03 09:07 Message: Logged In: YES user_id=671362 Thanks, Arigo and Perky. Hmm, maybe I should read PEP and the thread about namespace more carefully, not just skim the surface. Anyway, PEP has not been updated since last October, so I think it's good time to merge recent progress of genexpr and update PEP-289. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-02-02 12:58 Message: Logged In: YES user_id=4771 The behavior is indeed the one described by the PEP but it is still surprising. As far as I'm concerned it is yet another reason why free variable bindings ("nested scopes") are done wrong in Python currently :-( If you use the older trick "lambda x,y=y:x+y" instead, then both cases will agree (with the sensible result in my opinion). A few more issues and I'll start a campain for definition-time bindings of free variables :-( ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-02-02 12:37 Message: Logged In: YES user_id=55188 I think it's right for namespace difference between listcomp and genexpr. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-02-02 11:47 Message: Logged In: YES user_id=671362 I am not sure if I should call this a bug, but here it goes: >>> lst = [lambda x:x+y for y in range(3)] >>> for f in lst:print f(0) ... 2 2 2 >>> gen = (lambda x:x+y for y in range(3)) >>> for f in gen:print f(0) ... 0 1 2 Is this the intended behavior? ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-02-02 08:29 Message: Logged In: YES user_id=595483 ok. I fixed another bug reported by perky, and added related test to test_grammar.py. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-30 05:09 Message: Logged In: YES user_id=55188 Yet another Fatal Python error; >>> (a for a in (b for b in (a for a in 'xxx' if True) if False) if True) lookup 'True' in ? 3 -1 freevars of : ('True',) Fatal Python error: com_make_closure() zsh: abort (core dumped) ./python # applied patch as of 2004-01-30 ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-30 03:01 Message: Logged In: YES user_id=595483 ok, I've fixed the bug quiver commented, and added related test code to test_grammar.py. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-01-29 11:48 Message: Logged In: YES user_id=671362 yet another Fatal Python error; >>> (a for a in (b for b in (0,1))) >>> (a for a in (b for b in range(2))) Fatal Python error: unknown scope for range in ?(0) in symbols: {'range': 8} locals: {} globals: {} Aborted # applied patch as of 2004-01-27 ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-27 07:44 Message: Logged In: YES user_id=595483 I fixed the patch for the bug that arigo mentioned, and for what perky mentioned - PyList_GetSlice error handling - . now, I'll tackle python compiler package :) ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-01-26 15:15 Message: Logged In: YES user_id=4771 >>> (a for d in a) Fatal Python error: unknown scope for a in (1) in symbols: {'a': 2048, '_[1]': 4, 'd': 2} locals: {'_[1]': 0, 'd': 1} globals: {} ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-26 06:17 Message: Logged In: YES user_id=55188 Please ignore the previous comment. It was a result from an old revision of patches. It works on the recent. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-25 23:44 Message: Logged In: YES user_id=55188 BTW, does unavaliability of yield (genexpr) and return (genexpr) an intended behavior? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-01-21 18:13 Message: Logged In: YES user_id=6656 Documentation would be nice! ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-21 14:14 Message: Logged In: YES user_id=55188 Okay. My review is done. The revised patch "gexp.diff" looks fine for me. There're few minor tweaks still needed to get into CVS. - Some error exit cases are ignored. You need to provide safe exit paths for MemoryError. (eg. PyList_GetSlice usage of Python/ compiler.c) - A patch for 'compiler' pure python package. (there's an old implementation on SF #795947) ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-19 11:10 Message: Logged In: YES user_id=595483 ok. I modified the patch so that it evaluates iterator expr in generator expression creation time. That means, g = (x for x in range(10)) is equivalent to def __gen(iter1): for x in iter1: yield x g = __gen(range(10)) so, evalution of range(10) is not deferred anymore. I also changed testgenexpr to testlist_gexp in Grammar/Grammar, since Hye-Shik Chang advised as such. I'm willing to take any advice about this patch, so please do. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-12 03:26 Message: Logged In: YES user_id=595483 ok. I've implemented capturing variables part as arigo suggested. File genexpr-capture-vars-in-args.diff is that. If you look at the code, you'll find that the code for generator expression is much shorter, and there's lesser modification in the existing code. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-09 01:49 Message: Logged In: YES user_id=595483 What arigo wrote sounds reasonable, and not very difficult to implement. I'll try to do that way. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-01-08 18:50 Message: Logged In: YES user_id=4771 We may not need two levels of nested anonymous functions. It seems to me that the following equivalent code would do, because it captures the variable in an argument instead of via nested scopes: x = 1 def __gen(x): for i in range(10): yield x g = __gen(x) I don't know though if this is easy to implement in compile.c. Alternatively: x = 1 def __gen(x=x): for i in range(10): yield x g = __gen() ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-08 05:44 Message: Logged In: YES user_id=55188 Okay. I verified that basic characteristics mentioned on PEP are working. I started to review the implementation. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-01-08 04:50 Message: Logged In: YES user_id=595483 I added diff of Lib/symbol.py, so no need to run python Lib/symbol.py now. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-01-07 12:25 Message: Logged In: YES user_id=55188 Assigned to me. The originator is my friend and I have much interest on this. :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=872326&group_id=5470 From carrillo_fx at psico.uh.cu Sun May 23 19:47:25 2004 From: carrillo_fx at psico.uh.cu (Antonio Carrillo) Date: Sun May 23 16:51:44 2004 Subject: [Patches] =?iso-8859-1?q?Buy_V=EDagra_and_many_other_medicines_h?= =?iso-8859-1?q?ere=2E?= Message-ID: <133301c44120$c90ed845$da119a7e@xnear.cl> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040523/c42e2740/attachment.html From noreply at sourceforge.net Mon May 24 12:13:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 24 12:13:52 2004 Subject: [Patches] [ python-Patches-959534 ] Arrange 64bits detection for ReliantUnix Message-ID: Patches item #959534, was opened at 2004-05-24 18:13 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=959534&group_id=5470 Category: Build Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Jean-frederic Clere (jfclere) Assigned to: Nobody/Anonymous (nobody) Summary: Arrange 64bits detection for ReliantUnix Initial Comment: When compiling python on ReliantUnix (5.43) the compilation fails with unresolved externals like fstatvfs64, lseek64 etc Those routines belong to 5.45. The patch detects the missing lseek64() and suppress lfs support (use_lfs=no). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=959534&group_id=5470 From noreply at sourceforge.net Mon May 24 17:08:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 24 17:08:13 2004 Subject: [Patches] [ python-Patches-959726 ] sdist versus SVN Message-ID: Patches item #959726, was opened at 2004-05-24 17:08 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=959726&group_id=5470 Category: Distutils and setup.py Group: None Status: Open Resolution: None Priority: 5 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: sdist versus SVN Initial Comment: The particular magic the disutils sdist command uses to prune CVS and RCS directories doesn't appear available via anything I can type in MANIFEST.in. So when using sdist from an SVN checkout directory, huge steaming mounds of SVN admin files get sucked up, and then sdist fails because it can't delete the read-only SVN admin files it copied. The patch adds .svn to the same regexp used to special case CVS and RCS directories, which solves the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=959726&group_id=5470 From zkgnbjwvvwo at tmicha.net Mon May 24 18:17:53 2004 From: zkgnbjwvvwo at tmicha.net (Barbra Edwards) Date: Mon May 24 17:22:49 2004 Subject: [Patches] Take big discounts on Windows XP, Office, Adobe, Corel stuff from Lambert's SoftShop Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040524/103adcb0/attachment.html From bobbiemcMullensh at austarmetro.com.au Mon May 24 20:34:14 2004 From: bobbiemcMullensh at austarmetro.com.au (Bobbie McMullen) Date: Mon May 24 20:37:04 2004 Subject: [Patches] =?iso-8859-1?q?Buy_V=EDagra_and_many_other_medicines_h?= =?iso-8859-1?q?ere=2E?= Message-ID: <5f5401c441f0$a25f867c$9f1b363b@citynet.com.ar> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040524/9f2e0f33/attachment.html From dhfsdnnklol at bol.com.br Mon May 24 23:43:13 2004 From: dhfsdnnklol at bol.com.br (Pat Muniz) Date: Tue May 25 01:43:30 2004 Subject: [Patches] Amazing software deals about 1032 but Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040525/8686ae3d/attachment.html From noreply at sourceforge.net Tue May 25 15:32:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 25 15:32:53 2004 Subject: [Patches] [ python-Patches-736962 ] Port tests to unittest (Part 2) Message-ID: Patches item #736962, was opened at 2003-05-13 12:45 Message generated for change (Comment added) made by doerwalter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 Category: Tests Group: None Status: Open Resolution: Accepted Priority: 5 Submitted By: Walter D?rwald (doerwalter) Assigned to: Walter D?rwald (doerwalter) Summary: Port tests to unittest (Part 2) Initial Comment: Here are the next test scripts ported to PyUnit: test_winsound and test_array. For test_array many additional tests have been added (code coverage is at 91%) ---------------------------------------------------------------------- >Comment By: Walter D?rwald (doerwalter) Date: 2004-05-25 21:32 Message: Logged In: YES user_id=89016 Merging test_dict and test_userdict revealed a difference between dict and UserDict.UserDict: >>> import UserDict >>> d = UserDict.UserDict() >>> d.update(None) >>> d {} >>> d = {} >>> d.update(None) Traceback (most recent call last): File "", line 1, in ? TypeError: iteration over non-sequence Should we do anything about this? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 18:03 Message: Logged In: YES user_id=80475 test_dict is fine. Consider beefing up the tests for update(). As of Py2.4, it now takes all the same argument possibilites as dict(): d.update(a=1, b=2) d.update([('a', 1), ('b', 2)]) Also, try to link in: from test_userdict import TestMappingProtocol This was supposed to provide tests common to all mappings. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-04-03 13:06 Message: Logged In: YES user_id=89016 The attached test_dict.py moves the dict tests from test_types.py to a new PyUnit test script. Code coverage for dictobject.c is at 90.76% (up from 89.51%) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-03-15 13:17 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_binascii delete Lib/test/test_binascii.py 1.17 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 21:26 Message: Logged In: YES user_id=80475 Okay, this one is fine. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-19 00:49 Message: Logged In: YES user_id=89016 Here's the next one: test_binascii.py. Code coverage in binascii.c is at 92% (could be improved by enhancing test_binhex.py). ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 23:46 Message: Logged In: YES user_id=33168 Looked good to me, test_future_pyunit.diff checked in as: * Lib/test/badsyntax_future3.py 1.2 * Lib/test/badsyntax_future4.py 1.2 * Lib/test/badsyntax_future5.py 1.2 * Lib/test/badsyntax_future6.py 1.2 * Lib/test/badsyntax_future7.py 1.2 * Lib/test/badsyntax_future8.py 1.1 * Lib/test/badsyntax_future9.py 1.1 * Lib/test/test_future.py 1.7 * Lib/test/test_future1.py 1.3 * Lib/test/test_future2.py 1.2 * Lib/test/output/test_future 1.4 ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 22:47 Message: Logged In: YES user_id=33168 Walter, I didn't get a mail from SF either. After you do a cvs add, you need to use -N to cvs diff. For example, cvs diff -N new_file_added_to_cvs_but_not_committed. If you look carefully, you can see this in the patch on the diff line for each file. I'll take a look at the patch. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-13 21:18 Message: Logged In: YES user_id=89016 Here is a version of test_future ported to PyUnit (test_future_pyunit.diff). If this makes sense to you Neal, please check it in. (BTW, how did you convince CVS to include badsyntax_future8.py and badsyntax_future9.py in the diff? Doing a "cvs add" only results in " Lib/test/badsyntax_future8.py is a new entry, no comparison available") ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-11 13:36 Message: Logged In: YES user_id=89016 md5/weakref patch checked in as: Lib/test/test_weakref.py 1.33 Lib/test/test_md5.py 1.5 Lib/test/output/test_md5 remove ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 07:05 Message: Logged In: YES user_id=33168 Improve coverage for test_future too. I'm not sure if test future can be ported to PyUnit. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 06:52 Message: Logged In: YES user_id=33168 Attached are two tests (in one file, I'm being lazy, sorry) to improve test coverage for md5c.c and _weakref.c. test_md5 was ported to unittest, weakref, just adds two little tests to get coverage up to 100%. If you like, just go ahead and check in. You will need to remove Lib/test/output/test_md5 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-08 12:42 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/list_tests.py 1.1 Lib/test/seq_tests.py 1.1 Lib/test/test_list.py 1.1 Lib/test/test_tuple.py 1.1 Lib/test/test_types.py 1.56 Lib/test/test_userlist.py 1.11 Lib/test/output/test_types 1.3 Next will be test_binascii.py before I continue with test_types.py. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-12-08 00:49 Message: Logged In: YES user_id=80475 Looks good. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-11-27 20:48 Message: Logged In: YES user_id=89016 Here the first part of the test_types port: list and tuple tests have been moved to their own scripts: test_tuple.py and test_list.py. Common tests for tuple, list and UserList are shared (in seq_test.py and list_test.py) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-09-02 03:53 Message: Logged In: YES user_id=80475 Applied as: Lib/test/test_slice.py 1.5. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-09-01 01:52 Message: Logged In: YES user_id=80475 Looks good. I added a couple of minor tests. Revised patch attached. Okay to apply. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-31 21:49 Message: Logged In: YES user_id=89016 Thanks! Here's another one: test_slice.py ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-31 01:04 Message: Logged In: YES user_id=80475 Done. See: Lib/test/test_longexp.py 1.9; previous revision: 1.8 Lib/test/test_pep263.py 1.3; previous revision: 1.2 Lib/test/test_sets.py 1.27; previous revision: 1.26 Lib/test/test_structseq.py 1.5; previous revision: 1.4 Lib/test/output/test_longexp delete; previous revision: 1.3 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-30 19:10 Message: Logged In: YES user_id=80475 Will do! ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-30 19:06 Message: Logged In: YES user_id=89016 Here's test_structse.py converted with a few additional tests. If all three scripts are OK, could you check them in Raymond, as I'll be on vacation for three weeks? ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-09 17:47 Message: Logged In: YES user_id=89016 Here are two simple ones: test_pep263 and test_longexp. I can't think of any additional tests to add. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-23 15:38 Message: Logged In: YES user_id=80475 Fixed Walter's review comments and committed as Lib/test/test_compile.py 1.19 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-23 13:49 Message: Logged In: YES user_id=89016 Are you sure, that the code path is the same in the new test_argument_handling() as in the old test? I.e. is "eval('lambda a,a: 0')" the same as "exec 'def f(a, a): pass'"? The print statement in test_float_literals should be changed to a comment. Should the imports in test_unary_minus() be moved to the start of the script? Otherwise the test looks (and runs) OK. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-20 21:26 Message: Logged In: YES user_id=80475 Here's one for you: test_compile.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-19 11:46 Message: Logged In: YES user_id=89016 Maybe test_types.py should be split into several scripts: test_dict, test_tuple, test_list, test_int, test_long etc. Some of them already exist (like test_long), some don't. The constructor tests from test_builtin should probably be moved to the new test scripts as well. Furthermore we should try to share as much testing functionality as possible (e.g. between test_int and test_long, or between test_list and test_userlist) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-18 21:48 Message: Logged In: YES user_id=80475 Great! Can I suggest that test_types.py be next. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-18 16:27 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_builtin.py 1.21 Lib/test/test_complex.py 1.10 (I've moved the constructor tests from test_builtin to test_complex) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-18 01:49 Message: Logged In: YES user_id=80475 I added a few tests. If they are fine with you, go ahead and commit. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 23:36 Message: Logged In: YES user_id=89016 Here's the next one: text_complex.py. There one test that's currently commented out, because it crashes Python on Alpha (see http://www.python.org/sf/756093. The old test scripts states that tests for the constructor are in test_builtin, but I've added many tests to this script, so this is no longer true. We could move the tests to test_builtin, but IMHO that doesn't make sense, we'd better move the rest of the constructor tests from test_builtin to test_complex. I'd like to have a version of assertAlmostEqual() in unittest.py that can cope with complex numbers, but this would have to be coordinated with the standalone version of PyUnit (and it would probably have to wait until the 2.4 cycle starts) (I noticed that there is no assertAlmostEqual in the code on pyunit.sf.net anyway.) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 14:06 Message: Logged In: YES user_id=89016 I'd like to keep this patch open, as it is an ongoing task (the next test scripts to be converted will be test_complex and then maybe test_marshal) ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 23:55 Message: Logged In: YES user_id=357491 OK, all tests pass cleanly. Applied as revision 1.6. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 21:51 Message: Logged In: YES user_id=89016 The joys of unittesting: Breaking code to make it better! ;) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 21:41 Message: Logged In: YES user_id=80475 Okay, it runs fine here. Once Brett confirms that it runs on the Mac, go ahead and load it. P.S. Your improved test_mimetools.py helped detect a latent error in mimetools.py when it was run under Windows. Tim made the fix this weekend. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 21:33 Message: Logged In: YES user_id=89016 Strange, I didn't get this failure here, but I got it on my laptop at home. I've removed the comparison with the getatime() value from test_time(). I hope this fixes it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 21:09 Message: Logged In: YES user_id=80475 Walter, there is one failure left: ====================================== ================================ FAIL: test_time (__main__.PosixPathTest) ------------------------------------------------------------------- --- Traceback (most recent call last): File "test_posixpath.py", line 125, in test_time self.assert_( File "C:\PY23\lib\unittest.py", line 268, in failUnless if not expr: raise self.failureException, msg AssertionError Brett, after Walter revises the patch, just load the patch and make sure the test runs on the Mac. Between the three of us, we can validate the suite on three different platforms. Cheers. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 20:20 Message: Logged In: YES user_id=357491 Just wanting to work me like a dog, huh, Raymond? =) And to clarify for my and Walter's benefit, when you say guards, you mean that the tests don't crap out and say they failed on Windows, right? I thought posixpath was not meant to work under Windows. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 20:09 Message: Logged In: YES user_id=89016 I didn't realize that test_posixpath must work on Windows too. Here's a new version. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 18:04 Message: Logged In: YES user_id=80475 The previous comment applied to another patch. It should have said: Assigning to Brett to make sure the patch runs on the Mac. Don't accept this one until it has guards that allow the tests to run on Windows. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 17:59 Message: Logged In: YES user_id=80475 Assigning to Brett to give experience doing a detail review on this type of change. * examine every line of the diff and consider whether there is any semantic change (exceptions raised, etc). * apply the diff and run the test suite * in the interactive mode, call-up each function and make sure it behaves as expected (this is necessary because the test coverage is very low). * verify that the whitespace has been cleaned up. * look for missing changes (such as use of +=) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 17:44 Message: Logged In: YES user_id=80475 The test file now has dependencies that do not apply to windows. The failure messages are attached. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 14:48 Message: Logged In: YES user_id=89016 Here's the next one: test_posixpath.py with many additional tests. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 19:33 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_mimetools delete Lib/test/test_mimetools.py 1.4 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-22 18:18 Message: Logged In: YES user_id=80475 test_mimetools.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 17:05 Message: Logged In: YES user_id=89016 I've attached a third version of test_mimetools.py that does some checks for the mimetools.Message class. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-21 15:04 Message: Logged In: YES user_id=80475 Attaching a slightly modified test_mimetools which covers more encodings and has a stronger set test. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-19 01:46 Message: Logged In: YES user_id=89016 Agreed, this is too much magic for too little gain. Back to business: Here is test_mimetools ported to PyUnit. Tests for mimetools.Message are still missing. If you can think of any tests please add them. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 05:18 Message: Logged In: YES user_id=80475 Get the module with sys.modules: tests = test_support.findtestclasses(sys.modules [__name__]) test_support.unittest(*tests) Yeah, the inheritance thing is a problem. I was trying to avoid having to modify unittest.TestCase to have a metaclass. The control of the module is kept in a separate SF project and one of its goals is to be backward compatible through 1.5.2 (meaning no metaclasses). A possible workaround is to define a modified testcase in test_support so that people don't import unittest directly anymore: test_support.py ------------------------- import unittest class SmartTestCase(unittest.TestCase): __metaclass__ = autotracktests pass test_sets.py ------------------ class TestBasicOps(test_support.SmartTestCase): run = False . . . class TestBasicOpsEmpty(TestBasicOps): def setUp(self): . . . Still, this is starting to seem a bit magical and tricky. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 04:52 Message: Logged In: YES user_id=89016 But how do I pass the module object from inside the module? And skipping abstract classes seems to be more work in this version: If skipping is done via a class attribute, derived classes have to explicitely reset this flag because of interitance. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 03:59 Message: Logged In: YES user_id=80475 Good call. Instead of using metaclasses, perhaps add a module introspector function to test_support: def findtestclasses(mod): tests = [] for elem in dir(mod): member = getattr(mod, elem) if type(member) != type: continue if issubclass(member, unittest.TestCase): tests.append(member) return tests ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 03:45 Message: Logged In: YES user_id=89016 But this can be solved with a special non-inheritable class attribute: class BaseTest(unittest.TestCase): run = False Then the metaclass can do the following: def __new__(cls, name, bases, dict): if "run" not in dict: dict["run"] = True cls = type.__new__(cls, name, bases, dict) if cls.run: tests.append(cls) return cls ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 03:04 Message: Logged In: YES user_id=80475 I don't think metaclasses or module introspection would help whenever there are classes that derive from TestCase but are not meant to be run directly (their subclasses have the setup/teardown/or class data). test_sets.py has examples of that kind of thing. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 02:50 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_array.py 1.20 Lib/test/test_winsound.py 1.5 Lib/test/output/test_winsound delete > The approach of using tests.append() is elegant and > makes it easier to verify that no tests are being omitted. The most elegant approach would probably be a metaclass that collects all TestCase subclasses that get defined. Classes that only serve as a base class could be skipped by specifying a certain class attribute. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 01:35 Message: Logged In: YES user_id=80475 The approach of using tests.append() is elegant and makes it easier to verify that no tests are being omitted. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 From noreply at sourceforge.net Tue May 25 16:21:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 25 16:21:29 2004 Subject: [Patches] [ python-Patches-736962 ] Port tests to unittest (Part 2) Message-ID: Patches item #736962, was opened at 2003-05-13 05:45 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 Category: Tests Group: None Status: Open Resolution: Accepted Priority: 5 Submitted By: Walter D?rwald (doerwalter) Assigned to: Walter D?rwald (doerwalter) Summary: Port tests to unittest (Part 2) Initial Comment: Here are the next test scripts ported to PyUnit: test_winsound and test_array. For test_array many additional tests have been added (code coverage is at 91%) ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-25 15:21 Message: Logged In: YES user_id=80475 IMO, it is fine as is. I would be suprised if some code were relying on UserDict.update(None) raising an exception. This sort of thing also comes up also in places like the string and random modules whereever a function or method with an optional argument gets wrapped by another function. The only way around this is to use *args and then you break code that used the unfortunately named "dict" argument as a keyword. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-05-25 14:32 Message: Logged In: YES user_id=89016 Merging test_dict and test_userdict revealed a difference between dict and UserDict.UserDict: >>> import UserDict >>> d = UserDict.UserDict() >>> d.update(None) >>> d {} >>> d = {} >>> d.update(None) Traceback (most recent call last): File "", line 1, in ? TypeError: iteration over non-sequence Should we do anything about this? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 11:03 Message: Logged In: YES user_id=80475 test_dict is fine. Consider beefing up the tests for update(). As of Py2.4, it now takes all the same argument possibilites as dict(): d.update(a=1, b=2) d.update([('a', 1), ('b', 2)]) Also, try to link in: from test_userdict import TestMappingProtocol This was supposed to provide tests common to all mappings. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-04-03 06:06 Message: Logged In: YES user_id=89016 The attached test_dict.py moves the dict tests from test_types.py to a new PyUnit test script. Code coverage for dictobject.c is at 90.76% (up from 89.51%) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-03-15 07:17 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_binascii delete Lib/test/test_binascii.py 1.17 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 15:26 Message: Logged In: YES user_id=80475 Okay, this one is fine. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-18 18:49 Message: Logged In: YES user_id=89016 Here's the next one: test_binascii.py. Code coverage in binascii.c is at 92% (could be improved by enhancing test_binhex.py). ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 17:46 Message: Logged In: YES user_id=33168 Looked good to me, test_future_pyunit.diff checked in as: * Lib/test/badsyntax_future3.py 1.2 * Lib/test/badsyntax_future4.py 1.2 * Lib/test/badsyntax_future5.py 1.2 * Lib/test/badsyntax_future6.py 1.2 * Lib/test/badsyntax_future7.py 1.2 * Lib/test/badsyntax_future8.py 1.1 * Lib/test/badsyntax_future9.py 1.1 * Lib/test/test_future.py 1.7 * Lib/test/test_future1.py 1.3 * Lib/test/test_future2.py 1.2 * Lib/test/output/test_future 1.4 ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 16:47 Message: Logged In: YES user_id=33168 Walter, I didn't get a mail from SF either. After you do a cvs add, you need to use -N to cvs diff. For example, cvs diff -N new_file_added_to_cvs_but_not_committed. If you look carefully, you can see this in the patch on the diff line for each file. I'll take a look at the patch. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-13 15:18 Message: Logged In: YES user_id=89016 Here is a version of test_future ported to PyUnit (test_future_pyunit.diff). If this makes sense to you Neal, please check it in. (BTW, how did you convince CVS to include badsyntax_future8.py and badsyntax_future9.py in the diff? Doing a "cvs add" only results in " Lib/test/badsyntax_future8.py is a new entry, no comparison available") ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-11 07:36 Message: Logged In: YES user_id=89016 md5/weakref patch checked in as: Lib/test/test_weakref.py 1.33 Lib/test/test_md5.py 1.5 Lib/test/output/test_md5 remove ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 01:05 Message: Logged In: YES user_id=33168 Improve coverage for test_future too. I'm not sure if test future can be ported to PyUnit. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 00:52 Message: Logged In: YES user_id=33168 Attached are two tests (in one file, I'm being lazy, sorry) to improve test coverage for md5c.c and _weakref.c. test_md5 was ported to unittest, weakref, just adds two little tests to get coverage up to 100%. If you like, just go ahead and check in. You will need to remove Lib/test/output/test_md5 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-08 06:42 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/list_tests.py 1.1 Lib/test/seq_tests.py 1.1 Lib/test/test_list.py 1.1 Lib/test/test_tuple.py 1.1 Lib/test/test_types.py 1.56 Lib/test/test_userlist.py 1.11 Lib/test/output/test_types 1.3 Next will be test_binascii.py before I continue with test_types.py. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-12-07 18:49 Message: Logged In: YES user_id=80475 Looks good. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-11-27 14:48 Message: Logged In: YES user_id=89016 Here the first part of the test_types port: list and tuple tests have been moved to their own scripts: test_tuple.py and test_list.py. Common tests for tuple, list and UserList are shared (in seq_test.py and list_test.py) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-09-01 20:53 Message: Logged In: YES user_id=80475 Applied as: Lib/test/test_slice.py 1.5. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-31 18:52 Message: Logged In: YES user_id=80475 Looks good. I added a couple of minor tests. Revised patch attached. Okay to apply. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-31 14:49 Message: Logged In: YES user_id=89016 Thanks! Here's another one: test_slice.py ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-30 18:04 Message: Logged In: YES user_id=80475 Done. See: Lib/test/test_longexp.py 1.9; previous revision: 1.8 Lib/test/test_pep263.py 1.3; previous revision: 1.2 Lib/test/test_sets.py 1.27; previous revision: 1.26 Lib/test/test_structseq.py 1.5; previous revision: 1.4 Lib/test/output/test_longexp delete; previous revision: 1.3 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-30 12:10 Message: Logged In: YES user_id=80475 Will do! ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-30 12:06 Message: Logged In: YES user_id=89016 Here's test_structse.py converted with a few additional tests. If all three scripts are OK, could you check them in Raymond, as I'll be on vacation for three weeks? ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-09 10:47 Message: Logged In: YES user_id=89016 Here are two simple ones: test_pep263 and test_longexp. I can't think of any additional tests to add. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-23 08:38 Message: Logged In: YES user_id=80475 Fixed Walter's review comments and committed as Lib/test/test_compile.py 1.19 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-23 06:49 Message: Logged In: YES user_id=89016 Are you sure, that the code path is the same in the new test_argument_handling() as in the old test? I.e. is "eval('lambda a,a: 0')" the same as "exec 'def f(a, a): pass'"? The print statement in test_float_literals should be changed to a comment. Should the imports in test_unary_minus() be moved to the start of the script? Otherwise the test looks (and runs) OK. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-20 14:26 Message: Logged In: YES user_id=80475 Here's one for you: test_compile.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-19 04:46 Message: Logged In: YES user_id=89016 Maybe test_types.py should be split into several scripts: test_dict, test_tuple, test_list, test_int, test_long etc. Some of them already exist (like test_long), some don't. The constructor tests from test_builtin should probably be moved to the new test scripts as well. Furthermore we should try to share as much testing functionality as possible (e.g. between test_int and test_long, or between test_list and test_userlist) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-18 14:48 Message: Logged In: YES user_id=80475 Great! Can I suggest that test_types.py be next. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-18 09:27 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_builtin.py 1.21 Lib/test/test_complex.py 1.10 (I've moved the constructor tests from test_builtin to test_complex) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-17 18:49 Message: Logged In: YES user_id=80475 I added a few tests. If they are fine with you, go ahead and commit. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 16:36 Message: Logged In: YES user_id=89016 Here's the next one: text_complex.py. There one test that's currently commented out, because it crashes Python on Alpha (see http://www.python.org/sf/756093. The old test scripts states that tests for the constructor are in test_builtin, but I've added many tests to this script, so this is no longer true. We could move the tests to test_builtin, but IMHO that doesn't make sense, we'd better move the rest of the constructor tests from test_builtin to test_complex. I'd like to have a version of assertAlmostEqual() in unittest.py that can cope with complex numbers, but this would have to be coordinated with the standalone version of PyUnit (and it would probably have to wait until the 2.4 cycle starts) (I noticed that there is no assertAlmostEqual in the code on pyunit.sf.net anyway.) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 07:06 Message: Logged In: YES user_id=89016 I'd like to keep this patch open, as it is an ongoing task (the next test scripts to be converted will be test_complex and then maybe test_marshal) ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 16:55 Message: Logged In: YES user_id=357491 OK, all tests pass cleanly. Applied as revision 1.6. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 14:51 Message: Logged In: YES user_id=89016 The joys of unittesting: Breaking code to make it better! ;) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 14:41 Message: Logged In: YES user_id=80475 Okay, it runs fine here. Once Brett confirms that it runs on the Mac, go ahead and load it. P.S. Your improved test_mimetools.py helped detect a latent error in mimetools.py when it was run under Windows. Tim made the fix this weekend. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 14:33 Message: Logged In: YES user_id=89016 Strange, I didn't get this failure here, but I got it on my laptop at home. I've removed the comparison with the getatime() value from test_time(). I hope this fixes it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 14:09 Message: Logged In: YES user_id=80475 Walter, there is one failure left: ====================================== ================================ FAIL: test_time (__main__.PosixPathTest) ------------------------------------------------------------------- --- Traceback (most recent call last): File "test_posixpath.py", line 125, in test_time self.assert_( File "C:\PY23\lib\unittest.py", line 268, in failUnless if not expr: raise self.failureException, msg AssertionError Brett, after Walter revises the patch, just load the patch and make sure the test runs on the Mac. Between the three of us, we can validate the suite on three different platforms. Cheers. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 13:20 Message: Logged In: YES user_id=357491 Just wanting to work me like a dog, huh, Raymond? =) And to clarify for my and Walter's benefit, when you say guards, you mean that the tests don't crap out and say they failed on Windows, right? I thought posixpath was not meant to work under Windows. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 13:09 Message: Logged In: YES user_id=89016 I didn't realize that test_posixpath must work on Windows too. Here's a new version. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 11:04 Message: Logged In: YES user_id=80475 The previous comment applied to another patch. It should have said: Assigning to Brett to make sure the patch runs on the Mac. Don't accept this one until it has guards that allow the tests to run on Windows. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 10:59 Message: Logged In: YES user_id=80475 Assigning to Brett to give experience doing a detail review on this type of change. * examine every line of the diff and consider whether there is any semantic change (exceptions raised, etc). * apply the diff and run the test suite * in the interactive mode, call-up each function and make sure it behaves as expected (this is necessary because the test coverage is very low). * verify that the whitespace has been cleaned up. * look for missing changes (such as use of +=) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 10:44 Message: Logged In: YES user_id=80475 The test file now has dependencies that do not apply to windows. The failure messages are attached. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 07:48 Message: Logged In: YES user_id=89016 Here's the next one: test_posixpath.py with many additional tests. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 12:33 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_mimetools delete Lib/test/test_mimetools.py 1.4 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-22 11:18 Message: Logged In: YES user_id=80475 test_mimetools.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 10:05 Message: Logged In: YES user_id=89016 I've attached a third version of test_mimetools.py that does some checks for the mimetools.Message class. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-21 08:04 Message: Logged In: YES user_id=80475 Attaching a slightly modified test_mimetools which covers more encodings and has a stronger set test. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 18:46 Message: Logged In: YES user_id=89016 Agreed, this is too much magic for too little gain. Back to business: Here is test_mimetools ported to PyUnit. Tests for mimetools.Message are still missing. If you can think of any tests please add them. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 22:18 Message: Logged In: YES user_id=80475 Get the module with sys.modules: tests = test_support.findtestclasses(sys.modules [__name__]) test_support.unittest(*tests) Yeah, the inheritance thing is a problem. I was trying to avoid having to modify unittest.TestCase to have a metaclass. The control of the module is kept in a separate SF project and one of its goals is to be backward compatible through 1.5.2 (meaning no metaclasses). A possible workaround is to define a modified testcase in test_support so that people don't import unittest directly anymore: test_support.py ------------------------- import unittest class SmartTestCase(unittest.TestCase): __metaclass__ = autotracktests pass test_sets.py ------------------ class TestBasicOps(test_support.SmartTestCase): run = False . . . class TestBasicOpsEmpty(TestBasicOps): def setUp(self): . . . Still, this is starting to seem a bit magical and tricky. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-17 21:52 Message: Logged In: YES user_id=89016 But how do I pass the module object from inside the module? And skipping abstract classes seems to be more work in this version: If skipping is done via a class attribute, derived classes have to explicitely reset this flag because of interitance. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 20:59 Message: Logged In: YES user_id=80475 Good call. Instead of using metaclasses, perhaps add a module introspector function to test_support: def findtestclasses(mod): tests = [] for elem in dir(mod): member = getattr(mod, elem) if type(member) != type: continue if issubclass(member, unittest.TestCase): tests.append(member) return tests ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-17 20:45 Message: Logged In: YES user_id=89016 But this can be solved with a special non-inheritable class attribute: class BaseTest(unittest.TestCase): run = False Then the metaclass can do the following: def __new__(cls, name, bases, dict): if "run" not in dict: dict["run"] = True cls = type.__new__(cls, name, bases, dict) if cls.run: tests.append(cls) return cls ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 20:04 Message: Logged In: YES user_id=80475 I don't think metaclasses or module introspection would help whenever there are classes that derive from TestCase but are not meant to be run directly (their subclasses have the setup/teardown/or class data). test_sets.py has examples of that kind of thing. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-17 19:50 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_array.py 1.20 Lib/test/test_winsound.py 1.5 Lib/test/output/test_winsound delete > The approach of using tests.append() is elegant and > makes it easier to verify that no tests are being omitted. The most elegant approach would probably be a metaclass that collects all TestCase subclasses that get defined. Classes that only serve as a base class could be skipped by specifying a certain class attribute. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 18:35 Message: Logged In: YES user_id=80475 The approach of using tests.append() is elegant and makes it easier to verify that no tests are being omitted. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 From noreply at sourceforge.net Tue May 25 16:29:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 25 16:30:07 2004 Subject: [Patches] [ python-Patches-736962 ] Port tests to unittest (Part 2) Message-ID: Patches item #736962, was opened at 2003-05-13 06:45 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 Category: Tests Group: None Status: Open Resolution: Accepted Priority: 5 Submitted By: Walter D?rwald (doerwalter) Assigned to: Walter D?rwald (doerwalter) Summary: Port tests to unittest (Part 2) Initial Comment: Here are the next test scripts ported to PyUnit: test_winsound and test_array. For test_array many additional tests have been added (code coverage is at 91%) ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-25 16:29 Message: Logged In: YES user_id=31435 Don't really care myself. Would be nice if the docstring for dict.update() matched the new realities, though . BTW, there is a reliable and conventional way to detect whether the 'dict' argument was passed to UserDict.update: use a unique marker object instead of the universally visible None. Like _marker = object() at module level, and ...(self, dict=_marker, ...) in the method defn. That can't be fooled without deliberate intent to deceive. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-25 16:21 Message: Logged In: YES user_id=80475 IMO, it is fine as is. I would be suprised if some code were relying on UserDict.update(None) raising an exception. This sort of thing also comes up also in places like the string and random modules whereever a function or method with an optional argument gets wrapped by another function. The only way around this is to use *args and then you break code that used the unfortunately named "dict" argument as a keyword. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-05-25 15:32 Message: Logged In: YES user_id=89016 Merging test_dict and test_userdict revealed a difference between dict and UserDict.UserDict: >>> import UserDict >>> d = UserDict.UserDict() >>> d.update(None) >>> d {} >>> d = {} >>> d.update(None) Traceback (most recent call last): File "", line 1, in ? TypeError: iteration over non-sequence Should we do anything about this? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 12:03 Message: Logged In: YES user_id=80475 test_dict is fine. Consider beefing up the tests for update(). As of Py2.4, it now takes all the same argument possibilites as dict(): d.update(a=1, b=2) d.update([('a', 1), ('b', 2)]) Also, try to link in: from test_userdict import TestMappingProtocol This was supposed to provide tests common to all mappings. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-04-03 06:06 Message: Logged In: YES user_id=89016 The attached test_dict.py moves the dict tests from test_types.py to a new PyUnit test script. Code coverage for dictobject.c is at 90.76% (up from 89.51%) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-03-15 07:17 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_binascii delete Lib/test/test_binascii.py 1.17 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 15:26 Message: Logged In: YES user_id=80475 Okay, this one is fine. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-18 18:49 Message: Logged In: YES user_id=89016 Here's the next one: test_binascii.py. Code coverage in binascii.c is at 92% (could be improved by enhancing test_binhex.py). ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 17:46 Message: Logged In: YES user_id=33168 Looked good to me, test_future_pyunit.diff checked in as: * Lib/test/badsyntax_future3.py 1.2 * Lib/test/badsyntax_future4.py 1.2 * Lib/test/badsyntax_future5.py 1.2 * Lib/test/badsyntax_future6.py 1.2 * Lib/test/badsyntax_future7.py 1.2 * Lib/test/badsyntax_future8.py 1.1 * Lib/test/badsyntax_future9.py 1.1 * Lib/test/test_future.py 1.7 * Lib/test/test_future1.py 1.3 * Lib/test/test_future2.py 1.2 * Lib/test/output/test_future 1.4 ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 16:47 Message: Logged In: YES user_id=33168 Walter, I didn't get a mail from SF either. After you do a cvs add, you need to use -N to cvs diff. For example, cvs diff -N new_file_added_to_cvs_but_not_committed. If you look carefully, you can see this in the patch on the diff line for each file. I'll take a look at the patch. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-13 15:18 Message: Logged In: YES user_id=89016 Here is a version of test_future ported to PyUnit (test_future_pyunit.diff). If this makes sense to you Neal, please check it in. (BTW, how did you convince CVS to include badsyntax_future8.py and badsyntax_future9.py in the diff? Doing a "cvs add" only results in " Lib/test/badsyntax_future8.py is a new entry, no comparison available") ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-11 07:36 Message: Logged In: YES user_id=89016 md5/weakref patch checked in as: Lib/test/test_weakref.py 1.33 Lib/test/test_md5.py 1.5 Lib/test/output/test_md5 remove ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 01:05 Message: Logged In: YES user_id=33168 Improve coverage for test_future too. I'm not sure if test future can be ported to PyUnit. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 00:52 Message: Logged In: YES user_id=33168 Attached are two tests (in one file, I'm being lazy, sorry) to improve test coverage for md5c.c and _weakref.c. test_md5 was ported to unittest, weakref, just adds two little tests to get coverage up to 100%. If you like, just go ahead and check in. You will need to remove Lib/test/output/test_md5 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-08 06:42 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/list_tests.py 1.1 Lib/test/seq_tests.py 1.1 Lib/test/test_list.py 1.1 Lib/test/test_tuple.py 1.1 Lib/test/test_types.py 1.56 Lib/test/test_userlist.py 1.11 Lib/test/output/test_types 1.3 Next will be test_binascii.py before I continue with test_types.py. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-12-07 18:49 Message: Logged In: YES user_id=80475 Looks good. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-11-27 14:48 Message: Logged In: YES user_id=89016 Here the first part of the test_types port: list and tuple tests have been moved to their own scripts: test_tuple.py and test_list.py. Common tests for tuple, list and UserList are shared (in seq_test.py and list_test.py) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-09-01 21:53 Message: Logged In: YES user_id=80475 Applied as: Lib/test/test_slice.py 1.5. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-31 19:52 Message: Logged In: YES user_id=80475 Looks good. I added a couple of minor tests. Revised patch attached. Okay to apply. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-31 15:49 Message: Logged In: YES user_id=89016 Thanks! Here's another one: test_slice.py ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-30 19:04 Message: Logged In: YES user_id=80475 Done. See: Lib/test/test_longexp.py 1.9; previous revision: 1.8 Lib/test/test_pep263.py 1.3; previous revision: 1.2 Lib/test/test_sets.py 1.27; previous revision: 1.26 Lib/test/test_structseq.py 1.5; previous revision: 1.4 Lib/test/output/test_longexp delete; previous revision: 1.3 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-30 13:10 Message: Logged In: YES user_id=80475 Will do! ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-30 13:06 Message: Logged In: YES user_id=89016 Here's test_structse.py converted with a few additional tests. If all three scripts are OK, could you check them in Raymond, as I'll be on vacation for three weeks? ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-09 11:47 Message: Logged In: YES user_id=89016 Here are two simple ones: test_pep263 and test_longexp. I can't think of any additional tests to add. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-23 09:38 Message: Logged In: YES user_id=80475 Fixed Walter's review comments and committed as Lib/test/test_compile.py 1.19 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-23 07:49 Message: Logged In: YES user_id=89016 Are you sure, that the code path is the same in the new test_argument_handling() as in the old test? I.e. is "eval('lambda a,a: 0')" the same as "exec 'def f(a, a): pass'"? The print statement in test_float_literals should be changed to a comment. Should the imports in test_unary_minus() be moved to the start of the script? Otherwise the test looks (and runs) OK. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-20 15:26 Message: Logged In: YES user_id=80475 Here's one for you: test_compile.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-19 05:46 Message: Logged In: YES user_id=89016 Maybe test_types.py should be split into several scripts: test_dict, test_tuple, test_list, test_int, test_long etc. Some of them already exist (like test_long), some don't. The constructor tests from test_builtin should probably be moved to the new test scripts as well. Furthermore we should try to share as much testing functionality as possible (e.g. between test_int and test_long, or between test_list and test_userlist) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-18 15:48 Message: Logged In: YES user_id=80475 Great! Can I suggest that test_types.py be next. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-18 10:27 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_builtin.py 1.21 Lib/test/test_complex.py 1.10 (I've moved the constructor tests from test_builtin to test_complex) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-17 19:49 Message: Logged In: YES user_id=80475 I added a few tests. If they are fine with you, go ahead and commit. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 17:36 Message: Logged In: YES user_id=89016 Here's the next one: text_complex.py. There one test that's currently commented out, because it crashes Python on Alpha (see http://www.python.org/sf/756093. The old test scripts states that tests for the constructor are in test_builtin, but I've added many tests to this script, so this is no longer true. We could move the tests to test_builtin, but IMHO that doesn't make sense, we'd better move the rest of the constructor tests from test_builtin to test_complex. I'd like to have a version of assertAlmostEqual() in unittest.py that can cope with complex numbers, but this would have to be coordinated with the standalone version of PyUnit (and it would probably have to wait until the 2.4 cycle starts) (I noticed that there is no assertAlmostEqual in the code on pyunit.sf.net anyway.) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 08:06 Message: Logged In: YES user_id=89016 I'd like to keep this patch open, as it is an ongoing task (the next test scripts to be converted will be test_complex and then maybe test_marshal) ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 17:55 Message: Logged In: YES user_id=357491 OK, all tests pass cleanly. Applied as revision 1.6. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 15:51 Message: Logged In: YES user_id=89016 The joys of unittesting: Breaking code to make it better! ;) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 15:41 Message: Logged In: YES user_id=80475 Okay, it runs fine here. Once Brett confirms that it runs on the Mac, go ahead and load it. P.S. Your improved test_mimetools.py helped detect a latent error in mimetools.py when it was run under Windows. Tim made the fix this weekend. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 15:33 Message: Logged In: YES user_id=89016 Strange, I didn't get this failure here, but I got it on my laptop at home. I've removed the comparison with the getatime() value from test_time(). I hope this fixes it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 15:09 Message: Logged In: YES user_id=80475 Walter, there is one failure left: ====================================== ================================ FAIL: test_time (__main__.PosixPathTest) ------------------------------------------------------------------- --- Traceback (most recent call last): File "test_posixpath.py", line 125, in test_time self.assert_( File "C:\PY23\lib\unittest.py", line 268, in failUnless if not expr: raise self.failureException, msg AssertionError Brett, after Walter revises the patch, just load the patch and make sure the test runs on the Mac. Between the three of us, we can validate the suite on three different platforms. Cheers. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 14:20 Message: Logged In: YES user_id=357491 Just wanting to work me like a dog, huh, Raymond? =) And to clarify for my and Walter's benefit, when you say guards, you mean that the tests don't crap out and say they failed on Windows, right? I thought posixpath was not meant to work under Windows. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 14:09 Message: Logged In: YES user_id=89016 I didn't realize that test_posixpath must work on Windows too. Here's a new version. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 12:04 Message: Logged In: YES user_id=80475 The previous comment applied to another patch. It should have said: Assigning to Brett to make sure the patch runs on the Mac. Don't accept this one until it has guards that allow the tests to run on Windows. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 11:59 Message: Logged In: YES user_id=80475 Assigning to Brett to give experience doing a detail review on this type of change. * examine every line of the diff and consider whether there is any semantic change (exceptions raised, etc). * apply the diff and run the test suite * in the interactive mode, call-up each function and make sure it behaves as expected (this is necessary because the test coverage is very low). * verify that the whitespace has been cleaned up. * look for missing changes (such as use of +=) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 11:44 Message: Logged In: YES user_id=80475 The test file now has dependencies that do not apply to windows. The failure messages are attached. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 08:48 Message: Logged In: YES user_id=89016 Here's the next one: test_posixpath.py with many additional tests. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 13:33 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_mimetools delete Lib/test/test_mimetools.py 1.4 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-22 12:18 Message: Logged In: YES user_id=80475 test_mimetools.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 11:05 Message: Logged In: YES user_id=89016 I've attached a third version of test_mimetools.py that does some checks for the mimetools.Message class. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-21 09:04 Message: Logged In: YES user_id=80475 Attaching a slightly modified test_mimetools which covers more encodings and has a stronger set test. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 19:46 Message: Logged In: YES user_id=89016 Agreed, this is too much magic for too little gain. Back to business: Here is test_mimetools ported to PyUnit. Tests for mimetools.Message are still missing. If you can think of any tests please add them. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 23:18 Message: Logged In: YES user_id=80475 Get the module with sys.modules: tests = test_support.findtestclasses(sys.modules [__name__]) test_support.unittest(*tests) Yeah, the inheritance thing is a problem. I was trying to avoid having to modify unittest.TestCase to have a metaclass. The control of the module is kept in a separate SF project and one of its goals is to be backward compatible through 1.5.2 (meaning no metaclasses). A possible workaround is to define a modified testcase in test_support so that people don't import unittest directly anymore: test_support.py ------------------------- import unittest class SmartTestCase(unittest.TestCase): __metaclass__ = autotracktests pass test_sets.py ------------------ class TestBasicOps(test_support.SmartTestCase): run = False . . . class TestBasicOpsEmpty(TestBasicOps): def setUp(self): . . . Still, this is starting to seem a bit magical and tricky. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-17 22:52 Message: Logged In: YES user_id=89016 But how do I pass the module object from inside the module? And skipping abstract classes seems to be more work in this version: If skipping is done via a class attribute, derived classes have to explicitely reset this flag because of interitance. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 21:59 Message: Logged In: YES user_id=80475 Good call. Instead of using metaclasses, perhaps add a module introspector function to test_support: def findtestclasses(mod): tests = [] for elem in dir(mod): member = getattr(mod, elem) if type(member) != type: continue if issubclass(member, unittest.TestCase): tests.append(member) return tests ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-17 21:45 Message: Logged In: YES user_id=89016 But this can be solved with a special non-inheritable class attribute: class BaseTest(unittest.TestCase): run = False Then the metaclass can do the following: def __new__(cls, name, bases, dict): if "run" not in dict: dict["run"] = True cls = type.__new__(cls, name, bases, dict) if cls.run: tests.append(cls) return cls ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 21:04 Message: Logged In: YES user_id=80475 I don't think metaclasses or module introspection would help whenever there are classes that derive from TestCase but are not meant to be run directly (their subclasses have the setup/teardown/or class data). test_sets.py has examples of that kind of thing. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-17 20:50 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_array.py 1.20 Lib/test/test_winsound.py 1.5 Lib/test/output/test_winsound delete > The approach of using tests.append() is elegant and > makes it easier to verify that no tests are being omitted. The most elegant approach would probably be a metaclass that collects all TestCase subclasses that get defined. Classes that only serve as a base class could be skipped by specifying a certain class attribute. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 19:35 Message: Logged In: YES user_id=80475 The approach of using tests.append() is elegant and makes it easier to verify that no tests are being omitted. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 From noreply at sourceforge.net Tue May 25 17:00:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 25 17:00:28 2004 Subject: [Patches] [ python-Patches-960406 ] unblock signals in threads Message-ID: Patches item #960406, was opened at 2004-05-25 17:00 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: unblock signals in threads Initial Comment: This is a patch which will correct the issues some people have with python's handling of signal handling in threads. It allows any thread to initially catch the signal mark it as triggered, allowing the main thread to later process it. (This is actually just restoring access to the functionality that was in Python 2.1) The special SIGINT handling for the python readline module has been changed so that it can now see an EINTR error code, rather than needing a longjmp out of the readline library itself. If the readline library python is being linked to doesn't have the callback features necessary, it will fall back to its old behavior. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 From beardsleycounseling at charter.net Tue May 25 17:07:26 2004 From: beardsleycounseling at charter.net (Rajendra Hymes) Date: Tue May 25 17:06:45 2004 Subject: [Patches] Do you want your P'ENIS to be HARD a1l the time? Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040525/db43cd25/attachment.html From bert at wanadoo.fr Wed May 26 02:33:31 2004 From: bert at wanadoo.fr (chia-yin) Date: Wed May 26 02:47:49 2004 Subject: [Patches] Blanca Message-ID: <200405260647.i4Q6lf4W056665@mxzilla3.xs4all.nl> http://star-teens.com/mc/luc/ This is absolutely a stunning spot where you get to view superb images and videos. Bouncing lemons, tight slits. Please click and look at them petting - each other. http://star-teens.com/mc/luc/ From noreply at sourceforge.net Wed May 26 06:46:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 06:52:11 2004 Subject: [Patches] [ python-Patches-949332 ] synchronous signals blocked in pthreads Message-ID: Patches item #949332, was opened at 2004-05-06 17:07 Message generated for change (Comment added) made by dmaurer You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: synchronous signals blocked in pthreads Initial Comment: In a pthreads environment, signals sent asynchronously from things like kill() or the terminal driver are sent to the process and can be handled by any thread. Signals that are sent synchronously from the instruction stream (signals like SIGSEGV, SIGFPE, or SIGILL) are only delivered to the thread that executed the instruction. The current Python threads, when implemented via pthreads, blocks these synchronous signals and so thread that generates one can hang the entire process. The attached patch will remove some signals the list of signals to be blocked. The list of signals to be removed correspond to signals that can be generated synchronously. ---------------------------------------------------------------------- Comment By: Dieter Maurer (dmaurer) Date: 2004-05-26 10:46 Message: Logged In: YES user_id=265829 This patch addresses a serious problem in Zope2 (and probably other multi-threaded) production environments with LinuxThreads. A crash of one thread brings other threads in an unsane state which they leave only on "kill -9". This lets Guido's "zdeamon" automatic restart logic fail. Even manual restarts are a bit difficult as each individual thread must be sent a "kill -9". Applying the patch lets at least the process die properly. I am not yet sure whether the return code is appropriate for "zdeamon" to restart the process. I will check that soon and report back. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 13:47 Message: Logged In: YES user_id=6380 As Tim mentioned, this is a possible stopgap fix for bug 949332. While many objections have been raised, I still think this is a safer approach than some of the more advanced suggestions. (Too bad it doesn't solve my own, related problem, which is that SIGTERM is ignored by default in a process started from a Python thread.) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 18:11 Message: Logged In: YES user_id=31435 Boosted the priority in the hopes this will get looked at before 2.3.4. Unfortunately, I'm not in a position to test a pthreads patch. I'll note that the good comments about *why* this change is being entertained would be far more useful in the code than in the patch comment here. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 From noreply at sourceforge.net Wed May 26 08:00:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 08:00:07 2004 Subject: [Patches] [ python-Patches-949332 ] synchronous signals blocked in pthreads Message-ID: Patches item #949332, was opened at 2004-05-06 17:07 Message generated for change (Comment added) made by dmaurer You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: synchronous signals blocked in pthreads Initial Comment: In a pthreads environment, signals sent asynchronously from things like kill() or the terminal driver are sent to the process and can be handled by any thread. Signals that are sent synchronously from the instruction stream (signals like SIGSEGV, SIGFPE, or SIGILL) are only delivered to the thread that executed the instruction. The current Python threads, when implemented via pthreads, blocks these synchronous signals and so thread that generates one can hang the entire process. The attached patch will remove some signals the list of signals to be blocked. The list of signals to be removed correspond to signals that can be generated synchronously. ---------------------------------------------------------------------- Comment By: Dieter Maurer (dmaurer) Date: 2004-05-26 12:00 Message: Logged In: YES user_id=265829 I verified that the return code is appropriate. ---------------------------------------------------------------------- Comment By: Dieter Maurer (dmaurer) Date: 2004-05-26 10:46 Message: Logged In: YES user_id=265829 This patch addresses a serious problem in Zope2 (and probably other multi-threaded) production environments with LinuxThreads. A crash of one thread brings other threads in an unsane state which they leave only on "kill -9". This lets Guido's "zdeamon" automatic restart logic fail. Even manual restarts are a bit difficult as each individual thread must be sent a "kill -9". Applying the patch lets at least the process die properly. I am not yet sure whether the return code is appropriate for "zdeamon" to restart the process. I will check that soon and report back. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 13:47 Message: Logged In: YES user_id=6380 As Tim mentioned, this is a possible stopgap fix for bug 949332. While many objections have been raised, I still think this is a safer approach than some of the more advanced suggestions. (Too bad it doesn't solve my own, related problem, which is that SIGTERM is ignored by default in a process started from a Python thread.) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 18:11 Message: Logged In: YES user_id=31435 Boosted the priority in the hopes this will get looked at before 2.3.4. Unfortunately, I'm not in a position to test a pthreads patch. I'll note that the good comments about *why* this change is being entertained would be far more useful in the code than in the patch comment here. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 From noreply at sourceforge.net Wed May 26 10:52:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 10:52:04 2004 Subject: [Patches] [ python-Patches-948614 ] Linux signals during threads Message-ID: Patches item #948614, was opened at 2004-05-05 13:13 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=948614&group_id=5470 Category: Core (C code) Group: Python 2.3 >Status: Closed >Resolution: Out of Date Priority: 5 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: Linux signals during threads Initial Comment: The Linux implementation of pthreads in the 2.0 through 2.4 (called LinuxThreads ) has one notable place where it diverges from the pthreads standard. A signal caused by a thread will only trigger the signal handler on that thread, and it is not seen or able to be handled by any other thread. The LinuxThreads package was first created in 1996 and is only now starting to be replaced by the new Linux 2.6 threading facility (called NPTL.) The non-standard LinuxThreads behavior falls afoul with Python's thread semantics, since it assumes that only the main thread can set or act on signal handlers. For the pthreads environment, it enforces this by blocking signals on all threads except the main thread. On LinuxThreads, this causes these signals to become ignored and lost, sometimes causing deadlock. The enclosed patch tests for LinuxThread's specific peculiarities in the configure script and adjusts the handling of thread creation and signal handling in order to give threads the documented Python thread and signal behavior. ---------------------------------------------------------------------- >Comment By: Andrew Langmead (langmead) Date: 2004-05-26 10:51 Message: Logged In: YES user_id=119306 http://sourceforge.net/tracker/ ?func=detail&aid=960406&group_id=5470&atid=305470 might be a better solution. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=948614&group_id=5470 From noreply at sourceforge.net Wed May 26 12:30:22 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 12:30:59 2004 Subject: [Patches] [ python-Patches-876130 ] add C API to datetime module Message-ID: Patches item #876130, was opened at 2004-01-13 08:20 Message generated for change (Comment added) made by atuining You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 Category: Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Anthony Tuininga (atuining) Assigned to: M.-A. Lemburg (lemburg) Summary: add C API to datetime module Initial Comment: The datetime module does not have a C API which means that modules written in C cannot take advantage of the datetime module without incurring significant overhead. These patches supply this lack and are based on the C API exported by the mxDateTime module and the cStringIO module and enhanced as suggested by Guido. ---------------------------------------------------------------------- >Comment By: Anthony Tuininga (atuining) Date: 2004-05-26 10:30 Message: Logged In: YES user_id=619560 Attached are the modified patches based on your suggestions. I have tested them with my own module cx_Oracle and they appear to work correctly. Any further suggestions would be appreciated. I would like to see this included for Python 2.4 if at all possible. Please let me know what I need to do (if anything) to get this to happen. As for the TimeFromTicks() routine, Oracle has absolutely no use for a "time-only" data type and does not support it. It clearly does not have broad support so I really have no desire to implement it since I will never use it myself. If you consider it important, you could add it yourself or we could wait until someone else who requires it for their implementation of the DB API requires it. I'm not trying to be rude or trying to denigrate the DB API, simply trying to be practical. Please let me know if this is offensive in any way. Thanks. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-19 02:28 Message: Logged In: YES user_id=38388 >From the DB API specs: TimeFromTicks(ticks) This function constructs an object holding a time value from the given ticks value (number of seconds since the epoch; see the documentation of the standard Python time module for details). The reason for this having this API is to be able to construct an object suitable for passing to the database given a Unix ticks value. Since the ticks value contains both a date and a time part and most databases have a special column type for time value, we need two constructor APIs, one to extract just the date part and one for the time part. Here's the mxDateTime implementation for reference: def TimeFromTicks(ticks, # Locals: DateTimeDelta=DateTimeDelta,localtime=_time.localtime): """ TimeFromTicks(ticks) Constructs a DateTimeDelta instance pointing to the local time indicated by the given ticks value. The date part is ignored. """ return apply(DateTimeDelta, (0,) + localtime(ticks)[3:6]) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-03 11:46 Message: Logged In: YES user_id=31435 I don't understand what TimeFromTicks() is supposed to do. Seconds-from-the-epoch is a point in time, not a time-of- day. If some DB API requires guessing some transformation from seconds-from-the-epoch to time-of-day, that's fine, but it doesn't belong in Python's datetime module. The datetime module should certainly have a method to construct a datetime.datetime from a seconds-from-the- epoch argument -- seconds-from-the-epoch is a way of specifying a datetime.datetime. Given that, if the intent is to throw away the datetime.date portion of the datetime.datetime object, retaining only the datetime.time portion, then that's trivially accomplished by invoking dt.time () (where dt is the datetime.datetime object). Do any databases really store time-of-day as a seconds-from- the-epoch value? Google's top hit on TimeFromTicks() today happens to be a msg from Anthony saying that Oracle does not. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 11:24 Message: Logged In: YES user_id=38388 Macro: I don't know whether you need the macro or not (you probably do for the type checks). In any case, you can do without relying on Python providing you this information via the build process, since only datetimemodule.c is using the header file while Python is being built. TimeFromTicks(): I' ve stated my point. I don't think there's anything to add - I'm tired of fighting for these things... one of the reasons I stopped actively discussing things on python-dev. IMHO, a product that's aimed at developers should make things easy for the developer and try to avoid code duplication if possible. The API is needed in order to support database interfaces that use Unix ticks as basis for date/time, so if you don't add it to the module, the ten or twenty module authors out there will have to duplicate that piece of work in their implementation. Transition: This was already discussed on the db-api list. We love freedom of choice. Nothing to add. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 10:48 Message: Logged In: YES user_id=619560 About the Py_BUILD_CORE issue: You are suggesting that I define this in datetimemodule.c just prior to the include of datetime.h? Or are you suggesting some other macro? I am not sure about the comment about "not needing a macro in the first place" since there are two groups that will be importing datetime.h: the first being the datetime module itself and the second being the DB API modules. The macro is definitely required to distinguish between the two -- unless I am missing something? On the TimeFromTicks() issue: You obviously consider it important that the three methods be available at the C level for DB API module authors to use directly. My objection is simply a reiteration of an objection raised by Tim Peters. I think you'll have to argue that one with him. :-) I'll provide my implementations for cx_Oracle as part of the patches and then you can fight it out and I'll stay out of it. :-) As for the transition period, I assumed that with the introduction of a standard datetime module, that it would become the suggested implementation for database interfaace modules. That said, I don't really have any authority to say anything on this front so I'll leave that to you and others to decide. :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 09:26 Message: Logged In: YES user_id=38388 About the Py_BUILD_CORE issue: Sorry, I didn't know that Python does not define this macro for core modules. Nevertheless, looking at the datetime module code I don't really understand why you would need such a macro in the first place. Since the datetime module is the only code including the datetime.h header file it should be easy to add a macro definition just before importing datetime.h. On the TimeFromTicks() issue: The whole point of this discussion is to make the datetime module easily usable for authors of database modules. Since the DB API specifies that the module will need to export the three functions, two of which are available through the datetime module already, I don't really see your point in not wanting to add it. BTW, you are wrong in the assumption that the DB API spec will move away from a generic API for date/time objects. The DB API has always been defined in terms of interfaces and does not force a certain set of tools onto the developer. The datetime module will become one possible choice for DB API authors, others will want to stick to mxDateTime, use strings, ticks integers or more specific objects for interfacing. That won't change, so there is no "transition period". ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 08:40 Message: Logged In: YES user_id=619560 I spent some time getting the two include files merged together, only to discover that Py_BUILD_CORE is __NOT__ defined when building modules, only when building stuff linked directly into the core interpreter. Perhaps this ought to be defined when building the modules with setup.py? At any rate, I cannot proceed along the route suggested without this problem being resolved. On the DateFromTicks(), TimestampFromTicks(), TimeFromTicks() issue, I'm not sure that there is much benefit to including a C API for this since if the datetime module becomes the standard method for managing datetime objects, there is no point in having the DB API modules provide something that the datetime module already provides, right? During the transition period the few lines of code needed can be posted. If you disagree I can certainly provide the few lines as part of the patch but I suspect you will find resistance to having this included by Tim Peters, the maintainer (I believe) of the datetime module. Feel free to correct me... :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-04-26 10:19 Message: Logged In: YES user_id=38388 Sorry for not getting back to you earlier: I didn't see you last message. 1-4) Python defines a macro during the Python build process that is not defined when compiling extensions: Py_BUILD_CORE You can use that macro to compile things differently for core things vs. extensions. 5) Fair enough; I don't see a point in creating a new numbering for each and every module. Please just use 1, 2, 3, 4, ... 6) Hope this makes sense :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-26 09:35 Message: Logged In: YES user_id=619560 Any chance to look at this? Any hope of this being included in Python 2.4? I know a number of people (including Guido) who would appreciate that.... :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-05 16:04 Message: Logged In: YES user_id=619560 I am back at work after finishing my move so I think I can finally devote a little more time to this issue. 1-4) The main reason for datetimeapi.h as opposed to simply modifying datetime.h was that the internal build needs to define things __differently__ from an external build of a module and there doesn't appear to be any way of fixing that -- so if you know a way I think I can manage to create a reasonable patch; otherwise, I'm out of my depth! 5) Guido was suggesting that a new PyIMPORT macro be made available which would verify that the magic number defined in the header file actually matches the magic number exported by the built module; this is simply a developer protection scheme which should eliminate really stupid uses; you can ask him about this or I can e-mail you part of the thread that brought this up 6) Makes sense. I'd like to get the questions above resolved first before proceeding any further, though. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 11:23 Message: Logged In: YES user_id=38388 Thanks for the comments... 1) Please put *all* the code for the C API into datetimeapi.h. I don't like your current mix of putting some things into datetime.h and others into datetimeapi.h (e.g. the PyDateTime_CAPI definition). 2) Since it's the only API missing if you want to use datetime objects in database interfacing, please add it. 3) dito. 4) Yes, the header file is the right place. Have a look at unicodeobject.h for what I have in mind (or mxDateTime.h for that matter). 5) Why add yet another magic number ? Isn't the Python API number enough ? 6) Since you don't provide a doc patch, please put comments into the header file. There can never be enough documentation and developers tend to look at the code rather than the docs ;-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-03-25 11:08 Message: Logged In: YES user_id=619560 Sorry for the delay. Things here have been quite busy recently both at work and at home. Let me see if I can answer your questions. 1) are you suggesting comments that include the macro names in datetimeapi.h similar to what I put in the comments below? Or something else? See modified datetimeapi.h for more info. 2) TimeFromTicks() is a very DB API specific routine. In chatting with others about the datetime module, they were not keen on adding such a beast. I guess there will have to be some discussion about whether datetimeapi.h is also a sort of half implementation for a C module implementing the DB API. I was intending to do this in my cx_Oracle module but not in datetimeapi.h. Could you clarify? 3) I could add C APIS for the three DB API ticks interfaces but I was simply intending to do that in cx_Oracle. Again, see #2. I can do it either way. :-) 4) I have added limited documentation in datetimeapi.h but is that really the right place for it? Is it even remotely close to what is needed? I'm not sure what is needed here exactly. 5) the API magic number is simply an arbitrary number which is stored in datetime.h and need not be understood by the implementor. For your information I simply took the name "datetime", converted each letter to its ordinal value in the alphabet, modded by 16 and used that hex character -- does that make any sense? This can be changed to whatever makes sense, though. 6) Whether or not datetimeapi.h should be sufficient for a developer or whether he should look at documentation in the usual locations (html pages, for example) I'll leave up to the Python development team. Let me know how I can assist. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 10:35 Message: Logged In: YES user_id=38388 Anthony, have you had a chance to look at the comments ? ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 08:06 Message: Logged In: YES user_id=619560 Oops! It looks like my own misunderstanding of SourceForge and how it works is to blame. :-( I'll take a look at this and get back to you as soon as possible -- ignore my previous comment. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 08:03 Message: Logged In: YES user_id=619560 You haven't yet responded to my previous comments -- it looks like SourceForge put them __after__ your comments so it is quite understandable why you didn't notice them. If you can give me the answers to those questions or provide additional questions that I need to answer, then I think we can make progress again. I was waiting for __you__ :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-02-25 16:08 Message: Logged In: YES user_id=38388 Any progress on this ? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-28 04:49 Message: Logged In: YES user_id=38388 Thanks for the clarification. I had forgotten about the macros -- I'd suggest that you document them in the datetimeapi.h include file to not cause the same confusion on the developer side (the trick to achieve adoption is to make things easy on the developer). As for TimeFromTicks(): This should be rather straight forward to implement: you take the time part of the ticks timestamp and create a time object from it. This is the way the DB API constructor works. Could you add C APIs for the three DB API ticks interface methods ? Other things that are missing: * documentation of the way the CAPI object can be used in datetimeapi.h * documentation of the various CAPI entries * documentation of the API magic number (there should be some scheme for this) I think that datetimeapi.h should be enough for a developer to read in order to use the interface. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 14:02 Message: Logged In: YES user_id=619560 I didn't think any additional API would be necessary for extracting date time information since the following macros are available: PyDateTime_GET_YEAR(o) PyDateTime_GET_MONTH(o) PyDateTime_GET_DAY(o) PyDateTime_DATE_GET_HOUR(o) PyDateTime_DATE_GET_MINUTE(o) PyDateTime_DATE_GET_SECOND(o) PyDateTime_DATE_GET_MICROSECOND(o) PyDateTime_TIME_GET_HOUR(o) PyDateTime_TIME_GET_MINUTE(o) PyDateTime_TIME_GET_SECOND(o) PyDateTime_TIME_GET_MICROSECOND(o) Were you thinking of something else that is needed? As for interfacing at the C level for converting from Unix ticks, the following method is already available and could simply be used as a drop in replacement for DateFromTicks() and TimestampFromTicks() from the DB API document. DateFromTicks() --> datetime.date.fromtimestamp() TimestampFromTicks() --> datetime.datetime.fromtimestamp() TimeFromTicks() is not already exposed but discussions with Tim Peters already suggested that would not happen since the concept is rather strange. You'll have to discuss that with him if you disagree! :-) Any comments? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-26 13:46 Message: Logged In: YES user_id=38388 This is a good start. However, you should add more APIs for extracting date/time information to the C API. mxDateTime.h from egenix-mx-base can provide a good basis for this. If you want to make the datetime module usable for database interfacing at C level, you'll also have to add interfaces for Unix ticks conversion, since these are often used by database C level interfaces. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 11:36 Message: Logged In: YES user_id=619560 I have no objection to providing patches for doc and test. A quick look at _testcapimodule.c doesn't provide any obvious ideas as to how to patch it. I looked for cStringIO which has a C API already and found nothing in there at all. The documentation also simply says "see the source for the information you require". Any suggestions as to how to proceed? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-01-25 20:17 Message: Logged In: YES user_id=31435 Marc-Andre, can you review this? I think you have more experience building C APIs for modules than anyone else (while I don't have any). There's certainly no objection to giving datetime a C API, and Guido already said he doesn't want to rely on that Martin changed datetime to be built as part of the core (in 2.3 on Windows, datetime.pyd was a distinct DLL; in CVS, datetime is compiled into the core DLL now). Anthony, you probably need to add doc and test patches. _testcapimodule.c holds tests of things you can only get at from C. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 From noreply at sourceforge.net Wed May 26 12:38:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 12:38:27 2004 Subject: [Patches] [ python-Patches-949332 ] synchronous signals blocked in pthreads Message-ID: Patches item #949332, was opened at 2004-05-06 13:07 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: synchronous signals blocked in pthreads Initial Comment: In a pthreads environment, signals sent asynchronously from things like kill() or the terminal driver are sent to the process and can be handled by any thread. Signals that are sent synchronously from the instruction stream (signals like SIGSEGV, SIGFPE, or SIGILL) are only delivered to the thread that executed the instruction. The current Python threads, when implemented via pthreads, blocks these synchronous signals and so thread that generates one can hang the entire process. The attached patch will remove some signals the list of signals to be blocked. The list of signals to be removed correspond to signals that can be generated synchronously. ---------------------------------------------------------------------- >Comment By: Andrew Langmead (langmead) Date: 2004-05-26 12:38 Message: Logged In: YES user_id=119306 The changes in are probably more appropriate than this patch. ---------------------------------------------------------------------- Comment By: Dieter Maurer (dmaurer) Date: 2004-05-26 08:00 Message: Logged In: YES user_id=265829 I verified that the return code is appropriate. ---------------------------------------------------------------------- Comment By: Dieter Maurer (dmaurer) Date: 2004-05-26 06:46 Message: Logged In: YES user_id=265829 This patch addresses a serious problem in Zope2 (and probably other multi-threaded) production environments with LinuxThreads. A crash of one thread brings other threads in an unsane state which they leave only on "kill -9". This lets Guido's "zdeamon" automatic restart logic fail. Even manual restarts are a bit difficult as each individual thread must be sent a "kill -9". Applying the patch lets at least the process die properly. I am not yet sure whether the return code is appropriate for "zdeamon" to restart the process. I will check that soon and report back. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 09:47 Message: Logged In: YES user_id=6380 As Tim mentioned, this is a possible stopgap fix for bug 949332. While many objections have been raised, I still think this is a safer approach than some of the more advanced suggestions. (Too bad it doesn't solve my own, related problem, which is that SIGTERM is ignored by default in a process started from a Python thread.) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 14:11 Message: Logged In: YES user_id=31435 Boosted the priority in the hopes this will get looked at before 2.3.4. Unfortunately, I'm not in a position to test a pthreads patch. I'll note that the good comments about *why* this change is being entertained would be far more useful in the code than in the patch comment here. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 From noreply at sourceforge.net Wed May 26 13:22:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 13:22:33 2004 Subject: [Patches] [ python-Patches-960406 ] unblock signals in threads Message-ID: Patches item #960406, was opened at 2004-05-25 22:00 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: unblock signals in threads Initial Comment: This is a patch which will correct the issues some people have with python's handling of signal handling in threads. It allows any thread to initially catch the signal mark it as triggered, allowing the main thread to later process it. (This is actually just restoring access to the functionality that was in Python 2.1) The special SIGINT handling for the python readline module has been changed so that it can now see an EINTR error code, rather than needing a longjmp out of the readline library itself. If the readline library python is being linked to doesn't have the callback features necessary, it will fall back to its old behavior. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 18:22 Message: Logged In: YES user_id=6656 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. In addition, even if you *did* check this checkbox, a bug in SourceForge prevents attaching a file when *creating* an issue. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 From noreply at sourceforge.net Wed May 26 13:48:26 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 13:48:32 2004 Subject: [Patches] [ python-Patches-960406 ] unblock signals in threads Message-ID: Patches item #960406, was opened at 2004-05-25 17:00 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: unblock signals in threads Initial Comment: This is a patch which will correct the issues some people have with python's handling of signal handling in threads. It allows any thread to initially catch the signal mark it as triggered, allowing the main thread to later process it. (This is actually just restoring access to the functionality that was in Python 2.1) The special SIGINT handling for the python readline module has been changed so that it can now see an EINTR error code, rather than needing a longjmp out of the readline library itself. If the readline library python is being linked to doesn't have the callback features necessary, it will fall back to its old behavior. ---------------------------------------------------------------------- >Comment By: Andrew Langmead (langmead) Date: 2004-05-26 13:48 Message: Logged In: YES user_id=119306 I apologize that the missing patch. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 13:22 Message: Logged In: YES user_id=6656 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. In addition, even if you *did* check this checkbox, a bug in SourceForge prevents attaching a file when *creating* an issue. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 From noreply at sourceforge.net Wed May 26 14:41:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 14:41:14 2004 Subject: [Patches] [ python-Patches-960406 ] unblock signals in threads Message-ID: Patches item #960406, was opened at 2004-05-25 22:00 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: unblock signals in threads Initial Comment: This is a patch which will correct the issues some people have with python's handling of signal handling in threads. It allows any thread to initially catch the signal mark it as triggered, allowing the main thread to later process it. (This is actually just restoring access to the functionality that was in Python 2.1) The special SIGINT handling for the python readline module has been changed so that it can now see an EINTR error code, rather than needing a longjmp out of the readline library itself. If the readline library python is being linked to doesn't have the callback features necessary, it will fall back to its old behavior. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-26 19:41 Message: Logged In: YES user_id=6656 test_threadsignals hangs for me on OS X. Haven't done anything more thorough than that yet... ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-26 18:48 Message: Logged In: YES user_id=119306 I apologize that the missing patch. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 18:22 Message: Logged In: YES user_id=6656 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. In addition, even if you *did* check this checkbox, a bug in SourceForge prevents attaching a file when *creating* an issue. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 From noreply at sourceforge.net Wed May 26 15:22:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 15:22:39 2004 Subject: [Patches] [ python-Patches-876130 ] add C API to datetime module Message-ID: Patches item #876130, was opened at 2004-01-13 16:20 Message generated for change (Comment added) made by lemburg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 Category: Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Anthony Tuininga (atuining) Assigned to: M.-A. Lemburg (lemburg) Summary: add C API to datetime module Initial Comment: The datetime module does not have a C API which means that modules written in C cannot take advantage of the datetime module without incurring significant overhead. These patches supply this lack and are based on the C API exported by the mxDateTime module and the cStringIO module and enhanced as suggested by Guido. ---------------------------------------------------------------------- >Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 21:22 Message: Logged In: YES user_id=38388 Thanks, but I don't have time to review it. As for the TimeFromTicks() API: I can understand that you don't feel like implementing it, but hey, I won't use datetime either. The only reason I'm trying to push full support of the DB API specs is that Guido et al. wanted to see a note in the DB API that datetime also provides a usable interface to do date/time interfacing. I'm not very interested in datetime myself, since I'll continue to use mxDateTime, so the situation for me is somewhat similar to yours: investing time into something that I myself will probably not use much (I will add support to mxODBC for those who like to use datetime instead of mxDateTime, but won't use it myself). Feel free to check in your changes. Thanks. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-26 18:30 Message: Logged In: YES user_id=619560 Attached are the modified patches based on your suggestions. I have tested them with my own module cx_Oracle and they appear to work correctly. Any further suggestions would be appreciated. I would like to see this included for Python 2.4 if at all possible. Please let me know what I need to do (if anything) to get this to happen. As for the TimeFromTicks() routine, Oracle has absolutely no use for a "time-only" data type and does not support it. It clearly does not have broad support so I really have no desire to implement it since I will never use it myself. If you consider it important, you could add it yourself or we could wait until someone else who requires it for their implementation of the DB API requires it. I'm not trying to be rude or trying to denigrate the DB API, simply trying to be practical. Please let me know if this is offensive in any way. Thanks. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-19 10:28 Message: Logged In: YES user_id=38388 >From the DB API specs: TimeFromTicks(ticks) This function constructs an object holding a time value from the given ticks value (number of seconds since the epoch; see the documentation of the standard Python time module for details). The reason for this having this API is to be able to construct an object suitable for passing to the database given a Unix ticks value. Since the ticks value contains both a date and a time part and most databases have a special column type for time value, we need two constructor APIs, one to extract just the date part and one for the time part. Here's the mxDateTime implementation for reference: def TimeFromTicks(ticks, # Locals: DateTimeDelta=DateTimeDelta,localtime=_time.localtime): """ TimeFromTicks(ticks) Constructs a DateTimeDelta instance pointing to the local time indicated by the given ticks value. The date part is ignored. """ return apply(DateTimeDelta, (0,) + localtime(ticks)[3:6]) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-03 19:46 Message: Logged In: YES user_id=31435 I don't understand what TimeFromTicks() is supposed to do. Seconds-from-the-epoch is a point in time, not a time-of- day. If some DB API requires guessing some transformation from seconds-from-the-epoch to time-of-day, that's fine, but it doesn't belong in Python's datetime module. The datetime module should certainly have a method to construct a datetime.datetime from a seconds-from-the- epoch argument -- seconds-from-the-epoch is a way of specifying a datetime.datetime. Given that, if the intent is to throw away the datetime.date portion of the datetime.datetime object, retaining only the datetime.time portion, then that's trivially accomplished by invoking dt.time () (where dt is the datetime.datetime object). Do any databases really store time-of-day as a seconds-from- the-epoch value? Google's top hit on TimeFromTicks() today happens to be a msg from Anthony saying that Oracle does not. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 19:24 Message: Logged In: YES user_id=38388 Macro: I don't know whether you need the macro or not (you probably do for the type checks). In any case, you can do without relying on Python providing you this information via the build process, since only datetimemodule.c is using the header file while Python is being built. TimeFromTicks(): I' ve stated my point. I don't think there's anything to add - I'm tired of fighting for these things... one of the reasons I stopped actively discussing things on python-dev. IMHO, a product that's aimed at developers should make things easy for the developer and try to avoid code duplication if possible. The API is needed in order to support database interfaces that use Unix ticks as basis for date/time, so if you don't add it to the module, the ten or twenty module authors out there will have to duplicate that piece of work in their implementation. Transition: This was already discussed on the db-api list. We love freedom of choice. Nothing to add. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 18:48 Message: Logged In: YES user_id=619560 About the Py_BUILD_CORE issue: You are suggesting that I define this in datetimemodule.c just prior to the include of datetime.h? Or are you suggesting some other macro? I am not sure about the comment about "not needing a macro in the first place" since there are two groups that will be importing datetime.h: the first being the datetime module itself and the second being the DB API modules. The macro is definitely required to distinguish between the two -- unless I am missing something? On the TimeFromTicks() issue: You obviously consider it important that the three methods be available at the C level for DB API module authors to use directly. My objection is simply a reiteration of an objection raised by Tim Peters. I think you'll have to argue that one with him. :-) I'll provide my implementations for cx_Oracle as part of the patches and then you can fight it out and I'll stay out of it. :-) As for the transition period, I assumed that with the introduction of a standard datetime module, that it would become the suggested implementation for database interfaace modules. That said, I don't really have any authority to say anything on this front so I'll leave that to you and others to decide. :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 17:26 Message: Logged In: YES user_id=38388 About the Py_BUILD_CORE issue: Sorry, I didn't know that Python does not define this macro for core modules. Nevertheless, looking at the datetime module code I don't really understand why you would need such a macro in the first place. Since the datetime module is the only code including the datetime.h header file it should be easy to add a macro definition just before importing datetime.h. On the TimeFromTicks() issue: The whole point of this discussion is to make the datetime module easily usable for authors of database modules. Since the DB API specifies that the module will need to export the three functions, two of which are available through the datetime module already, I don't really see your point in not wanting to add it. BTW, you are wrong in the assumption that the DB API spec will move away from a generic API for date/time objects. The DB API has always been defined in terms of interfaces and does not force a certain set of tools onto the developer. The datetime module will become one possible choice for DB API authors, others will want to stick to mxDateTime, use strings, ticks integers or more specific objects for interfacing. That won't change, so there is no "transition period". ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 16:40 Message: Logged In: YES user_id=619560 I spent some time getting the two include files merged together, only to discover that Py_BUILD_CORE is __NOT__ defined when building modules, only when building stuff linked directly into the core interpreter. Perhaps this ought to be defined when building the modules with setup.py? At any rate, I cannot proceed along the route suggested without this problem being resolved. On the DateFromTicks(), TimestampFromTicks(), TimeFromTicks() issue, I'm not sure that there is much benefit to including a C API for this since if the datetime module becomes the standard method for managing datetime objects, there is no point in having the DB API modules provide something that the datetime module already provides, right? During the transition period the few lines of code needed can be posted. If you disagree I can certainly provide the few lines as part of the patch but I suspect you will find resistance to having this included by Tim Peters, the maintainer (I believe) of the datetime module. Feel free to correct me... :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-04-26 18:19 Message: Logged In: YES user_id=38388 Sorry for not getting back to you earlier: I didn't see you last message. 1-4) Python defines a macro during the Python build process that is not defined when compiling extensions: Py_BUILD_CORE You can use that macro to compile things differently for core things vs. extensions. 5) Fair enough; I don't see a point in creating a new numbering for each and every module. Please just use 1, 2, 3, 4, ... 6) Hope this makes sense :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-26 17:35 Message: Logged In: YES user_id=619560 Any chance to look at this? Any hope of this being included in Python 2.4? I know a number of people (including Guido) who would appreciate that.... :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-06 00:04 Message: Logged In: YES user_id=619560 I am back at work after finishing my move so I think I can finally devote a little more time to this issue. 1-4) The main reason for datetimeapi.h as opposed to simply modifying datetime.h was that the internal build needs to define things __differently__ from an external build of a module and there doesn't appear to be any way of fixing that -- so if you know a way I think I can manage to create a reasonable patch; otherwise, I'm out of my depth! 5) Guido was suggesting that a new PyIMPORT macro be made available which would verify that the magic number defined in the header file actually matches the magic number exported by the built module; this is simply a developer protection scheme which should eliminate really stupid uses; you can ask him about this or I can e-mail you part of the thread that brought this up 6) Makes sense. I'd like to get the questions above resolved first before proceeding any further, though. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 19:23 Message: Logged In: YES user_id=38388 Thanks for the comments... 1) Please put *all* the code for the C API into datetimeapi.h. I don't like your current mix of putting some things into datetime.h and others into datetimeapi.h (e.g. the PyDateTime_CAPI definition). 2) Since it's the only API missing if you want to use datetime objects in database interfacing, please add it. 3) dito. 4) Yes, the header file is the right place. Have a look at unicodeobject.h for what I have in mind (or mxDateTime.h for that matter). 5) Why add yet another magic number ? Isn't the Python API number enough ? 6) Since you don't provide a doc patch, please put comments into the header file. There can never be enough documentation and developers tend to look at the code rather than the docs ;-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-03-25 19:08 Message: Logged In: YES user_id=619560 Sorry for the delay. Things here have been quite busy recently both at work and at home. Let me see if I can answer your questions. 1) are you suggesting comments that include the macro names in datetimeapi.h similar to what I put in the comments below? Or something else? See modified datetimeapi.h for more info. 2) TimeFromTicks() is a very DB API specific routine. In chatting with others about the datetime module, they were not keen on adding such a beast. I guess there will have to be some discussion about whether datetimeapi.h is also a sort of half implementation for a C module implementing the DB API. I was intending to do this in my cx_Oracle module but not in datetimeapi.h. Could you clarify? 3) I could add C APIS for the three DB API ticks interfaces but I was simply intending to do that in cx_Oracle. Again, see #2. I can do it either way. :-) 4) I have added limited documentation in datetimeapi.h but is that really the right place for it? Is it even remotely close to what is needed? I'm not sure what is needed here exactly. 5) the API magic number is simply an arbitrary number which is stored in datetime.h and need not be understood by the implementor. For your information I simply took the name "datetime", converted each letter to its ordinal value in the alphabet, modded by 16 and used that hex character -- does that make any sense? This can be changed to whatever makes sense, though. 6) Whether or not datetimeapi.h should be sufficient for a developer or whether he should look at documentation in the usual locations (html pages, for example) I'll leave up to the Python development team. Let me know how I can assist. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 18:35 Message: Logged In: YES user_id=38388 Anthony, have you had a chance to look at the comments ? ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 16:06 Message: Logged In: YES user_id=619560 Oops! It looks like my own misunderstanding of SourceForge and how it works is to blame. :-( I'll take a look at this and get back to you as soon as possible -- ignore my previous comment. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 16:03 Message: Logged In: YES user_id=619560 You haven't yet responded to my previous comments -- it looks like SourceForge put them __after__ your comments so it is quite understandable why you didn't notice them. If you can give me the answers to those questions or provide additional questions that I need to answer, then I think we can make progress again. I was waiting for __you__ :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-02-26 00:08 Message: Logged In: YES user_id=38388 Any progress on this ? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-28 12:49 Message: Logged In: YES user_id=38388 Thanks for the clarification. I had forgotten about the macros -- I'd suggest that you document them in the datetimeapi.h include file to not cause the same confusion on the developer side (the trick to achieve adoption is to make things easy on the developer). As for TimeFromTicks(): This should be rather straight forward to implement: you take the time part of the ticks timestamp and create a time object from it. This is the way the DB API constructor works. Could you add C APIs for the three DB API ticks interface methods ? Other things that are missing: * documentation of the way the CAPI object can be used in datetimeapi.h * documentation of the various CAPI entries * documentation of the API magic number (there should be some scheme for this) I think that datetimeapi.h should be enough for a developer to read in order to use the interface. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 22:02 Message: Logged In: YES user_id=619560 I didn't think any additional API would be necessary for extracting date time information since the following macros are available: PyDateTime_GET_YEAR(o) PyDateTime_GET_MONTH(o) PyDateTime_GET_DAY(o) PyDateTime_DATE_GET_HOUR(o) PyDateTime_DATE_GET_MINUTE(o) PyDateTime_DATE_GET_SECOND(o) PyDateTime_DATE_GET_MICROSECOND(o) PyDateTime_TIME_GET_HOUR(o) PyDateTime_TIME_GET_MINUTE(o) PyDateTime_TIME_GET_SECOND(o) PyDateTime_TIME_GET_MICROSECOND(o) Were you thinking of something else that is needed? As for interfacing at the C level for converting from Unix ticks, the following method is already available and could simply be used as a drop in replacement for DateFromTicks() and TimestampFromTicks() from the DB API document. DateFromTicks() --> datetime.date.fromtimestamp() TimestampFromTicks() --> datetime.datetime.fromtimestamp() TimeFromTicks() is not already exposed but discussions with Tim Peters already suggested that would not happen since the concept is rather strange. You'll have to discuss that with him if you disagree! :-) Any comments? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-26 21:46 Message: Logged In: YES user_id=38388 This is a good start. However, you should add more APIs for extracting date/time information to the C API. mxDateTime.h from egenix-mx-base can provide a good basis for this. If you want to make the datetime module usable for database interfacing at C level, you'll also have to add interfaces for Unix ticks conversion, since these are often used by database C level interfaces. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 19:36 Message: Logged In: YES user_id=619560 I have no objection to providing patches for doc and test. A quick look at _testcapimodule.c doesn't provide any obvious ideas as to how to patch it. I looked for cStringIO which has a C API already and found nothing in there at all. The documentation also simply says "see the source for the information you require". Any suggestions as to how to proceed? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-01-26 04:17 Message: Logged In: YES user_id=31435 Marc-Andre, can you review this? I think you have more experience building C APIs for modules than anyone else (while I don't have any). There's certainly no objection to giving datetime a C API, and Guido already said he doesn't want to rely on that Martin changed datetime to be built as part of the core (in 2.3 on Windows, datetime.pyd was a distinct DLL; in CVS, datetime is compiled into the core DLL now). Anthony, you probably need to add doc and test patches. _testcapimodule.c holds tests of things you can only get at from C. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 From noreply at sourceforge.net Wed May 26 15:25:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 15:25:34 2004 Subject: [Patches] [ python-Patches-876130 ] add C API to datetime module Message-ID: Patches item #876130, was opened at 2004-01-13 16:20 Message generated for change (Comment added) made by lemburg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 Category: Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Anthony Tuininga (atuining) >Assigned to: Tim Peters (tim_one) Summary: add C API to datetime module Initial Comment: The datetime module does not have a C API which means that modules written in C cannot take advantage of the datetime module without incurring significant overhead. These patches supply this lack and are based on the C API exported by the mxDateTime module and the cStringIO module and enhanced as suggested by Guido. ---------------------------------------------------------------------- >Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 21:25 Message: Logged In: YES user_id=38388 Tim, please review. Thanks. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 21:22 Message: Logged In: YES user_id=38388 Thanks, but I don't have time to review it. As for the TimeFromTicks() API: I can understand that you don't feel like implementing it, but hey, I won't use datetime either. The only reason I'm trying to push full support of the DB API specs is that Guido et al. wanted to see a note in the DB API that datetime also provides a usable interface to do date/time interfacing. I'm not very interested in datetime myself, since I'll continue to use mxDateTime, so the situation for me is somewhat similar to yours: investing time into something that I myself will probably not use much (I will add support to mxODBC for those who like to use datetime instead of mxDateTime, but won't use it myself). Feel free to check in your changes. Thanks. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-26 18:30 Message: Logged In: YES user_id=619560 Attached are the modified patches based on your suggestions. I have tested them with my own module cx_Oracle and they appear to work correctly. Any further suggestions would be appreciated. I would like to see this included for Python 2.4 if at all possible. Please let me know what I need to do (if anything) to get this to happen. As for the TimeFromTicks() routine, Oracle has absolutely no use for a "time-only" data type and does not support it. It clearly does not have broad support so I really have no desire to implement it since I will never use it myself. If you consider it important, you could add it yourself or we could wait until someone else who requires it for their implementation of the DB API requires it. I'm not trying to be rude or trying to denigrate the DB API, simply trying to be practical. Please let me know if this is offensive in any way. Thanks. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-19 10:28 Message: Logged In: YES user_id=38388 >From the DB API specs: TimeFromTicks(ticks) This function constructs an object holding a time value from the given ticks value (number of seconds since the epoch; see the documentation of the standard Python time module for details). The reason for this having this API is to be able to construct an object suitable for passing to the database given a Unix ticks value. Since the ticks value contains both a date and a time part and most databases have a special column type for time value, we need two constructor APIs, one to extract just the date part and one for the time part. Here's the mxDateTime implementation for reference: def TimeFromTicks(ticks, # Locals: DateTimeDelta=DateTimeDelta,localtime=_time.localtime): """ TimeFromTicks(ticks) Constructs a DateTimeDelta instance pointing to the local time indicated by the given ticks value. The date part is ignored. """ return apply(DateTimeDelta, (0,) + localtime(ticks)[3:6]) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-03 19:46 Message: Logged In: YES user_id=31435 I don't understand what TimeFromTicks() is supposed to do. Seconds-from-the-epoch is a point in time, not a time-of- day. If some DB API requires guessing some transformation from seconds-from-the-epoch to time-of-day, that's fine, but it doesn't belong in Python's datetime module. The datetime module should certainly have a method to construct a datetime.datetime from a seconds-from-the- epoch argument -- seconds-from-the-epoch is a way of specifying a datetime.datetime. Given that, if the intent is to throw away the datetime.date portion of the datetime.datetime object, retaining only the datetime.time portion, then that's trivially accomplished by invoking dt.time () (where dt is the datetime.datetime object). Do any databases really store time-of-day as a seconds-from- the-epoch value? Google's top hit on TimeFromTicks() today happens to be a msg from Anthony saying that Oracle does not. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 19:24 Message: Logged In: YES user_id=38388 Macro: I don't know whether you need the macro or not (you probably do for the type checks). In any case, you can do without relying on Python providing you this information via the build process, since only datetimemodule.c is using the header file while Python is being built. TimeFromTicks(): I' ve stated my point. I don't think there's anything to add - I'm tired of fighting for these things... one of the reasons I stopped actively discussing things on python-dev. IMHO, a product that's aimed at developers should make things easy for the developer and try to avoid code duplication if possible. The API is needed in order to support database interfaces that use Unix ticks as basis for date/time, so if you don't add it to the module, the ten or twenty module authors out there will have to duplicate that piece of work in their implementation. Transition: This was already discussed on the db-api list. We love freedom of choice. Nothing to add. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 18:48 Message: Logged In: YES user_id=619560 About the Py_BUILD_CORE issue: You are suggesting that I define this in datetimemodule.c just prior to the include of datetime.h? Or are you suggesting some other macro? I am not sure about the comment about "not needing a macro in the first place" since there are two groups that will be importing datetime.h: the first being the datetime module itself and the second being the DB API modules. The macro is definitely required to distinguish between the two -- unless I am missing something? On the TimeFromTicks() issue: You obviously consider it important that the three methods be available at the C level for DB API module authors to use directly. My objection is simply a reiteration of an objection raised by Tim Peters. I think you'll have to argue that one with him. :-) I'll provide my implementations for cx_Oracle as part of the patches and then you can fight it out and I'll stay out of it. :-) As for the transition period, I assumed that with the introduction of a standard datetime module, that it would become the suggested implementation for database interfaace modules. That said, I don't really have any authority to say anything on this front so I'll leave that to you and others to decide. :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 17:26 Message: Logged In: YES user_id=38388 About the Py_BUILD_CORE issue: Sorry, I didn't know that Python does not define this macro for core modules. Nevertheless, looking at the datetime module code I don't really understand why you would need such a macro in the first place. Since the datetime module is the only code including the datetime.h header file it should be easy to add a macro definition just before importing datetime.h. On the TimeFromTicks() issue: The whole point of this discussion is to make the datetime module easily usable for authors of database modules. Since the DB API specifies that the module will need to export the three functions, two of which are available through the datetime module already, I don't really see your point in not wanting to add it. BTW, you are wrong in the assumption that the DB API spec will move away from a generic API for date/time objects. The DB API has always been defined in terms of interfaces and does not force a certain set of tools onto the developer. The datetime module will become one possible choice for DB API authors, others will want to stick to mxDateTime, use strings, ticks integers or more specific objects for interfacing. That won't change, so there is no "transition period". ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 16:40 Message: Logged In: YES user_id=619560 I spent some time getting the two include files merged together, only to discover that Py_BUILD_CORE is __NOT__ defined when building modules, only when building stuff linked directly into the core interpreter. Perhaps this ought to be defined when building the modules with setup.py? At any rate, I cannot proceed along the route suggested without this problem being resolved. On the DateFromTicks(), TimestampFromTicks(), TimeFromTicks() issue, I'm not sure that there is much benefit to including a C API for this since if the datetime module becomes the standard method for managing datetime objects, there is no point in having the DB API modules provide something that the datetime module already provides, right? During the transition period the few lines of code needed can be posted. If you disagree I can certainly provide the few lines as part of the patch but I suspect you will find resistance to having this included by Tim Peters, the maintainer (I believe) of the datetime module. Feel free to correct me... :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-04-26 18:19 Message: Logged In: YES user_id=38388 Sorry for not getting back to you earlier: I didn't see you last message. 1-4) Python defines a macro during the Python build process that is not defined when compiling extensions: Py_BUILD_CORE You can use that macro to compile things differently for core things vs. extensions. 5) Fair enough; I don't see a point in creating a new numbering for each and every module. Please just use 1, 2, 3, 4, ... 6) Hope this makes sense :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-26 17:35 Message: Logged In: YES user_id=619560 Any chance to look at this? Any hope of this being included in Python 2.4? I know a number of people (including Guido) who would appreciate that.... :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-06 00:04 Message: Logged In: YES user_id=619560 I am back at work after finishing my move so I think I can finally devote a little more time to this issue. 1-4) The main reason for datetimeapi.h as opposed to simply modifying datetime.h was that the internal build needs to define things __differently__ from an external build of a module and there doesn't appear to be any way of fixing that -- so if you know a way I think I can manage to create a reasonable patch; otherwise, I'm out of my depth! 5) Guido was suggesting that a new PyIMPORT macro be made available which would verify that the magic number defined in the header file actually matches the magic number exported by the built module; this is simply a developer protection scheme which should eliminate really stupid uses; you can ask him about this or I can e-mail you part of the thread that brought this up 6) Makes sense. I'd like to get the questions above resolved first before proceeding any further, though. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 19:23 Message: Logged In: YES user_id=38388 Thanks for the comments... 1) Please put *all* the code for the C API into datetimeapi.h. I don't like your current mix of putting some things into datetime.h and others into datetimeapi.h (e.g. the PyDateTime_CAPI definition). 2) Since it's the only API missing if you want to use datetime objects in database interfacing, please add it. 3) dito. 4) Yes, the header file is the right place. Have a look at unicodeobject.h for what I have in mind (or mxDateTime.h for that matter). 5) Why add yet another magic number ? Isn't the Python API number enough ? 6) Since you don't provide a doc patch, please put comments into the header file. There can never be enough documentation and developers tend to look at the code rather than the docs ;-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-03-25 19:08 Message: Logged In: YES user_id=619560 Sorry for the delay. Things here have been quite busy recently both at work and at home. Let me see if I can answer your questions. 1) are you suggesting comments that include the macro names in datetimeapi.h similar to what I put in the comments below? Or something else? See modified datetimeapi.h for more info. 2) TimeFromTicks() is a very DB API specific routine. In chatting with others about the datetime module, they were not keen on adding such a beast. I guess there will have to be some discussion about whether datetimeapi.h is also a sort of half implementation for a C module implementing the DB API. I was intending to do this in my cx_Oracle module but not in datetimeapi.h. Could you clarify? 3) I could add C APIS for the three DB API ticks interfaces but I was simply intending to do that in cx_Oracle. Again, see #2. I can do it either way. :-) 4) I have added limited documentation in datetimeapi.h but is that really the right place for it? Is it even remotely close to what is needed? I'm not sure what is needed here exactly. 5) the API magic number is simply an arbitrary number which is stored in datetime.h and need not be understood by the implementor. For your information I simply took the name "datetime", converted each letter to its ordinal value in the alphabet, modded by 16 and used that hex character -- does that make any sense? This can be changed to whatever makes sense, though. 6) Whether or not datetimeapi.h should be sufficient for a developer or whether he should look at documentation in the usual locations (html pages, for example) I'll leave up to the Python development team. Let me know how I can assist. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 18:35 Message: Logged In: YES user_id=38388 Anthony, have you had a chance to look at the comments ? ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 16:06 Message: Logged In: YES user_id=619560 Oops! It looks like my own misunderstanding of SourceForge and how it works is to blame. :-( I'll take a look at this and get back to you as soon as possible -- ignore my previous comment. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 16:03 Message: Logged In: YES user_id=619560 You haven't yet responded to my previous comments -- it looks like SourceForge put them __after__ your comments so it is quite understandable why you didn't notice them. If you can give me the answers to those questions or provide additional questions that I need to answer, then I think we can make progress again. I was waiting for __you__ :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-02-26 00:08 Message: Logged In: YES user_id=38388 Any progress on this ? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-28 12:49 Message: Logged In: YES user_id=38388 Thanks for the clarification. I had forgotten about the macros -- I'd suggest that you document them in the datetimeapi.h include file to not cause the same confusion on the developer side (the trick to achieve adoption is to make things easy on the developer). As for TimeFromTicks(): This should be rather straight forward to implement: you take the time part of the ticks timestamp and create a time object from it. This is the way the DB API constructor works. Could you add C APIs for the three DB API ticks interface methods ? Other things that are missing: * documentation of the way the CAPI object can be used in datetimeapi.h * documentation of the various CAPI entries * documentation of the API magic number (there should be some scheme for this) I think that datetimeapi.h should be enough for a developer to read in order to use the interface. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 22:02 Message: Logged In: YES user_id=619560 I didn't think any additional API would be necessary for extracting date time information since the following macros are available: PyDateTime_GET_YEAR(o) PyDateTime_GET_MONTH(o) PyDateTime_GET_DAY(o) PyDateTime_DATE_GET_HOUR(o) PyDateTime_DATE_GET_MINUTE(o) PyDateTime_DATE_GET_SECOND(o) PyDateTime_DATE_GET_MICROSECOND(o) PyDateTime_TIME_GET_HOUR(o) PyDateTime_TIME_GET_MINUTE(o) PyDateTime_TIME_GET_SECOND(o) PyDateTime_TIME_GET_MICROSECOND(o) Were you thinking of something else that is needed? As for interfacing at the C level for converting from Unix ticks, the following method is already available and could simply be used as a drop in replacement for DateFromTicks() and TimestampFromTicks() from the DB API document. DateFromTicks() --> datetime.date.fromtimestamp() TimestampFromTicks() --> datetime.datetime.fromtimestamp() TimeFromTicks() is not already exposed but discussions with Tim Peters already suggested that would not happen since the concept is rather strange. You'll have to discuss that with him if you disagree! :-) Any comments? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-26 21:46 Message: Logged In: YES user_id=38388 This is a good start. However, you should add more APIs for extracting date/time information to the C API. mxDateTime.h from egenix-mx-base can provide a good basis for this. If you want to make the datetime module usable for database interfacing at C level, you'll also have to add interfaces for Unix ticks conversion, since these are often used by database C level interfaces. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 19:36 Message: Logged In: YES user_id=619560 I have no objection to providing patches for doc and test. A quick look at _testcapimodule.c doesn't provide any obvious ideas as to how to patch it. I looked for cStringIO which has a C API already and found nothing in there at all. The documentation also simply says "see the source for the information you require". Any suggestions as to how to proceed? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-01-26 04:17 Message: Logged In: YES user_id=31435 Marc-Andre, can you review this? I think you have more experience building C APIs for modules than anyone else (while I don't have any). There's certainly no objection to giving datetime a C API, and Guido already said he doesn't want to rely on that Martin changed datetime to be built as part of the core (in 2.3 on Windows, datetime.pyd was a distinct DLL; in CVS, datetime is compiled into the core DLL now). Anthony, you probably need to add doc and test patches. _testcapimodule.c holds tests of things you can only get at from C. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 From noreply at sourceforge.net Wed May 26 15:32:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 15:42:25 2004 Subject: [Patches] [ python-Patches-876130 ] add C API to datetime module Message-ID: Patches item #876130, was opened at 2004-01-13 10:20 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 Category: Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Anthony Tuininga (atuining) >Assigned to: M.-A. Lemburg (lemburg) Summary: add C API to datetime module Initial Comment: The datetime module does not have a C API which means that modules written in C cannot take advantage of the datetime module without incurring significant overhead. These patches supply this lack and are based on the C API exported by the mxDateTime module and the cStringIO module and enhanced as suggested by Guido. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-26 15:32 Message: Logged In: YES user_id=31435 Under the belief that the *intent* of this patch is to add a C API to Python's datetime module, it should stick to exposing C ways to spell what Python programmers can already do from Python using datetime. Since nothing like TimeFromTicks() exists in the Python datetime API, it simply doesn't make sense to invent one available only from C. It *may* make sense to invent one in a DB API wrapper around datetime, but I view that as out of scope for this patch. I would not add TimeFromTicks() to the Python datetime API either, because the docs are ambiguous. The sample implementation Marc-Andre posted here uses localtime() to resolve the ambiguity in the docs, but that has to be a wrong choice for some time-of-day apps. For example, there's apparently no portable way to spell "noon" using TimeFromTicks(): the value you get back depends on which time zone localtime() happens to use at the time it's called. If people want time-of-day from a POSIX timestamp, it remains easy to get it from what datetime already supplies, but to do so you have to be explicit about whether you want UTC or local time interpretation. Both are easily spelled already, and it's a Good Thing that you need to be explicit if you want to do such a thing. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 15:25 Message: Logged In: YES user_id=38388 Tim, please review. Thanks. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 15:22 Message: Logged In: YES user_id=38388 Thanks, but I don't have time to review it. As for the TimeFromTicks() API: I can understand that you don't feel like implementing it, but hey, I won't use datetime either. The only reason I'm trying to push full support of the DB API specs is that Guido et al. wanted to see a note in the DB API that datetime also provides a usable interface to do date/time interfacing. I'm not very interested in datetime myself, since I'll continue to use mxDateTime, so the situation for me is somewhat similar to yours: investing time into something that I myself will probably not use much (I will add support to mxODBC for those who like to use datetime instead of mxDateTime, but won't use it myself). Feel free to check in your changes. Thanks. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-26 12:30 Message: Logged In: YES user_id=619560 Attached are the modified patches based on your suggestions. I have tested them with my own module cx_Oracle and they appear to work correctly. Any further suggestions would be appreciated. I would like to see this included for Python 2.4 if at all possible. Please let me know what I need to do (if anything) to get this to happen. As for the TimeFromTicks() routine, Oracle has absolutely no use for a "time-only" data type and does not support it. It clearly does not have broad support so I really have no desire to implement it since I will never use it myself. If you consider it important, you could add it yourself or we could wait until someone else who requires it for their implementation of the DB API requires it. I'm not trying to be rude or trying to denigrate the DB API, simply trying to be practical. Please let me know if this is offensive in any way. Thanks. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-19 04:28 Message: Logged In: YES user_id=38388 >From the DB API specs: TimeFromTicks(ticks) This function constructs an object holding a time value from the given ticks value (number of seconds since the epoch; see the documentation of the standard Python time module for details). The reason for this having this API is to be able to construct an object suitable for passing to the database given a Unix ticks value. Since the ticks value contains both a date and a time part and most databases have a special column type for time value, we need two constructor APIs, one to extract just the date part and one for the time part. Here's the mxDateTime implementation for reference: def TimeFromTicks(ticks, # Locals: DateTimeDelta=DateTimeDelta,localtime=_time.localtime): """ TimeFromTicks(ticks) Constructs a DateTimeDelta instance pointing to the local time indicated by the given ticks value. The date part is ignored. """ return apply(DateTimeDelta, (0,) + localtime(ticks)[3:6]) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-03 13:46 Message: Logged In: YES user_id=31435 I don't understand what TimeFromTicks() is supposed to do. Seconds-from-the-epoch is a point in time, not a time-of- day. If some DB API requires guessing some transformation from seconds-from-the-epoch to time-of-day, that's fine, but it doesn't belong in Python's datetime module. The datetime module should certainly have a method to construct a datetime.datetime from a seconds-from-the- epoch argument -- seconds-from-the-epoch is a way of specifying a datetime.datetime. Given that, if the intent is to throw away the datetime.date portion of the datetime.datetime object, retaining only the datetime.time portion, then that's trivially accomplished by invoking dt.time () (where dt is the datetime.datetime object). Do any databases really store time-of-day as a seconds-from- the-epoch value? Google's top hit on TimeFromTicks() today happens to be a msg from Anthony saying that Oracle does not. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 13:24 Message: Logged In: YES user_id=38388 Macro: I don't know whether you need the macro or not (you probably do for the type checks). In any case, you can do without relying on Python providing you this information via the build process, since only datetimemodule.c is using the header file while Python is being built. TimeFromTicks(): I' ve stated my point. I don't think there's anything to add - I'm tired of fighting for these things... one of the reasons I stopped actively discussing things on python-dev. IMHO, a product that's aimed at developers should make things easy for the developer and try to avoid code duplication if possible. The API is needed in order to support database interfaces that use Unix ticks as basis for date/time, so if you don't add it to the module, the ten or twenty module authors out there will have to duplicate that piece of work in their implementation. Transition: This was already discussed on the db-api list. We love freedom of choice. Nothing to add. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 12:48 Message: Logged In: YES user_id=619560 About the Py_BUILD_CORE issue: You are suggesting that I define this in datetimemodule.c just prior to the include of datetime.h? Or are you suggesting some other macro? I am not sure about the comment about "not needing a macro in the first place" since there are two groups that will be importing datetime.h: the first being the datetime module itself and the second being the DB API modules. The macro is definitely required to distinguish between the two -- unless I am missing something? On the TimeFromTicks() issue: You obviously consider it important that the three methods be available at the C level for DB API module authors to use directly. My objection is simply a reiteration of an objection raised by Tim Peters. I think you'll have to argue that one with him. :-) I'll provide my implementations for cx_Oracle as part of the patches and then you can fight it out and I'll stay out of it. :-) As for the transition period, I assumed that with the introduction of a standard datetime module, that it would become the suggested implementation for database interfaace modules. That said, I don't really have any authority to say anything on this front so I'll leave that to you and others to decide. :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 11:26 Message: Logged In: YES user_id=38388 About the Py_BUILD_CORE issue: Sorry, I didn't know that Python does not define this macro for core modules. Nevertheless, looking at the datetime module code I don't really understand why you would need such a macro in the first place. Since the datetime module is the only code including the datetime.h header file it should be easy to add a macro definition just before importing datetime.h. On the TimeFromTicks() issue: The whole point of this discussion is to make the datetime module easily usable for authors of database modules. Since the DB API specifies that the module will need to export the three functions, two of which are available through the datetime module already, I don't really see your point in not wanting to add it. BTW, you are wrong in the assumption that the DB API spec will move away from a generic API for date/time objects. The DB API has always been defined in terms of interfaces and does not force a certain set of tools onto the developer. The datetime module will become one possible choice for DB API authors, others will want to stick to mxDateTime, use strings, ticks integers or more specific objects for interfacing. That won't change, so there is no "transition period". ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 10:40 Message: Logged In: YES user_id=619560 I spent some time getting the two include files merged together, only to discover that Py_BUILD_CORE is __NOT__ defined when building modules, only when building stuff linked directly into the core interpreter. Perhaps this ought to be defined when building the modules with setup.py? At any rate, I cannot proceed along the route suggested without this problem being resolved. On the DateFromTicks(), TimestampFromTicks(), TimeFromTicks() issue, I'm not sure that there is much benefit to including a C API for this since if the datetime module becomes the standard method for managing datetime objects, there is no point in having the DB API modules provide something that the datetime module already provides, right? During the transition period the few lines of code needed can be posted. If you disagree I can certainly provide the few lines as part of the patch but I suspect you will find resistance to having this included by Tim Peters, the maintainer (I believe) of the datetime module. Feel free to correct me... :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-04-26 12:19 Message: Logged In: YES user_id=38388 Sorry for not getting back to you earlier: I didn't see you last message. 1-4) Python defines a macro during the Python build process that is not defined when compiling extensions: Py_BUILD_CORE You can use that macro to compile things differently for core things vs. extensions. 5) Fair enough; I don't see a point in creating a new numbering for each and every module. Please just use 1, 2, 3, 4, ... 6) Hope this makes sense :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-26 11:35 Message: Logged In: YES user_id=619560 Any chance to look at this? Any hope of this being included in Python 2.4? I know a number of people (including Guido) who would appreciate that.... :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-05 18:04 Message: Logged In: YES user_id=619560 I am back at work after finishing my move so I think I can finally devote a little more time to this issue. 1-4) The main reason for datetimeapi.h as opposed to simply modifying datetime.h was that the internal build needs to define things __differently__ from an external build of a module and there doesn't appear to be any way of fixing that -- so if you know a way I think I can manage to create a reasonable patch; otherwise, I'm out of my depth! 5) Guido was suggesting that a new PyIMPORT macro be made available which would verify that the magic number defined in the header file actually matches the magic number exported by the built module; this is simply a developer protection scheme which should eliminate really stupid uses; you can ask him about this or I can e-mail you part of the thread that brought this up 6) Makes sense. I'd like to get the questions above resolved first before proceeding any further, though. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 13:23 Message: Logged In: YES user_id=38388 Thanks for the comments... 1) Please put *all* the code for the C API into datetimeapi.h. I don't like your current mix of putting some things into datetime.h and others into datetimeapi.h (e.g. the PyDateTime_CAPI definition). 2) Since it's the only API missing if you want to use datetime objects in database interfacing, please add it. 3) dito. 4) Yes, the header file is the right place. Have a look at unicodeobject.h for what I have in mind (or mxDateTime.h for that matter). 5) Why add yet another magic number ? Isn't the Python API number enough ? 6) Since you don't provide a doc patch, please put comments into the header file. There can never be enough documentation and developers tend to look at the code rather than the docs ;-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-03-25 13:08 Message: Logged In: YES user_id=619560 Sorry for the delay. Things here have been quite busy recently both at work and at home. Let me see if I can answer your questions. 1) are you suggesting comments that include the macro names in datetimeapi.h similar to what I put in the comments below? Or something else? See modified datetimeapi.h for more info. 2) TimeFromTicks() is a very DB API specific routine. In chatting with others about the datetime module, they were not keen on adding such a beast. I guess there will have to be some discussion about whether datetimeapi.h is also a sort of half implementation for a C module implementing the DB API. I was intending to do this in my cx_Oracle module but not in datetimeapi.h. Could you clarify? 3) I could add C APIS for the three DB API ticks interfaces but I was simply intending to do that in cx_Oracle. Again, see #2. I can do it either way. :-) 4) I have added limited documentation in datetimeapi.h but is that really the right place for it? Is it even remotely close to what is needed? I'm not sure what is needed here exactly. 5) the API magic number is simply an arbitrary number which is stored in datetime.h and need not be understood by the implementor. For your information I simply took the name "datetime", converted each letter to its ordinal value in the alphabet, modded by 16 and used that hex character -- does that make any sense? This can be changed to whatever makes sense, though. 6) Whether or not datetimeapi.h should be sufficient for a developer or whether he should look at documentation in the usual locations (html pages, for example) I'll leave up to the Python development team. Let me know how I can assist. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 12:35 Message: Logged In: YES user_id=38388 Anthony, have you had a chance to look at the comments ? ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 10:06 Message: Logged In: YES user_id=619560 Oops! It looks like my own misunderstanding of SourceForge and how it works is to blame. :-( I'll take a look at this and get back to you as soon as possible -- ignore my previous comment. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 10:03 Message: Logged In: YES user_id=619560 You haven't yet responded to my previous comments -- it looks like SourceForge put them __after__ your comments so it is quite understandable why you didn't notice them. If you can give me the answers to those questions or provide additional questions that I need to answer, then I think we can make progress again. I was waiting for __you__ :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-02-25 18:08 Message: Logged In: YES user_id=38388 Any progress on this ? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-28 06:49 Message: Logged In: YES user_id=38388 Thanks for the clarification. I had forgotten about the macros -- I'd suggest that you document them in the datetimeapi.h include file to not cause the same confusion on the developer side (the trick to achieve adoption is to make things easy on the developer). As for TimeFromTicks(): This should be rather straight forward to implement: you take the time part of the ticks timestamp and create a time object from it. This is the way the DB API constructor works. Could you add C APIs for the three DB API ticks interface methods ? Other things that are missing: * documentation of the way the CAPI object can be used in datetimeapi.h * documentation of the various CAPI entries * documentation of the API magic number (there should be some scheme for this) I think that datetimeapi.h should be enough for a developer to read in order to use the interface. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 16:02 Message: Logged In: YES user_id=619560 I didn't think any additional API would be necessary for extracting date time information since the following macros are available: PyDateTime_GET_YEAR(o) PyDateTime_GET_MONTH(o) PyDateTime_GET_DAY(o) PyDateTime_DATE_GET_HOUR(o) PyDateTime_DATE_GET_MINUTE(o) PyDateTime_DATE_GET_SECOND(o) PyDateTime_DATE_GET_MICROSECOND(o) PyDateTime_TIME_GET_HOUR(o) PyDateTime_TIME_GET_MINUTE(o) PyDateTime_TIME_GET_SECOND(o) PyDateTime_TIME_GET_MICROSECOND(o) Were you thinking of something else that is needed? As for interfacing at the C level for converting from Unix ticks, the following method is already available and could simply be used as a drop in replacement for DateFromTicks() and TimestampFromTicks() from the DB API document. DateFromTicks() --> datetime.date.fromtimestamp() TimestampFromTicks() --> datetime.datetime.fromtimestamp() TimeFromTicks() is not already exposed but discussions with Tim Peters already suggested that would not happen since the concept is rather strange. You'll have to discuss that with him if you disagree! :-) Any comments? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-26 15:46 Message: Logged In: YES user_id=38388 This is a good start. However, you should add more APIs for extracting date/time information to the C API. mxDateTime.h from egenix-mx-base can provide a good basis for this. If you want to make the datetime module usable for database interfacing at C level, you'll also have to add interfaces for Unix ticks conversion, since these are often used by database C level interfaces. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 13:36 Message: Logged In: YES user_id=619560 I have no objection to providing patches for doc and test. A quick look at _testcapimodule.c doesn't provide any obvious ideas as to how to patch it. I looked for cStringIO which has a C API already and found nothing in there at all. The documentation also simply says "see the source for the information you require". Any suggestions as to how to proceed? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-01-25 22:17 Message: Logged In: YES user_id=31435 Marc-Andre, can you review this? I think you have more experience building C APIs for modules than anyone else (while I don't have any). There's certainly no objection to giving datetime a C API, and Guido already said he doesn't want to rely on that Martin changed datetime to be built as part of the core (in 2.3 on Windows, datetime.pyd was a distinct DLL; in CVS, datetime is compiled into the core DLL now). Anthony, you probably need to add doc and test patches. _testcapimodule.c holds tests of things you can only get at from C. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 From noreply at sourceforge.net Wed May 26 15:39:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 15:44:52 2004 Subject: [Patches] [ python-Patches-876130 ] add C API to datetime module Message-ID: Patches item #876130, was opened at 2004-01-13 08:20 Message generated for change (Comment added) made by atuining You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 Category: Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Anthony Tuininga (atuining) Assigned to: M.-A. Lemburg (lemburg) Summary: add C API to datetime module Initial Comment: The datetime module does not have a C API which means that modules written in C cannot take advantage of the datetime module without incurring significant overhead. These patches supply this lack and are based on the C API exported by the mxDateTime module and the cStringIO module and enhanced as suggested by Guido. ---------------------------------------------------------------------- >Comment By: Anthony Tuininga (atuining) Date: 2004-05-26 13:39 Message: Logged In: YES user_id=619560 I would agree with your assessment of the issue regarding the TimeFromTicks() API and find it quite reasonable. Either implement both at the C and Python levels or not at all. Since there is no consensus on this, none it is. :-) Did you have a chance to review the actual code? The changes are fairly minimal but you might have questions about readability, names, etc. It would be great if this could be included in an upcoming 2.4 alpha/beta so that I can provide support in cx_Oracle for the next release. Thanks for your time. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-26 13:32 Message: Logged In: YES user_id=31435 Under the belief that the *intent* of this patch is to add a C API to Python's datetime module, it should stick to exposing C ways to spell what Python programmers can already do from Python using datetime. Since nothing like TimeFromTicks() exists in the Python datetime API, it simply doesn't make sense to invent one available only from C. It *may* make sense to invent one in a DB API wrapper around datetime, but I view that as out of scope for this patch. I would not add TimeFromTicks() to the Python datetime API either, because the docs are ambiguous. The sample implementation Marc-Andre posted here uses localtime() to resolve the ambiguity in the docs, but that has to be a wrong choice for some time-of-day apps. For example, there's apparently no portable way to spell "noon" using TimeFromTicks(): the value you get back depends on which time zone localtime() happens to use at the time it's called. If people want time-of-day from a POSIX timestamp, it remains easy to get it from what datetime already supplies, but to do so you have to be explicit about whether you want UTC or local time interpretation. Both are easily spelled already, and it's a Good Thing that you need to be explicit if you want to do such a thing. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 13:25 Message: Logged In: YES user_id=38388 Tim, please review. Thanks. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 13:22 Message: Logged In: YES user_id=38388 Thanks, but I don't have time to review it. As for the TimeFromTicks() API: I can understand that you don't feel like implementing it, but hey, I won't use datetime either. The only reason I'm trying to push full support of the DB API specs is that Guido et al. wanted to see a note in the DB API that datetime also provides a usable interface to do date/time interfacing. I'm not very interested in datetime myself, since I'll continue to use mxDateTime, so the situation for me is somewhat similar to yours: investing time into something that I myself will probably not use much (I will add support to mxODBC for those who like to use datetime instead of mxDateTime, but won't use it myself). Feel free to check in your changes. Thanks. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-26 10:30 Message: Logged In: YES user_id=619560 Attached are the modified patches based on your suggestions. I have tested them with my own module cx_Oracle and they appear to work correctly. Any further suggestions would be appreciated. I would like to see this included for Python 2.4 if at all possible. Please let me know what I need to do (if anything) to get this to happen. As for the TimeFromTicks() routine, Oracle has absolutely no use for a "time-only" data type and does not support it. It clearly does not have broad support so I really have no desire to implement it since I will never use it myself. If you consider it important, you could add it yourself or we could wait until someone else who requires it for their implementation of the DB API requires it. I'm not trying to be rude or trying to denigrate the DB API, simply trying to be practical. Please let me know if this is offensive in any way. Thanks. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-19 02:28 Message: Logged In: YES user_id=38388 >From the DB API specs: TimeFromTicks(ticks) This function constructs an object holding a time value from the given ticks value (number of seconds since the epoch; see the documentation of the standard Python time module for details). The reason for this having this API is to be able to construct an object suitable for passing to the database given a Unix ticks value. Since the ticks value contains both a date and a time part and most databases have a special column type for time value, we need two constructor APIs, one to extract just the date part and one for the time part. Here's the mxDateTime implementation for reference: def TimeFromTicks(ticks, # Locals: DateTimeDelta=DateTimeDelta,localtime=_time.localtime): """ TimeFromTicks(ticks) Constructs a DateTimeDelta instance pointing to the local time indicated by the given ticks value. The date part is ignored. """ return apply(DateTimeDelta, (0,) + localtime(ticks)[3:6]) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-03 11:46 Message: Logged In: YES user_id=31435 I don't understand what TimeFromTicks() is supposed to do. Seconds-from-the-epoch is a point in time, not a time-of- day. If some DB API requires guessing some transformation from seconds-from-the-epoch to time-of-day, that's fine, but it doesn't belong in Python's datetime module. The datetime module should certainly have a method to construct a datetime.datetime from a seconds-from-the- epoch argument -- seconds-from-the-epoch is a way of specifying a datetime.datetime. Given that, if the intent is to throw away the datetime.date portion of the datetime.datetime object, retaining only the datetime.time portion, then that's trivially accomplished by invoking dt.time () (where dt is the datetime.datetime object). Do any databases really store time-of-day as a seconds-from- the-epoch value? Google's top hit on TimeFromTicks() today happens to be a msg from Anthony saying that Oracle does not. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 11:24 Message: Logged In: YES user_id=38388 Macro: I don't know whether you need the macro or not (you probably do for the type checks). In any case, you can do without relying on Python providing you this information via the build process, since only datetimemodule.c is using the header file while Python is being built. TimeFromTicks(): I' ve stated my point. I don't think there's anything to add - I'm tired of fighting for these things... one of the reasons I stopped actively discussing things on python-dev. IMHO, a product that's aimed at developers should make things easy for the developer and try to avoid code duplication if possible. The API is needed in order to support database interfaces that use Unix ticks as basis for date/time, so if you don't add it to the module, the ten or twenty module authors out there will have to duplicate that piece of work in their implementation. Transition: This was already discussed on the db-api list. We love freedom of choice. Nothing to add. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 10:48 Message: Logged In: YES user_id=619560 About the Py_BUILD_CORE issue: You are suggesting that I define this in datetimemodule.c just prior to the include of datetime.h? Or are you suggesting some other macro? I am not sure about the comment about "not needing a macro in the first place" since there are two groups that will be importing datetime.h: the first being the datetime module itself and the second being the DB API modules. The macro is definitely required to distinguish between the two -- unless I am missing something? On the TimeFromTicks() issue: You obviously consider it important that the three methods be available at the C level for DB API module authors to use directly. My objection is simply a reiteration of an objection raised by Tim Peters. I think you'll have to argue that one with him. :-) I'll provide my implementations for cx_Oracle as part of the patches and then you can fight it out and I'll stay out of it. :-) As for the transition period, I assumed that with the introduction of a standard datetime module, that it would become the suggested implementation for database interfaace modules. That said, I don't really have any authority to say anything on this front so I'll leave that to you and others to decide. :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 09:26 Message: Logged In: YES user_id=38388 About the Py_BUILD_CORE issue: Sorry, I didn't know that Python does not define this macro for core modules. Nevertheless, looking at the datetime module code I don't really understand why you would need such a macro in the first place. Since the datetime module is the only code including the datetime.h header file it should be easy to add a macro definition just before importing datetime.h. On the TimeFromTicks() issue: The whole point of this discussion is to make the datetime module easily usable for authors of database modules. Since the DB API specifies that the module will need to export the three functions, two of which are available through the datetime module already, I don't really see your point in not wanting to add it. BTW, you are wrong in the assumption that the DB API spec will move away from a generic API for date/time objects. The DB API has always been defined in terms of interfaces and does not force a certain set of tools onto the developer. The datetime module will become one possible choice for DB API authors, others will want to stick to mxDateTime, use strings, ticks integers or more specific objects for interfacing. That won't change, so there is no "transition period". ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 08:40 Message: Logged In: YES user_id=619560 I spent some time getting the two include files merged together, only to discover that Py_BUILD_CORE is __NOT__ defined when building modules, only when building stuff linked directly into the core interpreter. Perhaps this ought to be defined when building the modules with setup.py? At any rate, I cannot proceed along the route suggested without this problem being resolved. On the DateFromTicks(), TimestampFromTicks(), TimeFromTicks() issue, I'm not sure that there is much benefit to including a C API for this since if the datetime module becomes the standard method for managing datetime objects, there is no point in having the DB API modules provide something that the datetime module already provides, right? During the transition period the few lines of code needed can be posted. If you disagree I can certainly provide the few lines as part of the patch but I suspect you will find resistance to having this included by Tim Peters, the maintainer (I believe) of the datetime module. Feel free to correct me... :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-04-26 10:19 Message: Logged In: YES user_id=38388 Sorry for not getting back to you earlier: I didn't see you last message. 1-4) Python defines a macro during the Python build process that is not defined when compiling extensions: Py_BUILD_CORE You can use that macro to compile things differently for core things vs. extensions. 5) Fair enough; I don't see a point in creating a new numbering for each and every module. Please just use 1, 2, 3, 4, ... 6) Hope this makes sense :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-26 09:35 Message: Logged In: YES user_id=619560 Any chance to look at this? Any hope of this being included in Python 2.4? I know a number of people (including Guido) who would appreciate that.... :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-05 16:04 Message: Logged In: YES user_id=619560 I am back at work after finishing my move so I think I can finally devote a little more time to this issue. 1-4) The main reason for datetimeapi.h as opposed to simply modifying datetime.h was that the internal build needs to define things __differently__ from an external build of a module and there doesn't appear to be any way of fixing that -- so if you know a way I think I can manage to create a reasonable patch; otherwise, I'm out of my depth! 5) Guido was suggesting that a new PyIMPORT macro be made available which would verify that the magic number defined in the header file actually matches the magic number exported by the built module; this is simply a developer protection scheme which should eliminate really stupid uses; you can ask him about this or I can e-mail you part of the thread that brought this up 6) Makes sense. I'd like to get the questions above resolved first before proceeding any further, though. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 11:23 Message: Logged In: YES user_id=38388 Thanks for the comments... 1) Please put *all* the code for the C API into datetimeapi.h. I don't like your current mix of putting some things into datetime.h and others into datetimeapi.h (e.g. the PyDateTime_CAPI definition). 2) Since it's the only API missing if you want to use datetime objects in database interfacing, please add it. 3) dito. 4) Yes, the header file is the right place. Have a look at unicodeobject.h for what I have in mind (or mxDateTime.h for that matter). 5) Why add yet another magic number ? Isn't the Python API number enough ? 6) Since you don't provide a doc patch, please put comments into the header file. There can never be enough documentation and developers tend to look at the code rather than the docs ;-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-03-25 11:08 Message: Logged In: YES user_id=619560 Sorry for the delay. Things here have been quite busy recently both at work and at home. Let me see if I can answer your questions. 1) are you suggesting comments that include the macro names in datetimeapi.h similar to what I put in the comments below? Or something else? See modified datetimeapi.h for more info. 2) TimeFromTicks() is a very DB API specific routine. In chatting with others about the datetime module, they were not keen on adding such a beast. I guess there will have to be some discussion about whether datetimeapi.h is also a sort of half implementation for a C module implementing the DB API. I was intending to do this in my cx_Oracle module but not in datetimeapi.h. Could you clarify? 3) I could add C APIS for the three DB API ticks interfaces but I was simply intending to do that in cx_Oracle. Again, see #2. I can do it either way. :-) 4) I have added limited documentation in datetimeapi.h but is that really the right place for it? Is it even remotely close to what is needed? I'm not sure what is needed here exactly. 5) the API magic number is simply an arbitrary number which is stored in datetime.h and need not be understood by the implementor. For your information I simply took the name "datetime", converted each letter to its ordinal value in the alphabet, modded by 16 and used that hex character -- does that make any sense? This can be changed to whatever makes sense, though. 6) Whether or not datetimeapi.h should be sufficient for a developer or whether he should look at documentation in the usual locations (html pages, for example) I'll leave up to the Python development team. Let me know how I can assist. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 10:35 Message: Logged In: YES user_id=38388 Anthony, have you had a chance to look at the comments ? ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 08:06 Message: Logged In: YES user_id=619560 Oops! It looks like my own misunderstanding of SourceForge and how it works is to blame. :-( I'll take a look at this and get back to you as soon as possible -- ignore my previous comment. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 08:03 Message: Logged In: YES user_id=619560 You haven't yet responded to my previous comments -- it looks like SourceForge put them __after__ your comments so it is quite understandable why you didn't notice them. If you can give me the answers to those questions or provide additional questions that I need to answer, then I think we can make progress again. I was waiting for __you__ :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-02-25 16:08 Message: Logged In: YES user_id=38388 Any progress on this ? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-28 04:49 Message: Logged In: YES user_id=38388 Thanks for the clarification. I had forgotten about the macros -- I'd suggest that you document them in the datetimeapi.h include file to not cause the same confusion on the developer side (the trick to achieve adoption is to make things easy on the developer). As for TimeFromTicks(): This should be rather straight forward to implement: you take the time part of the ticks timestamp and create a time object from it. This is the way the DB API constructor works. Could you add C APIs for the three DB API ticks interface methods ? Other things that are missing: * documentation of the way the CAPI object can be used in datetimeapi.h * documentation of the various CAPI entries * documentation of the API magic number (there should be some scheme for this) I think that datetimeapi.h should be enough for a developer to read in order to use the interface. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 14:02 Message: Logged In: YES user_id=619560 I didn't think any additional API would be necessary for extracting date time information since the following macros are available: PyDateTime_GET_YEAR(o) PyDateTime_GET_MONTH(o) PyDateTime_GET_DAY(o) PyDateTime_DATE_GET_HOUR(o) PyDateTime_DATE_GET_MINUTE(o) PyDateTime_DATE_GET_SECOND(o) PyDateTime_DATE_GET_MICROSECOND(o) PyDateTime_TIME_GET_HOUR(o) PyDateTime_TIME_GET_MINUTE(o) PyDateTime_TIME_GET_SECOND(o) PyDateTime_TIME_GET_MICROSECOND(o) Were you thinking of something else that is needed? As for interfacing at the C level for converting from Unix ticks, the following method is already available and could simply be used as a drop in replacement for DateFromTicks() and TimestampFromTicks() from the DB API document. DateFromTicks() --> datetime.date.fromtimestamp() TimestampFromTicks() --> datetime.datetime.fromtimestamp() TimeFromTicks() is not already exposed but discussions with Tim Peters already suggested that would not happen since the concept is rather strange. You'll have to discuss that with him if you disagree! :-) Any comments? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-26 13:46 Message: Logged In: YES user_id=38388 This is a good start. However, you should add more APIs for extracting date/time information to the C API. mxDateTime.h from egenix-mx-base can provide a good basis for this. If you want to make the datetime module usable for database interfacing at C level, you'll also have to add interfaces for Unix ticks conversion, since these are often used by database C level interfaces. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 11:36 Message: Logged In: YES user_id=619560 I have no objection to providing patches for doc and test. A quick look at _testcapimodule.c doesn't provide any obvious ideas as to how to patch it. I looked for cStringIO which has a C API already and found nothing in there at all. The documentation also simply says "see the source for the information you require". Any suggestions as to how to proceed? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-01-25 20:17 Message: Logged In: YES user_id=31435 Marc-Andre, can you review this? I think you have more experience building C APIs for modules than anyone else (while I don't have any). There's certainly no objection to giving datetime a C API, and Guido already said he doesn't want to rely on that Martin changed datetime to be built as part of the core (in 2.3 on Windows, datetime.pyd was a distinct DLL; in CVS, datetime is compiled into the core DLL now). Anthony, you probably need to add doc and test patches. _testcapimodule.c holds tests of things you can only get at from C. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 From EXYHTFIRSDB at themail.com Wed May 26 18:12:09 2004 From: EXYHTFIRSDB at themail.com (Viola Tuttle) Date: Wed May 26 16:04:34 2004 Subject: [Patches] Hot Sale on all Systemworks products at Polly's Softshop Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040527/153c449a/attachment.html From noreply at sourceforge.net Thu May 27 02:04:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 27 02:04:38 2004 Subject: [Patches] [ python-Patches-960406 ] unblock signals in threads Message-ID: Patches item #960406, was opened at 2004-05-25 17:00 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: unblock signals in threads Initial Comment: This is a patch which will correct the issues some people have with python's handling of signal handling in threads. It allows any thread to initially catch the signal mark it as triggered, allowing the main thread to later process it. (This is actually just restoring access to the functionality that was in Python 2.1) The special SIGINT handling for the python readline module has been changed so that it can now see an EINTR error code, rather than needing a longjmp out of the readline library itself. If the readline library python is being linked to doesn't have the callback features necessary, it will fall back to its old behavior. ---------------------------------------------------------------------- >Comment By: Andrew Langmead (langmead) Date: 2004-05-27 02:04 Message: Logged In: YES user_id=119306 It seems that at least OS X, sending the kill to the process schedules that the receiving process will run the signal handler at some later time. (it seems to be the only one to frequently run the signal handlers in the opposite order than they were sent) This revised version of the test seems to work better on OS X. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 14:41 Message: Logged In: YES user_id=6656 test_threadsignals hangs for me on OS X. Haven't done anything more thorough than that yet... ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-26 13:48 Message: Logged In: YES user_id=119306 I apologize that the missing patch. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 13:22 Message: Logged In: YES user_id=6656 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. In addition, even if you *did* check this checkbox, a bug in SourceForge prevents attaching a file when *creating* an issue. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 From noreply at sourceforge.net Thu May 27 03:39:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 27 03:39:24 2004 Subject: [Patches] [ python-Patches-961387 ] Make IDLE's paragraph reformatting width configurable Message-ID: Patches item #961387, was opened at 2004-05-27 02:39 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=961387&group_id=5470 Category: IDLE Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Kurt B. Kaiser (kbk) Summary: Make IDLE's paragraph reformatting width configurable Initial Comment: The current reformatting width is hardcoded at 70. This patch makes that option configurable. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=961387&group_id=5470 From noreply at sourceforge.net Thu May 27 06:14:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 27 06:15:12 2004 Subject: [Patches] [ python-Patches-961465 ] New Misc/RPM/python-2.3.spec file Message-ID: Patches item #961465, was opened at 2004-05-27 10:14 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=961465&group_id=5470 Category: Build Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Sean Reifschneider (jafo) Assigned to: Nobody/Anonymous (nobody) Summary: New Misc/RPM/python-2.3.spec file Initial Comment: Includes patches to allow building on Red Hat 7.*/RHEL 2.*, and a bugfix in the fixing of /usr/local/bin/python references. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=961465&group_id=5470 From noreply at sourceforge.net Thu May 27 15:04:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 27 15:05:02 2004 Subject: [Patches] [ python-Patches-736962 ] Port tests to unittest (Part 2) Message-ID: Patches item #736962, was opened at 2003-05-13 12:45 Message generated for change (Comment added) made by doerwalter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 Category: Tests Group: None Status: Open Resolution: Accepted Priority: 5 Submitted By: Walter D?rwald (doerwalter) >Assigned to: Raymond Hettinger (rhettinger) Summary: Port tests to unittest (Part 2) Initial Comment: Here are the next test scripts ported to PyUnit: test_winsound and test_array. For test_array many additional tests have been added (code coverage is at 91%) ---------------------------------------------------------------------- >Comment By: Walter D?rwald (doerwalter) Date: 2004-05-27 21:04 Message: Logged In: YES user_id=89016 dict_diff.txt is a patch that tries to share tests between test_dict, test_userdict, test_os, test_shelve and test_weakref. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-25 22:29 Message: Logged In: YES user_id=31435 Don't really care myself. Would be nice if the docstring for dict.update() matched the new realities, though . BTW, there is a reliable and conventional way to detect whether the 'dict' argument was passed to UserDict.update: use a unique marker object instead of the universally visible None. Like _marker = object() at module level, and ...(self, dict=_marker, ...) in the method defn. That can't be fooled without deliberate intent to deceive. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-25 22:21 Message: Logged In: YES user_id=80475 IMO, it is fine as is. I would be suprised if some code were relying on UserDict.update(None) raising an exception. This sort of thing also comes up also in places like the string and random modules whereever a function or method with an optional argument gets wrapped by another function. The only way around this is to use *args and then you break code that used the unfortunately named "dict" argument as a keyword. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-05-25 21:32 Message: Logged In: YES user_id=89016 Merging test_dict and test_userdict revealed a difference between dict and UserDict.UserDict: >>> import UserDict >>> d = UserDict.UserDict() >>> d.update(None) >>> d {} >>> d = {} >>> d.update(None) Traceback (most recent call last): File "", line 1, in ? TypeError: iteration over non-sequence Should we do anything about this? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 18:03 Message: Logged In: YES user_id=80475 test_dict is fine. Consider beefing up the tests for update(). As of Py2.4, it now takes all the same argument possibilites as dict(): d.update(a=1, b=2) d.update([('a', 1), ('b', 2)]) Also, try to link in: from test_userdict import TestMappingProtocol This was supposed to provide tests common to all mappings. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-04-03 13:06 Message: Logged In: YES user_id=89016 The attached test_dict.py moves the dict tests from test_types.py to a new PyUnit test script. Code coverage for dictobject.c is at 90.76% (up from 89.51%) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-03-15 13:17 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_binascii delete Lib/test/test_binascii.py 1.17 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 21:26 Message: Logged In: YES user_id=80475 Okay, this one is fine. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-19 00:49 Message: Logged In: YES user_id=89016 Here's the next one: test_binascii.py. Code coverage in binascii.c is at 92% (could be improved by enhancing test_binhex.py). ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 23:46 Message: Logged In: YES user_id=33168 Looked good to me, test_future_pyunit.diff checked in as: * Lib/test/badsyntax_future3.py 1.2 * Lib/test/badsyntax_future4.py 1.2 * Lib/test/badsyntax_future5.py 1.2 * Lib/test/badsyntax_future6.py 1.2 * Lib/test/badsyntax_future7.py 1.2 * Lib/test/badsyntax_future8.py 1.1 * Lib/test/badsyntax_future9.py 1.1 * Lib/test/test_future.py 1.7 * Lib/test/test_future1.py 1.3 * Lib/test/test_future2.py 1.2 * Lib/test/output/test_future 1.4 ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 22:47 Message: Logged In: YES user_id=33168 Walter, I didn't get a mail from SF either. After you do a cvs add, you need to use -N to cvs diff. For example, cvs diff -N new_file_added_to_cvs_but_not_committed. If you look carefully, you can see this in the patch on the diff line for each file. I'll take a look at the patch. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-13 21:18 Message: Logged In: YES user_id=89016 Here is a version of test_future ported to PyUnit (test_future_pyunit.diff). If this makes sense to you Neal, please check it in. (BTW, how did you convince CVS to include badsyntax_future8.py and badsyntax_future9.py in the diff? Doing a "cvs add" only results in " Lib/test/badsyntax_future8.py is a new entry, no comparison available") ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-11 13:36 Message: Logged In: YES user_id=89016 md5/weakref patch checked in as: Lib/test/test_weakref.py 1.33 Lib/test/test_md5.py 1.5 Lib/test/output/test_md5 remove ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 07:05 Message: Logged In: YES user_id=33168 Improve coverage for test_future too. I'm not sure if test future can be ported to PyUnit. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 06:52 Message: Logged In: YES user_id=33168 Attached are two tests (in one file, I'm being lazy, sorry) to improve test coverage for md5c.c and _weakref.c. test_md5 was ported to unittest, weakref, just adds two little tests to get coverage up to 100%. If you like, just go ahead and check in. You will need to remove Lib/test/output/test_md5 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-08 12:42 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/list_tests.py 1.1 Lib/test/seq_tests.py 1.1 Lib/test/test_list.py 1.1 Lib/test/test_tuple.py 1.1 Lib/test/test_types.py 1.56 Lib/test/test_userlist.py 1.11 Lib/test/output/test_types 1.3 Next will be test_binascii.py before I continue with test_types.py. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-12-08 00:49 Message: Logged In: YES user_id=80475 Looks good. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-11-27 20:48 Message: Logged In: YES user_id=89016 Here the first part of the test_types port: list and tuple tests have been moved to their own scripts: test_tuple.py and test_list.py. Common tests for tuple, list and UserList are shared (in seq_test.py and list_test.py) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-09-02 03:53 Message: Logged In: YES user_id=80475 Applied as: Lib/test/test_slice.py 1.5. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-09-01 01:52 Message: Logged In: YES user_id=80475 Looks good. I added a couple of minor tests. Revised patch attached. Okay to apply. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-31 21:49 Message: Logged In: YES user_id=89016 Thanks! Here's another one: test_slice.py ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-31 01:04 Message: Logged In: YES user_id=80475 Done. See: Lib/test/test_longexp.py 1.9; previous revision: 1.8 Lib/test/test_pep263.py 1.3; previous revision: 1.2 Lib/test/test_sets.py 1.27; previous revision: 1.26 Lib/test/test_structseq.py 1.5; previous revision: 1.4 Lib/test/output/test_longexp delete; previous revision: 1.3 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-30 19:10 Message: Logged In: YES user_id=80475 Will do! ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-30 19:06 Message: Logged In: YES user_id=89016 Here's test_structse.py converted with a few additional tests. If all three scripts are OK, could you check them in Raymond, as I'll be on vacation for three weeks? ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-09 17:47 Message: Logged In: YES user_id=89016 Here are two simple ones: test_pep263 and test_longexp. I can't think of any additional tests to add. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-23 15:38 Message: Logged In: YES user_id=80475 Fixed Walter's review comments and committed as Lib/test/test_compile.py 1.19 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-23 13:49 Message: Logged In: YES user_id=89016 Are you sure, that the code path is the same in the new test_argument_handling() as in the old test? I.e. is "eval('lambda a,a: 0')" the same as "exec 'def f(a, a): pass'"? The print statement in test_float_literals should be changed to a comment. Should the imports in test_unary_minus() be moved to the start of the script? Otherwise the test looks (and runs) OK. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-20 21:26 Message: Logged In: YES user_id=80475 Here's one for you: test_compile.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-19 11:46 Message: Logged In: YES user_id=89016 Maybe test_types.py should be split into several scripts: test_dict, test_tuple, test_list, test_int, test_long etc. Some of them already exist (like test_long), some don't. The constructor tests from test_builtin should probably be moved to the new test scripts as well. Furthermore we should try to share as much testing functionality as possible (e.g. between test_int and test_long, or between test_list and test_userlist) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-18 21:48 Message: Logged In: YES user_id=80475 Great! Can I suggest that test_types.py be next. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-18 16:27 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_builtin.py 1.21 Lib/test/test_complex.py 1.10 (I've moved the constructor tests from test_builtin to test_complex) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-18 01:49 Message: Logged In: YES user_id=80475 I added a few tests. If they are fine with you, go ahead and commit. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 23:36 Message: Logged In: YES user_id=89016 Here's the next one: text_complex.py. There one test that's currently commented out, because it crashes Python on Alpha (see http://www.python.org/sf/756093. The old test scripts states that tests for the constructor are in test_builtin, but I've added many tests to this script, so this is no longer true. We could move the tests to test_builtin, but IMHO that doesn't make sense, we'd better move the rest of the constructor tests from test_builtin to test_complex. I'd like to have a version of assertAlmostEqual() in unittest.py that can cope with complex numbers, but this would have to be coordinated with the standalone version of PyUnit (and it would probably have to wait until the 2.4 cycle starts) (I noticed that there is no assertAlmostEqual in the code on pyunit.sf.net anyway.) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 14:06 Message: Logged In: YES user_id=89016 I'd like to keep this patch open, as it is an ongoing task (the next test scripts to be converted will be test_complex and then maybe test_marshal) ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 23:55 Message: Logged In: YES user_id=357491 OK, all tests pass cleanly. Applied as revision 1.6. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 21:51 Message: Logged In: YES user_id=89016 The joys of unittesting: Breaking code to make it better! ;) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 21:41 Message: Logged In: YES user_id=80475 Okay, it runs fine here. Once Brett confirms that it runs on the Mac, go ahead and load it. P.S. Your improved test_mimetools.py helped detect a latent error in mimetools.py when it was run under Windows. Tim made the fix this weekend. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 21:33 Message: Logged In: YES user_id=89016 Strange, I didn't get this failure here, but I got it on my laptop at home. I've removed the comparison with the getatime() value from test_time(). I hope this fixes it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 21:09 Message: Logged In: YES user_id=80475 Walter, there is one failure left: ====================================== ================================ FAIL: test_time (__main__.PosixPathTest) ------------------------------------------------------------------- --- Traceback (most recent call last): File "test_posixpath.py", line 125, in test_time self.assert_( File "C:\PY23\lib\unittest.py", line 268, in failUnless if not expr: raise self.failureException, msg AssertionError Brett, after Walter revises the patch, just load the patch and make sure the test runs on the Mac. Between the three of us, we can validate the suite on three different platforms. Cheers. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 20:20 Message: Logged In: YES user_id=357491 Just wanting to work me like a dog, huh, Raymond? =) And to clarify for my and Walter's benefit, when you say guards, you mean that the tests don't crap out and say they failed on Windows, right? I thought posixpath was not meant to work under Windows. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 20:09 Message: Logged In: YES user_id=89016 I didn't realize that test_posixpath must work on Windows too. Here's a new version. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 18:04 Message: Logged In: YES user_id=80475 The previous comment applied to another patch. It should have said: Assigning to Brett to make sure the patch runs on the Mac. Don't accept this one until it has guards that allow the tests to run on Windows. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 17:59 Message: Logged In: YES user_id=80475 Assigning to Brett to give experience doing a detail review on this type of change. * examine every line of the diff and consider whether there is any semantic change (exceptions raised, etc). * apply the diff and run the test suite * in the interactive mode, call-up each function and make sure it behaves as expected (this is necessary because the test coverage is very low). * verify that the whitespace has been cleaned up. * look for missing changes (such as use of +=) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 17:44 Message: Logged In: YES user_id=80475 The test file now has dependencies that do not apply to windows. The failure messages are attached. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 14:48 Message: Logged In: YES user_id=89016 Here's the next one: test_posixpath.py with many additional tests. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 19:33 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_mimetools delete Lib/test/test_mimetools.py 1.4 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-22 18:18 Message: Logged In: YES user_id=80475 test_mimetools.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 17:05 Message: Logged In: YES user_id=89016 I've attached a third version of test_mimetools.py that does some checks for the mimetools.Message class. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-21 15:04 Message: Logged In: YES user_id=80475 Attaching a slightly modified test_mimetools which covers more encodings and has a stronger set test. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-19 01:46 Message: Logged In: YES user_id=89016 Agreed, this is too much magic for too little gain. Back to business: Here is test_mimetools ported to PyUnit. Tests for mimetools.Message are still missing. If you can think of any tests please add them. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 05:18 Message: Logged In: YES user_id=80475 Get the module with sys.modules: tests = test_support.findtestclasses(sys.modules [__name__]) test_support.unittest(*tests) Yeah, the inheritance thing is a problem. I was trying to avoid having to modify unittest.TestCase to have a metaclass. The control of the module is kept in a separate SF project and one of its goals is to be backward compatible through 1.5.2 (meaning no metaclasses). A possible workaround is to define a modified testcase in test_support so that people don't import unittest directly anymore: test_support.py ------------------------- import unittest class SmartTestCase(unittest.TestCase): __metaclass__ = autotracktests pass test_sets.py ------------------ class TestBasicOps(test_support.SmartTestCase): run = False . . . class TestBasicOpsEmpty(TestBasicOps): def setUp(self): . . . Still, this is starting to seem a bit magical and tricky. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 04:52 Message: Logged In: YES user_id=89016 But how do I pass the module object from inside the module? And skipping abstract classes seems to be more work in this version: If skipping is done via a class attribute, derived classes have to explicitely reset this flag because of interitance. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 03:59 Message: Logged In: YES user_id=80475 Good call. Instead of using metaclasses, perhaps add a module introspector function to test_support: def findtestclasses(mod): tests = [] for elem in dir(mod): member = getattr(mod, elem) if type(member) != type: continue if issubclass(member, unittest.TestCase): tests.append(member) return tests ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 03:45 Message: Logged In: YES user_id=89016 But this can be solved with a special non-inheritable class attribute: class BaseTest(unittest.TestCase): run = False Then the metaclass can do the following: def __new__(cls, name, bases, dict): if "run" not in dict: dict["run"] = True cls = type.__new__(cls, name, bases, dict) if cls.run: tests.append(cls) return cls ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 03:04 Message: Logged In: YES user_id=80475 I don't think metaclasses or module introspection would help whenever there are classes that derive from TestCase but are not meant to be run directly (their subclasses have the setup/teardown/or class data). test_sets.py has examples of that kind of thing. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 02:50 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_array.py 1.20 Lib/test/test_winsound.py 1.5 Lib/test/output/test_winsound delete > The approach of using tests.append() is elegant and > makes it easier to verify that no tests are being omitted. The most elegant approach would probably be a metaclass that collects all TestCase subclasses that get defined. Classes that only serve as a base class could be skipped by specifying a certain class attribute. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 01:35 Message: Logged In: YES user_id=80475 The approach of using tests.append() is elegant and makes it easier to verify that no tests are being omitted. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 From noreply at sourceforge.net Fri May 28 04:50:44 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 04:53:50 2004 Subject: [Patches] [ python-Patches-949332 ] synchronous signals blocked in pthreads Message-ID: Patches item #949332, was opened at 2004-05-06 18:07 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 Category: Core (C code) Group: Python 2.3 >Status: Closed Resolution: None Priority: 7 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: synchronous signals blocked in pthreads Initial Comment: In a pthreads environment, signals sent asynchronously from things like kill() or the terminal driver are sent to the process and can be handled by any thread. Signals that are sent synchronously from the instruction stream (signals like SIGSEGV, SIGFPE, or SIGILL) are only delivered to the thread that executed the instruction. The current Python threads, when implemented via pthreads, blocks these synchronous signals and so thread that generates one can hang the entire process. The attached patch will remove some signals the list of signals to be blocked. The list of signals to be removed correspond to signals that can be generated synchronously. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-28 09:50 Message: Logged In: YES user_id=6656 Closing in favour of 960406. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-26 17:38 Message: Logged In: YES user_id=119306 The changes in are probably more appropriate than this patch. ---------------------------------------------------------------------- Comment By: Dieter Maurer (dmaurer) Date: 2004-05-26 13:00 Message: Logged In: YES user_id=265829 I verified that the return code is appropriate. ---------------------------------------------------------------------- Comment By: Dieter Maurer (dmaurer) Date: 2004-05-26 11:46 Message: Logged In: YES user_id=265829 This patch addresses a serious problem in Zope2 (and probably other multi-threaded) production environments with LinuxThreads. A crash of one thread brings other threads in an unsane state which they leave only on "kill -9". This lets Guido's "zdeamon" automatic restart logic fail. Even manual restarts are a bit difficult as each individual thread must be sent a "kill -9". Applying the patch lets at least the process die properly. I am not yet sure whether the return code is appropriate for "zdeamon" to restart the process. I will check that soon and report back. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 14:47 Message: Logged In: YES user_id=6380 As Tim mentioned, this is a possible stopgap fix for bug 949332. While many objections have been raised, I still think this is a safer approach than some of the more advanced suggestions. (Too bad it doesn't solve my own, related problem, which is that SIGTERM is ignored by default in a process started from a Python thread.) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 19:11 Message: Logged In: YES user_id=31435 Boosted the priority in the hopes this will get looked at before 2.3.4. Unfortunately, I'm not in a position to test a pthreads patch. I'll note that the good comments about *why* this change is being entertained would be far more useful in the code than in the patch comment here. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=949332&group_id=5470 From noreply at sourceforge.net Fri May 28 04:54:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 04:59:06 2004 Subject: [Patches] [ python-Patches-960406 ] unblock signals in threads Message-ID: Patches item #960406, was opened at 2004-05-25 22:00 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: unblock signals in threads Initial Comment: This is a patch which will correct the issues some people have with python's handling of signal handling in threads. It allows any thread to initially catch the signal mark it as triggered, allowing the main thread to later process it. (This is actually just restoring access to the functionality that was in Python 2.1) The special SIGINT handling for the python readline module has been changed so that it can now see an EINTR error code, rather than needing a longjmp out of the readline library itself. If the readline library python is being linked to doesn't have the callback features necessary, it will fall back to its old behavior. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-28 09:54 Message: Logged In: YES user_id=6656 I haven't been able to test on MacOS X further, unfortunately. The patch works on linux/x86 though (after fixing the TabError :-) but this is with an NTPL kernel, so I didn't have a problem anyway. The C doesn't all conform to the Python style -- see PEP 7. Can you fix that? Why the change to Python/ceval.c? After all that -- thanks a lot! I really want to get this checked in ASAP so we can find out which platforms it breaks at the earliest point in the 2.4 cycle. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-27 07:04 Message: Logged In: YES user_id=119306 It seems that at least OS X, sending the kill to the process schedules that the receiving process will run the signal handler at some later time. (it seems to be the only one to frequently run the signal handlers in the opposite order than they were sent) This revised version of the test seems to work better on OS X. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 19:41 Message: Logged In: YES user_id=6656 test_threadsignals hangs for me on OS X. Haven't done anything more thorough than that yet... ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-26 18:48 Message: Logged In: YES user_id=119306 I apologize that the missing patch. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 18:22 Message: Logged In: YES user_id=6656 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. In addition, even if you *did* check this checkbox, a bug in SourceForge prevents attaching a file when *creating* an issue. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 From Christa at SAFe-mail.net Fri May 28 07:12:14 2004 From: Christa at SAFe-mail.net (Christa@SAFe-mail.net) Date: Fri May 28 06:21:43 2004 Subject: [Patches] How's yourself? Message-ID: >embalm bathe bose fallen panama detour lavoisier dobbin bennington corpul= ent macroprocessor theologian SensaAtion!! We opeRned a NEW site with unbeKatable prices and prodDucts. 800 WORTLD BEST softMware with 90% disZcount - that is a really BEST offTe= r just imagine, you can buy ALL softIware that you ever need and pay price o= f just one of it! OfficeW 2003 for 50$ - nice deal right ? ;) retGail price is 700$ - great = saviQngs, huh? Please spend few momentsN of yours preciousL time to check our offerI - it= is more than Iworth it! http://roast.cheap-oem-license.biz/?dispensary >What will the world be quite overturned when you die? >He who has never failed somewhere, that man can not be great. >Where God builds a church the devil builds a chapel. >The nice thing about egotists is that they don't talk about other people.= >Money is good for bribing yourself through the inconveniences of life. >We always hear about the haves and the have-nots. Why don't we hear about= the doers and the do-nots. >God made man merely to hear some praise of what he'd done on those Five D= ays. >Courage is spelled I-N-T-E-G-R-I-T-Y. >Dying is the most embarrassing thing that can ever happen to you, because= someone's got to take care of all your details. >Show me a frigid women and, nine times out of ten, I'll show you a little= man. >My books kept me from the ring, the dog-pit, the tavern, and the saloon. >Don't be just another member of society, be a living example of your drea= ms and goals. >Business is like a wheelbarrow--it stands still until someone pushes it. >Happiness consists in activity -- it is a running stream, not a stagnant = pool. >To hazard much to get much has more of avarice than wisdom. >I am walking over hot coals suspended over a deep pit at the bottom of wh= ich are a large number of vipers baring their fangs. >Actors die so loud. >The person who in shaky times also wavers only increases the evil, but th= e person of firm decision fashions the universe. >The hero draws inspiration from the virtue of his ancestors. From noreply at sourceforge.net Fri May 28 08:37:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 08:45:13 2004 Subject: [Patches] [ python-Patches-960406 ] unblock signals in threads Message-ID: Patches item #960406, was opened at 2004-05-25 17:00 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: unblock signals in threads Initial Comment: This is a patch which will correct the issues some people have with python's handling of signal handling in threads. It allows any thread to initially catch the signal mark it as triggered, allowing the main thread to later process it. (This is actually just restoring access to the functionality that was in Python 2.1) The special SIGINT handling for the python readline module has been changed so that it can now see an EINTR error code, rather than needing a longjmp out of the readline library itself. If the readline library python is being linked to doesn't have the callback features necessary, it will fall back to its old behavior. ---------------------------------------------------------------------- >Comment By: Andrew Langmead (langmead) Date: 2004-05-28 08:37 Message: Logged In: YES user_id=119306 Thank you for pointing me to PEP 7. I'll take a look at where I am amiss and fix it up. For the change in ceval.c, I took a look at gcc's x86 assembly output of the file, and noticed that the optimizer was altering the order of the busy flag test. Since busy is set from other concurrent execution (other signal handlers), changing the variable to volatile told gcc not to optimize accesses to the variable. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-28 04:54 Message: Logged In: YES user_id=6656 I haven't been able to test on MacOS X further, unfortunately. The patch works on linux/x86 though (after fixing the TabError :-) but this is with an NTPL kernel, so I didn't have a problem anyway. The C doesn't all conform to the Python style -- see PEP 7. Can you fix that? Why the change to Python/ceval.c? After all that -- thanks a lot! I really want to get this checked in ASAP so we can find out which platforms it breaks at the earliest point in the 2.4 cycle. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-27 02:04 Message: Logged In: YES user_id=119306 It seems that at least OS X, sending the kill to the process schedules that the receiving process will run the signal handler at some later time. (it seems to be the only one to frequently run the signal handlers in the opposite order than they were sent) This revised version of the test seems to work better on OS X. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 14:41 Message: Logged In: YES user_id=6656 test_threadsignals hangs for me on OS X. Haven't done anything more thorough than that yet... ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-26 13:48 Message: Logged In: YES user_id=119306 I apologize that the missing patch. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 13:22 Message: Logged In: YES user_id=6656 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. In addition, even if you *did* check this checkbox, a bug in SourceForge prevents attaching a file when *creating* an issue. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 From noreply at sourceforge.net Fri May 28 10:25:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 10:26:02 2004 Subject: [Patches] [ python-Patches-960406 ] unblock signals in threads Message-ID: Patches item #960406, was opened at 2004-05-25 17:00 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: unblock signals in threads Initial Comment: This is a patch which will correct the issues some people have with python's handling of signal handling in threads. It allows any thread to initially catch the signal mark it as triggered, allowing the main thread to later process it. (This is actually just restoring access to the functionality that was in Python 2.1) The special SIGINT handling for the python readline module has been changed so that it can now see an EINTR error code, rather than needing a longjmp out of the readline library itself. If the readline library python is being linked to doesn't have the callback features necessary, it will fall back to its old behavior. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-28 10:25 Message: Logged In: YES user_id=31435 I agree that "busy" always should have been volatile -- once again, good eye! Python C style is basically K&R Classic, hard tab for indentation, open curly at the end of the line opening a block except for first line of function definition. Just make it look like the other C code, but be careful to pick one of the .c files Guido approves of . ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-28 08:37 Message: Logged In: YES user_id=119306 Thank you for pointing me to PEP 7. I'll take a look at where I am amiss and fix it up. For the change in ceval.c, I took a look at gcc's x86 assembly output of the file, and noticed that the optimizer was altering the order of the busy flag test. Since busy is set from other concurrent execution (other signal handlers), changing the variable to volatile told gcc not to optimize accesses to the variable. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-28 04:54 Message: Logged In: YES user_id=6656 I haven't been able to test on MacOS X further, unfortunately. The patch works on linux/x86 though (after fixing the TabError :-) but this is with an NTPL kernel, so I didn't have a problem anyway. The C doesn't all conform to the Python style -- see PEP 7. Can you fix that? Why the change to Python/ceval.c? After all that -- thanks a lot! I really want to get this checked in ASAP so we can find out which platforms it breaks at the earliest point in the 2.4 cycle. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-27 02:04 Message: Logged In: YES user_id=119306 It seems that at least OS X, sending the kill to the process schedules that the receiving process will run the signal handler at some later time. (it seems to be the only one to frequently run the signal handlers in the opposite order than they were sent) This revised version of the test seems to work better on OS X. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 14:41 Message: Logged In: YES user_id=6656 test_threadsignals hangs for me on OS X. Haven't done anything more thorough than that yet... ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-26 13:48 Message: Logged In: YES user_id=119306 I apologize that the missing patch. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 13:22 Message: Logged In: YES user_id=6656 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. In addition, even if you *did* check this checkbox, a bug in SourceForge prevents attaching a file when *creating* an issue. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 From noreply at sourceforge.net Fri May 28 16:59:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 17:01:23 2004 Subject: [Patches] [ python-Patches-736962 ] Port tests to unittest (Part 2) Message-ID: Patches item #736962, was opened at 2003-05-13 12:45 Message generated for change (Comment added) made by doerwalter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 Category: Tests Group: None Status: Open Resolution: Accepted Priority: 5 Submitted By: Walter D?rwald (doerwalter) Assigned to: Raymond Hettinger (rhettinger) Summary: Port tests to unittest (Part 2) Initial Comment: Here are the next test scripts ported to PyUnit: test_winsound and test_array. For test_array many additional tests have been added (code coverage is at 91%) ---------------------------------------------------------------------- >Comment By: Walter D?rwald (doerwalter) Date: 2004-05-28 22:59 Message: Logged In: YES user_id=89016 OK, I've updated the docstring for dict.update() (Object/dictobject.c 2.160) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-05-27 21:04 Message: Logged In: YES user_id=89016 dict_diff.txt is a patch that tries to share tests between test_dict, test_userdict, test_os, test_shelve and test_weakref. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-25 22:29 Message: Logged In: YES user_id=31435 Don't really care myself. Would be nice if the docstring for dict.update() matched the new realities, though . BTW, there is a reliable and conventional way to detect whether the 'dict' argument was passed to UserDict.update: use a unique marker object instead of the universally visible None. Like _marker = object() at module level, and ...(self, dict=_marker, ...) in the method defn. That can't be fooled without deliberate intent to deceive. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-25 22:21 Message: Logged In: YES user_id=80475 IMO, it is fine as is. I would be suprised if some code were relying on UserDict.update(None) raising an exception. This sort of thing also comes up also in places like the string and random modules whereever a function or method with an optional argument gets wrapped by another function. The only way around this is to use *args and then you break code that used the unfortunately named "dict" argument as a keyword. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-05-25 21:32 Message: Logged In: YES user_id=89016 Merging test_dict and test_userdict revealed a difference between dict and UserDict.UserDict: >>> import UserDict >>> d = UserDict.UserDict() >>> d.update(None) >>> d {} >>> d = {} >>> d.update(None) Traceback (most recent call last): File "", line 1, in ? TypeError: iteration over non-sequence Should we do anything about this? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 18:03 Message: Logged In: YES user_id=80475 test_dict is fine. Consider beefing up the tests for update(). As of Py2.4, it now takes all the same argument possibilites as dict(): d.update(a=1, b=2) d.update([('a', 1), ('b', 2)]) Also, try to link in: from test_userdict import TestMappingProtocol This was supposed to provide tests common to all mappings. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-04-03 13:06 Message: Logged In: YES user_id=89016 The attached test_dict.py moves the dict tests from test_types.py to a new PyUnit test script. Code coverage for dictobject.c is at 90.76% (up from 89.51%) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-03-15 13:17 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_binascii delete Lib/test/test_binascii.py 1.17 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 21:26 Message: Logged In: YES user_id=80475 Okay, this one is fine. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-19 00:49 Message: Logged In: YES user_id=89016 Here's the next one: test_binascii.py. Code coverage in binascii.c is at 92% (could be improved by enhancing test_binhex.py). ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 23:46 Message: Logged In: YES user_id=33168 Looked good to me, test_future_pyunit.diff checked in as: * Lib/test/badsyntax_future3.py 1.2 * Lib/test/badsyntax_future4.py 1.2 * Lib/test/badsyntax_future5.py 1.2 * Lib/test/badsyntax_future6.py 1.2 * Lib/test/badsyntax_future7.py 1.2 * Lib/test/badsyntax_future8.py 1.1 * Lib/test/badsyntax_future9.py 1.1 * Lib/test/test_future.py 1.7 * Lib/test/test_future1.py 1.3 * Lib/test/test_future2.py 1.2 * Lib/test/output/test_future 1.4 ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 22:47 Message: Logged In: YES user_id=33168 Walter, I didn't get a mail from SF either. After you do a cvs add, you need to use -N to cvs diff. For example, cvs diff -N new_file_added_to_cvs_but_not_committed. If you look carefully, you can see this in the patch on the diff line for each file. I'll take a look at the patch. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-13 21:18 Message: Logged In: YES user_id=89016 Here is a version of test_future ported to PyUnit (test_future_pyunit.diff). If this makes sense to you Neal, please check it in. (BTW, how did you convince CVS to include badsyntax_future8.py and badsyntax_future9.py in the diff? Doing a "cvs add" only results in " Lib/test/badsyntax_future8.py is a new entry, no comparison available") ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-11 13:36 Message: Logged In: YES user_id=89016 md5/weakref patch checked in as: Lib/test/test_weakref.py 1.33 Lib/test/test_md5.py 1.5 Lib/test/output/test_md5 remove ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 07:05 Message: Logged In: YES user_id=33168 Improve coverage for test_future too. I'm not sure if test future can be ported to PyUnit. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 06:52 Message: Logged In: YES user_id=33168 Attached are two tests (in one file, I'm being lazy, sorry) to improve test coverage for md5c.c and _weakref.c. test_md5 was ported to unittest, weakref, just adds two little tests to get coverage up to 100%. If you like, just go ahead and check in. You will need to remove Lib/test/output/test_md5 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-08 12:42 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/list_tests.py 1.1 Lib/test/seq_tests.py 1.1 Lib/test/test_list.py 1.1 Lib/test/test_tuple.py 1.1 Lib/test/test_types.py 1.56 Lib/test/test_userlist.py 1.11 Lib/test/output/test_types 1.3 Next will be test_binascii.py before I continue with test_types.py. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-12-08 00:49 Message: Logged In: YES user_id=80475 Looks good. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-11-27 20:48 Message: Logged In: YES user_id=89016 Here the first part of the test_types port: list and tuple tests have been moved to their own scripts: test_tuple.py and test_list.py. Common tests for tuple, list and UserList are shared (in seq_test.py and list_test.py) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-09-02 03:53 Message: Logged In: YES user_id=80475 Applied as: Lib/test/test_slice.py 1.5. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-09-01 01:52 Message: Logged In: YES user_id=80475 Looks good. I added a couple of minor tests. Revised patch attached. Okay to apply. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-31 21:49 Message: Logged In: YES user_id=89016 Thanks! Here's another one: test_slice.py ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-31 01:04 Message: Logged In: YES user_id=80475 Done. See: Lib/test/test_longexp.py 1.9; previous revision: 1.8 Lib/test/test_pep263.py 1.3; previous revision: 1.2 Lib/test/test_sets.py 1.27; previous revision: 1.26 Lib/test/test_structseq.py 1.5; previous revision: 1.4 Lib/test/output/test_longexp delete; previous revision: 1.3 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-30 19:10 Message: Logged In: YES user_id=80475 Will do! ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-30 19:06 Message: Logged In: YES user_id=89016 Here's test_structse.py converted with a few additional tests. If all three scripts are OK, could you check them in Raymond, as I'll be on vacation for three weeks? ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-09 17:47 Message: Logged In: YES user_id=89016 Here are two simple ones: test_pep263 and test_longexp. I can't think of any additional tests to add. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-23 15:38 Message: Logged In: YES user_id=80475 Fixed Walter's review comments and committed as Lib/test/test_compile.py 1.19 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-23 13:49 Message: Logged In: YES user_id=89016 Are you sure, that the code path is the same in the new test_argument_handling() as in the old test? I.e. is "eval('lambda a,a: 0')" the same as "exec 'def f(a, a): pass'"? The print statement in test_float_literals should be changed to a comment. Should the imports in test_unary_minus() be moved to the start of the script? Otherwise the test looks (and runs) OK. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-20 21:26 Message: Logged In: YES user_id=80475 Here's one for you: test_compile.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-19 11:46 Message: Logged In: YES user_id=89016 Maybe test_types.py should be split into several scripts: test_dict, test_tuple, test_list, test_int, test_long etc. Some of them already exist (like test_long), some don't. The constructor tests from test_builtin should probably be moved to the new test scripts as well. Furthermore we should try to share as much testing functionality as possible (e.g. between test_int and test_long, or between test_list and test_userlist) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-18 21:48 Message: Logged In: YES user_id=80475 Great! Can I suggest that test_types.py be next. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-18 16:27 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_builtin.py 1.21 Lib/test/test_complex.py 1.10 (I've moved the constructor tests from test_builtin to test_complex) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-18 01:49 Message: Logged In: YES user_id=80475 I added a few tests. If they are fine with you, go ahead and commit. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 23:36 Message: Logged In: YES user_id=89016 Here's the next one: text_complex.py. There one test that's currently commented out, because it crashes Python on Alpha (see http://www.python.org/sf/756093. The old test scripts states that tests for the constructor are in test_builtin, but I've added many tests to this script, so this is no longer true. We could move the tests to test_builtin, but IMHO that doesn't make sense, we'd better move the rest of the constructor tests from test_builtin to test_complex. I'd like to have a version of assertAlmostEqual() in unittest.py that can cope with complex numbers, but this would have to be coordinated with the standalone version of PyUnit (and it would probably have to wait until the 2.4 cycle starts) (I noticed that there is no assertAlmostEqual in the code on pyunit.sf.net anyway.) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 14:06 Message: Logged In: YES user_id=89016 I'd like to keep this patch open, as it is an ongoing task (the next test scripts to be converted will be test_complex and then maybe test_marshal) ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 23:55 Message: Logged In: YES user_id=357491 OK, all tests pass cleanly. Applied as revision 1.6. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 21:51 Message: Logged In: YES user_id=89016 The joys of unittesting: Breaking code to make it better! ;) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 21:41 Message: Logged In: YES user_id=80475 Okay, it runs fine here. Once Brett confirms that it runs on the Mac, go ahead and load it. P.S. Your improved test_mimetools.py helped detect a latent error in mimetools.py when it was run under Windows. Tim made the fix this weekend. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 21:33 Message: Logged In: YES user_id=89016 Strange, I didn't get this failure here, but I got it on my laptop at home. I've removed the comparison with the getatime() value from test_time(). I hope this fixes it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 21:09 Message: Logged In: YES user_id=80475 Walter, there is one failure left: ====================================== ================================ FAIL: test_time (__main__.PosixPathTest) ------------------------------------------------------------------- --- Traceback (most recent call last): File "test_posixpath.py", line 125, in test_time self.assert_( File "C:\PY23\lib\unittest.py", line 268, in failUnless if not expr: raise self.failureException, msg AssertionError Brett, after Walter revises the patch, just load the patch and make sure the test runs on the Mac. Between the three of us, we can validate the suite on three different platforms. Cheers. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 20:20 Message: Logged In: YES user_id=357491 Just wanting to work me like a dog, huh, Raymond? =) And to clarify for my and Walter's benefit, when you say guards, you mean that the tests don't crap out and say they failed on Windows, right? I thought posixpath was not meant to work under Windows. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 20:09 Message: Logged In: YES user_id=89016 I didn't realize that test_posixpath must work on Windows too. Here's a new version. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 18:04 Message: Logged In: YES user_id=80475 The previous comment applied to another patch. It should have said: Assigning to Brett to make sure the patch runs on the Mac. Don't accept this one until it has guards that allow the tests to run on Windows. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 17:59 Message: Logged In: YES user_id=80475 Assigning to Brett to give experience doing a detail review on this type of change. * examine every line of the diff and consider whether there is any semantic change (exceptions raised, etc). * apply the diff and run the test suite * in the interactive mode, call-up each function and make sure it behaves as expected (this is necessary because the test coverage is very low). * verify that the whitespace has been cleaned up. * look for missing changes (such as use of +=) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 17:44 Message: Logged In: YES user_id=80475 The test file now has dependencies that do not apply to windows. The failure messages are attached. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 14:48 Message: Logged In: YES user_id=89016 Here's the next one: test_posixpath.py with many additional tests. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 19:33 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_mimetools delete Lib/test/test_mimetools.py 1.4 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-22 18:18 Message: Logged In: YES user_id=80475 test_mimetools.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 17:05 Message: Logged In: YES user_id=89016 I've attached a third version of test_mimetools.py that does some checks for the mimetools.Message class. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-21 15:04 Message: Logged In: YES user_id=80475 Attaching a slightly modified test_mimetools which covers more encodings and has a stronger set test. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-19 01:46 Message: Logged In: YES user_id=89016 Agreed, this is too much magic for too little gain. Back to business: Here is test_mimetools ported to PyUnit. Tests for mimetools.Message are still missing. If you can think of any tests please add them. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 05:18 Message: Logged In: YES user_id=80475 Get the module with sys.modules: tests = test_support.findtestclasses(sys.modules [__name__]) test_support.unittest(*tests) Yeah, the inheritance thing is a problem. I was trying to avoid having to modify unittest.TestCase to have a metaclass. The control of the module is kept in a separate SF project and one of its goals is to be backward compatible through 1.5.2 (meaning no metaclasses). A possible workaround is to define a modified testcase in test_support so that people don't import unittest directly anymore: test_support.py ------------------------- import unittest class SmartTestCase(unittest.TestCase): __metaclass__ = autotracktests pass test_sets.py ------------------ class TestBasicOps(test_support.SmartTestCase): run = False . . . class TestBasicOpsEmpty(TestBasicOps): def setUp(self): . . . Still, this is starting to seem a bit magical and tricky. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 04:52 Message: Logged In: YES user_id=89016 But how do I pass the module object from inside the module? And skipping abstract classes seems to be more work in this version: If skipping is done via a class attribute, derived classes have to explicitely reset this flag because of interitance. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 03:59 Message: Logged In: YES user_id=80475 Good call. Instead of using metaclasses, perhaps add a module introspector function to test_support: def findtestclasses(mod): tests = [] for elem in dir(mod): member = getattr(mod, elem) if type(member) != type: continue if issubclass(member, unittest.TestCase): tests.append(member) return tests ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 03:45 Message: Logged In: YES user_id=89016 But this can be solved with a special non-inheritable class attribute: class BaseTest(unittest.TestCase): run = False Then the metaclass can do the following: def __new__(cls, name, bases, dict): if "run" not in dict: dict["run"] = True cls = type.__new__(cls, name, bases, dict) if cls.run: tests.append(cls) return cls ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 03:04 Message: Logged In: YES user_id=80475 I don't think metaclasses or module introspection would help whenever there are classes that derive from TestCase but are not meant to be run directly (their subclasses have the setup/teardown/or class data). test_sets.py has examples of that kind of thing. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 02:50 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_array.py 1.20 Lib/test/test_winsound.py 1.5 Lib/test/output/test_winsound delete > The approach of using tests.append() is elegant and > makes it easier to verify that no tests are being omitted. The most elegant approach would probably be a metaclass that collects all TestCase subclasses that get defined. Classes that only serve as a base class could be skipped by specifying a certain class attribute. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 01:35 Message: Logged In: YES user_id=80475 The approach of using tests.append() is elegant and makes it easier to verify that no tests are being omitted. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 From noreply at sourceforge.net Fri May 28 18:27:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 18:27:18 2004 Subject: [Patches] [ python-Patches-962487 ] locale.getdefaultlocale fails with empty env. variable Message-ID: Patches item #962487, was opened at 2004-05-28 22:27 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=962487&group_id=5470 Category: Library (Lib) Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Matthias Klose (doko) Assigned to: Nobody/Anonymous (nobody) Summary: locale.getdefaultlocale fails with empty env. variable Initial Comment: [forwarded from http://bugs.debian.org/249816] When getdefaultlocale receives an empty string in one of the LANGUAGE, LC_ALL, LC_CTYPE or LANG variables it passes it to _parse_localename function causing a ValueError. It can be easily fixed by checking the truth value of the os.environ.get instead of checking if it's None. The added patch fixes the problem. 342c342 < if localename is not None: --- > if localename: ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=962487&group_id=5470 From noreply at sourceforge.net Fri May 28 18:59:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 18:59:28 2004 Subject: [Patches] [ python-Patches-962502 ] East Asian Width support for Unicode Message-ID: Patches item #962502, was opened at 2004-05-29 07:59 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=962502&group_id=5470 Category: Core (C code) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Hye-Shik Chang (perky) Assigned to: Nobody/Anonymous (nobody) Summary: East Asian Width support for Unicode Initial Comment: As David Goodger's inspiration, I thought that it would be great if we have some unicode methods that manipulates East Asian Width (http://www.unicode.org/reports/tr11/tr11-13.html#UCD). The attached patch implements rough first-time idea. >>> u'1'.iswide() False >>> u'\uac00'.iswide() True >>> u'\ud55c\uae00'.iswide() True >>> u'\ud55c\uae00'.width() 4 >>> u'ab\ud55c\uae00'.width() 6 >>> u'ab\ud55c\uae00'.iswide() False ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=962502&group_id=5470 From noreply at sourceforge.net Fri May 28 19:02:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 19:02:44 2004 Subject: [Patches] [ python-Patches-736962 ] Port tests to unittest (Part 2) Message-ID: Patches item #736962, was opened at 2003-05-13 05:45 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 Category: Tests Group: None Status: Open Resolution: Accepted Priority: 5 Submitted By: Walter D?rwald (doerwalter) >Assigned to: Walter D?rwald (doerwalter) Summary: Port tests to unittest (Part 2) Initial Comment: Here are the next test scripts ported to PyUnit: test_winsound and test_array. For test_array many additional tests have been added (code coverage is at 91%) ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-28 18:02 Message: Logged In: YES user_id=80475 Looks good. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-05-28 15:59 Message: Logged In: YES user_id=89016 OK, I've updated the docstring for dict.update() (Object/dictobject.c 2.160) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-05-27 14:04 Message: Logged In: YES user_id=89016 dict_diff.txt is a patch that tries to share tests between test_dict, test_userdict, test_os, test_shelve and test_weakref. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-25 15:29 Message: Logged In: YES user_id=31435 Don't really care myself. Would be nice if the docstring for dict.update() matched the new realities, though . BTW, there is a reliable and conventional way to detect whether the 'dict' argument was passed to UserDict.update: use a unique marker object instead of the universally visible None. Like _marker = object() at module level, and ...(self, dict=_marker, ...) in the method defn. That can't be fooled without deliberate intent to deceive. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-25 15:21 Message: Logged In: YES user_id=80475 IMO, it is fine as is. I would be suprised if some code were relying on UserDict.update(None) raising an exception. This sort of thing also comes up also in places like the string and random modules whereever a function or method with an optional argument gets wrapped by another function. The only way around this is to use *args and then you break code that used the unfortunately named "dict" argument as a keyword. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-05-25 14:32 Message: Logged In: YES user_id=89016 Merging test_dict and test_userdict revealed a difference between dict and UserDict.UserDict: >>> import UserDict >>> d = UserDict.UserDict() >>> d.update(None) >>> d {} >>> d = {} >>> d.update(None) Traceback (most recent call last): File "", line 1, in ? TypeError: iteration over non-sequence Should we do anything about this? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 11:03 Message: Logged In: YES user_id=80475 test_dict is fine. Consider beefing up the tests for update(). As of Py2.4, it now takes all the same argument possibilites as dict(): d.update(a=1, b=2) d.update([('a', 1), ('b', 2)]) Also, try to link in: from test_userdict import TestMappingProtocol This was supposed to provide tests common to all mappings. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-04-03 06:06 Message: Logged In: YES user_id=89016 The attached test_dict.py moves the dict tests from test_types.py to a new PyUnit test script. Code coverage for dictobject.c is at 90.76% (up from 89.51%) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-03-15 07:17 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_binascii delete Lib/test/test_binascii.py 1.17 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 15:26 Message: Logged In: YES user_id=80475 Okay, this one is fine. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-18 18:49 Message: Logged In: YES user_id=89016 Here's the next one: test_binascii.py. Code coverage in binascii.c is at 92% (could be improved by enhancing test_binhex.py). ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 17:46 Message: Logged In: YES user_id=33168 Looked good to me, test_future_pyunit.diff checked in as: * Lib/test/badsyntax_future3.py 1.2 * Lib/test/badsyntax_future4.py 1.2 * Lib/test/badsyntax_future5.py 1.2 * Lib/test/badsyntax_future6.py 1.2 * Lib/test/badsyntax_future7.py 1.2 * Lib/test/badsyntax_future8.py 1.1 * Lib/test/badsyntax_future9.py 1.1 * Lib/test/test_future.py 1.7 * Lib/test/test_future1.py 1.3 * Lib/test/test_future2.py 1.2 * Lib/test/output/test_future 1.4 ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 16:47 Message: Logged In: YES user_id=33168 Walter, I didn't get a mail from SF either. After you do a cvs add, you need to use -N to cvs diff. For example, cvs diff -N new_file_added_to_cvs_but_not_committed. If you look carefully, you can see this in the patch on the diff line for each file. I'll take a look at the patch. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-13 15:18 Message: Logged In: YES user_id=89016 Here is a version of test_future ported to PyUnit (test_future_pyunit.diff). If this makes sense to you Neal, please check it in. (BTW, how did you convince CVS to include badsyntax_future8.py and badsyntax_future9.py in the diff? Doing a "cvs add" only results in " Lib/test/badsyntax_future8.py is a new entry, no comparison available") ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-11 07:36 Message: Logged In: YES user_id=89016 md5/weakref patch checked in as: Lib/test/test_weakref.py 1.33 Lib/test/test_md5.py 1.5 Lib/test/output/test_md5 remove ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 01:05 Message: Logged In: YES user_id=33168 Improve coverage for test_future too. I'm not sure if test future can be ported to PyUnit. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 00:52 Message: Logged In: YES user_id=33168 Attached are two tests (in one file, I'm being lazy, sorry) to improve test coverage for md5c.c and _weakref.c. test_md5 was ported to unittest, weakref, just adds two little tests to get coverage up to 100%. If you like, just go ahead and check in. You will need to remove Lib/test/output/test_md5 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-08 06:42 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/list_tests.py 1.1 Lib/test/seq_tests.py 1.1 Lib/test/test_list.py 1.1 Lib/test/test_tuple.py 1.1 Lib/test/test_types.py 1.56 Lib/test/test_userlist.py 1.11 Lib/test/output/test_types 1.3 Next will be test_binascii.py before I continue with test_types.py. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-12-07 18:49 Message: Logged In: YES user_id=80475 Looks good. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-11-27 14:48 Message: Logged In: YES user_id=89016 Here the first part of the test_types port: list and tuple tests have been moved to their own scripts: test_tuple.py and test_list.py. Common tests for tuple, list and UserList are shared (in seq_test.py and list_test.py) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-09-01 20:53 Message: Logged In: YES user_id=80475 Applied as: Lib/test/test_slice.py 1.5. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-31 18:52 Message: Logged In: YES user_id=80475 Looks good. I added a couple of minor tests. Revised patch attached. Okay to apply. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-31 14:49 Message: Logged In: YES user_id=89016 Thanks! Here's another one: test_slice.py ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-30 18:04 Message: Logged In: YES user_id=80475 Done. See: Lib/test/test_longexp.py 1.9; previous revision: 1.8 Lib/test/test_pep263.py 1.3; previous revision: 1.2 Lib/test/test_sets.py 1.27; previous revision: 1.26 Lib/test/test_structseq.py 1.5; previous revision: 1.4 Lib/test/output/test_longexp delete; previous revision: 1.3 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-30 12:10 Message: Logged In: YES user_id=80475 Will do! ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-30 12:06 Message: Logged In: YES user_id=89016 Here's test_structse.py converted with a few additional tests. If all three scripts are OK, could you check them in Raymond, as I'll be on vacation for three weeks? ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-09 10:47 Message: Logged In: YES user_id=89016 Here are two simple ones: test_pep263 and test_longexp. I can't think of any additional tests to add. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-23 08:38 Message: Logged In: YES user_id=80475 Fixed Walter's review comments and committed as Lib/test/test_compile.py 1.19 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-23 06:49 Message: Logged In: YES user_id=89016 Are you sure, that the code path is the same in the new test_argument_handling() as in the old test? I.e. is "eval('lambda a,a: 0')" the same as "exec 'def f(a, a): pass'"? The print statement in test_float_literals should be changed to a comment. Should the imports in test_unary_minus() be moved to the start of the script? Otherwise the test looks (and runs) OK. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-20 14:26 Message: Logged In: YES user_id=80475 Here's one for you: test_compile.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-19 04:46 Message: Logged In: YES user_id=89016 Maybe test_types.py should be split into several scripts: test_dict, test_tuple, test_list, test_int, test_long etc. Some of them already exist (like test_long), some don't. The constructor tests from test_builtin should probably be moved to the new test scripts as well. Furthermore we should try to share as much testing functionality as possible (e.g. between test_int and test_long, or between test_list and test_userlist) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-18 14:48 Message: Logged In: YES user_id=80475 Great! Can I suggest that test_types.py be next. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-18 09:27 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_builtin.py 1.21 Lib/test/test_complex.py 1.10 (I've moved the constructor tests from test_builtin to test_complex) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-17 18:49 Message: Logged In: YES user_id=80475 I added a few tests. If they are fine with you, go ahead and commit. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 16:36 Message: Logged In: YES user_id=89016 Here's the next one: text_complex.py. There one test that's currently commented out, because it crashes Python on Alpha (see http://www.python.org/sf/756093. The old test scripts states that tests for the constructor are in test_builtin, but I've added many tests to this script, so this is no longer true. We could move the tests to test_builtin, but IMHO that doesn't make sense, we'd better move the rest of the constructor tests from test_builtin to test_complex. I'd like to have a version of assertAlmostEqual() in unittest.py that can cope with complex numbers, but this would have to be coordinated with the standalone version of PyUnit (and it would probably have to wait until the 2.4 cycle starts) (I noticed that there is no assertAlmostEqual in the code on pyunit.sf.net anyway.) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 07:06 Message: Logged In: YES user_id=89016 I'd like to keep this patch open, as it is an ongoing task (the next test scripts to be converted will be test_complex and then maybe test_marshal) ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 16:55 Message: Logged In: YES user_id=357491 OK, all tests pass cleanly. Applied as revision 1.6. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 14:51 Message: Logged In: YES user_id=89016 The joys of unittesting: Breaking code to make it better! ;) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 14:41 Message: Logged In: YES user_id=80475 Okay, it runs fine here. Once Brett confirms that it runs on the Mac, go ahead and load it. P.S. Your improved test_mimetools.py helped detect a latent error in mimetools.py when it was run under Windows. Tim made the fix this weekend. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 14:33 Message: Logged In: YES user_id=89016 Strange, I didn't get this failure here, but I got it on my laptop at home. I've removed the comparison with the getatime() value from test_time(). I hope this fixes it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 14:09 Message: Logged In: YES user_id=80475 Walter, there is one failure left: ====================================== ================================ FAIL: test_time (__main__.PosixPathTest) ------------------------------------------------------------------- --- Traceback (most recent call last): File "test_posixpath.py", line 125, in test_time self.assert_( File "C:\PY23\lib\unittest.py", line 268, in failUnless if not expr: raise self.failureException, msg AssertionError Brett, after Walter revises the patch, just load the patch and make sure the test runs on the Mac. Between the three of us, we can validate the suite on three different platforms. Cheers. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 13:20 Message: Logged In: YES user_id=357491 Just wanting to work me like a dog, huh, Raymond? =) And to clarify for my and Walter's benefit, when you say guards, you mean that the tests don't crap out and say they failed on Windows, right? I thought posixpath was not meant to work under Windows. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 13:09 Message: Logged In: YES user_id=89016 I didn't realize that test_posixpath must work on Windows too. Here's a new version. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 11:04 Message: Logged In: YES user_id=80475 The previous comment applied to another patch. It should have said: Assigning to Brett to make sure the patch runs on the Mac. Don't accept this one until it has guards that allow the tests to run on Windows. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 10:59 Message: Logged In: YES user_id=80475 Assigning to Brett to give experience doing a detail review on this type of change. * examine every line of the diff and consider whether there is any semantic change (exceptions raised, etc). * apply the diff and run the test suite * in the interactive mode, call-up each function and make sure it behaves as expected (this is necessary because the test coverage is very low). * verify that the whitespace has been cleaned up. * look for missing changes (such as use of +=) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 10:44 Message: Logged In: YES user_id=80475 The test file now has dependencies that do not apply to windows. The failure messages are attached. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 07:48 Message: Logged In: YES user_id=89016 Here's the next one: test_posixpath.py with many additional tests. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 12:33 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_mimetools delete Lib/test/test_mimetools.py 1.4 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-22 11:18 Message: Logged In: YES user_id=80475 test_mimetools.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 10:05 Message: Logged In: YES user_id=89016 I've attached a third version of test_mimetools.py that does some checks for the mimetools.Message class. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-21 08:04 Message: Logged In: YES user_id=80475 Attaching a slightly modified test_mimetools which covers more encodings and has a stronger set test. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 18:46 Message: Logged In: YES user_id=89016 Agreed, this is too much magic for too little gain. Back to business: Here is test_mimetools ported to PyUnit. Tests for mimetools.Message are still missing. If you can think of any tests please add them. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 22:18 Message: Logged In: YES user_id=80475 Get the module with sys.modules: tests = test_support.findtestclasses(sys.modules [__name__]) test_support.unittest(*tests) Yeah, the inheritance thing is a problem. I was trying to avoid having to modify unittest.TestCase to have a metaclass. The control of the module is kept in a separate SF project and one of its goals is to be backward compatible through 1.5.2 (meaning no metaclasses). A possible workaround is to define a modified testcase in test_support so that people don't import unittest directly anymore: test_support.py ------------------------- import unittest class SmartTestCase(unittest.TestCase): __metaclass__ = autotracktests pass test_sets.py ------------------ class TestBasicOps(test_support.SmartTestCase): run = False . . . class TestBasicOpsEmpty(TestBasicOps): def setUp(self): . . . Still, this is starting to seem a bit magical and tricky. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-17 21:52 Message: Logged In: YES user_id=89016 But how do I pass the module object from inside the module? And skipping abstract classes seems to be more work in this version: If skipping is done via a class attribute, derived classes have to explicitely reset this flag because of interitance. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 20:59 Message: Logged In: YES user_id=80475 Good call. Instead of using metaclasses, perhaps add a module introspector function to test_support: def findtestclasses(mod): tests = [] for elem in dir(mod): member = getattr(mod, elem) if type(member) != type: continue if issubclass(member, unittest.TestCase): tests.append(member) return tests ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-17 20:45 Message: Logged In: YES user_id=89016 But this can be solved with a special non-inheritable class attribute: class BaseTest(unittest.TestCase): run = False Then the metaclass can do the following: def __new__(cls, name, bases, dict): if "run" not in dict: dict["run"] = True cls = type.__new__(cls, name, bases, dict) if cls.run: tests.append(cls) return cls ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 20:04 Message: Logged In: YES user_id=80475 I don't think metaclasses or module introspection would help whenever there are classes that derive from TestCase but are not meant to be run directly (their subclasses have the setup/teardown/or class data). test_sets.py has examples of that kind of thing. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-17 19:50 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_array.py 1.20 Lib/test/test_winsound.py 1.5 Lib/test/output/test_winsound delete > The approach of using tests.append() is elegant and > makes it easier to verify that no tests are being omitted. The most elegant approach would probably be a metaclass that collects all TestCase subclasses that get defined. Classes that only serve as a base class could be skipped by specifying a certain class attribute. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-17 18:35 Message: Logged In: YES user_id=80475 The approach of using tests.append() is elegant and makes it easier to verify that no tests are being omitted. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 From noreply at sourceforge.net Fri May 28 19:03:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 19:03:52 2004 Subject: [Patches] [ python-Patches-929502 ] Remove ROT_FOUR Message-ID: Patches item #929502, was opened at 2004-04-04 21:01 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=929502&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Michael Chermside (mcherm) Assigned to: Jeremy Hylton (jhylton) Summary: Remove ROT_FOUR Initial Comment: I'll apologize in advance for the excessively lengthy description here. This patch results from an observation that Brett made during the PyCon sprints... that the ROT_FOUR opcode is only used in one incredibly obscure situation... augmented slice assignment (and then only when the slice has exactly 2 arguments). In case it's not obvious what augmented slice assignment IS, here's an example: >>> x = range(6) >>> x[2:4] += 'abc' >>> x [0, 1, 2, 3, 'a', 'b', 'c', 4, 5] The feature needs to be supported but is of no practical use whatsoever. Thus, one could ALMOST say that the ROT_FOUR opcode is never used in Python. Meanwhile, Raymond and others are hard at work trying to squeeze better performance out of Python, and every time they propose adding a new opcode or fiddling with the core interpreter loop, someone somewhere complains about cache effects. If we could drop support for some completely unused opcode, then it might give Raymond and others enough elbow room to do something truly useful. So this patch re-implements augmented slice assignment WITHOUT using ROT_FOUR (some fancy stack juggling and a temporary tuple do the trick). I'll leave it to others to decide whether it is worth keeping the ROT_FOUR opcode around "just because we might need it someday" (I'll note that we dropped ROT_FIVE and ROT_SIX some time ago as they were never used). But keeping it around just to support augmented assignments is no longer an issue. I tried to update everything relevent (ceval and compile obviously, but also docs, the compiler package and a mention in NEWS). I did NOT, however, update a version number for the changed bytecodes, since I'm not sure where such a thing lives (although SOMETHING must be changed whenever a new release has modified the opcodes). One technical note: I apologize for not using a context diff. If someone can give me pointers on how to do that on Windows (I've been using Tortoise CVS) I'll re-run the diff. And finally... after hanging around here for a number of years, this is my first half-way decent patch submission! Thanks to everyone at the PyCon sprints who helped get me set up and compiling. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-28 18:03 Message: Logged In: YES user_id=80475 Perhaps this patch can be closed (I'm -0 on it also). ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-12 10:29 Message: Logged In: YES user_id=80475 Jeremy, what do you think about this one? ---------------------------------------------------------------------- Comment By: Michael Chermside (mcherm) Date: 2004-04-15 20:42 Message: Logged In: YES user_id=99874 All right, I finally got a chance to re-do the diff as a context diff using Jim's approach. And I removed the FOURTH and SET_FOURTH macros as Raymond suggested. Now we just need to decide whether to apply the patch or reject it. Although I'm loath to "waste" the effort that's gone into this so far, I think in the end it is probably NOT a simplification. So I guess I am (reluctantly) a -0 on my own patch. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-08 10:54 Message: Logged In: YES user_id=80475 ROT_FOUR is somewhat obscure and I won't miss it. Once it is gone, you can also eliminate the macro definitions for FOURTH and SET_FOURTH which are only used by ROT_FOUR. The motivation ought to be simplification rather than the vague notion of "cache effects" which gets tossed around too much as if it had more meaning than it does. Size changes to the eval_loop do cause it to hop in and out of local minimums on one compiler or another. While smaller is generally better, do not expect to see a measurable benefits from removing the opcode. It's up to Brett to decide whether the code base is actually simpler after the patch. The "fancy stack juggling and temporary tuple" are not beautiful and add more complication to the compiler than they remove from the eval loop. When disassembling something that used to contain ROT_FOUR, the new byte code is much less obvious. ---------------------------------------------------------------------- Comment By: Michael Chermside (mcherm) Date: 2004-04-07 13:43 Message: Logged In: YES user_id=99874 Jim Jewette writes me separately to say that he handles it by keeping a separate CVS checkout to diff against (as I described before). TortoiseCVS provides "cvs diff" (that's what I used to build this patch) but it defaults to unified diffs, not context diffs. There oughta be a better diff tool in the world. Perhaps I should write one (in Python)! In the meantime, I guess I'll follow Jim's advice and just keep two separate CVS checkouts. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-04-07 13:27 Message: Logged In: YES user_id=357491 There is the ``cvs diff`` command if you have a checkout of CVS. That will diff the specified files (not sure if it will check the whole tree if you don't give it any args). Don't know how TortoiseCVS handles it, though. As for making a single diff file, all that takes is concatenating all the individual diffs into a single file. On UNIX it literally is just taking individual diff files and tacking on to the end each other. As for the better solution, there is always using UNIX. =) ---------------------------------------------------------------------- Comment By: Michael Chermside (mcherm) Date: 2004-04-07 11:33 Message: Logged In: YES user_id=99874 It's fairly easy to run a context diff between two files, what I'm not sure how to do is to run a context diff over the entire tree, comparing my files against those in CVS and collecting the results into a single diff file. I can do that with a command-line CVS, but I'm not sure how to do it with the tools I have on Windows. I suppose I could modify diff.py so it was capable of comparing two trees, but then I'd need to keep an extra up- to-date copy of the entire python CVS tree on my drive to compare against. Seems like there's gotta be a better solution. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-04-06 21:14 Message: Logged In: YES user_id=357491 I think you can actually use the difflib module from the stdlib. The docs say that Tools/scripts/diff.py is a front-end for doing this on files. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=929502&group_id=5470 From noreply at sourceforge.net Sat May 29 01:49:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 29 01:50:16 2004 Subject: [Patches] [ python-Patches-960406 ] unblock signals in threads Message-ID: Patches item #960406, was opened at 2004-05-25 17:00 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Andrew Langmead (langmead) Assigned to: Nobody/Anonymous (nobody) Summary: unblock signals in threads Initial Comment: This is a patch which will correct the issues some people have with python's handling of signal handling in threads. It allows any thread to initially catch the signal mark it as triggered, allowing the main thread to later process it. (This is actually just restoring access to the functionality that was in Python 2.1) The special SIGINT handling for the python readline module has been changed so that it can now see an EINTR error code, rather than needing a longjmp out of the readline library itself. If the readline library python is being linked to doesn't have the callback features necessary, it will fall back to its old behavior. ---------------------------------------------------------------------- >Comment By: Andrew Langmead (langmead) Date: 2004-05-29 01:49 Message: Logged In: YES user_id=119306 Here is a reformatted version of the patch. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-28 10:25 Message: Logged In: YES user_id=31435 I agree that "busy" always should have been volatile -- once again, good eye! Python C style is basically K&R Classic, hard tab for indentation, open curly at the end of the line opening a block except for first line of function definition. Just make it look like the other C code, but be careful to pick one of the .c files Guido approves of . ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-28 08:37 Message: Logged In: YES user_id=119306 Thank you for pointing me to PEP 7. I'll take a look at where I am amiss and fix it up. For the change in ceval.c, I took a look at gcc's x86 assembly output of the file, and noticed that the optimizer was altering the order of the busy flag test. Since busy is set from other concurrent execution (other signal handlers), changing the variable to volatile told gcc not to optimize accesses to the variable. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-28 04:54 Message: Logged In: YES user_id=6656 I haven't been able to test on MacOS X further, unfortunately. The patch works on linux/x86 though (after fixing the TabError :-) but this is with an NTPL kernel, so I didn't have a problem anyway. The C doesn't all conform to the Python style -- see PEP 7. Can you fix that? Why the change to Python/ceval.c? After all that -- thanks a lot! I really want to get this checked in ASAP so we can find out which platforms it breaks at the earliest point in the 2.4 cycle. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-27 02:04 Message: Logged In: YES user_id=119306 It seems that at least OS X, sending the kill to the process schedules that the receiving process will run the signal handler at some later time. (it seems to be the only one to frequently run the signal handlers in the opposite order than they were sent) This revised version of the test seems to work better on OS X. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 14:41 Message: Logged In: YES user_id=6656 test_threadsignals hangs for me on OS X. Haven't done anything more thorough than that yet... ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-26 13:48 Message: Logged In: YES user_id=119306 I apologize that the missing patch. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 13:22 Message: Logged In: YES user_id=6656 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. In addition, even if you *did* check this checkbox, a bug in SourceForge prevents attaching a file when *creating* an issue. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=960406&group_id=5470 From noreply at sourceforge.net Sat May 29 05:22:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 29 05:22:13 2004 Subject: [Patches] [ python-Patches-929502 ] Remove ROT_FOUR Message-ID: Patches item #929502, was opened at 2004-04-04 19:01 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=929502&group_id=5470 Category: Core (C code) Group: Python 2.4 >Status: Closed >Resolution: Rejected Priority: 3 Submitted By: Michael Chermside (mcherm) Assigned to: Jeremy Hylton (jhylton) Summary: Remove ROT_FOUR Initial Comment: I'll apologize in advance for the excessively lengthy description here. This patch results from an observation that Brett made during the PyCon sprints... that the ROT_FOUR opcode is only used in one incredibly obscure situation... augmented slice assignment (and then only when the slice has exactly 2 arguments). In case it's not obvious what augmented slice assignment IS, here's an example: >>> x = range(6) >>> x[2:4] += 'abc' >>> x [0, 1, 2, 3, 'a', 'b', 'c', 4, 5] The feature needs to be supported but is of no practical use whatsoever. Thus, one could ALMOST say that the ROT_FOUR opcode is never used in Python. Meanwhile, Raymond and others are hard at work trying to squeeze better performance out of Python, and every time they propose adding a new opcode or fiddling with the core interpreter loop, someone somewhere complains about cache effects. If we could drop support for some completely unused opcode, then it might give Raymond and others enough elbow room to do something truly useful. So this patch re-implements augmented slice assignment WITHOUT using ROT_FOUR (some fancy stack juggling and a temporary tuple do the trick). I'll leave it to others to decide whether it is worth keeping the ROT_FOUR opcode around "just because we might need it someday" (I'll note that we dropped ROT_FIVE and ROT_SIX some time ago as they were never used). But keeping it around just to support augmented assignments is no longer an issue. I tried to update everything relevent (ceval and compile obviously, but also docs, the compiler package and a mention in NEWS). I did NOT, however, update a version number for the changed bytecodes, since I'm not sure where such a thing lives (although SOMETHING must be changed whenever a new release has modified the opcodes). One technical note: I apologize for not using a context diff. If someone can give me pointers on how to do that on Windows (I've been using Tortoise CVS) I'll re-run the diff. And finally... after hanging around here for a number of years, this is my first half-way decent patch submission! Thanks to everyone at the PyCon sprints who helped get me set up and compiling. ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-05-29 02:22 Message: Logged In: YES user_id=357491 Well, with both Raymond and Michael -0 on this, I am going to take Raymond's hint and close this patch myself. It was definitely a learning experience to figure this sucker out. Maybe we should try to eliminate an opcode at every PyCON. =) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-28 16:03 Message: Logged In: YES user_id=80475 Perhaps this patch can be closed (I'm -0 on it also). ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-12 08:29 Message: Logged In: YES user_id=80475 Jeremy, what do you think about this one? ---------------------------------------------------------------------- Comment By: Michael Chermside (mcherm) Date: 2004-04-15 18:42 Message: Logged In: YES user_id=99874 All right, I finally got a chance to re-do the diff as a context diff using Jim's approach. And I removed the FOURTH and SET_FOURTH macros as Raymond suggested. Now we just need to decide whether to apply the patch or reject it. Although I'm loath to "waste" the effort that's gone into this so far, I think in the end it is probably NOT a simplification. So I guess I am (reluctantly) a -0 on my own patch. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-08 08:54 Message: Logged In: YES user_id=80475 ROT_FOUR is somewhat obscure and I won't miss it. Once it is gone, you can also eliminate the macro definitions for FOURTH and SET_FOURTH which are only used by ROT_FOUR. The motivation ought to be simplification rather than the vague notion of "cache effects" which gets tossed around too much as if it had more meaning than it does. Size changes to the eval_loop do cause it to hop in and out of local minimums on one compiler or another. While smaller is generally better, do not expect to see a measurable benefits from removing the opcode. It's up to Brett to decide whether the code base is actually simpler after the patch. The "fancy stack juggling and temporary tuple" are not beautiful and add more complication to the compiler than they remove from the eval loop. When disassembling something that used to contain ROT_FOUR, the new byte code is much less obvious. ---------------------------------------------------------------------- Comment By: Michael Chermside (mcherm) Date: 2004-04-07 11:43 Message: Logged In: YES user_id=99874 Jim Jewette writes me separately to say that he handles it by keeping a separate CVS checkout to diff against (as I described before). TortoiseCVS provides "cvs diff" (that's what I used to build this patch) but it defaults to unified diffs, not context diffs. There oughta be a better diff tool in the world. Perhaps I should write one (in Python)! In the meantime, I guess I'll follow Jim's advice and just keep two separate CVS checkouts. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-04-07 11:27 Message: Logged In: YES user_id=357491 There is the ``cvs diff`` command if you have a checkout of CVS. That will diff the specified files (not sure if it will check the whole tree if you don't give it any args). Don't know how TortoiseCVS handles it, though. As for making a single diff file, all that takes is concatenating all the individual diffs into a single file. On UNIX it literally is just taking individual diff files and tacking on to the end each other. As for the better solution, there is always using UNIX. =) ---------------------------------------------------------------------- Comment By: Michael Chermside (mcherm) Date: 2004-04-07 09:33 Message: Logged In: YES user_id=99874 It's fairly easy to run a context diff between two files, what I'm not sure how to do is to run a context diff over the entire tree, comparing my files against those in CVS and collecting the results into a single diff file. I can do that with a command-line CVS, but I'm not sure how to do it with the tools I have on Windows. I suppose I could modify diff.py so it was capable of comparing two trees, but then I'd need to keep an extra up- to-date copy of the entire python CVS tree on my drive to compare against. Seems like there's gotta be a better solution. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-04-06 19:14 Message: Logged In: YES user_id=357491 I think you can actually use the difflib module from the stdlib. The docs say that Tools/scripts/diff.py is a front-end for doing this on files. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=929502&group_id=5470 From Rowena at nothingbutdate.com Sat May 29 10:34:09 2004 From: Rowena at nothingbutdate.com (Angelina Dyer) Date: Sat May 29 06:36:33 2004 Subject: [Patches] confirmation.. References: Message-ID: <522D8IR01HX6RBM$z51Y3t7$VJ265MO306@Q7> One of your buddies hooked you up on a date with another buddy. http://nothingbutdate.com/confirm/?oc=50797777 Don't want these notifications? http://nothingbutdate.com/remove/?oc=50797777 cgeophysicsmoscow raven bmw freeport charley cortege heave alberta re joust nearby legend stupefy otoeague dissertation bulky breadfruit atom area dairyman whore quintic cabbage bausch fabulous traipse bagley queue parish perpetrate glucose borealis bum mcgraw digestible rustle clannish craft amide bessel acidulous bacterial conservatism bern shipbuild ito burbank adoptive candace From noreply at sourceforge.net Sun May 30 19:39:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 30 19:39:17 2004 Subject: [Patches] [ python-Patches-963318 ] New module cookielib Message-ID: Patches item #963318, was opened at 2004-05-31 00:39 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963318&group_id=5470 Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: John J Lee (jjlee) Assigned to: Nobody/Anonymous (nobody) Summary: New module cookielib Initial Comment: cookielib itself is here: http://codespeak.net/svn/user/jjlee/wwwsearch/ClientCookie/branch/python-2.4_candidate/ All I'm uploading here are patches to the various other files in Python that are affected. Remaining issues: -The .unverifiable and .origin_req_host attributes I've added to urllib2.Request are a bit ugly (yet another bit of HTTP-specific state). But I guess it's too late to start having subclasses for specific protocol schemes. -Implementation wart: ._now attributes. -test_urllib2.py depends on test_cookielib.py. What's the right way to fix this? -Thread synchronization needs checking by someone with more of a clue about this: I have great confidence that it IS currently broken, since I've never tested it. -Update magic string in LWPCookieJar, if Gisle agrees (I'm waiting for a reply). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963318&group_id=5470 From ECHMAENFWFE at can.es Sun May 30 23:50:36 2004 From: ECHMAENFWFE at can.es (Tommy Benson) Date: Sun May 30 22:56:43 2004 Subject: [Patches] 7* how would you like to be paid for your time? Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040531/7bdbe53d/attachment.html From noreply at sourceforge.net Mon May 31 07:07:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 07:07:24 2004 Subject: [Patches] [ python-Patches-963318 ] New module cookielib Message-ID: Patches item #963318, was opened at 2004-05-31 00:39 Message generated for change (Comment added) made by jjlee You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963318&group_id=5470 Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: John J Lee (jjlee) Assigned to: Nobody/Anonymous (nobody) Summary: New module cookielib Initial Comment: cookielib itself is here: http://codespeak.net/svn/user/jjlee/wwwsearch/ClientCookie/branch/python-2.4_candidate/ All I'm uploading here are patches to the various other files in Python that are affected. Remaining issues: -The .unverifiable and .origin_req_host attributes I've added to urllib2.Request are a bit ugly (yet another bit of HTTP-specific state). But I guess it's too late to start having subclasses for specific protocol schemes. -Implementation wart: ._now attributes. -test_urllib2.py depends on test_cookielib.py. What's the right way to fix this? -Thread synchronization needs checking by someone with more of a clue about this: I have great confidence that it IS currently broken, since I've never tested it. -Update magic string in LWPCookieJar, if Gisle agrees (I'm waiting for a reply). ---------------------------------------------------------------------- >Comment By: John J Lee (jjlee) Date: 2004-05-31 12:07 Message: Logged In: YES user_id=261020 The files that should be copied from the subversion repository are: _LWPCookieJar.py _MozillaCookieJar.py cookielib.py test/test_cookielib.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963318&group_id=5470 From noreply at sourceforge.net Mon May 31 10:20:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 10:23:14 2004 Subject: [Patches] [ python-Patches-963318 ] New module cookielib Message-ID: Patches item #963318, was opened at 2004-05-31 00:39 Message generated for change (Comment added) made by jjlee You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963318&group_id=5470 Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: John J Lee (jjlee) Assigned to: Nobody/Anonymous (nobody) Summary: New module cookielib Initial Comment: cookielib itself is here: http://codespeak.net/svn/user/jjlee/wwwsearch/ClientCookie/branch/python-2.4_candidate/ All I'm uploading here are patches to the various other files in Python that are affected. Remaining issues: -The .unverifiable and .origin_req_host attributes I've added to urllib2.Request are a bit ugly (yet another bit of HTTP-specific state). But I guess it's too late to start having subclasses for specific protocol schemes. -Implementation wart: ._now attributes. -test_urllib2.py depends on test_cookielib.py. What's the right way to fix this? -Thread synchronization needs checking by someone with more of a clue about this: I have great confidence that it IS currently broken, since I've never tested it. -Update magic string in LWPCookieJar, if Gisle agrees (I'm waiting for a reply). ---------------------------------------------------------------------- >Comment By: John J Lee (jjlee) Date: 2004-05-31 15:20 Message: Logged In: YES user_id=261020 Martin, in case you are just about to apply this: I've finally stopped dithering about the first 'remaining issue' above (unverifiable and origin_req_host), and am just about to update liburllib2.tex.patch, urllib2.py.patch and test_urllib2.py.patch (and cookielib itself). ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-31 12:07 Message: Logged In: YES user_id=261020 The files that should be copied from the subversion repository are: _LWPCookieJar.py _MozillaCookieJar.py cookielib.py test/test_cookielib.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963318&group_id=5470 From noreply at sourceforge.net Mon May 31 12:31:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 12:31:29 2004 Subject: [Patches] [ python-Patches-736962 ] Port tests to unittest (Part 2) Message-ID: Patches item #736962, was opened at 2003-05-13 12:45 Message generated for change (Comment added) made by doerwalter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 Category: Tests Group: None Status: Open Resolution: Accepted Priority: 5 Submitted By: Walter D?rwald (doerwalter) Assigned to: Walter D?rwald (doerwalter) Summary: Port tests to unittest (Part 2) Initial Comment: Here are the next test scripts ported to PyUnit: test_winsound and test_array. For test_array many additional tests have been added (code coverage is at 91%) ---------------------------------------------------------------------- >Comment By: Walter D?rwald (doerwalter) Date: 2004-05-31 18:31 Message: Logged In: YES user_id=89016 OK, checked in as: Lib/test/mapping_tests.py 1.1 Lib/test/test_os.py 1.22 Lib/test/test_shelve.py 1.7 Lib/test/test_types.py 1.60 Lib/test/test_userdict.py 1.19 Lib/test/test_weakref.py 1.39 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-29 01:02 Message: Logged In: YES user_id=80475 Looks good. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-05-28 22:59 Message: Logged In: YES user_id=89016 OK, I've updated the docstring for dict.update() (Object/dictobject.c 2.160) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-05-27 21:04 Message: Logged In: YES user_id=89016 dict_diff.txt is a patch that tries to share tests between test_dict, test_userdict, test_os, test_shelve and test_weakref. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-25 22:29 Message: Logged In: YES user_id=31435 Don't really care myself. Would be nice if the docstring for dict.update() matched the new realities, though . BTW, there is a reliable and conventional way to detect whether the 'dict' argument was passed to UserDict.update: use a unique marker object instead of the universally visible None. Like _marker = object() at module level, and ...(self, dict=_marker, ...) in the method defn. That can't be fooled without deliberate intent to deceive. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-25 22:21 Message: Logged In: YES user_id=80475 IMO, it is fine as is. I would be suprised if some code were relying on UserDict.update(None) raising an exception. This sort of thing also comes up also in places like the string and random modules whereever a function or method with an optional argument gets wrapped by another function. The only way around this is to use *args and then you break code that used the unfortunately named "dict" argument as a keyword. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-05-25 21:32 Message: Logged In: YES user_id=89016 Merging test_dict and test_userdict revealed a difference between dict and UserDict.UserDict: >>> import UserDict >>> d = UserDict.UserDict() >>> d.update(None) >>> d {} >>> d = {} >>> d.update(None) Traceback (most recent call last): File "", line 1, in ? TypeError: iteration over non-sequence Should we do anything about this? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-20 18:03 Message: Logged In: YES user_id=80475 test_dict is fine. Consider beefing up the tests for update(). As of Py2.4, it now takes all the same argument possibilites as dict(): d.update(a=1, b=2) d.update([('a', 1), ('b', 2)]) Also, try to link in: from test_userdict import TestMappingProtocol This was supposed to provide tests common to all mappings. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-04-03 13:06 Message: Logged In: YES user_id=89016 The attached test_dict.py moves the dict tests from test_types.py to a new PyUnit test script. Code coverage for dictobject.c is at 90.76% (up from 89.51%) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-03-15 13:17 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_binascii delete Lib/test/test_binascii.py 1.17 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 21:26 Message: Logged In: YES user_id=80475 Okay, this one is fine. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-19 00:49 Message: Logged In: YES user_id=89016 Here's the next one: test_binascii.py. Code coverage in binascii.c is at 92% (could be improved by enhancing test_binhex.py). ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 23:46 Message: Logged In: YES user_id=33168 Looked good to me, test_future_pyunit.diff checked in as: * Lib/test/badsyntax_future3.py 1.2 * Lib/test/badsyntax_future4.py 1.2 * Lib/test/badsyntax_future5.py 1.2 * Lib/test/badsyntax_future6.py 1.2 * Lib/test/badsyntax_future7.py 1.2 * Lib/test/badsyntax_future8.py 1.1 * Lib/test/badsyntax_future9.py 1.1 * Lib/test/test_future.py 1.7 * Lib/test/test_future1.py 1.3 * Lib/test/test_future2.py 1.2 * Lib/test/output/test_future 1.4 ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-13 22:47 Message: Logged In: YES user_id=33168 Walter, I didn't get a mail from SF either. After you do a cvs add, you need to use -N to cvs diff. For example, cvs diff -N new_file_added_to_cvs_but_not_committed. If you look carefully, you can see this in the patch on the diff line for each file. I'll take a look at the patch. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-13 21:18 Message: Logged In: YES user_id=89016 Here is a version of test_future ported to PyUnit (test_future_pyunit.diff). If this makes sense to you Neal, please check it in. (BTW, how did you convince CVS to include badsyntax_future8.py and badsyntax_future9.py in the diff? Doing a "cvs add" only results in " Lib/test/badsyntax_future8.py is a new entry, no comparison available") ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-11 13:36 Message: Logged In: YES user_id=89016 md5/weakref patch checked in as: Lib/test/test_weakref.py 1.33 Lib/test/test_md5.py 1.5 Lib/test/output/test_md5 remove ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 07:05 Message: Logged In: YES user_id=33168 Improve coverage for test_future too. I'm not sure if test future can be ported to PyUnit. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-12-11 06:52 Message: Logged In: YES user_id=33168 Attached are two tests (in one file, I'm being lazy, sorry) to improve test coverage for md5c.c and _weakref.c. test_md5 was ported to unittest, weakref, just adds two little tests to get coverage up to 100%. If you like, just go ahead and check in. You will need to remove Lib/test/output/test_md5 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-12-08 12:42 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/list_tests.py 1.1 Lib/test/seq_tests.py 1.1 Lib/test/test_list.py 1.1 Lib/test/test_tuple.py 1.1 Lib/test/test_types.py 1.56 Lib/test/test_userlist.py 1.11 Lib/test/output/test_types 1.3 Next will be test_binascii.py before I continue with test_types.py. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-12-08 00:49 Message: Logged In: YES user_id=80475 Looks good. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-11-27 20:48 Message: Logged In: YES user_id=89016 Here the first part of the test_types port: list and tuple tests have been moved to their own scripts: test_tuple.py and test_list.py. Common tests for tuple, list and UserList are shared (in seq_test.py and list_test.py) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-09-02 03:53 Message: Logged In: YES user_id=80475 Applied as: Lib/test/test_slice.py 1.5. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-09-01 01:52 Message: Logged In: YES user_id=80475 Looks good. I added a couple of minor tests. Revised patch attached. Okay to apply. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-31 21:49 Message: Logged In: YES user_id=89016 Thanks! Here's another one: test_slice.py ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-31 01:04 Message: Logged In: YES user_id=80475 Done. See: Lib/test/test_longexp.py 1.9; previous revision: 1.8 Lib/test/test_pep263.py 1.3; previous revision: 1.2 Lib/test/test_sets.py 1.27; previous revision: 1.26 Lib/test/test_structseq.py 1.5; previous revision: 1.4 Lib/test/output/test_longexp delete; previous revision: 1.3 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-30 19:10 Message: Logged In: YES user_id=80475 Will do! ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-30 19:06 Message: Logged In: YES user_id=89016 Here's test_structse.py converted with a few additional tests. If all three scripts are OK, could you check them in Raymond, as I'll be on vacation for three weeks? ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-08-09 17:47 Message: Logged In: YES user_id=89016 Here are two simple ones: test_pep263 and test_longexp. I can't think of any additional tests to add. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-23 15:38 Message: Logged In: YES user_id=80475 Fixed Walter's review comments and committed as Lib/test/test_compile.py 1.19 ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-23 13:49 Message: Logged In: YES user_id=89016 Are you sure, that the code path is the same in the new test_argument_handling() as in the old test? I.e. is "eval('lambda a,a: 0')" the same as "exec 'def f(a, a): pass'"? The print statement in test_float_literals should be changed to a comment. Should the imports in test_unary_minus() be moved to the start of the script? Otherwise the test looks (and runs) OK. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-20 21:26 Message: Logged In: YES user_id=80475 Here's one for you: test_compile.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-19 11:46 Message: Logged In: YES user_id=89016 Maybe test_types.py should be split into several scripts: test_dict, test_tuple, test_list, test_int, test_long etc. Some of them already exist (like test_long), some don't. The constructor tests from test_builtin should probably be moved to the new test scripts as well. Furthermore we should try to share as much testing functionality as possible (e.g. between test_int and test_long, or between test_list and test_userlist) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-18 21:48 Message: Logged In: YES user_id=80475 Great! Can I suggest that test_types.py be next. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-18 16:27 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_builtin.py 1.21 Lib/test/test_complex.py 1.10 (I've moved the constructor tests from test_builtin to test_complex) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-18 01:49 Message: Logged In: YES user_id=80475 I added a few tests. If they are fine with you, go ahead and commit. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 23:36 Message: Logged In: YES user_id=89016 Here's the next one: text_complex.py. There one test that's currently commented out, because it crashes Python on Alpha (see http://www.python.org/sf/756093. The old test scripts states that tests for the constructor are in test_builtin, but I've added many tests to this script, so this is no longer true. We could move the tests to test_builtin, but IMHO that doesn't make sense, we'd better move the rest of the constructor tests from test_builtin to test_complex. I'd like to have a version of assertAlmostEqual() in unittest.py that can cope with complex numbers, but this would have to be coordinated with the standalone version of PyUnit (and it would probably have to wait until the 2.4 cycle starts) (I noticed that there is no assertAlmostEqual in the code on pyunit.sf.net anyway.) ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-17 14:06 Message: Logged In: YES user_id=89016 I'd like to keep this patch open, as it is an ongoing task (the next test scripts to be converted will be test_complex and then maybe test_marshal) ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 23:55 Message: Logged In: YES user_id=357491 OK, all tests pass cleanly. Applied as revision 1.6. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 21:51 Message: Logged In: YES user_id=89016 The joys of unittesting: Breaking code to make it better! ;) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 21:41 Message: Logged In: YES user_id=80475 Okay, it runs fine here. Once Brett confirms that it runs on the Mac, go ahead and load it. P.S. Your improved test_mimetools.py helped detect a latent error in mimetools.py when it was run under Windows. Tim made the fix this weekend. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 21:33 Message: Logged In: YES user_id=89016 Strange, I didn't get this failure here, but I got it on my laptop at home. I've removed the comparison with the getatime() value from test_time(). I hope this fixes it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 21:09 Message: Logged In: YES user_id=80475 Walter, there is one failure left: ====================================== ================================ FAIL: test_time (__main__.PosixPathTest) ------------------------------------------------------------------- --- Traceback (most recent call last): File "test_posixpath.py", line 125, in test_time self.assert_( File "C:\PY23\lib\unittest.py", line 268, in failUnless if not expr: raise self.failureException, msg AssertionError Brett, after Walter revises the patch, just load the patch and make sure the test runs on the Mac. Between the three of us, we can validate the suite on three different platforms. Cheers. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-16 20:20 Message: Logged In: YES user_id=357491 Just wanting to work me like a dog, huh, Raymond? =) And to clarify for my and Walter's benefit, when you say guards, you mean that the tests don't crap out and say they failed on Windows, right? I thought posixpath was not meant to work under Windows. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 20:09 Message: Logged In: YES user_id=89016 I didn't realize that test_posixpath must work on Windows too. Here's a new version. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 18:04 Message: Logged In: YES user_id=80475 The previous comment applied to another patch. It should have said: Assigning to Brett to make sure the patch runs on the Mac. Don't accept this one until it has guards that allow the tests to run on Windows. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 17:59 Message: Logged In: YES user_id=80475 Assigning to Brett to give experience doing a detail review on this type of change. * examine every line of the diff and consider whether there is any semantic change (exceptions raised, etc). * apply the diff and run the test suite * in the interactive mode, call-up each function and make sure it behaves as expected (this is necessary because the test coverage is very low). * verify that the whitespace has been cleaned up. * look for missing changes (such as use of +=) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-16 17:44 Message: Logged In: YES user_id=80475 The test file now has dependencies that do not apply to windows. The failure messages are attached. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-16 14:48 Message: Logged In: YES user_id=89016 Here's the next one: test_posixpath.py with many additional tests. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 19:33 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/output/test_mimetools delete Lib/test/test_mimetools.py 1.4 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-22 18:18 Message: Logged In: YES user_id=80475 test_mimetools.py is ready. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-22 17:05 Message: Logged In: YES user_id=89016 I've attached a third version of test_mimetools.py that does some checks for the mimetools.Message class. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-21 15:04 Message: Logged In: YES user_id=80475 Attaching a slightly modified test_mimetools which covers more encodings and has a stronger set test. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-19 01:46 Message: Logged In: YES user_id=89016 Agreed, this is too much magic for too little gain. Back to business: Here is test_mimetools ported to PyUnit. Tests for mimetools.Message are still missing. If you can think of any tests please add them. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 05:18 Message: Logged In: YES user_id=80475 Get the module with sys.modules: tests = test_support.findtestclasses(sys.modules [__name__]) test_support.unittest(*tests) Yeah, the inheritance thing is a problem. I was trying to avoid having to modify unittest.TestCase to have a metaclass. The control of the module is kept in a separate SF project and one of its goals is to be backward compatible through 1.5.2 (meaning no metaclasses). A possible workaround is to define a modified testcase in test_support so that people don't import unittest directly anymore: test_support.py ------------------------- import unittest class SmartTestCase(unittest.TestCase): __metaclass__ = autotracktests pass test_sets.py ------------------ class TestBasicOps(test_support.SmartTestCase): run = False . . . class TestBasicOpsEmpty(TestBasicOps): def setUp(self): . . . Still, this is starting to seem a bit magical and tricky. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 04:52 Message: Logged In: YES user_id=89016 But how do I pass the module object from inside the module? And skipping abstract classes seems to be more work in this version: If skipping is done via a class attribute, derived classes have to explicitely reset this flag because of interitance. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 03:59 Message: Logged In: YES user_id=80475 Good call. Instead of using metaclasses, perhaps add a module introspector function to test_support: def findtestclasses(mod): tests = [] for elem in dir(mod): member = getattr(mod, elem) if type(member) != type: continue if issubclass(member, unittest.TestCase): tests.append(member) return tests ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 03:45 Message: Logged In: YES user_id=89016 But this can be solved with a special non-inheritable class attribute: class BaseTest(unittest.TestCase): run = False Then the metaclass can do the following: def __new__(cls, name, bases, dict): if "run" not in dict: dict["run"] = True cls = type.__new__(cls, name, bases, dict) if cls.run: tests.append(cls) return cls ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 03:04 Message: Logged In: YES user_id=80475 I don't think metaclasses or module introspection would help whenever there are classes that derive from TestCase but are not meant to be run directly (their subclasses have the setup/teardown/or class data). test_sets.py has examples of that kind of thing. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-05-18 02:50 Message: Logged In: YES user_id=89016 Checked in as: Lib/test/test_array.py 1.20 Lib/test/test_winsound.py 1.5 Lib/test/output/test_winsound delete > The approach of using tests.append() is elegant and > makes it easier to verify that no tests are being omitted. The most elegant approach would probably be a metaclass that collects all TestCase subclasses that get defined. Classes that only serve as a base class could be skipped by specifying a certain class attribute. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-05-18 01:35 Message: Logged In: YES user_id=80475 The approach of using tests.append() is elegant and makes it easier to verify that no tests are being omitted. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=736962&group_id=5470 From noreply at sourceforge.net Mon May 31 12:51:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 12:51:43 2004 Subject: [Patches] [ python-Patches-839496 ] SimpleHTTPServer reports wrong content-length for text files Message-ID: Patches item #839496, was opened at 2003-11-10 21:42 Message generated for change (Comment added) made by irmen You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=839496&group_id=5470 Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Irmen de Jong (irmen) Assigned to: Nobody/Anonymous (nobody) Summary: SimpleHTTPServer reports wrong content-length for text files Initial Comment: (Python 2.3.2 on Windows) SimpleHTTPServer reports the size of the file on disk as Content-Length. This works except for text files. If the content type starts with "text/" it is opening the file in 'text' mode rather than 'binary' mode. At least on Windows this causes newline translations, thereby making the actual size of the content transmitted *less* than the content-length! I don't know why SimpleHTTPServer is reading text files with text mode. The included patch removes this distinction so all files are opened in binary mode (and, also on windows, the actual size transmitted is the same as the reported content-length). --Irmen de Jong ---------------------------------------------------------------------- >Comment By: Irmen de Jong (irmen) Date: 2004-05-31 18:51 Message: Logged In: YES user_id=129426 The attached trivial patch removes the special case for text files. ---------------------------------------------------------------------- Comment By: Irmen de Jong (irmen) Date: 2004-05-13 13:21 Message: Logged In: YES user_id=129426 This bug is also still present in the SimpleHTTPServer.py from Python 2.3.3 (and in the current CVS version, too). Is there a reason why it treats text files differently? If so, then at least the reported content-length must be fixed. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=839496&group_id=5470 From RSFZVMPUT at ciens.ula.ve Mon May 31 14:03:24 2004 From: RSFZVMPUT at ciens.ula.ve (Derrick Sanders) Date: Mon May 31 13:07:40 2004 Subject: [Patches] university degrees to the highest bidder :) Message-ID: <4239E24A7AD1FE.9D286@RSFZVMPUT@ciens.ula.ve> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20040531/d5692a75/attachment-0001.html From noreply at sourceforge.net Mon May 31 13:09:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 13:13:50 2004 Subject: [Patches] [ python-Patches-963318 ] New module cookielib Message-ID: Patches item #963318, was opened at 2004-05-31 00:39 Message generated for change (Comment added) made by jjlee You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963318&group_id=5470 Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: John J Lee (jjlee) Assigned to: Nobody/Anonymous (nobody) Summary: New module cookielib Initial Comment: cookielib itself is here: http://codespeak.net/svn/user/jjlee/wwwsearch/ClientCookie/branch/python-2.4_candidate/ All I'm uploading here are patches to the various other files in Python that are affected. Remaining issues: -The .unverifiable and .origin_req_host attributes I've added to urllib2.Request are a bit ugly (yet another bit of HTTP-specific state). But I guess it's too late to start having subclasses for specific protocol schemes. -Implementation wart: ._now attributes. -test_urllib2.py depends on test_cookielib.py. What's the right way to fix this? -Thread synchronization needs checking by someone with more of a clue about this: I have great confidence that it IS currently broken, since I've never tested it. -Update magic string in LWPCookieJar, if Gisle agrees (I'm waiting for a reply). ---------------------------------------------------------------------- >Comment By: John J Lee (jjlee) Date: 2004-05-31 18:09 Message: Logged In: YES user_id=261020 OK, done. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-31 15:20 Message: Logged In: YES user_id=261020 Martin, in case you are just about to apply this: I've finally stopped dithering about the first 'remaining issue' above (unverifiable and origin_req_host), and am just about to update liburllib2.tex.patch, urllib2.py.patch and test_urllib2.py.patch (and cookielib itself). ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-31 12:07 Message: Logged In: YES user_id=261020 The files that should be copied from the subversion repository are: _LWPCookieJar.py _MozillaCookieJar.py cookielib.py test/test_cookielib.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963318&group_id=5470 From noreply at sourceforge.net Mon May 31 14:25:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 14:26:01 2004 Subject: [Patches] [ python-Patches-959726 ] sdist versus SVN Message-ID: Patches item #959726, was opened at 2004-05-24 23:08 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=959726&group_id=5470 Category: Distutils and setup.py Group: None Status: Open >Resolution: Accepted Priority: 5 Submitted By: Tim Peters (tim_one) >Assigned to: Tim Peters (tim_one) Summary: sdist versus SVN Initial Comment: The particular magic the disutils sdist command uses to prune CVS and RCS directories doesn't appear available via anything I can type in MANIFEST.in. So when using sdist from an SVN checkout directory, huge steaming mounds of SVN admin files get sucked up, and then sdist fails because it can't delete the read-only SVN admin files it copied. The patch adds .svn to the same regexp used to special case CVS and RCS directories, which solves the problem. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 20:25 Message: Logged In: YES user_id=21627 The patch is fine, please apply. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=959726&group_id=5470 From noreply at sourceforge.net Mon May 31 14:23:44 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 14:26:13 2004 Subject: [Patches] [ python-Patches-963318 ] New module cookielib Message-ID: Patches item #963318, was opened at 2004-05-31 01:39 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963318&group_id=5470 Category: Library (Lib) Group: Python 2.4 >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: John J Lee (jjlee) Assigned to: Nobody/Anonymous (nobody) Summary: New module cookielib Initial Comment: cookielib itself is here: http://codespeak.net/svn/user/jjlee/wwwsearch/ClientCookie/branch/python-2.4_candidate/ All I'm uploading here are patches to the various other files in Python that are affected. Remaining issues: -The .unverifiable and .origin_req_host attributes I've added to urllib2.Request are a bit ugly (yet another bit of HTTP-specific state). But I guess it's too late to start having subclasses for specific protocol schemes. -Implementation wart: ._now attributes. -test_urllib2.py depends on test_cookielib.py. What's the right way to fix this? -Thread synchronization needs checking by someone with more of a clue about this: I have great confidence that it IS currently broken, since I've never tested it. -Update magic string in LWPCookieJar, if Gisle agrees (I'm waiting for a reply). ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 20:23 Message: Logged In: YES user_id=21627 Thanks for the patch. Applied as lib.tex 1.226 libcookie.tex 1.12 libcookielib.tex 1.1 liburllib2.tex 1.18 whatsnew24.tex 1.49 _LWPCookieJar.py 1.1 _MozillaCookieJar.py 1.1 cookielib.py 1.1 urllib2.py 1.67 test_cookielib.py 1.1 test_urllib2.py 1.13 NEWS 1.978 ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-31 19:09 Message: Logged In: YES user_id=261020 OK, done. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-31 16:20 Message: Logged In: YES user_id=261020 Martin, in case you are just about to apply this: I've finally stopped dithering about the first 'remaining issue' above (unverifiable and origin_req_host), and am just about to update liburllib2.tex.patch, urllib2.py.patch and test_urllib2.py.patch (and cookielib itself). ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-31 13:07 Message: Logged In: YES user_id=261020 The files that should be copied from the subversion repository are: _LWPCookieJar.py _MozillaCookieJar.py cookielib.py test/test_cookielib.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963318&group_id=5470 From noreply at sourceforge.net Mon May 31 14:27:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 14:27:29 2004 Subject: [Patches] [ python-Patches-959534 ] Arrange 64bits detection for ReliantUnix Message-ID: Patches item #959534, was opened at 2004-05-24 18:13 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=959534&group_id=5470 Category: Build Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Jean-frederic Clere (jfclere) Assigned to: Nobody/Anonymous (nobody) Summary: Arrange 64bits detection for ReliantUnix Initial Comment: When compiling python on ReliantUnix (5.43) the compilation fails with unresolved externals like fstatvfs64, lseek64 etc Those routines belong to 5.45. The patch detects the missing lseek64() and suppress lfs support (use_lfs=no). ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 20:27 Message: Logged In: YES user_id=21627 Why do you check for SINIX after the link test fails? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=959534&group_id=5470 From noreply at sourceforge.net Mon May 31 14:28:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 14:28:15 2004 Subject: [Patches] [ python-Patches-961465 ] New Misc/RPM/python-2.3.spec file Message-ID: Patches item #961465, was opened at 2004-05-27 12:14 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=961465&group_id=5470 Category: Build Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Sean Reifschneider (jafo) Assigned to: Nobody/Anonymous (nobody) Summary: New Misc/RPM/python-2.3.spec file Initial Comment: Includes patches to allow building on Red Hat 7.*/RHEL 2.*, and a bugfix in the fixing of /usr/local/bin/python references. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 20:28 Message: Logged In: YES user_id=21627 There may not be any further Python 2.3 releases. Should I still commit the patch? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=961465&group_id=5470 From noreply at sourceforge.net Mon May 31 14:30:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 14:30:52 2004 Subject: [Patches] [ python-Patches-957398 ] Public Generator Object/Type Message-ID: Patches item #957398, was opened at 2004-05-20 15:59 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957398&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: James William Pye (jwpye) Assigned to: Nobody/Anonymous (nobody) Summary: Public Generator Object/Type Initial Comment: Here the patches/files for public generator object/type. removal patch for ceval.c ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 20:30 Message: Logged In: YES user_id=21627 What is the purpose of this patch? ---------------------------------------------------------------------- Comment By: James William Pye (jwpye) Date: 2004-05-20 18:09 Message: Logged In: YES user_id=1044177 Ok, got it to compile, and tried out a simple generator; seems to be fine.. Probably should have asked first, but I'm assuming(yay) that it was desired to draw it inline with *object.[ch] organization protocol.. Only gotcha appears to be that genobject.c, of course, references eval_frame, so I just wrapped a call to it in eval.c with a non-static function(PyEval_EvaluateFrame()), this is probably not be what python-dev would like.. I mostly did it like this to get it to compile.(probably a bad idea, and should probably reference it directly. no need for an extra layer of function call overhead...) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957398&group_id=5470 From noreply at sourceforge.net Mon May 31 14:32:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 14:32:35 2004 Subject: [Patches] [ python-Patches-957240 ] Patch for feature request 491033 Message-ID: Patches item #957240, was opened at 2004-05-20 08:23 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957240&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Josiah Carlson (josiahcarlson) Assigned to: Nobody/Anonymous (nobody) Summary: Patch for feature request 491033 Initial Comment: The attached patch is to offer the functionality desired by feature request: python.org/sf/491033 On machines that use standard IEEE floating point doubles, the 'infinity' needs only to be 1e16 due to the limited precision of floating point. 1e309 is used for convenience sake, and will continue to be a precision infinity until 1024 bits of precision in floating point values is available. At such point in time where 1e309 is available in hardware, it will still sufficient due to the practical limitations of counting. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 20:32 Message: Logged In: YES user_id=21627 This patch lacks changes to the documentation and the test suite. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-20 08:26 Message: Logged In: YES user_id=341410 Sorry about that, I generated the patch by hand and got the ordering messed up. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957240&group_id=5470 From noreply at sourceforge.net Mon May 31 14:53:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 14:53:19 2004 Subject: [Patches] [ python-Patches-961465 ] New Misc/RPM/python-2.3.spec file Message-ID: Patches item #961465, was opened at 2004-05-27 10:14 Message generated for change (Comment added) made by jafo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=961465&group_id=5470 Category: Build Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Sean Reifschneider (jafo) Assigned to: Nobody/Anonymous (nobody) Summary: New Misc/RPM/python-2.3.spec file Initial Comment: Includes patches to allow building on Red Hat 7.*/RHEL 2.*, and a bugfix in the fixing of /usr/local/bin/python references. ---------------------------------------------------------------------- >Comment By: Sean Reifschneider (jafo) Date: 2004-05-31 18:53 Message: Logged In: YES user_id=81797 Yes, please. Please also commit it for 2.4, up the version from 2.3.3 to 2.4. Thanks, Sean ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 18:28 Message: Logged In: YES user_id=21627 There may not be any further Python 2.3 releases. Should I still commit the patch? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=961465&group_id=5470 From noreply at sourceforge.net Mon May 31 14:56:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 14:57:00 2004 Subject: [Patches] [ python-Patches-933238 ] doctest: add a special (dedented) marker for blank lines Message-ID: Patches item #933238, was opened at 2004-04-11 17:11 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=933238&group_id=5470 Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Edward Loper (edloper) Assigned to: Nobody/Anonymous (nobody) Summary: doctest: add a special (dedented) marker for blank lines Initial Comment: A current limitation of doctest is that it can't handle output containing blank lines, since it uses blank lines to detect the end of the example. This patch uses adds special "blank line marker" to mark blank lines in the expected output. In order to distinguish this marker from actual output, it must be dedented such that it ends just before the doctest example's indentation. Here's an example use, with BLANKLINE_MARKER set to "-" (as it is in the patch): def _test_blankline_marker(): r""" This test example uses BLANKLINE_MARKER to signify a blank line: >>> print 'first line\n\nthis is after a blank line' first line - this is after a blank line >>> print 'trailing blanks can be marked too\n' trailing blanks can be marked too - """ If desired, BLANKLINE_MARKER could be changed to another string. (This is an option for the developers of doctest, *not* for users.) E.g., here's the same example with BLANKLINE_MARKER="BL>": def _test_blankline_marker(): r""" This test example uses BLANKLINE_MARKER to signify a blank line: >>> print 'first line\n\nthis is after a blank line' first line BL> this is after a blank line >>> print 'trailing blanks can be marked too\n' trailing blanks can be marked too BL> """ In the patch, the indentation rules for BLANKLINE_MARKER are fairly strict: it must be dedented to end one character before the indentation defined by the doctest example's prompt (>>>). But if desired, we could relax this restriction, and say that it simply has to be dedented further than the prompt. E.g., this would make the following legal (it currently is not): def _test_blankline_marker(): r""" This test example uses BLANKLINE_MARKER to signify a blank line: >>> print 'first line\n\nthis is after a blank line' first line - this is after a blank line >>> print 'trailing blanks can be marked too\n' trailing blanks can be marked too - """ This patch is probably mutually exclusive with patch #932933 [1], since they both have the same goal: to allow blank lines in doctest output. I thought of the other solution first, but I think I prefer this patch. Here's a comparison of pros and cons of this patch vs #932933. [+] it doesn't need a special option to turn it on [+] it handles trailing blank lines [+] it won't break external tools like epytext and restructuredtext that look for doctest blocks. [-] the user must manually add blank line markers. [1] The patch was made avainst revision 1.33 of doctest.py. If this patch looks good, I'd be happy to write a patch for docs & test cases. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 20:56 Message: Logged In: YES user_id=21627 This patch is incomplete: It lacks changes to the documentation (libdoctest.tex) and to the test suite (test_libdoctest.tex). Pasting examples into the patch submission is futile: they are lost once the patch is accepted. So if you think users of your patch should know them, you should incorporate them into your patch - if not in the documentation, then perhaps in the Demo directory. Please submit a single patch file containing all changes. SF has not stripped the indentation; if you look at the HTML source, you'll see it is still there. However, in HTML, indentation is stripped by the web browsers. That is one reason why the submission guidelines say not to include the code in the description. ---------------------------------------------------------------------- Comment By: Edward Loper (edloper) Date: 2004-04-11 17:18 Message: Logged In: YES user_id=195958 I just noticed that sourceforge strips leading whitespace from every line. (Why does it do that?) You should be able to reconstruct what the examples should look like from the text, but if not, here's the first example with leading spaces replaced by periods: def _test_blankline_marker(): ....r""" ....This test example uses BLANKLINE_MARKER to signify ....a blank line: .... ........>>> print 'first line\n\nthis is after a blank line' ........first line .......- ........this is after a blank line ........>>> print 'trailing blanks can be marked too\n' ........trailing blank lines can be marked too .......- ....""" ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=933238&group_id=5470 From noreply at sourceforge.net Mon May 31 15:02:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 15:02:39 2004 Subject: [Patches] [ python-Patches-932930 ] doctest: suggest the use of rawstrings for backslashes Message-ID: Patches item #932930, was opened at 2004-04-10 21:55 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=932930&group_id=5470 Category: Library (Lib) Group: Python 2.4 >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: Edward Loper (edloper) Assigned to: Nobody/Anonymous (nobody) Summary: doctest: suggest the use of rawstrings for backslashes Initial Comment: Previously, the doctest documentation suggested the use of double backslashes for backslashes in docstrings. This patch updates the documentation to suggest the use of raw strings instead (but still notes that double backslashes are an option). Raw strings typically lead to docstrings that are much less error prone and easier to read & maintain than double backslashing. Especially it the backslashes are in some complex string (such as a regular expression). The patch was made against revision 1.33 of doctest.py and revision 1.19 of libdoctest.tex ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 21:02 Message: Logged In: YES user_id=21627 Thanks for the patch. Applied as libdoctest.tex 1.20 doctest.py 1.34 Please include path names in the chunks of the patch. This is best done by invoking "cvs diff" from the toplevel directory. That way, the patch can be applied with "patch -p0", instead of splitting it into multiple files again, and applying each chunk separately. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=932930&group_id=5470 From noreply at sourceforge.net Mon May 31 15:04:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 15:04:08 2004 Subject: [Patches] [ python-Patches-932935 ] doctest: allow custom matchers for testing if got==want Message-ID: Patches item #932935, was opened at 2004-04-10 21:57 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=932935&group_id=5470 Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Edward Loper (edloper) Assigned to: Nobody/Anonymous (nobody) Summary: doctest: allow custom matchers for testing if got==want Initial Comment: To determine the success of a doctest example, doctest compares the actual output to the expected output. The test succeeds if the two match exactly. This patch gives the user the ability to supply a custom "matcher" function, which will be used instead of string equality to test whether the test succeeds. This function should take two string parameters, "got" and "want", which will contain the actual and expected output, respectively; and it should return True if they should be considered to match (i.e., test succeeds), and False if they should not (i.e., test fails). Two sample matcher functions are provided, as well: - match_ignoring_whitespace returns true if the actual output and expected output are equal, ignoring differences in amount of whitespace (eg 2 spaces vs 1). - match_ignoring_trailing_blanklines returns true if the actual output and expected output are equal, once any trailing blank lines are discarded. This can be useful with the END_WITH_DEDENT option, suggested in patch #932933 [1] The patch was made avainst revision 1.33 of doctest.py. [1] http://sourceforge.net/tracker/index.php? func=detail&aid=932933&group_id=5470&atid=3054 70 ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 21:04 Message: Logged In: YES user_id=21627 Please resubmit the patch using unified diffs. You might just as well include test cases and documentation right away. ---------------------------------------------------------------------- Comment By: Edward Loper (edloper) Date: 2004-04-10 22:00 Message: Logged In: YES user_id=195958 If this patch looks good, I'd be happy to write a patch for the docs & test cases. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=932935&group_id=5470 From noreply at sourceforge.net Mon May 31 15:05:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 15:05:50 2004 Subject: [Patches] [ python-Patches-932932 ] doctest: Add Tester params to DocTestSuite Message-ID: Patches item #932932, was opened at 2004-04-10 21:56 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=932932&group_id=5470 Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Edward Loper (edloper) Assigned to: Nobody/Anonymous (nobody) Summary: doctest: Add Tester params to DocTestSuite Initial Comment: This diff adds the following parameters to DocTestSuite: globs, verbose, isprivate, optionflags. The new parameters are simply passed on to the Tester object that the DocTestSuite wraps. This match was made against revision 1.33 of doctest.py ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 21:05 Message: Logged In: YES user_id=21627 The patch looks good in principle. Please submit test cases and documentation. ---------------------------------------------------------------------- Comment By: Edward Loper (edloper) Date: 2004-04-10 21:59 Message: Logged In: YES user_id=195958 If this patch looks good, I'd be happy to write a patch for the docs & test cases. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=932932&group_id=5470 From noreply at sourceforge.net Mon May 31 15:30:09 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 15:30:15 2004 Subject: [Patches] [ python-Patches-959726 ] sdist versus SVN Message-ID: Patches item #959726, was opened at 2004-05-24 17:08 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=959726&group_id=5470 Category: Distutils and setup.py Group: None >Status: Closed Resolution: Accepted Priority: 5 Submitted By: Tim Peters (tim_one) Assigned to: Tim Peters (tim_one) Summary: sdist versus SVN Initial Comment: The particular magic the disutils sdist command uses to prune CVS and RCS directories doesn't appear available via anything I can type in MANIFEST.in. So when using sdist from an SVN checkout directory, huge steaming mounds of SVN admin files get sucked up, and then sdist fails because it can't delete the read-only SVN admin files it copied. The patch adds .svn to the same regexp used to special case CVS and RCS directories, which solves the problem. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-31 15:30 Message: Logged In: YES user_id=31435 Checked in on HEAD, Doc/dist/dist.tex; new revision: 1.74 Lib/distutils/command/sdist.py; new revision: 1.58 Misc/NEWS; new revision: 1.979 ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 14:25 Message: Logged In: YES user_id=21627 The patch is fine, please apply. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=959726&group_id=5470 From noreply at sourceforge.net Mon May 31 15:39:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 15:39:45 2004 Subject: [Patches] [ python-Patches-957398 ] Public Generator Object/Type Message-ID: Patches item #957398, was opened at 2004-05-20 06:59 Message generated for change (Comment added) made by jwpye You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957398&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: James William Pye (jwpye) Assigned to: Nobody/Anonymous (nobody) Summary: Public Generator Object/Type Initial Comment: Here the patches/files for public generator object/type. removal patch for ceval.c ---------------------------------------------------------------------- >Comment By: James William Pye (jwpye) Date: 2004-05-31 12:39 Message: Logged In: YES user_id=1044177 Externalize the generator type so that it can be easily accessed/referenced by extension mod authors and embedders.. http://mail.python.org/pipermail/python-dev/2004-May/044742.html I would use it, as an embedder, in my postgresql extension: http://gborg.postgresql.org/project/postgrespy/projdisplay.php Currently, I compile a little generator and snag the type(this is the same thing as from types import GeneratorType, I suppose), and will do so if I don't have direct access. I also access parts of the generator, currently using getattr, but if I had access to it, I would do it directly. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 11:30 Message: Logged In: YES user_id=21627 What is the purpose of this patch? ---------------------------------------------------------------------- Comment By: James William Pye (jwpye) Date: 2004-05-20 09:09 Message: Logged In: YES user_id=1044177 Ok, got it to compile, and tried out a simple generator; seems to be fine.. Probably should have asked first, but I'm assuming(yay) that it was desired to draw it inline with *object.[ch] organization protocol.. Only gotcha appears to be that genobject.c, of course, references eval_frame, so I just wrapped a call to it in eval.c with a non-static function(PyEval_EvaluateFrame()), this is probably not be what python-dev would like.. I mostly did it like this to get it to compile.(probably a bad idea, and should probably reference it directly. no need for an extra layer of function call overhead...) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957398&group_id=5470 From noreply at sourceforge.net Mon May 31 15:41:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 15:42:01 2004 Subject: [Patches] [ python-Patches-961465 ] New Misc/RPM/python-2.3.spec file Message-ID: Patches item #961465, was opened at 2004-05-27 12:14 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=961465&group_id=5470 Category: Build Group: Python 2.3 >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: Sean Reifschneider (jafo) Assigned to: Nobody/Anonymous (nobody) Summary: New Misc/RPM/python-2.3.spec file Initial Comment: Includes patches to allow building on Red Hat 7.*/RHEL 2.*, and a bugfix in the fixing of /usr/local/bin/python references. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 21:41 Message: Logged In: YES user_id=21627 Ok, applied as python-2.3.spec 1.2.12.13 python-2.3.spec delete python-2.4.spec 1.1 Please review my changes in python-2.4.spec for correctness. ---------------------------------------------------------------------- Comment By: Sean Reifschneider (jafo) Date: 2004-05-31 20:53 Message: Logged In: YES user_id=81797 Yes, please. Please also commit it for 2.4, up the version from 2.3.3 to 2.4. Thanks, Sean ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 20:28 Message: Logged In: YES user_id=21627 There may not be any further Python 2.3 releases. Should I still commit the patch? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=961465&group_id=5470 From noreply at sourceforge.net Mon May 31 15:46:26 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 15:46:30 2004 Subject: [Patches] [ python-Patches-954115 ] Fix os.stat handling of UNC roots Message-ID: Patches item #954115, was opened at 2004-05-14 19:57 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=954115&group_id=5470 Category: Core (C code) Group: Python 2.3 Status: Open Resolution: None >Priority: 6 Submitted By: Greg Chapman (glchapman) >Assigned to: Martin v. L?wis (loewis) Summary: Fix os.stat handling of UNC roots Initial Comment: This is a patch for the bug described at: www.python.org/sf/513572 The patch adds two new routines to posixmodule.c: IsUNCRootA and IsUNCRootW which test if a supplied path matches the pattern for a UNC root directory. These routines are called by posix_do_stat in two different places: 1) when testing if a path with a trailing slash is a drive root and 2) to see if a path without a trailing slash is a UNC root. The first of these is the real bug fix; it allows UNC roots to be specified as "//server/share/" rather than "//server/share//". The second was added because it is easy to forget to add the trailing slash to "//server/share", but stats of UNC roots without the trailing slash will fail. However, I think this is an optional addition. I'm not sure what to do about tests, since existing shares will obviously vary from system to system, and it seems to me that the test suite should not temporarily add a share just to test this feature. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=954115&group_id=5470 From noreply at sourceforge.net Mon May 31 15:50:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 15:50:38 2004 Subject: [Patches] [ python-Patches-957398 ] Public Generator Object/Type Message-ID: Patches item #957398, was opened at 2004-05-20 15:59 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957398&group_id=5470 Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None >Priority: 6 Submitted By: James William Pye (jwpye) >Assigned to: Martin v. L?wis (loewis) Summary: Public Generator Object/Type Initial Comment: Here the patches/files for public generator object/type. removal patch for ceval.c ---------------------------------------------------------------------- Comment By: James William Pye (jwpye) Date: 2004-05-31 21:39 Message: Logged In: YES user_id=1044177 Externalize the generator type so that it can be easily accessed/referenced by extension mod authors and embedders.. http://mail.python.org/pipermail/python-dev/2004-May/044742.html I would use it, as an embedder, in my postgresql extension: http://gborg.postgresql.org/project/postgrespy/projdisplay.php Currently, I compile a little generator and snag the type(this is the same thing as from types import GeneratorType, I suppose), and will do so if I don't have direct access. I also access parts of the generator, currently using getattr, but if I had access to it, I would do it directly. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 20:30 Message: Logged In: YES user_id=21627 What is the purpose of this patch? ---------------------------------------------------------------------- Comment By: James William Pye (jwpye) Date: 2004-05-20 18:09 Message: Logged In: YES user_id=1044177 Ok, got it to compile, and tried out a simple generator; seems to be fine.. Probably should have asked first, but I'm assuming(yay) that it was desired to draw it inline with *object.[ch] organization protocol.. Only gotcha appears to be that genobject.c, of course, references eval_frame, so I just wrapped a call to it in eval.c with a non-static function(PyEval_EvaluateFrame()), this is probably not be what python-dev would like.. I mostly did it like this to get it to compile.(probably a bad idea, and should probably reference it directly. no need for an extra layer of function call overhead...) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957398&group_id=5470 From noreply at sourceforge.net Mon May 31 16:36:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 16:37:09 2004 Subject: [Patches] [ python-Patches-957240 ] Patch for feature request 491033 Message-ID: Patches item #957240, was opened at 2004-05-19 23:23 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957240&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Josiah Carlson (josiahcarlson) Assigned to: Nobody/Anonymous (nobody) Summary: Patch for feature request 491033 Initial Comment: The attached patch is to offer the functionality desired by feature request: python.org/sf/491033 On machines that use standard IEEE floating point doubles, the 'infinity' needs only to be 1e16 due to the limited precision of floating point. 1e309 is used for convenience sake, and will continue to be a precision infinity until 1024 bits of precision in floating point values is available. At such point in time where 1e309 is available in hardware, it will still sufficient due to the practical limitations of counting. ---------------------------------------------------------------------- >Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-31 13:36 Message: Logged In: YES user_id=341410 The attached patch includes both a documentation patch, as well as a modified loop patch to explicitly match the documentation. The patches are concatenated and in standard context diff format (as per the Python patch submission guidelines, sorry about the previous unified diff). The test suite does not seem to contain tests for asyncore. It does contain a test for asynchat, which will be unaffected by this patch. One thing to note: the use of 'count = count - 1' rather than 'count -= 1' is to remove potential bugs in the case where a mutable number-like object with a modifying __isub__ method were to be passed by a user. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 11:32 Message: Logged In: YES user_id=21627 This patch lacks changes to the documentation and the test suite. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-19 23:26 Message: Logged In: YES user_id=341410 Sorry about that, I generated the patch by hand and got the ordering messed up. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957240&group_id=5470 From noreply at sourceforge.net Mon May 31 16:59:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 16:59:16 2004 Subject: [Patches] [ python-Patches-957240 ] Patch for feature request 491033 Message-ID: Patches item #957240, was opened at 2004-05-20 08:23 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957240&group_id=5470 Category: None Group: None Status: Open Resolution: None >Priority: 6 Submitted By: Josiah Carlson (josiahcarlson) >Assigned to: Martin v. L?wis (loewis) Summary: Patch for feature request 491033 Initial Comment: The attached patch is to offer the functionality desired by feature request: python.org/sf/491033 On machines that use standard IEEE floating point doubles, the 'infinity' needs only to be 1e16 due to the limited precision of floating point. 1e309 is used for convenience sake, and will continue to be a precision infinity until 1024 bits of precision in floating point values is available. At such point in time where 1e309 is available in hardware, it will still sufficient due to the practical limitations of counting. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-31 22:36 Message: Logged In: YES user_id=341410 The attached patch includes both a documentation patch, as well as a modified loop patch to explicitly match the documentation. The patches are concatenated and in standard context diff format (as per the Python patch submission guidelines, sorry about the previous unified diff). The test suite does not seem to contain tests for asyncore. It does contain a test for asynchat, which will be unaffected by this patch. One thing to note: the use of 'count = count - 1' rather than 'count -= 1' is to remove potential bugs in the case where a mutable number-like object with a modifying __isub__ method were to be passed by a user. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 20:32 Message: Logged In: YES user_id=21627 This patch lacks changes to the documentation and the test suite. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-20 08:26 Message: Logged In: YES user_id=341410 Sorry about that, I generated the patch by hand and got the ordering messed up. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=957240&group_id=5470 From noreply at sourceforge.net Mon May 31 17:07:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 17:07:33 2004 Subject: [Patches] [ python-Patches-962502 ] East Asian Width support for Unicode Message-ID: Patches item #962502, was opened at 2004-05-29 00:59 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=962502&group_id=5470 Category: Core (C code) Group: None Status: Open Resolution: None Priority: 5 Submitted By: Hye-Shik Chang (perky) Assigned to: Nobody/Anonymous (nobody) Summary: East Asian Width support for Unicode Initial Comment: As David Goodger's inspiration, I thought that it would be great if we have some unicode methods that manipulates East Asian Width (http://www.unicode.org/reports/tr11/tr11-13.html#UCD). The attached patch implements rough first-time idea. >>> u'1'.iswide() False >>> u'\uac00'.iswide() True >>> u'\ud55c\uae00'.iswide() True >>> u'\ud55c\uae00'.width() 4 >>> u'ab\ud55c\uae00'.width() 6 >>> u'ab\ud55c\uae00'.iswide() False ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 23:07 Message: Logged In: YES user_id=21627 Updating to the Unicode 4.0 database is risky. It will break IDNA, which specifies that IDN must use the 3.2 version of the unicode database. It would be ok if you could arrange to provide both versions of the database. Ideally, the database would only store the deltas from 4.0 to 3.2, so we don't get any increase in space for cases where the data didn't change between Unicode versions. It might be reasonable to leave that issue alone for this patch, and proceed with the 3.2 version of EastAsianWidth. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=962502&group_id=5470 From noreply at sourceforge.net Mon May 31 19:21:34 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 19:21:53 2004 Subject: [Patches] [ python-Patches-963318 ] New module cookielib Message-ID: Patches item #963318, was opened at 2004-05-31 00:39 Message generated for change (Comment added) made by jjlee You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963318&group_id=5470 Category: Library (Lib) Group: Python 2.4 >Status: Open Resolution: Accepted Priority: 5 Submitted By: John J Lee (jjlee) Assigned to: Nobody/Anonymous (nobody) Summary: New module cookielib Initial Comment: cookielib itself is here: http://codespeak.net/svn/user/jjlee/wwwsearch/ClientCookie/branch/python-2.4_candidate/ All I'm uploading here are patches to the various other files in Python that are affected. Remaining issues: -The .unverifiable and .origin_req_host attributes I've added to urllib2.Request are a bit ugly (yet another bit of HTTP-specific state). But I guess it's too late to start having subclasses for specific protocol schemes. -Implementation wart: ._now attributes. -test_urllib2.py depends on test_cookielib.py. What's the right way to fix this? -Thread synchronization needs checking by someone with more of a clue about this: I have great confidence that it IS currently broken, since I've never tested it. -Update magic string in LWPCookieJar, if Gisle agrees (I'm waiting for a reply). ---------------------------------------------------------------------- >Comment By: John J Lee (jjlee) Date: 2004-06-01 00:21 Message: Logged In: YES user_id=261020 Here's a fix for the noisy unit tests. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 19:23 Message: Logged In: YES user_id=21627 Thanks for the patch. Applied as lib.tex 1.226 libcookie.tex 1.12 libcookielib.tex 1.1 liburllib2.tex 1.18 whatsnew24.tex 1.49 _LWPCookieJar.py 1.1 _MozillaCookieJar.py 1.1 cookielib.py 1.1 urllib2.py 1.67 test_cookielib.py 1.1 test_urllib2.py 1.13 NEWS 1.978 ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-31 18:09 Message: Logged In: YES user_id=261020 OK, done. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-31 15:20 Message: Logged In: YES user_id=261020 Martin, in case you are just about to apply this: I've finally stopped dithering about the first 'remaining issue' above (unverifiable and origin_req_host), and am just about to update liburllib2.tex.patch, urllib2.py.patch and test_urllib2.py.patch (and cookielib itself). ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2004-05-31 12:07 Message: Logged In: YES user_id=261020 The files that should be copied from the subversion repository are: _LWPCookieJar.py _MozillaCookieJar.py cookielib.py test/test_cookielib.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963318&group_id=5470 From noreply at sourceforge.net Mon May 31 19:36:08 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 19:36:11 2004 Subject: [Patches] [ python-Patches-963906 ] Unicode email address helper Message-ID: Patches item #963906, was opened at 2004-06-01 09:36 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963906&group_id=5470 Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Stuart Bishop (zenzen) Assigned to: Nobody/Anonymous (nobody) Summary: Unicode email address helper Initial Comment: Converting email addresses between Unicode and ASCII is non trivial, as three different encodings are used (RFC2047, IDNA and ASCII). Here is an EmailAddress I use and a test suite, which I feel should be integrated into the email package. I'm quite happy to implement a different interface if the 'unicode subclass' design is unsuitable, although I got no feedback from the Email-SIG so they are either happy with it or asleep ;) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=963906&group_id=5470 From noreply at sourceforge.net Mon May 31 19:39:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 19:39:59 2004 Subject: [Patches] [ python-Patches-921793 ] Rewrite of site.py Message-ID: Patches item #921793, was opened at 2004-03-23 07:05 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=921793&group_id=5470 Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Brett Cannon (bcannon) Assigned to: Nobody/Anonymous (nobody) Summary: Rewrite of site.py Initial Comment: Brought to you by the PyCon core sprint, I rewrote Lib/site.py to clean it up and modernize it. The biggest change is putting all the code into appropriate functions and defining a main() function that gets executed at import. Cleans up the code and allows others to run the functions after import in case they want the functionality. Also modernized it by using sets and universal newlines. A new test suite is also being included in this tracker item. You can partially run it against the original site.py, but some tests will fail since they call new functions. ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-05-31 16:39 Message: Logged In: YES user_id=357491 Re-generated the diff for site.py; turned out it didn't apply cleanly. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-03-23 09:44 Message: Logged In: YES user_id=357491 OK, now the test passes. Problem test was not matching what the code did by creating a normalized, absolute path of what was in sys.path . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-03-23 09:37 Message: Logged In: YES user_id=357491 Just discovered one of the regression tests fails if you run all the tests with regrtest.py. Trying to fix now since it is a bug in the test. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=921793&group_id=5470 From noreply at sourceforge.net Mon May 31 20:23:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 20:23:12 2004 Subject: [Patches] [ python-Patches-932933 ] doctest: add an option to end examples w/ dedent Message-ID: Patches item #932933, was opened at 2004-04-10 15:56 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=932933&group_id=5470 Category: Library (Lib) Group: Python 2.4 >Status: Closed >Resolution: Rejected Priority: 5 Submitted By: Edward Loper (edloper) >Assigned to: Tim Peters (tim_one) Summary: doctest: add an option to end examples w/ dedent Initial Comment: A current limitation of doctest is that it can't handle output containing blank lines, since it uses blank lines to detect the end of the example. This patch adds a new option, END_WITH_DEDENT, that gets around that problem by using a dedent to detect the end of examples instead. In particular, when END_WITH_DEDENT is specified, the end of each indented doctest example is marked by a dedent past its original prompt (>>>). Thus, blank lines may appear within the output. However, trailing blank lines are still disallowed. Unindented docstring examples continue to be termined by a blank line (since dedenting past the prompt is impossible). Here's an example use: def f(): r""" A doctest example with a blank line: >>> print 'One\n\nTwo' One Two Note that ending with dedents also means that we can skip the blank lines around doctest examples if we want: >>> print 1 1 This dedent signifies the end of the example; there's no need for a blank line. """ The END_WITH_DEDENT uses the same optionflags system that is already used by DONT_ACCEPT_TRUE_FOR_1. The patch was made avainst revision 1.33 of doctest.py. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-31 20:23 Message: Logged In: YES user_id=31435 Rejecting this one; let's focus on your other patch instead. ---------------------------------------------------------------------- Comment By: Edward Loper (edloper) Date: 2004-04-11 11:15 Message: Logged In: YES user_id=195958 This patch is probably mutually exclusive with patch #933238 [1], since they both have the same goal: to allow blank lines in doctest output. I thought of this solution first, but I think I prefer the other patch. Here's a comparison of pros and cons of this patch vs #933238. [-] it needs a special option to turn it on [-] it doesn't handles trailing blank lines [-] it will break external tools like epytext and restructuredtext that look for doctest blocks. [+] the user doesn't have to manually add blank line markers. ---------------------------------------------------------------------- Comment By: Edward Loper (edloper) Date: 2004-04-10 16:00 Message: Logged In: YES user_id=195958 If this patch looks good, I'd be happy to write a patch for the docs & test cases. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=932933&group_id=5470 From noreply at sourceforge.net Mon May 31 22:56:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 22:56:21 2004 Subject: [Patches] [ python-Patches-961465 ] New Misc/RPM/python-2.3.spec file Message-ID: Patches item #961465, was opened at 2004-05-27 20:14 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=961465&group_id=5470 Category: Build Group: Python 2.3 Status: Closed Resolution: Accepted Priority: 5 Submitted By: Sean Reifschneider (jafo) Assigned to: Nobody/Anonymous (nobody) Summary: New Misc/RPM/python-2.3.spec file Initial Comment: Includes patches to allow building on Red Hat 7.*/RHEL 2.*, and a bugfix in the fixing of /usr/local/bin/python references. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-06-01 12:56 Message: Logged In: YES user_id=29957 FWIW, I do plan a 2.3.5 after 2.4, as the final 2.3 release. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-06-01 05:41 Message: Logged In: YES user_id=21627 Ok, applied as python-2.3.spec 1.2.12.13 python-2.3.spec delete python-2.4.spec 1.1 Please review my changes in python-2.4.spec for correctness. ---------------------------------------------------------------------- Comment By: Sean Reifschneider (jafo) Date: 2004-06-01 04:53 Message: Logged In: YES user_id=81797 Yes, please. Please also commit it for 2.4, up the version from 2.3.3 to 2.4. Thanks, Sean ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-06-01 04:28 Message: Logged In: YES user_id=21627 There may not be any further Python 2.3 releases. Should I still commit the patch? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=961465&group_id=5470 From noreply at sourceforge.net Mon May 31 23:12:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 23:12:41 2004 Subject: [Patches] [ python-Patches-876130 ] add C API to datetime module Message-ID: Patches item #876130, was opened at 2004-01-13 10:20 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470 Category: Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Anthony Tuininga (atuining) >Assigned to: Tim Peters (tim_one) Summary: add C API to datetime module Initial Comment: The datetime module does not have a C API which means that modules written in C cannot take advantage of the datetime module without incurring significant overhead. These patches supply this lack and are based on the C API exported by the mxDateTime module and the cStringIO module and enhanced as suggested by Guido. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-31 23:12 Message: Logged In: YES user_id=31435 Assigned to me for the next review. Can't do it immediately, but will do my best to free up some time for it "soon". ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-26 15:39 Message: Logged In: YES user_id=619560 I would agree with your assessment of the issue regarding the TimeFromTicks() API and find it quite reasonable. Either implement both at the C and Python levels or not at all. Since there is no consensus on this, none it is. :-) Did you have a chance to review the actual code? The changes are fairly minimal but you might have questions about readability, names, etc. It would be great if this could be included in an upcoming 2.4 alpha/beta so that I can provide support in cx_Oracle for the next release. Thanks for your time. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-26 15:32 Message: Logged In: YES user_id=31435 Under the belief that the *intent* of this patch is to add a C API to Python's datetime module, it should stick to exposing C ways to spell what Python programmers can already do from Python using datetime. Since nothing like TimeFromTicks() exists in the Python datetime API, it simply doesn't make sense to invent one available only from C. It *may* make sense to invent one in a DB API wrapper around datetime, but I view that as out of scope for this patch. I would not add TimeFromTicks() to the Python datetime API either, because the docs are ambiguous. The sample implementation Marc-Andre posted here uses localtime() to resolve the ambiguity in the docs, but that has to be a wrong choice for some time-of-day apps. For example, there's apparently no portable way to spell "noon" using TimeFromTicks(): the value you get back depends on which time zone localtime() happens to use at the time it's called. If people want time-of-day from a POSIX timestamp, it remains easy to get it from what datetime already supplies, but to do so you have to be explicit about whether you want UTC or local time interpretation. Both are easily spelled already, and it's a Good Thing that you need to be explicit if you want to do such a thing. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 15:25 Message: Logged In: YES user_id=38388 Tim, please review. Thanks. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 15:22 Message: Logged In: YES user_id=38388 Thanks, but I don't have time to review it. As for the TimeFromTicks() API: I can understand that you don't feel like implementing it, but hey, I won't use datetime either. The only reason I'm trying to push full support of the DB API specs is that Guido et al. wanted to see a note in the DB API that datetime also provides a usable interface to do date/time interfacing. I'm not very interested in datetime myself, since I'll continue to use mxDateTime, so the situation for me is somewhat similar to yours: investing time into something that I myself will probably not use much (I will add support to mxODBC for those who like to use datetime instead of mxDateTime, but won't use it myself). Feel free to check in your changes. Thanks. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-26 12:30 Message: Logged In: YES user_id=619560 Attached are the modified patches based on your suggestions. I have tested them with my own module cx_Oracle and they appear to work correctly. Any further suggestions would be appreciated. I would like to see this included for Python 2.4 if at all possible. Please let me know what I need to do (if anything) to get this to happen. As for the TimeFromTicks() routine, Oracle has absolutely no use for a "time-only" data type and does not support it. It clearly does not have broad support so I really have no desire to implement it since I will never use it myself. If you consider it important, you could add it yourself or we could wait until someone else who requires it for their implementation of the DB API requires it. I'm not trying to be rude or trying to denigrate the DB API, simply trying to be practical. Please let me know if this is offensive in any way. Thanks. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-19 04:28 Message: Logged In: YES user_id=38388 >From the DB API specs: TimeFromTicks(ticks) This function constructs an object holding a time value from the given ticks value (number of seconds since the epoch; see the documentation of the standard Python time module for details). The reason for this having this API is to be able to construct an object suitable for passing to the database given a Unix ticks value. Since the ticks value contains both a date and a time part and most databases have a special column type for time value, we need two constructor APIs, one to extract just the date part and one for the time part. Here's the mxDateTime implementation for reference: def TimeFromTicks(ticks, # Locals: DateTimeDelta=DateTimeDelta,localtime=_time.localtime): """ TimeFromTicks(ticks) Constructs a DateTimeDelta instance pointing to the local time indicated by the given ticks value. The date part is ignored. """ return apply(DateTimeDelta, (0,) + localtime(ticks)[3:6]) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-03 13:46 Message: Logged In: YES user_id=31435 I don't understand what TimeFromTicks() is supposed to do. Seconds-from-the-epoch is a point in time, not a time-of- day. If some DB API requires guessing some transformation from seconds-from-the-epoch to time-of-day, that's fine, but it doesn't belong in Python's datetime module. The datetime module should certainly have a method to construct a datetime.datetime from a seconds-from-the- epoch argument -- seconds-from-the-epoch is a way of specifying a datetime.datetime. Given that, if the intent is to throw away the datetime.date portion of the datetime.datetime object, retaining only the datetime.time portion, then that's trivially accomplished by invoking dt.time () (where dt is the datetime.datetime object). Do any databases really store time-of-day as a seconds-from- the-epoch value? Google's top hit on TimeFromTicks() today happens to be a msg from Anthony saying that Oracle does not. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 13:24 Message: Logged In: YES user_id=38388 Macro: I don't know whether you need the macro or not (you probably do for the type checks). In any case, you can do without relying on Python providing you this information via the build process, since only datetimemodule.c is using the header file while Python is being built. TimeFromTicks(): I' ve stated my point. I don't think there's anything to add - I'm tired of fighting for these things... one of the reasons I stopped actively discussing things on python-dev. IMHO, a product that's aimed at developers should make things easy for the developer and try to avoid code duplication if possible. The API is needed in order to support database interfaces that use Unix ticks as basis for date/time, so if you don't add it to the module, the ten or twenty module authors out there will have to duplicate that piece of work in their implementation. Transition: This was already discussed on the db-api list. We love freedom of choice. Nothing to add. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 12:48 Message: Logged In: YES user_id=619560 About the Py_BUILD_CORE issue: You are suggesting that I define this in datetimemodule.c just prior to the include of datetime.h? Or are you suggesting some other macro? I am not sure about the comment about "not needing a macro in the first place" since there are two groups that will be importing datetime.h: the first being the datetime module itself and the second being the DB API modules. The macro is definitely required to distinguish between the two -- unless I am missing something? On the TimeFromTicks() issue: You obviously consider it important that the three methods be available at the C level for DB API module authors to use directly. My objection is simply a reiteration of an objection raised by Tim Peters. I think you'll have to argue that one with him. :-) I'll provide my implementations for cx_Oracle as part of the patches and then you can fight it out and I'll stay out of it. :-) As for the transition period, I assumed that with the introduction of a standard datetime module, that it would become the suggested implementation for database interfaace modules. That said, I don't really have any authority to say anything on this front so I'll leave that to you and others to decide. :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-03 11:26 Message: Logged In: YES user_id=38388 About the Py_BUILD_CORE issue: Sorry, I didn't know that Python does not define this macro for core modules. Nevertheless, looking at the datetime module code I don't really understand why you would need such a macro in the first place. Since the datetime module is the only code including the datetime.h header file it should be easy to add a macro definition just before importing datetime.h. On the TimeFromTicks() issue: The whole point of this discussion is to make the datetime module easily usable for authors of database modules. Since the DB API specifies that the module will need to export the three functions, two of which are available through the datetime module already, I don't really see your point in not wanting to add it. BTW, you are wrong in the assumption that the DB API spec will move away from a generic API for date/time objects. The DB API has always been defined in terms of interfaces and does not force a certain set of tools onto the developer. The datetime module will become one possible choice for DB API authors, others will want to stick to mxDateTime, use strings, ticks integers or more specific objects for interfacing. That won't change, so there is no "transition period". ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-05-03 10:40 Message: Logged In: YES user_id=619560 I spent some time getting the two include files merged together, only to discover that Py_BUILD_CORE is __NOT__ defined when building modules, only when building stuff linked directly into the core interpreter. Perhaps this ought to be defined when building the modules with setup.py? At any rate, I cannot proceed along the route suggested without this problem being resolved. On the DateFromTicks(), TimestampFromTicks(), TimeFromTicks() issue, I'm not sure that there is much benefit to including a C API for this since if the datetime module becomes the standard method for managing datetime objects, there is no point in having the DB API modules provide something that the datetime module already provides, right? During the transition period the few lines of code needed can be posted. If you disagree I can certainly provide the few lines as part of the patch but I suspect you will find resistance to having this included by Tim Peters, the maintainer (I believe) of the datetime module. Feel free to correct me... :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-04-26 12:19 Message: Logged In: YES user_id=38388 Sorry for not getting back to you earlier: I didn't see you last message. 1-4) Python defines a macro during the Python build process that is not defined when compiling extensions: Py_BUILD_CORE You can use that macro to compile things differently for core things vs. extensions. 5) Fair enough; I don't see a point in creating a new numbering for each and every module. Please just use 1, 2, 3, 4, ... 6) Hope this makes sense :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-26 11:35 Message: Logged In: YES user_id=619560 Any chance to look at this? Any hope of this being included in Python 2.4? I know a number of people (including Guido) who would appreciate that.... :-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-04-05 18:04 Message: Logged In: YES user_id=619560 I am back at work after finishing my move so I think I can finally devote a little more time to this issue. 1-4) The main reason for datetimeapi.h as opposed to simply modifying datetime.h was that the internal build needs to define things __differently__ from an external build of a module and there doesn't appear to be any way of fixing that -- so if you know a way I think I can manage to create a reasonable patch; otherwise, I'm out of my depth! 5) Guido was suggesting that a new PyIMPORT macro be made available which would verify that the magic number defined in the header file actually matches the magic number exported by the built module; this is simply a developer protection scheme which should eliminate really stupid uses; you can ask him about this or I can e-mail you part of the thread that brought this up 6) Makes sense. I'd like to get the questions above resolved first before proceeding any further, though. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 13:23 Message: Logged In: YES user_id=38388 Thanks for the comments... 1) Please put *all* the code for the C API into datetimeapi.h. I don't like your current mix of putting some things into datetime.h and others into datetimeapi.h (e.g. the PyDateTime_CAPI definition). 2) Since it's the only API missing if you want to use datetime objects in database interfacing, please add it. 3) dito. 4) Yes, the header file is the right place. Have a look at unicodeobject.h for what I have in mind (or mxDateTime.h for that matter). 5) Why add yet another magic number ? Isn't the Python API number enough ? 6) Since you don't provide a doc patch, please put comments into the header file. There can never be enough documentation and developers tend to look at the code rather than the docs ;-) ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-03-25 13:08 Message: Logged In: YES user_id=619560 Sorry for the delay. Things here have been quite busy recently both at work and at home. Let me see if I can answer your questions. 1) are you suggesting comments that include the macro names in datetimeapi.h similar to what I put in the comments below? Or something else? See modified datetimeapi.h for more info. 2) TimeFromTicks() is a very DB API specific routine. In chatting with others about the datetime module, they were not keen on adding such a beast. I guess there will have to be some discussion about whether datetimeapi.h is also a sort of half implementation for a C module implementing the DB API. I was intending to do this in my cx_Oracle module but not in datetimeapi.h. Could you clarify? 3) I could add C APIS for the three DB API ticks interfaces but I was simply intending to do that in cx_Oracle. Again, see #2. I can do it either way. :-) 4) I have added limited documentation in datetimeapi.h but is that really the right place for it? Is it even remotely close to what is needed? I'm not sure what is needed here exactly. 5) the API magic number is simply an arbitrary number which is stored in datetime.h and need not be understood by the implementor. For your information I simply took the name "datetime", converted each letter to its ordinal value in the alphabet, modded by 16 and used that hex character -- does that make any sense? This can be changed to whatever makes sense, though. 6) Whether or not datetimeapi.h should be sufficient for a developer or whether he should look at documentation in the usual locations (html pages, for example) I'll leave up to the Python development team. Let me know how I can assist. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-25 12:35 Message: Logged In: YES user_id=38388 Anthony, have you had a chance to look at the comments ? ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 10:06 Message: Logged In: YES user_id=619560 Oops! It looks like my own misunderstanding of SourceForge and how it works is to blame. :-( I'll take a look at this and get back to you as soon as possible -- ignore my previous comment. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-02-26 10:03 Message: Logged In: YES user_id=619560 You haven't yet responded to my previous comments -- it looks like SourceForge put them __after__ your comments so it is quite understandable why you didn't notice them. If you can give me the answers to those questions or provide additional questions that I need to answer, then I think we can make progress again. I was waiting for __you__ :-) ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-02-25 18:08 Message: Logged In: YES user_id=38388 Any progress on this ? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-28 06:49 Message: Logged In: YES user_id=38388 Thanks for the clarification. I had forgotten about the macros -- I'd suggest that you document them in the datetimeapi.h include file to not cause the same confusion on the developer side (the trick to achieve adoption is to make things easy on the developer). As for TimeFromTicks(): This should be rather straight forward to implement: you take the time part of the ticks timestamp and create a time object from it. This is the way the DB API constructor works. Could you add C APIs for the three DB API ticks interface methods ? Other things that are missing: * documentation of the way the CAPI object can be used in datetimeapi.h * documentation of the various CAPI entries * documentation of the API magic number (there should be some scheme for this) I think that datetimeapi.h should be enough for a developer to read in order to use the interface. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 16:02 Message: Logged In: YES user_id=619560 I didn't think any additional API would be necessary for extracting date time information since the following macros are available: PyDateTime_GET_YEAR(o) PyDateTime_GET_MONTH(o) PyDateTime_GET_DAY(o) PyDateTime_DATE_GET_HOUR(o) PyDateTime_DATE_GET_MINUTE(o) PyDateTime_DATE_GET_SECOND(o) PyDateTime_DATE_GET_MICROSECOND(o) PyDateTime_TIME_GET_HOUR(o) PyDateTime_TIME_GET_MINUTE(o) PyDateTime_TIME_GET_SECOND(o) PyDateTime_TIME_GET_MICROSECOND(o) Were you thinking of something else that is needed? As for interfacing at the C level for converting from Unix ticks, the following method is already available and could simply be used as a drop in replacement for DateFromTicks() and TimestampFromTicks() from the DB API document. DateFromTicks() --> datetime.date.fromtimestamp() TimestampFromTicks() --> datetime.datetime.fromtimestamp() TimeFromTicks() is not already exposed but discussions with Tim Peters already suggested that would not happen since the concept is rather strange. You'll have to discuss that with him if you disagree! :-) Any comments? ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-01-26 15:46 Message: Logged In: YES user_id=38388 This is a good start. However, you should add more APIs for extracting date/time information to the C API. mxDateTime.h from egenix-mx-base can provide a good basis for this. If you want to make the datetime module usable for database interfacing at C level, you'll also have to add interfaces for Unix ticks conversion, since these are often used by database C level interfaces. ---------------------------------------------------------------------- Comment By: Anthony Tuininga (atuining) Date: 2004-01-26 13:36 Message: Logged In: YES user_id=619560 I have no objection to providing patches for doc and test. A quick look at _testcapimodule.c doesn't provide any obvious ideas as to how to patch it. I looked for cStringIO which has a C API already and found nothing in there at all. The documentation also simply says "see the source for the information you require". Any suggestions as to how to proceed? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-01-25 22:17 Message: Logged In: YES user_id=31435 Marc-Andre, can you review this? I think you have more experience building C APIs for modules than anyone else (while I don't have any). There's certainly no objection to giving datetime a C API, and Guido already said he doesn't want to rely on that Martin changed datetime to be built as part of the core (in 2.3 on Windows, datetime.pyd was a distinct DLL; in CVS, datetime is compiled into the core DLL now). Anthony, you probably need to add doc and test patches. _testcapimodule.c holds tests of things you can only get at from C. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=876130&group_id=5470