From noreply at sourceforge.net Fri Oct 1 06:19:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 1 06:19:22 2004 Subject: [ python-Bugs-780725 ] Compile error messages and PEP-263 Message-ID: Bugs item #780725, was opened at 2003-07-31 18:11 Message generated for change (Comment added) made by ishimoto You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=780725&group_id=5470 Category: Parser/Compiler Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: atsuo ishimoto (ishimoto) Assigned to: Nobody/Anonymous (nobody) Summary: Compile error messages and PEP-263 Initial Comment: If a source file contains encoding definition and it has syntax errors, Python prints line of source file in utf-8. Here's simple example. # -*- coding: Shift_JIS -*- print "XXXXX" abcdefg # "XXXXX" is Japanese ShiftJIS string. With this source file, Python prints a second line as a part of error message in utf-8 encoding. Lines printed here should be reconverted into original encoding. Also, error column that '^' character marks is wrong. ---------------------------------------------------------------------- >Comment By: atsuo ishimoto (ishimoto) Date: 2004-10-01 13:19 Message: Logged In: YES user_id=463672 I filed a patch for this bug. http://sourceforge.net/tracker/? func=detail&aid=1031213&group_id=5470&atid=305470 Any chance this bug to be fixed in Python 2.4? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=780725&group_id=5470 From noreply at sourceforge.net Fri Oct 1 12:42:22 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 1 12:42:24 2004 Subject: [ python-Bugs-1038410 ] silent installation on windows: no drive in registry values Message-ID: Bugs item #1038410, was opened at 2004-10-01 12:42 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038410&group_id=5470 Category: Installation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Matteo Gallivanoni (sigmud) Assigned to: Nobody/Anonymous (nobody) Summary: silent installation on windows: no drive in registry values Initial Comment: I used to install python 2.2 with the option /s (silent) on windows (win 2k server). Now I migrated to python 2.3 (Python-2.3.4.exe) and it seems to me that in all the registry values the drive is missing EG: under HKEY_CLASSES_ROOT\Python.NoConFile\DefaultIcon The (Default) values I found is \Python23\Py.ico instead of the one that I have if I performed a non-silent installation C:\Python23\Py.ico (likewise it is C:\Python22\Py.ico in the silent python 22 installation) TIA! ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038410&group_id=5470 From noreply at sourceforge.net Fri Oct 1 14:00:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 1 14:00:38 2004 Subject: [ python-Bugs-1038410 ] silent installation on windows: no drive in registry values Message-ID: Bugs item #1038410, was opened at 2004-10-01 12:42 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038410&group_id=5470 Category: Installation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Matteo Gallivanoni (sigmud) >Assigned to: Tim Peters (tim_one) Summary: silent installation on windows: no drive in registry values Initial Comment: I used to install python 2.2 with the option /s (silent) on windows (win 2k server). Now I migrated to python 2.3 (Python-2.3.4.exe) and it seems to me that in all the registry values the drive is missing EG: under HKEY_CLASSES_ROOT\Python.NoConFile\DefaultIcon The (Default) values I found is \Python23\Py.ico instead of the one that I have if I performed a non-silent installation C:\Python23\Py.ico (likewise it is C:\Python22\Py.ico in the silent python 22 installation) TIA! ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-10-01 14:00 Message: Logged In: YES user_id=11105 IIRC, this change was made for good reasons by Tim. Maybe he still knows the details? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038410&group_id=5470 From noreply at sourceforge.net Fri Oct 1 18:56:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 1 18:56:08 2004 Subject: [ python-Bugs-1038591 ] Python 2.3+ socket._fileobject handles EAGAIN with data loss Message-ID: Bugs item #1038591, was opened at 2004-10-01 11:56 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038591&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jon Nelson (jnelson) Assigned to: Nobody/Anonymous (nobody) Summary: Python 2.3+ socket._fileobject handles EAGAIN with data loss Initial Comment: If one has a non-blocking socket, 'sock', and uses 'sock.makefile()' to create file-like object out of it, there exists the possibility (even probability) of data loss in the event of an EAGAIN error. The code that comprises _fileobject in socket does not try to catch EAGAIN. In the event that the read() operation does not encounter EOF, but *has* read data (and then gets an EAGAIN) such as when there is less than 8KB of data to read, the socket.error traceback is handed back to the caller of read and the read data is discarded. This is horrible. The _fileobject read routine (and probably write, too) should catch socket.error and check it for EAGAIN, and in that case, return what it has already read (if anything). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038591&group_id=5470 From noreply at sourceforge.net Fri Oct 1 19:27:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 1 19:27:14 2004 Subject: [ python-Bugs-1038410 ] silent installation on windows: no drive in registry values Message-ID: Bugs item #1038410, was opened at 2004-10-01 06:42 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038410&group_id=5470 Category: Installation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Matteo Gallivanoni (sigmud) >Assigned to: Nobody/Anonymous (nobody) Summary: silent installation on windows: no drive in registry values Initial Comment: I used to install python 2.2 with the option /s (silent) on windows (win 2k server). Now I migrated to python 2.3 (Python-2.3.4.exe) and it seems to me that in all the registry values the drive is missing EG: under HKEY_CLASSES_ROOT\Python.NoConFile\DefaultIcon The (Default) values I found is \Python23\Py.ico instead of the one that I have if I performed a non-silent installation C:\Python23\Py.ico (likewise it is C:\Python22\Py.ico in the silent python 22 installation) TIA! ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-01 13:27 Message: Logged In: YES user_id=31435 The Wise system variable MAINDIR had "C:" hardcoded before revision 1.123 of python20.wse. Rev 1.123 removed the "C:", because Wise uses the value of MAINDIR to suggest the default install directory in a GUI install, and at least two users complained that they wanted the default to be their system drive, for which whatever reasons they changed to be something other than C -- and they complained endlessly about how hard it was to select the drive they wanted instead from the dropdown list. Since I couldn't find a way to get Wise to tell us what "the system drive" is (Wise appeared to assume it *must* be C), and since they said that removing the hardcoded "C:" from MAINDIR caused the GUI install to do what they wanted, it was removed. It would be foolish to put it back. So unless someone who has a better understanding of Wise can figure out how to get it to reveal the system drive, and cares enough to take over Python's Wise installer, it won't change. Perhaps some form of the installer's /d switch would allow setting MAINDIR during a silent install (I don't know): http://www.i386.com/?k=3&p=1013 ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-10-01 08:00 Message: Logged In: YES user_id=11105 IIRC, this change was made for good reasons by Tim. Maybe he still knows the details? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038410&group_id=5470 From noreply at sourceforge.net Fri Oct 1 19:36:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 1 19:36:10 2004 Subject: [ python-Bugs-1038410 ] silent installation on windows: no drive in registry values Message-ID: Bugs item #1038410, was opened at 2004-10-01 12:42 Message generated for change (Settings changed) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038410&group_id=5470 Category: Installation Group: Python 2.3 >Status: Closed >Resolution: Wont Fix Priority: 5 Submitted By: Matteo Gallivanoni (sigmud) Assigned to: Nobody/Anonymous (nobody) Summary: silent installation on windows: no drive in registry values Initial Comment: I used to install python 2.2 with the option /s (silent) on windows (win 2k server). Now I migrated to python 2.3 (Python-2.3.4.exe) and it seems to me that in all the registry values the drive is missing EG: under HKEY_CLASSES_ROOT\Python.NoConFile\DefaultIcon The (Default) values I found is \Python23\Py.ico instead of the one that I have if I performed a non-silent installation C:\Python23\Py.ico (likewise it is C:\Python22\Py.ico in the silent python 22 installation) TIA! ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-10-01 19:36 Message: Logged In: YES user_id=11105 Closing as won't fix. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-01 19:27 Message: Logged In: YES user_id=31435 The Wise system variable MAINDIR had "C:" hardcoded before revision 1.123 of python20.wse. Rev 1.123 removed the "C:", because Wise uses the value of MAINDIR to suggest the default install directory in a GUI install, and at least two users complained that they wanted the default to be their system drive, for which whatever reasons they changed to be something other than C -- and they complained endlessly about how hard it was to select the drive they wanted instead from the dropdown list. Since I couldn't find a way to get Wise to tell us what "the system drive" is (Wise appeared to assume it *must* be C), and since they said that removing the hardcoded "C:" from MAINDIR caused the GUI install to do what they wanted, it was removed. It would be foolish to put it back. So unless someone who has a better understanding of Wise can figure out how to get it to reveal the system drive, and cares enough to take over Python's Wise installer, it won't change. Perhaps some form of the installer's /d switch would allow setting MAINDIR during a silent install (I don't know): http://www.i386.com/?k=3&p=1013 ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-10-01 14:00 Message: Logged In: YES user_id=11105 IIRC, this change was made for good reasons by Tim. Maybe he still knows the details? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038410&group_id=5470 From noreply at sourceforge.net Fri Oct 1 20:25:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 1 20:25:48 2004 Subject: [ python-Bugs-1038591 ] Python 2.3+ socket._fileobject handles EAGAIN with data loss Message-ID: Bugs item #1038591, was opened at 2004-10-01 11:56 Message generated for change (Comment added) made by jnelson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038591&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jon Nelson (jnelson) Assigned to: Nobody/Anonymous (nobody) Summary: Python 2.3+ socket._fileobject handles EAGAIN with data loss Initial Comment: If one has a non-blocking socket, 'sock', and uses 'sock.makefile()' to create file-like object out of it, there exists the possibility (even probability) of data loss in the event of an EAGAIN error. The code that comprises _fileobject in socket does not try to catch EAGAIN. In the event that the read() operation does not encounter EOF, but *has* read data (and then gets an EAGAIN) such as when there is less than 8KB of data to read, the socket.error traceback is handed back to the caller of read and the read data is discarded. This is horrible. The _fileobject read routine (and probably write, too) should catch socket.error and check it for EAGAIN, and in that case, return what it has already read (if anything). ---------------------------------------------------------------------- >Comment By: Jon Nelson (jnelson) Date: 2004-10-01 13:25 Message: Logged In: YES user_id=8446 An improvement would be to allow EAGAIN to surface only if it happens as the /first/ read. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038591&group_id=5470 From noreply at sourceforge.net Fri Oct 1 22:26:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 1 22:26:58 2004 Subject: [ python-Bugs-1038693 ] bad link in Tkinter docs Message-ID: Bugs item #1038693, was opened at 2004-10-01 13:26 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038693&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Brett Cannon (bcannon) Assigned to: Nobody/Anonymous (nobody) Summary: bad link in Tkinter docs Initial Comment: http://docs.python.org/lib/node639.html has a dead link pointing to the "Packer class interface". Is that supposed to be pointing to the next page, "Packer options"? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038693&group_id=5470 From noreply at sourceforge.net Sat Oct 2 06:16:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 2 06:16:06 2004 Subject: [ python-Bugs-513866 ] Float/long comparison anomaly Message-ID: Bugs item #513866, was opened at 2002-02-06 18:33 Message generated for change (Comment added) made by arkoenig You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=513866&group_id=5470 Category: Python Interpreter Core Group: None Status: Closed Resolution: Fixed Priority: 5 Submitted By: Andrew Koenig (arkoenig) Assigned to: Tim Peters (tim_one) Summary: Float/long comparison anomaly Initial Comment: Comparing a float and a long appears to convert the long to float and then compare the two floats. This strategy is a problem because the conversion might lose precision. As a result, == is not an equivalence relation and < is not an order relation. For example, it is possible to create three numbers a, b, and c such that a==b, b==c, and a!=c. ---------------------------------------------------------------------- >Comment By: Andrew Koenig (arkoenig) Date: 2004-10-02 04:16 Message: Logged In: YES user_id=418174 Thank you! This is seriously cool... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-09-23 08:08 Message: Logged In: YES user_id=31435 I'm on vacation, so I fixed this . The changes are so messy I doubt it will get backported to the 2.3 line, though (I'm not going to try -- someone else would need to volunteer for that). Lib/test/test_long.py 1.25 Misc/NEWS 1.1142 Objects/floatobject.c 2.133 ---------------------------------------------------------------------- Comment By: paul rubin (phr) Date: 2002-02-17 16:22 Message: Logged In: YES user_id=72053 It looks like the complication is not in finding an algorithm but rather in fitting it into the implementation. I'm not at all sure this is right, but glancing at the code, the comparison seems to happen in the function try_3way_compare in Objects/object.c, which calls PyNumber_CoerceEx if the types aren't equal. PyNumber_CoerceEx ends up calling float_coerce on x,n which "promotes" n to float, similar to what happens when you do mixed arithmetic (like x+n). My guess is that a suitable patch would go into try_3way_compare to specially notice when you're comparing a float and a long, and avoid the coercion. I'm unfamiliar enough with the implementation that I'd probably take a while to get it right, and still possibly end up forgetting to update a refcount or something, leading to leaked memory or mysterious crashes later. Anyway, no, this isn't real important to me, at least at the moment. It just wasn't clear whether there was any difficulty figuring out a useable algorithm. ---------------------------------------------------------------------- Comment By: Andrew Koenig (arkoenig) Date: 2002-02-17 15:46 Message: Logged In: YES user_id=418174 I think there's a slightly more straightforward algorithm than the one that Paul Rubin (phr) suggested. Again, assume that x is a float and n is a long. We note first that the comparison is trivial unless x and n are both nonzero and have the same sign. We will therefore assume in the rest of this discussion that x and n are strictly positive; the case where they are negative is analogous. Every floating-point implementation has many numbers with the property that the least significant bit in those numbers' representations has a value of 1. In general, if the floating-point representation has k bits, then any integer in the range [2**(k-1),2**k) qualifies. Let K be any of these numbers; it doesn't matter which one. Precompute K and store it in both float and long form. This computation is exact because K is an integer that has an exact representation in floating-point form. It is now possible to compare x with K and n with K exactly, without conversion, because we already have K exactly in both forms. If x < K and n >= K, then x < n and we're done. If x > K and n <= K, then x > n and we're done. Otherwise, x and n are on the same side of K (possibly being equal to K). If x >= K and n >= K, then the LSB of x is at least 1, so we can convert x to long without losing information. Therefore, cmp(x, n) is cmp(long(x), n). If x <= K and n <= K, then then n is small enough that it has an exact representation as a float. Therefore cmp(x, n) is cmp(x, float(n)). So I don't think there's any profound algorithmic problem here. Unfortunately, I don't know enough about the details of how comparison is implemented to be willing to try my hand at a patch. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2002-02-17 15:31 Message: Logged In: YES user_id=31435 Paul, this isn't an intellectual challenge -- I expect any numerical programmer of ordinary skill could write code to compare a float to a long delivering the mathematically sensible result. There are several ways to do it. Adding "me too" votes doesn't change the priority. How about taking a whack at writing a patch if this is important to you? It's so low on the list of PythonLabs priorities I doubt I'll ever get to it (which is why I unassigned myself: an unassigned bug report is looking for someone to fix it, not a cheerleader <wink>). ---------------------------------------------------------------------- Comment By: paul rubin (phr) Date: 2002-02-17 14:58 Message: Logged In: YES user_id=72053 Oops, I got confused about the order of the two args in the example below. I meant cmp(x,n) in the description and cmp(xl, n) in the code, rather than having n first. Anyway you get the idea. Now I should go back to bed ;-). ---------------------------------------------------------------------- Comment By: paul rubin (phr) Date: 2002-02-17 14:43 Message: Logged In: YES user_id=72053 I hope there's a simple solution to this--it's obvious what the right result should be mathematically if you compare 1L<<10000 with 0.0. It should not raise an error. If the documented behavior leads to raising an error, then there's a bug in the document. I agree that it's not the highest priority bug in the world, but it doesn't seem that complicated. If n is a long and x is a float, both >= 0, what happens if you do this, to implement cmp(n,x): xl = long(x) # if x has a fraction part and int part is == n, then x>n if float(xl)!=x and xl==n: return 1 return cmp(n, xl) If both are < 0, change 1 to -1 above. If x and n are of opposite sign, the positive one is greater. Unless I missed something (which is possible--I'm not too alert right now) the above should be ok in all cases. Basically you use long as the common type to convert to; you do lose information when converting a non-integer, but for the comparison with an integer, you don't need the lost information other than knowing whether it was nonzero, which you find out by converting the long back to a float. ---------------------------------------------------------------------- Comment By: Andrew Koenig (arkoenig) Date: 2002-02-09 15:42 Message: Logged In: YES user_id=418174 I completely agree it's not a high-priority item, especially because it may be complicated to fix. I think that the fundamental problem is that there is no common type to which both float and long can be converted without losing information, which complicates both the definition and implementation of comparison. Accordingly, it might make sense to think about this issue in conjunction with future consideration of rational numbers. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2002-02-09 07:33 Message: Logged In: YES user_id=31435 I reopened this, but unassigned it since I can't justify working on it (the benefit/cost ratio of fixing it is down in the noise compared to other things that should be done). I no longer think we'd need a PEP to change the behavior, and agree it would be nice to change it. Changing it may surprise people expecting Python to work like C (C99 says that when integral -> floating conversion is in range but can't be done exactly, either of the closest representable floating numbers may be returned; Python inherits the platform C's behavior here for Python int -> Python float conversion (C long -> C double); when the conversion is out of range, C doesn't define what happens, and Python inherits that too before 2.2 (Infinities and NaNs are what I've seen most often, varying by platform); in 2.2 it raises OverflowError). I'm not sure it's possible for a<b and b<c and a==c, unless the platform C is inconsistent (meaning that (double)i for a fixed i returns the next-lowest double on some calls but the next-higher on others). This brute-force searcher didn't turn up any examples on my box: f = 2L**53 - 5 # straddle the representable-as-double limit nums = [f+i for i in range(50)] nums.extend(map(float, nums)) for a in nums: . for b in nums: . if not a < b: . continue . for c in nums: . if not b < c: . continue . if a >= c: . print `a`, `b`, `c` ---------------------------------------------------------------------- Comment By: Andrew Koenig (arkoenig) Date: 2002-02-07 15:33 Message: Logged In: YES user_id=418174 Here is yet another surprise: x=[1L<10000] y=[0.0] z=x+y Now I can execute x.sort() and y.sort() successfully, but z.sort blows up. ---------------------------------------------------------------------- Comment By: Andrew Koenig (arkoenig) Date: 2002-02-07 13:28 Message: Logged In: YES user_id=418174 The difficulty is that as defined, < is not an order relation, because there exist values a, b, c such that a<b, b==c, and a==c. I believe that there also exist values such that a<b, b<c, and a==c. Under such circumstances, it is hard to understand how sort can work properly, whicn is my real concern. Do you really want to warn people that they shouldn't sort lists containing floats and longs? Moreover, it is not terribly difficult to define the comparisons so that == is an equivalence relation and < is an order relation. The idea is that for any floating-point system, there is a threshold T such that if x is a float value >=T, converting x to long will not lose information, and if x is a long value <=T, converting x to float will not lose information. Therefore, instead of always converting to long, it suffices to convert in a direction chosen by comparing the operands to T (without conversion) first. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2002-02-07 04:59 Message: Logged In: YES user_id=31435 Oops! I meant """ could lead to a different result than the explicit coercion in somefloat == float(somelong) """ ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2002-02-07 04:52 Message: Logged In: YES user_id=31435 Since the coercion to float is documented and intended, it's not "a bug" (it's functioning as designed), although you may wish to argue for a different design, in which case making an incompatible change would first require a PEP and community debate. Information loss in operations involving floats comes with the territory, and I don't see a reason to single this particular case out as especially surprising. OTOH, I expect it would be especially surprising to a majority of users if the implicit coercion in somefloat == somelong could lead to a different result than the explicit coercion in long(somefloat) == somelong Note that the "long" type isn't unique here: the same is true of mixing Python ints with Python floats on boxes where C longs have more bits of precision than C doubles (e.g., Linux for IA64, and Crays). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=513866&group_id=5470 From noreply at sourceforge.net Sat Oct 2 13:40:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 2 13:40:55 2004 Subject: [ python-Bugs-1038935 ] Typo "signiture" in HTML generated by DocXMLRPCServer Message-ID: Bugs item #1038935, was opened at 2004-10-02 13:40 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038935&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Malte Helmert (maltehelmert) Assigned to: Nobody/Anonymous (nobody) Summary: Typo "signiture" in HTML generated by DocXMLRPCServer Initial Comment: There is a typo in the docstring of SimpleXMLRPCServer.system_methodSignature, where the word "signature" reads "signiture". If this method is exported by a DocXMLRPCServer using register_introspection_functions, the wrong text appears on the generated HTML content. This is with Python 2.4a3. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038935&group_id=5470 From noreply at sourceforge.net Sat Oct 2 20:12:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 2 21:02:56 2004 Subject: [ python-Bugs-1039092 ] apply.__doc__ says "Deprecated since release 2.3" Message-ID: Bugs item #1039092, was opened at 2004-10-02 14:12 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039092&group_id=5470 Category: Documentation Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Jp Calderone (kuran) Assigned to: Nobody/Anonymous (nobody) Summary: apply.__doc__ says "Deprecated since release 2.3" Initial Comment: This is confusing, since using apply does not log a DeprecationWarning, nor a PendingDeprecationWarning, even in Python 2.4. I think "Deprecated since release 2.3" should be removed or replaced with text indicating that apply() is simply not preferred by all people. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039092&group_id=5470 From noreply at sourceforge.net Sat Oct 2 23:04:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 2 23:04:25 2004 Subject: [ python-Bugs-854102 ] Message.get_filename() problem Message-ID: Bugs item #854102, was opened at 2003-12-04 10:26 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854102&group_id=5470 Category: Python Library Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Ervin N?meth (nemethe) Assigned to: Barry A. Warsaw (bwarsaw) Summary: Message.get_filename() problem Initial Comment: There are obviously e-mails with non-standard encodings (who wants to join me hating pine?) See attached backtrace. A local patch of - return unicode(newvalue[2], newvalue[0] or 'us-ascii') + try: + return unicode(newvalue[2], newvalue[0] or 'us-ascii') + except LookupError: + return unicode(newvalue[2], 'us-ascii', 'replace') solved the problem here. ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-10-02 17:04 Message: Logged In: YES user_id=12800 This will be fixed in email 3.0/Python 2.4. ---------------------------------------------------------------------- Comment By: Ervin N?meth (nemethe) Date: 2004-05-12 10:28 Message: Logged In: YES user_id=768522 Barry, sorry but it is 6 month since the report. The mail causing the problem is long lost. You could try contacting Gregor Maier who commented on mailman bug 803729 recently. He is at http://sourceforge.net/users/maiergre/ ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-10 10:50 Message: Logged In: YES user_id=12800 Please attach a sample test case to illustrate the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854102&group_id=5470 From noreply at sourceforge.net Sun Oct 3 00:33:44 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 3 00:33:49 2004 Subject: [ python-Bugs-1039092 ] apply.__doc__ says "Deprecated since release 2.3" Message-ID: Bugs item #1039092, was opened at 2004-10-02 13:12 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039092&group_id=5470 Category: Documentation Group: Python 2.4 >Status: Closed >Resolution: Wont Fix Priority: 5 Submitted By: Jp Calderone (kuran) Assigned to: Nobody/Anonymous (nobody) Summary: apply.__doc__ says "Deprecated since release 2.3" Initial Comment: This is confusing, since using apply does not log a DeprecationWarning, nor a PendingDeprecationWarning, even in Python 2.4. I think "Deprecated since release 2.3" should be removed or replaced with text indicating that apply() is simply not preferred by all people. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-02 17:33 Message: Logged In: YES user_id=80475 The function is in-fact deprecated (albiet at a glacial rate waiting for Py3.0). Rather that being "not preferred", it should not be used at all. The f(*args, **kwds) approach is now the one way to do it. In the docs, apply() was moved to a section for "things you don't need to learn about but are still there." We removed the warning in Py2.4 because it was doing more harm than good. IOW, we can deprecate it without making people's lives miserable. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039092&group_id=5470 From noreply at sourceforge.net Sun Oct 3 01:00:08 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 3 01:00:14 2004 Subject: [ python-Feature Requests-1023290 ] proposed struct module format code addition Message-ID: Feature Requests item #1023290, was opened at 2004-09-06 15:42 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1023290&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Josiah Carlson (josiahcarlson) Assigned to: Nobody/Anonymous (nobody) Summary: proposed struct module format code addition Initial Comment: I believe there should be a mechanism to load and unload arbitrarily large integers via the struct module. Currently, one would likely start with the 'Q' format character, creating the integer in a block-wise fashion with multiplies and shifts. This is OK, though it tends to lend itself to certain kinds of bugs. There is currently another method for getting large integers from strings and going back without the struct module: long(stri.encode('hex'), 16) hex(inte)[2:].decode('hex') Arguably, such things shouldn't be done for the packing and unpacking of binary data in general (the string slicing especially). I propose a new format character for the struct module, specifically because the struct module is to "Interpret strings as packed binary data". Perhaps 'g' and 'G' (eg. biGint) is sufficient, though any reasonable character should suffice. Endianness should be handled, and the number of bytes representing the object would be the same as with the 's' formatting code. That is, '>60G' would be an unsigned big-endian integer represented by 60 bytes (null filled if the magnitude of the passed integer is not large enough). The only reason why one wouldn't want this functionality in the struct module is "This module performs conversions between Python values and C structs represented as Python strings." and arbitrarily large integers are not traditionally part of a C struct (though I am sure many of us have implemented arbitrary precision integers with structs). The reason "not a C type" has been used to quash the 'bit' and 'nibble' format character, because "masks and shifts" are able to emulate them, and though "masks and shifts" could also be used here, I have heard myself and others state that there should be an easy method for converting between large longs and strings. A side-effect for allowing arbitrarily large integers to be represented in this fashion is that its functionality could, if desired, subsume the other integer type characters, as well as fill in the gaps for nonstandard size integers (3, 5, 6, 7 etc. byte integers), that I (and I am sure others) have used in various applications. Currently no implementation exists, and I don't have time to do one now. Having taken a look at longobject.c and structmodule.c, I would likely be able to make a patch to the documentation, structmodule.c, and test_struct.py around mid October, if this functionality is desireable to others and accepted. While I doubt that a PEP for this is required, if necessary I would write one up with a sample implementation around mid October. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-02 18:00 Message: Logged In: YES user_id=80475 If no one other that the OP supports this, I would like to reject putting this in the struct module. Initially, it seemed like a fit because the endian options and whatnot are already in place; however, in one way or another each of the posters except the OP has stated good reasons for it not being in the struct module. Variable length C structure members are not what the module is about. Having to know the length in advance of the call is a killer. The learning curve issues with struct are also a problem. And, the use cases jsut don't point to struct. Put in a simple function in binascii or let's drop it. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-10-02 17:34 Message: Logged In: YES user_id=341410 I have just attached a unified diff against structmodule.c 2.62 in CVS. It implements the semantics I have been describing, compiles cleanly, and produces proper results. >>> pickle.encode_long(83726) '\x0eG\x01' >>> struct.pack('<3g', 83726) '\x0eG\x01' >>> struct.unpack('<3g', struct.pack('<3g', 83726)) (83726L,) If the functionality is accepted, I will submit diffs for test_struct.py and libstruct.tex . ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-16 13:11 Message: Logged In: YES user_id=341410 Curse you Tim, for your core Python experience *wink*. Pickle is one example where a pascal-like encoding of longs was an encoding decision made to be flexible and space efficient. Certainly we have disparate use cases. Mine is for fixed struct-like records with multiple types. With pickle, any /thought/ of fixed records are tossed out the window with variable-lengthed types like strings, longs, lists, tuples and dicts, and I believe aren't really comparable. Now, variable-lengthed longs packed in little-endian format already have a mechanism for encoding and decoding via pickle.en/decode_long (though it is wholly undocumented), and seemingly is going to get another in binascii. Fixed-lengthed, optional signed/unsigned, optional little-endian/big-endian longs do not have a mechanism for encoding and decoding, which is what I am asking for. I will point out that 128 bit integers are gaining support on newer 32 and 64 bit processors and C compilers for them (SSE on x86, Itanium, etc.). In the future, a new code for these 128 bit integers may be asked for inclusion. With a variable-width integer type, all future "hey, we now have x-byte types in C, where is struct support in Python?", can be answered with the proposed, "choose your integer size" format code. That is to say, this format code is future proof, unless integer types start wandering from integer byte widths. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-09-15 21:52 Message: Logged In: YES user_id=31435 Use cases are important. Oddly(?) enough, I've never had a need for a bigint conversion in a C struct, and have a hard time imagining I will someday. All the cases I've had (and I've had more than a few) were one-shot str->long or long->str conversions. An obvious example in the core is the tedious encode_long() and decode_long() functions in pickle.py. Note that a pickle.encode_long() workalike doesn't know in advance how many bytes it needs, which would make using struct a PITA for that particular use case. If a proposal isn't convenient for taking over existing conversions of this nature, that counts against it. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-15 21:31 Message: Logged In: YES user_id=341410 And as I stated, the 's' format character is also a "variable lengthed type". It just so happens that in most use cases I've had and observed for both the 's' format AND proposed 'g' format, the type size, is in fact, fixed at 'compile' time. It also happens that for the 'g' format, this fixed size is not in the set {1,2,4,8}, which are not limitations for the pre-existing 's' format. Please note that the only fundamental difference between the pre-existing 's' format and the proposed 'g' format, is that of a quick call to appropriate PyLong_* functions, and a range check as required by other integer types. Python is a tool. Struct is a tool. By changing the tool only slightly, we can add flexibility. The code is already there, minor glue would make it work, and would make it convenient for, I believe, more people than binascii. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-15 19:26 Message: Logged In: YES user_id=80475 I agree with Michael and Martin that variable length types do not belong in struct. The module is about working with fixed record layouts. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-15 19:17 Message: Logged In: YES user_id=341410 Raymond, from your first post on this topic, it seems as though you were previously implementing this functionality in binascii for some particular reason, and it seems as though it is to be included with binascii in the future, regardless of the outcome of this particular feature request. The only reason the binascii solution is better than status quo, is because a user doesn't need to implement arbitrarily large integer packing and unpacking themselves. On the other hand, it still requires the user make manual binascii.str_to_long(str_obj) calls in the case of it being part of a struct, so doesn't gain significantly. Now, one of the reasons why I requested a format code addition was because one can (un)pack multiple data types simultaneously with a single function call via struct. In nearly all of the use cases I have for packing and unpacking large integers, they are a part of other structures. In the cases where I have been packing and unpacking single integers, floats, etc., I still use struct because it is has nearly all of the functionality I need (signed, unsigned, big endian, little endian, char, short, long, long long, etc., lacking only arbitrarily large integer packing and unpacking). ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-15 17:08 Message: Logged In: YES user_id=80475 My vote is for binascii functions to parallel hexlify and unhexlify. Ideally, it would give the same result as long(hexlify(s), 16). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-15 13:53 Message: Logged In: YES user_id=341410 As you state, it already supports packing and unpacking a variable-lengthed type: strings. In the use cases I've had and seen for (un)packing strings with struct, it is the most common to define a static format code, and use that all the time. That is, you see things like ">HLLHB25s", that become string constants in a module. On the _very rare_ occasion where people want more flexibility in their types, I have seen both the use of fixed and variable pascal strings... def packit(arg1, arg2, arg3, strng): return struct.pack(">LHH%ip"%len(strng), arg1, arg2, arg3, strng) I would not expect any pascal-string-like packing of a large integer, though it is possible. I do expect that most people have similar use cases as I, and would pre-define their struct formatting code. In the case of other similar requests (long to string, string to long via a base256 representation, etc.) for use in cryptography, I expect that the regularity of structures used in cryptography would almost certainly result in formatting codes being module constants. To sum up, both in the case for the 's' and 'p' format codes, and the proposed 'g'/'G' formatting codes, the vast majority of use cases pre-define the length of the string and large integer on a per-structure basis via "25s", "25p", or "25g". Rarely are the lengths truely variable in the case of "%ip"%len(strng). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-15 12:03 Message: Logged In: YES user_id=6656 Oops, I see you actually address that (ish). But I still feel packing what is an essentially variable length type using the struct module is a bit strange. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-15 12:02 Message: Logged In: YES user_id=6656 Josiah, what do you suggest struct.calcsize does with the format code your proposing? I think this question encapsulates why I find this feature request a bit misdirected. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-12 13:55 Message: Logged In: YES user_id=341410 Hexlify and unhexlify make sense for translating strings. Nowhere in the binascii module is there any mention of translating anything to/from integers, longs or floats. The only reason (un)hexlify make any sense at all is because we can get integers from hexlified strings, and get strings from hexlified integers with relative ease. I guess the trick with struct, at least with me, is not that I use it because it translates to/from C types, it is because it translates to/from types that I find useful. Its intersection with C types, as well as Python's intersection with C types, and my use intersection with the types is a convenient (but very engineered and understandable) coincidence. It would be another very convenient (but also engineered *wink*) coincidence if I didn't have to first extract a section of data, then translate it, in order to get one large integer. In the cases that I would truely find useful, big integers are a part of what would be called structs in the C world, and wouldn't require additional processing over what I'm already doing for other integers and floats. I was looking around, and it turns out that in 2001, Paul Rubin requested that one be able to translate to/from arbitrary bases via the C-level format function. In that discussion, Paul made the case that there should be a method to get arbitrarily long integers to and from strings: "The struct module doesn't give any way of converting arbitrary ints (meaning longs) to binary. Really, it's needed. Some people do it with gmpy, but if Python is going to support longs as a built-in type, one shouldn't have to resort to 3rd-party modules to read and write them in binary." Guido followed up with: "OK, I believe you. Can you submit a patch?" It seems like this was in reference to being able to use functions in binascii for converting to/from arbitrary packed binary integer types in base 256 (http://sourceforge.net/tracker/?func=detail&atid=105470&aid=465045&group_id=5470 if you are interested). That request seems to have died because Paul dropped the ball. Me, I would prefer struct to binascii, if only because the code for doing this is already waiting to be used in struct, and because you can pull multiple objects from a single packed binary string, rather than one object per call. This would seemingly also satisfy complaints of being able to translate to/from base 256 for arbitrarily large integers. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-09-11 22:40 Message: Logged In: YES user_id=31435 binascii makes sense because that's where the hexlify and unhexlify functions live, which are small conceptual steps away from what's needed here. Methods on numbers make sense too, and only seem strange because so few are clearly visible now (although, e.g., there are lots of them already, like number.__abs__ and number.__add__). The struct module makes sense too, although it would be darned ugly to document a refusal to accept the new codes in "native" mode; and struct has a high learning curve; and struct obviously never intended to support types that aren't supplied directly by C compilers (the "Pascal string" code seems weird now, but not at the time). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-10 18:18 Message: Logged In: YES user_id=341410 (sorry it took me a few days to get back to you, I am on a contract deadline crunch...just taking a break now) The *HTTPServer heirarchy is interesting in its own right, but really, each piece in the heirarchy adds functionality. A similar thing can be said of asyncore and all the modules that derive from it (asynchat, *HTTPServer, *XMLRPCServer, smtpd, etc.). In this case, since the struct module is already in C and the functions are not subclassable, creating another module that parses strings and sends pieces off to struct for actual decoding seems like a waste of a module, especially when the change is so minor. Now, binascii is being used in such a fashion by uu and binhex, but that is because binascii is the data processing component, where uu and binhex make a 'pretty' interface. Struct doesn't need a pretty interface, it is already pretty. Though as I have said before, I think it could use this small addition. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-09-08 17:53 Message: Logged In: YES user_id=21627 Since you were asking: it is quite common that modules refer to related functionality. For example, BaseHTTPServer refers to SimpleHTTPServer and CGIHTTPServer. One might expect that a HTTP server also supports files and does CGI - but not this one; go elsewhere. Likewise, module binascii refers to modules uu and binhex. The math documentation points out that it does not support complex numbers, and that cmath is needed. The audioop documentation gives the function echocancel in the documentation, instead of implementing it. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-08 17:38 Message: Logged In: YES user_id=341410 Martin, I was typing as you submitted your most recent comment. I am honestly shocked that you would suggest that longs should gain a method for encoding themselves as binary strings. Such a thing would then suggest that standard ints and floats also gain such methods. It would also imply that since one can go to strings, one should equivalently be able to come from strings via equivalent methods. Goodness, int.tostring(width) and int.fromstring(str)? But what about endianness? Looks like a big can of worms to me. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-08 17:03 Message: Logged In: YES user_id=341410 Structures (aka C Structs) can contain arbitrarily large or small numbers of basic types inside them. As such, 'single long values' are still a valid use. I use struct for packing and unpacking of single items (8,4,2 byte integers, 1 byte integers are faster served via chr and ord) when necessary (because it is the most convenient), as well as a current contract where it is not uncommon to be packing and unpacking 256 byte structs. Those large structs contains various 1,2,4 and 8 byte integers, as well as a handful of 16 and 20 byte integers (which I must manually shift and mask during packing and unpacking). I'm a big boy, and can do it, but that doesn't mean that such functionality should be left out of Python. As for 'document the approach of going through hex inside the documentation of the struct module', I am curious about whether other modules do the same thing, that is to tell users "this functionality conceptually fits here X%, which is why it is documented here, but because it does not fit 100%, here is how you can do the same thing, which will likely look like a strange hack, require slicing potentially large strings, and be significantly slower than if we had just added the functionality, but here you go anyways." Now, I don't /need/ the feature, but I believe myself and others would find it useful. I also don't /require/ it be in struct, but no other modules offer equivalent functionality; Pickle and Marshal are Python-only, binascii (and bin2hex) are for converting between binary and ascii representations for transferring over non-8-bit channels (email, web, etc.), and no other module even comes close to offering a similar bit of "packs various types into a binary format, the same way C would" as struct. If anyone has a better place for it, I'm all ears (or eyes). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-09-08 16:45 Message: Logged In: YES user_id=21627 I would think that def long_as_bytes(lvalue, width): fmt = '%%.%dx' % (2*width) return unhexlify(fmt % (lvalue & ((1L<<8*width)-1))) is short enough for a recipe to not really make a C function necessary for that feature. However, if they are going to be provided somewhere, I would suggest that static methods on the long type might be the right place. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-08 14:30 Message: Logged In: YES user_id=80475 The idea is to expose the _PyLong_FromByteArray() and _PyLong_AsByteArray() functions. While long(hexlify(b),16) is doable for bin2long, going the other way is not so simple. I agree that these are not struct related. Originally, I proposed the binascii module because one of the operations is so similar to hexlify(). As there other suggestions? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-09-08 14:15 Message: Logged In: YES user_id=21627 Apparently, the point of this request is that the method for converting long ints to binary should be "easily found in documentation". And also apparently, the submitter thinks that the struct module would be the place where people look. Now, that allows for a simple solution: document the approach of going through hex inside the documentation of the struct module. There is one other reason (beyond being primarily for C APIs) why such a feature should *not* be in the struct module: The struct module, most naturally, is about structures. However, I understand that the intended usage of this feature would not be structures, but single long values. Therefore, I consider it counter-intuitive to extend struct for that. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-06 19:02 Message: Logged In: YES user_id=80475 Okay, submit a patch with docs and unittests. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-06 18:44 Message: Logged In: YES user_id=341410 As I provide in the feature request, there is already a method for translating string <-> long. The problem with current methods for converting between large integers and strings is that they do not lend themselves to generally being understandable or to being documented. The struct module already provides two appropriate functions for handling packed binary data, a place for documenting functions involving packing and unpacking binary data, and whose implementation seems to be simple enough (one more format character, much of which borrowed from 's' character, and a call to _PyLong_FromByteArray seems to be sufficient). As for the binascii module, many of the functions listed seem like they should be wrapped into the encode/decode string methods, hexlify already being so in str.encode('hex'). To me, just being able to translate doesn't seem sufficient (we already can translate), but being able to do it well, have it documented well, and placed in a location that is obvious, fast and optimized for these kinds of things seems to be the right thing. >From what I can tell, the only reason why struct doesn't already have an equivalent format character to the proposed 'g' and 'G', is because the module was created to handle packed C structs and seemingly "nothing else". Considering there doesn't seem to be any other reasonable or easily documentable location for placing equivalent functionality (both packing and unpacking), I am of the opinion that restricting the packing and unpacking to C types in the struct module (when there are other useful types) is overkill. As I said, I will provide an implementation if desired. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-06 17:34 Message: Logged In: YES user_id=80475 FWIW, I'm working str/long conversion functions for the binascii module. Will that suit your needs? The tolong function is equivalent to: long(hexlify(b), 16) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1023290&group_id=5470 From noreply at sourceforge.net Sun Oct 3 00:34:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 3 02:31:26 2004 Subject: [ python-Feature Requests-1023290 ] proposed struct module format code addition Message-ID: Feature Requests item #1023290, was opened at 2004-09-06 13:42 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1023290&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Josiah Carlson (josiahcarlson) Assigned to: Nobody/Anonymous (nobody) Summary: proposed struct module format code addition Initial Comment: I believe there should be a mechanism to load and unload arbitrarily large integers via the struct module. Currently, one would likely start with the 'Q' format character, creating the integer in a block-wise fashion with multiplies and shifts. This is OK, though it tends to lend itself to certain kinds of bugs. There is currently another method for getting large integers from strings and going back without the struct module: long(stri.encode('hex'), 16) hex(inte)[2:].decode('hex') Arguably, such things shouldn't be done for the packing and unpacking of binary data in general (the string slicing especially). I propose a new format character for the struct module, specifically because the struct module is to "Interpret strings as packed binary data". Perhaps 'g' and 'G' (eg. biGint) is sufficient, though any reasonable character should suffice. Endianness should be handled, and the number of bytes representing the object would be the same as with the 's' formatting code. That is, '>60G' would be an unsigned big-endian integer represented by 60 bytes (null filled if the magnitude of the passed integer is not large enough). The only reason why one wouldn't want this functionality in the struct module is "This module performs conversions between Python values and C structs represented as Python strings." and arbitrarily large integers are not traditionally part of a C struct (though I am sure many of us have implemented arbitrary precision integers with structs). The reason "not a C type" has been used to quash the 'bit' and 'nibble' format character, because "masks and shifts" are able to emulate them, and though "masks and shifts" could also be used here, I have heard myself and others state that there should be an easy method for converting between large longs and strings. A side-effect for allowing arbitrarily large integers to be represented in this fashion is that its functionality could, if desired, subsume the other integer type characters, as well as fill in the gaps for nonstandard size integers (3, 5, 6, 7 etc. byte integers), that I (and I am sure others) have used in various applications. Currently no implementation exists, and I don't have time to do one now. Having taken a look at longobject.c and structmodule.c, I would likely be able to make a patch to the documentation, structmodule.c, and test_struct.py around mid October, if this functionality is desireable to others and accepted. While I doubt that a PEP for this is required, if necessary I would write one up with a sample implementation around mid October. ---------------------------------------------------------------------- >Comment By: Josiah Carlson (josiahcarlson) Date: 2004-10-02 15:34 Message: Logged In: YES user_id=341410 I have just attached a unified diff against structmodule.c 2.62 in CVS. It implements the semantics I have been describing, compiles cleanly, and produces proper results. >>> pickle.encode_long(83726) '\x0eG\x01' >>> struct.pack('<3g', 83726) '\x0eG\x01' >>> struct.unpack('<3g', struct.pack('<3g', 83726)) (83726L,) If the functionality is accepted, I will submit diffs for test_struct.py and libstruct.tex . ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-16 11:11 Message: Logged In: YES user_id=341410 Curse you Tim, for your core Python experience *wink*. Pickle is one example where a pascal-like encoding of longs was an encoding decision made to be flexible and space efficient. Certainly we have disparate use cases. Mine is for fixed struct-like records with multiple types. With pickle, any /thought/ of fixed records are tossed out the window with variable-lengthed types like strings, longs, lists, tuples and dicts, and I believe aren't really comparable. Now, variable-lengthed longs packed in little-endian format already have a mechanism for encoding and decoding via pickle.en/decode_long (though it is wholly undocumented), and seemingly is going to get another in binascii. Fixed-lengthed, optional signed/unsigned, optional little-endian/big-endian longs do not have a mechanism for encoding and decoding, which is what I am asking for. I will point out that 128 bit integers are gaining support on newer 32 and 64 bit processors and C compilers for them (SSE on x86, Itanium, etc.). In the future, a new code for these 128 bit integers may be asked for inclusion. With a variable-width integer type, all future "hey, we now have x-byte types in C, where is struct support in Python?", can be answered with the proposed, "choose your integer size" format code. That is to say, this format code is future proof, unless integer types start wandering from integer byte widths. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-09-15 19:52 Message: Logged In: YES user_id=31435 Use cases are important. Oddly(?) enough, I've never had a need for a bigint conversion in a C struct, and have a hard time imagining I will someday. All the cases I've had (and I've had more than a few) were one-shot str->long or long->str conversions. An obvious example in the core is the tedious encode_long() and decode_long() functions in pickle.py. Note that a pickle.encode_long() workalike doesn't know in advance how many bytes it needs, which would make using struct a PITA for that particular use case. If a proposal isn't convenient for taking over existing conversions of this nature, that counts against it. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-15 19:31 Message: Logged In: YES user_id=341410 And as I stated, the 's' format character is also a "variable lengthed type". It just so happens that in most use cases I've had and observed for both the 's' format AND proposed 'g' format, the type size, is in fact, fixed at 'compile' time. It also happens that for the 'g' format, this fixed size is not in the set {1,2,4,8}, which are not limitations for the pre-existing 's' format. Please note that the only fundamental difference between the pre-existing 's' format and the proposed 'g' format, is that of a quick call to appropriate PyLong_* functions, and a range check as required by other integer types. Python is a tool. Struct is a tool. By changing the tool only slightly, we can add flexibility. The code is already there, minor glue would make it work, and would make it convenient for, I believe, more people than binascii. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-15 17:26 Message: Logged In: YES user_id=80475 I agree with Michael and Martin that variable length types do not belong in struct. The module is about working with fixed record layouts. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-15 17:17 Message: Logged In: YES user_id=341410 Raymond, from your first post on this topic, it seems as though you were previously implementing this functionality in binascii for some particular reason, and it seems as though it is to be included with binascii in the future, regardless of the outcome of this particular feature request. The only reason the binascii solution is better than status quo, is because a user doesn't need to implement arbitrarily large integer packing and unpacking themselves. On the other hand, it still requires the user make manual binascii.str_to_long(str_obj) calls in the case of it being part of a struct, so doesn't gain significantly. Now, one of the reasons why I requested a format code addition was because one can (un)pack multiple data types simultaneously with a single function call via struct. In nearly all of the use cases I have for packing and unpacking large integers, they are a part of other structures. In the cases where I have been packing and unpacking single integers, floats, etc., I still use struct because it is has nearly all of the functionality I need (signed, unsigned, big endian, little endian, char, short, long, long long, etc., lacking only arbitrarily large integer packing and unpacking). ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-15 15:08 Message: Logged In: YES user_id=80475 My vote is for binascii functions to parallel hexlify and unhexlify. Ideally, it would give the same result as long(hexlify(s), 16). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-15 11:53 Message: Logged In: YES user_id=341410 As you state, it already supports packing and unpacking a variable-lengthed type: strings. In the use cases I've had and seen for (un)packing strings with struct, it is the most common to define a static format code, and use that all the time. That is, you see things like ">HLLHB25s", that become string constants in a module. On the _very rare_ occasion where people want more flexibility in their types, I have seen both the use of fixed and variable pascal strings... def packit(arg1, arg2, arg3, strng): return struct.pack(">LHH%ip"%len(strng), arg1, arg2, arg3, strng) I would not expect any pascal-string-like packing of a large integer, though it is possible. I do expect that most people have similar use cases as I, and would pre-define their struct formatting code. In the case of other similar requests (long to string, string to long via a base256 representation, etc.) for use in cryptography, I expect that the regularity of structures used in cryptography would almost certainly result in formatting codes being module constants. To sum up, both in the case for the 's' and 'p' format codes, and the proposed 'g'/'G' formatting codes, the vast majority of use cases pre-define the length of the string and large integer on a per-structure basis via "25s", "25p", or "25g". Rarely are the lengths truely variable in the case of "%ip"%len(strng). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-15 10:03 Message: Logged In: YES user_id=6656 Oops, I see you actually address that (ish). But I still feel packing what is an essentially variable length type using the struct module is a bit strange. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-15 10:02 Message: Logged In: YES user_id=6656 Josiah, what do you suggest struct.calcsize does with the format code your proposing? I think this question encapsulates why I find this feature request a bit misdirected. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-12 11:55 Message: Logged In: YES user_id=341410 Hexlify and unhexlify make sense for translating strings. Nowhere in the binascii module is there any mention of translating anything to/from integers, longs or floats. The only reason (un)hexlify make any sense at all is because we can get integers from hexlified strings, and get strings from hexlified integers with relative ease. I guess the trick with struct, at least with me, is not that I use it because it translates to/from C types, it is because it translates to/from types that I find useful. Its intersection with C types, as well as Python's intersection with C types, and my use intersection with the types is a convenient (but very engineered and understandable) coincidence. It would be another very convenient (but also engineered *wink*) coincidence if I didn't have to first extract a section of data, then translate it, in order to get one large integer. In the cases that I would truely find useful, big integers are a part of what would be called structs in the C world, and wouldn't require additional processing over what I'm already doing for other integers and floats. I was looking around, and it turns out that in 2001, Paul Rubin requested that one be able to translate to/from arbitrary bases via the C-level format function. In that discussion, Paul made the case that there should be a method to get arbitrarily long integers to and from strings: "The struct module doesn't give any way of converting arbitrary ints (meaning longs) to binary. Really, it's needed. Some people do it with gmpy, but if Python is going to support longs as a built-in type, one shouldn't have to resort to 3rd-party modules to read and write them in binary." Guido followed up with: "OK, I believe you. Can you submit a patch?" It seems like this was in reference to being able to use functions in binascii for converting to/from arbitrary packed binary integer types in base 256 (http://sourceforge.net/tracker/?func=detail&atid=105470&aid=465045&group_id=5470 if you are interested). That request seems to have died because Paul dropped the ball. Me, I would prefer struct to binascii, if only because the code for doing this is already waiting to be used in struct, and because you can pull multiple objects from a single packed binary string, rather than one object per call. This would seemingly also satisfy complaints of being able to translate to/from base 256 for arbitrarily large integers. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-09-11 20:40 Message: Logged In: YES user_id=31435 binascii makes sense because that's where the hexlify and unhexlify functions live, which are small conceptual steps away from what's needed here. Methods on numbers make sense too, and only seem strange because so few are clearly visible now (although, e.g., there are lots of them already, like number.__abs__ and number.__add__). The struct module makes sense too, although it would be darned ugly to document a refusal to accept the new codes in "native" mode; and struct has a high learning curve; and struct obviously never intended to support types that aren't supplied directly by C compilers (the "Pascal string" code seems weird now, but not at the time). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-10 16:18 Message: Logged In: YES user_id=341410 (sorry it took me a few days to get back to you, I am on a contract deadline crunch...just taking a break now) The *HTTPServer heirarchy is interesting in its own right, but really, each piece in the heirarchy adds functionality. A similar thing can be said of asyncore and all the modules that derive from it (asynchat, *HTTPServer, *XMLRPCServer, smtpd, etc.). In this case, since the struct module is already in C and the functions are not subclassable, creating another module that parses strings and sends pieces off to struct for actual decoding seems like a waste of a module, especially when the change is so minor. Now, binascii is being used in such a fashion by uu and binhex, but that is because binascii is the data processing component, where uu and binhex make a 'pretty' interface. Struct doesn't need a pretty interface, it is already pretty. Though as I have said before, I think it could use this small addition. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-09-08 15:53 Message: Logged In: YES user_id=21627 Since you were asking: it is quite common that modules refer to related functionality. For example, BaseHTTPServer refers to SimpleHTTPServer and CGIHTTPServer. One might expect that a HTTP server also supports files and does CGI - but not this one; go elsewhere. Likewise, module binascii refers to modules uu and binhex. The math documentation points out that it does not support complex numbers, and that cmath is needed. The audioop documentation gives the function echocancel in the documentation, instead of implementing it. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-08 15:38 Message: Logged In: YES user_id=341410 Martin, I was typing as you submitted your most recent comment. I am honestly shocked that you would suggest that longs should gain a method for encoding themselves as binary strings. Such a thing would then suggest that standard ints and floats also gain such methods. It would also imply that since one can go to strings, one should equivalently be able to come from strings via equivalent methods. Goodness, int.tostring(width) and int.fromstring(str)? But what about endianness? Looks like a big can of worms to me. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-08 15:03 Message: Logged In: YES user_id=341410 Structures (aka C Structs) can contain arbitrarily large or small numbers of basic types inside them. As such, 'single long values' are still a valid use. I use struct for packing and unpacking of single items (8,4,2 byte integers, 1 byte integers are faster served via chr and ord) when necessary (because it is the most convenient), as well as a current contract where it is not uncommon to be packing and unpacking 256 byte structs. Those large structs contains various 1,2,4 and 8 byte integers, as well as a handful of 16 and 20 byte integers (which I must manually shift and mask during packing and unpacking). I'm a big boy, and can do it, but that doesn't mean that such functionality should be left out of Python. As for 'document the approach of going through hex inside the documentation of the struct module', I am curious about whether other modules do the same thing, that is to tell users "this functionality conceptually fits here X%, which is why it is documented here, but because it does not fit 100%, here is how you can do the same thing, which will likely look like a strange hack, require slicing potentially large strings, and be significantly slower than if we had just added the functionality, but here you go anyways." Now, I don't /need/ the feature, but I believe myself and others would find it useful. I also don't /require/ it be in struct, but no other modules offer equivalent functionality; Pickle and Marshal are Python-only, binascii (and bin2hex) are for converting between binary and ascii representations for transferring over non-8-bit channels (email, web, etc.), and no other module even comes close to offering a similar bit of "packs various types into a binary format, the same way C would" as struct. If anyone has a better place for it, I'm all ears (or eyes). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-09-08 14:45 Message: Logged In: YES user_id=21627 I would think that def long_as_bytes(lvalue, width): fmt = '%%.%dx' % (2*width) return unhexlify(fmt % (lvalue & ((1L<<8*width)-1))) is short enough for a recipe to not really make a C function necessary for that feature. However, if they are going to be provided somewhere, I would suggest that static methods on the long type might be the right place. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-08 12:30 Message: Logged In: YES user_id=80475 The idea is to expose the _PyLong_FromByteArray() and _PyLong_AsByteArray() functions. While long(hexlify(b),16) is doable for bin2long, going the other way is not so simple. I agree that these are not struct related. Originally, I proposed the binascii module because one of the operations is so similar to hexlify(). As there other suggestions? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-09-08 12:15 Message: Logged In: YES user_id=21627 Apparently, the point of this request is that the method for converting long ints to binary should be "easily found in documentation". And also apparently, the submitter thinks that the struct module would be the place where people look. Now, that allows for a simple solution: document the approach of going through hex inside the documentation of the struct module. There is one other reason (beyond being primarily for C APIs) why such a feature should *not* be in the struct module: The struct module, most naturally, is about structures. However, I understand that the intended usage of this feature would not be structures, but single long values. Therefore, I consider it counter-intuitive to extend struct for that. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-06 17:02 Message: Logged In: YES user_id=80475 Okay, submit a patch with docs and unittests. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-06 16:44 Message: Logged In: YES user_id=341410 As I provide in the feature request, there is already a method for translating string <-> long. The problem with current methods for converting between large integers and strings is that they do not lend themselves to generally being understandable or to being documented. The struct module already provides two appropriate functions for handling packed binary data, a place for documenting functions involving packing and unpacking binary data, and whose implementation seems to be simple enough (one more format character, much of which borrowed from 's' character, and a call to _PyLong_FromByteArray seems to be sufficient). As for the binascii module, many of the functions listed seem like they should be wrapped into the encode/decode string methods, hexlify already being so in str.encode('hex'). To me, just being able to translate doesn't seem sufficient (we already can translate), but being able to do it well, have it documented well, and placed in a location that is obvious, fast and optimized for these kinds of things seems to be the right thing. >From what I can tell, the only reason why struct doesn't already have an equivalent format character to the proposed 'g' and 'G', is because the module was created to handle packed C structs and seemingly "nothing else". Considering there doesn't seem to be any other reasonable or easily documentable location for placing equivalent functionality (both packing and unpacking), I am of the opinion that restricting the packing and unpacking to C types in the struct module (when there are other useful types) is overkill. As I said, I will provide an implementation if desired. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-06 15:34 Message: Logged In: YES user_id=80475 FWIW, I'm working str/long conversion functions for the binascii module. Will that suit your needs? The tolong function is equivalent to: long(hexlify(b), 16) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1023290&group_id=5470 From noreply at sourceforge.net Sun Oct 3 05:44:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 3 05:44:52 2004 Subject: [ python-Bugs-1039270 ] time zone tests fail on Windows Message-ID: Bugs item #1039270, was opened at 2004-10-03 12:44 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Nobody/Anonymous (nobody) Summary: time zone tests fail on Windows Initial Comment: Following tests fail on Win 2K(Japanese locale): # test_strptime.py test_compile (__main__.TimeRETests) ... FAIL test_bad_timezone (__main__.StrptimeTests) ... ERROR test_timezone (__main__.StrptimeTests) ... ERROR test_day_of_week_calculation (__main__.CalculationTests) ... ERROR test_gregorian_calculation (__main__.CalculationTests) ... ERROR test_julian_calculation (__main__.CalculationTests) ... ERROR # test_time.py test_strptime (test.test_time.TimeTestCase) ... FAIL === They all stem from time zone tests and can be divided into two groups: FAIL of test_compile is basically same as #bug 883604. http://www.python.org/sf/883604 Local time values include regular expression's metacharacters, but they are not escaped. The rest is caused because strptime can't parse the values of strftime. >>> import time >>> time.tzname ('\x93\x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)', '\x93 \x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)') >>> time.strptime(time.strftime('%Z', time.gmtime())) Traceback (most recent call last): File "", line 1, in -toplevel- time.strptime(time.strftime('%Z', time.gmtime())) File "C:\Python24\lib\_strptime.py", line 291, in strptime raise ValueError("time data did not match format: data=%s fmt=%s" % ValueError: time data did not match format: data=東京 (標準時) fmt=%a %b %d %H:%M:%S %Y The output of running test_time.py and test_strptime.py is attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 From noreply at sourceforge.net Sun Oct 3 08:54:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 3 08:54:44 2004 Subject: [ python-Bugs-1039270 ] time zone tests fail on Windows Message-ID: Bugs item #1039270, was opened at 2004-10-02 22:44 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: George Yoshida (quiver) >Assigned to: Brett Cannon (bcannon) Summary: time zone tests fail on Windows Initial Comment: Following tests fail on Win 2K(Japanese locale): # test_strptime.py test_compile (__main__.TimeRETests) ... FAIL test_bad_timezone (__main__.StrptimeTests) ... ERROR test_timezone (__main__.StrptimeTests) ... ERROR test_day_of_week_calculation (__main__.CalculationTests) ... ERROR test_gregorian_calculation (__main__.CalculationTests) ... ERROR test_julian_calculation (__main__.CalculationTests) ... ERROR # test_time.py test_strptime (test.test_time.TimeTestCase) ... FAIL === They all stem from time zone tests and can be divided into two groups: FAIL of test_compile is basically same as #bug 883604. http://www.python.org/sf/883604 Local time values include regular expression's metacharacters, but they are not escaped. The rest is caused because strptime can't parse the values of strftime. >>> import time >>> time.tzname ('\x93\x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)', '\x93 \x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)') >>> time.strptime(time.strftime('%Z', time.gmtime())) Traceback (most recent call last): File "", line 1, in -toplevel- time.strptime(time.strftime('%Z', time.gmtime())) File "C:\Python24\lib\_strptime.py", line 291, in strptime raise ValueError("time data did not match format: data=%s fmt=%s" % ValueError: time data did not match format: data=東京 (標準時) fmt=%a %b %d %H:%M:%S %Y The output of running test_time.py and test_strptime.py is attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 From noreply at sourceforge.net Sun Oct 3 10:52:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 3 10:52:15 2004 Subject: [ python-Bugs-1036752 ] printf() in dynload_shlib.c should be PySys_WriteStderr Message-ID: Bugs item #1036752, was opened at 2004-09-29 05:21 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1036752&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Jp Calderone (kuran) Assigned to: Martin v. L?wis (loewis) Summary: printf() in dynload_shlib.c should be PySys_WriteStderr Initial Comment: Line 116 of dist/src/Python/dynload_shlib.c contains a printf() guarded by a Py_VerboseFlag. This should be a PySys_WriteStderr instead. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-03 10:52 Message: Logged In: YES user_id=21627 Thanks for your report. This is now fixed in dynload_shlib.c 2.17 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-30 02:48 Message: Logged In: YES user_id=80475 Martin, I believe this is your code. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1036752&group_id=5470 From noreply at sourceforge.net Sun Oct 3 17:05:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 3 17:05:52 2004 Subject: [ python-Bugs-1039270 ] time zone tests fail on Windows Message-ID: Bugs item #1039270, was opened at 2004-10-03 12:44 Message generated for change (Comment added) made by quiver You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Brett Cannon (bcannon) Summary: time zone tests fail on Windows Initial Comment: Following tests fail on Win 2K(Japanese locale): # test_strptime.py test_compile (__main__.TimeRETests) ... FAIL test_bad_timezone (__main__.StrptimeTests) ... ERROR test_timezone (__main__.StrptimeTests) ... ERROR test_day_of_week_calculation (__main__.CalculationTests) ... ERROR test_gregorian_calculation (__main__.CalculationTests) ... ERROR test_julian_calculation (__main__.CalculationTests) ... ERROR # test_time.py test_strptime (test.test_time.TimeTestCase) ... FAIL === They all stem from time zone tests and can be divided into two groups: FAIL of test_compile is basically same as #bug 883604. http://www.python.org/sf/883604 Local time values include regular expression's metacharacters, but they are not escaped. The rest is caused because strptime can't parse the values of strftime. >>> import time >>> time.tzname ('\x93\x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)', '\x93 \x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)') >>> time.strptime(time.strftime('%Z', time.gmtime())) Traceback (most recent call last): File "", line 1, in -toplevel- time.strptime(time.strftime('%Z', time.gmtime())) File "C:\Python24\lib\_strptime.py", line 291, in strptime raise ValueError("time data did not match format: data=%s fmt=%s" % ValueError: time data did not match format: data=東京 (標準時) fmt=%a %b %d %H:%M:%S %Y The output of running test_time.py and test_strptime.py is attached. ---------------------------------------------------------------------- >Comment By: George Yoshida (quiver) Date: 2004-10-04 00:05 Message: Logged In: YES user_id=671362 I've found another bug. Line 167 & 169 of Lib/_strptime.py contains the expression: time.tzname[0].lower() I guess this is intended to normalize alphabets, but for multibyte characters this is really dangerous. >>> import time >>> time.tzname[0] '\x93\x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)' >>> _.lower() '\x93\x8c\x8b\x9e (\x95w\x8f\x80\x8e\x9e)' \x95W and \x95w is not the same character. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 From noreply at sourceforge.net Sun Oct 3 18:03:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 3 18:03:40 2004 Subject: [ python-Bugs-1017864 ] Case sensitivity bug in ConfigParser Message-ID: Bugs item #1017864, was opened at 2004-08-27 16:26 Message generated for change (Comment added) made by goodger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017864&group_id=5470 Category: Python Library >Group: Python 2.4 >Status: Closed >Resolution: Accepted Priority: 6 Submitted By: Dani (asqui) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: Case sensitivity bug in ConfigParser Initial Comment: (Assume an instantiated ConfigParser, c, with a loaded file) >>> c.get("DEFAULT", "foo", False, {"foo": "Bar"}) 'Bar' >>> c.get("DEFAULT", "Foo", False, {"Foo": "Bar"}) Traceback (most recent call last): File "", line 1, in ? c.get("DEFAULT", "Foo", False, {"Foo": "Bar"}) File "C:\Python23\lib\ConfigParser.py", line 513, in get raise NoOptionError(option, section) NoOptionError: No option 'foo' in section: 'DEFAULT' The offending code appears to be as follows (in ConfigParser.py): def get(self, section, option): opt = self.optionxform(option) if section not in self._sections: if section != DEFAULTSECT: raise NoSectionError(section) if opt in self._defaults: return self._defaults[opt] else: raise NoOptionError(option, section) elif opt in self._sections[section]: return self._sections[section][opt] elif opt in self._defaults: return self._defaults[opt] else: raise NoOptionError(option, section) It uses optionxform on the supplied option, but not on the one in the defaults dictionary. If you're going to impose a transform then you have to do it consistently, imho... >>> c.get("DEFAULT", "Foo", False, {"foo": "Bar"}) 'Bar' The supplied "Foo" gets transformed to "foo" and matches the one in the defaults dictionary. Seems a bit counterintuitive to expect that you supply any defaults as pre-transformed according to your specified optionxform. I'm not sure if this is also a problem when reading a file, but there seems to be a related post about this (with no replies!) dating back to 2000: http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&selm=8suc1f%24mf9%241%40nnrp1.deja.com I don't have time to look into it now but I'd guess this problem was resolved in the more obvious location (when reading a file) but patching of the defaults dictionary case was omitted. ---------------------------------------------------------------------- >Comment By: David Goodger (goodger) Date: 2004-10-03 12:03 Message: Logged In: YES user_id=7733 Applied the patches in CVS. This is a Python 2.3 backport candidate. Lib/ConfigParser.py 1.68 Lib/test/test_cfgparser.py 1.25 Doc/lib/libcfgparser.tex 1.39 ---------------------------------------------------------------------- Comment By: David Goodger (goodger) Date: 2004-09-23 10:22 Message: Logged In: YES user_id=7733 Should be applied prior to 2.4-beta-1. ---------------------------------------------------------------------- Comment By: David Goodger (goodger) Date: 2004-09-23 10:19 Message: Logged In: YES user_id=7733 Patches added for ConfigParser.py and test_cfgparser.py. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017864&group_id=5470 From noreply at sourceforge.net Sun Oct 3 18:05:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 3 18:05:18 2004 Subject: [ python-Bugs-1017864 ] Case sensitivity bug in ConfigParser Message-ID: Bugs item #1017864, was opened at 2004-08-27 16:26 Message generated for change (Comment added) made by goodger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017864&group_id=5470 Category: Python Library Group: Python 2.4 Status: Closed Resolution: Accepted Priority: 6 Submitted By: Dani (asqui) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: Case sensitivity bug in ConfigParser Initial Comment: (Assume an instantiated ConfigParser, c, with a loaded file) >>> c.get("DEFAULT", "foo", False, {"foo": "Bar"}) 'Bar' >>> c.get("DEFAULT", "Foo", False, {"Foo": "Bar"}) Traceback (most recent call last): File "", line 1, in ? c.get("DEFAULT", "Foo", False, {"Foo": "Bar"}) File "C:\Python23\lib\ConfigParser.py", line 513, in get raise NoOptionError(option, section) NoOptionError: No option 'foo' in section: 'DEFAULT' The offending code appears to be as follows (in ConfigParser.py): def get(self, section, option): opt = self.optionxform(option) if section not in self._sections: if section != DEFAULTSECT: raise NoSectionError(section) if opt in self._defaults: return self._defaults[opt] else: raise NoOptionError(option, section) elif opt in self._sections[section]: return self._sections[section][opt] elif opt in self._defaults: return self._defaults[opt] else: raise NoOptionError(option, section) It uses optionxform on the supplied option, but not on the one in the defaults dictionary. If you're going to impose a transform then you have to do it consistently, imho... >>> c.get("DEFAULT", "Foo", False, {"foo": "Bar"}) 'Bar' The supplied "Foo" gets transformed to "foo" and matches the one in the defaults dictionary. Seems a bit counterintuitive to expect that you supply any defaults as pre-transformed according to your specified optionxform. I'm not sure if this is also a problem when reading a file, but there seems to be a related post about this (with no replies!) dating back to 2000: http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&selm=8suc1f%24mf9%241%40nnrp1.deja.com I don't have time to look into it now but I'd guess this problem was resolved in the more obvious location (when reading a file) but patching of the defaults dictionary case was omitted. ---------------------------------------------------------------------- >Comment By: David Goodger (goodger) Date: 2004-10-03 12:05 Message: Logged In: YES user_id=7733 correction: Lib/ConfigParser.py 1.67 Lib/test/test_cfgparser.py 1.24 no change to Doc/lib/libcfgparser.tex ---------------------------------------------------------------------- Comment By: David Goodger (goodger) Date: 2004-10-03 12:03 Message: Logged In: YES user_id=7733 Applied the patches in CVS. This is a Python 2.3 backport candidate. Lib/ConfigParser.py 1.68 Lib/test/test_cfgparser.py 1.25 Doc/lib/libcfgparser.tex 1.39 ---------------------------------------------------------------------- Comment By: David Goodger (goodger) Date: 2004-09-23 10:22 Message: Logged In: YES user_id=7733 Should be applied prior to 2.4-beta-1. ---------------------------------------------------------------------- Comment By: David Goodger (goodger) Date: 2004-09-23 10:19 Message: Logged In: YES user_id=7733 Patches added for ConfigParser.py and test_cfgparser.py. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017864&group_id=5470 From noreply at sourceforge.net Sun Oct 3 18:07:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 3 18:07:22 2004 Subject: [ python-Bugs-997050 ] ConfigParser behavior change Message-ID: Bugs item #997050, was opened at 2004-07-24 08:30 Message generated for change (Comment added) made by goodger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=997050&group_id=5470 Category: Python Library Group: Python 2.4 >Status: Closed >Resolution: Accepted Priority: 6 Submitted By: David Goodger (goodger) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: ConfigParser behavior change Initial Comment: ConfigParser.set() doesn't allow non-string arguments for 'value' any more. This breaks Docutils code. I submit that the behavior should not have been changed, rather that the documentation needed updating. I volunteer to undo the change and update the documentation. ConfigParser.py rev. 1.65 implements the fix detailed in bug report 810843 (http://python.org/sf/810843): Fixed using methods (2) and (3): ConfigParser.set() now checks that the value is some flavor of string, and raises TypeError if it isn't. This is documented behavior; regression tests have been added. "This is documented behavior": where was this behavior documented before the bug fix? I couldn't find it. If it wasn't documented, wouldn't method 4 (from the bug report, below) have been more appropriate? (4) Stating in the documentation that the raw parameter should be used when an option's value might not be of type string (this might be the intended usage, but it isn't made clear). IOW, perhaps it wasn't a code bug but rather an omission in the docs. By adding the restriction and raising TypeError, it breaks Docutils, which subclasses ConfigParser. If the string-only restriction *was* documented and I just missed it, I'll accept that the Docutils code was at fault (it's not the first time ;-) and rework it. But if that restriction wasn't documented, I don't think ConfigParser's behavior should change. See also . ---------------------------------------------------------------------- >Comment By: David Goodger (goodger) Date: 2004-10-03 12:07 Message: Logged In: YES user_id=7733 Applied the patches in CVS. Lib/ConfigParser.py 1.68 Lib/test/test_cfgparser.py 1.25 Doc/lib/libcfgparser.tex 1.39 ---------------------------------------------------------------------- Comment By: David Goodger (goodger) Date: 2004-09-23 10:28 Message: Logged In: YES user_id=7733 I have come up with a solution that should satisfy everyone. Python 2.4's ConfigParser.py has added a new SafeConfigParser class, so the new string-only restriction should be added to the new class. Existing behavior is now documented and remains valid, so there should be no code breakage. But new users will get the "safe" string-only behavior from SafeConfigParser. Patches added for ConfigParser.py, test_cfgparser.py, and libcfgparser.tex. This should be applied prior to Python 2.4-beta-1. ---------------------------------------------------------------------- Comment By: David Goodger (goodger) Date: 2004-08-30 13:26 Message: Logged In: YES user_id=7733 [anthonybaxter] > RCP isn't in __all__, Perhaps it should be, because: > and isn't documented, Actually, it is documented: http://www.python.org/dev/doc/devel/lib/RawConfigParser-objects.html > we can also put a type check in the > output code to make sure it gets picked up there... It gets more and more complicated, doesn't it? I still think that the code itself wasn't broken so there was no need to fix it. The simplest solution would be to revert the rev 1.65 code changes, and just add some documentation. Something along the lines of: "Output only works with string values. If used for non-string values, you're on your own." We shouldn't be *removing* functionality without a darned good reason. I don't see such a reason here. And since this change broke preexisting code that relied on that functionality, it seems obvious that it should be reverted. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-08-30 12:18 Message: Logged In: YES user_id=29957 I can live with David's last solution (moving the type check to CP.set). RCP isn't in __all__, and isn't documented, so I don't think it matters so much if it has internally inconsistent results - we can also put a type check in the output code to make sure it gets picked up there... ---------------------------------------------------------------------- Comment By: David Goodger (goodger) Date: 2004-08-24 23:34 Message: Logged In: YES user_id=7733 [rhettinger] > Perhaps, the set() method should coerce to a string rather > than test for it. Perhaps. But regardless of whether it's a test or coercion, if it's in RawConfigParser.set, it prevents non-string use. Perhaps ConfigParser.set should extend RawConfigParser.set with the string test or coercion, leaving RawConfigParser.set as it was before rev 1.65. That may be the best solution. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-08-24 12:10 Message: Logged In: YES user_id=80475 Perhaps, the set() method should coerce to a string rather than test for it. ---------------------------------------------------------------------- Comment By: David Goodger (goodger) Date: 2004-08-24 11:14 Message: Logged In: YES user_id=7733 I strongly feel that the 2.3 behavior should be restored, and documented, before 2.4b1. I will do the doc change. I have updated the bug report with links and discussions. ---------------------------------------------------------------------- Comment By: David Goodger (goodger) Date: 2004-08-24 11:10 Message: Logged In: YES user_id=7733 Also see the python-dev thread at . The third message in the thread was garbled due to a GPG problem; reproduced below. [Fred L. Drake, Jr.] > David has (properly) been asking me to look into this, and i've > managed not to have enough time. Sorry, David! I thought a python-dev nudge might get you off your keister ;-) > The ConfigParser documentation was certainly too vague, but the > problem, as I see it, is that the module was never intended to store > non-string values. BUT ConfigParser *did* allow non-strings to be stored, for internal use only (can't write them out, of course). And changing it *did* break code. I think a doc change acknowledging that ability but qualifying it ("for internal use only, can't be used to write out config files or interpolate values") would have been a better fix. The change in rev. 1.65 makes ConfigParser.py *less* useful. When I used ConfigParser in Docutils, I wasn't being especially clever when I used it to set non-string values. It will take a lot more cleverness post-rev-1.65 to enable that functionality. Of course, the fix to Docutils is pretty simple: just override ConfigParser.set with the older version. But that depends on private implementation details (self._defaults, self._sections). This is *much* more dangerous and impossible to future-proof against. > I think allowing them is just asking for still more trouble from > that module down the road. Has there been any trouble until now? Why "fix" something that wasn't really broke? The original bug report (http://www.python.org/sf/810843) was really only asking for clarification. It ends with: Nonetheless, I was shocked to see what I thought was a straightforward use of ConfigParser throw off errors. Imagine developers' shock now that we can't do what we want at all! Docutils is not the only project using ConfigParser to store non-strings. I wouldn't be surprised to see other code breaking, which we probably wouldn't find out about until after 2.4-final. > Sure, the module could be made happy by reverting the patch you > cite, but happy-by-accident is a very fragile state to be in. So let's remove the accident, and make the happiness official! >> The comment for bug 810843 says "This is documented behavior", but >> I couldn't find any such documentation pre-dating this change. > > This may have been a bug in my memory. Now that the memory bug is fixed, how about reconsidering the resultant decision? I'll write a doc patch ASAP. -- David Goodger ---------------------------------------------------------------------- Comment By: David Goodger (goodger) Date: 2004-08-24 11:04 Message: Logged In: YES user_id=7733 The alternative is the following change in Docutils. Unfortunately, it requires access to internal implementation details marked with single leading underscores. diff -u -r1.64 frontend.py --- docutils/frontend.py 24 Jul 2004 14:13:38 -0000 1.64 +++ docutils/frontend.py 24 Aug 2004 15:01:32 -0000 @@ -637,6 +637,22 @@ section_dict[option] = self.get(section, option, raw=1) return section_dict + def set(self, section, option, value): + """ + Set an option. + + Overrides stdlib ConfigParser's set() method to allow non-string + values. Required for compatibility with Python 2.4. + """ + if not section or section == CP.DEFAULTSECT: + sectdict = self._defaults + else: + try: + sectdict = self._sections[section] + except KeyError: + raise CP.NoSectionError(section) + sectdict[self.optionxform(option)] = value + class ConfigDeprecationWarning(DeprecationWarning): """Warning for deprecated configuration file features.""" ---------------------------------------------------------------------- Comment By: David Goodger (goodger) Date: 2004-08-24 11:00 Message: Logged In: YES user_id=7733 The change that I'm proposing to revert: http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Lib/ConfigParser.py?r1=1.64&r2=1.65 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=997050&group_id=5470 From noreply at sourceforge.net Sun Oct 3 18:15:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 3 18:15:24 2004 Subject: [ python-Bugs-810843 ] Support for non-string data in ConfigParser unclear/buggy Message-ID: Bugs item #810843, was opened at 2003-09-22 17:58 Message generated for change (Comment added) made by goodger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=810843&group_id=5470 Category: None Group: Python 2.3 Status: Closed Resolution: Fixed Priority: 5 Submitted By: curby (kirbysdl) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: Support for non-string data in ConfigParser unclear/buggy Initial Comment: In my current project, I'm using ConfigParser to read configuration files as well as manage internal configuration. The following test script generates an error because values of options must be strings in order for ConfigParser to run smoothly. (It's short enough that I'm including it here for reference; I'm also attaching it). #----- BEGIN TEST ----- #!/usr/bin/env python import ConfigParser import sys conf = ConfigParser.ConfigParser() conf.add_section("sect") conf.set("sect","option",2) conf.write(sys.stderr) conf.get("sect","option") #----- END TEST ----- >From the ConfigParser code, I see that the value isn't checked to make sure it is a string before string methods are applied to it. I'm currently using the raw parameter to get() (see http://www.python.org/doc/lib/ConfigParser-objects.html#l2h-1261) in order to circumvent the problem. I'd suggest one or more of the following: (1) Casting the option to a string explicitly instead of assuming it is a string (this could also be done in the set() method). (2) Checking that the option is a string instead of assuming it is a string. (3) Stating in the documentation that the set() method should only take values of type string. (4) Stating in the documentation that the raw parameter should be used when an option's value might not be of type string (this might be the intended usage, but it isn't made clear). I hope none of these are too difficult to implement, but I understand that this is a noncritical problem. Nonetheless, I was shocked to see what I thought was a straightforward use of ConfigParser throw off errors. ---------------------------------------------------------------------- >Comment By: David Goodger (goodger) Date: 2004-10-03 12:15 Message: Logged In: YES user_id=7733 FYI: Moved the new type-check to SafeConfigParser.set(); see http://www.python.org/sf/997050. Lib/ConfigParser.py 1.68 Lib/test/test_cfgparser.py 1.25 Doc/lib/libcfgparser.tex 1.39 ---------------------------------------------------------------------- Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-17 23:32 Message: Logged In: YES user_id=3066 Fixed using methods (2) and (3): ConfigParser.set() now checks that the value is some flavor of string, and raises TypeError if it isn't. This is documented behavior; regression tests have been added. Doc/lib/libcfgparser.tex 1.36 Lib/ConfigParser.py 1.65 Lib/test/test_cfgparser.py 1.22 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=810843&group_id=5470 From noreply at sourceforge.net Mon Oct 4 01:16:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 4 01:16:24 2004 Subject: [ python-Bugs-1039270 ] time zone tests fail on Windows Message-ID: Bugs item #1039270, was opened at 2004-10-02 20:44 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Brett Cannon (bcannon) Summary: time zone tests fail on Windows Initial Comment: Following tests fail on Win 2K(Japanese locale): # test_strptime.py test_compile (__main__.TimeRETests) ... FAIL test_bad_timezone (__main__.StrptimeTests) ... ERROR test_timezone (__main__.StrptimeTests) ... ERROR test_day_of_week_calculation (__main__.CalculationTests) ... ERROR test_gregorian_calculation (__main__.CalculationTests) ... ERROR test_julian_calculation (__main__.CalculationTests) ... ERROR # test_time.py test_strptime (test.test_time.TimeTestCase) ... FAIL === They all stem from time zone tests and can be divided into two groups: FAIL of test_compile is basically same as #bug 883604. http://www.python.org/sf/883604 Local time values include regular expression's metacharacters, but they are not escaped. The rest is caused because strptime can't parse the values of strftime. >>> import time >>> time.tzname ('\x93\x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)', '\x93 \x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)') >>> time.strptime(time.strftime('%Z', time.gmtime())) Traceback (most recent call last): File "", line 1, in -toplevel- time.strptime(time.strftime('%Z', time.gmtime())) File "C:\Python24\lib\_strptime.py", line 291, in strptime raise ValueError("time data did not match format: data=%s fmt=%s" % ValueError: time data did not match format: data=東京 (標準時) fmt=%a %b %d %H:%M:%S %Y The output of running test_time.py and test_strptime.py is attached. ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-10-03 16:16 Message: Logged In: YES user_id=357491 The .lower() call is intended to normalize since capitalization is not standard across OSs. But if it is a Unicode string it should be fine. And even if it isn't, it is all lowercased for comparison anyway, so as long as it is consistent, shouldn't it still work? As for your example of strptime not being able to parse, you have a bug in it; you forgot the format string. It should have been ``time.strptime(time.strftime('%Z'), '%Z')``. Give that a run and let me know what the output is. As for this whole multi-byte issue, is it all being returned as Unicode strings, or is it just a regular string? In other words, what is ``type(time.tzname[0])`` spitting out? And what character encoding is all of this in (i.e., what should I pass to unicode so as to not have it raise UnicodeDecodeError)? And finally, for the regex metacharacter stuff, why the hell are there parentheses in a timezone?!? Whoever decided that was good did it just to upset me. That does need to be fixed. Apply the patch I just uploaded and let me know if it at least deals with that problem. Have I mentioned I hate timezones? In case I haven't, I do. Thanks for catching this all, though, George. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-03 08:05 Message: Logged In: YES user_id=671362 I've found another bug. Line 167 & 169 of Lib/_strptime.py contains the expression: time.tzname[0].lower() I guess this is intended to normalize alphabets, but for multibyte characters this is really dangerous. >>> import time >>> time.tzname[0] '\x93\x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)' >>> _.lower() '\x93\x8c\x8b\x9e (\x95w\x8f\x80\x8e\x9e)' \x95W and \x95w is not the same character. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 From noreply at sourceforge.net Mon Oct 4 01:23:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 4 01:23:37 2004 Subject: [ python-Bugs-1038935 ] Typo "signiture" in HTML generated by DocXMLRPCServer Message-ID: Bugs item #1038935, was opened at 2004-10-02 04:40 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038935&group_id=5470 Category: Python Library Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Malte Helmert (maltehelmert) >Assigned to: Brett Cannon (bcannon) Summary: Typo "signiture" in HTML generated by DocXMLRPCServer Initial Comment: There is a typo in the docstring of SimpleXMLRPCServer.system_methodSignature, where the word "signature" reads "signiture". If this method is exported by a DocXMLRPCServer using register_introspection_functions, the wrong text appears on the generated HTML content. This is with Python 2.4a3. ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-10-03 16:23 Message: Logged In: YES user_id=357491 Fixed in rev. 1.9 for HEAD and rev. 1.7.8.1 for 2.3 . Thanks, Malte, for spotting this. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038935&group_id=5470 From noreply at sourceforge.net Mon Oct 4 12:23:34 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 4 12:23:37 2004 Subject: [ python-Feature Requests-1039857 ] win32 silent installation: allow MAINDIR on command line Message-ID: Feature Requests item #1039857, was opened at 2004-10-04 12:23 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1039857&group_id=5470 Category: Installation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Matteo Gallivanoni (sigmud) Assigned to: Nobody/Anonymous (nobody) Summary: win32 silent installation: allow MAINDIR on command line Initial Comment: This request is related to my (false) "Bugs item #1038410" since it is by design and it won't be fixed I thought I could be suggested as a new feature. Request: Add a command line switch to be able to specify python install directory in silent installation mode in other word it should be something like this: Python-X.Y.Z.exe /s /d MAINDIR=C:\some_dir\python BTW: IIRC the windows wise installer configuration file is not provided into python cvs source tree so I can only suggest this google-groups thread as a hint: Newsgroups:wise.wise7.general Subject: "Passing the path name to a silent install. How?" Data:2000/01/17 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1039857&group_id=5470 From noreply at sourceforge.net Mon Oct 4 18:34:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 4 18:34:39 2004 Subject: [ python-Bugs-1040026 ] os.times() is bogus Message-ID: Bugs item #1040026, was opened at 2004-10-04 12:34 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1040026&group_id=5470 Category: Extension Modules Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Nobody/Anonymous (nobody) Summary: os.times() is bogus Initial Comment: At least on Linux, all values the tuple returned by os.times() are bogus. The code always divides the return value from the times() system call by HZ, but according to the man page times(2) they are in clock ticks. The man page adds: NOTES The number of clock ticks per second can be obtained using sysconf(_SC_CLK_TCK); In POSIX-1996 the symbol CLK_TCK (defined in ) is mentioned as obsolescent. It is obsolete now. [...] Note that clock(3) returns values of type clock_t that are not measured in clock ticks but in CLOCKS_PER_SEC. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1040026&group_id=5470 From noreply at sourceforge.net Tue Oct 5 01:00:26 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 5 01:00:29 2004 Subject: [ python-Feature Requests-1040267 ] os.run function for convinience and robustness Message-ID: Feature Requests item #1040267, was opened at 2004-10-04 23:00 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1040267&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Alexander Schmolck (aschmolck) Assigned to: Nobody/Anonymous (nobody) Summary: os.run function for convinience and robustness Initial Comment: I propose to add the the following convenience function: os.run('some_program', infilename, '-o', outfilename) as equivalent to os.spawnvp(os.P_WAIT, 'some_program', ['some_program', infilename, '-o', outfilename]) Rationale: Newbies and more experienced programmers alike are likely to make incorrect use of os.system for the common task of running a process passing a couple of arguments, when they should really be using os.spawnvp, but the latter has an awkward interface and is much more difficult to find in the docs (in a recent thread [1]; a couple of regulars admitted either having been bitten or at least finding os.spawnvp difficult to use or awkward to remember). As a consequence this (broken): os.system('some_program %s -o %s' % (infilename, outfilename) is often used instead of os.spawnvp(os.P_WAIT, 'some_program', ['some_program', infilename, '-o', outfilename]) which works correctly, but is more awkward (and also easy to get wrong, e.g. note the double occurence of 'some_program'). I propose to add the covenience function for the common case (or at least some doc improvements) to make such brittle code less common and especially to aid less experienced programmers who are unlikely to even hit upon os.spawnvp in the docs. A similar function would also be useful for popen* style use (which currently cannot be done at all without shell-escaping by hand). Further rationale and an implementation can be found in [1]. [1] Message-ID: or http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&c2coff=1&threadm=yfsacv24cdn.fsf%40black4.ex.ac.uk&rnum=1&prev=/groups%3Fq%3Dos.system%26hl%3Den%26lr%3D%26ie%3DUTF-8%26c2coff%3D1%26scoring%3Dd%26selm%3Dyfsacv24cdn.fsf%2540black4.ex.ac.uk%26rnum%3D1 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1040267&group_id=5470 From noreply at sourceforge.net Tue Oct 5 08:51:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 5 08:51:14 2004 Subject: [ python-Bugs-1040439 ] Missing documentation on How To Link With libpython Message-ID: Bugs item #1040439, was opened at 2004-10-05 10:51 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1040439&group_id=5470 Category: Documentation Group: Feature Request Status: Open Resolution: None Priority: 5 Submitted By: Sir Raorn (raorn) Assigned to: Nobody/Anonymous (nobody) Summary: Missing documentation on How To Link With libpython Initial Comment: Missing documentation part about linking program with libpython (embedding python). Questions are: How one can get needed CFLAGS (-I... -D...) How one can get needed LDFLAGS (-L... -l...) For now, applications assumes that headers located in PREFIX/include/pythonX.Y, libraries located in EXEC_PREFIX/lib/pythonX.Y/config ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1040439&group_id=5470 From noreply at sourceforge.net Tue Oct 5 12:46:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 5 12:46:52 2004 Subject: [ python-Bugs-1038693 ] bad link in Tkinter docs Message-ID: Bugs item #1038693, was opened at 2004-10-01 22:26 Message generated for change (Comment added) made by grubert You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038693&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Brett Cannon (bcannon) Assigned to: Nobody/Anonymous (nobody) Summary: bad link in Tkinter docs Initial Comment: http://docs.python.org/lib/node639.html has a dead link pointing to the "Packer class interface". Is that supposed to be pointing to the next page, "Packer options"? ---------------------------------------------------------------------- Comment By: engelbert gruber (grubert) Date: 2004-10-05 12:46 Message: Logged In: YES user_id=147070 googling gave the impression it is stale since 2001 http://www.ssg.co.at/python/tkinter.tex.patch ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038693&group_id=5470 From noreply at sourceforge.net Tue Oct 5 15:55:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 5 15:56:02 2004 Subject: [ python-Bugs-1036626 ] file.next() info hidden Message-ID: Bugs item #1036626, was opened at 2004-09-29 00:42 Message generated for change (Comment added) made by grubert You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1036626&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Nick Jacobson (nickjacobson) Assigned to: Nobody/Anonymous (nobody) Summary: file.next() info hidden Initial Comment: (A.M. Kuchling told me to post this bug report.) In section 2.3.8, "File Objects", of the Python Library Reference, the following section is given for file.next(): << In order to make a for loop the most efficient way of looping over the lines of a file (a very common operation), the next() method uses a hidden read-ahead buffer. As a consequence of using a read-ahead buffer, combining next() with other file methods (like readline()) does not work right. >> This information is necessary for anyone who mixes "for line in file:" with read()/tell()/etc. However, in the statement "for line in file:", next() is called implicitly, and therefore it may not occur to the programmer to find the information there. Therefore the documentation should be reorganized. One solution is to have the read()/tell()/etc. methods refer to the above section. i.e. "Don't use this method with next() or 'for line in file:'; see next()." A second solution is to move the above section to before the methods section (i.e. after the second paragraph after the 2.3.8 header), and to mention 'for line in file:'. i.e. "As a consequence of using a read-ahead buffer, combining next() or 'for line in file:' with other file methods (like readline()) does not work right." (Also, saying "with other file methods (like readline())" is too vague. IMHO it should specify which ones not to use. That's debatable though.) ---------------------------------------------------------------------- Comment By: engelbert gruber (grubert) Date: 2004-10-05 15:55 Message: Logged In: YES user_id=147070 see also http://www.python.org/sf/645594 for lin in file: file.tell() tells wrong ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1036626&group_id=5470 From noreply at sourceforge.net Tue Oct 5 16:19:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 5 16:19:39 2004 Subject: [ python-Bugs-1032875 ] ftplib has incomplete transfer when sending files in Windows Message-ID: Bugs item #1032875, was opened at 2004-09-22 21:21 Message generated for change (Comment added) made by grubert You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1032875&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Ed Sanville (infinity314) Assigned to: Nobody/Anonymous (nobody) Summary: ftplib has incomplete transfer when sending files in Windows Initial Comment: I wrote the following short script: from ftplib import FTP ftp=FTP('ftp.edsanville.us', username, password) print ftp.getwelcome() print ftp.storbinary('stor test.xls', file('test.xls', 'r')) ftp.quit() It transfers the entire file under Red Hat Linux 9, but under Windows XP, it only transfers a couple kilobytes at most. Using a hex editor, I found that the first 5 or 6 bytes of the file are OK on the remote side after transfer, but the rest of the file is totally wrong. I'm at a loss, it doesn't make any sense. Please, somebody tell me I'm not insane. ---------------------------------------------------------------------- Comment By: engelbert gruber (grubert) Date: 2004-10-05 16:19 Message: Logged In: YES user_id=147070 looks like a file problem: f=open('test.xls');d=f.read() reads all the file. on winxp/py234 not making a binary file: for i in range(0,50):for d in range(0,255): f.write(d) reading it: d=f.read() 1.round len(d) = 26, f.tell() 12288 2.round len(d) = 26, f.tell() 12800 chr(26) is the end of a dos textfile. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1032875&group_id=5470 From noreply at sourceforge.net Tue Oct 5 19:45:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 5 19:45:54 2004 Subject: [ python-Bugs-1019882 ] hotshot start / stop stats bug Message-ID: Bugs item #1019882, was opened at 2004-08-31 18:06 Message generated for change (Comment added) made by aminusfu You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1019882&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Barry A. Warsaw (bwarsaw) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: hotshot start / stop stats bug Initial Comment: Hotshot has a bug that manifests when using start/stop instead of run(), however it appears to only occur in the interactive interpreter or in an embedded application. If you run the attached program like "python hotshotbug.py" it works just fine. However, if you start an interactive session and paste each line one by one into the shell, you get the following exception: Traceback (most recent call last): File "", line 1, in ? File "/usr/local/lib/python2.3/hotshot/stats.py", line 12, in load return StatsLoader(filename).load() File "/usr/local/lib/python2.3/hotshot/stats.py", line 29, in load for event in log: File "/usr/local/lib/python2.3/hotshot/log.py", line 110, in next return what, self._pop(), tdelta IndexError: pop from empty list We discovered this when trying to profile an embedded application which doesn't have a single entrypoint suitable for run(). Unfortunately, I can't provide that application, but it shouldn't matter, since the interpreter crash is the same. ---------------------------------------------------------------------- Comment By: Robert Brewer (aminusfu) Date: 2004-10-05 17:45 Message: Logged In: YES user_id=967320 I get the same traceback even when not running interactively. I can provide .prof files if needed. fumanchu@amor.org ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-09-27 14:42 Message: Logged In: YES user_id=12800 Could this be related to 900092? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1019882&group_id=5470 From noreply at sourceforge.net Tue Oct 5 20:56:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 5 20:56:52 2004 Subject: [ python-Bugs-1039270 ] time zone tests fail on Windows Message-ID: Bugs item #1039270, was opened at 2004-10-03 12:44 Message generated for change (Comment added) made by quiver You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Brett Cannon (bcannon) Summary: time zone tests fail on Windows Initial Comment: Following tests fail on Win 2K(Japanese locale): # test_strptime.py test_compile (__main__.TimeRETests) ... FAIL test_bad_timezone (__main__.StrptimeTests) ... ERROR test_timezone (__main__.StrptimeTests) ... ERROR test_day_of_week_calculation (__main__.CalculationTests) ... ERROR test_gregorian_calculation (__main__.CalculationTests) ... ERROR test_julian_calculation (__main__.CalculationTests) ... ERROR # test_time.py test_strptime (test.test_time.TimeTestCase) ... FAIL === They all stem from time zone tests and can be divided into two groups: FAIL of test_compile is basically same as #bug 883604. http://www.python.org/sf/883604 Local time values include regular expression's metacharacters, but they are not escaped. The rest is caused because strptime can't parse the values of strftime. >>> import time >>> time.tzname ('\x93\x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)', '\x93 \x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)') >>> time.strptime(time.strftime('%Z', time.gmtime())) Traceback (most recent call last): File "", line 1, in -toplevel- time.strptime(time.strftime('%Z', time.gmtime())) File "C:\Python24\lib\_strptime.py", line 291, in strptime raise ValueError("time data did not match format: data=%s fmt=%s" % ValueError: time data did not match format: data=東京 (標準時) fmt=%a %b %d %H:%M:%S %Y The output of running test_time.py and test_strptime.py is attached. ---------------------------------------------------------------------- >Comment By: George Yoshida (quiver) Date: 2004-10-06 03:56 Message: Logged In: YES user_id=671362 bcannon write: > The .lower() call is intended to normalize since capitalization > is not standard across OSs. But if it is a Unicode string it > should be fine. And even if it isn't, it is all lowercased for > comparison anyway, so as long as it is consistent, shouldn't it > still work? Hmm. > As for your example of strptime not being able to parse, you have > a bug in it; you forgot the format string. It should have been > ``time.strptime(time.strftime('%Z'), '%Z')``. Give that a run > and let me know what the output is. Yeah, it's my fault. I forget to specify a format. Even so, strptime couldn't parse timezone. > As for this whole multi-byte issue, is it all being returned as > Unicod e strings, or is it just a regular string? In other > words, what is ``type(time.tzname[0])`` spitting out? And what > character encoding is all of this in (i.e., what should I pass > to unicode so as to not have it raise UnicodeDecodeError)? It returns strings(not a unicode), and the encoding is cp932. This is a default encoding of Japanese Windows. >>> unicode(time.tzname[0], 'cp932') u'\u6771\u4eac (\u6a19\u6e96\u6642)' > And finally, for the regex metacharacter stuff, why the hell ar > e there parentheses in a timezone?!? Whoever decided that wa > s good did it just to upset me. Ask M$ Japan :-; I don't regard 'Tokyo (standard time)' as an acceptable representation for time zone at all, but this is what Windows returns as a time zone on my box. > That does need to be fixed. Apply the patch I just uploaded and let > me know if it at least deals with that problem. With your patch, all tests succeed without any Error or Fail, and strftime <-> strptime conversions work well. This is a backport candidate, so I created a new patch against Python 2.3 with listcomps instead of genexprs. But there is one problem left. On IDLE, strptime still can't parse. I haven't looked into it in details, but probably patch #590913 has something to do with it. This patch sets locale at IDLE's start up time and this can affect behaviors of string-related functions and constants. [PEP 263 support in IDLE] http://www.python.org/sf/590913 # patch applied >>> time.strptime(time.strptime('%Z'), '%Z') Traceback (most recent call last): File "", line 1, in -toplevel- time.strptime(time.strptime('%Z'), '%Z') File "C:\Python24\lib\_strptime.py", line 291, in strptime if not found: ValueError: time data did not match format: data=%Z fmt=% a %b %d %H:%M:%S %Y >>> import locale >>> locale.getlocale() ['Japanese_Japan', '932'] # culprit? > Have I mentioned I hate timezones? In case I haven't, I do. I agree with you one hundred percent. --George ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-04 08:16 Message: Logged In: YES user_id=357491 The .lower() call is intended to normalize since capitalization is not standard across OSs. But if it is a Unicode string it should be fine. And even if it isn't, it is all lowercased for comparison anyway, so as long as it is consistent, shouldn't it still work? As for your example of strptime not being able to parse, you have a bug in it; you forgot the format string. It should have been ``time.strptime(time.strftime('%Z'), '%Z')``. Give that a run and let me know what the output is. As for this whole multi-byte issue, is it all being returned as Unicode strings, or is it just a regular string? In other words, what is ``type(time.tzname[0])`` spitting out? And what character encoding is all of this in (i.e., what should I pass to unicode so as to not have it raise UnicodeDecodeError)? And finally, for the regex metacharacter stuff, why the hell are there parentheses in a timezone?!? Whoever decided that was good did it just to upset me. That does need to be fixed. Apply the patch I just uploaded and let me know if it at least deals with that problem. Have I mentioned I hate timezones? In case I haven't, I do. Thanks for catching this all, though, George. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-04 00:05 Message: Logged In: YES user_id=671362 I've found another bug. Line 167 & 169 of Lib/_strptime.py contains the expression: time.tzname[0].lower() I guess this is intended to normalize alphabets, but for multibyte characters this is really dangerous. >>> import time >>> time.tzname[0] '\x93\x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)' >>> _.lower() '\x93\x8c\x8b\x9e (\x95w\x8f\x80\x8e\x9e)' \x95W and \x95w is not the same character. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 From noreply at sourceforge.net Tue Oct 5 21:12:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 5 21:12:31 2004 Subject: [ python-Bugs-1039270 ] time zone tests fail on Windows Message-ID: Bugs item #1039270, was opened at 2004-10-03 12:44 Message generated for change (Comment added) made by quiver You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Brett Cannon (bcannon) Summary: time zone tests fail on Windows Initial Comment: Following tests fail on Win 2K(Japanese locale): # test_strptime.py test_compile (__main__.TimeRETests) ... FAIL test_bad_timezone (__main__.StrptimeTests) ... ERROR test_timezone (__main__.StrptimeTests) ... ERROR test_day_of_week_calculation (__main__.CalculationTests) ... ERROR test_gregorian_calculation (__main__.CalculationTests) ... ERROR test_julian_calculation (__main__.CalculationTests) ... ERROR # test_time.py test_strptime (test.test_time.TimeTestCase) ... FAIL === They all stem from time zone tests and can be divided into two groups: FAIL of test_compile is basically same as #bug 883604. http://www.python.org/sf/883604 Local time values include regular expression's metacharacters, but they are not escaped. The rest is caused because strptime can't parse the values of strftime. >>> import time >>> time.tzname ('\x93\x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)', '\x93 \x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)') >>> time.strptime(time.strftime('%Z', time.gmtime())) Traceback (most recent call last): File "", line 1, in -toplevel- time.strptime(time.strftime('%Z', time.gmtime())) File "C:\Python24\lib\_strptime.py", line 291, in strptime raise ValueError("time data did not match format: data=%s fmt=%s" % ValueError: time data did not match format: data=東京 (標準時) fmt=%a %b %d %H:%M:%S %Y The output of running test_time.py and test_strptime.py is attached. ---------------------------------------------------------------------- >Comment By: George Yoshida (quiver) Date: 2004-10-06 04:12 Message: Logged In: YES user_id=671362 Correct my previous post. There's nothing wrong with strptime on IDLE. >>> import time >>> time.strptime(time.strftime('%Z'), '%Z') (1900, 1, 1, 0, 0, 0, 0, 1, 0) Please close this bug and apply the patches. Thanks Brett! ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-06 03:56 Message: Logged In: YES user_id=671362 bcannon write: > The .lower() call is intended to normalize since capitalization > is not standard across OSs. But if it is a Unicode string it > should be fine. And even if it isn't, it is all lowercased for > comparison anyway, so as long as it is consistent, shouldn't it > still work? Hmm. > As for your example of strptime not being able to parse, you have > a bug in it; you forgot the format string. It should have been > ``time.strptime(time.strftime('%Z'), '%Z')``. Give that a run > and let me know what the output is. Yeah, it's my fault. I forget to specify a format. Even so, strptime couldn't parse timezone. > As for this whole multi-byte issue, is it all being returned as > Unicod e strings, or is it just a regular string? In other > words, what is ``type(time.tzname[0])`` spitting out? And what > character encoding is all of this in (i.e., what should I pass > to unicode so as to not have it raise UnicodeDecodeError)? It returns strings(not a unicode), and the encoding is cp932. This is a default encoding of Japanese Windows. >>> unicode(time.tzname[0], 'cp932') u'\u6771\u4eac (\u6a19\u6e96\u6642)' > And finally, for the regex metacharacter stuff, why the hell ar > e there parentheses in a timezone?!? Whoever decided that wa > s good did it just to upset me. Ask M$ Japan :-; I don't regard 'Tokyo (standard time)' as an acceptable representation for time zone at all, but this is what Windows returns as a time zone on my box. > That does need to be fixed. Apply the patch I just uploaded and let > me know if it at least deals with that problem. With your patch, all tests succeed without any Error or Fail, and strftime <-> strptime conversions work well. This is a backport candidate, so I created a new patch against Python 2.3 with listcomps instead of genexprs. But there is one problem left. On IDLE, strptime still can't parse. I haven't looked into it in details, but probably patch #590913 has something to do with it. This patch sets locale at IDLE's start up time and this can affect behaviors of string-related functions and constants. [PEP 263 support in IDLE] http://www.python.org/sf/590913 # patch applied >>> time.strptime(time.strptime('%Z'), '%Z') Traceback (most recent call last): File "", line 1, in -toplevel- time.strptime(time.strptime('%Z'), '%Z') File "C:\Python24\lib\_strptime.py", line 291, in strptime if not found: ValueError: time data did not match format: data=%Z fmt=% a %b %d %H:%M:%S %Y >>> import locale >>> locale.getlocale() ['Japanese_Japan', '932'] # culprit? > Have I mentioned I hate timezones? In case I haven't, I do. I agree with you one hundred percent. --George ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-04 08:16 Message: Logged In: YES user_id=357491 The .lower() call is intended to normalize since capitalization is not standard across OSs. But if it is a Unicode string it should be fine. And even if it isn't, it is all lowercased for comparison anyway, so as long as it is consistent, shouldn't it still work? As for your example of strptime not being able to parse, you have a bug in it; you forgot the format string. It should have been ``time.strptime(time.strftime('%Z'), '%Z')``. Give that a run and let me know what the output is. As for this whole multi-byte issue, is it all being returned as Unicode strings, or is it just a regular string? In other words, what is ``type(time.tzname[0])`` spitting out? And what character encoding is all of this in (i.e., what should I pass to unicode so as to not have it raise UnicodeDecodeError)? And finally, for the regex metacharacter stuff, why the hell are there parentheses in a timezone?!? Whoever decided that was good did it just to upset me. That does need to be fixed. Apply the patch I just uploaded and let me know if it at least deals with that problem. Have I mentioned I hate timezones? In case I haven't, I do. Thanks for catching this all, though, George. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-04 00:05 Message: Logged In: YES user_id=671362 I've found another bug. Line 167 & 169 of Lib/_strptime.py contains the expression: time.tzname[0].lower() I guess this is intended to normalize alphabets, but for multibyte characters this is really dangerous. >>> import time >>> time.tzname[0] '\x93\x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)' >>> _.lower() '\x93\x8c\x8b\x9e (\x95w\x8f\x80\x8e\x9e)' \x95W and \x95w is not the same character. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 From noreply at sourceforge.net Wed Oct 6 03:59:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 6 03:59:46 2004 Subject: [ python-Feature Requests-1023290 ] proposed struct module format code addition Message-ID: Feature Requests item #1023290, was opened at 2004-09-06 16:42 Message generated for change (Comment added) made by etrepum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1023290&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Josiah Carlson (josiahcarlson) Assigned to: Nobody/Anonymous (nobody) Summary: proposed struct module format code addition Initial Comment: I believe there should be a mechanism to load and unload arbitrarily large integers via the struct module. Currently, one would likely start with the 'Q' format character, creating the integer in a block-wise fashion with multiplies and shifts. This is OK, though it tends to lend itself to certain kinds of bugs. There is currently another method for getting large integers from strings and going back without the struct module: long(stri.encode('hex'), 16) hex(inte)[2:].decode('hex') Arguably, such things shouldn't be done for the packing and unpacking of binary data in general (the string slicing especially). I propose a new format character for the struct module, specifically because the struct module is to "Interpret strings as packed binary data". Perhaps 'g' and 'G' (eg. biGint) is sufficient, though any reasonable character should suffice. Endianness should be handled, and the number of bytes representing the object would be the same as with the 's' formatting code. That is, '>60G' would be an unsigned big-endian integer represented by 60 bytes (null filled if the magnitude of the passed integer is not large enough). The only reason why one wouldn't want this functionality in the struct module is "This module performs conversions between Python values and C structs represented as Python strings." and arbitrarily large integers are not traditionally part of a C struct (though I am sure many of us have implemented arbitrary precision integers with structs). The reason "not a C type" has been used to quash the 'bit' and 'nibble' format character, because "masks and shifts" are able to emulate them, and though "masks and shifts" could also be used here, I have heard myself and others state that there should be an easy method for converting between large longs and strings. A side-effect for allowing arbitrarily large integers to be represented in this fashion is that its functionality could, if desired, subsume the other integer type characters, as well as fill in the gaps for nonstandard size integers (3, 5, 6, 7 etc. byte integers), that I (and I am sure others) have used in various applications. Currently no implementation exists, and I don't have time to do one now. Having taken a look at longobject.c and structmodule.c, I would likely be able to make a patch to the documentation, structmodule.c, and test_struct.py around mid October, if this functionality is desireable to others and accepted. While I doubt that a PEP for this is required, if necessary I would write one up with a sample implementation around mid October. ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-10-05 21:59 Message: Logged In: YES user_id=139309 I would definitely have an immediate use for 3 byte integers.. the Mach- O executable format has a couple fields that are 3 byte unsigned integers (bit flags). py2app's supporting library macholib reads and writes this format directly. Currently I have several places that look like this: class dylib_reference(Structure): _fields_ = ( # XXX - ick, fix ('isym_flags', p_ulong), #('isym', p_ubyte * 3), #('flags', p_ubyte), ) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-02 19:00 Message: Logged In: YES user_id=80475 If no one other that the OP supports this, I would like to reject putting this in the struct module. Initially, it seemed like a fit because the endian options and whatnot are already in place; however, in one way or another each of the posters except the OP has stated good reasons for it not being in the struct module. Variable length C structure members are not what the module is about. Having to know the length in advance of the call is a killer. The learning curve issues with struct are also a problem. And, the use cases jsut don't point to struct. Put in a simple function in binascii or let's drop it. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-10-02 18:34 Message: Logged In: YES user_id=341410 I have just attached a unified diff against structmodule.c 2.62 in CVS. It implements the semantics I have been describing, compiles cleanly, and produces proper results. >>> pickle.encode_long(83726) '\x0eG\x01' >>> struct.pack('<3g', 83726) '\x0eG\x01' >>> struct.unpack('<3g', struct.pack('<3g', 83726)) (83726L,) If the functionality is accepted, I will submit diffs for test_struct.py and libstruct.tex . ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-16 14:11 Message: Logged In: YES user_id=341410 Curse you Tim, for your core Python experience *wink*. Pickle is one example where a pascal-like encoding of longs was an encoding decision made to be flexible and space efficient. Certainly we have disparate use cases. Mine is for fixed struct-like records with multiple types. With pickle, any /thought/ of fixed records are tossed out the window with variable-lengthed types like strings, longs, lists, tuples and dicts, and I believe aren't really comparable. Now, variable-lengthed longs packed in little-endian format already have a mechanism for encoding and decoding via pickle.en/decode_long (though it is wholly undocumented), and seemingly is going to get another in binascii. Fixed-lengthed, optional signed/unsigned, optional little-endian/big-endian longs do not have a mechanism for encoding and decoding, which is what I am asking for. I will point out that 128 bit integers are gaining support on newer 32 and 64 bit processors and C compilers for them (SSE on x86, Itanium, etc.). In the future, a new code for these 128 bit integers may be asked for inclusion. With a variable-width integer type, all future "hey, we now have x-byte types in C, where is struct support in Python?", can be answered with the proposed, "choose your integer size" format code. That is to say, this format code is future proof, unless integer types start wandering from integer byte widths. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-09-15 22:52 Message: Logged In: YES user_id=31435 Use cases are important. Oddly(?) enough, I've never had a need for a bigint conversion in a C struct, and have a hard time imagining I will someday. All the cases I've had (and I've had more than a few) were one-shot str->long or long->str conversions. An obvious example in the core is the tedious encode_long() and decode_long() functions in pickle.py. Note that a pickle.encode_long() workalike doesn't know in advance how many bytes it needs, which would make using struct a PITA for that particular use case. If a proposal isn't convenient for taking over existing conversions of this nature, that counts against it. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-15 22:31 Message: Logged In: YES user_id=341410 And as I stated, the 's' format character is also a "variable lengthed type". It just so happens that in most use cases I've had and observed for both the 's' format AND proposed 'g' format, the type size, is in fact, fixed at 'compile' time. It also happens that for the 'g' format, this fixed size is not in the set {1,2,4,8}, which are not limitations for the pre-existing 's' format. Please note that the only fundamental difference between the pre-existing 's' format and the proposed 'g' format, is that of a quick call to appropriate PyLong_* functions, and a range check as required by other integer types. Python is a tool. Struct is a tool. By changing the tool only slightly, we can add flexibility. The code is already there, minor glue would make it work, and would make it convenient for, I believe, more people than binascii. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-15 20:26 Message: Logged In: YES user_id=80475 I agree with Michael and Martin that variable length types do not belong in struct. The module is about working with fixed record layouts. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-15 20:17 Message: Logged In: YES user_id=341410 Raymond, from your first post on this topic, it seems as though you were previously implementing this functionality in binascii for some particular reason, and it seems as though it is to be included with binascii in the future, regardless of the outcome of this particular feature request. The only reason the binascii solution is better than status quo, is because a user doesn't need to implement arbitrarily large integer packing and unpacking themselves. On the other hand, it still requires the user make manual binascii.str_to_long(str_obj) calls in the case of it being part of a struct, so doesn't gain significantly. Now, one of the reasons why I requested a format code addition was because one can (un)pack multiple data types simultaneously with a single function call via struct. In nearly all of the use cases I have for packing and unpacking large integers, they are a part of other structures. In the cases where I have been packing and unpacking single integers, floats, etc., I still use struct because it is has nearly all of the functionality I need (signed, unsigned, big endian, little endian, char, short, long, long long, etc., lacking only arbitrarily large integer packing and unpacking). ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-15 18:08 Message: Logged In: YES user_id=80475 My vote is for binascii functions to parallel hexlify and unhexlify. Ideally, it would give the same result as long(hexlify(s), 16). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-15 14:53 Message: Logged In: YES user_id=341410 As you state, it already supports packing and unpacking a variable-lengthed type: strings. In the use cases I've had and seen for (un)packing strings with struct, it is the most common to define a static format code, and use that all the time. That is, you see things like ">HLLHB25s", that become string constants in a module. On the _very rare_ occasion where people want more flexibility in their types, I have seen both the use of fixed and variable pascal strings... def packit(arg1, arg2, arg3, strng): return struct.pack(">LHH%ip"%len(strng), arg1, arg2, arg3, strng) I would not expect any pascal-string-like packing of a large integer, though it is possible. I do expect that most people have similar use cases as I, and would pre-define their struct formatting code. In the case of other similar requests (long to string, string to long via a base256 representation, etc.) for use in cryptography, I expect that the regularity of structures used in cryptography would almost certainly result in formatting codes being module constants. To sum up, both in the case for the 's' and 'p' format codes, and the proposed 'g'/'G' formatting codes, the vast majority of use cases pre-define the length of the string and large integer on a per-structure basis via "25s", "25p", or "25g". Rarely are the lengths truely variable in the case of "%ip"%len(strng). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-15 13:03 Message: Logged In: YES user_id=6656 Oops, I see you actually address that (ish). But I still feel packing what is an essentially variable length type using the struct module is a bit strange. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-15 13:02 Message: Logged In: YES user_id=6656 Josiah, what do you suggest struct.calcsize does with the format code your proposing? I think this question encapsulates why I find this feature request a bit misdirected. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-12 14:55 Message: Logged In: YES user_id=341410 Hexlify and unhexlify make sense for translating strings. Nowhere in the binascii module is there any mention of translating anything to/from integers, longs or floats. The only reason (un)hexlify make any sense at all is because we can get integers from hexlified strings, and get strings from hexlified integers with relative ease. I guess the trick with struct, at least with me, is not that I use it because it translates to/from C types, it is because it translates to/from types that I find useful. Its intersection with C types, as well as Python's intersection with C types, and my use intersection with the types is a convenient (but very engineered and understandable) coincidence. It would be another very convenient (but also engineered *wink*) coincidence if I didn't have to first extract a section of data, then translate it, in order to get one large integer. In the cases that I would truely find useful, big integers are a part of what would be called structs in the C world, and wouldn't require additional processing over what I'm already doing for other integers and floats. I was looking around, and it turns out that in 2001, Paul Rubin requested that one be able to translate to/from arbitrary bases via the C-level format function. In that discussion, Paul made the case that there should be a method to get arbitrarily long integers to and from strings: "The struct module doesn't give any way of converting arbitrary ints (meaning longs) to binary. Really, it's needed. Some people do it with gmpy, but if Python is going to support longs as a built-in type, one shouldn't have to resort to 3rd-party modules to read and write them in binary." Guido followed up with: "OK, I believe you. Can you submit a patch?" It seems like this was in reference to being able to use functions in binascii for converting to/from arbitrary packed binary integer types in base 256 (http://sourceforge.net/tracker/?func=detail&atid=105470&aid=465045&group_id=5470 if you are interested). That request seems to have died because Paul dropped the ball. Me, I would prefer struct to binascii, if only because the code for doing this is already waiting to be used in struct, and because you can pull multiple objects from a single packed binary string, rather than one object per call. This would seemingly also satisfy complaints of being able to translate to/from base 256 for arbitrarily large integers. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-09-11 23:40 Message: Logged In: YES user_id=31435 binascii makes sense because that's where the hexlify and unhexlify functions live, which are small conceptual steps away from what's needed here. Methods on numbers make sense too, and only seem strange because so few are clearly visible now (although, e.g., there are lots of them already, like number.__abs__ and number.__add__). The struct module makes sense too, although it would be darned ugly to document a refusal to accept the new codes in "native" mode; and struct has a high learning curve; and struct obviously never intended to support types that aren't supplied directly by C compilers (the "Pascal string" code seems weird now, but not at the time). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-10 19:18 Message: Logged In: YES user_id=341410 (sorry it took me a few days to get back to you, I am on a contract deadline crunch...just taking a break now) The *HTTPServer heirarchy is interesting in its own right, but really, each piece in the heirarchy adds functionality. A similar thing can be said of asyncore and all the modules that derive from it (asynchat, *HTTPServer, *XMLRPCServer, smtpd, etc.). In this case, since the struct module is already in C and the functions are not subclassable, creating another module that parses strings and sends pieces off to struct for actual decoding seems like a waste of a module, especially when the change is so minor. Now, binascii is being used in such a fashion by uu and binhex, but that is because binascii is the data processing component, where uu and binhex make a 'pretty' interface. Struct doesn't need a pretty interface, it is already pretty. Though as I have said before, I think it could use this small addition. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-09-08 18:53 Message: Logged In: YES user_id=21627 Since you were asking: it is quite common that modules refer to related functionality. For example, BaseHTTPServer refers to SimpleHTTPServer and CGIHTTPServer. One might expect that a HTTP server also supports files and does CGI - but not this one; go elsewhere. Likewise, module binascii refers to modules uu and binhex. The math documentation points out that it does not support complex numbers, and that cmath is needed. The audioop documentation gives the function echocancel in the documentation, instead of implementing it. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-08 18:38 Message: Logged In: YES user_id=341410 Martin, I was typing as you submitted your most recent comment. I am honestly shocked that you would suggest that longs should gain a method for encoding themselves as binary strings. Such a thing would then suggest that standard ints and floats also gain such methods. It would also imply that since one can go to strings, one should equivalently be able to come from strings via equivalent methods. Goodness, int.tostring(width) and int.fromstring(str)? But what about endianness? Looks like a big can of worms to me. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-08 18:03 Message: Logged In: YES user_id=341410 Structures (aka C Structs) can contain arbitrarily large or small numbers of basic types inside them. As such, 'single long values' are still a valid use. I use struct for packing and unpacking of single items (8,4,2 byte integers, 1 byte integers are faster served via chr and ord) when necessary (because it is the most convenient), as well as a current contract where it is not uncommon to be packing and unpacking 256 byte structs. Those large structs contains various 1,2,4 and 8 byte integers, as well as a handful of 16 and 20 byte integers (which I must manually shift and mask during packing and unpacking). I'm a big boy, and can do it, but that doesn't mean that such functionality should be left out of Python. As for 'document the approach of going through hex inside the documentation of the struct module', I am curious about whether other modules do the same thing, that is to tell users "this functionality conceptually fits here X%, which is why it is documented here, but because it does not fit 100%, here is how you can do the same thing, which will likely look like a strange hack, require slicing potentially large strings, and be significantly slower than if we had just added the functionality, but here you go anyways." Now, I don't /need/ the feature, but I believe myself and others would find it useful. I also don't /require/ it be in struct, but no other modules offer equivalent functionality; Pickle and Marshal are Python-only, binascii (and bin2hex) are for converting between binary and ascii representations for transferring over non-8-bit channels (email, web, etc.), and no other module even comes close to offering a similar bit of "packs various types into a binary format, the same way C would" as struct. If anyone has a better place for it, I'm all ears (or eyes). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-09-08 17:45 Message: Logged In: YES user_id=21627 I would think that def long_as_bytes(lvalue, width): fmt = '%%.%dx' % (2*width) return unhexlify(fmt % (lvalue & ((1L<<8*width)-1))) is short enough for a recipe to not really make a C function necessary for that feature. However, if they are going to be provided somewhere, I would suggest that static methods on the long type might be the right place. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-08 15:30 Message: Logged In: YES user_id=80475 The idea is to expose the _PyLong_FromByteArray() and _PyLong_AsByteArray() functions. While long(hexlify(b),16) is doable for bin2long, going the other way is not so simple. I agree that these are not struct related. Originally, I proposed the binascii module because one of the operations is so similar to hexlify(). As there other suggestions? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-09-08 15:15 Message: Logged In: YES user_id=21627 Apparently, the point of this request is that the method for converting long ints to binary should be "easily found in documentation". And also apparently, the submitter thinks that the struct module would be the place where people look. Now, that allows for a simple solution: document the approach of going through hex inside the documentation of the struct module. There is one other reason (beyond being primarily for C APIs) why such a feature should *not* be in the struct module: The struct module, most naturally, is about structures. However, I understand that the intended usage of this feature would not be structures, but single long values. Therefore, I consider it counter-intuitive to extend struct for that. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-06 20:02 Message: Logged In: YES user_id=80475 Okay, submit a patch with docs and unittests. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-09-06 19:44 Message: Logged In: YES user_id=341410 As I provide in the feature request, there is already a method for translating string <-> long. The problem with current methods for converting between large integers and strings is that they do not lend themselves to generally being understandable or to being documented. The struct module already provides two appropriate functions for handling packed binary data, a place for documenting functions involving packing and unpacking binary data, and whose implementation seems to be simple enough (one more format character, much of which borrowed from 's' character, and a call to _PyLong_FromByteArray seems to be sufficient). As for the binascii module, many of the functions listed seem like they should be wrapped into the encode/decode string methods, hexlify already being so in str.encode('hex'). To me, just being able to translate doesn't seem sufficient (we already can translate), but being able to do it well, have it documented well, and placed in a location that is obvious, fast and optimized for these kinds of things seems to be the right thing. >From what I can tell, the only reason why struct doesn't already have an equivalent format character to the proposed 'g' and 'G', is because the module was created to handle packed C structs and seemingly "nothing else". Considering there doesn't seem to be any other reasonable or easily documentable location for placing equivalent functionality (both packing and unpacking), I am of the opinion that restricting the packing and unpacking to C types in the struct module (when there are other useful types) is overkill. As I said, I will provide an implementation if desired. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-06 18:34 Message: Logged In: YES user_id=80475 FWIW, I'm working str/long conversion functions for the binascii module. Will that suit your needs? The tolong function is equivalent to: long(hexlify(b), 16) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1023290&group_id=5470 From noreply at sourceforge.net Wed Oct 6 04:17:44 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 6 04:17:48 2004 Subject: [ python-Bugs-1039270 ] time zone tests fail on Windows Message-ID: Bugs item #1039270, was opened at 2004-10-02 20:44 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Brett Cannon (bcannon) Summary: time zone tests fail on Windows Initial Comment: Following tests fail on Win 2K(Japanese locale): # test_strptime.py test_compile (__main__.TimeRETests) ... FAIL test_bad_timezone (__main__.StrptimeTests) ... ERROR test_timezone (__main__.StrptimeTests) ... ERROR test_day_of_week_calculation (__main__.CalculationTests) ... ERROR test_gregorian_calculation (__main__.CalculationTests) ... ERROR test_julian_calculation (__main__.CalculationTests) ... ERROR # test_time.py test_strptime (test.test_time.TimeTestCase) ... FAIL === They all stem from time zone tests and can be divided into two groups: FAIL of test_compile is basically same as #bug 883604. http://www.python.org/sf/883604 Local time values include regular expression's metacharacters, but they are not escaped. The rest is caused because strptime can't parse the values of strftime. >>> import time >>> time.tzname ('\x93\x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)', '\x93 \x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)') >>> time.strptime(time.strftime('%Z', time.gmtime())) Traceback (most recent call last): File "", line 1, in -toplevel- time.strptime(time.strftime('%Z', time.gmtime())) File "C:\Python24\lib\_strptime.py", line 291, in strptime raise ValueError("time data did not match format: data=%s fmt=%s" % ValueError: time data did not match format: data=東京 (標準時) fmt=%a %b %d %H:%M:%S %Y The output of running test_time.py and test_strptime.py is attached. ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-10-05 19:17 Message: Logged In: YES user_id=357491 rev. 1.33 on HEAD and rev. 1.23.4.5 on 2.3 have the fix. Thanks for the help, George. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-05 12:12 Message: Logged In: YES user_id=671362 Correct my previous post. There's nothing wrong with strptime on IDLE. >>> import time >>> time.strptime(time.strftime('%Z'), '%Z') (1900, 1, 1, 0, 0, 0, 0, 1, 0) Please close this bug and apply the patches. Thanks Brett! ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-05 11:56 Message: Logged In: YES user_id=671362 bcannon write: > The .lower() call is intended to normalize since capitalization > is not standard across OSs. But if it is a Unicode string it > should be fine. And even if it isn't, it is all lowercased for > comparison anyway, so as long as it is consistent, shouldn't it > still work? Hmm. > As for your example of strptime not being able to parse, you have > a bug in it; you forgot the format string. It should have been > ``time.strptime(time.strftime('%Z'), '%Z')``. Give that a run > and let me know what the output is. Yeah, it's my fault. I forget to specify a format. Even so, strptime couldn't parse timezone. > As for this whole multi-byte issue, is it all being returned as > Unicod e strings, or is it just a regular string? In other > words, what is ``type(time.tzname[0])`` spitting out? And what > character encoding is all of this in (i.e., what should I pass > to unicode so as to not have it raise UnicodeDecodeError)? It returns strings(not a unicode), and the encoding is cp932. This is a default encoding of Japanese Windows. >>> unicode(time.tzname[0], 'cp932') u'\u6771\u4eac (\u6a19\u6e96\u6642)' > And finally, for the regex metacharacter stuff, why the hell ar > e there parentheses in a timezone?!? Whoever decided that wa > s good did it just to upset me. Ask M$ Japan :-; I don't regard 'Tokyo (standard time)' as an acceptable representation for time zone at all, but this is what Windows returns as a time zone on my box. > That does need to be fixed. Apply the patch I just uploaded and let > me know if it at least deals with that problem. With your patch, all tests succeed without any Error or Fail, and strftime <-> strptime conversions work well. This is a backport candidate, so I created a new patch against Python 2.3 with listcomps instead of genexprs. But there is one problem left. On IDLE, strptime still can't parse. I haven't looked into it in details, but probably patch #590913 has something to do with it. This patch sets locale at IDLE's start up time and this can affect behaviors of string-related functions and constants. [PEP 263 support in IDLE] http://www.python.org/sf/590913 # patch applied >>> time.strptime(time.strptime('%Z'), '%Z') Traceback (most recent call last): File "", line 1, in -toplevel- time.strptime(time.strptime('%Z'), '%Z') File "C:\Python24\lib\_strptime.py", line 291, in strptime if not found: ValueError: time data did not match format: data=%Z fmt=% a %b %d %H:%M:%S %Y >>> import locale >>> locale.getlocale() ['Japanese_Japan', '932'] # culprit? > Have I mentioned I hate timezones? In case I haven't, I do. I agree with you one hundred percent. --George ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-03 16:16 Message: Logged In: YES user_id=357491 The .lower() call is intended to normalize since capitalization is not standard across OSs. But if it is a Unicode string it should be fine. And even if it isn't, it is all lowercased for comparison anyway, so as long as it is consistent, shouldn't it still work? As for your example of strptime not being able to parse, you have a bug in it; you forgot the format string. It should have been ``time.strptime(time.strftime('%Z'), '%Z')``. Give that a run and let me know what the output is. As for this whole multi-byte issue, is it all being returned as Unicode strings, or is it just a regular string? In other words, what is ``type(time.tzname[0])`` spitting out? And what character encoding is all of this in (i.e., what should I pass to unicode so as to not have it raise UnicodeDecodeError)? And finally, for the regex metacharacter stuff, why the hell are there parentheses in a timezone?!? Whoever decided that was good did it just to upset me. That does need to be fixed. Apply the patch I just uploaded and let me know if it at least deals with that problem. Have I mentioned I hate timezones? In case I haven't, I do. Thanks for catching this all, though, George. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-03 08:05 Message: Logged In: YES user_id=671362 I've found another bug. Line 167 & 169 of Lib/_strptime.py contains the expression: time.tzname[0].lower() I guess this is intended to normalize alphabets, but for multibyte characters this is really dangerous. >>> import time >>> time.tzname[0] '\x93\x8c\x8b\x9e (\x95W\x8f\x80\x8e\x9e)' >>> _.lower() '\x93\x8c\x8b\x9e (\x95w\x8f\x80\x8e\x9e)' \x95W and \x95w is not the same character. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 From noreply at sourceforge.net Wed Oct 6 04:50:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 6 04:52:18 2004 Subject: [ python-Bugs-1032875 ] ftplib has incomplete transfer when sending files in Windows Message-ID: Bugs item #1032875, was opened at 2004-09-22 21:21 Message generated for change (Comment added) made by grubert You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1032875&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Ed Sanville (infinity314) Assigned to: Nobody/Anonymous (nobody) Summary: ftplib has incomplete transfer when sending files in Windows Initial Comment: I wrote the following short script: from ftplib import FTP ftp=FTP('ftp.edsanville.us', username, password) print ftp.getwelcome() print ftp.storbinary('stor test.xls', file('test.xls', 'r')) ftp.quit() It transfers the entire file under Red Hat Linux 9, but under Windows XP, it only transfers a couple kilobytes at most. Using a hex editor, I found that the first 5 or 6 bytes of the file are OK on the remote side after transfer, but the rest of the file is totally wrong. I'm at a loss, it doesn't make any sense. Please, somebody tell me I'm not insane. ---------------------------------------------------------------------- Comment By: engelbert gruber (grubert) Date: 2004-10-06 04:50 Message: Logged In: YES user_id=147070 For binary files on windows one has to use file('test.xls','rb'). if this works please close the bug. ---------------------------------------------------------------------- Comment By: engelbert gruber (grubert) Date: 2004-10-05 16:19 Message: Logged In: YES user_id=147070 looks like a file problem: f=open('test.xls');d=f.read() reads all the file. on winxp/py234 not making a binary file: for i in range(0,50):for d in range(0,255): f.write(d) reading it: d=f.read() 1.round len(d) = 26, f.tell() 12288 2.round len(d) = 26, f.tell() 12800 chr(26) is the end of a dos textfile. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1032875&group_id=5470 From noreply at sourceforge.net Wed Oct 6 12:36:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 6 12:36:03 2004 Subject: [ python-Bugs-1041325 ] copy.deepcopy exceptions.TypeError: 'NoneType' object ... Message-ID: Bugs item #1041325, was opened at 2004-10-06 10:35 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041325&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Arthur Lutz (arthur_lutz) Assigned to: Nobody/Anonymous (nobody) Summary: copy.deepcopy exceptions.TypeError: 'NoneType' object ... Initial Comment: when using copy.deepcopy on certain objects I get (after a large number of copier/_deepcopy_something): filename="/usr/lib/python2.3/copy.py" line_number="179" function_name="deepcopy" y = copier(x, memo) filename="/usr/lib/python2.3/copy.py" line_number="250" function_name="_deepcopy_tuple" y.append(deepcopy(a, memo)) filename="/usr/lib/python2.3/copy.py" line_number="179" function_name="deepcopy" y = copier(x, memo) filename="/usr/lib/python2.3/copy.py" line_number="270" function_name="_deepcopy_dict" y[deepcopy(key, memo)] = deepcopy(value, memo) filename="/usr/lib/python2.3/copy.py" line_number="179" function_name="deepcopy" y = copier(x, memo) filename="/usr/lib/python2.3/copy.py" line_number="243" function_name="_deepcopy_list" y.append(deepcopy(a, memo)) filename="/usr/lib/python2.3/copy.py" line_number="206" function_name="deepcopy" y = _reconstruct(x, rv, 1, memo) filename="/usr/lib/python2.3/copy.py" line_number="353" function_name="_reconstruct" state = deepcopy(state, memo) filename="/usr/lib/python2.3/copy.py" line_number="179" function_name="deepcopy" y = copier(x, memo) filename="/usr/lib/python2.3/copy.py" line_number="270" function_name="_deepcopy_dict" y[deepcopy(key, memo)] = deepcopy(value, memo) filename="/usr/lib/python2.3/copy.py" line_number="198" function_name="deepcopy" rv = reductor(2) exceptions.TypeError: 'NoneType' object is not callable Hope this is enough traceback ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041325&group_id=5470 From noreply at sourceforge.net Wed Oct 6 16:31:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 6 16:31:46 2004 Subject: [ python-Bugs-1041501 ] SimpleXMLRPCServer example is broken Message-ID: Bugs item #1041501, was opened at 2004-10-06 14:31 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041501&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Kent Johnson (kjohnson) Assigned to: Nobody/Anonymous (nobody) Summary: SimpleXMLRPCServer example is broken Initial Comment: Section 11.21.1 of the library reference (SimpleXMLRPCServer Objects) includes this class definition: class MyFuncs: def div(self, x, y) : return div(x,y) This is given to the sample server as in instance object. The problem is that there is no global div() function so attempting to call div() from XML-RPC returns a Fault with a NameError. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041501&group_id=5470 From noreply at sourceforge.net Wed Oct 6 17:32:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 6 17:32:38 2004 Subject: [ python-Bugs-1041325 ] copy.deepcopy exceptions.TypeError: 'NoneType' object ... Message-ID: Bugs item #1041325, was opened at 2004-10-06 20:35 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041325&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Arthur Lutz (arthur_lutz) Assigned to: Nobody/Anonymous (nobody) Summary: copy.deepcopy exceptions.TypeError: 'NoneType' object ... Initial Comment: when using copy.deepcopy on certain objects I get (after a large number of copier/_deepcopy_something): filename="/usr/lib/python2.3/copy.py" line_number="179" function_name="deepcopy" y = copier(x, memo) filename="/usr/lib/python2.3/copy.py" line_number="250" function_name="_deepcopy_tuple" y.append(deepcopy(a, memo)) filename="/usr/lib/python2.3/copy.py" line_number="179" function_name="deepcopy" y = copier(x, memo) filename="/usr/lib/python2.3/copy.py" line_number="270" function_name="_deepcopy_dict" y[deepcopy(key, memo)] = deepcopy(value, memo) filename="/usr/lib/python2.3/copy.py" line_number="179" function_name="deepcopy" y = copier(x, memo) filename="/usr/lib/python2.3/copy.py" line_number="243" function_name="_deepcopy_list" y.append(deepcopy(a, memo)) filename="/usr/lib/python2.3/copy.py" line_number="206" function_name="deepcopy" y = _reconstruct(x, rv, 1, memo) filename="/usr/lib/python2.3/copy.py" line_number="353" function_name="_reconstruct" state = deepcopy(state, memo) filename="/usr/lib/python2.3/copy.py" line_number="179" function_name="deepcopy" y = copier(x, memo) filename="/usr/lib/python2.3/copy.py" line_number="270" function_name="_deepcopy_dict" y[deepcopy(key, memo)] = deepcopy(value, memo) filename="/usr/lib/python2.3/copy.py" line_number="198" function_name="deepcopy" rv = reductor(2) exceptions.TypeError: 'NoneType' object is not callable Hope this is enough traceback ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 01:32 Message: Logged In: YES user_id=1038590 'Certain objects' is not very specific. Are these builtin objects, or your own classes? (I would assume the latter) Which version of Python 2.3 are you running? ("python -V" will tell you if you don't already know) Also, are you able to retry this with the latest Python 2.4 pre-release? It looks like it has changes which are relevant (i.e there are guards before all of the reductor calls to prevent the above exception). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041325&group_id=5470 From noreply at sourceforge.net Wed Oct 6 17:37:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 6 17:37:24 2004 Subject: [ python-Bugs-1041325 ] copy.deepcopy exceptions.TypeError: 'NoneType' object ... Message-ID: Bugs item #1041325, was opened at 2004-10-06 10:35 Message generated for change (Comment added) made by arthur_lutz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041325&group_id=5470 Category: None Group: None >Status: Deleted Resolution: None Priority: 5 Submitted By: Arthur Lutz (arthur_lutz) Assigned to: Nobody/Anonymous (nobody) Summary: copy.deepcopy exceptions.TypeError: 'NoneType' object ... Initial Comment: when using copy.deepcopy on certain objects I get (after a large number of copier/_deepcopy_something): filename="/usr/lib/python2.3/copy.py" line_number="179" function_name="deepcopy" y = copier(x, memo) filename="/usr/lib/python2.3/copy.py" line_number="250" function_name="_deepcopy_tuple" y.append(deepcopy(a, memo)) filename="/usr/lib/python2.3/copy.py" line_number="179" function_name="deepcopy" y = copier(x, memo) filename="/usr/lib/python2.3/copy.py" line_number="270" function_name="_deepcopy_dict" y[deepcopy(key, memo)] = deepcopy(value, memo) filename="/usr/lib/python2.3/copy.py" line_number="179" function_name="deepcopy" y = copier(x, memo) filename="/usr/lib/python2.3/copy.py" line_number="243" function_name="_deepcopy_list" y.append(deepcopy(a, memo)) filename="/usr/lib/python2.3/copy.py" line_number="206" function_name="deepcopy" y = _reconstruct(x, rv, 1, memo) filename="/usr/lib/python2.3/copy.py" line_number="353" function_name="_reconstruct" state = deepcopy(state, memo) filename="/usr/lib/python2.3/copy.py" line_number="179" function_name="deepcopy" y = copier(x, memo) filename="/usr/lib/python2.3/copy.py" line_number="270" function_name="_deepcopy_dict" y[deepcopy(key, memo)] = deepcopy(value, memo) filename="/usr/lib/python2.3/copy.py" line_number="198" function_name="deepcopy" rv = reductor(2) exceptions.TypeError: 'NoneType' object is not callable Hope this is enough traceback ---------------------------------------------------------------------- >Comment By: Arthur Lutz (arthur_lutz) Date: 2004-10-06 15:37 Message: Logged In: YES user_id=848508 Sorry, The bug was indeed in one of my classes. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 15:32 Message: Logged In: YES user_id=1038590 'Certain objects' is not very specific. Are these builtin objects, or your own classes? (I would assume the latter) Which version of Python 2.3 are you running? ("python -V" will tell you if you don't already know) Also, are you able to retry this with the latest Python 2.4 pre-release? It looks like it has changes which are relevant (i.e there are guards before all of the reductor calls to prevent the above exception). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041325&group_id=5470 From noreply at sourceforge.net Wed Oct 6 20:15:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 6 20:15:42 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 14:15 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: benson margulies (benson_basis) Assigned to: Nobody/Anonymous (nobody) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Wed Oct 6 20:16:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 6 20:16:54 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 14:15 Message generated for change (Settings changed) made by benson_basis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None >Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Nobody/Anonymous (nobody) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Thu Oct 7 01:31:26 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 01:31:30 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-07 04:15 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Nobody/Anonymous (nobody) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 09:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Thu Oct 7 01:32:26 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 01:32:30 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-07 04:15 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Nobody/Anonymous (nobody) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 09:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 09:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Thu Oct 7 01:49:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 01:49:41 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 14:15 Message generated for change (Comment added) made by benson_basis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Nobody/Anonymous (nobody) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- >Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Thu Oct 7 01:56:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 01:56:43 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 14:15 Message generated for change (Comment added) made by benson_basis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Nobody/Anonymous (nobody) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- >Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Thu Oct 7 02:15:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 02:15:50 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-07 04:15 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Nobody/Anonymous (nobody) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 10:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-07 09:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-07 09:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 09:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 09:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Thu Oct 7 02:43:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 02:43:22 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-07 04:15 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Nobody/Anonymous (nobody) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 10:43 Message: Logged In: YES user_id=1038590 *rereads bug description and comments* I think I'm on the right page now. The offending call would appear to be PyMem_NEW inside PyThreadState_New. Under a release build, this resolves into a direct call to malloc(), but under debug it resolves to _PyObjectDebugMalloc. Which is what you said in the bug report - I just read it incorrectly. Sorry for the confusion. I just remembered that this came up a few weeks ago on Python dev, with a proposed solution (changing this particular call to use malloc() directly): http://mail.python.org/pipermail/python-dev/2004-September/048683.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 10:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-07 09:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-07 09:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 09:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 09:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Thu Oct 7 02:46:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 02:46:14 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-07 04:15 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Nobody/Anonymous (nobody) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 10:46 Message: Logged In: YES user_id=1038590 I'd suggest assigning the bug report to Tim Peters (aka tim_one), to see if he still agrees with the fix he proposed last month. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 10:43 Message: Logged In: YES user_id=1038590 *rereads bug description and comments* I think I'm on the right page now. The offending call would appear to be PyMem_NEW inside PyThreadState_New. Under a release build, this resolves into a direct call to malloc(), but under debug it resolves to _PyObjectDebugMalloc. Which is what you said in the bug report - I just read it incorrectly. Sorry for the confusion. I just remembered that this came up a few weeks ago on Python dev, with a proposed solution (changing this particular call to use malloc() directly): http://mail.python.org/pipermail/python-dev/2004-September/048683.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 10:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-07 09:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-07 09:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 09:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 09:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Thu Oct 7 03:57:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 03:57:08 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 14:15 Message generated for change (Comment added) made by benson_basis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Nobody/Anonymous (nobody) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- >Comment By: benson margulies (benson_basis) Date: 2004-10-06 21:57 Message: Logged In: YES user_id=876734 The rather funny-looking initialization was recommended by the author of the PEP. Once InitThreads is called, the GIL is held. To get it unheld, you have to release it. Here's his explanation .. OK - here is my explanation which you can adapt A simple application is one that initializes Python, and then executes some Python code. This Python code may itself cause other threads, or may indirectly cause other applications to "callback" into Python, but once the main thread returns, all the code has finished - you application terminates. With these applications, your init code need take no further action - just initialize Python and execute your code. Only your extension functions or callback entry points need manage the GIL state. python.exe is an example of this kind of app - it initializes Python, then executes the code specified on the command line. More complex applications may need to initialize Python, perform a little bit of initialization, but then do nothing else on that thread. An example would be where that initialization code bootstraps a few threads, then terminates immediately. The application however continues to live until explicitly terminated by the running code. In this scenario, you must use the following code - this code calls into Python. Once this call has been made, the thread still holds the GIL. This thread must release the GIL before other threads can work. [Technical aside: This works in the first case as the main thread which holds the lock continues to run. When it calls out to external libraries and at periodic times during execution, the thread-lock is unlocked allowing other threads to run. Once the main thread returns, it does still hold the lock - but as the application is terminating, that is OK - no other threads are competing for the lock. In our complicated example, the main thread returning does *not* indicate application termination, so the lock must be manually dropped.] That doesn't make a huge amount of sense, but I hope it is useful. All this being said, I have two contradictory bits of evidence. 1) We all agree that the code, as written, would fail due to the unsafe debug malloc. 2) When I reverse the calls, it produces a very different situation. I think, perhaps, that the GIL never gets unlocked properly, and then instead of crashing everything just gets stuck somehow. The change to call malloc seems pretty reasonable to me. Some tests seem called for in the tree on this. If order is not supposed to matter, the test should test both orders. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:46 Message: Logged In: YES user_id=1038590 I'd suggest assigning the bug report to Tim Peters (aka tim_one), to see if he still agrees with the fix he proposed last month. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:43 Message: Logged In: YES user_id=1038590 *rereads bug description and comments* I think I'm on the right page now. The offending call would appear to be PyMem_NEW inside PyThreadState_New. Under a release build, this resolves into a direct call to malloc(), but under debug it resolves to _PyObjectDebugMalloc. Which is what you said in the bug report - I just read it incorrectly. Sorry for the confusion. I just remembered that this came up a few weeks ago on Python dev, with a proposed solution (changing this particular call to use malloc() directly): http://mail.python.org/pipermail/python-dev/2004-September/048683.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Thu Oct 7 04:24:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 04:24:35 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 14:15 Message generated for change (Comment added) made by benson_basis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Nobody/Anonymous (nobody) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- >Comment By: benson margulies (benson_basis) Date: 2004-10-06 22:24 Message: Logged In: YES user_id=876734 I'm afraid that something is more seriously wrong here than just the malloc in Ensure. Here's what I did in our production code. I protected all calls to Ensure with a mutex. The result was a similiar problem with the debug heap happening inside code that was executing under the GIL. Given the structure of our code, I'm reasonably confident that there is no rogue thread running around outside the GIL. In fact, at the time of the explosion here, the other threads are all waiting their turn to get the GIL. btpython23_d.dll!PyObject_Malloc(unsigned int nbytes=98) Line 581 + 0x6 C btpython23_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=82) Line 990 + 0x9 C btpython23_d.dll!PyString_FromStringAndSize(const char * str=0x03176384, int size=50) Line 78 + 0xc C btpython23_d.dll!string_slice(PyStringObject * a=0x03176368, int i=0, int j=50) Line 1006 + 0x17 C btpython23_d.dll!PySequence_GetSlice(_object * s=0x03176368, int i1=0, int i2=50) Line 1218 + 0x12 C btpython23_d.dll!apply_slice(_object * u=0x03176368, _object * v=0x00000000, _object * w=0x02fb7acc) Line 3806 + 0x11 C btpython23_d.dll!eval_frame(_frame * f=0x031aeeb0) Line 1391 + 0x11 C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x030c9130, _object * globals=0x03101ab8, _object * locals=0x00000000, _object * * args=0x0323650c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x030eed0c, int defcount=2, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyEval_CallObjectWithKeywords (_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 3346 + 0x11 C btpython23_d.dll!PyErr_Warn(_object * category=0x0306da10, char * message=0x1e14df70) Line 627 + 0xf C > btpython23_d.dll!builtin_apply(_object * self=0x00000000, _object * args=0x03238a78) Line 80 + 0x10 C btpython23_d.dll!PyCFunction_Call(_object * func=0x0305fb38, _object * arg=0x03238a78, _object * kw=0x00000000) Line 73 + 0xb C btpython23_d.dll!call_function(_object * * * pp_stack=0x0364f2b0, int oparg=2) Line 3439 + 0xf C btpython23_d.dll!eval_frame(_frame * f=0x0314a310) Line 2116 + 0xd C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x03178868, _object * globals=0x03174818, _object * locals=0x00000000, _object * * args=0x0d558f0c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!instancemethod_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 2432 + 0x11 C btpython23_d.dll!PyObject_Call(_object * func=0x0321fd38, _object * arg=0x030728f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyObject_CallMethod(_object * o=0x0317a4f8, char * name=0x64c453f4, char * format=0x64c453f0, ...) Line 1844 + 0xf C btrexcoreC230.dll!BT_REX_LP_PyProxy::Run (BT_Blackboard * ref=0x02eacc88) Line 136 + 0x1b C++ btrexcoreC230.dll!BT_Blackboard::Run() Line 166 + 0x29 C++ btrexcoreC230.dll! BT_REX_ContextImp::ProcessUTF16Buffer(const unsigned short * inbuf=0x00ab78bc, unsigned int inlen=673, int lid=30) Line 609 C++ basistechnology.rlp.dll! BasisTechnology.RLP.ContextImp.ProcessBuffer(String* data = "South African answers U.S. message in a bottle. JOHANNESBURG 08-22-1996 A South African boy is writing back to an American girl whose message in a bottle he found washed up on President Nelson Mandela's old prison island. But Carlo Hoffmann, an 11-year- old jailer's son who found the bottle on the beach at Robben Island off Cape Town after winter storms, will send his letter back by ordinary mail on Thursday, the post office said. It will be sent for free. Danielle Murray from Sandusky, Ohio, the same age as her new penfriend, asked for a reply from whoever received the message she flung on its journey months ago on the other side of the Atlantic Ocean. ", BasisTechnology.RLP.LanguageID __unnamed001 = ENGLISH) Line 84 + 0x33 bytes C++ ThreadStress.exe! BasisTechnology.RLPUnitTest.Worker.Go() Line 63 C# ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 21:57 Message: Logged In: YES user_id=876734 The rather funny-looking initialization was recommended by the author of the PEP. Once InitThreads is called, the GIL is held. To get it unheld, you have to release it. Here's his explanation .. OK - here is my explanation which you can adapt A simple application is one that initializes Python, and then executes some Python code. This Python code may itself cause other threads, or may indirectly cause other applications to "callback" into Python, but once the main thread returns, all the code has finished - you application terminates. With these applications, your init code need take no further action - just initialize Python and execute your code. Only your extension functions or callback entry points need manage the GIL state. python.exe is an example of this kind of app - it initializes Python, then executes the code specified on the command line. More complex applications may need to initialize Python, perform a little bit of initialization, but then do nothing else on that thread. An example would be where that initialization code bootstraps a few threads, then terminates immediately. The application however continues to live until explicitly terminated by the running code. In this scenario, you must use the following code - this code calls into Python. Once this call has been made, the thread still holds the GIL. This thread must release the GIL before other threads can work. [Technical aside: This works in the first case as the main thread which holds the lock continues to run. When it calls out to external libraries and at periodic times during execution, the thread-lock is unlocked allowing other threads to run. Once the main thread returns, it does still hold the lock - but as the application is terminating, that is OK - no other threads are competing for the lock. In our complicated example, the main thread returning does *not* indicate application termination, so the lock must be manually dropped.] That doesn't make a huge amount of sense, but I hope it is useful. All this being said, I have two contradictory bits of evidence. 1) We all agree that the code, as written, would fail due to the unsafe debug malloc. 2) When I reverse the calls, it produces a very different situation. I think, perhaps, that the GIL never gets unlocked properly, and then instead of crashing everything just gets stuck somehow. The change to call malloc seems pretty reasonable to me. Some tests seem called for in the tree on this. If order is not supposed to matter, the test should test both orders. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:46 Message: Logged In: YES user_id=1038590 I'd suggest assigning the bug report to Tim Peters (aka tim_one), to see if he still agrees with the fix he proposed last month. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:43 Message: Logged In: YES user_id=1038590 *rereads bug description and comments* I think I'm on the right page now. The offending call would appear to be PyMem_NEW inside PyThreadState_New. Under a release build, this resolves into a direct call to malloc(), but under debug it resolves to _PyObjectDebugMalloc. Which is what you said in the bug report - I just read it incorrectly. Sorry for the confusion. I just remembered that this came up a few weeks ago on Python dev, with a proposed solution (changing this particular call to use malloc() directly): http://mail.python.org/pipermail/python-dev/2004-September/048683.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Thu Oct 7 04:35:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 04:35:53 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-07 04:15 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Nobody/Anonymous (nobody) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 12:35 Message: Logged In: YES user_id=1038590 I'm well out of my depth now - I definitely recommend assigning the bug report to Tim (I'd do it, but I can't edit bugs I didn't create). ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-07 12:24 Message: Logged In: YES user_id=876734 I'm afraid that something is more seriously wrong here than just the malloc in Ensure. Here's what I did in our production code. I protected all calls to Ensure with a mutex. The result was a similiar problem with the debug heap happening inside code that was executing under the GIL. Given the structure of our code, I'm reasonably confident that there is no rogue thread running around outside the GIL. In fact, at the time of the explosion here, the other threads are all waiting their turn to get the GIL. btpython23_d.dll!PyObject_Malloc(unsigned int nbytes=98) Line 581 + 0x6 C btpython23_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=82) Line 990 + 0x9 C btpython23_d.dll!PyString_FromStringAndSize(const char * str=0x03176384, int size=50) Line 78 + 0xc C btpython23_d.dll!string_slice(PyStringObject * a=0x03176368, int i=0, int j=50) Line 1006 + 0x17 C btpython23_d.dll!PySequence_GetSlice(_object * s=0x03176368, int i1=0, int i2=50) Line 1218 + 0x12 C btpython23_d.dll!apply_slice(_object * u=0x03176368, _object * v=0x00000000, _object * w=0x02fb7acc) Line 3806 + 0x11 C btpython23_d.dll!eval_frame(_frame * f=0x031aeeb0) Line 1391 + 0x11 C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x030c9130, _object * globals=0x03101ab8, _object * locals=0x00000000, _object * * args=0x0323650c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x030eed0c, int defcount=2, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyEval_CallObjectWithKeywords (_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 3346 + 0x11 C btpython23_d.dll!PyErr_Warn(_object * category=0x0306da10, char * message=0x1e14df70) Line 627 + 0xf C > btpython23_d.dll!builtin_apply(_object * self=0x00000000, _object * args=0x03238a78) Line 80 + 0x10 C btpython23_d.dll!PyCFunction_Call(_object * func=0x0305fb38, _object * arg=0x03238a78, _object * kw=0x00000000) Line 73 + 0xb C btpython23_d.dll!call_function(_object * * * pp_stack=0x0364f2b0, int oparg=2) Line 3439 + 0xf C btpython23_d.dll!eval_frame(_frame * f=0x0314a310) Line 2116 + 0xd C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x03178868, _object * globals=0x03174818, _object * locals=0x00000000, _object * * args=0x0d558f0c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!instancemethod_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 2432 + 0x11 C btpython23_d.dll!PyObject_Call(_object * func=0x0321fd38, _object * arg=0x030728f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyObject_CallMethod(_object * o=0x0317a4f8, char * name=0x64c453f4, char * format=0x64c453f0, ...) Line 1844 + 0xf C btrexcoreC230.dll!BT_REX_LP_PyProxy::Run (BT_Blackboard * ref=0x02eacc88) Line 136 + 0x1b C++ btrexcoreC230.dll!BT_Blackboard::Run() Line 166 + 0x29 C++ btrexcoreC230.dll! BT_REX_ContextImp::ProcessUTF16Buffer(const unsigned short * inbuf=0x00ab78bc, unsigned int inlen=673, int lid=30) Line 609 C++ basistechnology.rlp.dll! BasisTechnology.RLP.ContextImp.ProcessBuffer(String* data = "South African answers U.S. message in a bottle. JOHANNESBURG 08-22-1996 A South African boy is writing back to an American girl whose message in a bottle he found washed up on President Nelson Mandela's old prison island. But Carlo Hoffmann, an 11-year- old jailer's son who found the bottle on the beach at Robben Island off Cape Town after winter storms, will send his letter back by ordinary mail on Thursday, the post office said. It will be sent for free. Danielle Murray from Sandusky, Ohio, the same age as her new penfriend, asked for a reply from whoever received the message she flung on its journey months ago on the other side of the Atlantic Ocean. ", BasisTechnology.RLP.LanguageID __unnamed001 = ENGLISH) Line 84 + 0x33 bytes C++ ThreadStress.exe! BasisTechnology.RLPUnitTest.Worker.Go() Line 63 C# ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-07 11:57 Message: Logged In: YES user_id=876734 The rather funny-looking initialization was recommended by the author of the PEP. Once InitThreads is called, the GIL is held. To get it unheld, you have to release it. Here's his explanation .. OK - here is my explanation which you can adapt A simple application is one that initializes Python, and then executes some Python code. This Python code may itself cause other threads, or may indirectly cause other applications to "callback" into Python, but once the main thread returns, all the code has finished - you application terminates. With these applications, your init code need take no further action - just initialize Python and execute your code. Only your extension functions or callback entry points need manage the GIL state. python.exe is an example of this kind of app - it initializes Python, then executes the code specified on the command line. More complex applications may need to initialize Python, perform a little bit of initialization, but then do nothing else on that thread. An example would be where that initialization code bootstraps a few threads, then terminates immediately. The application however continues to live until explicitly terminated by the running code. In this scenario, you must use the following code - this code calls into Python. Once this call has been made, the thread still holds the GIL. This thread must release the GIL before other threads can work. [Technical aside: This works in the first case as the main thread which holds the lock continues to run. When it calls out to external libraries and at periodic times during execution, the thread-lock is unlocked allowing other threads to run. Once the main thread returns, it does still hold the lock - but as the application is terminating, that is OK - no other threads are competing for the lock. In our complicated example, the main thread returning does *not* indicate application termination, so the lock must be manually dropped.] That doesn't make a huge amount of sense, but I hope it is useful. All this being said, I have two contradictory bits of evidence. 1) We all agree that the code, as written, would fail due to the unsafe debug malloc. 2) When I reverse the calls, it produces a very different situation. I think, perhaps, that the GIL never gets unlocked properly, and then instead of crashing everything just gets stuck somehow. The change to call malloc seems pretty reasonable to me. Some tests seem called for in the tree on this. If order is not supposed to matter, the test should test both orders. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 10:46 Message: Logged In: YES user_id=1038590 I'd suggest assigning the bug report to Tim Peters (aka tim_one), to see if he still agrees with the fix he proposed last month. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 10:43 Message: Logged In: YES user_id=1038590 *rereads bug description and comments* I think I'm on the right page now. The offending call would appear to be PyMem_NEW inside PyThreadState_New. Under a release build, this resolves into a direct call to malloc(), but under debug it resolves to _PyObjectDebugMalloc. Which is what you said in the bug report - I just read it incorrectly. Sorry for the confusion. I just remembered that this came up a few weeks ago on Python dev, with a proposed solution (changing this particular call to use malloc() directly): http://mail.python.org/pipermail/python-dev/2004-September/048683.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 10:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-07 09:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-07 09:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 09:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-07 09:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Thu Oct 7 06:00:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 06:00:39 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 13:15 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) >Assigned to: Tim Peters (tim_one) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 21:35 Message: Logged In: YES user_id=1038590 I'm well out of my depth now - I definitely recommend assigning the bug report to Tim (I'd do it, but I can't edit bugs I didn't create). ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 21:24 Message: Logged In: YES user_id=876734 I'm afraid that something is more seriously wrong here than just the malloc in Ensure. Here's what I did in our production code. I protected all calls to Ensure with a mutex. The result was a similiar problem with the debug heap happening inside code that was executing under the GIL. Given the structure of our code, I'm reasonably confident that there is no rogue thread running around outside the GIL. In fact, at the time of the explosion here, the other threads are all waiting their turn to get the GIL. btpython23_d.dll!PyObject_Malloc(unsigned int nbytes=98) Line 581 + 0x6 C btpython23_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=82) Line 990 + 0x9 C btpython23_d.dll!PyString_FromStringAndSize(const char * str=0x03176384, int size=50) Line 78 + 0xc C btpython23_d.dll!string_slice(PyStringObject * a=0x03176368, int i=0, int j=50) Line 1006 + 0x17 C btpython23_d.dll!PySequence_GetSlice(_object * s=0x03176368, int i1=0, int i2=50) Line 1218 + 0x12 C btpython23_d.dll!apply_slice(_object * u=0x03176368, _object * v=0x00000000, _object * w=0x02fb7acc) Line 3806 + 0x11 C btpython23_d.dll!eval_frame(_frame * f=0x031aeeb0) Line 1391 + 0x11 C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x030c9130, _object * globals=0x03101ab8, _object * locals=0x00000000, _object * * args=0x0323650c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x030eed0c, int defcount=2, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyEval_CallObjectWithKeywords (_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 3346 + 0x11 C btpython23_d.dll!PyErr_Warn(_object * category=0x0306da10, char * message=0x1e14df70) Line 627 + 0xf C > btpython23_d.dll!builtin_apply(_object * self=0x00000000, _object * args=0x03238a78) Line 80 + 0x10 C btpython23_d.dll!PyCFunction_Call(_object * func=0x0305fb38, _object * arg=0x03238a78, _object * kw=0x00000000) Line 73 + 0xb C btpython23_d.dll!call_function(_object * * * pp_stack=0x0364f2b0, int oparg=2) Line 3439 + 0xf C btpython23_d.dll!eval_frame(_frame * f=0x0314a310) Line 2116 + 0xd C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x03178868, _object * globals=0x03174818, _object * locals=0x00000000, _object * * args=0x0d558f0c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!instancemethod_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 2432 + 0x11 C btpython23_d.dll!PyObject_Call(_object * func=0x0321fd38, _object * arg=0x030728f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyObject_CallMethod(_object * o=0x0317a4f8, char * name=0x64c453f4, char * format=0x64c453f0, ...) Line 1844 + 0xf C btrexcoreC230.dll!BT_REX_LP_PyProxy::Run (BT_Blackboard * ref=0x02eacc88) Line 136 + 0x1b C++ btrexcoreC230.dll!BT_Blackboard::Run() Line 166 + 0x29 C++ btrexcoreC230.dll! BT_REX_ContextImp::ProcessUTF16Buffer(const unsigned short * inbuf=0x00ab78bc, unsigned int inlen=673, int lid=30) Line 609 C++ basistechnology.rlp.dll! BasisTechnology.RLP.ContextImp.ProcessBuffer(String* data = "South African answers U.S. message in a bottle. JOHANNESBURG 08-22-1996 A South African boy is writing back to an American girl whose message in a bottle he found washed up on President Nelson Mandela's old prison island. But Carlo Hoffmann, an 11-year- old jailer's son who found the bottle on the beach at Robben Island off Cape Town after winter storms, will send his letter back by ordinary mail on Thursday, the post office said. It will be sent for free. Danielle Murray from Sandusky, Ohio, the same age as her new penfriend, asked for a reply from whoever received the message she flung on its journey months ago on the other side of the Atlantic Ocean. ", BasisTechnology.RLP.LanguageID __unnamed001 = ENGLISH) Line 84 + 0x33 bytes C++ ThreadStress.exe! BasisTechnology.RLPUnitTest.Worker.Go() Line 63 C# ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 20:57 Message: Logged In: YES user_id=876734 The rather funny-looking initialization was recommended by the author of the PEP. Once InitThreads is called, the GIL is held. To get it unheld, you have to release it. Here's his explanation .. OK - here is my explanation which you can adapt A simple application is one that initializes Python, and then executes some Python code. This Python code may itself cause other threads, or may indirectly cause other applications to "callback" into Python, but once the main thread returns, all the code has finished - you application terminates. With these applications, your init code need take no further action - just initialize Python and execute your code. Only your extension functions or callback entry points need manage the GIL state. python.exe is an example of this kind of app - it initializes Python, then executes the code specified on the command line. More complex applications may need to initialize Python, perform a little bit of initialization, but then do nothing else on that thread. An example would be where that initialization code bootstraps a few threads, then terminates immediately. The application however continues to live until explicitly terminated by the running code. In this scenario, you must use the following code - this code calls into Python. Once this call has been made, the thread still holds the GIL. This thread must release the GIL before other threads can work. [Technical aside: This works in the first case as the main thread which holds the lock continues to run. When it calls out to external libraries and at periodic times during execution, the thread-lock is unlocked allowing other threads to run. Once the main thread returns, it does still hold the lock - but as the application is terminating, that is OK - no other threads are competing for the lock. In our complicated example, the main thread returning does *not* indicate application termination, so the lock must be manually dropped.] That doesn't make a huge amount of sense, but I hope it is useful. All this being said, I have two contradictory bits of evidence. 1) We all agree that the code, as written, would fail due to the unsafe debug malloc. 2) When I reverse the calls, it produces a very different situation. I think, perhaps, that the GIL never gets unlocked properly, and then instead of crashing everything just gets stuck somehow. The change to call malloc seems pretty reasonable to me. Some tests seem called for in the tree on this. If order is not supposed to matter, the test should test both orders. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:46 Message: Logged In: YES user_id=1038590 I'd suggest assigning the bug report to Tim Peters (aka tim_one), to see if he still agrees with the fix he proposed last month. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:43 Message: Logged In: YES user_id=1038590 *rereads bug description and comments* I think I'm on the right page now. The offending call would appear to be PyMem_NEW inside PyThreadState_New. Under a release build, this resolves into a direct call to malloc(), but under debug it resolves to _PyObjectDebugMalloc. Which is what you said in the bug report - I just read it incorrectly. Sorry for the confusion. I just remembered that this came up a few weeks ago on Python dev, with a proposed solution (changing this particular call to use malloc() directly): http://mail.python.org/pipermail/python-dev/2004-September/048683.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 18:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 18:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 18:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 18:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Thu Oct 7 13:20:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 13:20:27 2004 Subject: [ python-Bugs-1030629 ] PyOS_InputHook broken Message-ID: Bugs item #1030629, was opened at 2004-09-19 11:27 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Michael Hudson (mwh) Summary: PyOS_InputHook broken Initial Comment: PyOS_InputHook is a pointer to a function to be called periodically when Python is idle. It is used, for example, to get messages delivered to graphics windows. In Python 2.3.4, calling PyOS_InputHook was handled by readline via rl_event_hook = PyOS_InputHook; ... p = readline(prompt); The readline library takes care of calling rl_event_hook (and hence PyOS_InputHook) ten times per second while Python idles at the call to readline(prompt). In Python 2.4a3, the call to readline is replaced by a call to readline_until_enter_or_signal. Here, the "select" function is used: has_input = select(fileno(rl_instream) + 1, &selectset, NULL, NULL, NULL); if(has_input > 0) { rl_callback_read_char(); } Now Python idles at "select", but PyOS_InputHook is not being called. Instead, PyOS_InputHook gets called via rl_event_hook at rl_callback_read_char. This means that PyOS_InputHook is called once for every character the user types, and is not called at all if the user is not typing something. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-07 12:20 Message: Logged In: YES user_id=6656 er, ping? i'm willing to believe this is a fairly bad bug, but i could do with some encouragement to fix it :-) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-19 19:35 Message: Logged In: YES user_id=6656 Well, this is probably my fault. However, I'm about to go on holiday for a week and thinking about readline is definitely not part of my plans :-) Can you work up a patch for this? It shouldn't be terribly hard -- a non-NULL fifth argument to select seems in order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 From noreply at sourceforge.net Thu Oct 7 14:02:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 14:02:34 2004 Subject: [ python-Bugs-1042179 ] Patch Message-ID: Bugs item #1042179, was opened at 2004-10-07 21:02 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042179&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Nobody/Anonymous (nobody) Summary: Patch Initial Comment: I submitted a patch a couple of weeks ago. It's available from the the bug tracker on sourceforge. Same place as where you are reading this message. Let me know if you can't find the patch there. Actually I assumed you would get a message automatically when I submitted the patch, but apparently that didn't happen. One comment about the patch: I wasn't able to check this patch on Windows, because I don't know how to compile Python there. I would like to check it on Windows though, because there were some inconsistencies in the behavior of PyOS_InputHook between Windows and other platforms in previous Python versions. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042179&group_id=5470 From noreply at sourceforge.net Thu Oct 7 14:07:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 14:07:23 2004 Subject: [ python-Bugs-1042179 ] Patch Message-ID: Bugs item #1042179, was opened at 2004-10-07 13:02 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042179&group_id=5470 Category: None Group: None >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Nobody/Anonymous (nobody) Summary: Patch Initial Comment: I submitted a patch a couple of weeks ago. It's available from the the bug tracker on sourceforge. Same place as where you are reading this message. Let me know if you can't find the patch there. Actually I assumed you would get a message automatically when I submitted the patch, but apparently that didn't happen. One comment about the patch: I wasn't able to check this patch on Windows, because I don't know how to compile Python there. I would like to check it on Windows though, because there were some inconsistencies in the behavior of PyOS_InputHook between Windows and other platforms in previous Python versions. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-07 13:07 Message: Logged In: YES user_id=6656 Oops. No notfication goes out when a file is attached to a report, so I simply didn't know. Sorry about that. Filing a new bug report as a way of getting in touch with one particular developer is ... uh ... strange. You can add comments to existing issues, you know :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042179&group_id=5470 From noreply at sourceforge.net Thu Oct 7 14:08:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 14:08:39 2004 Subject: [ python-Bugs-1030629 ] PyOS_InputHook broken Message-ID: Bugs item #1030629, was opened at 2004-09-19 19:27 Message generated for change (Comment added) made by mdehoon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Michael Hudson (mwh) Summary: PyOS_InputHook broken Initial Comment: PyOS_InputHook is a pointer to a function to be called periodically when Python is idle. It is used, for example, to get messages delivered to graphics windows. In Python 2.3.4, calling PyOS_InputHook was handled by readline via rl_event_hook = PyOS_InputHook; ... p = readline(prompt); The readline library takes care of calling rl_event_hook (and hence PyOS_InputHook) ten times per second while Python idles at the call to readline(prompt). In Python 2.4a3, the call to readline is replaced by a call to readline_until_enter_or_signal. Here, the "select" function is used: has_input = select(fileno(rl_instream) + 1, &selectset, NULL, NULL, NULL); if(has_input > 0) { rl_callback_read_char(); } Now Python idles at "select", but PyOS_InputHook is not being called. Instead, PyOS_InputHook gets called via rl_event_hook at rl_callback_read_char. This means that PyOS_InputHook is called once for every character the user types, and is not called at all if the user is not typing something. ---------------------------------------------------------------------- >Comment By: Michiel de Hoon (mdehoon) Date: 2004-10-07 21:08 Message: Logged In: YES user_id=488897 I submitted a patch a couple of weeks ago. Let me know if you can't find it on sourceforge. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 20:20 Message: Logged In: YES user_id=6656 er, ping? i'm willing to believe this is a fairly bad bug, but i could do with some encouragement to fix it :-) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-20 03:35 Message: Logged In: YES user_id=6656 Well, this is probably my fault. However, I'm about to go on holiday for a week and thinking about readline is definitely not part of my plans :-) Can you work up a patch for this? It shouldn't be terribly hard -- a non-NULL fifth argument to select seems in order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 From noreply at sourceforge.net Thu Oct 7 14:10:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 14:10:40 2004 Subject: [ python-Bugs-1030629 ] PyOS_InputHook broken Message-ID: Bugs item #1030629, was opened at 2004-09-19 11:27 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Michael Hudson (mwh) Summary: PyOS_InputHook broken Initial Comment: PyOS_InputHook is a pointer to a function to be called periodically when Python is idle. It is used, for example, to get messages delivered to graphics windows. In Python 2.3.4, calling PyOS_InputHook was handled by readline via rl_event_hook = PyOS_InputHook; ... p = readline(prompt); The readline library takes care of calling rl_event_hook (and hence PyOS_InputHook) ten times per second while Python idles at the call to readline(prompt). In Python 2.4a3, the call to readline is replaced by a call to readline_until_enter_or_signal. Here, the "select" function is used: has_input = select(fileno(rl_instream) + 1, &selectset, NULL, NULL, NULL); if(has_input > 0) { rl_callback_read_char(); } Now Python idles at "select", but PyOS_InputHook is not being called. Instead, PyOS_InputHook gets called via rl_event_hook at rl_callback_read_char. This means that PyOS_InputHook is called once for every character the user types, and is not called at all if the user is not typing something. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-07 13:10 Message: Logged In: YES user_id=6656 Oops, didn't see the patch. While I'm updating my checkout and testing the patch, my first thought is that you probably still want to assign to rl_event_hook, just in case we're *not* using the callback interface to readline (old versions of readline for example). ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2004-10-07 13:08 Message: Logged In: YES user_id=488897 I submitted a patch a couple of weeks ago. Let me know if you can't find it on sourceforge. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 12:20 Message: Logged In: YES user_id=6656 er, ping? i'm willing to believe this is a fairly bad bug, but i could do with some encouragement to fix it :-) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-19 19:35 Message: Logged In: YES user_id=6656 Well, this is probably my fault. However, I'm about to go on holiday for a week and thinking about readline is definitely not part of my plans :-) Can you work up a patch for this? It shouldn't be terribly hard -- a non-NULL fifth argument to select seems in order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 From noreply at sourceforge.net Thu Oct 7 14:21:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 14:22:02 2004 Subject: [ python-Bugs-1030629 ] PyOS_InputHook broken Message-ID: Bugs item #1030629, was opened at 2004-09-19 19:27 Message generated for change (Comment added) made by mdehoon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Michael Hudson (mwh) Summary: PyOS_InputHook broken Initial Comment: PyOS_InputHook is a pointer to a function to be called periodically when Python is idle. It is used, for example, to get messages delivered to graphics windows. In Python 2.3.4, calling PyOS_InputHook was handled by readline via rl_event_hook = PyOS_InputHook; ... p = readline(prompt); The readline library takes care of calling rl_event_hook (and hence PyOS_InputHook) ten times per second while Python idles at the call to readline(prompt). In Python 2.4a3, the call to readline is replaced by a call to readline_until_enter_or_signal. Here, the "select" function is used: has_input = select(fileno(rl_instream) + 1, &selectset, NULL, NULL, NULL); if(has_input > 0) { rl_callback_read_char(); } Now Python idles at "select", but PyOS_InputHook is not being called. Instead, PyOS_InputHook gets called via rl_event_hook at rl_callback_read_char. This means that PyOS_InputHook is called once for every character the user types, and is not called at all if the user is not typing something. ---------------------------------------------------------------------- >Comment By: Michiel de Hoon (mdehoon) Date: 2004-10-07 21:21 Message: Logged In: YES user_id=488897 That is right. I didn't think about that. You might consider assigning to rl_event_hook inside the second readline_until_enter_or_signal (the one if not defined(HAVE_RL_CALLBACK) && defined(HAVE_SELECT)), because it is only needed there. Though it probably wouldn't hurt to set rl_event_hook for both cases. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 21:10 Message: Logged In: YES user_id=6656 Oops, didn't see the patch. While I'm updating my checkout and testing the patch, my first thought is that you probably still want to assign to rl_event_hook, just in case we're *not* using the callback interface to readline (old versions of readline for example). ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2004-10-07 21:08 Message: Logged In: YES user_id=488897 I submitted a patch a couple of weeks ago. Let me know if you can't find it on sourceforge. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 20:20 Message: Logged In: YES user_id=6656 er, ping? i'm willing to believe this is a fairly bad bug, but i could do with some encouragement to fix it :-) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-20 03:35 Message: Logged In: YES user_id=6656 Well, this is probably my fault. However, I'm about to go on holiday for a week and thinking about readline is definitely not part of my plans :-) Can you work up a patch for this? It shouldn't be terribly hard -- a non-NULL fifth argument to select seems in order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 From noreply at sourceforge.net Thu Oct 7 15:35:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 15:35:41 2004 Subject: [ python-Bugs-1042238 ] Lib/compiler chokes on certain genexps Message-ID: Bugs item #1042238, was opened at 2004-10-07 14:35 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: Lib/compiler chokes on certain genexps Initial Comment: Noticed this because test_compiler fails on _strptime as of yesterday. Here's what seems to be a fairly minimal example: f((x for x in y), 1) The traceback is huge, but ends with: File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 687, in power node = self.com_apply_trailer(node, elt) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1144, in com_apply_trailer return self.com_call_function(primaryNode, nodelist[2]) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1169, in com_call_function raise SyntaxError, 'generator expression needs parenthesis' SyntaxError: generator expression needs parenthesis Might be a problem in transformer.py, not really sure. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 From noreply at sourceforge.net Thu Oct 7 15:38:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 15:38:25 2004 Subject: [ python-Bugs-1042238 ] Lib/compiler chokes on certain genexps Message-ID: Bugs item #1042238, was opened at 2004-10-07 14:35 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: Lib/compiler chokes on certain genexps Initial Comment: Noticed this because test_compiler fails on _strptime as of yesterday. Here's what seems to be a fairly minimal example: f((x for x in y), 1) The traceback is huge, but ends with: File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 687, in power node = self.com_apply_trailer(node, elt) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1144, in com_apply_trailer return self.com_call_function(primaryNode, nodelist[2]) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1169, in com_call_function raise SyntaxError, 'generator expression needs parenthesis' SyntaxError: generator expression needs parenthesis Might be a problem in transformer.py, not really sure. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-07 14:38 Message: Logged In: YES user_id=6656 Hang on, the exception is *inside* transformer.py. Duh. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 From noreply at sourceforge.net Thu Oct 7 15:47:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 15:47:42 2004 Subject: [ python-Bugs-1030629 ] PyOS_InputHook broken Message-ID: Bugs item #1030629, was opened at 2004-09-19 11:27 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Michael Hudson (mwh) Summary: PyOS_InputHook broken Initial Comment: PyOS_InputHook is a pointer to a function to be called periodically when Python is idle. It is used, for example, to get messages delivered to graphics windows. In Python 2.3.4, calling PyOS_InputHook was handled by readline via rl_event_hook = PyOS_InputHook; ... p = readline(prompt); The readline library takes care of calling rl_event_hook (and hence PyOS_InputHook) ten times per second while Python idles at the call to readline(prompt). In Python 2.4a3, the call to readline is replaced by a call to readline_until_enter_or_signal. Here, the "select" function is used: has_input = select(fileno(rl_instream) + 1, &selectset, NULL, NULL, NULL); if(has_input > 0) { rl_callback_read_char(); } Now Python idles at "select", but PyOS_InputHook is not being called. Instead, PyOS_InputHook gets called via rl_event_hook at rl_callback_read_char. This means that PyOS_InputHook is called once for every character the user types, and is not called at all if the user is not typing something. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-07 14:47 Message: Logged In: YES user_id=6656 Ok, I made a couple of trivial changes, including moving the rl_event_hook assignment. Seems to work for me. Modules/readline.c revision 2.78. Thanks for the patch! ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2004-10-07 13:21 Message: Logged In: YES user_id=488897 That is right. I didn't think about that. You might consider assigning to rl_event_hook inside the second readline_until_enter_or_signal (the one if not defined(HAVE_RL_CALLBACK) && defined(HAVE_SELECT)), because it is only needed there. Though it probably wouldn't hurt to set rl_event_hook for both cases. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 13:10 Message: Logged In: YES user_id=6656 Oops, didn't see the patch. While I'm updating my checkout and testing the patch, my first thought is that you probably still want to assign to rl_event_hook, just in case we're *not* using the callback interface to readline (old versions of readline for example). ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2004-10-07 13:08 Message: Logged In: YES user_id=488897 I submitted a patch a couple of weeks ago. Let me know if you can't find it on sourceforge. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 12:20 Message: Logged In: YES user_id=6656 er, ping? i'm willing to believe this is a fairly bad bug, but i could do with some encouragement to fix it :-) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-19 19:35 Message: Logged In: YES user_id=6656 Well, this is probably my fault. However, I'm about to go on holiday for a week and thinking about readline is definitely not part of my plans :-) Can you work up a patch for this? It shouldn't be terribly hard -- a non-NULL fifth argument to select seems in order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 From noreply at sourceforge.net Thu Oct 7 22:57:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 22:57:24 2004 Subject: [ python-Bugs-751124 ] Enhancements to pdb.py when invoked as script Message-ID: Bugs item #751124, was opened at 2003-06-09 04:19 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=751124&group_id=5470 Category: Python Library Group: Feature Request Status: Open Resolution: None Priority: 5 Submitted By: Christian Stork (cst) >Assigned to: Johannes Gijsbers (jlgijsbers) Summary: Enhancements to pdb.py when invoked as script Initial Comment: Three little enhancements/mini-bugs: 1) Adding the following lines to the "main part" of pdb.py gives it the ability to stay in pdb if the program crashes. Very handy! # Ensure that we end up in pdb if we crash def pdb_on_crash(type, value, tb): sys.__excepthook__(type, value, tb) post_mortem(tb) sys.excepthook = pdb_on_crash 2) There's also a problem in the last lines of pdb.py: # Insert script directory in front of module search path sys.path.insert(0, os.path.dirname(filename)) This leaves the directory path under which pdb.py was invoked in sys.path. This is a problem if pdb.py was invoked e.g. via a link a la /usr/pdb. This leads to the last and related suggestion. 3) It would be very convenient if a pyhon installation would also install a link /usr/pdb -> /usr/lib/python2.3/pdb.py. This provides for easy invocation and, for example, it's also the default command used by Emacs' pdb-mode. ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-07 22:57 Message: Logged In: YES user_id=469548 Note that 3 is no longer necessary now we have 'python -m'. ---------------------------------------------------------------------- Comment By: Ilya Sandler (isandler) Date: 2004-03-12 18:24 Message: Logged In: YES user_id=971153 a note for the reviewer: patch #896011 fixes bug (2) and implements (in a different manner but with the same result) feature (1), PS. I would also like to add my vote for (3). Being a command line tool pdb.py deserves to be installed right next to python binary..(on Linux that would often be /usr/bin) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=751124&group_id=5470 From noreply at sourceforge.net Thu Oct 7 23:13:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 7 23:13:35 2004 Subject: [ python-Bugs-1025127 ] shutils.rmtree() uses excessive amounts of memory Message-ID: Bugs item #1025127, was opened at 2004-09-09 15:52 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1025127&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: James Henstridge (jhenstridge) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: shutils.rmtree() uses excessive amounts of memory Initial Comment: The shutils.rmtree() implementation uses an excessive amount of memory when deleting large directory heirarchies. Before actually deleting any files, it builds up a list of (function, filename) tuples for all the files that it is going to remove. If there are a lot of files, this results in a lot of memory for a large heirarchy (I had a Python process using 800MB in one case). I'm not sure why it is doing things this way. It isn't using the list to avoid recursion, so the depth of directories it can remove is still limited by Python's recursion limit. Replacing _build_cmdtuple() with a generator might be a good way to reduce the memory usage while leaving the rest of the code unchanged. I checked in CVS, and this issue is still present on HEAD. ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-07 23:13 Message: Logged In: YES user_id=469548 I just looked at my own patch again, added the _raise_err function back in and checked it in as rev 1.33 of shutil.py. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-09-12 00:56 Message: Logged In: YES user_id=31435 I don't really have time for a thorough review. I'll note that stuff like func = something1 arg = something2 func(arg) looks, to my eye, like a convoluted way to say something1(something2) I suppose that's to keep the onerror= gimmick working, though. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-09-11 23:36 Message: Logged In: YES user_id=469548 Please attach changes as a patch next time. I've attached shutils-rmtree.py as a patch this time. I gave it a quick review and added a test to test_shutil.py to ensure some not-very-obvious behavior (don't delete a path passed to rmtree if it's a file) would be preserved. The new version seems fine to me. Tim, could you take a look at it as well? ---------------------------------------------------------------------- Comment By: James Henstridge (jhenstridge) Date: 2004-09-10 03:33 Message: Logged In: YES user_id=146903 Attached is a Python file including a fixed up shutils.rmtree() using os.walk(). It seems to work for me, and should have the same error behaviour. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-09-09 16:17 Message: Logged In: YES user_id=31435 Rewrite it using os.walk() (not os.path.walk()) with topdown=False. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1025127&group_id=5470 From noreply at sourceforge.net Fri Oct 8 17:44:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 8 17:44:20 2004 Subject: [ python-Bugs-1043134 ] mimetypes.guess_extension('text/plain') == '.ksh' ??? Message-ID: Bugs item #1043134, was opened at 2004-10-08 17:44 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1043134&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Robert Kiendl (kxroberto) Assigned to: Nobody/Anonymous (nobody) Summary: mimetypes.guess_extension('text/plain') == '.ksh' ??? Initial Comment: Instead of returning the first in the list of extensions it should return the most reasonable . here: to have a *.txt on disk after saveing? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1043134&group_id=5470 From noreply at sourceforge.net Fri Oct 8 20:29:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 8 20:29:57 2004 Subject: [ python-Bugs-1031897 ] Conflicting descriptions of application order of decorators Message-ID: Bugs item #1031897, was opened at 2004-09-21 09:44 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1031897&group_id=5470 Category: Documentation Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 6 Submitted By: Hamish Lawson (hamishlawson) Assigned to: A.M. Kuchling (akuchling) Summary: Conflicting descriptions of application order of decorators Initial Comment: The explanation of decorators in "What's new in Python 2.4" has the application order opposite to that stated in the Reference Manual. ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-10-08 14:29 Message: Logged In: YES user_id=11375 Fixed; thanks! ---------------------------------------------------------------------- Comment By: Philippe Fremy (pfremy) Date: 2004-09-21 11:16 Message: Logged In: YES user_id=233844 And the PEP 318 is in conflict with the "What's new in Python 2.4" but in accordance with the manual. So it is very probably a problem in the what'snew. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1031897&group_id=5470 From noreply at sourceforge.net Fri Oct 8 20:34:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 8 20:34:39 2004 Subject: [ python-Bugs-1041501 ] SimpleXMLRPCServer example is broken Message-ID: Bugs item #1041501, was opened at 2004-10-06 10:31 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041501&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Kent Johnson (kjohnson) >Assigned to: A.M. Kuchling (akuchling) Summary: SimpleXMLRPCServer example is broken Initial Comment: Section 11.21.1 of the library reference (SimpleXMLRPCServer Objects) includes this class definition: class MyFuncs: def div(self, x, y) : return div(x,y) This is given to the sample server as in instance object. The problem is that there is no global div() function so attempting to call div() from XML-RPC returns a Fault with a NameError. ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-10-08 14:34 Message: Logged In: YES user_id=11375 Fixed; thanks! ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041501&group_id=5470 From noreply at sourceforge.net Fri Oct 8 20:57:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 8 20:57:19 2004 Subject: [ python-Bugs-1022311 ] x, y in curses window object documentation Message-ID: Bugs item #1022311, was opened at 2004-09-04 08:54 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1022311&group_id=5470 Category: Documentation Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Felix Wiemann (felixwiemann) >Assigned to: A.M. Kuchling (akuchling) Summary: x, y in curses window object documentation Initial Comment: lib/curses-window-objects.html says "x, y" several times for method parameters. I think this is wrong; it must be "y, x". ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-10-08 14:57 Message: Logged In: YES user_id=11375 Fixed; thanks! (The only exception is ungetmouse(), which really is x,y. Oh, well.) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1022311&group_id=5470 From noreply at sourceforge.net Sat Oct 9 06:26:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 9 06:26:43 2004 Subject: [ python-Bugs-1043446 ] Uninterruptable loop Message-ID: Bugs item #1043446, was opened at 2004-10-09 00:26 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1043446&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Jp Calderone (kuran) Assigned to: Nobody/Anonymous (nobody) Summary: Uninterruptable loop Initial Comment: import itertools list(itertools.repeat('x')) ^C will not interrupt this. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1043446&group_id=5470 From noreply at sourceforge.net Sat Oct 9 06:52:57 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 9 06:53:05 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 14:15 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Tim Peters (tim_one) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-09 00:52 Message: Logged In: YES user_id=31435 Random: - Arguments "1 100 40" don't make sense on my box, as MAXIMUM_WAIT_OBJECTS is 64, and so the call to WaitForMultipleObjects() fails immediately when passed 100 handles. - The failures in a debug build are much more dramatic if the memory leak is repaired (as Nick said, "trash" leaks; freeing the memory again is much more effective at stressing the heap). - Is benson_basis the KSR Benson M? If so, you'll appreciate that debugging someone else's thread problems is a lot harder than resolving the Banach-Tarski Paradox . - While I was a fan of Mark's new thread API, I've never used it, so it may take some time to figure out what this is all about. Alas, spare time is in short supply. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 22:35 Message: Logged In: YES user_id=1038590 I'm well out of my depth now - I definitely recommend assigning the bug report to Tim (I'd do it, but I can't edit bugs I didn't create). ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 22:24 Message: Logged In: YES user_id=876734 I'm afraid that something is more seriously wrong here than just the malloc in Ensure. Here's what I did in our production code. I protected all calls to Ensure with a mutex. The result was a similiar problem with the debug heap happening inside code that was executing under the GIL. Given the structure of our code, I'm reasonably confident that there is no rogue thread running around outside the GIL. In fact, at the time of the explosion here, the other threads are all waiting their turn to get the GIL. btpython23_d.dll!PyObject_Malloc(unsigned int nbytes=98) Line 581 + 0x6 C btpython23_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=82) Line 990 + 0x9 C btpython23_d.dll!PyString_FromStringAndSize(const char * str=0x03176384, int size=50) Line 78 + 0xc C btpython23_d.dll!string_slice(PyStringObject * a=0x03176368, int i=0, int j=50) Line 1006 + 0x17 C btpython23_d.dll!PySequence_GetSlice(_object * s=0x03176368, int i1=0, int i2=50) Line 1218 + 0x12 C btpython23_d.dll!apply_slice(_object * u=0x03176368, _object * v=0x00000000, _object * w=0x02fb7acc) Line 3806 + 0x11 C btpython23_d.dll!eval_frame(_frame * f=0x031aeeb0) Line 1391 + 0x11 C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x030c9130, _object * globals=0x03101ab8, _object * locals=0x00000000, _object * * args=0x0323650c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x030eed0c, int defcount=2, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyEval_CallObjectWithKeywords (_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 3346 + 0x11 C btpython23_d.dll!PyErr_Warn(_object * category=0x0306da10, char * message=0x1e14df70) Line 627 + 0xf C > btpython23_d.dll!builtin_apply(_object * self=0x00000000, _object * args=0x03238a78) Line 80 + 0x10 C btpython23_d.dll!PyCFunction_Call(_object * func=0x0305fb38, _object * arg=0x03238a78, _object * kw=0x00000000) Line 73 + 0xb C btpython23_d.dll!call_function(_object * * * pp_stack=0x0364f2b0, int oparg=2) Line 3439 + 0xf C btpython23_d.dll!eval_frame(_frame * f=0x0314a310) Line 2116 + 0xd C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x03178868, _object * globals=0x03174818, _object * locals=0x00000000, _object * * args=0x0d558f0c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!instancemethod_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 2432 + 0x11 C btpython23_d.dll!PyObject_Call(_object * func=0x0321fd38, _object * arg=0x030728f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyObject_CallMethod(_object * o=0x0317a4f8, char * name=0x64c453f4, char * format=0x64c453f0, ...) Line 1844 + 0xf C btrexcoreC230.dll!BT_REX_LP_PyProxy::Run (BT_Blackboard * ref=0x02eacc88) Line 136 + 0x1b C++ btrexcoreC230.dll!BT_Blackboard::Run() Line 166 + 0x29 C++ btrexcoreC230.dll! BT_REX_ContextImp::ProcessUTF16Buffer(const unsigned short * inbuf=0x00ab78bc, unsigned int inlen=673, int lid=30) Line 609 C++ basistechnology.rlp.dll! BasisTechnology.RLP.ContextImp.ProcessBuffer(String* data = "South African answers U.S. message in a bottle. JOHANNESBURG 08-22-1996 A South African boy is writing back to an American girl whose message in a bottle he found washed up on President Nelson Mandela's old prison island. But Carlo Hoffmann, an 11-year- old jailer's son who found the bottle on the beach at Robben Island off Cape Town after winter storms, will send his letter back by ordinary mail on Thursday, the post office said. It will be sent for free. Danielle Murray from Sandusky, Ohio, the same age as her new penfriend, asked for a reply from whoever received the message she flung on its journey months ago on the other side of the Atlantic Ocean. ", BasisTechnology.RLP.LanguageID __unnamed001 = ENGLISH) Line 84 + 0x33 bytes C++ ThreadStress.exe! BasisTechnology.RLPUnitTest.Worker.Go() Line 63 C# ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 21:57 Message: Logged In: YES user_id=876734 The rather funny-looking initialization was recommended by the author of the PEP. Once InitThreads is called, the GIL is held. To get it unheld, you have to release it. Here's his explanation .. OK - here is my explanation which you can adapt A simple application is one that initializes Python, and then executes some Python code. This Python code may itself cause other threads, or may indirectly cause other applications to "callback" into Python, but once the main thread returns, all the code has finished - you application terminates. With these applications, your init code need take no further action - just initialize Python and execute your code. Only your extension functions or callback entry points need manage the GIL state. python.exe is an example of this kind of app - it initializes Python, then executes the code specified on the command line. More complex applications may need to initialize Python, perform a little bit of initialization, but then do nothing else on that thread. An example would be where that initialization code bootstraps a few threads, then terminates immediately. The application however continues to live until explicitly terminated by the running code. In this scenario, you must use the following code - this code calls into Python. Once this call has been made, the thread still holds the GIL. This thread must release the GIL before other threads can work. [Technical aside: This works in the first case as the main thread which holds the lock continues to run. When it calls out to external libraries and at periodic times during execution, the thread-lock is unlocked allowing other threads to run. Once the main thread returns, it does still hold the lock - but as the application is terminating, that is OK - no other threads are competing for the lock. In our complicated example, the main thread returning does *not* indicate application termination, so the lock must be manually dropped.] That doesn't make a huge amount of sense, but I hope it is useful. All this being said, I have two contradictory bits of evidence. 1) We all agree that the code, as written, would fail due to the unsafe debug malloc. 2) When I reverse the calls, it produces a very different situation. I think, perhaps, that the GIL never gets unlocked properly, and then instead of crashing everything just gets stuck somehow. The change to call malloc seems pretty reasonable to me. Some tests seem called for in the tree on this. If order is not supposed to matter, the test should test both orders. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:46 Message: Logged In: YES user_id=1038590 I'd suggest assigning the bug report to Tim Peters (aka tim_one), to see if he still agrees with the fix he proposed last month. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:43 Message: Logged In: YES user_id=1038590 *rereads bug description and comments* I think I'm on the right page now. The offending call would appear to be PyMem_NEW inside PyThreadState_New. Under a release build, this resolves into a direct call to malloc(), but under debug it resolves to _PyObjectDebugMalloc. Which is what you said in the bug report - I just read it incorrectly. Sorry for the confusion. I just remembered that this came up a few weeks ago on Python dev, with a proposed solution (changing this particular call to use malloc() directly): http://mail.python.org/pipermail/python-dev/2004-September/048683.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Sat Oct 9 07:04:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 9 07:04:33 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 14:15 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Tim Peters (tim_one) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-09 01:04 Message: Logged In: YES user_id=31435 For the record, in a release build this reliably dies, on a hyper- threaded 3.2GHz box, with """ Fatal Python error: auto-releasing thread-state, but no thread-state for this thread """ given arguments "200 64 100". WRT this failure, it doesn't matter in which order Py_Initialize(); PyEval_InitThreads(); are done. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 00:52 Message: Logged In: YES user_id=31435 Random: - Arguments "1 100 40" don't make sense on my box, as MAXIMUM_WAIT_OBJECTS is 64, and so the call to WaitForMultipleObjects() fails immediately when passed 100 handles. - The failures in a debug build are much more dramatic if the memory leak is repaired (as Nick said, "trash" leaks; freeing the memory again is much more effective at stressing the heap). - Is benson_basis the KSR Benson M? If so, you'll appreciate that debugging someone else's thread problems is a lot harder than resolving the Banach-Tarski Paradox . - While I was a fan of Mark's new thread API, I've never used it, so it may take some time to figure out what this is all about. Alas, spare time is in short supply. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 22:35 Message: Logged In: YES user_id=1038590 I'm well out of my depth now - I definitely recommend assigning the bug report to Tim (I'd do it, but I can't edit bugs I didn't create). ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 22:24 Message: Logged In: YES user_id=876734 I'm afraid that something is more seriously wrong here than just the malloc in Ensure. Here's what I did in our production code. I protected all calls to Ensure with a mutex. The result was a similiar problem with the debug heap happening inside code that was executing under the GIL. Given the structure of our code, I'm reasonably confident that there is no rogue thread running around outside the GIL. In fact, at the time of the explosion here, the other threads are all waiting their turn to get the GIL. btpython23_d.dll!PyObject_Malloc(unsigned int nbytes=98) Line 581 + 0x6 C btpython23_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=82) Line 990 + 0x9 C btpython23_d.dll!PyString_FromStringAndSize(const char * str=0x03176384, int size=50) Line 78 + 0xc C btpython23_d.dll!string_slice(PyStringObject * a=0x03176368, int i=0, int j=50) Line 1006 + 0x17 C btpython23_d.dll!PySequence_GetSlice(_object * s=0x03176368, int i1=0, int i2=50) Line 1218 + 0x12 C btpython23_d.dll!apply_slice(_object * u=0x03176368, _object * v=0x00000000, _object * w=0x02fb7acc) Line 3806 + 0x11 C btpython23_d.dll!eval_frame(_frame * f=0x031aeeb0) Line 1391 + 0x11 C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x030c9130, _object * globals=0x03101ab8, _object * locals=0x00000000, _object * * args=0x0323650c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x030eed0c, int defcount=2, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyEval_CallObjectWithKeywords (_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 3346 + 0x11 C btpython23_d.dll!PyErr_Warn(_object * category=0x0306da10, char * message=0x1e14df70) Line 627 + 0xf C > btpython23_d.dll!builtin_apply(_object * self=0x00000000, _object * args=0x03238a78) Line 80 + 0x10 C btpython23_d.dll!PyCFunction_Call(_object * func=0x0305fb38, _object * arg=0x03238a78, _object * kw=0x00000000) Line 73 + 0xb C btpython23_d.dll!call_function(_object * * * pp_stack=0x0364f2b0, int oparg=2) Line 3439 + 0xf C btpython23_d.dll!eval_frame(_frame * f=0x0314a310) Line 2116 + 0xd C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x03178868, _object * globals=0x03174818, _object * locals=0x00000000, _object * * args=0x0d558f0c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!instancemethod_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 2432 + 0x11 C btpython23_d.dll!PyObject_Call(_object * func=0x0321fd38, _object * arg=0x030728f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyObject_CallMethod(_object * o=0x0317a4f8, char * name=0x64c453f4, char * format=0x64c453f0, ...) Line 1844 + 0xf C btrexcoreC230.dll!BT_REX_LP_PyProxy::Run (BT_Blackboard * ref=0x02eacc88) Line 136 + 0x1b C++ btrexcoreC230.dll!BT_Blackboard::Run() Line 166 + 0x29 C++ btrexcoreC230.dll! BT_REX_ContextImp::ProcessUTF16Buffer(const unsigned short * inbuf=0x00ab78bc, unsigned int inlen=673, int lid=30) Line 609 C++ basistechnology.rlp.dll! BasisTechnology.RLP.ContextImp.ProcessBuffer(String* data = "South African answers U.S. message in a bottle. JOHANNESBURG 08-22-1996 A South African boy is writing back to an American girl whose message in a bottle he found washed up on President Nelson Mandela's old prison island. But Carlo Hoffmann, an 11-year- old jailer's son who found the bottle on the beach at Robben Island off Cape Town after winter storms, will send his letter back by ordinary mail on Thursday, the post office said. It will be sent for free. Danielle Murray from Sandusky, Ohio, the same age as her new penfriend, asked for a reply from whoever received the message she flung on its journey months ago on the other side of the Atlantic Ocean. ", BasisTechnology.RLP.LanguageID __unnamed001 = ENGLISH) Line 84 + 0x33 bytes C++ ThreadStress.exe! BasisTechnology.RLPUnitTest.Worker.Go() Line 63 C# ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 21:57 Message: Logged In: YES user_id=876734 The rather funny-looking initialization was recommended by the author of the PEP. Once InitThreads is called, the GIL is held. To get it unheld, you have to release it. Here's his explanation .. OK - here is my explanation which you can adapt A simple application is one that initializes Python, and then executes some Python code. This Python code may itself cause other threads, or may indirectly cause other applications to "callback" into Python, but once the main thread returns, all the code has finished - you application terminates. With these applications, your init code need take no further action - just initialize Python and execute your code. Only your extension functions or callback entry points need manage the GIL state. python.exe is an example of this kind of app - it initializes Python, then executes the code specified on the command line. More complex applications may need to initialize Python, perform a little bit of initialization, but then do nothing else on that thread. An example would be where that initialization code bootstraps a few threads, then terminates immediately. The application however continues to live until explicitly terminated by the running code. In this scenario, you must use the following code - this code calls into Python. Once this call has been made, the thread still holds the GIL. This thread must release the GIL before other threads can work. [Technical aside: This works in the first case as the main thread which holds the lock continues to run. When it calls out to external libraries and at periodic times during execution, the thread-lock is unlocked allowing other threads to run. Once the main thread returns, it does still hold the lock - but as the application is terminating, that is OK - no other threads are competing for the lock. In our complicated example, the main thread returning does *not* indicate application termination, so the lock must be manually dropped.] That doesn't make a huge amount of sense, but I hope it is useful. All this being said, I have two contradictory bits of evidence. 1) We all agree that the code, as written, would fail due to the unsafe debug malloc. 2) When I reverse the calls, it produces a very different situation. I think, perhaps, that the GIL never gets unlocked properly, and then instead of crashing everything just gets stuck somehow. The change to call malloc seems pretty reasonable to me. Some tests seem called for in the tree on this. If order is not supposed to matter, the test should test both orders. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:46 Message: Logged In: YES user_id=1038590 I'd suggest assigning the bug report to Tim Peters (aka tim_one), to see if he still agrees with the fix he proposed last month. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:43 Message: Logged In: YES user_id=1038590 *rereads bug description and comments* I think I'm on the right page now. The offending call would appear to be PyMem_NEW inside PyThreadState_New. Under a release build, this resolves into a direct call to malloc(), but under debug it resolves to _PyObjectDebugMalloc. Which is what you said in the bug report - I just read it incorrectly. Sorry for the confusion. I just remembered that this came up a few weeks ago on Python dev, with a proposed solution (changing this particular call to use malloc() directly): http://mail.python.org/pipermail/python-dev/2004-September/048683.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Sat Oct 9 17:26:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 9 17:26:54 2004 Subject: [ python-Bugs-1043134 ] mimetypes.guess_extension('text/plain') == '.ksh' ??? Message-ID: Bugs item #1043134, was opened at 2004-10-08 17:44 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1043134&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Robert Kiendl (kxroberto) Assigned to: Nobody/Anonymous (nobody) Summary: mimetypes.guess_extension('text/plain') == '.ksh' ??? Initial Comment: Instead of returning the first in the list of extensions it should return the most reasonable . here: to have a *.txt on disk after saveing? ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-09 17:26 Message: Logged In: YES user_id=469548 How would you suggest finding out what the most reasonable extension for a mime type is? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1043134&group_id=5470 From noreply at sourceforge.net Sat Oct 9 23:22:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 9 23:22:33 2004 Subject: [ python-Bugs-1032960 ] Confusing description of strict option for email.Parser Message-ID: Bugs item #1032960, was opened at 2004-09-22 16:50 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1032960&group_id=5470 Category: Documentation >Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Andrew Bennetts (spiv) Assigned to: Barry A. Warsaw (bwarsaw) Summary: Confusing description of strict option for email.Parser Initial Comment: The docs at http://www.python.org/dev/doc/devel/lib/node534.html say: "Normally, when things like MIME terminating boundaries are missing, or when messages contain other formatting problems, the Parser will raise a MessageParseError. However, when lax parsing is enabled, the Parser will attempt to work around such broken formatting ..." Which strongly suggests that the 'strict' flag defaults to True. However, it later says: "The strict flag defaults to False since lax parsing usually provides the most convenient behavior." I suggest the paragraph be reworded to avoid giving the impression that strict is True by default. Perhaps something like: "The optional strict flag specifies whether strict or lax parsing should be performed. When things like MIME terminating boundaries are missing, or when messages contain other formatting problems, the Parser will raise a MessageParseError if the strict flag is True. Otherwise, it uses lax parsing, and the Parser will attempt to work around such broken formatting to produce a usable message structure (this doesn't mean MessageParseErrors are never raised; some ill-formatted messages just can't be parsed). The strict flag defaults to False since lax parsing usually provides the most convenient behavior." ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-10-09 17:22 Message: Logged In: YES user_id=12800 Changing group to Python 2.3, since the 'strict' argument is deprecated in Python 2.4. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1032960&group_id=5470 From noreply at sourceforge.net Sat Oct 9 23:29:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 9 23:29:53 2004 Subject: [ python-Bugs-1032960 ] Confusing description of strict option for email.Parser Message-ID: Bugs item #1032960, was opened at 2004-09-22 16:50 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1032960&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Andrew Bennetts (spiv) Assigned to: Barry A. Warsaw (bwarsaw) Summary: Confusing description of strict option for email.Parser Initial Comment: The docs at http://www.python.org/dev/doc/devel/lib/node534.html say: "Normally, when things like MIME terminating boundaries are missing, or when messages contain other formatting problems, the Parser will raise a MessageParseError. However, when lax parsing is enabled, the Parser will attempt to work around such broken formatting ..." Which strongly suggests that the 'strict' flag defaults to True. However, it later says: "The strict flag defaults to False since lax parsing usually provides the most convenient behavior." I suggest the paragraph be reworded to avoid giving the impression that strict is True by default. Perhaps something like: "The optional strict flag specifies whether strict or lax parsing should be performed. When things like MIME terminating boundaries are missing, or when messages contain other formatting problems, the Parser will raise a MessageParseError if the strict flag is True. Otherwise, it uses lax parsing, and the Parser will attempt to work around such broken formatting to produce a usable message structure (this doesn't mean MessageParseErrors are never raised; some ill-formatted messages just can't be parsed). The strict flag defaults to False since lax parsing usually provides the most convenient behavior." ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-10-09 17:29 Message: Logged In: YES user_id=12800 I've improved the docs for the Python 2.3 version; it should show up in Python 2.3.5. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-10-09 17:22 Message: Logged In: YES user_id=12800 Changing group to Python 2.3, since the 'strict' argument is deprecated in Python 2.4. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1032960&group_id=5470 From noreply at sourceforge.net Sat Oct 9 23:42:13 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 9 23:42:20 2004 Subject: [ python-Bugs-964433 ] email package uses \n to rebuild content of a message Message-ID: Bugs item #964433, was opened at 2004-06-01 13:57 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=964433&group_id=5470 Category: Extension Modules >Group: Python 2.3 >Status: Closed >Resolution: Wont Fix Priority: 5 Submitted By: Marco Bizzarri (emmebi) Assigned to: Barry A. Warsaw (bwarsaw) Summary: email package uses \n to rebuild content of a message Initial Comment: As stated, the email.Parser class uses '\n' to add the firstbodyline to the rest of the message. This is done *AFTER* the splitlines() have been used to remove the first line from the body of a multipart message. Even though this is not a problem in many cases, it can be a great problem when you are dealing with signed files, as in my case. I've indeed a multipart message where I have: a pdf file a pkcs7 signature If I use the parser to analyze the message, the pdf file is actually one byte less, because the original file was \r\n terminated, rather than \n. When the parser tries to parse, it splits the first line (containing the %PDF1.4\r\n), and translates it to %PDF1.4, and then it is joined to the rest of the PDF file using a simple \n. In this way, the file is exactly one byte less of the original file, and, therefore, the signature can't be verified. I think we could avoid this problem using a splitlines(1)[0][:-1] which would keep the original \r\n, remove the \n, which can then be safely added. ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-10-09 17:42 Message: Logged In: YES user_id=12800 Doesn't this only happen if there is no separating blank line between the headers and the body, as mandated by RFC 2822? I'm not inclined to fix such problems because 1) it's not standard compliant; 2) Python 2.4's parser is completely rewritten so this analysis won't be relevant anymore. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=964433&group_id=5470 From noreply at sourceforge.net Sat Oct 9 23:44:26 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 9 23:44:29 2004 Subject: [ python-Bugs-1010102 ] smtpd.py docstring says wrong default class Message-ID: Bugs item #1010102, was opened at 2004-08-16 11:57 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010102&group_id=5470 Category: Python Library Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Chris Green (chrisgreen) Assigned to: Barry A. Warsaw (bwarsaw) Summary: smtpd.py docstring says wrong default class Initial Comment: Python 2.3.3 smtpd.py Says it uses SMTPProxy by default where it's actually "PureProxy" -c classname Use `classname' as the concrete SMTP proxy class. Uses `SMTPProxy' by default. [...] class Options: setuid = 1 classname = 'PureProxy' ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-10-09 17:44 Message: Logged In: YES user_id=12800 Fixed in Python 2.3 and 2.4. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010102&group_id=5470 From noreply at sourceforge.net Sat Oct 9 23:50:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 9 23:51:03 2004 Subject: [ python-Bugs-950747 ] email: mishandles Content-Transfer-Enc. for text/* messages Message-ID: Bugs item #950747, was opened at 2004-05-09 06:19 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950747&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Closed >Resolution: Wont Fix Priority: 5 Submitted By: Matthias Klose (doko) Assigned to: Barry A. Warsaw (bwarsaw) Summary: email: mishandles Content-Transfer-Enc. for text/* messages Initial Comment: [forwarded from http://bugs.debian.org/247792] Normally when creating MIME attachements with the email module, the user must explicitly encode (base64 or quoted-printable) the message using encode_base64() or encode_quopri(), for example. That sets the Content-Transfer-Encoding header, and replaces the payload with an encoded version which is then emitted verbatim by the Generator. However, for text/ objects, the Content-Transfer-Encoding header is set when the payload is attached, according to the character set. The payload is not encoded at that point, but is instead encoded according to the charset's default body encoding by the Generator at flatten() time. This means in particular that using encode_*() on a text/* message with a non-None charset will result in a malformed message: it will have multiple Content-Transfer-Encoding headers, and multiply encoded body. For added confusion the multiple body encodings won't even be applied in the same order as the duplicate headers appear (as they would if multiple encode_*() functions were applied in sequence). It also means it is impossible to override a charset's default encoding. For example utf-8 text will always be base64 encoded, even though utf-8 text in western languages is likely to be almost entirely 7bit, so quoted-printable or even 8bit encoding would be more appropriate. The payload data of the Message object should consistently hold either encoded or unencoded data. If the latter, the Generator should take its cue from the Content-Transfer-Encoding header, not from the charset. If the former, the encode_* functions should recognize an already encoded message and recode it, or at least throw an exception rather than generating a malformed MIME message. ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-10-09 17:50 Message: Logged In: YES user_id=12800 In Python 2.3, the _encoder argument to MIMEText.__init__() is deprecated, and in Python 2.4 it is removed. The right way to encode the payload for a text/* message is to provide the charset, not to provide an _encoder. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950747&group_id=5470 From noreply at sourceforge.net Sun Oct 10 01:00:08 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 10 01:00:13 2004 Subject: [ python-Bugs-1030941 ] Email message croaks the new email pkg parser Message-ID: Bugs item #1030941, was opened at 2004-09-19 20:34 Message generated for change (Settings changed) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030941&group_id=5470 Category: Python Library Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Skip Montanaro (montanaro) Assigned to: Barry A. Warsaw (bwarsaw) Summary: Email message croaks the new email pkg parser Initial Comment: The new parser for the email package croaks with an AssertionError on the attached email message. Here's the traceback: Traceback (most recent call last): File "/Users/skip/local/lib/python2.4/site-packages/ spambayes/mboxutils.py", line 129, in get_message msg = email.message_from_string(obj) File "/Users/skip/local/lib/python2.4/email/__init__.py", line 45, in message_from_string return Parser(_class, strict=strict).parsestr(s) File "/Users/skip/local/lib/python2.4/email/Parser.py", line 67, in parsestr return self.parse(StringIO(text), headersonly=headersonly) File "/Users/skip/local/lib/python2.4/email/Parser.py", line 56, in parse feedparser.feed(data) File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 145, in feed self._call_parse() File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 149, in _call_parse self._parse() File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 359, in _parsegen assert capturing_preamble Assigning to Barry on the outside chance that he'll have enough round tuits to fix this before the 2.4 release. ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-10-09 19:00 Message: Logged In: YES user_id=12800 I figured out what was wrong with msg_42.txt -- it has no start boundary. In that case the epilogue between the --BBB-- end boundary and the --AAA-- end boundary wasn't being consumed. Fixed in FeedParser.py 1.11, with a test case. Note that while the message will get parsed without an assertion now, it the innermost message (i.e. the [*] below) will have a StartBoundaryNotFoundDefect set. structure of msg_42.txt: multipart/mixed text/plain message/rfc822 multipart/mixed [*] ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-09-29 23:59 Message: Logged In: YES user_id=12800 msg_42.txt is the boiled down example that evokes the assertion. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030941&group_id=5470 From noreply at sourceforge.net Sun Oct 10 02:40:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 10 02:40:56 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 14:15 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Tim Peters (tim_one) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-09 20:40 Message: Logged In: YES user_id=31435 One problem here is that the portable TLS implementation isn't actually threadsafe -- find_key() can get fatally confused if another thread is deleting a key at the same time. That appears to account for the release-build crashes in my immediately preceding comment. That should be repaired soon. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 01:04 Message: Logged In: YES user_id=31435 For the record, in a release build this reliably dies, on a hyper- threaded 3.2GHz box, with """ Fatal Python error: auto-releasing thread-state, but no thread-state for this thread """ given arguments "200 64 100". WRT this failure, it doesn't matter in which order Py_Initialize(); PyEval_InitThreads(); are done. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 00:52 Message: Logged In: YES user_id=31435 Random: - Arguments "1 100 40" don't make sense on my box, as MAXIMUM_WAIT_OBJECTS is 64, and so the call to WaitForMultipleObjects() fails immediately when passed 100 handles. - The failures in a debug build are much more dramatic if the memory leak is repaired (as Nick said, "trash" leaks; freeing the memory again is much more effective at stressing the heap). - Is benson_basis the KSR Benson M? If so, you'll appreciate that debugging someone else's thread problems is a lot harder than resolving the Banach-Tarski Paradox . - While I was a fan of Mark's new thread API, I've never used it, so it may take some time to figure out what this is all about. Alas, spare time is in short supply. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 22:35 Message: Logged In: YES user_id=1038590 I'm well out of my depth now - I definitely recommend assigning the bug report to Tim (I'd do it, but I can't edit bugs I didn't create). ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 22:24 Message: Logged In: YES user_id=876734 I'm afraid that something is more seriously wrong here than just the malloc in Ensure. Here's what I did in our production code. I protected all calls to Ensure with a mutex. The result was a similiar problem with the debug heap happening inside code that was executing under the GIL. Given the structure of our code, I'm reasonably confident that there is no rogue thread running around outside the GIL. In fact, at the time of the explosion here, the other threads are all waiting their turn to get the GIL. btpython23_d.dll!PyObject_Malloc(unsigned int nbytes=98) Line 581 + 0x6 C btpython23_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=82) Line 990 + 0x9 C btpython23_d.dll!PyString_FromStringAndSize(const char * str=0x03176384, int size=50) Line 78 + 0xc C btpython23_d.dll!string_slice(PyStringObject * a=0x03176368, int i=0, int j=50) Line 1006 + 0x17 C btpython23_d.dll!PySequence_GetSlice(_object * s=0x03176368, int i1=0, int i2=50) Line 1218 + 0x12 C btpython23_d.dll!apply_slice(_object * u=0x03176368, _object * v=0x00000000, _object * w=0x02fb7acc) Line 3806 + 0x11 C btpython23_d.dll!eval_frame(_frame * f=0x031aeeb0) Line 1391 + 0x11 C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x030c9130, _object * globals=0x03101ab8, _object * locals=0x00000000, _object * * args=0x0323650c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x030eed0c, int defcount=2, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyEval_CallObjectWithKeywords (_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 3346 + 0x11 C btpython23_d.dll!PyErr_Warn(_object * category=0x0306da10, char * message=0x1e14df70) Line 627 + 0xf C > btpython23_d.dll!builtin_apply(_object * self=0x00000000, _object * args=0x03238a78) Line 80 + 0x10 C btpython23_d.dll!PyCFunction_Call(_object * func=0x0305fb38, _object * arg=0x03238a78, _object * kw=0x00000000) Line 73 + 0xb C btpython23_d.dll!call_function(_object * * * pp_stack=0x0364f2b0, int oparg=2) Line 3439 + 0xf C btpython23_d.dll!eval_frame(_frame * f=0x0314a310) Line 2116 + 0xd C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x03178868, _object * globals=0x03174818, _object * locals=0x00000000, _object * * args=0x0d558f0c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!instancemethod_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 2432 + 0x11 C btpython23_d.dll!PyObject_Call(_object * func=0x0321fd38, _object * arg=0x030728f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyObject_CallMethod(_object * o=0x0317a4f8, char * name=0x64c453f4, char * format=0x64c453f0, ...) Line 1844 + 0xf C btrexcoreC230.dll!BT_REX_LP_PyProxy::Run (BT_Blackboard * ref=0x02eacc88) Line 136 + 0x1b C++ btrexcoreC230.dll!BT_Blackboard::Run() Line 166 + 0x29 C++ btrexcoreC230.dll! BT_REX_ContextImp::ProcessUTF16Buffer(const unsigned short * inbuf=0x00ab78bc, unsigned int inlen=673, int lid=30) Line 609 C++ basistechnology.rlp.dll! BasisTechnology.RLP.ContextImp.ProcessBuffer(String* data = "South African answers U.S. message in a bottle. JOHANNESBURG 08-22-1996 A South African boy is writing back to an American girl whose message in a bottle he found washed up on President Nelson Mandela's old prison island. But Carlo Hoffmann, an 11-year- old jailer's son who found the bottle on the beach at Robben Island off Cape Town after winter storms, will send his letter back by ordinary mail on Thursday, the post office said. It will be sent for free. Danielle Murray from Sandusky, Ohio, the same age as her new penfriend, asked for a reply from whoever received the message she flung on its journey months ago on the other side of the Atlantic Ocean. ", BasisTechnology.RLP.LanguageID __unnamed001 = ENGLISH) Line 84 + 0x33 bytes C++ ThreadStress.exe! BasisTechnology.RLPUnitTest.Worker.Go() Line 63 C# ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 21:57 Message: Logged In: YES user_id=876734 The rather funny-looking initialization was recommended by the author of the PEP. Once InitThreads is called, the GIL is held. To get it unheld, you have to release it. Here's his explanation .. OK - here is my explanation which you can adapt A simple application is one that initializes Python, and then executes some Python code. This Python code may itself cause other threads, or may indirectly cause other applications to "callback" into Python, but once the main thread returns, all the code has finished - you application terminates. With these applications, your init code need take no further action - just initialize Python and execute your code. Only your extension functions or callback entry points need manage the GIL state. python.exe is an example of this kind of app - it initializes Python, then executes the code specified on the command line. More complex applications may need to initialize Python, perform a little bit of initialization, but then do nothing else on that thread. An example would be where that initialization code bootstraps a few threads, then terminates immediately. The application however continues to live until explicitly terminated by the running code. In this scenario, you must use the following code - this code calls into Python. Once this call has been made, the thread still holds the GIL. This thread must release the GIL before other threads can work. [Technical aside: This works in the first case as the main thread which holds the lock continues to run. When it calls out to external libraries and at periodic times during execution, the thread-lock is unlocked allowing other threads to run. Once the main thread returns, it does still hold the lock - but as the application is terminating, that is OK - no other threads are competing for the lock. In our complicated example, the main thread returning does *not* indicate application termination, so the lock must be manually dropped.] That doesn't make a huge amount of sense, but I hope it is useful. All this being said, I have two contradictory bits of evidence. 1) We all agree that the code, as written, would fail due to the unsafe debug malloc. 2) When I reverse the calls, it produces a very different situation. I think, perhaps, that the GIL never gets unlocked properly, and then instead of crashing everything just gets stuck somehow. The change to call malloc seems pretty reasonable to me. Some tests seem called for in the tree on this. If order is not supposed to matter, the test should test both orders. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:46 Message: Logged In: YES user_id=1038590 I'd suggest assigning the bug report to Tim Peters (aka tim_one), to see if he still agrees with the fix he proposed last month. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:43 Message: Logged In: YES user_id=1038590 *rereads bug description and comments* I think I'm on the right page now. The offending call would appear to be PyMem_NEW inside PyThreadState_New. Under a release build, this resolves into a direct call to malloc(), but under debug it resolves to _PyObjectDebugMalloc. Which is what you said in the bug report - I just read it incorrectly. Sorry for the confusion. I just remembered that this came up a few weeks ago on Python dev, with a proposed solution (changing this particular call to use malloc() directly): http://mail.python.org/pipermail/python-dev/2004-September/048683.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Sun Oct 10 04:00:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 10 04:00:23 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 14:15 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Tim Peters (tim_one) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-09 22:00 Message: Logged In: YES user_id=31435 A fix for find_key() was checked in, and the test program (repaired to plug the memory leaks) passed a long release- build run with arguments "5000 64 1000" after. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 20:40 Message: Logged In: YES user_id=31435 One problem here is that the portable TLS implementation isn't actually threadsafe -- find_key() can get fatally confused if another thread is deleting a key at the same time. That appears to account for the release-build crashes in my immediately preceding comment. That should be repaired soon. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 01:04 Message: Logged In: YES user_id=31435 For the record, in a release build this reliably dies, on a hyper- threaded 3.2GHz box, with """ Fatal Python error: auto-releasing thread-state, but no thread-state for this thread """ given arguments "200 64 100". WRT this failure, it doesn't matter in which order Py_Initialize(); PyEval_InitThreads(); are done. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 00:52 Message: Logged In: YES user_id=31435 Random: - Arguments "1 100 40" don't make sense on my box, as MAXIMUM_WAIT_OBJECTS is 64, and so the call to WaitForMultipleObjects() fails immediately when passed 100 handles. - The failures in a debug build are much more dramatic if the memory leak is repaired (as Nick said, "trash" leaks; freeing the memory again is much more effective at stressing the heap). - Is benson_basis the KSR Benson M? If so, you'll appreciate that debugging someone else's thread problems is a lot harder than resolving the Banach-Tarski Paradox . - While I was a fan of Mark's new thread API, I've never used it, so it may take some time to figure out what this is all about. Alas, spare time is in short supply. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 22:35 Message: Logged In: YES user_id=1038590 I'm well out of my depth now - I definitely recommend assigning the bug report to Tim (I'd do it, but I can't edit bugs I didn't create). ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 22:24 Message: Logged In: YES user_id=876734 I'm afraid that something is more seriously wrong here than just the malloc in Ensure. Here's what I did in our production code. I protected all calls to Ensure with a mutex. The result was a similiar problem with the debug heap happening inside code that was executing under the GIL. Given the structure of our code, I'm reasonably confident that there is no rogue thread running around outside the GIL. In fact, at the time of the explosion here, the other threads are all waiting their turn to get the GIL. btpython23_d.dll!PyObject_Malloc(unsigned int nbytes=98) Line 581 + 0x6 C btpython23_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=82) Line 990 + 0x9 C btpython23_d.dll!PyString_FromStringAndSize(const char * str=0x03176384, int size=50) Line 78 + 0xc C btpython23_d.dll!string_slice(PyStringObject * a=0x03176368, int i=0, int j=50) Line 1006 + 0x17 C btpython23_d.dll!PySequence_GetSlice(_object * s=0x03176368, int i1=0, int i2=50) Line 1218 + 0x12 C btpython23_d.dll!apply_slice(_object * u=0x03176368, _object * v=0x00000000, _object * w=0x02fb7acc) Line 3806 + 0x11 C btpython23_d.dll!eval_frame(_frame * f=0x031aeeb0) Line 1391 + 0x11 C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x030c9130, _object * globals=0x03101ab8, _object * locals=0x00000000, _object * * args=0x0323650c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x030eed0c, int defcount=2, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyEval_CallObjectWithKeywords (_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 3346 + 0x11 C btpython23_d.dll!PyErr_Warn(_object * category=0x0306da10, char * message=0x1e14df70) Line 627 + 0xf C > btpython23_d.dll!builtin_apply(_object * self=0x00000000, _object * args=0x03238a78) Line 80 + 0x10 C btpython23_d.dll!PyCFunction_Call(_object * func=0x0305fb38, _object * arg=0x03238a78, _object * kw=0x00000000) Line 73 + 0xb C btpython23_d.dll!call_function(_object * * * pp_stack=0x0364f2b0, int oparg=2) Line 3439 + 0xf C btpython23_d.dll!eval_frame(_frame * f=0x0314a310) Line 2116 + 0xd C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x03178868, _object * globals=0x03174818, _object * locals=0x00000000, _object * * args=0x0d558f0c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!instancemethod_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 2432 + 0x11 C btpython23_d.dll!PyObject_Call(_object * func=0x0321fd38, _object * arg=0x030728f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyObject_CallMethod(_object * o=0x0317a4f8, char * name=0x64c453f4, char * format=0x64c453f0, ...) Line 1844 + 0xf C btrexcoreC230.dll!BT_REX_LP_PyProxy::Run (BT_Blackboard * ref=0x02eacc88) Line 136 + 0x1b C++ btrexcoreC230.dll!BT_Blackboard::Run() Line 166 + 0x29 C++ btrexcoreC230.dll! BT_REX_ContextImp::ProcessUTF16Buffer(const unsigned short * inbuf=0x00ab78bc, unsigned int inlen=673, int lid=30) Line 609 C++ basistechnology.rlp.dll! BasisTechnology.RLP.ContextImp.ProcessBuffer(String* data = "South African answers U.S. message in a bottle. JOHANNESBURG 08-22-1996 A South African boy is writing back to an American girl whose message in a bottle he found washed up on President Nelson Mandela's old prison island. But Carlo Hoffmann, an 11-year- old jailer's son who found the bottle on the beach at Robben Island off Cape Town after winter storms, will send his letter back by ordinary mail on Thursday, the post office said. It will be sent for free. Danielle Murray from Sandusky, Ohio, the same age as her new penfriend, asked for a reply from whoever received the message she flung on its journey months ago on the other side of the Atlantic Ocean. ", BasisTechnology.RLP.LanguageID __unnamed001 = ENGLISH) Line 84 + 0x33 bytes C++ ThreadStress.exe! BasisTechnology.RLPUnitTest.Worker.Go() Line 63 C# ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 21:57 Message: Logged In: YES user_id=876734 The rather funny-looking initialization was recommended by the author of the PEP. Once InitThreads is called, the GIL is held. To get it unheld, you have to release it. Here's his explanation .. OK - here is my explanation which you can adapt A simple application is one that initializes Python, and then executes some Python code. This Python code may itself cause other threads, or may indirectly cause other applications to "callback" into Python, but once the main thread returns, all the code has finished - you application terminates. With these applications, your init code need take no further action - just initialize Python and execute your code. Only your extension functions or callback entry points need manage the GIL state. python.exe is an example of this kind of app - it initializes Python, then executes the code specified on the command line. More complex applications may need to initialize Python, perform a little bit of initialization, but then do nothing else on that thread. An example would be where that initialization code bootstraps a few threads, then terminates immediately. The application however continues to live until explicitly terminated by the running code. In this scenario, you must use the following code - this code calls into Python. Once this call has been made, the thread still holds the GIL. This thread must release the GIL before other threads can work. [Technical aside: This works in the first case as the main thread which holds the lock continues to run. When it calls out to external libraries and at periodic times during execution, the thread-lock is unlocked allowing other threads to run. Once the main thread returns, it does still hold the lock - but as the application is terminating, that is OK - no other threads are competing for the lock. In our complicated example, the main thread returning does *not* indicate application termination, so the lock must be manually dropped.] That doesn't make a huge amount of sense, but I hope it is useful. All this being said, I have two contradictory bits of evidence. 1) We all agree that the code, as written, would fail due to the unsafe debug malloc. 2) When I reverse the calls, it produces a very different situation. I think, perhaps, that the GIL never gets unlocked properly, and then instead of crashing everything just gets stuck somehow. The change to call malloc seems pretty reasonable to me. Some tests seem called for in the tree on this. If order is not supposed to matter, the test should test both orders. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:46 Message: Logged In: YES user_id=1038590 I'd suggest assigning the bug report to Tim Peters (aka tim_one), to see if he still agrees with the fix he proposed last month. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:43 Message: Logged In: YES user_id=1038590 *rereads bug description and comments* I think I'm on the right page now. The offending call would appear to be PyMem_NEW inside PyThreadState_New. Under a release build, this resolves into a direct call to malloc(), but under debug it resolves to _PyObjectDebugMalloc. Which is what you said in the bug report - I just read it incorrectly. Sorry for the confusion. I just remembered that this came up a few weeks ago on Python dev, with a proposed solution (changing this particular call to use malloc() directly): http://mail.python.org/pipermail/python-dev/2004-September/048683.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Sun Oct 10 04:24:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 10 04:24:23 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 14:15 Message generated for change (Comment added) made by benson_basis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Tim Peters (tim_one) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- >Comment By: benson margulies (benson_basis) Date: 2004-10-09 22:24 Message: Logged In: YES user_id=876734 I am indeed the benson in question. Hi there. I seem doomed to be in the thread business. b i m 2 0 0 4 splat b a s i s t e c h blot c o m I'll try to backport these fixes into our 2.3 codebase. I'd appreciate your view on the following question: given a choice between protecting all of our calls into python with a mutex, and on the other hand picking up these fixes and using the GIL, does python manage to internally unlock enough to make it worth the bother? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 22:00 Message: Logged In: YES user_id=31435 A fix for find_key() was checked in, and the test program (repaired to plug the memory leaks) passed a long release- build run with arguments "5000 64 1000" after. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 20:40 Message: Logged In: YES user_id=31435 One problem here is that the portable TLS implementation isn't actually threadsafe -- find_key() can get fatally confused if another thread is deleting a key at the same time. That appears to account for the release-build crashes in my immediately preceding comment. That should be repaired soon. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 01:04 Message: Logged In: YES user_id=31435 For the record, in a release build this reliably dies, on a hyper- threaded 3.2GHz box, with """ Fatal Python error: auto-releasing thread-state, but no thread-state for this thread """ given arguments "200 64 100". WRT this failure, it doesn't matter in which order Py_Initialize(); PyEval_InitThreads(); are done. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 00:52 Message: Logged In: YES user_id=31435 Random: - Arguments "1 100 40" don't make sense on my box, as MAXIMUM_WAIT_OBJECTS is 64, and so the call to WaitForMultipleObjects() fails immediately when passed 100 handles. - The failures in a debug build are much more dramatic if the memory leak is repaired (as Nick said, "trash" leaks; freeing the memory again is much more effective at stressing the heap). - Is benson_basis the KSR Benson M? If so, you'll appreciate that debugging someone else's thread problems is a lot harder than resolving the Banach-Tarski Paradox . - While I was a fan of Mark's new thread API, I've never used it, so it may take some time to figure out what this is all about. Alas, spare time is in short supply. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 22:35 Message: Logged In: YES user_id=1038590 I'm well out of my depth now - I definitely recommend assigning the bug report to Tim (I'd do it, but I can't edit bugs I didn't create). ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 22:24 Message: Logged In: YES user_id=876734 I'm afraid that something is more seriously wrong here than just the malloc in Ensure. Here's what I did in our production code. I protected all calls to Ensure with a mutex. The result was a similiar problem with the debug heap happening inside code that was executing under the GIL. Given the structure of our code, I'm reasonably confident that there is no rogue thread running around outside the GIL. In fact, at the time of the explosion here, the other threads are all waiting their turn to get the GIL. btpython23_d.dll!PyObject_Malloc(unsigned int nbytes=98) Line 581 + 0x6 C btpython23_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=82) Line 990 + 0x9 C btpython23_d.dll!PyString_FromStringAndSize(const char * str=0x03176384, int size=50) Line 78 + 0xc C btpython23_d.dll!string_slice(PyStringObject * a=0x03176368, int i=0, int j=50) Line 1006 + 0x17 C btpython23_d.dll!PySequence_GetSlice(_object * s=0x03176368, int i1=0, int i2=50) Line 1218 + 0x12 C btpython23_d.dll!apply_slice(_object * u=0x03176368, _object * v=0x00000000, _object * w=0x02fb7acc) Line 3806 + 0x11 C btpython23_d.dll!eval_frame(_frame * f=0x031aeeb0) Line 1391 + 0x11 C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x030c9130, _object * globals=0x03101ab8, _object * locals=0x00000000, _object * * args=0x0323650c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x030eed0c, int defcount=2, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyEval_CallObjectWithKeywords (_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 3346 + 0x11 C btpython23_d.dll!PyErr_Warn(_object * category=0x0306da10, char * message=0x1e14df70) Line 627 + 0xf C > btpython23_d.dll!builtin_apply(_object * self=0x00000000, _object * args=0x03238a78) Line 80 + 0x10 C btpython23_d.dll!PyCFunction_Call(_object * func=0x0305fb38, _object * arg=0x03238a78, _object * kw=0x00000000) Line 73 + 0xb C btpython23_d.dll!call_function(_object * * * pp_stack=0x0364f2b0, int oparg=2) Line 3439 + 0xf C btpython23_d.dll!eval_frame(_frame * f=0x0314a310) Line 2116 + 0xd C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x03178868, _object * globals=0x03174818, _object * locals=0x00000000, _object * * args=0x0d558f0c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!instancemethod_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 2432 + 0x11 C btpython23_d.dll!PyObject_Call(_object * func=0x0321fd38, _object * arg=0x030728f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyObject_CallMethod(_object * o=0x0317a4f8, char * name=0x64c453f4, char * format=0x64c453f0, ...) Line 1844 + 0xf C btrexcoreC230.dll!BT_REX_LP_PyProxy::Run (BT_Blackboard * ref=0x02eacc88) Line 136 + 0x1b C++ btrexcoreC230.dll!BT_Blackboard::Run() Line 166 + 0x29 C++ btrexcoreC230.dll! BT_REX_ContextImp::ProcessUTF16Buffer(const unsigned short * inbuf=0x00ab78bc, unsigned int inlen=673, int lid=30) Line 609 C++ basistechnology.rlp.dll! BasisTechnology.RLP.ContextImp.ProcessBuffer(String* data = "South African answers U.S. message in a bottle. JOHANNESBURG 08-22-1996 A South African boy is writing back to an American girl whose message in a bottle he found washed up on President Nelson Mandela's old prison island. But Carlo Hoffmann, an 11-year- old jailer's son who found the bottle on the beach at Robben Island off Cape Town after winter storms, will send his letter back by ordinary mail on Thursday, the post office said. It will be sent for free. Danielle Murray from Sandusky, Ohio, the same age as her new penfriend, asked for a reply from whoever received the message she flung on its journey months ago on the other side of the Atlantic Ocean. ", BasisTechnology.RLP.LanguageID __unnamed001 = ENGLISH) Line 84 + 0x33 bytes C++ ThreadStress.exe! BasisTechnology.RLPUnitTest.Worker.Go() Line 63 C# ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 21:57 Message: Logged In: YES user_id=876734 The rather funny-looking initialization was recommended by the author of the PEP. Once InitThreads is called, the GIL is held. To get it unheld, you have to release it. Here's his explanation .. OK - here is my explanation which you can adapt A simple application is one that initializes Python, and then executes some Python code. This Python code may itself cause other threads, or may indirectly cause other applications to "callback" into Python, but once the main thread returns, all the code has finished - you application terminates. With these applications, your init code need take no further action - just initialize Python and execute your code. Only your extension functions or callback entry points need manage the GIL state. python.exe is an example of this kind of app - it initializes Python, then executes the code specified on the command line. More complex applications may need to initialize Python, perform a little bit of initialization, but then do nothing else on that thread. An example would be where that initialization code bootstraps a few threads, then terminates immediately. The application however continues to live until explicitly terminated by the running code. In this scenario, you must use the following code - this code calls into Python. Once this call has been made, the thread still holds the GIL. This thread must release the GIL before other threads can work. [Technical aside: This works in the first case as the main thread which holds the lock continues to run. When it calls out to external libraries and at periodic times during execution, the thread-lock is unlocked allowing other threads to run. Once the main thread returns, it does still hold the lock - but as the application is terminating, that is OK - no other threads are competing for the lock. In our complicated example, the main thread returning does *not* indicate application termination, so the lock must be manually dropped.] That doesn't make a huge amount of sense, but I hope it is useful. All this being said, I have two contradictory bits of evidence. 1) We all agree that the code, as written, would fail due to the unsafe debug malloc. 2) When I reverse the calls, it produces a very different situation. I think, perhaps, that the GIL never gets unlocked properly, and then instead of crashing everything just gets stuck somehow. The change to call malloc seems pretty reasonable to me. Some tests seem called for in the tree on this. If order is not supposed to matter, the test should test both orders. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:46 Message: Logged In: YES user_id=1038590 I'd suggest assigning the bug report to Tim Peters (aka tim_one), to see if he still agrees with the fix he proposed last month. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:43 Message: Logged In: YES user_id=1038590 *rereads bug description and comments* I think I'm on the right page now. The offending call would appear to be PyMem_NEW inside PyThreadState_New. Under a release build, this resolves into a direct call to malloc(), but under debug it resolves to _PyObjectDebugMalloc. Which is what you said in the bug report - I just read it incorrectly. Sorry for the confusion. I just remembered that this came up a few weeks ago on Python dev, with a proposed solution (changing this particular call to use malloc() directly): http://mail.python.org/pipermail/python-dev/2004-September/048683.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Sun Oct 10 04:50:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 10 04:50:08 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 14:15 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Tim Peters (tim_one) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-09 22:50 Message: Logged In: YES user_id=31435 Hi, Benson! Long time. Quickie on your question: it depends on your app. In general, the Python implementation releases the GIL around (and only around) potentially blocking C calls, when it's safe to do so. So think I/O here -- operations on files and sockets, and miscellaneous operations like sleep() and select(). That can be extremely helpful if you're doing I/O, and more helpful the longer the I/O latency. If you're doing pure computation, it doesn't help at all -- unless you're also using, e.g., a C or Fortran extension module that releases the GIL around its internal computations. ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-09 22:24 Message: Logged In: YES user_id=876734 I am indeed the benson in question. Hi there. I seem doomed to be in the thread business. b i m 2 0 0 4 splat b a s i s t e c h blot c o m I'll try to backport these fixes into our 2.3 codebase. I'd appreciate your view on the following question: given a choice between protecting all of our calls into python with a mutex, and on the other hand picking up these fixes and using the GIL, does python manage to internally unlock enough to make it worth the bother? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 22:00 Message: Logged In: YES user_id=31435 A fix for find_key() was checked in, and the test program (repaired to plug the memory leaks) passed a long release- build run with arguments "5000 64 1000" after. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 20:40 Message: Logged In: YES user_id=31435 One problem here is that the portable TLS implementation isn't actually threadsafe -- find_key() can get fatally confused if another thread is deleting a key at the same time. That appears to account for the release-build crashes in my immediately preceding comment. That should be repaired soon. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 01:04 Message: Logged In: YES user_id=31435 For the record, in a release build this reliably dies, on a hyper- threaded 3.2GHz box, with """ Fatal Python error: auto-releasing thread-state, but no thread-state for this thread """ given arguments "200 64 100". WRT this failure, it doesn't matter in which order Py_Initialize(); PyEval_InitThreads(); are done. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 00:52 Message: Logged In: YES user_id=31435 Random: - Arguments "1 100 40" don't make sense on my box, as MAXIMUM_WAIT_OBJECTS is 64, and so the call to WaitForMultipleObjects() fails immediately when passed 100 handles. - The failures in a debug build are much more dramatic if the memory leak is repaired (as Nick said, "trash" leaks; freeing the memory again is much more effective at stressing the heap). - Is benson_basis the KSR Benson M? If so, you'll appreciate that debugging someone else's thread problems is a lot harder than resolving the Banach-Tarski Paradox . - While I was a fan of Mark's new thread API, I've never used it, so it may take some time to figure out what this is all about. Alas, spare time is in short supply. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 22:35 Message: Logged In: YES user_id=1038590 I'm well out of my depth now - I definitely recommend assigning the bug report to Tim (I'd do it, but I can't edit bugs I didn't create). ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 22:24 Message: Logged In: YES user_id=876734 I'm afraid that something is more seriously wrong here than just the malloc in Ensure. Here's what I did in our production code. I protected all calls to Ensure with a mutex. The result was a similiar problem with the debug heap happening inside code that was executing under the GIL. Given the structure of our code, I'm reasonably confident that there is no rogue thread running around outside the GIL. In fact, at the time of the explosion here, the other threads are all waiting their turn to get the GIL. btpython23_d.dll!PyObject_Malloc(unsigned int nbytes=98) Line 581 + 0x6 C btpython23_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=82) Line 990 + 0x9 C btpython23_d.dll!PyString_FromStringAndSize(const char * str=0x03176384, int size=50) Line 78 + 0xc C btpython23_d.dll!string_slice(PyStringObject * a=0x03176368, int i=0, int j=50) Line 1006 + 0x17 C btpython23_d.dll!PySequence_GetSlice(_object * s=0x03176368, int i1=0, int i2=50) Line 1218 + 0x12 C btpython23_d.dll!apply_slice(_object * u=0x03176368, _object * v=0x00000000, _object * w=0x02fb7acc) Line 3806 + 0x11 C btpython23_d.dll!eval_frame(_frame * f=0x031aeeb0) Line 1391 + 0x11 C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x030c9130, _object * globals=0x03101ab8, _object * locals=0x00000000, _object * * args=0x0323650c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x030eed0c, int defcount=2, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyEval_CallObjectWithKeywords (_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 3346 + 0x11 C btpython23_d.dll!PyErr_Warn(_object * category=0x0306da10, char * message=0x1e14df70) Line 627 + 0xf C > btpython23_d.dll!builtin_apply(_object * self=0x00000000, _object * args=0x03238a78) Line 80 + 0x10 C btpython23_d.dll!PyCFunction_Call(_object * func=0x0305fb38, _object * arg=0x03238a78, _object * kw=0x00000000) Line 73 + 0xb C btpython23_d.dll!call_function(_object * * * pp_stack=0x0364f2b0, int oparg=2) Line 3439 + 0xf C btpython23_d.dll!eval_frame(_frame * f=0x0314a310) Line 2116 + 0xd C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x03178868, _object * globals=0x03174818, _object * locals=0x00000000, _object * * args=0x0d558f0c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!instancemethod_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 2432 + 0x11 C btpython23_d.dll!PyObject_Call(_object * func=0x0321fd38, _object * arg=0x030728f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyObject_CallMethod(_object * o=0x0317a4f8, char * name=0x64c453f4, char * format=0x64c453f0, ...) Line 1844 + 0xf C btrexcoreC230.dll!BT_REX_LP_PyProxy::Run (BT_Blackboard * ref=0x02eacc88) Line 136 + 0x1b C++ btrexcoreC230.dll!BT_Blackboard::Run() Line 166 + 0x29 C++ btrexcoreC230.dll! BT_REX_ContextImp::ProcessUTF16Buffer(const unsigned short * inbuf=0x00ab78bc, unsigned int inlen=673, int lid=30) Line 609 C++ basistechnology.rlp.dll! BasisTechnology.RLP.ContextImp.ProcessBuffer(String* data = "South African answers U.S. message in a bottle. JOHANNESBURG 08-22-1996 A South African boy is writing back to an American girl whose message in a bottle he found washed up on President Nelson Mandela's old prison island. But Carlo Hoffmann, an 11-year- old jailer's son who found the bottle on the beach at Robben Island off Cape Town after winter storms, will send his letter back by ordinary mail on Thursday, the post office said. It will be sent for free. Danielle Murray from Sandusky, Ohio, the same age as her new penfriend, asked for a reply from whoever received the message she flung on its journey months ago on the other side of the Atlantic Ocean. ", BasisTechnology.RLP.LanguageID __unnamed001 = ENGLISH) Line 84 + 0x33 bytes C++ ThreadStress.exe! BasisTechnology.RLPUnitTest.Worker.Go() Line 63 C# ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 21:57 Message: Logged In: YES user_id=876734 The rather funny-looking initialization was recommended by the author of the PEP. Once InitThreads is called, the GIL is held. To get it unheld, you have to release it. Here's his explanation .. OK - here is my explanation which you can adapt A simple application is one that initializes Python, and then executes some Python code. This Python code may itself cause other threads, or may indirectly cause other applications to "callback" into Python, but once the main thread returns, all the code has finished - you application terminates. With these applications, your init code need take no further action - just initialize Python and execute your code. Only your extension functions or callback entry points need manage the GIL state. python.exe is an example of this kind of app - it initializes Python, then executes the code specified on the command line. More complex applications may need to initialize Python, perform a little bit of initialization, but then do nothing else on that thread. An example would be where that initialization code bootstraps a few threads, then terminates immediately. The application however continues to live until explicitly terminated by the running code. In this scenario, you must use the following code - this code calls into Python. Once this call has been made, the thread still holds the GIL. This thread must release the GIL before other threads can work. [Technical aside: This works in the first case as the main thread which holds the lock continues to run. When it calls out to external libraries and at periodic times during execution, the thread-lock is unlocked allowing other threads to run. Once the main thread returns, it does still hold the lock - but as the application is terminating, that is OK - no other threads are competing for the lock. In our complicated example, the main thread returning does *not* indicate application termination, so the lock must be manually dropped.] That doesn't make a huge amount of sense, but I hope it is useful. All this being said, I have two contradictory bits of evidence. 1) We all agree that the code, as written, would fail due to the unsafe debug malloc. 2) When I reverse the calls, it produces a very different situation. I think, perhaps, that the GIL never gets unlocked properly, and then instead of crashing everything just gets stuck somehow. The change to call malloc seems pretty reasonable to me. Some tests seem called for in the tree on this. If order is not supposed to matter, the test should test both orders. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:46 Message: Logged In: YES user_id=1038590 I'd suggest assigning the bug report to Tim Peters (aka tim_one), to see if he still agrees with the fix he proposed last month. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:43 Message: Logged In: YES user_id=1038590 *rereads bug description and comments* I think I'm on the right page now. The offending call would appear to be PyMem_NEW inside PyThreadState_New. Under a release build, this resolves into a direct call to malloc(), but under debug it resolves to _PyObjectDebugMalloc. Which is what you said in the bug report - I just read it incorrectly. Sorry for the confusion. I just remembered that this came up a few weeks ago on Python dev, with a proposed solution (changing this particular call to use malloc() directly): http://mail.python.org/pipermail/python-dev/2004-September/048683.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Sun Oct 10 04:58:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 10 06:17:18 2004 Subject: [ python-Bugs-1041645 ] Thread management corrupts heap Message-ID: Bugs item #1041645, was opened at 2004-10-06 14:15 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 Category: None Group: None >Status: Closed >Resolution: Fixed Priority: 7 Submitted By: benson margulies (benson_basis) Assigned to: Tim Peters (tim_one) Summary: Thread management corrupts heap Initial Comment: The PyGILState_Ensure mechanism appears to have a built-in heap-corrupting race condition. If a new thread calls PyGILState_Ensure, then the code allocates a new 'tstate' for it on the heap. This allocation is not protected by any lock. So, multiple racing threads can hit the heap at the same time, and corrupt it. I have observed this with both 2.3 and with 2.4a3. I will attach a sample application. The application is Win32, but should be easy enough to adapt to Unix if someone cares to. Since the stated purpose of this mechanism, in PEP311, is to allow any-old-thread to call into Python, I believe that the usage model here is legitimate. To watch this explode, run the attached with arguments like, oh, 1 100 40 against the debug python build. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-09 22:58 Message: Logged In: YES user_id=31435 OK, PyInterpreterState_New() and PyThreadState_New() were changed to call the platform malloc()/free() directly, and that appears to fix the remaining problems in a debug build. Two debug runs of the test program, with memory leaks repaired, using arguments "5000 64 100", completed without incident. One did initialization in the order: Py_Initialize(); PyEval_InitThreads(); and the other in order: PyEval_InitThreads(); Py_Initialize(); I saw no difference in behavior between those runs. There's not enough info in the 2004-10-06 22:24 comment to know what that was about, so I'm closing this bug report. If you continue to see problems, please open a new bug report (or reopen this one -- but this one is pretty "busy" already). Note that the problem with find_key() was horrid, and could lead to arbitrary memory corruption. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 22:50 Message: Logged In: YES user_id=31435 Hi, Benson! Long time. Quickie on your question: it depends on your app. In general, the Python implementation releases the GIL around (and only around) potentially blocking C calls, when it's safe to do so. So think I/O here -- operations on files and sockets, and miscellaneous operations like sleep() and select(). That can be extremely helpful if you're doing I/O, and more helpful the longer the I/O latency. If you're doing pure computation, it doesn't help at all -- unless you're also using, e.g., a C or Fortran extension module that releases the GIL around its internal computations. ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-09 22:24 Message: Logged In: YES user_id=876734 I am indeed the benson in question. Hi there. I seem doomed to be in the thread business. b i m 2 0 0 4 splat b a s i s t e c h blot c o m I'll try to backport these fixes into our 2.3 codebase. I'd appreciate your view on the following question: given a choice between protecting all of our calls into python with a mutex, and on the other hand picking up these fixes and using the GIL, does python manage to internally unlock enough to make it worth the bother? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 22:00 Message: Logged In: YES user_id=31435 A fix for find_key() was checked in, and the test program (repaired to plug the memory leaks) passed a long release- build run with arguments "5000 64 1000" after. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 20:40 Message: Logged In: YES user_id=31435 One problem here is that the portable TLS implementation isn't actually threadsafe -- find_key() can get fatally confused if another thread is deleting a key at the same time. That appears to account for the release-build crashes in my immediately preceding comment. That should be repaired soon. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 01:04 Message: Logged In: YES user_id=31435 For the record, in a release build this reliably dies, on a hyper- threaded 3.2GHz box, with """ Fatal Python error: auto-releasing thread-state, but no thread-state for this thread """ given arguments "200 64 100". WRT this failure, it doesn't matter in which order Py_Initialize(); PyEval_InitThreads(); are done. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-09 00:52 Message: Logged In: YES user_id=31435 Random: - Arguments "1 100 40" don't make sense on my box, as MAXIMUM_WAIT_OBJECTS is 64, and so the call to WaitForMultipleObjects() fails immediately when passed 100 handles. - The failures in a debug build are much more dramatic if the memory leak is repaired (as Nick said, "trash" leaks; freeing the memory again is much more effective at stressing the heap). - Is benson_basis the KSR Benson M? If so, you'll appreciate that debugging someone else's thread problems is a lot harder than resolving the Banach-Tarski Paradox . - While I was a fan of Mark's new thread API, I've never used it, so it may take some time to figure out what this is all about. Alas, spare time is in short supply. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 22:35 Message: Logged In: YES user_id=1038590 I'm well out of my depth now - I definitely recommend assigning the bug report to Tim (I'd do it, but I can't edit bugs I didn't create). ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 22:24 Message: Logged In: YES user_id=876734 I'm afraid that something is more seriously wrong here than just the malloc in Ensure. Here's what I did in our production code. I protected all calls to Ensure with a mutex. The result was a similiar problem with the debug heap happening inside code that was executing under the GIL. Given the structure of our code, I'm reasonably confident that there is no rogue thread running around outside the GIL. In fact, at the time of the explosion here, the other threads are all waiting their turn to get the GIL. btpython23_d.dll!PyObject_Malloc(unsigned int nbytes=98) Line 581 + 0x6 C btpython23_d.dll!_PyObject_DebugMalloc(unsigned int nbytes=82) Line 990 + 0x9 C btpython23_d.dll!PyString_FromStringAndSize(const char * str=0x03176384, int size=50) Line 78 + 0xc C btpython23_d.dll!string_slice(PyStringObject * a=0x03176368, int i=0, int j=50) Line 1006 + 0x17 C btpython23_d.dll!PySequence_GetSlice(_object * s=0x03176368, int i1=0, int i2=50) Line 1218 + 0x12 C btpython23_d.dll!apply_slice(_object * u=0x03176368, _object * v=0x00000000, _object * w=0x02fb7acc) Line 3806 + 0x11 C btpython23_d.dll!eval_frame(_frame * f=0x031aeeb0) Line 1391 + 0x11 C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x030c9130, _object * globals=0x03101ab8, _object * locals=0x00000000, _object * * args=0x0323650c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x030eed0c, int defcount=2, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyEval_CallObjectWithKeywords (_object * func=0x0316eea8, _object * arg=0x032364f8, _object * kw=0x00000000) Line 3346 + 0x11 C btpython23_d.dll!PyErr_Warn(_object * category=0x0306da10, char * message=0x1e14df70) Line 627 + 0xf C > btpython23_d.dll!builtin_apply(_object * self=0x00000000, _object * args=0x03238a78) Line 80 + 0x10 C btpython23_d.dll!PyCFunction_Call(_object * func=0x0305fb38, _object * arg=0x03238a78, _object * kw=0x00000000) Line 73 + 0xb C btpython23_d.dll!call_function(_object * * * pp_stack=0x0364f2b0, int oparg=2) Line 3439 + 0xf C btpython23_d.dll!eval_frame(_frame * f=0x0314a310) Line 2116 + 0xd C btpython23_d.dll!PyEval_EvalCodeEx(PyCodeObject * co=0x03178868, _object * globals=0x03174818, _object * locals=0x00000000, _object * * args=0x0d558f0c, int argcount=2, _object * * kws=0x00000000, int kwcount=0, _object * * defs=0x00000000, int defcount=0, _object * closure=0x00000000) Line 2663 + 0x9 C btpython23_d.dll!function_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 509 + 0x40 C btpython23_d.dll!PyObject_Call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!instancemethod_call(_object * func=0x0320f198, _object * arg=0x0d558ef8, _object * kw=0x00000000) Line 2432 + 0x11 C btpython23_d.dll!PyObject_Call(_object * func=0x0321fd38, _object * arg=0x030728f8, _object * kw=0x00000000) Line 1755 + 0xf C btpython23_d.dll!PyObject_CallMethod(_object * o=0x0317a4f8, char * name=0x64c453f4, char * format=0x64c453f0, ...) Line 1844 + 0xf C btrexcoreC230.dll!BT_REX_LP_PyProxy::Run (BT_Blackboard * ref=0x02eacc88) Line 136 + 0x1b C++ btrexcoreC230.dll!BT_Blackboard::Run() Line 166 + 0x29 C++ btrexcoreC230.dll! BT_REX_ContextImp::ProcessUTF16Buffer(const unsigned short * inbuf=0x00ab78bc, unsigned int inlen=673, int lid=30) Line 609 C++ basistechnology.rlp.dll! BasisTechnology.RLP.ContextImp.ProcessBuffer(String* data = "South African answers U.S. message in a bottle. JOHANNESBURG 08-22-1996 A South African boy is writing back to an American girl whose message in a bottle he found washed up on President Nelson Mandela's old prison island. But Carlo Hoffmann, an 11-year- old jailer's son who found the bottle on the beach at Robben Island off Cape Town after winter storms, will send his letter back by ordinary mail on Thursday, the post office said. It will be sent for free. Danielle Murray from Sandusky, Ohio, the same age as her new penfriend, asked for a reply from whoever received the message she flung on its journey months ago on the other side of the Atlantic Ocean. ", BasisTechnology.RLP.LanguageID __unnamed001 = ENGLISH) Line 84 + 0x33 bytes C++ ThreadStress.exe! BasisTechnology.RLPUnitTest.Worker.Go() Line 63 C# ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 21:57 Message: Logged In: YES user_id=876734 The rather funny-looking initialization was recommended by the author of the PEP. Once InitThreads is called, the GIL is held. To get it unheld, you have to release it. Here's his explanation .. OK - here is my explanation which you can adapt A simple application is one that initializes Python, and then executes some Python code. This Python code may itself cause other threads, or may indirectly cause other applications to "callback" into Python, but once the main thread returns, all the code has finished - you application terminates. With these applications, your init code need take no further action - just initialize Python and execute your code. Only your extension functions or callback entry points need manage the GIL state. python.exe is an example of this kind of app - it initializes Python, then executes the code specified on the command line. More complex applications may need to initialize Python, perform a little bit of initialization, but then do nothing else on that thread. An example would be where that initialization code bootstraps a few threads, then terminates immediately. The application however continues to live until explicitly terminated by the running code. In this scenario, you must use the following code - this code calls into Python. Once this call has been made, the thread still holds the GIL. This thread must release the GIL before other threads can work. [Technical aside: This works in the first case as the main thread which holds the lock continues to run. When it calls out to external libraries and at periodic times during execution, the thread-lock is unlocked allowing other threads to run. Once the main thread returns, it does still hold the lock - but as the application is terminating, that is OK - no other threads are competing for the lock. In our complicated example, the main thread returning does *not* indicate application termination, so the lock must be manually dropped.] That doesn't make a huge amount of sense, but I hope it is useful. All this being said, I have two contradictory bits of evidence. 1) We all agree that the code, as written, would fail due to the unsafe debug malloc. 2) When I reverse the calls, it produces a very different situation. I think, perhaps, that the GIL never gets unlocked properly, and then instead of crashing everything just gets stuck somehow. The change to call malloc seems pretty reasonable to me. Some tests seem called for in the tree on this. If order is not supposed to matter, the test should test both orders. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:46 Message: Logged In: YES user_id=1038590 I'd suggest assigning the bug report to Tim Peters (aka tim_one), to see if he still agrees with the fix he proposed last month. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:43 Message: Logged In: YES user_id=1038590 *rereads bug description and comments* I think I'm on the right page now. The offending call would appear to be PyMem_NEW inside PyThreadState_New. Under a release build, this resolves into a direct call to malloc(), but under debug it resolves to _PyObjectDebugMalloc. Which is what you said in the bug report - I just read it incorrectly. Sorry for the confusion. I just remembered that this came up a few weeks ago on Python dev, with a proposed solution (changing this particular call to use malloc() directly): http://mail.python.org/pipermail/python-dev/2004-September/048683.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 20:15 Message: Logged In: YES user_id=1038590 Disregard the previous comment regarding PyEval_InitThreads and Py_Initialize - I was reading an older version of the documentation which suggested the order of the calls mattered. This does leak the constructed strings (no call to Py_XDECREF(trash)), but that shouldn't cause anything too catastrophic. However, the call to PyGILState_Release without a preceding call to PyGILState_Ensure looks dubious (it ends up deleting the main thread's thread state, which seems unhealthy). Do the same symptoms occur when using the Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros around the call to run_group()? (These macros are recommended when you *know* you already have the GIL. The PyGILState functions are for when you don't know if you have the GIL or not) ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:56 Message: Logged In: YES user_id=876734 On the one hand, you are correct that changing the order improves the behavior. On the other hand, the doc is as I recalled it, asserting that you can call it first. Perhaps this should be registered as a doc problem? This is a no-op when called for a second time. It is safe to call this function before calling Py_Initialize() . ---------------------------------------------------------------------- Comment By: benson margulies (benson_basis) Date: 2004-10-06 19:49 Message: Logged In: YES user_id=876734 The documentation very specifically states that the order doesn't matter. Further, I've read the code, and the problem is unprotected data structures in the python debug heap, which won't be any more protected in the other order. I'll switch them and report back, just in case. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:32 Message: Logged In: YES user_id=1038590 I haven't checked if that would fix the problem, since I'm currently on Linux. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-06 19:31 Message: Logged In: YES user_id=1038590 The attached program calls PyEval_InitThreads and Py_Initialize in the wrong order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1041645&group_id=5470 From noreply at sourceforge.net Sun Oct 10 10:06:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 10 10:06:28 2004 Subject: [ python-Bugs-1043446 ] Uninterruptable loop Message-ID: Bugs item #1043446, was opened at 2004-10-08 23:26 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1043446&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None >Priority: 3 Submitted By: Jp Calderone (kuran) Assigned to: Nobody/Anonymous (nobody) Summary: Uninterruptable loop Initial Comment: import itertools list(itertools.repeat('x')) ^C will not interrupt this. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-10 03:06 Message: Logged In: YES user_id=80475 FWIW, there are many variations of this theme using almost anything accepting an iterable input (dict.fromkeys, tuple, set, etc) paired with any long running or infinite iterator (itertools.count, xrange(sys.maxint), etc). Even the tp_print slot can get caught up in emitting oversized output in a way that is uninterruptable. I don't see a clean way of teaching all of these functions to periodically check for signals, and either handle them, raise an exception or continue. Fixing this could muck-up and complicate a bunch of otherwise clean code. Still, it would be nice if everything were interruptable. I'm just not sure it's worth it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1043446&group_id=5470 From noreply at sourceforge.net Sun Oct 10 10:07:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 10 10:07:36 2004 Subject: [ python-Bugs-1042238 ] Lib/compiler chokes on certain genexps Message-ID: Bugs item #1042238, was opened at 2004-10-07 08:35 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: Lib/compiler chokes on certain genexps Initial Comment: Noticed this because test_compiler fails on _strptime as of yesterday. Here's what seems to be a fairly minimal example: f((x for x in y), 1) The traceback is huge, but ends with: File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 687, in power node = self.com_apply_trailer(node, elt) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1144, in com_apply_trailer return self.com_call_function(primaryNode, nodelist[2]) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1169, in com_call_function raise SyntaxError, 'generator expression needs parenthesis' SyntaxError: generator expression needs parenthesis Might be a problem in transformer.py, not really sure. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-10 03:07 Message: Logged In: YES user_id=80475 Do you know how to fix this? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 08:38 Message: Logged In: YES user_id=6656 Hang on, the exception is *inside* transformer.py. Duh. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 From noreply at sourceforge.net Sun Oct 10 10:44:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 10 10:44:42 2004 Subject: [ python-Bugs-1043134 ] mimetypes.guess_extension('text/plain') == '.ksh' ??? Message-ID: Bugs item #1043134, was opened at 2004-10-08 17:44 Message generated for change (Comment added) made by kxroberto You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1043134&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Robert Kiendl (kxroberto) Assigned to: Nobody/Anonymous (nobody) Summary: mimetypes.guess_extension('text/plain') == '.ksh' ??? Initial Comment: Instead of returning the first in the list of extensions it should return the most reasonable . here: to have a *.txt on disk after saveing? ---------------------------------------------------------------------- >Comment By: Robert Kiendl (kxroberto) Date: 2004-10-10 10:44 Message: Logged In: YES user_id=972995 in mimetypes.py there is already a common_types = { '.jpg' : 'image/jpg', ... .txt could be added, mayby guess_extension should first reverse-take it out of there, not random ...? background: my intent was to save MIME attachment as (startable) temporary file. yet got wonderful .ksh's for textfiles, and had to fumble ... ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-09 17:26 Message: Logged In: YES user_id=469548 How would you suggest finding out what the most reasonable extension for a mime type is? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1043134&group_id=5470 From noreply at sourceforge.net Sun Oct 10 17:47:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 10 17:47:28 2004 Subject: [ python-Bugs-1011380 ] Make GIL acquire/release functions noops if PyEval_Init Message-ID: Bugs item #1011380, was opened at 2004-08-18 22:07 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1011380&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Gustavo J. A. M. Carneiro (gustavo) Assigned to: Nobody/Anonymous (nobody) Summary: Make GIL acquire/release functions noops if PyEval_Init Initial Comment: The patch makes GIL acquire/release functions idempotent if PyEval_InitThreads hasn't been called. Motivation for this change can be found in my post to python-dev, here: http://mail.python.org/pipermail/python-dev/2004-August/047870.html http://mail.python.org/pipermail/python-dev/2004-August/047871.html Some feedback here: http://mail.python.org/pipermail/python-dev/2004-August/047949.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-11 01:47 Message: Logged In: YES user_id=1038590 The patch doesn't apply cleanly to current CVS (no actual conflict that I can see - just the patch machinery getting confused by a cautionary comment added to the top of pystate.c) Inspecting the patch manually, it looks like it does the right thing, given that Python instructs that Py_Initialise and PyEval_InitThreads *both* be called prior while the application is still single-threaded. However, _PyThread_Started seems like an odd bit of code - prior to this patch, it is set (initialised to 0 in pythonrun.c, set to 1 by PyEval_InitThreads), but never referenced anywhere (despite the comment saying that it is a "Flag for Py_Exit"). It's also not part of any API (even an internal one). It is defined in pythonrun.c, then declared as extern in ceval.c. The patch adds another extern declaration in pystate.c It would seem better (and only slightly more work) to add a real API (such as adding PyThread_IsInitialized() to pythread.h), rather than entrenching further use of _PyThread_Started. ---------------------------------------------------------------------- Comment By: Gustavo J. A. M. Carneiro (gustavo) Date: 2004-09-29 00:25 Message: Logged In: YES user_id=908 Can someone please take a look at this patch? It's fairly simple, but very important. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1011380&group_id=5470 From noreply at sourceforge.net Sun Oct 10 19:08:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 10 19:08:55 2004 Subject: [ python-Bugs-1011380 ] Make GIL acquire/release functions noops if PyEval_Init Message-ID: Bugs item #1011380, was opened at 2004-08-18 22:07 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1011380&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Gustavo J. A. M. Carneiro (gustavo) Assigned to: Nobody/Anonymous (nobody) Summary: Make GIL acquire/release functions noops if PyEval_Init Initial Comment: The patch makes GIL acquire/release functions idempotent if PyEval_InitThreads hasn't been called. Motivation for this change can be found in my post to python-dev, here: http://mail.python.org/pipermail/python-dev/2004-August/047870.html http://mail.python.org/pipermail/python-dev/2004-August/047871.html Some feedback here: http://mail.python.org/pipermail/python-dev/2004-August/047949.html ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-11 03:08 Message: Logged In: YES user_id=1038590 Patch #1044089 has been created with an alternate implementation of the same concept. Rather than using _PyThread_Started, it: - uses interpreter_lock within ceval.c, which matches the existing ceval.c code that uses the existence of the GIL to determine if PyEval_InitThreads has been called - adds a new API function PyEval_ThreadsInitialized which allows external code to check if the GIL exists or not - uses this new API in the PyGILState_* functions The same functions get modified as in the patch attached here - the only difference is in how the code asks the question "Has PyEval_InitThreads been called?" ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-11 01:47 Message: Logged In: YES user_id=1038590 The patch doesn't apply cleanly to current CVS (no actual conflict that I can see - just the patch machinery getting confused by a cautionary comment added to the top of pystate.c) Inspecting the patch manually, it looks like it does the right thing, given that Python instructs that Py_Initialise and PyEval_InitThreads *both* be called prior while the application is still single-threaded. However, _PyThread_Started seems like an odd bit of code - prior to this patch, it is set (initialised to 0 in pythonrun.c, set to 1 by PyEval_InitThreads), but never referenced anywhere (despite the comment saying that it is a "Flag for Py_Exit"). It's also not part of any API (even an internal one). It is defined in pythonrun.c, then declared as extern in ceval.c. The patch adds another extern declaration in pystate.c It would seem better (and only slightly more work) to add a real API (such as adding PyThread_IsInitialized() to pythread.h), rather than entrenching further use of _PyThread_Started. ---------------------------------------------------------------------- Comment By: Gustavo J. A. M. Carneiro (gustavo) Date: 2004-09-29 00:25 Message: Logged In: YES user_id=908 Can someone please take a look at this patch? It's fairly simple, but very important. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1011380&group_id=5470 From noreply at sourceforge.net Sun Oct 10 19:12:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 10 19:12:47 2004 Subject: [ python-Bugs-1011380 ] Make GIL acquire/release functions noops if PyEval_Init Message-ID: Bugs item #1011380, was opened at 2004-08-18 08:07 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1011380&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Gustavo J. A. M. Carneiro (gustavo) Assigned to: Nobody/Anonymous (nobody) Summary: Make GIL acquire/release functions noops if PyEval_Init Initial Comment: The patch makes GIL acquire/release functions idempotent if PyEval_InitThreads hasn't been called. Motivation for this change can be found in my post to python-dev, here: http://mail.python.org/pipermail/python-dev/2004-August/047870.html http://mail.python.org/pipermail/python-dev/2004-August/047871.html Some feedback here: http://mail.python.org/pipermail/python-dev/2004-August/047949.html ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-10 13:12 Message: Logged In: YES user_id=31435 Sorry, I'm opposed to this patch. Slowing every lock operation in the core by an additional test+branch makes everyone pay for what you want, and muddies semantics that haven't changed since 1991 . If you need to do something here, it would be better to go back to your first python-dev msg's approach. Note that the core never calls lock operations unless ceval.c's interperter_lock is non-NULL. That's how *it* avoids the expense of lock fiddling so long as only one thread is in use. It's not a goal to make that easy to spell (we're coding in C here, not Python). It may be a goal to expose enough machinery so that motivated extension module authors can endure the same kind of pain. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-10 13:08 Message: Logged In: YES user_id=1038590 Patch #1044089 has been created with an alternate implementation of the same concept. Rather than using _PyThread_Started, it: - uses interpreter_lock within ceval.c, which matches the existing ceval.c code that uses the existence of the GIL to determine if PyEval_InitThreads has been called - adds a new API function PyEval_ThreadsInitialized which allows external code to check if the GIL exists or not - uses this new API in the PyGILState_* functions The same functions get modified as in the patch attached here - the only difference is in how the code asks the question "Has PyEval_InitThreads been called?" ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-10 11:47 Message: Logged In: YES user_id=1038590 The patch doesn't apply cleanly to current CVS (no actual conflict that I can see - just the patch machinery getting confused by a cautionary comment added to the top of pystate.c) Inspecting the patch manually, it looks like it does the right thing, given that Python instructs that Py_Initialise and PyEval_InitThreads *both* be called prior while the application is still single-threaded. However, _PyThread_Started seems like an odd bit of code - prior to this patch, it is set (initialised to 0 in pythonrun.c, set to 1 by PyEval_InitThreads), but never referenced anywhere (despite the comment saying that it is a "Flag for Py_Exit"). It's also not part of any API (even an internal one). It is defined in pythonrun.c, then declared as extern in ceval.c. The patch adds another extern declaration in pystate.c It would seem better (and only slightly more work) to add a real API (such as adding PyThread_IsInitialized() to pythread.h), rather than entrenching further use of _PyThread_Started. ---------------------------------------------------------------------- Comment By: Gustavo J. A. M. Carneiro (gustavo) Date: 2004-09-28 10:25 Message: Logged In: YES user_id=908 Can someone please take a look at this patch? It's fairly simple, but very important. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1011380&group_id=5470 From noreply at sourceforge.net Mon Oct 11 03:48:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 03:48:51 2004 Subject: [ python-Bugs-1011380 ] Make GIL acquire/release functions noops if PyEval_Init Message-ID: Bugs item #1011380, was opened at 2004-08-18 08:07 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1011380&group_id=5470 Category: Threads Group: Python 2.4 >Status: Closed >Resolution: Rejected Priority: 5 Submitted By: Gustavo J. A. M. Carneiro (gustavo) Assigned to: Nobody/Anonymous (nobody) Summary: Make GIL acquire/release functions noops if PyEval_Init Initial Comment: The patch makes GIL acquire/release functions idempotent if PyEval_InitThreads hasn't been called. Motivation for this change can be found in my post to python-dev, here: http://mail.python.org/pipermail/python-dev/2004-August/047870.html http://mail.python.org/pipermail/python-dev/2004-August/047871.html Some feedback here: http://mail.python.org/pipermail/python-dev/2004-August/047949.html ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-10 21:48 Message: Logged In: YES user_id=31435 Rejected in favor of 1044089. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-10 13:12 Message: Logged In: YES user_id=31435 Sorry, I'm opposed to this patch. Slowing every lock operation in the core by an additional test+branch makes everyone pay for what you want, and muddies semantics that haven't changed since 1991 . If you need to do something here, it would be better to go back to your first python-dev msg's approach. Note that the core never calls lock operations unless ceval.c's interperter_lock is non-NULL. That's how *it* avoids the expense of lock fiddling so long as only one thread is in use. It's not a goal to make that easy to spell (we're coding in C here, not Python). It may be a goal to expose enough machinery so that motivated extension module authors can endure the same kind of pain. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-10 13:08 Message: Logged In: YES user_id=1038590 Patch #1044089 has been created with an alternate implementation of the same concept. Rather than using _PyThread_Started, it: - uses interpreter_lock within ceval.c, which matches the existing ceval.c code that uses the existence of the GIL to determine if PyEval_InitThreads has been called - adds a new API function PyEval_ThreadsInitialized which allows external code to check if the GIL exists or not - uses this new API in the PyGILState_* functions The same functions get modified as in the patch attached here - the only difference is in how the code asks the question "Has PyEval_InitThreads been called?" ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-10 11:47 Message: Logged In: YES user_id=1038590 The patch doesn't apply cleanly to current CVS (no actual conflict that I can see - just the patch machinery getting confused by a cautionary comment added to the top of pystate.c) Inspecting the patch manually, it looks like it does the right thing, given that Python instructs that Py_Initialise and PyEval_InitThreads *both* be called prior while the application is still single-threaded. However, _PyThread_Started seems like an odd bit of code - prior to this patch, it is set (initialised to 0 in pythonrun.c, set to 1 by PyEval_InitThreads), but never referenced anywhere (despite the comment saying that it is a "Flag for Py_Exit"). It's also not part of any API (even an internal one). It is defined in pythonrun.c, then declared as extern in ceval.c. The patch adds another extern declaration in pystate.c It would seem better (and only slightly more work) to add a real API (such as adding PyThread_IsInitialized() to pythread.h), rather than entrenching further use of _PyThread_Started. ---------------------------------------------------------------------- Comment By: Gustavo J. A. M. Carneiro (gustavo) Date: 2004-09-28 10:25 Message: Logged In: YES user_id=908 Can someone please take a look at this patch? It's fairly simple, but very important. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1011380&group_id=5470 From noreply at sourceforge.net Mon Oct 11 03:58:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 04:05:39 2004 Subject: [ python-Bugs-1044299 ] compile error with stlport Message-ID: Bugs item #1044299, was opened at 2004-10-10 20:58 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044299&group_id=5470 Category: Build Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Debug (nbdebug) Assigned to: Nobody/Anonymous (nobody) Summary: compile error with stlport Initial Comment: using the MSVC 7.1 .NET 2003 .sln and .vcproj in the PCBuild folder truncation of identifier warnings and errors compiling resources are generated from STLPort which breaks compilation as STLPort requires those to be unchanged. no other project i have compiled causes this i am using python latest CVS tree, and i use STLPort 4.62 and 5.0cvs niether of qhich display this behaviour with the 50 or so active projects i compile regularly. \STLport-4.6.2\stlport/stl/_config_compat.h(33) : warning RC4011: identifier truncated to '__STL_MINIMUM_DEFAULT_TEMPLATE_' \STLport-4.6.2\stlport/stl/_config_compat.h(33) : warning RC4011: identifier truncated to '_STLP_MINIMUM_DEFAULT_TEMPLATE_' \STLport-4.6.2\stlport/config/stl_mycomp.h(33) : error RC2188: \STLport-4.6.2\stlport/stl/_config.h(102) : warning RC4011: identifier truncated to '_STLP_NO_MEMBER_TEMPLATE_CLASSE' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_NO_MEMBER_TEMPLATE_CLASSE' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_DONT_USE_NESTED_TCLASS_TH' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_USE_NESTED_TCLASS_THROUGH' \STLport-4.6.2\stlport/stl/_config.h(110) : warning RC4011: identifier truncated to '_STLP_NO_CLASS_PARTIAL_SPECIALI' \STLport-4.6.2\stlport/stl/_config.h(110) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(114) : warning RC4011: identifier truncated to '_STLP_FUNCTION_TMPL_PARTIAL_ORD' \STLport-4.6.2\stlport/stl/_config.h(114) : warning RC4011: identifier truncated to '_STLP_NO_FUNCTION_TMPL_PARTIAL_' \STLport-4.6.2\stlport/stl/_config.h(211) : warning RC4011: identifier truncated to '_STLP_NATIVE_OLD_STREAMS_INCLUD' \STLport-4.6.2\stlport/stl/_config.h(219) : warning RC4011: identifier truncated to '_STLP_NATIVE_CPP_RUNTIME_INCLUD' \STLport-4.6.2\stlport/stl/_config.h(440) : warning RC4011: identifier truncated to '_STLP_USE_WRAPPER_FOR_ALLOC_PAR' \STLport-4.6.2\stlport/stl/_config.h(584) : warning RC4011: identifier truncated to '_STLP_BOGUS_TEMPLATE_TYPE_MATCH' \STLport-4.6.2\stlport/stl/_config.h(619) : warning RC4011: identifier truncated to '_STLP_NO_TYPENAME_ON_RETURN_TYP' \STLport-4.6.2\stlport/stl/_config.h(708) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(716) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(905) : warning RC4011: identifier truncated to '_STLP_DLLEXPORT_NEEDS_PREDECLAR' \STLport-4.6.2\stlport/stl/_config.h(1009) : warning RC4011: identifier truncated to '_STLP_DONT_USE_PARTIAL_SPEC_WRK' \STLport-4.6.2\stlport/stl/_config.h(1010) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(1010) : warning RC4011: identifier truncated to '_STLP_FUNCTION_TMPL_PARTIAL_ORD' \STLport-4.6.2\stlport/stl/_prolog.h(15) : warning RC4011: identifier truncated to '_STLP_HAS_SPECIFIC_PROLOG_EPILO' \STLport-4.6.2\stlport/stdarg.h(23) : fatal error RC1015: cannot open include file '.../include/stdarg.h'. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044299&group_id=5470 From noreply at sourceforge.net Mon Oct 11 05:00:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 05:00:15 2004 Subject: [ python-Bugs-1044299 ] compile error with stlport Message-ID: Bugs item #1044299, was opened at 2004-10-10 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044299&group_id=5470 Category: Build Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Debug (nbdebug) Assigned to: Nobody/Anonymous (nobody) Summary: compile error with stlport Initial Comment: using the MSVC 7.1 .NET 2003 .sln and .vcproj in the PCBuild folder truncation of identifier warnings and errors compiling resources are generated from STLPort which breaks compilation as STLPort requires those to be unchanged. no other project i have compiled causes this i am using python latest CVS tree, and i use STLPort 4.62 and 5.0cvs niether of qhich display this behaviour with the 50 or so active projects i compile regularly. \STLport-4.6.2\stlport/stl/_config_compat.h(33) : warning RC4011: identifier truncated to '__STL_MINIMUM_DEFAULT_TEMPLATE_' \STLport-4.6.2\stlport/stl/_config_compat.h(33) : warning RC4011: identifier truncated to '_STLP_MINIMUM_DEFAULT_TEMPLATE_' \STLport-4.6.2\stlport/config/stl_mycomp.h(33) : error RC2188: \STLport-4.6.2\stlport/stl/_config.h(102) : warning RC4011: identifier truncated to '_STLP_NO_MEMBER_TEMPLATE_CLASSE' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_NO_MEMBER_TEMPLATE_CLASSE' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_DONT_USE_NESTED_TCLASS_TH' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_USE_NESTED_TCLASS_THROUGH' \STLport-4.6.2\stlport/stl/_config.h(110) : warning RC4011: identifier truncated to '_STLP_NO_CLASS_PARTIAL_SPECIALI' \STLport-4.6.2\stlport/stl/_config.h(110) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(114) : warning RC4011: identifier truncated to '_STLP_FUNCTION_TMPL_PARTIAL_ORD' \STLport-4.6.2\stlport/stl/_config.h(114) : warning RC4011: identifier truncated to '_STLP_NO_FUNCTION_TMPL_PARTIAL_' \STLport-4.6.2\stlport/stl/_config.h(211) : warning RC4011: identifier truncated to '_STLP_NATIVE_OLD_STREAMS_INCLUD' \STLport-4.6.2\stlport/stl/_config.h(219) : warning RC4011: identifier truncated to '_STLP_NATIVE_CPP_RUNTIME_INCLUD' \STLport-4.6.2\stlport/stl/_config.h(440) : warning RC4011: identifier truncated to '_STLP_USE_WRAPPER_FOR_ALLOC_PAR' \STLport-4.6.2\stlport/stl/_config.h(584) : warning RC4011: identifier truncated to '_STLP_BOGUS_TEMPLATE_TYPE_MATCH' \STLport-4.6.2\stlport/stl/_config.h(619) : warning RC4011: identifier truncated to '_STLP_NO_TYPENAME_ON_RETURN_TYP' \STLport-4.6.2\stlport/stl/_config.h(708) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(716) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(905) : warning RC4011: identifier truncated to '_STLP_DLLEXPORT_NEEDS_PREDECLAR' \STLport-4.6.2\stlport/stl/_config.h(1009) : warning RC4011: identifier truncated to '_STLP_DONT_USE_PARTIAL_SPEC_WRK' \STLport-4.6.2\stlport/stl/_config.h(1010) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(1010) : warning RC4011: identifier truncated to '_STLP_FUNCTION_TMPL_PARTIAL_ORD' \STLport-4.6.2\stlport/stl/_prolog.h(15) : warning RC4011: identifier truncated to '_STLP_HAS_SPECIFIC_PROLOG_EPILO' \STLport-4.6.2\stlport/stdarg.h(23) : fatal error RC1015: cannot open include file '.../include/stdarg.h'. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-10 23:00 Message: Logged In: YES user_id=31435 Can you explain what this has to do with Python? There is no reference to STLPort in the Python codebase, so I don't understand why you posted this to a Python bug tracker. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044299&group_id=5470 From noreply at sourceforge.net Mon Oct 11 09:11:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 09:12:04 2004 Subject: [ python-Bugs-950747 ] email: mishandles Content-Transfer-Enc. for text/* messages Message-ID: Bugs item #950747, was opened at 2004-05-09 20:19 Message generated for change (Comment added) made by dgibson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950747&group_id=5470 Category: Python Library Group: Python 2.3 Status: Closed Resolution: Wont Fix Priority: 5 Submitted By: Matthias Klose (doko) Assigned to: Barry A. Warsaw (bwarsaw) Summary: email: mishandles Content-Transfer-Enc. for text/* messages Initial Comment: [forwarded from http://bugs.debian.org/247792] Normally when creating MIME attachements with the email module, the user must explicitly encode (base64 or quoted-printable) the message using encode_base64() or encode_quopri(), for example. That sets the Content-Transfer-Encoding header, and replaces the payload with an encoded version which is then emitted verbatim by the Generator. However, for text/ objects, the Content-Transfer-Encoding header is set when the payload is attached, according to the character set. The payload is not encoded at that point, but is instead encoded according to the charset's default body encoding by the Generator at flatten() time. This means in particular that using encode_*() on a text/* message with a non-None charset will result in a malformed message: it will have multiple Content-Transfer-Encoding headers, and multiply encoded body. For added confusion the multiple body encodings won't even be applied in the same order as the duplicate headers appear (as they would if multiple encode_*() functions were applied in sequence). It also means it is impossible to override a charset's default encoding. For example utf-8 text will always be base64 encoded, even though utf-8 text in western languages is likely to be almost entirely 7bit, so quoted-printable or even 8bit encoding would be more appropriate. The payload data of the Message object should consistently hold either encoded or unencoded data. If the latter, the Generator should take its cue from the Content-Transfer-Encoding header, not from the charset. If the former, the encode_* functions should recognize an already encoded message and recode it, or at least throw an exception rather than generating a malformed MIME message. ---------------------------------------------------------------------- Comment By: David Gibson (dgibson) Date: 2004-10-11 17:11 Message: Logged In: YES user_id=30682 This reply seems to miss the point. Having the semantics of the payload string (encoded at insertion vs. encoded at flattening) be different for text/ and non text/ messages is conceptually broken and confusing. Not being able to override a charset's default encoder is also a genuine deficiency. It is not sane to send out english, 7-bit messages which happen to be UTF-8 encoded as ungreppable BASE64 blobs. I really don't think this is hard to fix properly. I'll try to get a patch together, to demonstrate. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-10-10 07:50 Message: Logged In: YES user_id=12800 In Python 2.3, the _encoder argument to MIMEText.__init__() is deprecated, and in Python 2.4 it is removed. The right way to encode the payload for a text/* message is to provide the charset, not to provide an _encoder. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950747&group_id=5470 From noreply at sourceforge.net Mon Oct 11 09:22:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 09:22:43 2004 Subject: [ python-Bugs-1044382 ] Can't raise "C API version mismatch" warning Message-ID: Bugs item #1044382, was opened at 2004-10-11 09:22 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044382&group_id=5470 Category: Extension Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Frank Sonnenburg (sonnenburg) Assigned to: Nobody/Anonymous (nobody) Summary: Can't raise "C API version mismatch" warning Initial Comment: See following example (that's what we know): # Importing 2.1 module with Python 2.3 frank@theta:~> python Python 2.3+ (#1, Jan 7 2004, 09:17:35) [GCC 3.3.1 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.insert(0, '/usr/lib/python2.1/lib-dynload') >>> import cmath __main__:1: RuntimeWarning: Python C API version mismatch for module cmath: This Python has API version 1012, module cmath has version 1010. >>> ================================== Ignoring this warning works without problem, but raising an exception will lead to segmentation fault: # Starting python with warnings turned into exceptions frank@theta:~> python -W error Python 2.3+ (#1, Jan 7 2004, 09:17:35) [GCC 3.3.1 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.insert(0, '/usr/lib/python2.1/lib-dynload') >>> import cmath Speicherzugriffsfehler (seg fault) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044382&group_id=5470 From noreply at sourceforge.net Mon Oct 11 12:28:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 12:28:48 2004 Subject: [ python-Bugs-1044479 ] docs for Py_UNICODE are wrong Message-ID: Bugs item #1044479, was opened at 2004-10-11 12:28 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044479&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 5 Submitted By: Thomas Heller (theller) Assigned to: M.-A. Lemburg (lemburg) Summary: docs for Py_UNICODE are wrong Initial Comment: Quoting the docs (Python C/API manual, section 7.3.2, unicode objects): Py_UNICODE This type represents a 16-bit unsigned storage type which is used by Python internally as basis for holding Unicode ordinals. On platforms where wchar_t is available and also has 16-bits, Py_UNICODE is a typedef alias for wchar_t to enhance native platform compatibility. On all other platforms, Py_UNICODE is a typedef alias for unsigned short. This is apparently wrong for wide unicode builds. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044479&group_id=5470 From noreply at sourceforge.net Mon Oct 11 14:07:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 14:07:46 2004 Subject: [ python-Bugs-1042238 ] Lib/compiler chokes on certain genexps Message-ID: Bugs item #1042238, was opened at 2004-10-07 14:35 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: Lib/compiler chokes on certain genexps Initial Comment: Noticed this because test_compiler fails on _strptime as of yesterday. Here's what seems to be a fairly minimal example: f((x for x in y), 1) The traceback is huge, but ends with: File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 687, in power node = self.com_apply_trailer(node, elt) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1144, in com_apply_trailer return self.com_call_function(primaryNode, nodelist[2]) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1169, in com_call_function raise SyntaxError, 'generator expression needs parenthesis' SyntaxError: generator expression needs parenthesis Might be a problem in transformer.py, not really sure. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-11 13:07 Message: Logged In: YES user_id=6656 No. I'm hoping jiwon (he who wrote the genexps patch) does -- I've sent him a mail. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-10 09:07 Message: Logged In: YES user_id=80475 Do you know how to fix this? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 14:38 Message: Logged In: YES user_id=6656 Hang on, the exception is *inside* transformer.py. Duh. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 From noreply at sourceforge.net Mon Oct 11 14:40:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 14:40:34 2004 Subject: [ python-Bugs-1044299 ] compile error with stlport Message-ID: Bugs item #1044299, was opened at 2004-10-10 20:58 Message generated for change (Comment added) made by nbdebug You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044299&group_id=5470 Category: Build Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Debug (nbdebug) Assigned to: Nobody/Anonymous (nobody) Summary: compile error with stlport Initial Comment: using the MSVC 7.1 .NET 2003 .sln and .vcproj in the PCBuild folder truncation of identifier warnings and errors compiling resources are generated from STLPort which breaks compilation as STLPort requires those to be unchanged. no other project i have compiled causes this i am using python latest CVS tree, and i use STLPort 4.62 and 5.0cvs niether of qhich display this behaviour with the 50 or so active projects i compile regularly. \STLport-4.6.2\stlport/stl/_config_compat.h(33) : warning RC4011: identifier truncated to '__STL_MINIMUM_DEFAULT_TEMPLATE_' \STLport-4.6.2\stlport/stl/_config_compat.h(33) : warning RC4011: identifier truncated to '_STLP_MINIMUM_DEFAULT_TEMPLATE_' \STLport-4.6.2\stlport/config/stl_mycomp.h(33) : error RC2188: \STLport-4.6.2\stlport/stl/_config.h(102) : warning RC4011: identifier truncated to '_STLP_NO_MEMBER_TEMPLATE_CLASSE' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_NO_MEMBER_TEMPLATE_CLASSE' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_DONT_USE_NESTED_TCLASS_TH' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_USE_NESTED_TCLASS_THROUGH' \STLport-4.6.2\stlport/stl/_config.h(110) : warning RC4011: identifier truncated to '_STLP_NO_CLASS_PARTIAL_SPECIALI' \STLport-4.6.2\stlport/stl/_config.h(110) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(114) : warning RC4011: identifier truncated to '_STLP_FUNCTION_TMPL_PARTIAL_ORD' \STLport-4.6.2\stlport/stl/_config.h(114) : warning RC4011: identifier truncated to '_STLP_NO_FUNCTION_TMPL_PARTIAL_' \STLport-4.6.2\stlport/stl/_config.h(211) : warning RC4011: identifier truncated to '_STLP_NATIVE_OLD_STREAMS_INCLUD' \STLport-4.6.2\stlport/stl/_config.h(219) : warning RC4011: identifier truncated to '_STLP_NATIVE_CPP_RUNTIME_INCLUD' \STLport-4.6.2\stlport/stl/_config.h(440) : warning RC4011: identifier truncated to '_STLP_USE_WRAPPER_FOR_ALLOC_PAR' \STLport-4.6.2\stlport/stl/_config.h(584) : warning RC4011: identifier truncated to '_STLP_BOGUS_TEMPLATE_TYPE_MATCH' \STLport-4.6.2\stlport/stl/_config.h(619) : warning RC4011: identifier truncated to '_STLP_NO_TYPENAME_ON_RETURN_TYP' \STLport-4.6.2\stlport/stl/_config.h(708) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(716) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(905) : warning RC4011: identifier truncated to '_STLP_DLLEXPORT_NEEDS_PREDECLAR' \STLport-4.6.2\stlport/stl/_config.h(1009) : warning RC4011: identifier truncated to '_STLP_DONT_USE_PARTIAL_SPEC_WRK' \STLport-4.6.2\stlport/stl/_config.h(1010) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(1010) : warning RC4011: identifier truncated to '_STLP_FUNCTION_TMPL_PARTIAL_ORD' \STLport-4.6.2\stlport/stl/_prolog.h(15) : warning RC4011: identifier truncated to '_STLP_HAS_SPECIFIC_PROLOG_EPILO' \STLport-4.6.2\stlport/stdarg.h(23) : fatal error RC1015: cannot open include file '.../include/stdarg.h'. ---------------------------------------------------------------------- >Comment By: Debug (nbdebug) Date: 2004-10-11 07:40 Message: Logged In: YES user_id=1136723 well my point is its the python .rc files that are generating this behaviour no other project does this and since its an stl error i can only assume something must use stl... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-10 22:00 Message: Logged In: YES user_id=31435 Can you explain what this has to do with Python? There is no reference to STLPort in the Python codebase, so I don't understand why you posted this to a Python bug tracker. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044299&group_id=5470 From noreply at sourceforge.net Mon Oct 11 15:09:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 15:09:33 2004 Subject: [ python-Bugs-1042238 ] Lib/compiler chokes on certain genexps Message-ID: Bugs item #1042238, was opened at 2004-10-07 23:35 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 Category: None Group: None Status: Open Resolution: None >Priority: 9 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: Lib/compiler chokes on certain genexps Initial Comment: Noticed this because test_compiler fails on _strptime as of yesterday. Here's what seems to be a fairly minimal example: f((x for x in y), 1) The traceback is huge, but ends with: File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 687, in power node = self.com_apply_trailer(node, elt) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1144, in com_apply_trailer return self.com_call_function(primaryNode, nodelist[2]) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1169, in com_call_function raise SyntaxError, 'generator expression needs parenthesis' SyntaxError: generator expression needs parenthesis Might be a problem in transformer.py, not really sure. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-11 23:09 Message: Logged In: YES user_id=29957 Failing case, distilled down: import compiler a="""def foo(*x): return x foo((a for b in [[1,2,3],[4,5,6]] for a in b),'Z')""" compiler.compile(a,'', 'exec') ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-11 22:07 Message: Logged In: YES user_id=6656 No. I'm hoping jiwon (he who wrote the genexps patch) does -- I've sent him a mail. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-10 18:07 Message: Logged In: YES user_id=80475 Do you know how to fix this? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 23:38 Message: Logged In: YES user_id=6656 Hang on, the exception is *inside* transformer.py. Duh. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 From noreply at sourceforge.net Mon Oct 11 15:12:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 15:12:26 2004 Subject: [ python-Bugs-1042238 ] Lib/compiler chokes on certain genexps Message-ID: Bugs item #1042238, was opened at 2004-10-07 14:35 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 9 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: Lib/compiler chokes on certain genexps Initial Comment: Noticed this because test_compiler fails on _strptime as of yesterday. Here's what seems to be a fairly minimal example: f((x for x in y), 1) The traceback is huge, but ends with: File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 687, in power node = self.com_apply_trailer(node, elt) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1144, in com_apply_trailer return self.com_call_function(primaryNode, nodelist[2]) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1169, in com_call_function raise SyntaxError, 'generator expression needs parenthesis' SyntaxError: generator expression needs parenthesis Might be a problem in transformer.py, not really sure. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-11 14:12 Message: Logged In: YES user_id=6656 Rather more minimal example: from compiler import transformer transformer.parse("f((x for x in y), 1)") The problem is in transformer.py. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-11 14:09 Message: Logged In: YES user_id=29957 Failing case, distilled down: import compiler a="""def foo(*x): return x foo((a for b in [[1,2,3],[4,5,6]] for a in b),'Z')""" compiler.compile(a,'', 'exec') ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-11 13:07 Message: Logged In: YES user_id=6656 No. I'm hoping jiwon (he who wrote the genexps patch) does -- I've sent him a mail. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-10 09:07 Message: Logged In: YES user_id=80475 Do you know how to fix this? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 14:38 Message: Logged In: YES user_id=6656 Hang on, the exception is *inside* transformer.py. Duh. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 From noreply at sourceforge.net Mon Oct 11 16:24:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 16:24:13 2004 Subject: [ python-Bugs-1042238 ] Lib/compiler chokes on certain genexps Message-ID: Bugs item #1042238, was opened at 2004-10-07 22:35 Message generated for change (Comment added) made by jiwon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 9 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: Lib/compiler chokes on certain genexps Initial Comment: Noticed this because test_compiler fails on _strptime as of yesterday. Here's what seems to be a fairly minimal example: f((x for x in y), 1) The traceback is huge, but ends with: File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 687, in power node = self.com_apply_trailer(node, elt) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1144, in com_apply_trailer return self.com_call_function(primaryNode, nodelist[2]) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1169, in com_call_function raise SyntaxError, 'generator expression needs parenthesis' SyntaxError: generator expression needs parenthesis Might be a problem in transformer.py, not really sure. ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-10-11 23:24 Message: Logged In: YES user_id=595483 I fixed the bug. The bug was due to the wrong check condition for allowing "f(x for x in y)", but rejecting "f(x for x in y, 1)" The patch is http://seojiwon.dnip.net:8000/~jiwon/transformer.diff I hope mwh will upload that file here, since I don't have the permission. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-11 22:12 Message: Logged In: YES user_id=6656 Rather more minimal example: from compiler import transformer transformer.parse("f((x for x in y), 1)") The problem is in transformer.py. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-11 22:09 Message: Logged In: YES user_id=29957 Failing case, distilled down: import compiler a="""def foo(*x): return x foo((a for b in [[1,2,3],[4,5,6]] for a in b),'Z')""" compiler.compile(a,'', 'exec') ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-11 21:07 Message: Logged In: YES user_id=6656 No. I'm hoping jiwon (he who wrote the genexps patch) does -- I've sent him a mail. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-10 17:07 Message: Logged In: YES user_id=80475 Do you know how to fix this? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 22:38 Message: Logged In: YES user_id=6656 Hang on, the exception is *inside* transformer.py. Duh. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 From noreply at sourceforge.net Mon Oct 11 16:31:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 16:31:23 2004 Subject: [ python-Bugs-1042238 ] Lib/compiler chokes on certain genexps Message-ID: Bugs item #1042238, was opened at 2004-10-07 14:35 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 9 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: Lib/compiler chokes on certain genexps Initial Comment: Noticed this because test_compiler fails on _strptime as of yesterday. Here's what seems to be a fairly minimal example: f((x for x in y), 1) The traceback is huge, but ends with: File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 687, in power node = self.com_apply_trailer(node, elt) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1144, in com_apply_trailer return self.com_call_function(primaryNode, nodelist[2]) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1169, in com_call_function raise SyntaxError, 'generator expression needs parenthesis' SyntaxError: generator expression needs parenthesis Might be a problem in transformer.py, not really sure. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-11 15:31 Message: Logged In: YES user_id=6656 Attaching Jiwon's patch (after converting it to a context diff, *thwack* :-) ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-10-11 15:24 Message: Logged In: YES user_id=595483 I fixed the bug. The bug was due to the wrong check condition for allowing "f(x for x in y)", but rejecting "f(x for x in y, 1)" The patch is http://seojiwon.dnip.net:8000/~jiwon/transformer.diff I hope mwh will upload that file here, since I don't have the permission. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-11 14:12 Message: Logged In: YES user_id=6656 Rather more minimal example: from compiler import transformer transformer.parse("f((x for x in y), 1)") The problem is in transformer.py. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-11 14:09 Message: Logged In: YES user_id=29957 Failing case, distilled down: import compiler a="""def foo(*x): return x foo((a for b in [[1,2,3],[4,5,6]] for a in b),'Z')""" compiler.compile(a,'', 'exec') ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-11 13:07 Message: Logged In: YES user_id=6656 No. I'm hoping jiwon (he who wrote the genexps patch) does -- I've sent him a mail. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-10 09:07 Message: Logged In: YES user_id=80475 Do you know how to fix this? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 14:38 Message: Logged In: YES user_id=6656 Hang on, the exception is *inside* transformer.py. Duh. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 From noreply at sourceforge.net Mon Oct 11 17:36:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 17:36:46 2004 Subject: [ python-Bugs-1042238 ] Lib/compiler chokes on certain genexps Message-ID: Bugs item #1042238, was opened at 2004-10-07 14:35 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 Category: None Group: None >Status: Closed >Resolution: Fixed Priority: 9 Submitted By: Michael Hudson (mwh) >Assigned to: Michael Hudson (mwh) Summary: Lib/compiler chokes on certain genexps Initial Comment: Noticed this because test_compiler fails on _strptime as of yesterday. Here's what seems to be a fairly minimal example: f((x for x in y), 1) The traceback is huge, but ends with: File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 687, in power node = self.com_apply_trailer(node, elt) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1144, in com_apply_trailer return self.com_call_function(primaryNode, nodelist[2]) File "/home/mwh/src/python/dist/src/Lib/compiler/transformer.py", line 1169, in com_call_function raise SyntaxError, 'generator expression needs parenthesis' SyntaxError: generator expression needs parenthesis Might be a problem in transformer.py, not really sure. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-11 16:36 Message: Logged In: YES user_id=6656 Checked in jiwon's fix as Lib/compiler/transformer.py revision 1.48. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-11 15:31 Message: Logged In: YES user_id=6656 Attaching Jiwon's patch (after converting it to a context diff, *thwack* :-) ---------------------------------------------------------------------- Comment By: Jiwon Seo (jiwon) Date: 2004-10-11 15:24 Message: Logged In: YES user_id=595483 I fixed the bug. The bug was due to the wrong check condition for allowing "f(x for x in y)", but rejecting "f(x for x in y, 1)" The patch is http://seojiwon.dnip.net:8000/~jiwon/transformer.diff I hope mwh will upload that file here, since I don't have the permission. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-11 14:12 Message: Logged In: YES user_id=6656 Rather more minimal example: from compiler import transformer transformer.parse("f((x for x in y), 1)") The problem is in transformer.py. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-11 14:09 Message: Logged In: YES user_id=29957 Failing case, distilled down: import compiler a="""def foo(*x): return x foo((a for b in [[1,2,3],[4,5,6]] for a in b),'Z')""" compiler.compile(a,'', 'exec') ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-11 13:07 Message: Logged In: YES user_id=6656 No. I'm hoping jiwon (he who wrote the genexps patch) does -- I've sent him a mail. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-10 09:07 Message: Logged In: YES user_id=80475 Do you know how to fix this? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 14:38 Message: Logged In: YES user_id=6656 Hang on, the exception is *inside* transformer.py. Duh. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1042238&group_id=5470 From noreply at sourceforge.net Mon Oct 11 22:15:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 22:15:24 2004 Subject: [ python-Feature Requests-1043134 ] mimetypes.guess_extension('text/plain') == '.ksh' ??? Message-ID: Feature Requests item #1043134, was opened at 2004-10-08 17:44 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1043134&group_id=5470 >Category: None >Group: None Status: Open Resolution: None Priority: 5 Submitted By: Robert Kiendl (kxroberto) Assigned to: Nobody/Anonymous (nobody) Summary: mimetypes.guess_extension('text/plain') == '.ksh' ??? Initial Comment: Instead of returning the first in the list of extensions it should return the most reasonable . here: to have a *.txt on disk after saveing? ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-11 22:15 Message: Logged In: YES user_id=469548 common_types is for adding some non-standard types, not for determining which extension is most reasonable. I'll be happy to look at a decent patch, but I'm moving this to feature request until then. ---------------------------------------------------------------------- Comment By: Robert Kiendl (kxroberto) Date: 2004-10-10 10:44 Message: Logged In: YES user_id=972995 in mimetypes.py there is already a common_types = { '.jpg' : 'image/jpg', ... .txt could be added, mayby guess_extension should first reverse-take it out of there, not random ...? background: my intent was to save MIME attachment as (startable) temporary file. yet got wonderful .ksh's for textfiles, and had to fumble ... ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-09 17:26 Message: Logged In: YES user_id=469548 How would you suggest finding out what the most reasonable extension for a mime type is? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1043134&group_id=5470 From noreply at sourceforge.net Mon Oct 11 22:15:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 22:15:53 2004 Subject: [ python-Feature Requests-1043134 ] mimetypes.guess_extension('text/plain') == '.ksh' ??? Message-ID: Feature Requests item #1043134, was opened at 2004-10-08 17:44 Message generated for change (Settings changed) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1043134&group_id=5470 >Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Robert Kiendl (kxroberto) Assigned to: Nobody/Anonymous (nobody) Summary: mimetypes.guess_extension('text/plain') == '.ksh' ??? Initial Comment: Instead of returning the first in the list of extensions it should return the most reasonable . here: to have a *.txt on disk after saveing? ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-11 22:15 Message: Logged In: YES user_id=469548 common_types is for adding some non-standard types, not for determining which extension is most reasonable. I'll be happy to look at a decent patch, but I'm moving this to feature request until then. ---------------------------------------------------------------------- Comment By: Robert Kiendl (kxroberto) Date: 2004-10-10 10:44 Message: Logged In: YES user_id=972995 in mimetypes.py there is already a common_types = { '.jpg' : 'image/jpg', ... .txt could be added, mayby guess_extension should first reverse-take it out of there, not random ...? background: my intent was to save MIME attachment as (startable) temporary file. yet got wonderful .ksh's for textfiles, and had to fumble ... ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-09 17:26 Message: Logged In: YES user_id=469548 How would you suggest finding out what the most reasonable extension for a mime type is? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1043134&group_id=5470 From noreply at sourceforge.net Mon Oct 11 22:54:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 11 22:54:36 2004 Subject: [ python-Bugs-1044872 ] Unable to install Docs - Linux Message-ID: Bugs item #1044872, was opened at 2004-10-11 16:54 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044872&group_id=5470 Category: Build Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Colin J. Williams (cjwhrh) Assigned to: Nobody/Anonymous (nobody) Summary: Unable to install Docs - Linux Initial Comment: Please see the tail of the make output below: [6654 refs] TEXINPUTS=/usr/Downloads/Python/Python-2.3.4/Doc/commontex: mkhowto --html --about html/stdabout.dat --iconserver ../icons --favicon ../icons/pyfav.gif --address "See About this document... for information on suggesting changes." --up-link ../index.html --up-title "Python Documentation Index" --global-module-index "../modindex.html" --dvips-safe --dir html/api api/api.tex /bin/sh: line 1: mkhowto: command not found make: *** [html/api/api.html] Error 127 root@P4:/usr/Downloads/Python/Python-2.3.4/Doc# Attempted installation with Debian from Libranet 2.8.1 Latex2HTML has been installed. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044872&group_id=5470 From noreply at sourceforge.net Tue Oct 12 04:27:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 12 04:27:19 2004 Subject: [ python-Bugs-1044299 ] compile error with stlport Message-ID: Bugs item #1044299, was opened at 2004-10-10 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044299&group_id=5470 Category: Build Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Debug (nbdebug) Assigned to: Nobody/Anonymous (nobody) Summary: compile error with stlport Initial Comment: using the MSVC 7.1 .NET 2003 .sln and .vcproj in the PCBuild folder truncation of identifier warnings and errors compiling resources are generated from STLPort which breaks compilation as STLPort requires those to be unchanged. no other project i have compiled causes this i am using python latest CVS tree, and i use STLPort 4.62 and 5.0cvs niether of qhich display this behaviour with the 50 or so active projects i compile regularly. \STLport-4.6.2\stlport/stl/_config_compat.h(33) : warning RC4011: identifier truncated to '__STL_MINIMUM_DEFAULT_TEMPLATE_' \STLport-4.6.2\stlport/stl/_config_compat.h(33) : warning RC4011: identifier truncated to '_STLP_MINIMUM_DEFAULT_TEMPLATE_' \STLport-4.6.2\stlport/config/stl_mycomp.h(33) : error RC2188: \STLport-4.6.2\stlport/stl/_config.h(102) : warning RC4011: identifier truncated to '_STLP_NO_MEMBER_TEMPLATE_CLASSE' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_NO_MEMBER_TEMPLATE_CLASSE' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_DONT_USE_NESTED_TCLASS_TH' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_USE_NESTED_TCLASS_THROUGH' \STLport-4.6.2\stlport/stl/_config.h(110) : warning RC4011: identifier truncated to '_STLP_NO_CLASS_PARTIAL_SPECIALI' \STLport-4.6.2\stlport/stl/_config.h(110) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(114) : warning RC4011: identifier truncated to '_STLP_FUNCTION_TMPL_PARTIAL_ORD' \STLport-4.6.2\stlport/stl/_config.h(114) : warning RC4011: identifier truncated to '_STLP_NO_FUNCTION_TMPL_PARTIAL_' \STLport-4.6.2\stlport/stl/_config.h(211) : warning RC4011: identifier truncated to '_STLP_NATIVE_OLD_STREAMS_INCLUD' \STLport-4.6.2\stlport/stl/_config.h(219) : warning RC4011: identifier truncated to '_STLP_NATIVE_CPP_RUNTIME_INCLUD' \STLport-4.6.2\stlport/stl/_config.h(440) : warning RC4011: identifier truncated to '_STLP_USE_WRAPPER_FOR_ALLOC_PAR' \STLport-4.6.2\stlport/stl/_config.h(584) : warning RC4011: identifier truncated to '_STLP_BOGUS_TEMPLATE_TYPE_MATCH' \STLport-4.6.2\stlport/stl/_config.h(619) : warning RC4011: identifier truncated to '_STLP_NO_TYPENAME_ON_RETURN_TYP' \STLport-4.6.2\stlport/stl/_config.h(708) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(716) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(905) : warning RC4011: identifier truncated to '_STLP_DLLEXPORT_NEEDS_PREDECLAR' \STLport-4.6.2\stlport/stl/_config.h(1009) : warning RC4011: identifier truncated to '_STLP_DONT_USE_PARTIAL_SPEC_WRK' \STLport-4.6.2\stlport/stl/_config.h(1010) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(1010) : warning RC4011: identifier truncated to '_STLP_FUNCTION_TMPL_PARTIAL_ORD' \STLport-4.6.2\stlport/stl/_prolog.h(15) : warning RC4011: identifier truncated to '_STLP_HAS_SPECIFIC_PROLOG_EPILO' \STLport-4.6.2\stlport/stdarg.h(23) : fatal error RC1015: cannot open include file '.../include/stdarg.h'. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-11 22:27 Message: Logged In: YES user_id=31435 No, Python is written to the C89 std. If you look, you won't find a .cpp file in the entire project. Python makes no use of C++, not the STL or anything else. So you need to do some work here -- for example, you haven't said which subproject was being compiled when you got these msgs. Since Python doesn't use the STL, they don't mean anything to me. Python compiles fine on my box, and yours is the only report of this, so you have to identify what you've done that nobody else has done. For example (pure guess), perhaps as part of installing STLPort you fiddled paths so that 7.1 finds STLPort's .h files ahead of its own. If so, that's not Python's problem, but may be something to report on an STLPort bug tracker (like "MS 7.1 can no longer compile C programs after STLPort is installed"). ---------------------------------------------------------------------- Comment By: Debug (nbdebug) Date: 2004-10-11 08:40 Message: Logged In: YES user_id=1136723 well my point is its the python .rc files that are generating this behaviour no other project does this and since its an stl error i can only assume something must use stl... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-10 23:00 Message: Logged In: YES user_id=31435 Can you explain what this has to do with Python? There is no reference to STLPort in the Python codebase, so I don't understand why you posted this to a Python bug tracker. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044299&group_id=5470 From noreply at sourceforge.net Tue Oct 12 16:04:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 12 16:04:20 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 16:04 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Nobody/Anonymous (nobody) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Tue Oct 12 17:34:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 12 17:35:01 2004 Subject: [ python-Bugs-1045509 ] Problems with os.system and ulimit -f Message-ID: Bugs item #1045509, was opened at 2004-10-12 12:34 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045509&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Tomasz Kowaltowski (kowaltowski) Assigned to: Nobody/Anonymous (nobody) Summary: Problems with os.system and ulimit -f Initial Comment: Python version (running under Fedora Core 2 Linux): Python 2.3.3 (#1, May 7 2004, 10:31:40) [GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2 --------------------------------------------------------------------------- I found a problem while executing the 'ulimit -f' bash command within the 'os.system' function. According to the documentation this function should behave exactly like the stdlib function 'system'. However it does not happen as illustrated by the minimal Python and C examples: testsystem.py and testsystem.c (see attached zipped file). In these examples 'largefile' is a compiled C program which writes an endless file into the stdout (also attached). The C program testsystem.c works as expected and prints the following output: command: ulimit -f 10; largefile > xxx; result = 153 The Python program testsystem.py **does not stop**; if interrupted by Ctrl-C it prints: command: ulimit -f 10; largefile > xxx; result = 0 In both cases the output file 'xxx' has 10240 bytes, ie, 10 blocks as limited by 'ulimit'. It is interesting though that the command 'ulimit -t 1' (CPU time) produces correct results under both Python and C versions, ie, interrupts the execution and prints: command: ulimit -t 1; largefile > xxx; result = 137 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045509&group_id=5470 From noreply at sourceforge.net Tue Oct 12 20:17:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 12 20:17:56 2004 Subject: [ python-Bugs-751124 ] Enhancements to pdb.py when invoked as script Message-ID: Bugs item #751124, was opened at 2003-06-09 04:19 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=751124&group_id=5470 Category: Python Library Group: Feature Request >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Christian Stork (cst) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: Enhancements to pdb.py when invoked as script Initial Comment: Three little enhancements/mini-bugs: 1) Adding the following lines to the "main part" of pdb.py gives it the ability to stay in pdb if the program crashes. Very handy! # Ensure that we end up in pdb if we crash def pdb_on_crash(type, value, tb): sys.__excepthook__(type, value, tb) post_mortem(tb) sys.excepthook = pdb_on_crash 2) There's also a problem in the last lines of pdb.py: # Insert script directory in front of module search path sys.path.insert(0, os.path.dirname(filename)) This leaves the directory path under which pdb.py was invoked in sys.path. This is a problem if pdb.py was invoked e.g. via a link a la /usr/pdb. This leads to the last and related suggestion. 3) It would be very convenient if a pyhon installation would also install a link /usr/pdb -> /usr/lib/python2.3/pdb.py. This provides for easy invocation and, for example, it's also the default command used by Emacs' pdb-mode. ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-12 20:17 Message: Logged In: YES user_id=469548 Fixed (1) and (2) by appling patch #896011. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-07 22:57 Message: Logged In: YES user_id=469548 Note that 3 is no longer necessary now we have 'python -m'. ---------------------------------------------------------------------- Comment By: Ilya Sandler (isandler) Date: 2004-03-12 18:24 Message: Logged In: YES user_id=971153 a note for the reviewer: patch #896011 fixes bug (2) and implements (in a different manner but with the same result) feature (1), PS. I would also like to add my vote for (3). Being a command line tool pdb.py deserves to be installed right next to python binary..(on Linux that would often be /usr/bin) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=751124&group_id=5470 From noreply at sourceforge.net Tue Oct 12 21:49:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 12 21:49:52 2004 Subject: [ python-Bugs-1044299 ] compile error with stlport Message-ID: Bugs item #1044299, was opened at 2004-10-10 20:58 Message generated for change (Comment added) made by nbdebug You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044299&group_id=5470 Category: Build Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Debug (nbdebug) Assigned to: Nobody/Anonymous (nobody) Summary: compile error with stlport Initial Comment: using the MSVC 7.1 .NET 2003 .sln and .vcproj in the PCBuild folder truncation of identifier warnings and errors compiling resources are generated from STLPort which breaks compilation as STLPort requires those to be unchanged. no other project i have compiled causes this i am using python latest CVS tree, and i use STLPort 4.62 and 5.0cvs niether of qhich display this behaviour with the 50 or so active projects i compile regularly. \STLport-4.6.2\stlport/stl/_config_compat.h(33) : warning RC4011: identifier truncated to '__STL_MINIMUM_DEFAULT_TEMPLATE_' \STLport-4.6.2\stlport/stl/_config_compat.h(33) : warning RC4011: identifier truncated to '_STLP_MINIMUM_DEFAULT_TEMPLATE_' \STLport-4.6.2\stlport/config/stl_mycomp.h(33) : error RC2188: \STLport-4.6.2\stlport/stl/_config.h(102) : warning RC4011: identifier truncated to '_STLP_NO_MEMBER_TEMPLATE_CLASSE' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_NO_MEMBER_TEMPLATE_CLASSE' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_DONT_USE_NESTED_TCLASS_TH' \STLport-4.6.2\stlport/stl/_config.h(106) : warning RC4011: identifier truncated to '_STLP_USE_NESTED_TCLASS_THROUGH' \STLport-4.6.2\stlport/stl/_config.h(110) : warning RC4011: identifier truncated to '_STLP_NO_CLASS_PARTIAL_SPECIALI' \STLport-4.6.2\stlport/stl/_config.h(110) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(114) : warning RC4011: identifier truncated to '_STLP_FUNCTION_TMPL_PARTIAL_ORD' \STLport-4.6.2\stlport/stl/_config.h(114) : warning RC4011: identifier truncated to '_STLP_NO_FUNCTION_TMPL_PARTIAL_' \STLport-4.6.2\stlport/stl/_config.h(211) : warning RC4011: identifier truncated to '_STLP_NATIVE_OLD_STREAMS_INCLUD' \STLport-4.6.2\stlport/stl/_config.h(219) : warning RC4011: identifier truncated to '_STLP_NATIVE_CPP_RUNTIME_INCLUD' \STLport-4.6.2\stlport/stl/_config.h(440) : warning RC4011: identifier truncated to '_STLP_USE_WRAPPER_FOR_ALLOC_PAR' \STLport-4.6.2\stlport/stl/_config.h(584) : warning RC4011: identifier truncated to '_STLP_BOGUS_TEMPLATE_TYPE_MATCH' \STLport-4.6.2\stlport/stl/_config.h(619) : warning RC4011: identifier truncated to '_STLP_NO_TYPENAME_ON_RETURN_TYP' \STLport-4.6.2\stlport/stl/_config.h(708) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(716) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(905) : warning RC4011: identifier truncated to '_STLP_DLLEXPORT_NEEDS_PREDECLAR' \STLport-4.6.2\stlport/stl/_config.h(1009) : warning RC4011: identifier truncated to '_STLP_DONT_USE_PARTIAL_SPEC_WRK' \STLport-4.6.2\stlport/stl/_config.h(1010) : warning RC4011: identifier truncated to '_STLP_CLASS_PARTIAL_SPECIALIZAT' \STLport-4.6.2\stlport/stl/_config.h(1010) : warning RC4011: identifier truncated to '_STLP_FUNCTION_TMPL_PARTIAL_ORD' \STLport-4.6.2\stlport/stl/_prolog.h(15) : warning RC4011: identifier truncated to '_STLP_HAS_SPECIFIC_PROLOG_EPILO' \STLport-4.6.2\stlport/stdarg.h(23) : fatal error RC1015: cannot open include file '.../include/stdarg.h'. ---------------------------------------------------------------------- >Comment By: Debug (nbdebug) Date: 2004-10-12 14:49 Message: Logged In: YES user_id=1136723 i discovered the python resources are including files that are part of stlport i managed to stop the behaviour by setting ignore default include diretory and adding the path to visual studio includes locations ( python required 2 of a possible 4 ) this fixed the compile ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-11 21:27 Message: Logged In: YES user_id=31435 No, Python is written to the C89 std. If you look, you won't find a .cpp file in the entire project. Python makes no use of C++, not the STL or anything else. So you need to do some work here -- for example, you haven't said which subproject was being compiled when you got these msgs. Since Python doesn't use the STL, they don't mean anything to me. Python compiles fine on my box, and yours is the only report of this, so you have to identify what you've done that nobody else has done. For example (pure guess), perhaps as part of installing STLPort you fiddled paths so that 7.1 finds STLPort's .h files ahead of its own. If so, that's not Python's problem, but may be something to report on an STLPort bug tracker (like "MS 7.1 can no longer compile C programs after STLPort is installed"). ---------------------------------------------------------------------- Comment By: Debug (nbdebug) Date: 2004-10-11 07:40 Message: Logged In: YES user_id=1136723 well my point is its the python .rc files that are generating this behaviour no other project does this and since its an stl error i can only assume something must use stl... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-10 22:00 Message: Logged In: YES user_id=31435 Can you explain what this has to do with Python? There is no reference to STLPort in the Python codebase, so I don't understand why you posted this to a Python bug tracker. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044299&group_id=5470 From noreply at sourceforge.net Tue Oct 12 23:58:02 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 12 23:58:06 2004 Subject: [ python-Bugs-1045748 ] test_subprocess vs Windows Message-ID: Bugs item #1045748, was opened at 2004-10-12 17:58 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Tim Peters (tim_one) Assigned to: Fredrik Lundh (effbot) Summary: test_subprocess vs Windows Initial Comment: Some glitches. This takes about a minute and 20 seconds to run on a 3.2 GHz WinXP Pro SP2 box w/ a gigabyte of RAM. That's waaaaay slow. Don't know whether it's a bug or a feature. If it's a feature, changes to regrtest should be made so this only runs when a new "resource" is specified. At one point, it prints "banana" to the DOS box running the tests. Extraneous output shouldn't happen. Again don't know whether this is intended, but it's a bug either way. Would be better if it didn't pop up a visible DOS box while it's running. As Barry noted, the test fails in a debug build. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 From noreply at sourceforge.net Wed Oct 13 00:37:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 00:38:03 2004 Subject: [ python-Bugs-1045748 ] test_subprocess vs Windows Message-ID: Bugs item #1045748, was opened at 2004-10-12 17:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Tim Peters (tim_one) Assigned to: Fredrik Lundh (effbot) Summary: test_subprocess vs Windows Initial Comment: Some glitches. This takes about a minute and 20 seconds to run on a 3.2 GHz WinXP Pro SP2 box w/ a gigabyte of RAM. That's waaaaay slow. Don't know whether it's a bug or a feature. If it's a feature, changes to regrtest should be made so this only runs when a new "resource" is specified. At one point, it prints "banana" to the DOS box running the tests. Extraneous output shouldn't happen. Again don't know whether this is intended, but it's a bug either way. Would be better if it didn't pop up a visible DOS box while it's running. As Barry noted, the test fails in a debug build. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-12 18:37 Message: Logged In: YES user_id=31435 More info: The test popping up a DOS box is test_creationflags(). Maybe that's what it's supposed to do. By far the biggest time hog is test_no_leaking(), which creates >= 1026 processes. That's very expensive under any flavor of Windows. Since it doesn't actually appear to test for leaks, maybe it could be cut to one iteration with no real loss of functionality . test_stdout_none() is the one that prints "banana" to the screen. Don't know whether it should. If so, changing its output to, e.g., "This line is expected extraneous output." would be better. That's it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 From noreply at sourceforge.net Wed Oct 13 03:35:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 03:35:48 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 07:04 Message generated for change (Settings changed) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Sebastien JUST (sebastienjust) >Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Wed Oct 13 03:42:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 03:42:34 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 07:04 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Wed Oct 13 03:57:02 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 03:57:06 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 07:04 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Wed Oct 13 03:57:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 03:57:59 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 07:04 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Wed Oct 13 03:58:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 03:58:21 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 07:04 Message generated for change (Settings changed) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None >Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Wed Oct 13 06:12:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 06:12:55 2004 Subject: [ python-Bugs-1045748 ] test_subprocess vs Windows Message-ID: Bugs item #1045748, was opened at 2004-10-12 17:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Tim Peters (tim_one) Assigned to: Fredrik Lundh (effbot) Summary: test_subprocess vs Windows Initial Comment: Some glitches. This takes about a minute and 20 seconds to run on a 3.2 GHz WinXP Pro SP2 box w/ a gigabyte of RAM. That's waaaaay slow. Don't know whether it's a bug or a feature. If it's a feature, changes to regrtest should be made so this only runs when a new "resource" is specified. At one point, it prints "banana" to the DOS box running the tests. Extraneous output shouldn't happen. Again don't know whether this is intended, but it's a bug either way. Would be better if it didn't pop up a visible DOS box while it's running. As Barry noted, the test fails in a debug build. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-13 00:12 Message: Logged In: YES user_id=31435 I fixed all these except for test_no_leaking(). Left an XXX comment in the test about that one. Don't understand it. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-12 18:37 Message: Logged In: YES user_id=31435 More info: The test popping up a DOS box is test_creationflags(). Maybe that's what it's supposed to do. By far the biggest time hog is test_no_leaking(), which creates >= 1026 processes. That's very expensive under any flavor of Windows. Since it doesn't actually appear to test for leaks, maybe it could be cut to one iteration with no real loss of functionality . test_stdout_none() is the one that prints "banana" to the screen. Don't know whether it should. If so, changing its output to, e.g., "This line is expected extraneous output." would be better. That's it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 From noreply at sourceforge.net Wed Oct 13 07:06:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 07:06:05 2004 Subject: [ python-Bugs-1045893 ] warning '"_POSIX_C_SOURCE" redefined' when include Python.h Message-ID: Bugs item #1045893, was opened at 2004-10-13 14:06 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045893&group_id=5470 Category: Extension Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Eugene Sizikov (eugene_beast) Assigned to: Nobody/Anonymous (nobody) Summary: warning '"_POSIX_C_SOURCE" redefined' when include Python.h Initial Comment: """ In file included from /usr/include/python2.3/Python.h:8, from script.h:21, from server.cpp:23: /usr/include/python2.3/pyconfig.h:847:1: warning: "_POSIX_C_SOURCE" redefined In file included from /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/i386-redhat-linux/bits/os_defines.h:39, from /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/i386-redhat-linux/bits/c++config.h:35, from /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/iostream:44, from server.cpp:18: /usr/include/features.h:131:1: warning: this is the location of the previous definition ""' The above message is shown whenever program (extending or embedding python with C/C++) using Python.h was compilled. OS: Fedora Core 2 compiller: GCC 3.4.0 libc: glibc-2.3 Python: 2.3.3 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045893&group_id=5470 From noreply at sourceforge.net Wed Oct 13 09:30:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 09:31:02 2004 Subject: [ python-Bugs-1045748 ] test_subprocess vs Windows Message-ID: Bugs item #1045748, was opened at 2004-10-12 23:58 Message generated for change (Comment added) made by astrand You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Tim Peters (tim_one) Assigned to: Fredrik Lundh (effbot) Summary: test_subprocess vs Windows Initial Comment: Some glitches. This takes about a minute and 20 seconds to run on a 3.2 GHz WinXP Pro SP2 box w/ a gigabyte of RAM. That's waaaaay slow. Don't know whether it's a bug or a feature. If it's a feature, changes to regrtest should be made so this only runs when a new "resource" is specified. At one point, it prints "banana" to the DOS box running the tests. Extraneous output shouldn't happen. Again don't know whether this is intended, but it's a bug either way. Would be better if it didn't pop up a visible DOS box while it's running. As Barry noted, the test fails in a debug build. ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-13 09:30 Message: Logged In: YES user_id=344921 >The test popping up a DOS box is test_creationflags(). >Maybe that's what it's supposed to do. Yes, it's supposed to do that. I couldn't come up with any better creationflag to test than CREATE_NEW_CONSOLE. (The list of valid flags is on http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/process_creation_flags.asp ) Of course, one possibility is to remove this test altogether. >By far the biggest time hog is test_no_leaking(), which >creates >= 1026 processes. That's very expensive under any >flavor of Windows. Since it doesn't actually appear to test >or leaks, maybe it could be cut to one iteration with no real >loss of functionality . Normally, you cannot have 1026 files open in a UNIX process. So, this test makes sure all used file descriptors are closed properly. This test has actually found bugs for me during the development. I'm not sure if Windows has a limitation of how many handles a process can have open, but in any case, the test is a pretty good stress test. However, since it takes so long, I suggest that we cut down the number of iterations to 65 or so on Windows. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-13 06:12 Message: Logged In: YES user_id=31435 I fixed all these except for test_no_leaking(). Left an XXX comment in the test about that one. Don't understand it. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-13 00:37 Message: Logged In: YES user_id=31435 More info: The test popping up a DOS box is test_creationflags(). Maybe that's what it's supposed to do. By far the biggest time hog is test_no_leaking(), which creates >= 1026 processes. That's very expensive under any flavor of Windows. Since it doesn't actually appear to test for leaks, maybe it could be cut to one iteration with no real loss of functionality . test_stdout_none() is the one that prints "banana" to the screen. Don't know whether it should. If so, changing its output to, e.g., "This line is expected extraneous output." would be better. That's it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 From noreply at sourceforge.net Wed Oct 13 11:41:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 11:42:02 2004 Subject: [ python-Bugs-1046077 ] urllib2: better import ftplib and gopherlib etc late Message-ID: Bugs item #1046077, was opened at 2004-10-13 11:41 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046077&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Robert Kiendl (kxroberto) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2: better import ftplib and gopherlib etc late Initial Comment: I've always trouble shrinking a py2exe'd package, because of this. its also a speed issue. urllib2 is most time used only with http(s) ...? *** urllib2_old.py Tue May 11 16:14:34 2004 --- urllib2.py Wed Oct 13 11:32:44 2004 *************** f = urllib2.urlopen('http://www.python.o *** 88,95 **** # check digest against correct (i.e. non-apache) implementation import base64 - import ftplib - import gopherlib import httplib import inspect import md5 --- 88,93 ---- *************** class FileHandler(BaseHandler): *** 1009,1014 **** --- 1007,1013 ---- class FTPHandler(BaseHandler): def ftp_open(self, req): + import ftplib host = req.get_host() if not host: raise IOError, ('ftp error', 'no host given') *************** class CacheFTPHandler(FTPHandler): *** 1110,1115 **** --- 1109,1115 ---- class GopherHandler(BaseHandler): def gopher_open(self, req): + import gopherlib host = req.get_host() if not host: raise GopherError('no host given') ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046077&group_id=5470 From noreply at sourceforge.net Wed Oct 13 12:11:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 12:11:27 2004 Subject: [ python-Bugs-1046092 ] HTMLParser fix to accept mailformed tag attributes Message-ID: Bugs item #1046092, was opened at 2004-10-13 14:11 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046092&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Vsevolod Novikov (nnseva) Assigned to: Nobody/Anonymous (nobody) Summary: HTMLParser fix to accept mailformed tag attributes Initial Comment: This is a patch to fix bugs #975556 and #921657. I think, it should be made just because the parser should accept as many pages as it can. At the other hand, the code near to fixed contains regexp to accept mailformed attributes in other cases: compare attrfind variable and locatestarttagend variable values. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046092&group_id=5470 From noreply at sourceforge.net Wed Oct 13 12:15:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 12:16:01 2004 Subject: [ python-Bugs-975556 ] HTMLParser lukewarm on bogus bare attribute chars Message-ID: Bugs item #975556, was opened at 2004-06-18 23:33 Message generated for change (Comment added) made by nnseva You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=975556&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Mike Coleman (mkc) Assigned to: Nobody/Anonymous (nobody) Summary: HTMLParser lukewarm on bogus bare attribute chars Initial Comment: I tripped over the same problem mentioned in bug #921657 (HTMLParser.py), except that my bogus attribute char is '|' instead of '@'. May I suggest that HTMLParser either require strict compliance with the HTML spec, or alternatively that it accept everything reasonable? The latter approach would be much more useful, and it would also be valuable to have this decision documented. In particular, 'attrfind' needs to be changed to accept (following the '=\s*') something like the subpattern given for 'locatestarttagend' (see the "bare value" line). ---------------------------------------------------------------------- Comment By: Vsevolod Novikov (nnseva) Date: 2004-10-13 14:15 Message: Logged In: YES user_id=325678 see request #1046092 to fix it ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=975556&group_id=5470 From noreply at sourceforge.net Wed Oct 13 12:16:26 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 12:16:32 2004 Subject: [ python-Bugs-921657 ] HTMLParser ParseError in start tag Message-ID: Bugs item #921657, was opened at 2004-03-23 13:17 Message generated for change (Comment added) made by nnseva You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=921657&group_id=5470 Category: Python Library Group: Python 2.3 Status: Closed Resolution: Fixed Priority: 5 Submitted By: Bernd Zimmermann (bernd_zedv) Assigned to: A.M. Kuchling (akuchling) Summary: HTMLParser ParseError in start tag Initial Comment: when this - obviously correct html - is parsed: xyz this exception is raised: HTMLParseError: junk characters in start tag: '@domain.com>', at line 1, column 1 I work around this by adding '@' to the allowed character's class: import HTMLParser HTMLParser.attrfind = re.compile( r'\s*([a-zA-Z_][-.:a-zA-Z_0-9]*)(\s*=\s*' r'(\'[^\']*\'|"[^"]*"|[-a-zA-Z0-9./,:;+*%?!&$\(\) _#=~@]*))?') myparser = HTMLParser.HTMLParser() myparser.feed(' Bugs item #1046092, was opened at 2004-10-13 12:11 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046092&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Vsevolod Novikov (nnseva) Assigned to: Nobody/Anonymous (nobody) Summary: HTMLParser fix to accept mailformed tag attributes Initial Comment: This is a patch to fix bugs #975556 and #921657. I think, it should be made just because the parser should accept as many pages as it can. At the other hand, the code near to fixed contains regexp to accept mailformed attributes in other cases: compare attrfind variable and locatestarttagend variable values. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-13 13:09 Message: Logged In: YES user_id=469548 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046092&group_id=5470 From noreply at sourceforge.net Wed Oct 13 16:56:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 16:57:03 2004 Subject: [ python-Bugs-969574 ] BSD restartable signals not correctly disabled Message-ID: Bugs item #969574, was opened at 2004-06-09 22:59 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=969574&group_id=5470 Category: Python Interpreter Core Group: Platform-specific >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Luke Mewburn (lukem) >Assigned to: Anthony Baxter (anthonybaxter) Summary: BSD restartable signals not correctly disabled Initial Comment: I noticed a problem with some python scripts not performing correctly when ^C (SIGINT) was pressed. I managed to isolate it to the following test case: import sys foo=sys.stdin.read() Once that's executed, you need to press ^C _twice_ for KeyboardInterrupt to be raised; the first ^C is ignored. If you manually enter that into an interactive python session this behaviour also occurs, although it appears to function correctly if you run the foo=sys.stdin.read() a second time (only one ^C is needed). This occurs on NetBSD 1.6 and 2.0, with python 2.1, 2.2 and 2.3, configured with and without pthreads. It also occurs on FreeBSD 4.8 with python 2.2.2. It does not occur on various Linux systems I asked people to test for me. (I have a NetBSD problem report about this issue: http://www.netbsd.org/cgi-bin/query-pr-single.pl?number=24797 ) I traced the process and noticed that the read() system call was being restarted when the first SIGINT was received. This hint, and the fact that Linux was unaffected indicated that python was probably not expecting restartable signal behaviour, and that behaviour is the default in BSD systems for signal(3) (but not sigaction(2) or the other POSIX signal calls). After doing some research in the python 2.3.4 source it appeared to me that the intent was to disable restartable signals, but that was not what was happening in practice. I noticed the following code issues: * not all code had been converted from using signal(3) to PyOS_getsig() and PyOS_setsig(). This is contrary to the HISTORY for python 2.0beta2. * siginterrupt(2) (an older BSD function) was being used in places in an attempt to ensure that signals were not restartable. However, in some cases this function was called _before_ signal(3)/PyOS_setsig(), which would mean that the call may be ineffective if PyOS_setsig() was implemented using signal(3) and not sigaction(2) * PyOS_setsig() was using sigaction(2) suboptimally, iand inheriting the sa_flags from the existing handler. If SA_RESTART happened to be already set for the signal, it would be copied. I provide the following patch, which does: * converts a few remaining signal(3) calls to PyOS_setsig(). There should be none left in a build on a UNIX system, although there may be on other systems. Converting any remaining calls to signal(3) is left as an exercise :) * moves siginterrupt(2) to PyOS_setsig() when the latter is implemented using signal(3) instead of sigaction(2). * when implementing PyOS_setsig() in terms of sigaction(2), use sigaction(2) in a more portable and "common" manner that explicitly clears the flags for the new handler, thus preventing SA_RESTART from being implicitly copied. With this patch applied, python passes all the same regression tests as without it, and my test case now exits on the first ^C as expected. Also, it is possible that this patch may also fix other outstanding signal issues on systems with BSD restartable signal semantics. Cheers, Luke. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-14 00:56 Message: Logged In: YES user_id=29957 Patch applied. Thanks! ---------------------------------------------------------------------- Comment By: Luke Mewburn (lukem) Date: 2004-06-18 10:06 Message: Logged In: YES user_id=135998 I've submitted patch 975056 which has a more complete version of the patch to fix this. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=969574&group_id=5470 From noreply at sourceforge.net Wed Oct 13 17:04:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 17:04:54 2004 Subject: [ python-Bugs-1045748 ] test_subprocess vs Windows Message-ID: Bugs item #1045748, was opened at 2004-10-13 07:58 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Tim Peters (tim_one) Assigned to: Fredrik Lundh (effbot) Summary: test_subprocess vs Windows Initial Comment: Some glitches. This takes about a minute and 20 seconds to run on a 3.2 GHz WinXP Pro SP2 box w/ a gigabyte of RAM. That's waaaaay slow. Don't know whether it's a bug or a feature. If it's a feature, changes to regrtest should be made so this only runs when a new "resource" is specified. At one point, it prints "banana" to the DOS box running the tests. Extraneous output shouldn't happen. Again don't know whether this is intended, but it's a bug either way. Would be better if it didn't pop up a visible DOS box while it's running. As Barry noted, the test fails in a debug build. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-14 01:04 Message: Logged In: YES user_id=29957 Er - sorry - do I misunderstand this? You're assuming that we can get 1024 FDs, and have a test that relies on this? Plenty of systems have lower limits by default, such as HP/UX. Please remove this test, if this is going to break these systems. (From memory, HP/UX defaults to 64 FDs, but there are plenty of others with < 1024 FD limits) If the test _does_ rely on being able to open 1024 FDs, it should be removed. ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-13 17:30 Message: Logged In: YES user_id=344921 >The test popping up a DOS box is test_creationflags(). >Maybe that's what it's supposed to do. Yes, it's supposed to do that. I couldn't come up with any better creationflag to test than CREATE_NEW_CONSOLE. (The list of valid flags is on http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/process_creation_flags.asp ) Of course, one possibility is to remove this test altogether. >By far the biggest time hog is test_no_leaking(), which >creates >= 1026 processes. That's very expensive under any >flavor of Windows. Since it doesn't actually appear to test >or leaks, maybe it could be cut to one iteration with no real >loss of functionality . Normally, you cannot have 1026 files open in a UNIX process. So, this test makes sure all used file descriptors are closed properly. This test has actually found bugs for me during the development. I'm not sure if Windows has a limitation of how many handles a process can have open, but in any case, the test is a pretty good stress test. However, since it takes so long, I suggest that we cut down the number of iterations to 65 or so on Windows. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-13 14:12 Message: Logged In: YES user_id=31435 I fixed all these except for test_no_leaking(). Left an XXX comment in the test about that one. Don't understand it. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-13 08:37 Message: Logged In: YES user_id=31435 More info: The test popping up a DOS box is test_creationflags(). Maybe that's what it's supposed to do. By far the biggest time hog is test_no_leaking(), which creates >= 1026 processes. That's very expensive under any flavor of Windows. Since it doesn't actually appear to test for leaks, maybe it could be cut to one iteration with no real loss of functionality . test_stdout_none() is the one that prints "banana" to the screen. Don't know whether it should. If so, changing its output to, e.g., "This line is expected extraneous output." would be better. That's it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 From noreply at sourceforge.net Wed Oct 13 17:11:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 17:11:58 2004 Subject: [ python-Bugs-1021756 ] 2.4a3: unhelpful error message from distutils Message-ID: Bugs item #1021756, was opened at 2004-09-03 21:34 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1021756&group_id=5470 Category: Distutils Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Fredrik Lundh (effbot) Assigned to: Nobody/Anonymous (nobody) Summary: 2.4a3: unhelpful error message from distutils Initial Comment: C:\somewhere> python Python 2.4a3 (#56, Sep 2 2004, 20:50:21) ... C:\somewhere> python setup.py build running build_ext Traceback (most recent call last): File "setup.py", line 19, in ? ext_modules = [ File "C:\python24\lib\distutils\core.py", line 150, in setup dist.run_commands() File "C:\python24\lib\distutils\dist.py", line 991, in run_commands self.run_command(cmd) File "C:\python24\lib\distutils\dist.py", line 1011, in run_command cmd_obj.run() File "C:\python24\lib\distutils\command\build_ext.py", line 243, in run force=self.force) File "C:\python24\lib\distutils\ccompiler.py", line 1177, in new_compiler return klass (None, dry_run, force) File "C:\python24\lib\distutils\msvccompiler.py", line 206, in __init__ self.__macros = MacroExpander(self.__version) File "C:\python24\lib\distutils\msvccompiler.py", line 112, in __init__ self.load_macros(version) File "C:\python24\lib\distutils\msvccompiler.py", line 128, in load_macros self.set_macro("FrameworkSDKDir", net, "sdkinstallrootv1.1") File "C:\python24\lib\distutils\msvccompiler.py", line 118, in set_macro self.macros["$(%s)" % macro] = d[key] KeyError: 'sdkinstallrootv1.1' I suppose it's trying to tell me something, but I'm not sure what. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-14 01:11 Message: Logged In: YES user_id=29957 It looks (to me) like it's expecting you to have some additional package (Framework SDK?) installed. The code in question was added as part of the MSVC7 support - are you building with MSVC6? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1021756&group_id=5470 From noreply at sourceforge.net Wed Oct 13 17:12:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 17:12:56 2004 Subject: [ python-Bugs-1010952 ] running test_codecmaps_* takes too much effort Message-ID: Bugs item #1010952, was opened at 2004-08-18 05:13 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010952&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Ronald Oussoren (ronaldoussoren) Assigned to: Hye-Shik Chang (perky) Summary: running test_codecmaps_* takes too much effort Initial Comment: The only way I've found to actually run the codecmap tests is by running it over and over again, downloading the missing files when a test complains in between, until the tests no longer complains. E.g $ make test - test_codecmap_jp complains about a missing file - download this file $ make test - test_codecmap_jp complains about another missing filie - ... Another problem: it is completely unclear where I should place the downloaded files. I've worked around this by placing the files in Lib/test and patching test_multibytecodec_support to look for the files in os.path.dirname(__file__): cvs diff: Diffing . Index: test_multibytecodec_support.py ========================================= ========================== RCS file: /cvsroot/python/python/dist/src/Lib/test/ test_multibytecodec_support.py,v retrieving revision 1.5 diff -r1.5 test_multibytecodec_support.py 165a166 > self.mapfilename = os.path.join(os.path.dirname(__file__), self.mapfilename) It would be nice if there were a document that described what should be done to run these tests, adding the required files to CVS would be fine too :-) ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-14 01:12 Message: Logged In: YES user_id=29957 Could we at least get a single tarball/zip file of all of the files? At the moment, it takes a serious amount of effort to fetch all the files. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-08-20 17:26 Message: Logged In: YES user_id=21627 Adding the files to the CVS is unacceptable, because they would then end up in the distribution, and their size is considered unacceptable for distribution (let alone copyright issues with these files). ---------------------------------------------------------------------- Comment By: Ronald Oussoren (ronaldoussoren) Date: 2004-08-18 05:19 Message: Logged In: YES user_id=580910 test_normalization suffers from a simular problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010952&group_id=5470 From noreply at sourceforge.net Wed Oct 13 17:26:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 17:26:08 2004 Subject: [ python-Bugs-1045748 ] test_subprocess vs Windows Message-ID: Bugs item #1045748, was opened at 2004-10-12 23:58 Message generated for change (Comment added) made by effbot You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Tim Peters (tim_one) Assigned to: Fredrik Lundh (effbot) Summary: test_subprocess vs Windows Initial Comment: Some glitches. This takes about a minute and 20 seconds to run on a 3.2 GHz WinXP Pro SP2 box w/ a gigabyte of RAM. That's waaaaay slow. Don't know whether it's a bug or a feature. If it's a feature, changes to regrtest should be made so this only runs when a new "resource" is specified. At one point, it prints "banana" to the DOS box running the tests. Extraneous output shouldn't happen. Again don't know whether this is intended, but it's a bug either way. Would be better if it didn't pop up a visible DOS box while it's running. As Barry noted, the test fails in a debug build. ---------------------------------------------------------------------- >Comment By: Fredrik Lundh (effbot) Date: 2004-10-13 17:26 Message: Logged In: YES user_id=38376 No, it relies on being able to do something *without* leaking handles; the 1026 value was chosen to make sure we notice a leak, on a reasonable number of platforms. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-13 17:04 Message: Logged In: YES user_id=29957 Er - sorry - do I misunderstand this? You're assuming that we can get 1024 FDs, and have a test that relies on this? Plenty of systems have lower limits by default, such as HP/UX. Please remove this test, if this is going to break these systems. (From memory, HP/UX defaults to 64 FDs, but there are plenty of others with < 1024 FD limits) If the test _does_ rely on being able to open 1024 FDs, it should be removed. ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-13 09:30 Message: Logged In: YES user_id=344921 >The test popping up a DOS box is test_creationflags(). >Maybe that's what it's supposed to do. Yes, it's supposed to do that. I couldn't come up with any better creationflag to test than CREATE_NEW_CONSOLE. (The list of valid flags is on http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/process_creation_flags.asp ) Of course, one possibility is to remove this test altogether. >By far the biggest time hog is test_no_leaking(), which >creates >= 1026 processes. That's very expensive under any >flavor of Windows. Since it doesn't actually appear to test >or leaks, maybe it could be cut to one iteration with no real >loss of functionality . Normally, you cannot have 1026 files open in a UNIX process. So, this test makes sure all used file descriptors are closed properly. This test has actually found bugs for me during the development. I'm not sure if Windows has a limitation of how many handles a process can have open, but in any case, the test is a pretty good stress test. However, since it takes so long, I suggest that we cut down the number of iterations to 65 or so on Windows. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-13 06:12 Message: Logged In: YES user_id=31435 I fixed all these except for test_no_leaking(). Left an XXX comment in the test about that one. Don't understand it. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-13 00:37 Message: Logged In: YES user_id=31435 More info: The test popping up a DOS box is test_creationflags(). Maybe that's what it's supposed to do. By far the biggest time hog is test_no_leaking(), which creates >= 1026 processes. That's very expensive under any flavor of Windows. Since it doesn't actually appear to test for leaks, maybe it could be cut to one iteration with no real loss of functionality . test_stdout_none() is the one that prints "banana" to the screen. Don't know whether it should. If so, changing its output to, e.g., "This line is expected extraneous output." would be better. That's it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 From noreply at sourceforge.net Wed Oct 13 17:29:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 17:29:56 2004 Subject: [ python-Bugs-1021756 ] 2.4a3: unhelpful error message from distutils Message-ID: Bugs item #1021756, was opened at 2004-09-03 13:34 Message generated for change (Comment added) made by effbot You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1021756&group_id=5470 Category: Distutils Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Fredrik Lundh (effbot) Assigned to: Nobody/Anonymous (nobody) Summary: 2.4a3: unhelpful error message from distutils Initial Comment: C:\somewhere> python Python 2.4a3 (#56, Sep 2 2004, 20:50:21) ... C:\somewhere> python setup.py build running build_ext Traceback (most recent call last): File "setup.py", line 19, in ? ext_modules = [ File "C:\python24\lib\distutils\core.py", line 150, in setup dist.run_commands() File "C:\python24\lib\distutils\dist.py", line 991, in run_commands self.run_command(cmd) File "C:\python24\lib\distutils\dist.py", line 1011, in run_command cmd_obj.run() File "C:\python24\lib\distutils\command\build_ext.py", line 243, in run force=self.force) File "C:\python24\lib\distutils\ccompiler.py", line 1177, in new_compiler return klass (None, dry_run, force) File "C:\python24\lib\distutils\msvccompiler.py", line 206, in __init__ self.__macros = MacroExpander(self.__version) File "C:\python24\lib\distutils\msvccompiler.py", line 112, in __init__ self.load_macros(version) File "C:\python24\lib\distutils\msvccompiler.py", line 128, in load_macros self.set_macro("FrameworkSDKDir", net, "sdkinstallrootv1.1") File "C:\python24\lib\distutils\msvccompiler.py", line 118, in set_macro self.macros["$(%s)" % macro] = d[key] KeyError: 'sdkinstallrootv1.1' I suppose it's trying to tell me something, but I'm not sure what. ---------------------------------------------------------------------- >Comment By: Fredrik Lundh (effbot) Date: 2004-10-13 17:29 Message: Logged In: YES user_id=38376 If I'm using the wrong compiler, distutils should say so, instead of showing me a 10-level KeyError traceback... ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-13 17:11 Message: Logged In: YES user_id=29957 It looks (to me) like it's expecting you to have some additional package (Framework SDK?) installed. The code in question was added as part of the MSVC7 support - are you building with MSVC6? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1021756&group_id=5470 From noreply at sourceforge.net Wed Oct 13 17:30:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 17:30:22 2004 Subject: [ python-Bugs-977343 ] Solaris likes sys/loadavg.h Message-ID: Bugs item #977343, was opened at 2004-06-22 20:19 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=977343&group_id=5470 Category: Build Group: Python 2.4 >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: Anthony Baxter (anthonybaxter) Assigned to: Anthony Baxter (anthonybaxter) Summary: Solaris likes sys/loadavg.h Initial Comment: Recent Solaris puts the prototype for getloadavg() in The following patch adds a configure check for sys/loadavg.h, and #includes it in Modules/posixmodule.c If someone can give it a cursory glance, I'll check it in. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-14 01:30 Message: Logged In: YES user_id=29957 Applied. ---------------------------------------------------------------------- Comment By: alan johnson (chomo) Date: 2004-07-02 07:57 Message: Logged In: YES user_id=943591 sounds alright ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-06-23 04:14 Message: Logged In: YES user_id=21627 Looks fine, please apply. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=977343&group_id=5470 From noreply at sourceforge.net Wed Oct 13 17:31:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 17:31:57 2004 Subject: [ python-Bugs-989338 ] test_unicode_file fails on win2k Message-ID: Bugs item #989338, was opened at 2004-07-12 21:11 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=989338&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Miki Tebeka (tebeka) Assigned to: Nobody/Anonymous (nobody) Summary: test_unicode_file fails on win2k Initial Comment: >>> test_support.verbose = 1 >>> test_unicode_file.test_main() test_directories (test.test_unicode_file.TestUnicodeFiles) ... ERROR test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ... ERROR test_single_files (test.test_unicode_file.TestUnicodeFiles) ... ERROR ====================================================================== ERROR: test_directories (test.test_unicode_file.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\apps\Python24\lib\test\test_unicode_file.py", line 178, in test_directories self._do_directory(TESTFN_ENCODED+ext, TESTFN_ENCODED+ext, True) File "C:\apps\Python24\lib\test\test_unicode_file.py", line 112, in _do_directory os.mkdir(make_name) OSError: [Errno 22] Invalid argument: '@test-??.dir' ====================================================================== ERROR: test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\apps\Python24\lib\test\test_unicode_file.py", line 170, in test_equivalent_files self._test_equivalent(TESTFN_ENCODED, TESTFN_UNICODE) File "C:\apps\Python24\lib\test\test_unicode_file.py", line 154, in _test_equivalent f = file(filename1, "w") IOError: [Errno 2] No such file or directory: '@test-??' ====================================================================== ERROR: test_single_files (test.test_unicode_file.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\apps\Python24\lib\test\test_unicode_file.py", line 164, in test_single_files self._test_single(TESTFN_ENCODED) File "C:\apps\Python24\lib\test\test_unicode_file.py", line 136, in _test_single f = file(filename, "w") IOError: [Errno 2] No such file or directory: '@test-??' ---------------------------------------------------------------------- Ran 3 tests in 0.061s FAILED (errors=3) Traceback (most recent call last): File "", line 1, in -toplevel- test_unicode_file.test_main() File "C:\apps\Python24\lib\test\test_unicode_file.py", line 191, in test_main run_suite(suite) File "C:\apps\Python24\lib\test\test_support.py", line 274, in run_suite raise TestFailed(msg) TestFailed: errors occurred; run in verbose mode for details >>> This is Python2.4a1 on win2k pro ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-14 01:31 Message: Logged In: YES user_id=29957 Sounds like a locale thing - do the people seeing the bug have the problem if they switch their locale to english? ---------------------------------------------------------------------- Comment By: Marek Baczynski (imbaczek) Date: 2004-09-17 09:30 Message: Logged In: YES user_id=838849 I have the very same errors as quiver on WinXP SP1, Python 2.4a3, NTFS, locale Polish. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-09-08 00:37 Message: Logged In: YES user_id=671362 I can reproduce this in Python 2.4a3 on Win 2K SP4(locale is set to Japanese). File system is NTFS. But on my box, test_single_files passes OK. Only test_equivalent_files and test_directories raise ERROR. Here is the result. test_unicode_file test_directories (test.test_unicode_file.TestUnicodeFiles) ... ERROR test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ... ERROR test_single_files (test.test_unicode_file.TestUnicodeFiles) ... ok ============================================== ======================== ERROR: test_directories (test.test_unicode_file.TestUnicodeFiles) ------------------------------------------------------------ ---------- Traceback (most recent call last): File "C:\Python24\lib\test\test_unicode_file.py", line 185, in test_directories self._do_directory(TESTFN_ENCODED+ext, TESTFN_UNICODE+ext, True) File "C:\Python24\lib\test\test_unicode_file.py", line 114, in _do_directory os.chdir(chdir_name) OSError: [Errno 2] No such file or directory: u'@test-\xe0 \xf2.dir' ============================================== ======================== ERROR: test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ------------------------------------------------------------ ---------- Traceback (most recent call last): File "C:\Python24\lib\test\test_unicode_file.py", line 176, in test_equivalent_files self._test_equivalent(TESTFN_ENCODED, TESTFN_UNICODE) File "C:\Python24\lib\test\test_unicode_file.py", line 157, in _test_equivalent self._do_equivilent(filename1, filename2) File "C:\Python24\lib\test\test_unicode_file.py", line 67, in _do_equivilent os.stat(filename2)) OSError: [Errno 2] No such file or directory: u'@test-\xe0\xf2' ------------------------------------------------------------ ---------- Ran 3 tests in 0.070s FAILED (errors=2) test test_unicode_file failed -- errors occurred; run in verbose mode for details 1 test failed: test_unicode_file ---------------------------------------------------------------------- Comment By: Miki Tebeka (tebeka) Date: 2004-09-07 16:56 Message: Logged In: YES user_id=358087 Yes it does :-( I have a winXP on IBM T-40. The other language that is installed is Hebrew. The problem that when encoding TESTFN_UNICODE with the file system ecoding (mbcs) the resulted string is "@test-??" which is not a legal file name on windows ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-07 16:31 Message: Logged In: YES user_id=80475 Does this still fail for you in Py2.4a3 ? If not, please note the result and close the bug. ---------------------------------------------------------------------- Comment By: Miki Tebeka (tebeka) Date: 2004-07-18 16:44 Message: Logged In: YES user_id=358087 File system is NTFS. winver gives: Version 5.1 (Build 2600.xpsp2.030422-1633: Service Pack 1) ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-07-16 02:55 Message: Logged In: YES user_id=21627 Are you using a FAT partition? What precise version string do you get in winver.exe? ---------------------------------------------------------------------- Comment By: Miki Tebeka (tebeka) Date: 2004-07-13 16:10 Message: Logged In: YES user_id=358087 Sorry, I forgot I've upgraded my OS lately. This *is* WinXP Pro. Any other data I can send? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-07-13 02:35 Message: Logged In: YES user_id=31435 Peculiar -- it works fine on WinXP Pro, which ought to work the same as Win2K here. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=989338&group_id=5470 From noreply at sourceforge.net Wed Oct 13 17:54:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 17:54:17 2004 Subject: [ python-Bugs-1045748 ] test_subprocess vs Windows Message-ID: Bugs item #1045748, was opened at 2004-10-12 17:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 Category: Windows Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 7 Submitted By: Tim Peters (tim_one) Assigned to: Fredrik Lundh (effbot) Summary: test_subprocess vs Windows Initial Comment: Some glitches. This takes about a minute and 20 seconds to run on a 3.2 GHz WinXP Pro SP2 box w/ a gigabyte of RAM. That's waaaaay slow. Don't know whether it's a bug or a feature. If it's a feature, changes to regrtest should be made so this only runs when a new "resource" is specified. At one point, it prints "banana" to the DOS box running the tests. Extraneous output shouldn't happen. Again don't know whether this is intended, but it's a bug either way. Would be better if it didn't pop up a visible DOS box while it's running. As Barry noted, the test fails in a debug build. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-13 11:54 Message: Logged In: YES user_id=31435 Closing this now, since all the issues mentioned reached satisfactory resolutions. ---------------------------------------------------------------------- Comment By: Fredrik Lundh (effbot) Date: 2004-10-13 11:26 Message: Logged In: YES user_id=38376 No, it relies on being able to do something *without* leaking handles; the 1026 value was chosen to make sure we notice a leak, on a reasonable number of platforms. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-13 11:04 Message: Logged In: YES user_id=29957 Er - sorry - do I misunderstand this? You're assuming that we can get 1024 FDs, and have a test that relies on this? Plenty of systems have lower limits by default, such as HP/UX. Please remove this test, if this is going to break these systems. (From memory, HP/UX defaults to 64 FDs, but there are plenty of others with < 1024 FD limits) If the test _does_ rely on being able to open 1024 FDs, it should be removed. ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-13 03:30 Message: Logged In: YES user_id=344921 >The test popping up a DOS box is test_creationflags(). >Maybe that's what it's supposed to do. Yes, it's supposed to do that. I couldn't come up with any better creationflag to test than CREATE_NEW_CONSOLE. (The list of valid flags is on http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dllproc/base/process_creation_flags.asp ) Of course, one possibility is to remove this test altogether. >By far the biggest time hog is test_no_leaking(), which >creates >= 1026 processes. That's very expensive under any >flavor of Windows. Since it doesn't actually appear to test >or leaks, maybe it could be cut to one iteration with no real >loss of functionality . Normally, you cannot have 1026 files open in a UNIX process. So, this test makes sure all used file descriptors are closed properly. This test has actually found bugs for me during the development. I'm not sure if Windows has a limitation of how many handles a process can have open, but in any case, the test is a pretty good stress test. However, since it takes so long, I suggest that we cut down the number of iterations to 65 or so on Windows. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-13 00:12 Message: Logged In: YES user_id=31435 I fixed all these except for test_no_leaking(). Left an XXX comment in the test about that one. Don't understand it. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-12 18:37 Message: Logged In: YES user_id=31435 More info: The test popping up a DOS box is test_creationflags(). Maybe that's what it's supposed to do. By far the biggest time hog is test_no_leaking(), which creates >= 1026 processes. That's very expensive under any flavor of Windows. Since it doesn't actually appear to test for leaks, maybe it could be cut to one iteration with no real loss of functionality . test_stdout_none() is the one that prints "banana" to the screen. Don't know whether it should. If so, changing its output to, e.g., "This line is expected extraneous output." would be better. That's it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045748&group_id=5470 From noreply at sourceforge.net Wed Oct 13 17:56:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 17:56:04 2004 Subject: [ python-Bugs-1021756 ] 2.4a3: unhelpful error message from distutils Message-ID: Bugs item #1021756, was opened at 2004-09-03 21:34 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1021756&group_id=5470 Category: Distutils Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Fredrik Lundh (effbot) Assigned to: Nobody/Anonymous (nobody) Summary: 2.4a3: unhelpful error message from distutils Initial Comment: C:\somewhere> python Python 2.4a3 (#56, Sep 2 2004, 20:50:21) ... C:\somewhere> python setup.py build running build_ext Traceback (most recent call last): File "setup.py", line 19, in ? ext_modules = [ File "C:\python24\lib\distutils\core.py", line 150, in setup dist.run_commands() File "C:\python24\lib\distutils\dist.py", line 991, in run_commands self.run_command(cmd) File "C:\python24\lib\distutils\dist.py", line 1011, in run_command cmd_obj.run() File "C:\python24\lib\distutils\command\build_ext.py", line 243, in run force=self.force) File "C:\python24\lib\distutils\ccompiler.py", line 1177, in new_compiler return klass (None, dry_run, force) File "C:\python24\lib\distutils\msvccompiler.py", line 206, in __init__ self.__macros = MacroExpander(self.__version) File "C:\python24\lib\distutils\msvccompiler.py", line 112, in __init__ self.load_macros(version) File "C:\python24\lib\distutils\msvccompiler.py", line 128, in load_macros self.set_macro("FrameworkSDKDir", net, "sdkinstallrootv1.1") File "C:\python24\lib\distutils\msvccompiler.py", line 118, in set_macro self.macros["$(%s)" % macro] = d[key] KeyError: 'sdkinstallrootv1.1' I suppose it's trying to tell me something, but I'm not sure what. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-14 01:56 Message: Logged In: YES user_id=29957 Yep. Can you produce a small patch for this? I have no access to MSVC 6 or 7 (in fact, my Windows box won't even boot at the moment). ---------------------------------------------------------------------- Comment By: Fredrik Lundh (effbot) Date: 2004-10-14 01:29 Message: Logged In: YES user_id=38376 If I'm using the wrong compiler, distutils should say so, instead of showing me a 10-level KeyError traceback... ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-14 01:11 Message: Logged In: YES user_id=29957 It looks (to me) like it's expecting you to have some additional package (Framework SDK?) installed. The code in question was added as part of the MSVC7 support - are you building with MSVC6? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1021756&group_id=5470 From noreply at sourceforge.net Wed Oct 13 19:14:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 19:14:14 2004 Subject: [ python-Bugs-1046404 ] improving distutils swig support Message-ID: Bugs item #1046404, was opened at 2004-10-14 03:14 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046404&group_id=5470 Category: Distutils Group: Python 2.4 Status: Open Resolution: None Priority: 8 Submitted By: Anthony Baxter (anthonybaxter) Assigned to: Anthony Baxter (anthonybaxter) Summary: improving distutils swig support Initial Comment: This patch (based on one posted to distutils-sig by Andi Vajda) adds a couple of options to build_ext to allow you to specify swig options and a swig binary. I've left the old swig-cpp option in, with a log.warn() on it. Posting here so that hopefully someone with better SWIG clue than I can provide feedback - it seems to work fine, and I plan to commit it tomorrow... ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046404&group_id=5470 From noreply at sourceforge.net Wed Oct 13 21:59:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 21:59:57 2004 Subject: [ python-Bugs-1010952 ] running test_codecmaps_* takes too much effort Message-ID: Bugs item #1010952, was opened at 2004-08-17 21:13 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010952&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Ronald Oussoren (ronaldoussoren) Assigned to: Hye-Shik Chang (perky) Summary: running test_codecmaps_* takes too much effort Initial Comment: The only way I've found to actually run the codecmap tests is by running it over and over again, downloading the missing files when a test complains in between, until the tests no longer complains. E.g $ make test - test_codecmap_jp complains about a missing file - download this file $ make test - test_codecmap_jp complains about another missing filie - ... Another problem: it is completely unclear where I should place the downloaded files. I've worked around this by placing the files in Lib/test and patching test_multibytecodec_support to look for the files in os.path.dirname(__file__): cvs diff: Diffing . Index: test_multibytecodec_support.py ========================================= ========================== RCS file: /cvsroot/python/python/dist/src/Lib/test/ test_multibytecodec_support.py,v retrieving revision 1.5 diff -r1.5 test_multibytecodec_support.py 165a166 > self.mapfilename = os.path.join(os.path.dirname(__file__), self.mapfilename) It would be nice if there were a document that described what should be done to run these tests, adding the required files to CVS would be fine too :-) ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-13 21:59 Message: Logged In: YES user_id=469548 How about adding the files to nondist? ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-13 17:12 Message: Logged In: YES user_id=29957 Could we at least get a single tarball/zip file of all of the files? At the moment, it takes a serious amount of effort to fetch all the files. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-08-20 09:26 Message: Logged In: YES user_id=21627 Adding the files to the CVS is unacceptable, because they would then end up in the distribution, and their size is considered unacceptable for distribution (let alone copyright issues with these files). ---------------------------------------------------------------------- Comment By: Ronald Oussoren (ronaldoussoren) Date: 2004-08-17 21:19 Message: Logged In: YES user_id=580910 test_normalization suffers from a simular problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010952&group_id=5470 From noreply at sourceforge.net Wed Oct 13 22:04:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 13 22:04:57 2004 Subject: [ python-Bugs-1010952 ] running test_codecmaps_* takes too much effort Message-ID: Bugs item #1010952, was opened at 2004-08-17 21:13 Message generated for change (Comment added) made by lemburg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010952&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Ronald Oussoren (ronaldoussoren) Assigned to: Hye-Shik Chang (perky) Summary: running test_codecmaps_* takes too much effort Initial Comment: The only way I've found to actually run the codecmap tests is by running it over and over again, downloading the missing files when a test complains in between, until the tests no longer complains. E.g $ make test - test_codecmap_jp complains about a missing file - download this file $ make test - test_codecmap_jp complains about another missing filie - ... Another problem: it is completely unclear where I should place the downloaded files. I've worked around this by placing the files in Lib/test and patching test_multibytecodec_support to look for the files in os.path.dirname(__file__): cvs diff: Diffing . Index: test_multibytecodec_support.py ========================================= ========================== RCS file: /cvsroot/python/python/dist/src/Lib/test/ test_multibytecodec_support.py,v retrieving revision 1.5 diff -r1.5 test_multibytecodec_support.py 165a166 > self.mapfilename = os.path.join(os.path.dirname(__file__), self.mapfilename) It would be nice if there were a document that described what should be done to run these tests, adding the required files to CVS would be fine too :-) ---------------------------------------------------------------------- >Comment By: M.-A. Lemburg (lemburg) Date: 2004-10-13 22:04 Message: Logged In: YES user_id=38388 Just a suggestion: Why don't we add a new resource option to the test scripts and then have the tests download the files from the Internet as necessary ?! ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-13 21:59 Message: Logged In: YES user_id=469548 How about adding the files to nondist? ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-13 17:12 Message: Logged In: YES user_id=29957 Could we at least get a single tarball/zip file of all of the files? At the moment, it takes a serious amount of effort to fetch all the files. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-08-20 09:26 Message: Logged In: YES user_id=21627 Adding the files to the CVS is unacceptable, because they would then end up in the distribution, and their size is considered unacceptable for distribution (let alone copyright issues with these files). ---------------------------------------------------------------------- Comment By: Ronald Oussoren (ronaldoussoren) Date: 2004-08-17 21:19 Message: Logged In: YES user_id=580910 test_normalization suffers from a simular problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010952&group_id=5470 From noreply at sourceforge.net Thu Oct 14 00:16:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 14 00:16:34 2004 Subject: [ python-Bugs-1046644 ] improving distutils swig support (2) Message-ID: Bugs item #1046644, was opened at 2004-10-14 00:16 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046644&group_id=5470 Category: Distutils Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Lars Immisch (larsimmisch) Assigned to: Nobody/Anonymous (nobody) Summary: improving distutils swig support (2) Initial Comment: This patch further improves SWIG support in distutils. With this patch, both these idioms will work: setup (name = "aculab", ext_modules = [Extension("._aculab, ['acu.i'], swig_opts=['-modern' '-I../include']]) and: python setup.py build_ext --swig-opts="-modern -I../include" This patch extends and includes bug 1046404 from Anthony Baxter: http://sourceforge.net/tracker/index.php?func=detail&aid=1046404&group_id=5470&atid=105470 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046644&group_id=5470 From noreply at sourceforge.net Thu Oct 14 01:45:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 14 01:45:07 2004 Subject: [ python-Bugs-1046690 ] difflib.HtmlDiff doc errors (EASY) Message-ID: Bugs item #1046690, was opened at 2004-10-13 18:45 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046690&group_id=5470 Category: Documentation Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Dan Gass (dmgass) Assigned to: Nobody/Anonymous (nobody) Summary: difflib.HtmlDiff doc errors (EASY) Initial Comment: The following documentation errors are in /python/dist/src/Doc/lib/libdifflib.tex: 1) make_table method of HtmlDiff class should have the same prototype as make_file (should include numlines as an optional argument). 2) In make_table method discussion the following sentence should be changed: FROM: The arguments of this method are a subset of those for the make_file method. TO: The arguments of this method are identical to those for the make_file method. --------------------------------------------------------- I was the original author of this patch. I apologize for this inconvenience. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046690&group_id=5470 From noreply at sourceforge.net Thu Oct 14 05:10:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 14 05:10:27 2004 Subject: [ python-Bugs-1046690 ] difflib.HtmlDiff doc errors (EASY) Message-ID: Bugs item #1046690, was opened at 2004-10-13 19:45 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046690&group_id=5470 Category: Documentation Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Dan Gass (dmgass) >Assigned to: Tim Peters (tim_one) Summary: difflib.HtmlDiff doc errors (EASY) Initial Comment: The following documentation errors are in /python/dist/src/Doc/lib/libdifflib.tex: 1) make_table method of HtmlDiff class should have the same prototype as make_file (should include numlines as an optional argument). 2) In make_table method discussion the following sentence should be changed: FROM: The arguments of this method are a subset of those for the make_file method. TO: The arguments of this method are identical to those for the make_file method. --------------------------------------------------------- I was the original author of this patch. I apologize for this inconvenience. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-13 23:10 Message: Logged In: YES user_id=31435 Thanks, Dan. These repairs have been made. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046690&group_id=5470 From noreply at sourceforge.net Thu Oct 14 06:55:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 14 06:55:07 2004 Subject: [ python-Bugs-1046800 ] Error in the given example Message-ID: Bugs item #1046800, was opened at 2004-10-14 04:55 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046800&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Vishnu (vishnube) Assigned to: Nobody/Anonymous (nobody) Summary: Error in the given example Initial Comment: Python Documentation - Release 2.3.3 - December 19, 2003 In section, 16.1.6.5 The Window Manager, there is one problem exists in the given example. Please change the first line of the example FROM: import Tkinter TO: from Tkinter import * Thank you, Vishnu ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046800&group_id=5470 From noreply at sourceforge.net Thu Oct 14 09:43:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 14 09:43:09 2004 Subject: [ python-Bugs-1046644 ] improving distutils swig support (2) Message-ID: Bugs item #1046644, was opened at 2004-10-14 08:16 Message generated for change (Settings changed) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046644&group_id=5470 Category: Distutils Group: Python 2.4 Status: Open Resolution: None >Priority: 8 Submitted By: Lars Immisch (larsimmisch) >Assigned to: Anthony Baxter (anthonybaxter) Summary: improving distutils swig support (2) Initial Comment: This patch further improves SWIG support in distutils. With this patch, both these idioms will work: setup (name = "aculab", ext_modules = [Extension("._aculab, ['acu.i'], swig_opts=['-modern' '-I../include']]) and: python setup.py build_ext --swig-opts="-modern -I../include" This patch extends and includes bug 1046404 from Anthony Baxter: http://sourceforge.net/tracker/index.php?func=detail&aid=1046404&group_id=5470&atid=105470 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046644&group_id=5470 From noreply at sourceforge.net Thu Oct 14 10:21:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 14 10:21:51 2004 Subject: [ python-Bugs-1046855 ] httplib index out of range Message-ID: Bugs item #1046855, was opened at 2004-10-14 10:21 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046855&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Grzegorz Makarewicz (makaron) Assigned to: Nobody/Anonymous (nobody) Summary: httplib index out of range Initial Comment: import httplib conn = httplib.HTTP() httplib.py:523, in _set_hostport if host[0] == '[' and host[-1] == ']': just because host is an empty string ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046855&group_id=5470 From noreply at sourceforge.net Thu Oct 14 10:59:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 14 10:59:39 2004 Subject: [ python-Bugs-989338 ] test_unicode_file fails on win2k Message-ID: Bugs item #989338, was opened at 2004-07-12 14:11 Message generated for change (Comment added) made by tebeka You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=989338&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Miki Tebeka (tebeka) Assigned to: Nobody/Anonymous (nobody) Summary: test_unicode_file fails on win2k Initial Comment: >>> test_support.verbose = 1 >>> test_unicode_file.test_main() test_directories (test.test_unicode_file.TestUnicodeFiles) ... ERROR test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ... ERROR test_single_files (test.test_unicode_file.TestUnicodeFiles) ... ERROR ====================================================================== ERROR: test_directories (test.test_unicode_file.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\apps\Python24\lib\test\test_unicode_file.py", line 178, in test_directories self._do_directory(TESTFN_ENCODED+ext, TESTFN_ENCODED+ext, True) File "C:\apps\Python24\lib\test\test_unicode_file.py", line 112, in _do_directory os.mkdir(make_name) OSError: [Errno 22] Invalid argument: '@test-??.dir' ====================================================================== ERROR: test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\apps\Python24\lib\test\test_unicode_file.py", line 170, in test_equivalent_files self._test_equivalent(TESTFN_ENCODED, TESTFN_UNICODE) File "C:\apps\Python24\lib\test\test_unicode_file.py", line 154, in _test_equivalent f = file(filename1, "w") IOError: [Errno 2] No such file or directory: '@test-??' ====================================================================== ERROR: test_single_files (test.test_unicode_file.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\apps\Python24\lib\test\test_unicode_file.py", line 164, in test_single_files self._test_single(TESTFN_ENCODED) File "C:\apps\Python24\lib\test\test_unicode_file.py", line 136, in _test_single f = file(filename, "w") IOError: [Errno 2] No such file or directory: '@test-??' ---------------------------------------------------------------------- Ran 3 tests in 0.061s FAILED (errors=3) Traceback (most recent call last): File "", line 1, in -toplevel- test_unicode_file.test_main() File "C:\apps\Python24\lib\test\test_unicode_file.py", line 191, in test_main run_suite(suite) File "C:\apps\Python24\lib\test\test_support.py", line 274, in run_suite raise TestFailed(msg) TestFailed: errors occurred; run in verbose mode for details >>> This is Python2.4a1 on win2k pro ---------------------------------------------------------------------- >Comment By: Miki Tebeka (tebeka) Date: 2004-10-14 10:59 Message: Logged In: YES user_id=358087 How do I switch my locale to English on winXP pro? ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-13 17:31 Message: Logged In: YES user_id=29957 Sounds like a locale thing - do the people seeing the bug have the problem if they switch their locale to english? ---------------------------------------------------------------------- Comment By: Marek Baczynski (imbaczek) Date: 2004-09-17 02:30 Message: Logged In: YES user_id=838849 I have the very same errors as quiver on WinXP SP1, Python 2.4a3, NTFS, locale Polish. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-09-07 17:37 Message: Logged In: YES user_id=671362 I can reproduce this in Python 2.4a3 on Win 2K SP4(locale is set to Japanese). File system is NTFS. But on my box, test_single_files passes OK. Only test_equivalent_files and test_directories raise ERROR. Here is the result. test_unicode_file test_directories (test.test_unicode_file.TestUnicodeFiles) ... ERROR test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ... ERROR test_single_files (test.test_unicode_file.TestUnicodeFiles) ... ok ============================================== ======================== ERROR: test_directories (test.test_unicode_file.TestUnicodeFiles) ------------------------------------------------------------ ---------- Traceback (most recent call last): File "C:\Python24\lib\test\test_unicode_file.py", line 185, in test_directories self._do_directory(TESTFN_ENCODED+ext, TESTFN_UNICODE+ext, True) File "C:\Python24\lib\test\test_unicode_file.py", line 114, in _do_directory os.chdir(chdir_name) OSError: [Errno 2] No such file or directory: u'@test-\xe0 \xf2.dir' ============================================== ======================== ERROR: test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ------------------------------------------------------------ ---------- Traceback (most recent call last): File "C:\Python24\lib\test\test_unicode_file.py", line 176, in test_equivalent_files self._test_equivalent(TESTFN_ENCODED, TESTFN_UNICODE) File "C:\Python24\lib\test\test_unicode_file.py", line 157, in _test_equivalent self._do_equivilent(filename1, filename2) File "C:\Python24\lib\test\test_unicode_file.py", line 67, in _do_equivilent os.stat(filename2)) OSError: [Errno 2] No such file or directory: u'@test-\xe0\xf2' ------------------------------------------------------------ ---------- Ran 3 tests in 0.070s FAILED (errors=2) test test_unicode_file failed -- errors occurred; run in verbose mode for details 1 test failed: test_unicode_file ---------------------------------------------------------------------- Comment By: Miki Tebeka (tebeka) Date: 2004-09-07 09:56 Message: Logged In: YES user_id=358087 Yes it does :-( I have a winXP on IBM T-40. The other language that is installed is Hebrew. The problem that when encoding TESTFN_UNICODE with the file system ecoding (mbcs) the resulted string is "@test-??" which is not a legal file name on windows ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-07 09:31 Message: Logged In: YES user_id=80475 Does this still fail for you in Py2.4a3 ? If not, please note the result and close the bug. ---------------------------------------------------------------------- Comment By: Miki Tebeka (tebeka) Date: 2004-07-18 09:44 Message: Logged In: YES user_id=358087 File system is NTFS. winver gives: Version 5.1 (Build 2600.xpsp2.030422-1633: Service Pack 1) ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-07-15 19:55 Message: Logged In: YES user_id=21627 Are you using a FAT partition? What precise version string do you get in winver.exe? ---------------------------------------------------------------------- Comment By: Miki Tebeka (tebeka) Date: 2004-07-13 09:10 Message: Logged In: YES user_id=358087 Sorry, I forgot I've upgraded my OS lately. This *is* WinXP Pro. Any other data I can send? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-07-12 19:35 Message: Logged In: YES user_id=31435 Peculiar -- it works fine on WinXP Pro, which ought to work the same as Win2K here. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=989338&group_id=5470 From noreply at sourceforge.net Thu Oct 14 12:05:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 14 12:05:12 2004 Subject: [ python-Bugs-1046404 ] improving distutils swig support Message-ID: Bugs item #1046404, was opened at 2004-10-14 03:14 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046404&group_id=5470 Category: Distutils Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 8 Submitted By: Anthony Baxter (anthonybaxter) Assigned to: Anthony Baxter (anthonybaxter) Summary: improving distutils swig support Initial Comment: This patch (based on one posted to distutils-sig by Andi Vajda) adds a couple of options to build_ext to allow you to specify swig options and a swig binary. I've left the old swig-cpp option in, with a log.warn() on it. Posting here so that hopefully someone with better SWIG clue than I can provide feedback - it seems to work fine, and I plan to commit it tomorrow... ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-14 20:05 Message: Logged In: YES user_id=29957 Improved version on patch 1046644 was applied. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046404&group_id=5470 From noreply at sourceforge.net Thu Oct 14 12:05:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 14 12:05:27 2004 Subject: [ python-Bugs-1046644 ] improving distutils swig support (2) Message-ID: Bugs item #1046644, was opened at 2004-10-14 08:16 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046644&group_id=5470 Category: Distutils Group: Python 2.4 >Status: Closed >Resolution: Accepted Priority: 8 Submitted By: Lars Immisch (larsimmisch) Assigned to: Anthony Baxter (anthonybaxter) Summary: improving distutils swig support (2) Initial Comment: This patch further improves SWIG support in distutils. With this patch, both these idioms will work: setup (name = "aculab", ext_modules = [Extension("._aculab, ['acu.i'], swig_opts=['-modern' '-I../include']]) and: python setup.py build_ext --swig-opts="-modern -I../include" This patch extends and includes bug 1046404 from Anthony Baxter: http://sourceforge.net/tracker/index.php?func=detail&aid=1046404&group_id=5470&atid=105470 ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-14 20:05 Message: Logged In: YES user_id=29957 Applied. Thanks for the patch! ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046644&group_id=5470 From noreply at sourceforge.net Thu Oct 14 12:38:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 14 12:38:50 2004 Subject: [ python-Bugs-1046945 ] improving distutils swig support - documentation Message-ID: Bugs item #1046945, was opened at 2004-10-14 12:38 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046945&group_id=5470 Category: Distutils Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Lars Immisch (larsimmisch) Assigned to: Nobody/Anonymous (nobody) Summary: improving distutils swig support - documentation Initial Comment: This patch updates the documentation of distutils to describe the new swig_opts Extension keyword argument and the new --swig-opts commandline option. The --swig and --swig-cpp commandline options are not mentioned. This patch belongs to patch 1046644 or: http://sourceforge.net/tracker/index.php?func=detail&aid=1046644&group_id=5470&atid=105470 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046945&group_id=5470 From noreply at sourceforge.net Thu Oct 14 15:18:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 14 15:18:44 2004 Subject: [ python-Bugs-1046855 ] httplib index out of range Message-ID: Bugs item #1046855, was opened at 2004-10-14 09:21 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046855&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None >Priority: 7 Submitted By: Grzegorz Makarewicz (makaron) >Assigned to: Brett Cannon (bcannon) Summary: httplib index out of range Initial Comment: import httplib conn = httplib.HTTP() httplib.py:523, in _set_hostport if host[0] == '[' and host[-1] == ']': just because host is an empty string ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-14 14:18 Message: Logged In: YES user_id=6656 Ugh. Brett, cvs annotate fingers you :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046855&group_id=5470 From noreply at sourceforge.net Thu Oct 14 16:01:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 14 16:01:59 2004 Subject: [ python-Bugs-1046945 ] improving distutils swig support - documentation Message-ID: Bugs item #1046945, was opened at 2004-10-14 06:38 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046945&group_id=5470 Category: Distutils Group: Python 2.4 Status: Open >Resolution: Accepted Priority: 5 Submitted By: Lars Immisch (larsimmisch) Assigned to: Nobody/Anonymous (nobody) Summary: improving distutils swig support - documentation Initial Comment: This patch updates the documentation of distutils to describe the new swig_opts Extension keyword argument and the new --swig-opts commandline option. The --swig and --swig-cpp commandline options are not mentioned. This patch belongs to patch 1046644 or: http://sourceforge.net/tracker/index.php?func=detail&aid=1046644&group_id=5470&atid=105470 ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-10-14 10:01 Message: Logged In: YES user_id=3066 This looks fine to go in after the 2.4b1 release; it's too late to change the docs for that one (the tarball has already been built). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046945&group_id=5470 From noreply at sourceforge.net Thu Oct 14 17:24:13 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 14 17:24:22 2004 Subject: [ python-Bugs-1046855 ] httplib index out of range Message-ID: Bugs item #1046855, was opened at 2004-10-14 03:21 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046855&group_id=5470 Category: Python Library Group: None >Status: Closed >Resolution: Fixed Priority: 7 Submitted By: Grzegorz Makarewicz (makaron) Assigned to: Brett Cannon (bcannon) Summary: httplib index out of range Initial Comment: import httplib conn = httplib.HTTP() httplib.py:523, in _set_hostport if host[0] == '[' and host[-1] == ']': just because host is an empty string ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-14 10:24 Message: Logged In: YES user_id=80475 Fixed. See Lib/httplib.py 1.93. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-14 08:18 Message: Logged In: YES user_id=6656 Ugh. Brett, cvs annotate fingers you :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046855&group_id=5470 From noreply at sourceforge.net Fri Oct 15 00:23:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 00:23:53 2004 Subject: [ python-Bugs-1047397 ] cgitb failures Message-ID: Bugs item #1047397, was opened at 2004-10-14 22:23 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047397&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Robin Becker (rgbecker) Assigned to: Nobody/Anonymous (nobody) Summary: cgitb failures Initial Comment: cgitb fails when evaluating various reprs/getattrs. The bug demonstrator is cgitbbug.py. Below is the run output for two bugs. Fixes are possible by putting try: except around various evaluations. Although the test is artificial we found these could happen in real code. This is with 2.4a3 Windows XP Sample output C:\Python24>.\python \tmp\cgitbbug.py repr1 2.4a3 (#56, Sep 2 2004, 20:50:21) [MSC v.1310 32 bit (Intel)] Traceback (most recent call last): File "\tmp\cgitbbug.py", line 36, in ? print cgitb.html(sys.exc_info(),context=3) File "C:\python24\lib\cgitb.py", line 155, in html dump.append('%s = %s' % (name, pydoc.html.repr(value))) File "C:\python24\lib\pydoc.py", line 352, in repr return Repr.repr(self, object) File "C:\python24\lib\repr.py", line 24, in repr return self.repr1(x, self.maxlevel) File "C:\python24\lib\pydoc.py", line 359, in repr1 return self.escape(cram(stripid(repr(x)), self.maxother)) File "\tmp\cgitbbug.py", line 12, in __repr__ if self._mad=='repr1': raise ValueError('repr1 madness') ValueError: repr1 madness C:\Python24>.\python \tmp\cgitbbug.py getattr1 2.4a3 (#56, Sep 2 2004, 20:50:21) [MSC v.1310 32 bit (Intel)] Traceback (most recent call last): File "\tmp\cgitbbug.py", line 36, in ? print cgitb.html(sys.exc_info(),context=3) File "C:\python24\lib\cgitb.py", line 129, in html vars = scanvars(reader, frame, locals) File "C:\python24\lib\cgitb.py", line 83, in scanvars value = getattr(parent, token, __UNDEF__) File "\tmp\cgitbbug.py", line 16, in __getattr__ if self._mad=='getattr1': raise ValueError('getattr1 madness') ValueError: getattr1 madness C:\Python24> ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047397&group_id=5470 From noreply at sourceforge.net Fri Oct 15 01:33:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 01:33:37 2004 Subject: [ python-Bugs-1047397 ] cgitb failures Message-ID: Bugs item #1047397, was opened at 2004-10-14 22:23 Message generated for change (Comment added) made by rgbecker You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047397&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Robin Becker (rgbecker) Assigned to: Nobody/Anonymous (nobody) Summary: cgitb failures Initial Comment: cgitb fails when evaluating various reprs/getattrs. The bug demonstrator is cgitbbug.py. Below is the run output for two bugs. Fixes are possible by putting try: except around various evaluations. Although the test is artificial we found these could happen in real code. This is with 2.4a3 Windows XP Sample output C:\Python24>.\python \tmp\cgitbbug.py repr1 2.4a3 (#56, Sep 2 2004, 20:50:21) [MSC v.1310 32 bit (Intel)] Traceback (most recent call last): File "\tmp\cgitbbug.py", line 36, in ? print cgitb.html(sys.exc_info(),context=3) File "C:\python24\lib\cgitb.py", line 155, in html dump.append('%s = %s' % (name, pydoc.html.repr(value))) File "C:\python24\lib\pydoc.py", line 352, in repr return Repr.repr(self, object) File "C:\python24\lib\repr.py", line 24, in repr return self.repr1(x, self.maxlevel) File "C:\python24\lib\pydoc.py", line 359, in repr1 return self.escape(cram(stripid(repr(x)), self.maxother)) File "\tmp\cgitbbug.py", line 12, in __repr__ if self._mad=='repr1': raise ValueError('repr1 madness') ValueError: repr1 madness C:\Python24>.\python \tmp\cgitbbug.py getattr1 2.4a3 (#56, Sep 2 2004, 20:50:21) [MSC v.1310 32 bit (Intel)] Traceback (most recent call last): File "\tmp\cgitbbug.py", line 36, in ? print cgitb.html(sys.exc_info(),context=3) File "C:\python24\lib\cgitb.py", line 129, in html vars = scanvars(reader, frame, locals) File "C:\python24\lib\cgitb.py", line 83, in scanvars value = getattr(parent, token, __UNDEF__) File "\tmp\cgitbbug.py", line 16, in __getattr__ if self._mad=='getattr1': raise ValueError('getattr1 madness') ValueError: getattr1 madness C:\Python24> ---------------------------------------------------------------------- >Comment By: Robin Becker (rgbecker) Date: 2004-10-14 23:33 Message: Logged In: YES user_id=6946 Bug fix added ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047397&group_id=5470 From noreply at sourceforge.net Fri Oct 15 05:51:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 05:51:24 2004 Subject: [ python-Bugs-1047540 ] Turtle.py hangs Idle Message-ID: Bugs item #1047540, was opened at 2004-10-15 13:51 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047540&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Taro Ogawa (tso) Assigned to: Nobody/Anonymous (nobody) Summary: Turtle.py hangs Idle Initial Comment: 1 line fix: After line 336's: root.destroy() insert root.quit() [Since IDLE is Tkinter based, the implict quit can't happen immediately. Worse, the shell has difficulty resetting and using a ^C can leave a zombie process] -T. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047540&group_id=5470 From noreply at sourceforge.net Fri Oct 15 06:12:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 06:12:35 2004 Subject: [ python-Bugs-1047549 ] Turtle.py hangs Idle Message-ID: Bugs item #1047549, was opened at 2004-10-15 14:12 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047549&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Taro Ogawa (tso) Assigned to: Nobody/Anonymous (nobody) Summary: Turtle.py hangs Idle Initial Comment: 1 line fix: After line 336's: root.destroy() insert root.quit() [Since IDLE is Tkinter based, the implict quit can't happen immediately. Worse, the shell has difficulty resetting and using a ^C can leave a zombie process] -T. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047549&group_id=5470 From noreply at sourceforge.net Fri Oct 15 06:23:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 06:23:16 2004 Subject: [ python-Bugs-1047549 ] Turtle.py hangs Idle Message-ID: Bugs item #1047549, was opened at 2004-10-15 14:12 Message generated for change (Settings changed) made by tso You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047549&group_id=5470 Category: None Group: None >Status: Deleted Resolution: None Priority: 5 Submitted By: Taro Ogawa (tso) Assigned to: Nobody/Anonymous (nobody) Summary: Turtle.py hangs Idle Initial Comment: 1 line fix: After line 336's: root.destroy() insert root.quit() [Since IDLE is Tkinter based, the implict quit can't happen immediately. Worse, the shell has difficulty resetting and using a ^C can leave a zombie process] -T. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047549&group_id=5470 From noreply at sourceforge.net Fri Oct 15 07:03:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 07:03:05 2004 Subject: [ python-Bugs-1022813 ] test_xrange fails on osf1 v5.1b Message-ID: Bugs item #1022813, was opened at 2004-09-05 20:10 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1022813&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 6 Submitted By: roadkill (dharma_roadkill) >Assigned to: Neal Norwitz (nnorwitz) Summary: test_xrange fails on osf1 v5.1b Initial Comment: On a build of Python 2.4a3: uname -a: OSF1 xxx.tnzi.com V5.1 2650 alpha alpha > ./python ./Lib/test/test_xrange.py test_xrange (__main__.XrangeTest) ... ERROR ========================================== ============================ ERROR: test_xrange (__main__.XrangeTest) ------------------------------------------------------ ---------------- Traceback (most recent call last): File "./Lib/test/test_xrange.py", line 56, in test_xrange self.assertEqual(len(xrange(-sys.maxint, sys.maxint, 2)), ValueError: xrange object size cannot be reported ------------------------------------------------------ ---------------- Ran 1 test in 0.001s FAILED (errors=1) Traceback (most recent call last): File "./Lib/test/test_xrange.py", line 64, in ? test_main() File "./Lib/test/test_xrange.py", line 61, in test_main test.test_support.run_unittest(XrangeTest) File "/u03/home/doug/python/Python- 2.4a3/Lib/test/test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "/u03/home/doug/python/Python- 2.4a3/Lib/test/test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "./Lib/test/test_xrange.py", line 56, in test_xrange self.assertEqual(len(xrange(-sys.maxint, sys.maxint, 2)), ValueError: xrange object size cannot be reported Everything else seems to work. ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-15 01:03 Message: Logged In: YES user_id=33168 roadkill, can you verify the test passes with the latest CVS or 2.4b1 that should come out today/tomorrow? I believe this was fixed last night. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2004-09-06 22:52 Message: Logged In: YES user_id=33168 I haven't thought too much about this, but ISTM that the code is correct and the test needs to be updated. Unfortunately, I don't think there's anyway to test for LONG_MAX != INT_MAX. I think the best we could do is disable this test if sys.maxint > 0x7fffffff. Hmmm, you could probably use the struct module and find out the size of ints and longs. (struct.calcsize('i') and struct.calcsize('l')). But I'm tired, so all this could be non-sense. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-05 20:36 Message: Logged In: YES user_id=80475 I haven't looked into a solution yet but want to record what I've found so far. In checkin 2.44 on 9/11/2002, guido noted, """ Untested code for 64-bit platforms. range_length() is declared as int but returns r->len which is a long. This doesn't even cause a warning on 32-bit platforms, but can return bogus values on 64-bit platforms (and should cause a compiler warning). Fix this by inserting a range check when LONG_MAX != INT_MAX, and adding an explicit cast to (int) when the test passes. When r->len is out of range, PySequence_Size() and hence len() will report an error (but an iterator will still work). """ The code for the check is: static int range_length(rangeobject *r) { #if LONG_MAX != INT_MAX if (r->len > INT_MAX) { PyErr_SetString(PyExc_ValueError, "xrange object size cannot be reported"); return -1; } #endif return (int)(r->len); } ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1022813&group_id=5470 From noreply at sourceforge.net Fri Oct 15 08:14:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 08:14:58 2004 Subject: [ python-Bugs-989338 ] test_unicode_file fails on win2k Message-ID: Bugs item #989338, was opened at 2004-07-12 13:11 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=989338&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Miki Tebeka (tebeka) Assigned to: Nobody/Anonymous (nobody) Summary: test_unicode_file fails on win2k Initial Comment: >>> test_support.verbose = 1 >>> test_unicode_file.test_main() test_directories (test.test_unicode_file.TestUnicodeFiles) ... ERROR test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ... ERROR test_single_files (test.test_unicode_file.TestUnicodeFiles) ... ERROR ====================================================================== ERROR: test_directories (test.test_unicode_file.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\apps\Python24\lib\test\test_unicode_file.py", line 178, in test_directories self._do_directory(TESTFN_ENCODED+ext, TESTFN_ENCODED+ext, True) File "C:\apps\Python24\lib\test\test_unicode_file.py", line 112, in _do_directory os.mkdir(make_name) OSError: [Errno 22] Invalid argument: '@test-??.dir' ====================================================================== ERROR: test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\apps\Python24\lib\test\test_unicode_file.py", line 170, in test_equivalent_files self._test_equivalent(TESTFN_ENCODED, TESTFN_UNICODE) File "C:\apps\Python24\lib\test\test_unicode_file.py", line 154, in _test_equivalent f = file(filename1, "w") IOError: [Errno 2] No such file or directory: '@test-??' ====================================================================== ERROR: test_single_files (test.test_unicode_file.TestUnicodeFiles) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\apps\Python24\lib\test\test_unicode_file.py", line 164, in test_single_files self._test_single(TESTFN_ENCODED) File "C:\apps\Python24\lib\test\test_unicode_file.py", line 136, in _test_single f = file(filename, "w") IOError: [Errno 2] No such file or directory: '@test-??' ---------------------------------------------------------------------- Ran 3 tests in 0.061s FAILED (errors=3) Traceback (most recent call last): File "", line 1, in -toplevel- test_unicode_file.test_main() File "C:\apps\Python24\lib\test\test_unicode_file.py", line 191, in test_main run_suite(suite) File "C:\apps\Python24\lib\test\test_support.py", line 274, in run_suite raise TestFailed(msg) TestFailed: errors occurred; run in verbose mode for details >>> This is Python2.4a1 on win2k pro ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-15 08:14 Message: Logged In: YES user_id=21627 In control panel/regional settings, change the first *and* the third tab to English. This requires a reboot. The failure itself is harmless. test_support assumes that TESTFN_UNICODE = unicode("@test-\xe0\xf2", "latin-1") is a filename that can be represented in the default file encoding. On Windows, if the default file encoding is not CP-1252 (as it is in the Western European installations), this file name cannot be represented correctly in "mbcs". To fix this, we would need to come up with procedure to generate a file name depending on the locale. Unfortunately, such a procedure is very difficult to implement. ---------------------------------------------------------------------- Comment By: Miki Tebeka (tebeka) Date: 2004-10-14 10:59 Message: Logged In: YES user_id=358087 How do I switch my locale to English on winXP pro? ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-13 17:31 Message: Logged In: YES user_id=29957 Sounds like a locale thing - do the people seeing the bug have the problem if they switch their locale to english? ---------------------------------------------------------------------- Comment By: Marek Baczynski (imbaczek) Date: 2004-09-17 01:30 Message: Logged In: YES user_id=838849 I have the very same errors as quiver on WinXP SP1, Python 2.4a3, NTFS, locale Polish. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-09-07 16:37 Message: Logged In: YES user_id=671362 I can reproduce this in Python 2.4a3 on Win 2K SP4(locale is set to Japanese). File system is NTFS. But on my box, test_single_files passes OK. Only test_equivalent_files and test_directories raise ERROR. Here is the result. test_unicode_file test_directories (test.test_unicode_file.TestUnicodeFiles) ... ERROR test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ... ERROR test_single_files (test.test_unicode_file.TestUnicodeFiles) ... ok ============================================== ======================== ERROR: test_directories (test.test_unicode_file.TestUnicodeFiles) ------------------------------------------------------------ ---------- Traceback (most recent call last): File "C:\Python24\lib\test\test_unicode_file.py", line 185, in test_directories self._do_directory(TESTFN_ENCODED+ext, TESTFN_UNICODE+ext, True) File "C:\Python24\lib\test\test_unicode_file.py", line 114, in _do_directory os.chdir(chdir_name) OSError: [Errno 2] No such file or directory: u'@test-\xe0 \xf2.dir' ============================================== ======================== ERROR: test_equivalent_files (test.test_unicode_file.TestUnicodeFiles) ------------------------------------------------------------ ---------- Traceback (most recent call last): File "C:\Python24\lib\test\test_unicode_file.py", line 176, in test_equivalent_files self._test_equivalent(TESTFN_ENCODED, TESTFN_UNICODE) File "C:\Python24\lib\test\test_unicode_file.py", line 157, in _test_equivalent self._do_equivilent(filename1, filename2) File "C:\Python24\lib\test\test_unicode_file.py", line 67, in _do_equivilent os.stat(filename2)) OSError: [Errno 2] No such file or directory: u'@test-\xe0\xf2' ------------------------------------------------------------ ---------- Ran 3 tests in 0.070s FAILED (errors=2) test test_unicode_file failed -- errors occurred; run in verbose mode for details 1 test failed: test_unicode_file ---------------------------------------------------------------------- Comment By: Miki Tebeka (tebeka) Date: 2004-09-07 08:56 Message: Logged In: YES user_id=358087 Yes it does :-( I have a winXP on IBM T-40. The other language that is installed is Hebrew. The problem that when encoding TESTFN_UNICODE with the file system ecoding (mbcs) the resulted string is "@test-??" which is not a legal file name on windows ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-07 08:31 Message: Logged In: YES user_id=80475 Does this still fail for you in Py2.4a3 ? If not, please note the result and close the bug. ---------------------------------------------------------------------- Comment By: Miki Tebeka (tebeka) Date: 2004-07-18 08:44 Message: Logged In: YES user_id=358087 File system is NTFS. winver gives: Version 5.1 (Build 2600.xpsp2.030422-1633: Service Pack 1) ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-07-15 18:55 Message: Logged In: YES user_id=21627 Are you using a FAT partition? What precise version string do you get in winver.exe? ---------------------------------------------------------------------- Comment By: Miki Tebeka (tebeka) Date: 2004-07-13 08:10 Message: Logged In: YES user_id=358087 Sorry, I forgot I've upgraded my OS lately. This *is* WinXP Pro. Any other data I can send? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-07-12 18:35 Message: Logged In: YES user_id=31435 Peculiar -- it works fine on WinXP Pro, which ought to work the same as Win2K here. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=989338&group_id=5470 From noreply at sourceforge.net Fri Oct 15 08:27:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 08:27:38 2004 Subject: [ python-Bugs-1046092 ] HTMLParser fix to accept mailformed tag attributes Message-ID: Bugs item #1046092, was opened at 2004-10-13 14:11 Message generated for change (Comment added) made by nnseva You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046092&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Vsevolod Novikov (nnseva) Assigned to: Nobody/Anonymous (nobody) Summary: HTMLParser fix to accept mailformed tag attributes Initial Comment: This is a patch to fix bugs #975556 and #921657. I think, it should be made just because the parser should accept as many pages as it can. At the other hand, the code near to fixed contains regexp to accept mailformed attributes in other cases: compare attrfind variable and locatestarttagend variable values. ---------------------------------------------------------------------- >Comment By: Vsevolod Novikov (nnseva) Date: 2004-10-15 10:27 Message: Logged In: YES user_id=325678 Missed patch, sorry ... ---------------------------------------------------------------------- Comment By: Vsevolod Novikov (nnseva) Date: 2004-10-15 10:27 Message: Logged In: YES user_id=325678 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-13 15:09 Message: Logged In: YES user_id=469548 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046092&group_id=5470 From noreply at sourceforge.net Fri Oct 15 15:31:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 15:31:24 2004 Subject: [ python-Bugs-1044382 ] Can't raise "C API version mismatch" warning Message-ID: Bugs item #1044382, was opened at 2004-10-11 03:22 Message generated for change (Comment added) made by etrepum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044382&group_id=5470 Category: Extension Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Frank Sonnenburg (sonnenburg) Assigned to: Nobody/Anonymous (nobody) Summary: Can't raise "C API version mismatch" warning Initial Comment: See following example (that's what we know): # Importing 2.1 module with Python 2.3 frank@theta:~> python Python 2.3+ (#1, Jan 7 2004, 09:17:35) [GCC 3.3.1 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.insert(0, '/usr/lib/python2.1/lib-dynload') >>> import cmath __main__:1: RuntimeWarning: Python C API version mismatch for module cmath: This Python has API version 1012, module cmath has version 1010. >>> ================================== Ignoring this warning works without problem, but raising an exception will lead to segmentation fault: # Starting python with warnings turned into exceptions frank@theta:~> python -W error Python 2.3+ (#1, Jan 7 2004, 09:17:35) [GCC 3.3.1 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.insert(0, '/usr/lib/python2.1/lib-dynload') >>> import cmath Speicherzugriffsfehler (seg fault) ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-10-15 09:31 Message: Logged In: YES user_id=139309 It's not valid to use extensions from one major version of Python with another, hence the C API Version Mismatch warning. What do you expect? You're lucky it doesn't segfault sooner :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044382&group_id=5470 From noreply at sourceforge.net Fri Oct 15 15:49:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 15:49:44 2004 Subject: [ python-Bugs-1044382 ] Can't raise "C API version mismatch" warning Message-ID: Bugs item #1044382, was opened at 2004-10-11 08:22 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044382&group_id=5470 Category: Extension Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Frank Sonnenburg (sonnenburg) Assigned to: Nobody/Anonymous (nobody) Summary: Can't raise "C API version mismatch" warning Initial Comment: See following example (that's what we know): # Importing 2.1 module with Python 2.3 frank@theta:~> python Python 2.3+ (#1, Jan 7 2004, 09:17:35) [GCC 3.3.1 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.insert(0, '/usr/lib/python2.1/lib-dynload') >>> import cmath __main__:1: RuntimeWarning: Python C API version mismatch for module cmath: This Python has API version 1012, module cmath has version 1010. >>> ================================== Ignoring this warning works without problem, but raising an exception will lead to segmentation fault: # Starting python with warnings turned into exceptions frank@theta:~> python -W error Python 2.3+ (#1, Jan 7 2004, 09:17:35) [GCC 3.3.1 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.insert(0, '/usr/lib/python2.1/lib-dynload') >>> import cmath Speicherzugriffsfehler (seg fault) ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-15 14:49 Message: Logged In: YES user_id=6656 Well, maybe but this still looks like a bug to me. ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-10-15 14:31 Message: Logged In: YES user_id=139309 It's not valid to use extensions from one major version of Python with another, hence the C API Version Mismatch warning. What do you expect? You're lucky it doesn't segfault sooner :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044382&group_id=5470 From noreply at sourceforge.net Fri Oct 15 16:27:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 16:27:56 2004 Subject: [ python-Bugs-1016626 ] distutils support for swig is under par Message-ID: Bugs item #1016626, was opened at 2004-08-26 11:11 Message generated for change (Comment added) made by sjoerd You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1016626&group_id=5470 Category: Distutils Group: None Status: Open Resolution: None Priority: 5 Submitted By: Sjoerd Mullender (sjoerd) Assigned to: Nobody/Anonymous (nobody) Summary: distutils support for swig is under par Initial Comment: To my delight there is support for swig built into distutils. However a few problems exist with the support. - there is no way to specify any extra flags to the swig command (I need -I/usr/include -DHAVE_LONG_LONG); - swig generates two files, the XXX_wrap.c file which is turned into a _XXX.so file, and a XXX.py file. distutils only deals with the XXX_wrap.c file. I have no suggested fix for these problems, but problems they are. (Python from CVS HEAD on Fedora Core 2). ---------------------------------------------------------------------- >Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-15 16:27 Message: Logged In: YES user_id=43607 Part of the problem described here (the extra options that need to be given to swig) is solved by [ 1046644 ] improving distutils swig support (2). However, the other part is still a problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1016626&group_id=5470 From noreply at sourceforge.net Fri Oct 15 16:37:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 16:37:21 2004 Subject: [ python-Bugs-1016626 ] distutils support for swig is under par Message-ID: Bugs item #1016626, was opened at 2004-08-26 11:11 Message generated for change (Comment added) made by sjoerd You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1016626&group_id=5470 Category: Distutils Group: None Status: Open Resolution: None Priority: 5 Submitted By: Sjoerd Mullender (sjoerd) Assigned to: Nobody/Anonymous (nobody) Summary: distutils support for swig is under par Initial Comment: To my delight there is support for swig built into distutils. However a few problems exist with the support. - there is no way to specify any extra flags to the swig command (I need -I/usr/include -DHAVE_LONG_LONG); - swig generates two files, the XXX_wrap.c file which is turned into a _XXX.so file, and a XXX.py file. distutils only deals with the XXX_wrap.c file. I have no suggested fix for these problems, but problems they are. (Python from CVS HEAD on Fedora Core 2). ---------------------------------------------------------------------- >Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-15 16:37 Message: Logged In: YES user_id=43607 I just came up with a possible solution: In command/build.py change the order of the build.sub_commands list and put the entry for build_ext before build_py. The question is, does this cause a problem for anybody? ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-15 16:27 Message: Logged In: YES user_id=43607 Part of the problem described here (the extra options that need to be given to swig) is solved by [ 1046644 ] improving distutils swig support (2). However, the other part is still a problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1016626&group_id=5470 From noreply at sourceforge.net Fri Oct 15 17:34:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 17:34:57 2004 Subject: [ python-Bugs-1016626 ] distutils support for swig is under par Message-ID: Bugs item #1016626, was opened at 2004-08-26 11:11 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1016626&group_id=5470 Category: Distutils Group: None Status: Open Resolution: None Priority: 5 Submitted By: Sjoerd Mullender (sjoerd) Assigned to: Nobody/Anonymous (nobody) Summary: distutils support for swig is under par Initial Comment: To my delight there is support for swig built into distutils. However a few problems exist with the support. - there is no way to specify any extra flags to the swig command (I need -I/usr/include -DHAVE_LONG_LONG); - swig generates two files, the XXX_wrap.c file which is turned into a _XXX.so file, and a XXX.py file. distutils only deals with the XXX_wrap.c file. I have no suggested fix for these problems, but problems they are. (Python from CVS HEAD on Fedora Core 2). ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-10-15 17:34 Message: Logged In: YES user_id=11105 Haven't read the original problem (I don't use swig myself), but if changing the order of the sub_commands list than it can as well be done in the setup script by installing a custom build command. ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-15 16:37 Message: Logged In: YES user_id=43607 I just came up with a possible solution: In command/build.py change the order of the build.sub_commands list and put the entry for build_ext before build_py. The question is, does this cause a problem for anybody? ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-15 16:27 Message: Logged In: YES user_id=43607 Part of the problem described here (the extra options that need to be given to swig) is solved by [ 1046644 ] improving distutils swig support (2). However, the other part is still a problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1016626&group_id=5470 From noreply at sourceforge.net Fri Oct 15 20:07:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 20:07:11 2004 Subject: [ python-Bugs-1016626 ] distutils support for swig is under par Message-ID: Bugs item #1016626, was opened at 2004-08-26 11:11 Message generated for change (Comment added) made by sjoerd You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1016626&group_id=5470 Category: Distutils Group: None Status: Open Resolution: None Priority: 5 Submitted By: Sjoerd Mullender (sjoerd) Assigned to: Nobody/Anonymous (nobody) Summary: distutils support for swig is under par Initial Comment: To my delight there is support for swig built into distutils. However a few problems exist with the support. - there is no way to specify any extra flags to the swig command (I need -I/usr/include -DHAVE_LONG_LONG); - swig generates two files, the XXX_wrap.c file which is turned into a _XXX.so file, and a XXX.py file. distutils only deals with the XXX_wrap.c file. I have no suggested fix for these problems, but problems they are. (Python from CVS HEAD on Fedora Core 2). ---------------------------------------------------------------------- >Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-15 20:07 Message: Logged In: YES user_id=43607 Hmm, you can't really call that "decent swig support" then, can you? If this "solution" were properly documented (in a place one would notice when trying to create a setup.py to do swig!) then it wouldn't be too much of a problem, but I wouldn't know where that would be. So, if my change doesn't cause any detrimental effects I'd much rather see it fixed that way. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-10-15 17:34 Message: Logged In: YES user_id=11105 Haven't read the original problem (I don't use swig myself), but if changing the order of the sub_commands list than it can as well be done in the setup script by installing a custom build command. ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-15 16:37 Message: Logged In: YES user_id=43607 I just came up with a possible solution: In command/build.py change the order of the build.sub_commands list and put the entry for build_ext before build_py. The question is, does this cause a problem for anybody? ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-15 16:27 Message: Logged In: YES user_id=43607 Part of the problem described here (the extra options that need to be given to swig) is solved by [ 1046644 ] improving distutils swig support (2). However, the other part is still a problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1016626&group_id=5470 From noreply at sourceforge.net Fri Oct 15 20:19:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 15 20:19:05 2004 Subject: [ python-Bugs-1016626 ] distutils support for swig is under par Message-ID: Bugs item #1016626, was opened at 2004-08-26 11:11 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1016626&group_id=5470 Category: Distutils Group: None Status: Open Resolution: None Priority: 5 Submitted By: Sjoerd Mullender (sjoerd) Assigned to: Nobody/Anonymous (nobody) Summary: distutils support for swig is under par Initial Comment: To my delight there is support for swig built into distutils. However a few problems exist with the support. - there is no way to specify any extra flags to the swig command (I need -I/usr/include -DHAVE_LONG_LONG); - swig generates two files, the XXX_wrap.c file which is turned into a _XXX.so file, and a XXX.py file. distutils only deals with the XXX_wrap.c file. I have no suggested fix for these problems, but problems they are. (Python from CVS HEAD on Fedora Core 2). ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-10-15 20:19 Message: Logged In: YES user_id=11105 > Hmm, you can't really call that "decent swig support" then, can you? Of course not ;-) And I have setup scripts myself which would be more happy with this change, anyway, because they need the extensions built with build_ext to generate python modules. So, personally, I'm +1 on this change (but what does this count). ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-15 20:07 Message: Logged In: YES user_id=43607 Hmm, you can't really call that "decent swig support" then, can you? If this "solution" were properly documented (in a place one would notice when trying to create a setup.py to do swig!) then it wouldn't be too much of a problem, but I wouldn't know where that would be. So, if my change doesn't cause any detrimental effects I'd much rather see it fixed that way. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-10-15 17:34 Message: Logged In: YES user_id=11105 Haven't read the original problem (I don't use swig myself), but if changing the order of the sub_commands list than it can as well be done in the setup script by installing a custom build command. ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-15 16:37 Message: Logged In: YES user_id=43607 I just came up with a possible solution: In command/build.py change the order of the build.sub_commands list and put the entry for build_ext before build_py. The question is, does this cause a problem for anybody? ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-15 16:27 Message: Logged In: YES user_id=43607 Part of the problem described here (the extra options that need to be given to swig) is solved by [ 1046644 ] improving distutils swig support (2). However, the other part is still a problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1016626&group_id=5470 From noreply at sourceforge.net Sat Oct 16 00:25:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 16 00:25:49 2004 Subject: [ python-Bugs-1043446 ] Uninterruptable loop Message-ID: Bugs item #1043446, was opened at 2004-10-09 00:26 Message generated for change (Comment added) made by tjreedy You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1043446&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Jp Calderone (kuran) Assigned to: Nobody/Anonymous (nobody) Summary: Uninterruptable loop Initial Comment: import itertools list(itertools.repeat('x')) ^C will not interrupt this. ---------------------------------------------------------------------- Comment By: Terry J. Reedy (tjreedy) Date: 2004-10-15 18:25 Message: Logged In: YES user_id=593130 Killing the interpreter will, of course, interrupt anything ;-). The ability to ask the interpreter, via an alternate non-code communication channel, to stop doing what one just requested via the normal code input channel, is an implementation-specific metafeature independent of the language definition. So I see this as a CPython feature- expansion request rather than a bug report. If the CPython interpreter documentation promises that ^C or whatever will always grab the interpreter's attention, then that overpromise is a bug that should be modified. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-10 04:06 Message: Logged In: YES user_id=80475 FWIW, there are many variations of this theme using almost anything accepting an iterable input (dict.fromkeys, tuple, set, etc) paired with any long running or infinite iterator (itertools.count, xrange(sys.maxint), etc). Even the tp_print slot can get caught up in emitting oversized output in a way that is uninterruptable. I don't see a clean way of teaching all of these functions to periodically check for signals, and either handle them, raise an exception or continue. Fixing this could muck-up and complicate a bunch of otherwise clean code. Still, it would be nice if everything were interruptable. I'm just not sure it's worth it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1043446&group_id=5470 From noreply at sourceforge.net Sat Oct 16 00:49:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 16 00:50:01 2004 Subject: [ python-Bugs-1044382 ] Can't raise "C API version mismatch" warning Message-ID: Bugs item #1044382, was opened at 2004-10-11 03:22 Message generated for change (Comment added) made by tjreedy You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044382&group_id=5470 Category: Extension Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Frank Sonnenburg (sonnenburg) Assigned to: Nobody/Anonymous (nobody) Summary: Can't raise "C API version mismatch" warning Initial Comment: See following example (that's what we know): # Importing 2.1 module with Python 2.3 frank@theta:~> python Python 2.3+ (#1, Jan 7 2004, 09:17:35) [GCC 3.3.1 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.insert(0, '/usr/lib/python2.1/lib-dynload') >>> import cmath __main__:1: RuntimeWarning: Python C API version mismatch for module cmath: This Python has API version 1012, module cmath has version 1010. >>> ================================== Ignoring this warning works without problem, but raising an exception will lead to segmentation fault: # Starting python with warnings turned into exceptions frank@theta:~> python -W error Python 2.3+ (#1, Jan 7 2004, 09:17:35) [GCC 3.3.1 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.insert(0, '/usr/lib/python2.1/lib-dynload') >>> import cmath Speicherzugriffsfehler (seg fault) ---------------------------------------------------------------------- Comment By: Terry J. Reedy (tjreedy) Date: 2004-10-15 18:49 Message: Logged In: YES user_id=593130 Do you get same 2.4? (b1 should be out within a day.) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-15 09:49 Message: Logged In: YES user_id=6656 Well, maybe but this still looks like a bug to me. ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-10-15 09:31 Message: Logged In: YES user_id=139309 It's not valid to use extensions from one major version of Python with another, hence the C API Version Mismatch warning. What do you expect? You're lucky it doesn't segfault sooner :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044382&group_id=5470 From noreply at sourceforge.net Sat Oct 16 00:55:22 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 16 00:55:37 2004 Subject: [ python-Bugs-1046077 ] urllib2: better import ftplib and gopherlib etc late Message-ID: Bugs item #1046077, was opened at 2004-10-13 05:41 Message generated for change (Comment added) made by tjreedy You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046077&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Robert Kiendl (kxroberto) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2: better import ftplib and gopherlib etc late Initial Comment: I've always trouble shrinking a py2exe'd package, because of this. its also a speed issue. urllib2 is most time used only with http(s) ...? *** urllib2_old.py Tue May 11 16:14:34 2004 --- urllib2.py Wed Oct 13 11:32:44 2004 *************** f = urllib2.urlopen('http://www.python.o *** 88,95 **** # check digest against correct (i.e. non-apache) implementation import base64 - import ftplib - import gopherlib import httplib import inspect import md5 --- 88,93 ---- *************** class FileHandler(BaseHandler): *** 1009,1014 **** --- 1007,1013 ---- class FTPHandler(BaseHandler): def ftp_open(self, req): + import ftplib host = req.get_host() if not host: raise IOError, ('ftp error', 'no host given') *************** class CacheFTPHandler(FTPHandler): *** 1110,1115 **** --- 1109,1115 ---- class GopherHandler(BaseHandler): def gopher_open(self, req): + import gopherlib host = req.get_host() if not host: raise GopherError('no host given') ---------------------------------------------------------------------- Comment By: Terry J. Reedy (tjreedy) Date: 2004-10-15 18:55 Message: Logged In: YES user_id=593130 Since you have a patch, this should have been submitted as a patch rather than a bug. In any case, the patch needs to be submitted as separate file ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046077&group_id=5470 From noreply at sourceforge.net Sat Oct 16 01:05:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 16 01:05:37 2004 Subject: [ python-Bugs-1046800 ] Error in the given example Message-ID: Bugs item #1046800, was opened at 2004-10-14 00:55 Message generated for change (Comment added) made by tjreedy You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046800&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Vishnu (vishnube) Assigned to: Nobody/Anonymous (nobody) Summary: Error in the given example Initial Comment: Python Documentation - Release 2.3.3 - December 19, 2003 In section, 16.1.6.5 The Window Manager, there is one problem exists in the given example. Please change the first line of the example FROM: import Tkinter TO: from Tkinter import * Thank you, Vishnu ---------------------------------------------------------------------- Comment By: Terry J. Reedy (tjreedy) Date: 2004-10-15 19:05 Message: Logged In: YES user_id=593130 Or to 'from Tkinter import Frame' or prefix 'Frame' with 'Tkinter.' in its two appearances. However, OP's version is consistent with, for instance, 16.1.2.2. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046800&group_id=5470 From noreply at sourceforge.net Sat Oct 16 10:22:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 16 10:22:26 2004 Subject: [ python-Bugs-1048234 ] stat documentation is incorrect Message-ID: Bugs item #1048234, was opened at 2004-10-16 01:22 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048234&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: libber ator (libber) Assigned to: Nobody/Anonymous (nobody) Summary: stat documentation is incorrect Initial Comment: http://www.python.org/doc/2.3.4/lib/module-stat.html You no longer do this: mode = os.stat(file)[ST_MODE] You instead do: mode = os.stat(file).st_mode I would happily update this doc, just tell me the correct procedure. Python has been a joy to use. libber@gmail.com Collin Greene ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048234&group_id=5470 From noreply at sourceforge.net Sat Oct 16 12:53:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 16 12:53:03 2004 Subject: [ python-Bugs-1048234 ] stat documentation is incorrect Message-ID: Bugs item #1048234, was opened at 2004-10-16 10:22 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048234&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: libber ator (libber) Assigned to: Nobody/Anonymous (nobody) Summary: stat documentation is incorrect Initial Comment: http://www.python.org/doc/2.3.4/lib/module-stat.html You no longer do this: mode = os.stat(file)[ST_MODE] You instead do: mode = os.stat(file).st_mode I would happily update this doc, just tell me the correct procedure. Python has been a joy to use. libber@gmail.com Collin Greene ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-16 12:53 Message: Logged In: YES user_id=469548 You should first get the Python source from CVS (there's a link in the bar above). When you've got the source, you should look for a file called 'libstat.text' (in dist/src/python/Doc/lib). You can update the docs for this module in that file, create a patch and submit it back to us. Oh, if some of the above was complete gibberish to you, http://python.org/dev/ is also a good source. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048234&group_id=5470 From noreply at sourceforge.net Sat Oct 16 12:57:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 16 12:57:23 2004 Subject: [ python-Bugs-1048234 ] stat documentation is incorrect Message-ID: Bugs item #1048234, was opened at 2004-10-16 10:22 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048234&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: libber ator (libber) Assigned to: Nobody/Anonymous (nobody) Summary: stat documentation is incorrect Initial Comment: http://www.python.org/doc/2.3.4/lib/module-stat.html You no longer do this: mode = os.stat(file)[ST_MODE] You instead do: mode = os.stat(file).st_mode I would happily update this doc, just tell me the correct procedure. Python has been a joy to use. libber@gmail.com Collin Greene ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-16 12:57 Message: Logged In: YES user_id=469548 By the way: just providing the text and where to put it would be great too! ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-16 12:53 Message: Logged In: YES user_id=469548 You should first get the Python source from CVS (there's a link in the bar above). When you've got the source, you should look for a file called 'libstat.text' (in dist/src/python/Doc/lib). You can update the docs for this module in that file, create a patch and submit it back to us. Oh, if some of the above was complete gibberish to you, http://python.org/dev/ is also a good source. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048234&group_id=5470 From noreply at sourceforge.net Sat Oct 16 12:59:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 16 12:59:22 2004 Subject: [ python-Bugs-907457 ] Docs for os, popen2 omit list argument to popen2() et al Message-ID: Bugs item #907457, was opened at 2004-03-01 12:00 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=907457&group_id=5470 Category: Documentation Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Michael Hoffman (hoffmanm) Assigned to: Nobody/Anonymous (nobody) Summary: Docs for os, popen2 omit list argument to popen2() et al Initial Comment: http://www.python.org/doc/current/lib/os-newstreams.html and http://www.python.org/doc/current/lib/module-popen2.html do not give any indication that you can use a list for cmd instead of a string and os.execvp() will be used instead of setting up a subshell. ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-16 12:59 Message: Logged In: YES user_id=469548 Patch #1042705, applied a while ago, fixed this. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=907457&group_id=5470 From noreply at sourceforge.net Sat Oct 16 13:46:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 16 13:46:39 2004 Subject: [ python-Bugs-1044872 ] Unable to install Docs - Linux Message-ID: Bugs item #1044872, was opened at 2004-10-11 22:54 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044872&group_id=5470 Category: Build Group: Python 2.3 >Status: Closed >Resolution: Out of Date Priority: 5 Submitted By: Colin J. Williams (cjwhrh) Assigned to: Nobody/Anonymous (nobody) Summary: Unable to install Docs - Linux Initial Comment: Please see the tail of the make output below: [6654 refs] TEXINPUTS=/usr/Downloads/Python/Python-2.3.4/Doc/commontex: mkhowto --html --about html/stdabout.dat --iconserver ../icons --favicon ../icons/pyfav.gif --address "See About this document... for information on suggesting changes." --up-link ../index.html --up-title "Python Documentation Index" --global-module-index "../modindex.html" --dvips-safe --dir html/api api/api.tex /bin/sh: line 1: mkhowto: command not found make: *** [html/api/api.html] Error 127 root@P4:/usr/Downloads/Python/Python-2.3.4/Doc# Attempted installation with Debian from Libranet 2.8.1 Latex2HTML has been installed. ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-16 13:46 Message: Logged In: YES user_id=469548 This is a problem with the Makefile shipped with Python 2.3.4. It has been fixed in 2.4. You can either install mkhowto as described in http://docs.python.org/doc/tools-internal.html, or uncomment this line: #MKHOWTO= TEXINPUTS=$(TEXINPUTS) $(PYTHON) $(PWD)/tools/mkhowto and comment this line: MKHOWTO= TEXINPUTS=$(TEXINPUTS) mkhowto ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044872&group_id=5470 From noreply at sourceforge.net Sat Oct 16 18:00:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 16 18:00:54 2004 Subject: [ python-Bugs-1035703 ] distutils.util.get_platform() should include sys.version[:3] Message-ID: Bugs item #1035703, was opened at 2004-09-27 16:06 Message generated for change (Comment added) made by etrepum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1035703&group_id=5470 Category: Distutils Group: Feature Request >Status: Deleted Resolution: None Priority: 5 Submitted By: Bob Ippolito (etrepum) Assigned to: Nobody/Anonymous (nobody) Summary: distutils.util.get_platform() should include sys.version[:3] Initial Comment: When dealing with systems that have multiple versions of Python installed, it would be nice if distutils.util.get_platform() included the major python version (sys.version[:3]) so that you could do: python2.3 setup.py install python2.4 setup.py install and expect everything to typically work correctly. ---------------------------------------------------------------------- >Comment By: Bob Ippolito (etrepum) Date: 2004-10-16 12:00 Message: Logged In: YES user_id=139309 I think this was my mistake, not a bug. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1035703&group_id=5470 From noreply at sourceforge.net Sat Oct 16 23:44:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 16 23:44:38 2004 Subject: [ python-Bugs-1046800 ] Error in the given example Message-ID: Bugs item #1046800, was opened at 2004-10-14 01:55 Message generated for change (Comment added) made by facundobatista You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046800&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Vishnu (vishnube) Assigned to: Nobody/Anonymous (nobody) Summary: Error in the given example Initial Comment: Python Documentation - Release 2.3.3 - December 19, 2003 In section, 16.1.6.5 The Window Manager, there is one problem exists in the given example. Please change the first line of the example FROM: import Tkinter TO: from Tkinter import * Thank you, Vishnu ---------------------------------------------------------------------- >Comment By: Facundo Batista (facundobatista) Date: 2004-10-16 18:44 Message: Logged In: YES user_id=752496 Changed to "from Tkinter import *". Thank you! ---------------------------------------------------------------------- Comment By: Terry J. Reedy (tjreedy) Date: 2004-10-15 20:05 Message: Logged In: YES user_id=593130 Or to 'from Tkinter import Frame' or prefix 'Frame' with 'Tkinter.' in its two appearances. However, OP's version is consistent with, for instance, 16.1.2.2. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1046800&group_id=5470 From noreply at sourceforge.net Sun Oct 17 00:49:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 00:49:44 2004 Subject: [ python-Bugs-1048495 ] Memory leaks? Message-ID: Bugs item #1048495, was opened at 2004-10-17 04:49 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048495&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Roman Mamedov (romanrm) Assigned to: Nobody/Anonymous (nobody) Summary: Memory leaks? Initial Comment: Open python command-line interpreter. Enter: >>> a = range (10000000) Observe Python memory usage. 20 Mb real, 159 Mb virtual memory here(I'm on windows). Enter: >>> a = 0 Observe memory usage again. 120 mb real/120 mb virtual. OK, this is a garbage collected language, lets try to garbage-collect. >>> import gc >>> gc.collect() 0 That didn't help. The memory usage is still at 120/120. So, the question is, when that "range" object will get deleted, or how to do delete it manually? Why garbage collection doesn't get rid of "orphaned" objects? Any comments? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048495&group_id=5470 From noreply at sourceforge.net Sun Oct 17 01:05:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 01:05:58 2004 Subject: [ python-Bugs-1048498 ] SystemError with deque Message-ID: Bugs item #1048498, was opened at 2004-10-17 08:05 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048498&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Nobody/Anonymous (nobody) Summary: SystemError with deque Initial Comment: If I run the following script, it causes a SystemError. # start of program from collections import deque d1 = deque() d2 = deque() class A: def __eq__(self, other): d1.pop() d1.appendleft(1) return 1 d1.extend((0, 1)) d2.extend((A(), 1)) d1 == d2 # end of program $ python2.4 deque_demo.py Traceback (most recent call last): File "deque_demo.py", line 15, in ? d1 == d2 SystemError: error return without exception set This error happens if - __eq__ returns values that are interpreted as True. - appendleft is used insted of append. Tested on Python 2.4a3(Win) and python2.4 from CVS as of 2004/10/15. I attached this program for convenience. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048498&group_id=5470 From noreply at sourceforge.net Sun Oct 17 02:04:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 02:05:04 2004 Subject: [ python-Bugs-1048495 ] Memory leaks? Message-ID: Bugs item #1048495, was opened at 2004-10-16 18:49 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048495&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Roman Mamedov (romanrm) Assigned to: Nobody/Anonymous (nobody) Summary: Memory leaks? Initial Comment: Open python command-line interpreter. Enter: >>> a = range (10000000) Observe Python memory usage. 20 Mb real, 159 Mb virtual memory here(I'm on windows). Enter: >>> a = 0 Observe memory usage again. 120 mb real/120 mb virtual. OK, this is a garbage collected language, lets try to garbage-collect. >>> import gc >>> gc.collect() 0 That didn't help. The memory usage is still at 120/120. So, the question is, when that "range" object will get deleted, or how to do delete it manually? Why garbage collection doesn't get rid of "orphaned" objects? Any comments? ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-16 20:04 Message: Logged In: YES user_id=31435 range() constructs a list. The list takes 4 bytes/entry, so you get about 40MB reclaimed when the list goes away. The space for integer objects happens to be immortal, though, and the approximately 12 bytes per integer doesn't go away. Space for floats is also immortal, BTW. There aren't easy resolutions. For example, the caching of space for integer objects in a dedicated internal int freelist speeds many programs. And if Python didn't do special memory allocation for ints, malloc overhead would probably boost the memory burden in your example to 16 bytes/int. So there are tradeoffs. Note that xrange() can usually be used instead to create one integer at a time (instead of creating 10 million simultaneously). Then the memory burden is trivial. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048495&group_id=5470 From noreply at sourceforge.net Sun Oct 17 03:03:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 03:03:24 2004 Subject: [ python-Bugs-1048498 ] SystemError with deque Message-ID: Bugs item #1048498, was opened at 2004-10-16 18:05 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048498&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: George Yoshida (quiver) >Assigned to: Raymond Hettinger (rhettinger) Summary: SystemError with deque Initial Comment: If I run the following script, it causes a SystemError. # start of program from collections import deque d1 = deque() d2 = deque() class A: def __eq__(self, other): d1.pop() d1.appendleft(1) return 1 d1.extend((0, 1)) d2.extend((A(), 1)) d1 == d2 # end of program $ python2.4 deque_demo.py Traceback (most recent call last): File "deque_demo.py", line 15, in ? d1 == d2 SystemError: error return without exception set This error happens if - __eq__ returns values that are interpreted as True. - appendleft is used insted of append. Tested on Python 2.4a3(Win) and python2.4 from CVS as of 2004/10/15. I attached this program for convenience. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-16 20:03 Message: Logged In: YES user_id=80475 I think that has been fixed. Please reverify with Py2.4b1 when it comes out. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048498&group_id=5470 From noreply at sourceforge.net Sun Oct 17 04:18:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 04:19:04 2004 Subject: [ python-Bugs-1048498 ] SystemError with deque Message-ID: Bugs item #1048498, was opened at 2004-10-16 18:05 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048498&group_id=5470 Category: Python Library Group: Python 2.4 >Status: Closed >Resolution: Out of Date Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Raymond Hettinger (rhettinger) Summary: SystemError with deque Initial Comment: If I run the following script, it causes a SystemError. # start of program from collections import deque d1 = deque() d2 = deque() class A: def __eq__(self, other): d1.pop() d1.appendleft(1) return 1 d1.extend((0, 1)) d2.extend((A(), 1)) d1 == d2 # end of program $ python2.4 deque_demo.py Traceback (most recent call last): File "deque_demo.py", line 15, in ? d1 == d2 SystemError: error return without exception set This error happens if - __eq__ returns values that are interpreted as True. - appendleft is used insted of append. Tested on Python 2.4a3(Win) and python2.4 from CVS as of 2004/10/15. I attached this program for convenience. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-16 20:03 Message: Logged In: YES user_id=80475 I think that has been fixed. Please reverify with Py2.4b1 when it comes out. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048498&group_id=5470 From noreply at sourceforge.net Sun Oct 17 07:43:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 07:43:23 2004 Subject: [ python-Bugs-1048234 ] stat documentation is incorrect Message-ID: Bugs item #1048234, was opened at 2004-10-16 01:22 Message generated for change (Comment added) made by libber You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048234&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: libber ator (libber) Assigned to: Nobody/Anonymous (nobody) Summary: stat documentation is incorrect Initial Comment: http://www.python.org/doc/2.3.4/lib/module-stat.html You no longer do this: mode = os.stat(file)[ST_MODE] You instead do: mode = os.stat(file).st_mode I would happily update this doc, just tell me the correct procedure. Python has been a joy to use. libber@gmail.com Collin Greene ---------------------------------------------------------------------- >Comment By: libber ator (libber) Date: 2004-10-16 22:43 Message: Logged In: YES user_id=705534 Whoops, I found out I was just wrong, there is nothing incorrect about this documentation, sorry! ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-16 03:57 Message: Logged In: YES user_id=469548 By the way: just providing the text and where to put it would be great too! ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-16 03:53 Message: Logged In: YES user_id=469548 You should first get the Python source from CVS (there's a link in the bar above). When you've got the source, you should look for a file called 'libstat.text' (in dist/src/python/Doc/lib). You can update the docs for this module in that file, create a patch and submit it back to us. Oh, if some of the above was complete gibberish to you, http://python.org/dev/ is also a good source. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048234&group_id=5470 From noreply at sourceforge.net Sun Oct 17 12:54:08 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 12:54:13 2004 Subject: [ python-Bugs-907457 ] Docs for os, popen2 omit list argument to popen2() et al Message-ID: Bugs item #907457, was opened at 2004-03-01 11:00 Message generated for change (Comment added) made by hoffmanm You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=907457&group_id=5470 Category: Documentation Group: None Status: Closed Resolution: Fixed Priority: 5 Submitted By: Michael Hoffman (hoffmanm) Assigned to: Nobody/Anonymous (nobody) Summary: Docs for os, popen2 omit list argument to popen2() et al Initial Comment: http://www.python.org/doc/current/lib/os-newstreams.html and http://www.python.org/doc/current/lib/module-popen2.html do not give any indication that you can use a list for cmd instead of a string and os.execvp() will be used instead of setting up a subshell. ---------------------------------------------------------------------- >Comment By: Michael Hoffman (hoffmanm) Date: 2004-10-17 10:54 Message: Logged In: YES user_id=987664 Thanks! ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-16 10:59 Message: Logged In: YES user_id=469548 Patch #1042705, applied a while ago, fixed this. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=907457&group_id=5470 From noreply at sourceforge.net Sun Oct 17 15:25:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 15:25:14 2004 Subject: [ python-Bugs-1048234 ] stat documentation is incorrect Message-ID: Bugs item #1048234, was opened at 2004-10-16 10:22 Message generated for change (Settings changed) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048234&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: libber ator (libber) Assigned to: Nobody/Anonymous (nobody) Summary: stat documentation is incorrect Initial Comment: http://www.python.org/doc/2.3.4/lib/module-stat.html You no longer do this: mode = os.stat(file)[ST_MODE] You instead do: mode = os.stat(file).st_mode I would happily update this doc, just tell me the correct procedure. Python has been a joy to use. libber@gmail.com Collin Greene ---------------------------------------------------------------------- Comment By: libber ator (libber) Date: 2004-10-17 07:43 Message: Logged In: YES user_id=705534 Whoops, I found out I was just wrong, there is nothing incorrect about this documentation, sorry! ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-16 12:57 Message: Logged In: YES user_id=469548 By the way: just providing the text and where to put it would be great too! ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-16 12:53 Message: Logged In: YES user_id=469548 You should first get the Python source from CVS (there's a link in the bar above). When you've got the source, you should look for a file called 'libstat.text' (in dist/src/python/Doc/lib). You can update the docs for this module in that file, create a patch and submit it back to us. Oh, if some of the above was complete gibberish to you, http://python.org/dev/ is also a good source. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048234&group_id=5470 From noreply at sourceforge.net Sun Oct 17 15:46:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 15:46:32 2004 Subject: [ python-Bugs-1038013 ] textReader: reading after close is a seg fault Message-ID: Bugs item #1038013, was opened at 2004-09-30 16:11 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038013&group_id=5470 Category: XML >Group: 3rd Party >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Jyrki Alakuijala (jyrkialakuijala) Assigned to: Nobody/Anonymous (nobody) Summary: textReader: reading after close is a seg fault Initial Comment: Python 2.3.3 (#1, May 7 2004, 10:31:40) [GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2 the code below crashes on fedora core 2: ------------------------------------ import libxml2 file = open("ihanmikavaan.xml", "w") file.write("hajoa!") file.close() reader = libxml2.newTextReaderFilename("ihanmikavaan.xml") reader.Close() # reading after Close is a seg fault, not an exception ret = reader.Read() -------------------------------------- ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-17 09:46 Message: Logged In: YES user_id=33168 libxml2 is not a standard part of python. You should report this bug to the libxml2 maintainers and/or RedHat. You can try starting from one of these links: http://xmlsoft.org/ http://bugzilla.redhat.com/bugzilla/ ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1038013&group_id=5470 From noreply at sourceforge.net Sun Oct 17 16:29:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 16:29:07 2004 Subject: [ python-Bugs-1048681 ] Add pythonX.dll to KnownDlls Message-ID: Bugs item #1048681, was opened at 2004-10-17 16:29 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048681&group_id=5470 Category: Installation Group: Feature Request Status: Open Resolution: None Priority: 5 Submitted By: Miki Tebeka (tebeka) Assigned to: Nobody/Anonymous (nobody) Summary: Add pythonX.dll to KnownDlls Initial Comment: For Windows systems, when installing pythonX.dll to system32 it should be added to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\KnownDLLs This will speed up the initial loading of python. Many applications the runs a lot of python processes (such as cgi ...) will run much faster. See http://support.microsoft.com/kb/q164501/ for more details. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048681&group_id=5470 From noreply at sourceforge.net Sun Oct 17 18:00:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 18:00:17 2004 Subject: [ python-Bugs-989337 ] test_descr fails on win2k Message-ID: Bugs item #989337, was opened at 2004-07-12 14:07 Message generated for change (Comment added) made by tebeka You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=989337&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Miki Tebeka (tebeka) Assigned to: Nobody/Anonymous (nobody) Summary: test_descr fails on win2k Initial Comment: >>> from test import test_descr >>> test_descr.verbose = 1 >>> test_descr.test_main() Testing weakref segfault... Testing SF bug 551412 ... Testing SF bug 570483... Testing list operations... checking a+b checking b in a checking b in a checking a[b] checking a[b:c] checking a+=b checking a*=b checking len(a) checking a*b checking b*a checking a[b]=c checking a[b:c]=d Testing dict operations... checking cmp(a,b) checking b in a checking b in a checking a[b] checking len(a) checking a[b]=c Testing dict constructor ... Testing dir() ... Testing int operations... checking a & b checking a ^ b checking a >> b checking a - b checking a ** b checking a + b checking divmod(a, b) checking a << b checking a * b checking a / b checking cmp(a, b) checking a | b checking a % b checking int(a) checking - a checking ~ a checking float(a) checking hex(a) checking + a checking long(a) checking abs(a) checking oct(a) Testing long operations... checking a & b checking a ^ b checking a >> b checking a - b checking a ** b checking a + b checking divmod(a, b) checking a << b checking a * b checking a / b checking cmp(a, b) checking a | b checking a % b checking int(a) checking - a checking ~ a checking float(a) checking hex(a) checking + a checking long(a) checking abs(a) checking oct(a) Testing float operations... checking a >= b checking a - b checking a > b checking a ** b checking a < b checking a != b checking a + b checking a <= b checking divmod(a, b) checking a * b checking a / b checking a == b checking a % b checking int(a) checking - a checking float(a) checking + a checking long(a) checking abs(a) Testing complex operations... checking a - b checking a ** b checking a != b checking a + b checking divmod(a, b) checking a * b checking a / b checking a == b checking a % b checking - a checking + a checking abs(a) Testing spamlist operations... checking a+b checking b in a checking b in a checking a[b] checking a[b:c] checking a+=b checking a*=b checking len(a) checking a*b checking b*a checking a[b]=c checking a[b:c]=d Testing spamdict operations... checking cmp(a,b) checking b in a checking b in a checking a[b] checking len(a) checking repr(a) checking a[b]=c Testing Python subclass of dict... pydict stress test ... Testing Python subclass of list... Testing __metaclass__... Testing Python subclass of module... Testing multiple inheritance... Testing error messages for MRO disagreement... Testing multiple inheritance special cases... Testing ex5 from C3 switch discussion... Testing MRO monotonicity... Testing consistentcy with EPG... Testing object class... Testing __slots__... Testing __dict__ and __weakref__ in __slots__... Testing class attribute propagation... Testing errors... Testing class methods... Testing C-based class methods... Testing static methods... Testing C-based static methods... Testing classic classes... Testing computed attributes... Testing __new__ slot override... Testing mro() and overriding it... Testing operator overloading... Testing methods... Testing special operators... Traceback (most recent call last): File "", line 1, in -toplevel- test_descr.test_main() File "C:\apps\Python24\lib\test\test_descr.py", line 4024, in test_main specials() File "C:\apps\Python24\lib\test\test_descr.py", line 1875, in specials print Letter('w') PicklingError: Can't pickle : attribute lookup test.test_descr.Letter failed >>> This is Python 2.4a1 on win2k pro ---------------------------------------------------------------------- >Comment By: Miki Tebeka (tebeka) Date: 2004-10-17 18:00 Message: Logged In: YES user_id=358087 Still here in Python-2.4b1 ---------------------------------------------------------------------- Comment By: Miki Tebeka (tebeka) Date: 2004-07-19 08:09 Message: Logged In: YES user_id=358087 Sorry, forgot that I've upgraded to winXP the other day. This *is* winXP pro SP1 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-07-18 22:46 Message: Logged In: YES user_id=80475 I cannot reproduce this on either WinME or WinXP. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=989337&group_id=5470 From noreply at sourceforge.net Sun Oct 17 18:06:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 18:06:16 2004 Subject: [ python-Bugs-1029561 ] test_pep277 fails Message-ID: Bugs item #1029561, was opened at 2004-09-17 02:24 Message generated for change (Comment added) made by tebeka You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1029561&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Marek Baczynski (imbaczek) Assigned to: Nobody/Anonymous (nobody) Summary: test_pep277 fails Initial Comment: Running regrtest.py: test_pep277 test test_pep277 produced unexpected output: ********************************************************************** *** mismatch between line 3 of expected output and line 3 of actual output: - ['???', '???', '??????', '????????????', '????G\xdf', 'Ge??-sa?', 'Gr\xfc\xdf-Gott', 'abc', 'ascii'] ? ^ ------------ + ['???', '???', '????-???', '??????', '????????????', '?????\xdf', 'Gr\xfc\xdf-Gott', 'abc', 'ascii'] ? ++++++++++++ ^ ********************************************************************** But when running just test_pep277.py: test_directory (__main__.UnicodeFileTests) ... u'\xdf-\u66e8\u66e9\u66eb' ok test_failures (__main__.UnicodeFileTests) ... ok test_listdir (__main__.UnicodeFileTests) ... ['???', '???', '????-???', '??????', '????????????', '?????\xdf', 'Gr\xfc\xdf-Gott', 'abc', 'ascii'] [u'Gr\xfc\xdf-Gott', u'abc', u'ascii', u'\u0393\u03b5\u03b9\u03ac-\u03c3\u03b1\u03c2', u'\u0417\u0434\u0440\u0430\u0432\u0441\u0442\u0432\u0443\u0439\u0442\u0435', u'\u05d4\u05e9\u05e7\u05e6\u05e5\u05e1', u'\u306b\u307d\u3093', u'\u66e8\u05e9\u3093\u0434\u0393\xdf', u'\u66e8\u66e9\u66eb'] ok test_open (__main__.UnicodeFileTests) ... ok test_rename (__main__.UnicodeFileTests) ... ok ---------------------------------------------------------------------- Ran 5 tests in 0.109s OK ---------------------------------------------------------------------- Comment By: Miki Tebeka (tebeka) Date: 2004-10-17 18:06 Message: Logged In: YES user_id=358087 Fails for me on 2.4b1 on winXP ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-09-17 12:55 Message: Logged In: YES user_id=671362 Same for me. Fail for regrtest.py, but OK for test_277.py in isolation. (Python 2.4a3, Windows 2K SP4, locale:Japan) But my output is a bit different from yours. test test_pep277 produced unexpected output: ********************************************** ************************ *** mismatch between line 3 of expected output and line 3 of actual output: - ['???', '???', '??????', '????????????', '????G\xdf', 'Ge??- sa?', 'Gr\xfc\xdf-Gott', 'abc', 'ascii'] + ['??????', '??\x82\xf1\x84t\x83\xa1s', '?\x9eG?', 'Grus- Gott', 'abc', 'ascii', '\x82\xc9\x82\xdb\x82\xf1', '\x83\xa1\x83 \xc3\x83\xc7?-\x83\xd0\x83\xbf?', '\x84H\x84t\x84\x82 \x84p\x84r\x84\x83\x84\x84\x84r\x84\x85\x84z\x84\x84\x84u'] ********************************************** ************************ This reminds me of the next thread: * More joy with test_strptime http://mail.python.org/pipermail/python-dev/2004- July/046046.html ---------------------------------------------------------------------- Comment By: Marek Baczynski (imbaczek) Date: 2004-09-17 02:25 Message: Logged In: YES user_id=838849 Forgot: Windows XP SP1, Python 2.4a3 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1029561&group_id=5470 From noreply at sourceforge.net Sun Oct 17 19:12:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 19:12:21 2004 Subject: [ python-Bugs-1048756 ] Doc typo Message-ID: Bugs item #1048756, was opened at 2004-10-17 17:12 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048756&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Scott David Daniels (scott_daniels) Assigned to: Nobody/Anonymous (nobody) Summary: Doc typo Initial Comment: Python 2.4b1 (but also 2.3 and ...) In the documentation for pdb, section 9.1 Debugger commands describes the 'pp' command as: pp expression Like the "p" command, except the value of the exception is pretty-printed using the pprint module. This should be: pp expression Like the "p" command, except the value of the _expression_ is pretty-printed using the pprint module. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048756&group_id=5470 From noreply at sourceforge.net Sun Oct 17 20:50:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 20:51:05 2004 Subject: [ python-Bugs-1048495 ] Memory leaks? Message-ID: Bugs item #1048495, was opened at 2004-10-17 04:49 Message generated for change (Comment added) made by romanrm You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048495&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Roman Mamedov (romanrm) Assigned to: Nobody/Anonymous (nobody) Summary: Memory leaks? Initial Comment: Open python command-line interpreter. Enter: >>> a = range (10000000) Observe Python memory usage. 20 Mb real, 159 Mb virtual memory here(I'm on windows). Enter: >>> a = 0 Observe memory usage again. 120 mb real/120 mb virtual. OK, this is a garbage collected language, lets try to garbage-collect. >>> import gc >>> gc.collect() 0 That didn't help. The memory usage is still at 120/120. So, the question is, when that "range" object will get deleted, or how to do delete it manually? Why garbage collection doesn't get rid of "orphaned" objects? Any comments? ---------------------------------------------------------------------- >Comment By: Roman Mamedov (romanrm) Date: 2004-10-18 00:50 Message: Logged In: YES user_id=943452 Thank you very much for a detailed explaination. In my opinion, this issue deserves more attention and consideration. There's a trend to create not just simple fire-off/throw-away scripts, but complex, long-running, GUI software in Python(as well as in other scripting/VM languages), and this tradeoff could make memory usage unnecessary high in not-so-rare usage patterns. That way, a split-second gain caused by having immortal integers could easily be eaten by VM trashing due to overconsumption of memory. I believe that comparable integer/float performance can be attained even without having these types as infinitely-immortal. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-17 06:04 Message: Logged In: YES user_id=31435 range() constructs a list. The list takes 4 bytes/entry, so you get about 40MB reclaimed when the list goes away. The space for integer objects happens to be immortal, though, and the approximately 12 bytes per integer doesn't go away. Space for floats is also immortal, BTW. There aren't easy resolutions. For example, the caching of space for integer objects in a dedicated internal int freelist speeds many programs. And if Python didn't do special memory allocation for ints, malloc overhead would probably boost the memory burden in your example to 16 bytes/int. So there are tradeoffs. Note that xrange() can usually be used instead to create one integer at a time (instead of creating 10 million simultaneously). Then the memory burden is trivial. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048495&group_id=5470 From noreply at sourceforge.net Sun Oct 17 21:24:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 21:24:14 2004 Subject: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 Message-ID: Bugs item #1048808, was opened at 2004-10-17 12:24 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Shalabh Chaturvedi (shalabh) Assigned to: Nobody/Anonymous (nobody) Summary: test_subprocess 2.4b1 fails on FreeBSD 5.2 Initial Comment: Platform: FreeBSD 5.2.1-RELEASE Python version: 2.4b1 After untarring I just did 'make' and then 'make test'. make test passes all except following (output of make test): ---- test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/usr/home/shalabh/down/Python-2.4b1/Lib/test/test_subprocess.py", line 432, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '5' != '3' ---- Also, make test ends with: ---- *** Error code 1 Stop in /usr/home/shalabh/down/Python-2.4b1. ---- Don't know if above is expected when a test fails. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 From noreply at sourceforge.net Sun Oct 17 21:51:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 21:51:50 2004 Subject: [ python-Bugs-1048816 ] C-k does not work for curses.textpad.Textbox Message-ID: Bugs item #1048816, was opened at 2004-10-17 21:51 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048816&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Stefan Heimann (stefanheimann) Assigned to: Nobody/Anonymous (nobody) Summary: C-k does not work for curses.textpad.Textbox Initial Comment: The bug can be reproduced by starting the textpad module directly (by executing 'python textpad.py' inside the curses directory). The you should type: 1 2 3 C-a C-k You would expected an empty textbox (because you first enter the string '123', the go the the beginning of the line and then delete until the end of the line). But the textbox contains the text '12'. My Python version is 2.3.3, a patch is attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048816&group_id=5470 From noreply at sourceforge.net Sun Oct 17 21:57:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 21:57:14 2004 Subject: [ python-Bugs-1048756 ] Doc typo Message-ID: Bugs item #1048756, was opened at 2004-10-17 13:12 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048756&group_id=5470 Category: Documentation Group: None >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: Scott David Daniels (scott_daniels) >Assigned to: Neal Norwitz (nnorwitz) Summary: Doc typo Initial Comment: Python 2.4b1 (but also 2.3 and ...) In the documentation for pdb, section 9.1 Debugger commands describes the 'pp' command as: pp expression Like the "p" command, except the value of the exception is pretty-printed using the pprint module. This should be: pp expression Like the "p" command, except the value of the _expression_ is pretty-printed using the pprint module. ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-17 15:57 Message: Logged In: YES user_id=33168 Thanks, checked in as: * Doc/lib/libpdb.tex 1.41 & 1.37.8.2 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048756&group_id=5470 From noreply at sourceforge.net Sun Oct 17 21:58:26 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 21:58:29 2004 Subject: [ python-Bugs-1048820 ] Only "Overwrite" mode possible with curses.textpad.Textbox Message-ID: Bugs item #1048820, was opened at 2004-10-17 21:58 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048820&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Stefan Heimann (stefanheimann) Assigned to: Nobody/Anonymous (nobody) Summary: Only "Overwrite" mode possible with curses.textpad.Textbox Initial Comment: When entering a character in a curses.textpad.Textbox, the character under the cursor is deleted instead of being shifted when position. I attached a patch which implements the "Insert" mode. Maybe you should make it customizable wheter the Textbox should use the old "Overwrite" or the new "Insert" mode. I'm using python 2.3.3 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048820&group_id=5470 From noreply at sourceforge.net Sun Oct 17 22:54:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 22:54:22 2004 Subject: [ python-Bugs-881812 ] 2.4a0 build fails in Modules/signalmodule.c Message-ID: Bugs item #881812, was opened at 2004-01-21 18:16 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=881812&group_id=5470 Category: Build Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Dyck (dcd) Assigned to: Nobody/Anonymous (nobody) Summary: 2.4a0 build fails in Modules/signalmodule.c Initial Comment: PYTHON_VERSION, 2.4a0 (Jan 21 CVS) gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -c ./Modules/signalmodule.c -o Modules/signalmodule.o ./Modules/signalmodule.c: In function `signal_handler': ./Modules/signalmodule.c:141: warning: implicit declaration of function `siginterrupt' ./Modules/signalmodule.c: In function `initsignal': ./Modules/signalmodule.c:550: `_NSIG' undeclared (first use in this function) ./Modules/signalmodule.c:550: (Each undeclared identifier is reported only once ./Modules/signalmodule.c:550: for each function it appears in.) make: *** [Modules/signalmodule.o] Error 1 (At one time my libc5 linux system had similar troubles building perl, but perl's Configure get's it right the hard way) ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-17 16:54 Message: Logged In: YES user_id=33168 Is this still a problem w/ 2.4b1? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=881812&group_id=5470 From noreply at sourceforge.net Sun Oct 17 23:06:08 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 23:06:13 2004 Subject: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 Message-ID: Bugs item #1048808, was opened at 2004-10-17 12:24 Message generated for change (Comment added) made by shalabh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Shalabh Chaturvedi (shalabh) Assigned to: Nobody/Anonymous (nobody) Summary: test_subprocess 2.4b1 fails on FreeBSD 5.2 Initial Comment: Platform: FreeBSD 5.2.1-RELEASE Python version: 2.4b1 After untarring I just did 'make' and then 'make test'. make test passes all except following (output of make test): ---- test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/usr/home/shalabh/down/Python-2.4b1/Lib/test/test_subprocess.py", line 432, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '5' != '3' ---- Also, make test ends with: ---- *** Error code 1 Stop in /usr/home/shalabh/down/Python-2.4b1. ---- Don't know if above is expected when a test fails. ---------------------------------------------------------------------- >Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-17 14:06 Message: Logged In: YES user_id=30293 Here's some more info. On my FreeBSD box, for both 2.3.4 and 2.4b1: >>> import os >>> os.dup(0) 5 But on my Win XP box for 2.3.3: >>> import os >>> os.dup(0) 3 I am guessing this may be relevant. Someone more knowledgable can comment. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 From noreply at sourceforge.net Sun Oct 17 23:39:02 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 23:39:04 2004 Subject: [ python-Bugs-1048864 ] unexpected floating point exponent behaviour Message-ID: Bugs item #1048864, was opened at 2004-10-17 23:39 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048864&group_id=5470 Category: Parser/Compiler Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Ruwen Hahn (palana) Assigned to: Nobody/Anonymous (nobody) Summary: unexpected floating point exponent behaviour Initial Comment: There's a problem with exponents and numeric literals: 017e4 should be about 150k, but python gives 170k. Same with hexints, 0x12e4 should be 180k, but python gives 4836. I think that python should either throw an error or do it right. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048864&group_id=5470 From noreply at sourceforge.net Sun Oct 17 23:40:57 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 23:40:59 2004 Subject: [ python-Bugs-1048865 ] codecs.StreamReader.readlines() broken Message-ID: Bugs item #1048865, was opened at 2004-10-17 23:40 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048865&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Thomas Waldmann (thomaswaldmann) Assigned to: Nobody/Anonymous (nobody) Summary: codecs.StreamReader.readlines() broken Initial Comment: This is a quite obvious one: $ diff -u codecs.py-orig codecs.py --- codecs.py-orig 2004-10-17 23:30:50.000000000 +0200 +++ codecs.py 2004-10-17 23:31:57.000000000 +0200 @@ -334,7 +334,7 @@ """ data = self.read() - return self.splitlines(keepends) + return data.splitlines(keepends) def reset(self): I am using Python 2.4 beta 1. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048865&group_id=5470 From noreply at sourceforge.net Sun Oct 17 23:48:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 23:48:14 2004 Subject: [ python-Bugs-1048864 ] unexpected floating point exponent behaviour Message-ID: Bugs item #1048864, was opened at 2004-10-17 17:39 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048864&group_id=5470 Category: Parser/Compiler >Group: Not a Bug >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Ruwen Hahn (palana) Assigned to: Nobody/Anonymous (nobody) Summary: unexpected floating point exponent behaviour Initial Comment: There's a problem with exponents and numeric literals: 017e4 should be about 150k, but python gives 170k. Same with hexints, 0x12e4 should be 180k, but python gives 4836. I think that python should either throw an error or do it right. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-17 17:48 Message: Logged In: YES user_id=31435 Closing as invalid. Octal and hex notations apply only to integer literals (see the reference manual). 0x12e4 is an integer hex literal ('e' is a hex digit! so are a, b, c, d, and f), and 4836 is its correct decimal equivalent. 017e4 is a decimal float literal, denoting the same value as 17e4 (i.e., the presence or absence of leading zeroes has no effect on the meaning of decimal float literals -- and all float literals in Python are decimal float literals). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048864&group_id=5470 From noreply at sourceforge.net Sun Oct 17 23:55:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 17 23:55:20 2004 Subject: [ python-Bugs-1048870 ] call arg of lambda not updating Message-ID: Bugs item #1048870, was opened at 2004-10-17 14:55 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Kevin Quick (kquick) Assigned to: Nobody/Anonymous (nobody) Summary: call arg of lambda not updating Initial Comment: The attachment contains a script intended to do performance testing on various types of dictionary keys. Lambda functions are used to "generate" a sequence of keys with bound arguments to arrays that are modified during execution. Script fails with pop() on empty list error, but examination shows that the lambda function from the *previous* call is being used for the current call; if proper lambda was used, list would not be empty and operation should succeed. Script is large and has been "grown", so it's not very elegant, but either I'm missing somewhere where I'm making a stupid mistake or else the lambda argument to the function call isn't being set properly. Don't be put off by the size of the script; the problem is easily demonstrated and fairly clearly understood in just a few minutes of looking at the generated stack trace relative to the code. As it exists, script will fail immediately due to the bug. If it worked as expected, at least 3-4 lines of output should be generated before encountering new code that hasn't yet been fully debugged. New code could be removed, but it was left in because some of the complexity of core code is due to this latter code, and it doesn't obstruct the primary bug, so it can be ignored unless of interest. python Python 2.3.3 (#1, May 27 2004, 20:44:16) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Thanks for looking at this! ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 From noreply at sourceforge.net Mon Oct 18 00:12:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 00:12:54 2004 Subject: [ python-Bugs-1048878 ] shutil.copytree copies stat of files, but not of dirs Message-ID: Bugs item #1048878, was opened at 2004-10-18 00:12 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048878&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Thomas Waldmann (thomaswaldmann) Assigned to: Nobody/Anonymous (nobody) Summary: shutil.copytree copies stat of files, but not of dirs Initial Comment: Caring for stat of files, but modifying stat of dirs is maybe not what's expected from that function. I discovered that when implementing a new function in MoinMoin when it changed directory modes from -rw-rw---- to -rw-------. It is easy to fix: $ diff -u shutil.py shutil.py-tw --- shutil.py 2004-10-17 23:19:25.000000000 +0200 +++ shutil.py-tw 2004-10-18 00:01:57.000000000 +0200 @@ -109,6 +109,7 @@ """ names = os.listdir(src) os.mkdir(dst) + copystat(src, dst) errors = [] for name in names: srcname = os.path.join(src, name) Maybe it is even better to do that copystat after the for loop (and thus, after the recursive calls modifying the timestamp of the directory) and before the if statement, so it may even conserve the dir timestamp (I didn't test that). It happens in 2.3.4 as well as in 2.4 beta 1. Strictly taken, it is not a bug, as the docs say nothing about what it does to directories. But well, maybe just the docs need to be changed as well. :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048878&group_id=5470 From noreply at sourceforge.net Mon Oct 18 01:00:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 01:00:34 2004 Subject: [ python-Bugs-1048900 ] Registry not updated by 2.4b1 installer Message-ID: Bugs item #1048900, was opened at 2004-10-17 23:00 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048900&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: B Sizer (kylotan) Assigned to: Nobody/Anonymous (nobody) Summary: Registry not updated by 2.4b1 installer Initial Comment: Running the python-2.4b1.msi installer on Win98 SE appeared to proceeded as expected. However, the registry appears not to have been updated and any add-on Python library installers are still asking to be put in my old Python 2.3 directory, effectively preventing me from using those libraries with 2.4b, or any scripts that rely on them. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048900&group_id=5470 From noreply at sourceforge.net Mon Oct 18 01:04:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 01:04:54 2004 Subject: [ python-Bugs-1048870 ] call arg of lambda not updating Message-ID: Bugs item #1048870, was opened at 2004-10-17 16:55 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Kevin Quick (kquick) Assigned to: Nobody/Anonymous (nobody) Summary: call arg of lambda not updating Initial Comment: The attachment contains a script intended to do performance testing on various types of dictionary keys. Lambda functions are used to "generate" a sequence of keys with bound arguments to arrays that are modified during execution. Script fails with pop() on empty list error, but examination shows that the lambda function from the *previous* call is being used for the current call; if proper lambda was used, list would not be empty and operation should succeed. Script is large and has been "grown", so it's not very elegant, but either I'm missing somewhere where I'm making a stupid mistake or else the lambda argument to the function call isn't being set properly. Don't be put off by the size of the script; the problem is easily demonstrated and fairly clearly understood in just a few minutes of looking at the generated stack trace relative to the code. As it exists, script will fail immediately due to the bug. If it worked as expected, at least 3-4 lines of output should be generated before encountering new code that hasn't yet been fully debugged. New code could be removed, but it was left in because some of the complexity of core code is due to this latter code, and it doesn't obstruct the primary bug, so it can be ignored unless of interest. python Python 2.3.3 (#1, May 27 2004, 20:44:16) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Thanks for looking at this! ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 18:04 Message: Logged In: YES user_id=80475 Please resubmit after boiling this down to the simplest possible thing that doesn't work according to your expectations. Submitting a whole application and playing the "find the bug" game is not good use of developer time. Often, when a submitter starts taking away the unnecessary pieces and isolates the issue, they find that there was a bug in their own code or a bug in their own understanding of how the language works. When it comes to lambdas, the most frequent misunderstanding is knowing that default arguments bind only once as the time of the lambda declaration and that free variables bind whenever the resulting function is invoked. Here's an example of code that looks similar but behaves differently: def once(x): return x def twice(x): return 2*x def thrice(x): return 3*x funcs = [once, twice, thrice] flim = [lambda x:funcs[0](x), lambda x:funcs[1](x), lambda x:funcs[2](x)] flam = [lambda x:f(x) for f in funcs] print flim[0](1), flim[1](1), flim[2](1) print flam[0](1), flam[1](1), flam[2](1) The difference in output is because nested scopes bind names, while argument binding binds values. Hope this helps. If not, please resubmit with the smallest case that demonstrates the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 From noreply at sourceforge.net Mon Oct 18 01:05:57 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 01:06:03 2004 Subject: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 Message-ID: Bugs item #1048808, was opened at 2004-10-17 14:24 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None >Priority: 7 Submitted By: Shalabh Chaturvedi (shalabh) Assigned to: Nobody/Anonymous (nobody) Summary: test_subprocess 2.4b1 fails on FreeBSD 5.2 Initial Comment: Platform: FreeBSD 5.2.1-RELEASE Python version: 2.4b1 After untarring I just did 'make' and then 'make test'. make test passes all except following (output of make test): ---- test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/usr/home/shalabh/down/Python-2.4b1/Lib/test/test_subprocess.py", line 432, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '5' != '3' ---- Also, make test ends with: ---- *** Error code 1 Stop in /usr/home/shalabh/down/Python-2.4b1. ---- Don't know if above is expected when a test fails. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-17 16:06 Message: Logged In: YES user_id=30293 Here's some more info. On my FreeBSD box, for both 2.3.4 and 2.4b1: >>> import os >>> os.dup(0) 5 But on my Win XP box for 2.3.3: >>> import os >>> os.dup(0) 3 I am guessing this may be relevant. Someone more knowledgable can comment. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 From noreply at sourceforge.net Mon Oct 18 01:10:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 01:11:03 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 07:04 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-10-17 16:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Mon Oct 18 01:52:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 01:52:38 2004 Subject: [ python-Bugs-1048865 ] codecs.StreamReader.readlines() broken Message-ID: Bugs item #1048865, was opened at 2004-10-18 06:40 Message generated for change (Comment added) made by perky You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048865&group_id=5470 Category: Python Library Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Thomas Waldmann (thomaswaldmann) >Assigned to: Hye-Shik Chang (perky) Summary: codecs.StreamReader.readlines() broken Initial Comment: This is a quite obvious one: $ diff -u codecs.py-orig codecs.py --- codecs.py-orig 2004-10-17 23:30:50.000000000 +0200 +++ codecs.py 2004-10-17 23:31:57.000000000 +0200 @@ -334,7 +334,7 @@ """ data = self.read() - return self.splitlines(keepends) + return data.splitlines(keepends) def reset(self): I am using Python 2.4 beta 1. ---------------------------------------------------------------------- >Comment By: Hye-Shik Chang (perky) Date: 2004-10-18 08:52 Message: Logged In: YES user_id=55188 Thanks for the reporting. Fixed in CVS: Lib/codecs.py 1.35 Lib/test/test_codecs.py 1.14 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048865&group_id=5470 From noreply at sourceforge.net Mon Oct 18 03:24:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 03:24:15 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-17 21:24 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Nobody/Anonymous (nobody) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Mon Oct 18 03:24:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 03:24:39 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-17 21:24 Message generated for change (Settings changed) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) >Assigned to: Barry A. Warsaw (bwarsaw) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Mon Oct 18 03:40:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 03:40:53 2004 Subject: [ python-Bugs-1048498 ] SystemError with deque Message-ID: Bugs item #1048498, was opened at 2004-10-17 08:05 Message generated for change (Comment added) made by quiver You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048498&group_id=5470 Category: Python Library Group: Python 2.4 Status: Closed Resolution: Out of Date Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Raymond Hettinger (rhettinger) Summary: SystemError with deque Initial Comment: If I run the following script, it causes a SystemError. # start of program from collections import deque d1 = deque() d2 = deque() class A: def __eq__(self, other): d1.pop() d1.appendleft(1) return 1 d1.extend((0, 1)) d2.extend((A(), 1)) d1 == d2 # end of program $ python2.4 deque_demo.py Traceback (most recent call last): File "deque_demo.py", line 15, in ? d1 == d2 SystemError: error return without exception set This error happens if - __eq__ returns values that are interpreted as True. - appendleft is used insted of append. Tested on Python 2.4a3(Win) and python2.4 from CVS as of 2004/10/15. I attached this program for convenience. ---------------------------------------------------------------------- >Comment By: George Yoshida (quiver) Date: 2004-10-18 10:40 Message: Logged In: YES user_id=671362 I've verified that recent Python raises "RuntimeError : deque mutated during iteration" against my program. Thanks Raymond. Before submitting this, I tested on Cygwin and it looks like my Cygwin had a problem with libinstall. That is why not all the libraries were installed properly in the process of "make install". Sorry about that. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 10:03 Message: Logged In: YES user_id=80475 I think that has been fixed. Please reverify with Py2.4b1 when it comes out. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048498&group_id=5470 From noreply at sourceforge.net Mon Oct 18 03:56:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 03:56:58 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 07:04 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-10-17 18:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-17 16:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Mon Oct 18 04:37:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 04:37:24 2004 Subject: [ python-Bugs-854823 ] Python-2.3.3c1, Solaris 2.7: socketmodule does not compile Message-ID: Bugs item #854823, was opened at 2003-12-05 11:01 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854823&group_id=5470 Category: Build Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Falko Sauermann (fsm2761) Assigned to: Nobody/Anonymous (nobody) Summary: Python-2.3.3c1, Solaris 2.7: socketmodule does not compile Initial Comment: Compiling Python-2.3.3c1 on Solaris 2.7 with gcc 2.95.3 gave me two errors: socketmodule.c:2975: `AF_INET6' undeclared (first use in this function) socketmodule.c:3019: `INET_ADDRSTRLEN' undeclared (first use in this function) The first problem was already reported for Solaris 2.7 with SunPro cc (see item 814613) but is also true for Solaris 2.7 with gcc. The real problem is that AF_INET6 is used when ENABLE_IPV6 is undefined. I believe this must fail for any system where IPV6 is not avalilable. The second problem was already reported for Irix (see item 818490) but is also true for Solaris 2.7. The solution for Irix is also valid for Solaris. If I change socketmodule.c line 204 to: #if (defined(__sgi) || defined(sun)) && !defined(INET_ADDRSTRLEN) the error is gone. ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-17 22:37 Message: Logged In: YES user_id=33168 Does this problem still exist in 2.3.4 or 2.4b1? Can this be closed? ---------------------------------------------------------------------- Comment By: Stephan A. Terre (sfiedler) Date: 2004-02-26 12:23 Message: Logged In: YES user_id=246063 I think it's worth noting that the preprocessor directive near line 2975 is #ifndef , whereas all other preprocessor directives in the file that refer to the IPV6 symbol are #ifdef . In other words, it looks like just that one directive is a typo. Changing #ifndef to #ifdef fixes it for me. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854823&group_id=5470 From noreply at sourceforge.net Mon Oct 18 05:32:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 05:32:16 2004 Subject: [ python-Bugs-1048498 ] SystemError with deque Message-ID: Bugs item #1048498, was opened at 2004-10-16 18:05 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048498&group_id=5470 Category: Python Library Group: Python 2.4 Status: Closed Resolution: Out of Date Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Raymond Hettinger (rhettinger) Summary: SystemError with deque Initial Comment: If I run the following script, it causes a SystemError. # start of program from collections import deque d1 = deque() d2 = deque() class A: def __eq__(self, other): d1.pop() d1.appendleft(1) return 1 d1.extend((0, 1)) d2.extend((A(), 1)) d1 == d2 # end of program $ python2.4 deque_demo.py Traceback (most recent call last): File "deque_demo.py", line 15, in ? d1 == d2 SystemError: error return without exception set This error happens if - __eq__ returns values that are interpreted as True. - appendleft is used insted of append. Tested on Python 2.4a3(Win) and python2.4 from CVS as of 2004/10/15. I attached this program for convenience. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 22:32 Message: Logged In: YES user_id=80475 I appreciate the efforts to break the new tools and review all the docs. Keep up the good work. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-17 20:40 Message: Logged In: YES user_id=671362 I've verified that recent Python raises "RuntimeError : deque mutated during iteration" against my program. Thanks Raymond. Before submitting this, I tested on Cygwin and it looks like my Cygwin had a problem with libinstall. That is why not all the libraries were installed properly in the process of "make install". Sorry about that. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-16 20:03 Message: Logged In: YES user_id=80475 I think that has been fixed. Please reverify with Py2.4b1 when it comes out. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048498&group_id=5470 From noreply at sourceforge.net Mon Oct 18 05:50:02 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 05:50:15 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-13 03:04 Message generated for change (Comment added) made by anadelonbrin You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Closed Resolution: Fixed Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-18 16:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 14:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 12:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 14:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 14:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 14:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Mon Oct 18 06:02:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 06:02:53 2004 Subject: [ python-Bugs-1048870 ] call arg of lambda not updating Message-ID: Bugs item #1048870, was opened at 2004-10-17 14:55 Message generated for change (Comment added) made by kquick You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Kevin Quick (kquick) Assigned to: Nobody/Anonymous (nobody) Summary: call arg of lambda not updating Initial Comment: The attachment contains a script intended to do performance testing on various types of dictionary keys. Lambda functions are used to "generate" a sequence of keys with bound arguments to arrays that are modified during execution. Script fails with pop() on empty list error, but examination shows that the lambda function from the *previous* call is being used for the current call; if proper lambda was used, list would not be empty and operation should succeed. Script is large and has been "grown", so it's not very elegant, but either I'm missing somewhere where I'm making a stupid mistake or else the lambda argument to the function call isn't being set properly. Don't be put off by the size of the script; the problem is easily demonstrated and fairly clearly understood in just a few minutes of looking at the generated stack trace relative to the code. As it exists, script will fail immediately due to the bug. If it worked as expected, at least 3-4 lines of output should be generated before encountering new code that hasn't yet been fully debugged. New code could be removed, but it was left in because some of the complexity of core code is due to this latter code, and it doesn't obstruct the primary bug, so it can be ignored unless of interest. python Python 2.3.3 (#1, May 27 2004, 20:44:16) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Thanks for looking at this! ---------------------------------------------------------------------- >Comment By: Kevin Quick (kquick) Date: 2004-10-17 21:02 Message: Logged In: YES user_id=6133 OK, example script minimized. Lambda arguments are not the problem; thanks for the pointer, but I'm not having trouble with those. I have determined that proper lambda is being used, but not indicated as shown by attached script, which forces an exception to occur in the lambda function but backtrace shows wrong lambda function. Reducing 2nd argument of 2nd call to seq_func1 to 'num_args' instead of 'num_args+2' will not encounter the error, but the nature of the implementation requires that it is using the correct lambda, so it's the backtrace attribution that is apparently wrong. Mea culpa: my original script version did request wrong 'num_args' causing actual bug, but incorrect backtrace led me down an alternate path... I suspected that I had some incorrect code, but I couldn't resolve that against the error report. Still can't. :) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 16:04 Message: Logged In: YES user_id=80475 Please resubmit after boiling this down to the simplest possible thing that doesn't work according to your expectations. Submitting a whole application and playing the "find the bug" game is not good use of developer time. Often, when a submitter starts taking away the unnecessary pieces and isolates the issue, they find that there was a bug in their own code or a bug in their own understanding of how the language works. When it comes to lambdas, the most frequent misunderstanding is knowing that default arguments bind only once as the time of the lambda declaration and that free variables bind whenever the resulting function is invoked. Here's an example of code that looks similar but behaves differently: def once(x): return x def twice(x): return 2*x def thrice(x): return 3*x funcs = [once, twice, thrice] flim = [lambda x:funcs[0](x), lambda x:funcs[1](x), lambda x:funcs[2](x)] flam = [lambda x:f(x) for f in funcs] print flim[0](1), flim[1](1), flim[2](1) print flam[0](1), flam[1](1), flam[2](1) The difference in output is because nested scopes bind names, while argument binding binds values. Hope this helps. If not, please resubmit with the smallest case that demonstrates the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 From noreply at sourceforge.net Mon Oct 18 06:34:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 06:34:51 2004 Subject: [ python-Bugs-1048870 ] call arg of lambda not updating Message-ID: Bugs item #1048870, was opened at 2004-10-17 16:55 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Kevin Quick (kquick) Assigned to: Nobody/Anonymous (nobody) Summary: call arg of lambda not updating Initial Comment: The attachment contains a script intended to do performance testing on various types of dictionary keys. Lambda functions are used to "generate" a sequence of keys with bound arguments to arrays that are modified during execution. Script fails with pop() on empty list error, but examination shows that the lambda function from the *previous* call is being used for the current call; if proper lambda was used, list would not be empty and operation should succeed. Script is large and has been "grown", so it's not very elegant, but either I'm missing somewhere where I'm making a stupid mistake or else the lambda argument to the function call isn't being set properly. Don't be put off by the size of the script; the problem is easily demonstrated and fairly clearly understood in just a few minutes of looking at the generated stack trace relative to the code. As it exists, script will fail immediately due to the bug. If it worked as expected, at least 3-4 lines of output should be generated before encountering new code that hasn't yet been fully debugged. New code could be removed, but it was left in because some of the complexity of core code is due to this latter code, and it doesn't obstruct the primary bug, so it can be ignored unless of interest. python Python 2.3.3 (#1, May 27 2004, 20:44:16) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Thanks for looking at this! ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 23:34 Message: Logged In: YES user_id=80475 Closing as invalid -- this is a bug in the OP's script, not in Python itself. The error is obious when the code is simplified further by in-lining the function. Also, it helps to explicitly set the initial values of variables between the two sections: #!/usr/bin/env python num_elem = 100 key_src = range(num_elem+1) keynums1 = key_src[:] assert key_src[-1] == 100 for _ in xrange(num_elem): keynums1[key_src.pop()] keynums2 = [0] for _ in xrange(102): key_src[keynums2.pop()] I think your coding error was in assuming that keynum2 held a different value before the second half was run. If so, that could have been found by using pdb or by inserting print statements to reveal what is going on. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 23:02 Message: Logged In: YES user_id=6133 OK, example script minimized. Lambda arguments are not the problem; thanks for the pointer, but I'm not having trouble with those. I have determined that proper lambda is being used, but not indicated as shown by attached script, which forces an exception to occur in the lambda function but backtrace shows wrong lambda function. Reducing 2nd argument of 2nd call to seq_func1 to 'num_args' instead of 'num_args+2' will not encounter the error, but the nature of the implementation requires that it is using the correct lambda, so it's the backtrace attribution that is apparently wrong. Mea culpa: my original script version did request wrong 'num_args' causing actual bug, but incorrect backtrace led me down an alternate path... I suspected that I had some incorrect code, but I couldn't resolve that against the error report. Still can't. :) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 18:04 Message: Logged In: YES user_id=80475 Please resubmit after boiling this down to the simplest possible thing that doesn't work according to your expectations. Submitting a whole application and playing the "find the bug" game is not good use of developer time. Often, when a submitter starts taking away the unnecessary pieces and isolates the issue, they find that there was a bug in their own code or a bug in their own understanding of how the language works. When it comes to lambdas, the most frequent misunderstanding is knowing that default arguments bind only once as the time of the lambda declaration and that free variables bind whenever the resulting function is invoked. Here's an example of code that looks similar but behaves differently: def once(x): return x def twice(x): return 2*x def thrice(x): return 3*x funcs = [once, twice, thrice] flim = [lambda x:funcs[0](x), lambda x:funcs[1](x), lambda x:funcs[2](x)] flam = [lambda x:f(x) for f in funcs] print flim[0](1), flim[1](1), flim[2](1) print flam[0](1), flam[1](1), flam[2](1) The difference in output is because nested scopes bind names, while argument binding binds values. Hope this helps. If not, please resubmit with the smallest case that demonstrates the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 From noreply at sourceforge.net Mon Oct 18 06:45:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 06:45:38 2004 Subject: [ python-Bugs-1049003 ] Install-time module compilation fails with 2.4b1 Message-ID: Bugs item #1049003, was opened at 2004-10-18 17:45 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049003&group_id=5470 Category: Installation Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Tony Meyer (anadelonbrin) Assigned to: Nobody/Anonymous (nobody) Summary: Install-time module compilation fails with 2.4b1 Initial Comment: If I tick the "compile modules" box in the advanced options of the 2.4b1 msi installer, it dies part way through this (it scrolls too fast for me to see exactly where, though). The installer then gives a dialog with: """ There is a problem with this Windows Installer package. A program run as part of the setup did not finish as expected. Contact your support personnel or package vendor. """ And goes to a "install did not work" final box, rather than the regular 'all good' one. This happens with Win2K Pro and WinXP Pro SP1. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049003&group_id=5470 From noreply at sourceforge.net Mon Oct 18 06:48:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 06:48:23 2004 Subject: [ python-Bugs-1048870 ] call arg of lambda not updating Message-ID: Bugs item #1048870, was opened at 2004-10-17 14:55 Message generated for change (Comment added) made by kquick You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 >Status: Open Resolution: Invalid Priority: 5 Submitted By: Kevin Quick (kquick) Assigned to: Nobody/Anonymous (nobody) Summary: call arg of lambda not updating Initial Comment: The attachment contains a script intended to do performance testing on various types of dictionary keys. Lambda functions are used to "generate" a sequence of keys with bound arguments to arrays that are modified during execution. Script fails with pop() on empty list error, but examination shows that the lambda function from the *previous* call is being used for the current call; if proper lambda was used, list would not be empty and operation should succeed. Script is large and has been "grown", so it's not very elegant, but either I'm missing somewhere where I'm making a stupid mistake or else the lambda argument to the function call isn't being set properly. Don't be put off by the size of the script; the problem is easily demonstrated and fairly clearly understood in just a few minutes of looking at the generated stack trace relative to the code. As it exists, script will fail immediately due to the bug. If it worked as expected, at least 3-4 lines of output should be generated before encountering new code that hasn't yet been fully debugged. New code could be removed, but it was left in because some of the complexity of core code is due to this latter code, and it doesn't obstruct the primary bug, so it can be ignored unless of interest. python Python 2.3.3 (#1, May 27 2004, 20:44:16) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Thanks for looking at this! ---------------------------------------------------------------------- >Comment By: Kevin Quick (kquick) Date: 2004-10-17 21:48 Message: Logged In: YES user_id=6133 Reopened: Problem not addressed. Please *read* my previous comment. Yes, the script has a bug in it. That's not what's being reported here. The bug is in the backtrace produced by the Python interpreter. See below. Line number reported for lambda function raising exception is line 14 below. Actual line number for lambda generating exception is line 18. $ python lambda_bug.py Traceback (most recent call last): File "lambda_bug.py", line 23, in ? seqtest() File "lambda_bug.py", line 19, in seqtest seq_func1(nextkey2, num_elem+2) File "lambda_bug.py", line 5, in seq_func1 key_genfunc() File "lambda_bug.py", line 14, in nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()] IndexError: pop from empty list ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 21:34 Message: Logged In: YES user_id=80475 Closing as invalid -- this is a bug in the OP's script, not in Python itself. The error is obious when the code is simplified further by in-lining the function. Also, it helps to explicitly set the initial values of variables between the two sections: #!/usr/bin/env python num_elem = 100 key_src = range(num_elem+1) keynums1 = key_src[:] assert key_src[-1] == 100 for _ in xrange(num_elem): keynums1[key_src.pop()] keynums2 = [0] for _ in xrange(102): key_src[keynums2.pop()] I think your coding error was in assuming that keynum2 held a different value before the second half was run. If so, that could have been found by using pdb or by inserting print statements to reveal what is going on. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 21:02 Message: Logged In: YES user_id=6133 OK, example script minimized. Lambda arguments are not the problem; thanks for the pointer, but I'm not having trouble with those. I have determined that proper lambda is being used, but not indicated as shown by attached script, which forces an exception to occur in the lambda function but backtrace shows wrong lambda function. Reducing 2nd argument of 2nd call to seq_func1 to 'num_args' instead of 'num_args+2' will not encounter the error, but the nature of the implementation requires that it is using the correct lambda, so it's the backtrace attribution that is apparently wrong. Mea culpa: my original script version did request wrong 'num_args' causing actual bug, but incorrect backtrace led me down an alternate path... I suspected that I had some incorrect code, but I couldn't resolve that against the error report. Still can't. :) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 16:04 Message: Logged In: YES user_id=80475 Please resubmit after boiling this down to the simplest possible thing that doesn't work according to your expectations. Submitting a whole application and playing the "find the bug" game is not good use of developer time. Often, when a submitter starts taking away the unnecessary pieces and isolates the issue, they find that there was a bug in their own code or a bug in their own understanding of how the language works. When it comes to lambdas, the most frequent misunderstanding is knowing that default arguments bind only once as the time of the lambda declaration and that free variables bind whenever the resulting function is invoked. Here's an example of code that looks similar but behaves differently: def once(x): return x def twice(x): return 2*x def thrice(x): return 3*x funcs = [once, twice, thrice] flim = [lambda x:funcs[0](x), lambda x:funcs[1](x), lambda x:funcs[2](x)] flam = [lambda x:f(x) for f in funcs] print flim[0](1), flim[1](1), flim[2](1) print flam[0](1), flam[1](1), flam[2](1) The difference in output is because nested scopes bind names, while argument binding binds values. Hope this helps. If not, please resubmit with the smallest case that demonstrates the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 From noreply at sourceforge.net Mon Oct 18 07:41:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 07:41:33 2004 Subject: [ python-Bugs-1048870 ] call arg of lambda not updating Message-ID: Bugs item #1048870, was opened at 2004-10-17 16:55 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 >Category: Parser/Compiler Group: Python 2.3 Status: Open >Resolution: None >Priority: 6 Submitted By: Kevin Quick (kquick) >Assigned to: Jeremy Hylton (jhylton) Summary: call arg of lambda not updating Initial Comment: The attachment contains a script intended to do performance testing on various types of dictionary keys. Lambda functions are used to "generate" a sequence of keys with bound arguments to arrays that are modified during execution. Script fails with pop() on empty list error, but examination shows that the lambda function from the *previous* call is being used for the current call; if proper lambda was used, list would not be empty and operation should succeed. Script is large and has been "grown", so it's not very elegant, but either I'm missing somewhere where I'm making a stupid mistake or else the lambda argument to the function call isn't being set properly. Don't be put off by the size of the script; the problem is easily demonstrated and fairly clearly understood in just a few minutes of looking at the generated stack trace relative to the code. As it exists, script will fail immediately due to the bug. If it worked as expected, at least 3-4 lines of output should be generated before encountering new code that hasn't yet been fully debugged. New code could be removed, but it was left in because some of the complexity of core code is due to this latter code, and it doesn't obstruct the primary bug, so it can be ignored unless of interest. python Python 2.3.3 (#1, May 27 2004, 20:44:16) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Thanks for looking at this! ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 00:41 Message: Logged In: YES user_id=80475 Confirmed. Also, the behavior persists into Py2.4b1. The error disappears when the lambda is replaced by an equivalent one line def statement: def nextkey2(N=keynums2,K=key_src): return K[N.pop()] The opcodes are the same for both; however, the co_firstlineno attribute is incorrect for the lambda version. The pure python compiler module does not make the same error. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 23:48 Message: Logged In: YES user_id=6133 Reopened: Problem not addressed. Please *read* my previous comment. Yes, the script has a bug in it. That's not what's being reported here. The bug is in the backtrace produced by the Python interpreter. See below. Line number reported for lambda function raising exception is line 14 below. Actual line number for lambda generating exception is line 18. $ python lambda_bug.py Traceback (most recent call last): File "lambda_bug.py", line 23, in ? seqtest() File "lambda_bug.py", line 19, in seqtest seq_func1(nextkey2, num_elem+2) File "lambda_bug.py", line 5, in seq_func1 key_genfunc() File "lambda_bug.py", line 14, in nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()] IndexError: pop from empty list ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 23:34 Message: Logged In: YES user_id=80475 Closing as invalid -- this is a bug in the OP's script, not in Python itself. The error is obious when the code is simplified further by in-lining the function. Also, it helps to explicitly set the initial values of variables between the two sections: #!/usr/bin/env python num_elem = 100 key_src = range(num_elem+1) keynums1 = key_src[:] assert key_src[-1] == 100 for _ in xrange(num_elem): keynums1[key_src.pop()] keynums2 = [0] for _ in xrange(102): key_src[keynums2.pop()] I think your coding error was in assuming that keynum2 held a different value before the second half was run. If so, that could have been found by using pdb or by inserting print statements to reveal what is going on. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 23:02 Message: Logged In: YES user_id=6133 OK, example script minimized. Lambda arguments are not the problem; thanks for the pointer, but I'm not having trouble with those. I have determined that proper lambda is being used, but not indicated as shown by attached script, which forces an exception to occur in the lambda function but backtrace shows wrong lambda function. Reducing 2nd argument of 2nd call to seq_func1 to 'num_args' instead of 'num_args+2' will not encounter the error, but the nature of the implementation requires that it is using the correct lambda, so it's the backtrace attribution that is apparently wrong. Mea culpa: my original script version did request wrong 'num_args' causing actual bug, but incorrect backtrace led me down an alternate path... I suspected that I had some incorrect code, but I couldn't resolve that against the error report. Still can't. :) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 18:04 Message: Logged In: YES user_id=80475 Please resubmit after boiling this down to the simplest possible thing that doesn't work according to your expectations. Submitting a whole application and playing the "find the bug" game is not good use of developer time. Often, when a submitter starts taking away the unnecessary pieces and isolates the issue, they find that there was a bug in their own code or a bug in their own understanding of how the language works. When it comes to lambdas, the most frequent misunderstanding is knowing that default arguments bind only once as the time of the lambda declaration and that free variables bind whenever the resulting function is invoked. Here's an example of code that looks similar but behaves differently: def once(x): return x def twice(x): return 2*x def thrice(x): return 3*x funcs = [once, twice, thrice] flim = [lambda x:funcs[0](x), lambda x:funcs[1](x), lambda x:funcs[2](x)] flam = [lambda x:f(x) for f in funcs] print flim[0](1), flim[1](1), flim[2](1) print flam[0](1), flam[1](1), flam[2](1) The difference in output is because nested scopes bind names, while argument binding binds values. Hope this helps. If not, please resubmit with the smallest case that demonstrates the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 From noreply at sourceforge.net Mon Oct 18 07:44:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 07:44:43 2004 Subject: [ python-Bugs-1021756 ] 2.4a3: unhelpful error message from distutils Message-ID: Bugs item #1021756, was opened at 2004-09-03 20:34 Message generated for change (Comment added) made by mdehoon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1021756&group_id=5470 Category: Distutils Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Fredrik Lundh (effbot) Assigned to: Nobody/Anonymous (nobody) Summary: 2.4a3: unhelpful error message from distutils Initial Comment: C:\somewhere> python Python 2.4a3 (#56, Sep 2 2004, 20:50:21) ... C:\somewhere> python setup.py build running build_ext Traceback (most recent call last): File "setup.py", line 19, in ? ext_modules = [ File "C:\python24\lib\distutils\core.py", line 150, in setup dist.run_commands() File "C:\python24\lib\distutils\dist.py", line 991, in run_commands self.run_command(cmd) File "C:\python24\lib\distutils\dist.py", line 1011, in run_command cmd_obj.run() File "C:\python24\lib\distutils\command\build_ext.py", line 243, in run force=self.force) File "C:\python24\lib\distutils\ccompiler.py", line 1177, in new_compiler return klass (None, dry_run, force) File "C:\python24\lib\distutils\msvccompiler.py", line 206, in __init__ self.__macros = MacroExpander(self.__version) File "C:\python24\lib\distutils\msvccompiler.py", line 112, in __init__ self.load_macros(version) File "C:\python24\lib\distutils\msvccompiler.py", line 128, in load_macros self.set_macro("FrameworkSDKDir", net, "sdkinstallrootv1.1") File "C:\python24\lib\distutils\msvccompiler.py", line 118, in set_macro self.macros["$(%s)" % macro] = d[key] KeyError: 'sdkinstallrootv1.1' I suppose it's trying to tell me something, but I'm not sure what. ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2004-10-18 14:44 Message: Logged In: YES user_id=488897 Note that the same error occurs with python setup.py install (using Python 2.4b1): C:\somewhere> python setup.py build .... File "c:\Python24\lib\distutils\msvccompiler.py", line 118, in set_macro self.macros["$(%s)" % macro] = d[key] KeyError: 'sdkinstallrootv1.1' Using MinGW for the compilation works: C:\somewhere> python setup.py build --compiler=mingw32 .... compiles without errors. However, the install still fails: C:\somewhere> python setup.py install File "c:\Python24\lib\distutils\msvccompiler.py", line 118, in set_macro self.macros["$(%s)" % macro] = d[key] KeyError: 'sdkinstallrootv1.1' ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-14 00:56 Message: Logged In: YES user_id=29957 Yep. Can you produce a small patch for this? I have no access to MSVC 6 or 7 (in fact, my Windows box won't even boot at the moment). ---------------------------------------------------------------------- Comment By: Fredrik Lundh (effbot) Date: 2004-10-14 00:29 Message: Logged In: YES user_id=38376 If I'm using the wrong compiler, distutils should say so, instead of showing me a 10-level KeyError traceback... ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-10-14 00:11 Message: Logged In: YES user_id=29957 It looks (to me) like it's expecting you to have some additional package (Framework SDK?) installed. The code in question was added as part of the MSVC7 support - are you building with MSVC6? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1021756&group_id=5470 From noreply at sourceforge.net Mon Oct 18 08:06:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 08:06:54 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-18 03:24 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) >Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 08:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Mon Oct 18 08:42:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 08:42:25 2004 Subject: [ python-Bugs-1048870 ] call arg of lambda not updating Message-ID: Bugs item #1048870, was opened at 2004-10-17 16:55 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 Category: Parser/Compiler Group: Python 2.3 Status: Open Resolution: None >Priority: 2 Submitted By: Kevin Quick (kquick) Assigned to: Jeremy Hylton (jhylton) Summary: call arg of lambda not updating Initial Comment: The attachment contains a script intended to do performance testing on various types of dictionary keys. Lambda functions are used to "generate" a sequence of keys with bound arguments to arrays that are modified during execution. Script fails with pop() on empty list error, but examination shows that the lambda function from the *previous* call is being used for the current call; if proper lambda was used, list would not be empty and operation should succeed. Script is large and has been "grown", so it's not very elegant, but either I'm missing somewhere where I'm making a stupid mistake or else the lambda argument to the function call isn't being set properly. Don't be put off by the size of the script; the problem is easily demonstrated and fairly clearly understood in just a few minutes of looking at the generated stack trace relative to the code. As it exists, script will fail immediately due to the bug. If it worked as expected, at least 3-4 lines of output should be generated before encountering new code that hasn't yet been fully debugged. New code could be removed, but it was left in because some of the complexity of core code is due to this latter code, and it doesn't obstruct the primary bug, so it can be ignored unless of interest. python Python 2.3.3 (#1, May 27 2004, 20:44:16) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Thanks for looking at this! ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 01:42 Message: Logged In: YES user_id=80475 Under the hood, the compiler is recognizing that the function bodies are identical and it produces just one code object instead of two. Hence, the co_firstline number with always reflect the first time the code is defined. A simple demo shows the id being the same when the bodies are identical: src = """ f1 = lambda x=1: x f2 = lambda x=2: x """ from dis import dis c = compile(src, 'example', 'exec') dis(c) For the OP's code, the effect can be shown by differentiating either lambda body by adding zero to the pop index: nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()+0] I conclude that there is less to the bug than meets the eye and that it may even be considered a feature. Referring to Jeremy for final disposition. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 00:41 Message: Logged In: YES user_id=80475 Confirmed. Also, the behavior persists into Py2.4b1. The error disappears when the lambda is replaced by an equivalent one line def statement: def nextkey2(N=keynums2,K=key_src): return K[N.pop()] The opcodes are the same for both; however, the co_firstlineno attribute is incorrect for the lambda version. The pure python compiler module does not make the same error. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 23:48 Message: Logged In: YES user_id=6133 Reopened: Problem not addressed. Please *read* my previous comment. Yes, the script has a bug in it. That's not what's being reported here. The bug is in the backtrace produced by the Python interpreter. See below. Line number reported for lambda function raising exception is line 14 below. Actual line number for lambda generating exception is line 18. $ python lambda_bug.py Traceback (most recent call last): File "lambda_bug.py", line 23, in ? seqtest() File "lambda_bug.py", line 19, in seqtest seq_func1(nextkey2, num_elem+2) File "lambda_bug.py", line 5, in seq_func1 key_genfunc() File "lambda_bug.py", line 14, in nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()] IndexError: pop from empty list ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 23:34 Message: Logged In: YES user_id=80475 Closing as invalid -- this is a bug in the OP's script, not in Python itself. The error is obious when the code is simplified further by in-lining the function. Also, it helps to explicitly set the initial values of variables between the two sections: #!/usr/bin/env python num_elem = 100 key_src = range(num_elem+1) keynums1 = key_src[:] assert key_src[-1] == 100 for _ in xrange(num_elem): keynums1[key_src.pop()] keynums2 = [0] for _ in xrange(102): key_src[keynums2.pop()] I think your coding error was in assuming that keynum2 held a different value before the second half was run. If so, that could have been found by using pdb or by inserting print statements to reveal what is going on. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 23:02 Message: Logged In: YES user_id=6133 OK, example script minimized. Lambda arguments are not the problem; thanks for the pointer, but I'm not having trouble with those. I have determined that proper lambda is being used, but not indicated as shown by attached script, which forces an exception to occur in the lambda function but backtrace shows wrong lambda function. Reducing 2nd argument of 2nd call to seq_func1 to 'num_args' instead of 'num_args+2' will not encounter the error, but the nature of the implementation requires that it is using the correct lambda, so it's the backtrace attribution that is apparently wrong. Mea culpa: my original script version did request wrong 'num_args' causing actual bug, but incorrect backtrace led me down an alternate path... I suspected that I had some incorrect code, but I couldn't resolve that against the error report. Still can't. :) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 18:04 Message: Logged In: YES user_id=80475 Please resubmit after boiling this down to the simplest possible thing that doesn't work according to your expectations. Submitting a whole application and playing the "find the bug" game is not good use of developer time. Often, when a submitter starts taking away the unnecessary pieces and isolates the issue, they find that there was a bug in their own code or a bug in their own understanding of how the language works. When it comes to lambdas, the most frequent misunderstanding is knowing that default arguments bind only once as the time of the lambda declaration and that free variables bind whenever the resulting function is invoked. Here's an example of code that looks similar but behaves differently: def once(x): return x def twice(x): return 2*x def thrice(x): return 3*x funcs = [once, twice, thrice] flim = [lambda x:funcs[0](x), lambda x:funcs[1](x), lambda x:funcs[2](x)] flam = [lambda x:f(x) for f in funcs] print flim[0](1), flim[1](1), flim[2](1) print flam[0](1), flam[1](1), flam[2](1) The difference in output is because nested scopes bind names, while argument binding binds values. Hope this helps. If not, please resubmit with the smallest case that demonstrates the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 From noreply at sourceforge.net Mon Oct 18 09:37:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 09:37:43 2004 Subject: [ python-Bugs-1048870 ] call arg of lambda not updating Message-ID: Bugs item #1048870, was opened at 2004-10-17 14:55 Message generated for change (Comment added) made by kquick You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 Category: Parser/Compiler Group: Python 2.3 Status: Open Resolution: None Priority: 2 Submitted By: Kevin Quick (kquick) Assigned to: Jeremy Hylton (jhylton) Summary: call arg of lambda not updating Initial Comment: The attachment contains a script intended to do performance testing on various types of dictionary keys. Lambda functions are used to "generate" a sequence of keys with bound arguments to arrays that are modified during execution. Script fails with pop() on empty list error, but examination shows that the lambda function from the *previous* call is being used for the current call; if proper lambda was used, list would not be empty and operation should succeed. Script is large and has been "grown", so it's not very elegant, but either I'm missing somewhere where I'm making a stupid mistake or else the lambda argument to the function call isn't being set properly. Don't be put off by the size of the script; the problem is easily demonstrated and fairly clearly understood in just a few minutes of looking at the generated stack trace relative to the code. As it exists, script will fail immediately due to the bug. If it worked as expected, at least 3-4 lines of output should be generated before encountering new code that hasn't yet been fully debugged. New code could be removed, but it was left in because some of the complexity of core code is due to this latter code, and it doesn't obstruct the primary bug, so it can be ignored unless of interest. python Python 2.3.3 (#1, May 27 2004, 20:44:16) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Thanks for looking at this! ---------------------------------------------------------------------- >Comment By: Kevin Quick (kquick) Date: 2004-10-18 00:37 Message: Logged In: YES user_id=6133 Thanks for confirming this issue. My position is that while minor, this is still a bug and not a feature. It's good to detect identical lambda bodies and optimize for that case, but since argument binding occurs at declaration time and not execution time for lambdas, the bound arguments should properly be considered part of the body; the two lambda functions in the attached script should be disjoint. >From the practical perspective, the python backtrace was misleading in my debug efforts. Internal optimizations should take second place to accuracy of displayed information. func_code.co_code (and possibly others) could still be the same, but func_code.co_firstlineno (& func_code.co_filename) should be uniquely set. Hmmm. In thinking more about this, it's not that argument binding occurs immediately, but moreso that defaults are supplied for arguments. If the backtrace/debugger points me to a lambda function with entirely different default args than the ones that are applicable to the problem, then that would be wrong, IMHO. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 23:42 Message: Logged In: YES user_id=80475 Under the hood, the compiler is recognizing that the function bodies are identical and it produces just one code object instead of two. Hence, the co_firstline number with always reflect the first time the code is defined. A simple demo shows the id being the same when the bodies are identical: src = """ f1 = lambda x=1: x f2 = lambda x=2: x """ from dis import dis c = compile(src, 'example', 'exec') dis(c) For the OP's code, the effect can be shown by differentiating either lambda body by adding zero to the pop index: nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()+0] I conclude that there is less to the bug than meets the eye and that it may even be considered a feature. Referring to Jeremy for final disposition. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 22:41 Message: Logged In: YES user_id=80475 Confirmed. Also, the behavior persists into Py2.4b1. The error disappears when the lambda is replaced by an equivalent one line def statement: def nextkey2(N=keynums2,K=key_src): return K[N.pop()] The opcodes are the same for both; however, the co_firstlineno attribute is incorrect for the lambda version. The pure python compiler module does not make the same error. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 21:48 Message: Logged In: YES user_id=6133 Reopened: Problem not addressed. Please *read* my previous comment. Yes, the script has a bug in it. That's not what's being reported here. The bug is in the backtrace produced by the Python interpreter. See below. Line number reported for lambda function raising exception is line 14 below. Actual line number for lambda generating exception is line 18. $ python lambda_bug.py Traceback (most recent call last): File "lambda_bug.py", line 23, in ? seqtest() File "lambda_bug.py", line 19, in seqtest seq_func1(nextkey2, num_elem+2) File "lambda_bug.py", line 5, in seq_func1 key_genfunc() File "lambda_bug.py", line 14, in nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()] IndexError: pop from empty list ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 21:34 Message: Logged In: YES user_id=80475 Closing as invalid -- this is a bug in the OP's script, not in Python itself. The error is obious when the code is simplified further by in-lining the function. Also, it helps to explicitly set the initial values of variables between the two sections: #!/usr/bin/env python num_elem = 100 key_src = range(num_elem+1) keynums1 = key_src[:] assert key_src[-1] == 100 for _ in xrange(num_elem): keynums1[key_src.pop()] keynums2 = [0] for _ in xrange(102): key_src[keynums2.pop()] I think your coding error was in assuming that keynum2 held a different value before the second half was run. If so, that could have been found by using pdb or by inserting print statements to reveal what is going on. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 21:02 Message: Logged In: YES user_id=6133 OK, example script minimized. Lambda arguments are not the problem; thanks for the pointer, but I'm not having trouble with those. I have determined that proper lambda is being used, but not indicated as shown by attached script, which forces an exception to occur in the lambda function but backtrace shows wrong lambda function. Reducing 2nd argument of 2nd call to seq_func1 to 'num_args' instead of 'num_args+2' will not encounter the error, but the nature of the implementation requires that it is using the correct lambda, so it's the backtrace attribution that is apparently wrong. Mea culpa: my original script version did request wrong 'num_args' causing actual bug, but incorrect backtrace led me down an alternate path... I suspected that I had some incorrect code, but I couldn't resolve that against the error report. Still can't. :) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 16:04 Message: Logged In: YES user_id=80475 Please resubmit after boiling this down to the simplest possible thing that doesn't work according to your expectations. Submitting a whole application and playing the "find the bug" game is not good use of developer time. Often, when a submitter starts taking away the unnecessary pieces and isolates the issue, they find that there was a bug in their own code or a bug in their own understanding of how the language works. When it comes to lambdas, the most frequent misunderstanding is knowing that default arguments bind only once as the time of the lambda declaration and that free variables bind whenever the resulting function is invoked. Here's an example of code that looks similar but behaves differently: def once(x): return x def twice(x): return 2*x def thrice(x): return 3*x funcs = [once, twice, thrice] flim = [lambda x:funcs[0](x), lambda x:funcs[1](x), lambda x:funcs[2](x)] flam = [lambda x:f(x) for f in funcs] print flim[0](1), flim[1](1), flim[2](1) print flam[0](1), flam[1](1), flam[2](1) The difference in output is because nested scopes bind names, while argument binding binds values. Hope this helps. If not, please resubmit with the smallest case that demonstrates the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 From noreply at sourceforge.net Mon Oct 18 11:03:22 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 11:03:31 2004 Subject: [ python-Bugs-1044382 ] Can't raise "C API version mismatch" warning Message-ID: Bugs item #1044382, was opened at 2004-10-11 09:22 Message generated for change (Comment added) made by sonnenburg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044382&group_id=5470 Category: Extension Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Frank Sonnenburg (sonnenburg) Assigned to: Nobody/Anonymous (nobody) Summary: Can't raise "C API version mismatch" warning Initial Comment: See following example (that's what we know): # Importing 2.1 module with Python 2.3 frank@theta:~> python Python 2.3+ (#1, Jan 7 2004, 09:17:35) [GCC 3.3.1 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.insert(0, '/usr/lib/python2.1/lib-dynload') >>> import cmath __main__:1: RuntimeWarning: Python C API version mismatch for module cmath: This Python has API version 1012, module cmath has version 1010. >>> ================================== Ignoring this warning works without problem, but raising an exception will lead to segmentation fault: # Starting python with warnings turned into exceptions frank@theta:~> python -W error Python 2.3+ (#1, Jan 7 2004, 09:17:35) [GCC 3.3.1 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.insert(0, '/usr/lib/python2.1/lib-dynload') >>> import cmath Speicherzugriffsfehler (seg fault) ---------------------------------------------------------------------- >Comment By: Frank Sonnenburg (sonnenburg) Date: 2004-10-18 11:03 Message: Logged In: YES user_id=459347 Still the same for 2.4b1 on SuSE 9.0: frank@theta:/backup/users/frank/div/download/python/2.4b1/ theta-SuSE9.0> ./python -W error Python 2.4b1 (#1, Oct 18 2004, 10:45:40) [GCC 3.3.1 (SuSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.path.insert(0, '/usr/lib/python2.1/lib-dynload') >>> import cmath Speicherzugriffsfehler To etrepum: I really don't expect a segfault, because Python2.1 is built on my machine as well as Python2.3 resp. Python2.4b1 is. I do would expect a program termination, iff there were unsatisfied symbols calling new or renamed Python C-API functions. ---------------------------------------------------------------------- Comment By: Terry J. Reedy (tjreedy) Date: 2004-10-16 00:49 Message: Logged In: YES user_id=593130 Do you get same 2.4? (b1 should be out within a day.) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-15 15:49 Message: Logged In: YES user_id=6656 Well, maybe but this still looks like a bug to me. ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-10-15 15:31 Message: Logged In: YES user_id=139309 It's not valid to use extensions from one major version of Python with another, hence the C API Version Mismatch warning. What do you expect? You're lucky it doesn't segfault sooner :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1044382&group_id=5470 From noreply at sourceforge.net Mon Oct 18 17:56:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 17:56:41 2004 Subject: [ python-Bugs-881812 ] 2.4a0 build fails in Modules/signalmodule.c Message-ID: Bugs item #881812, was opened at 2004-01-21 23:16 Message generated for change (Comment added) made by dcd You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=881812&group_id=5470 Category: Build Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Dyck (dcd) Assigned to: Nobody/Anonymous (nobody) Summary: 2.4a0 build fails in Modules/signalmodule.c Initial Comment: PYTHON_VERSION, 2.4a0 (Jan 21 CVS) gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -c ./Modules/signalmodule.c -o Modules/signalmodule.o ./Modules/signalmodule.c: In function `signal_handler': ./Modules/signalmodule.c:141: warning: implicit declaration of function `siginterrupt' ./Modules/signalmodule.c: In function `initsignal': ./Modules/signalmodule.c:550: `_NSIG' undeclared (first use in this function) ./Modules/signalmodule.c:550: (Each undeclared identifier is reported only once ./Modules/signalmodule.c:550: for each function it appears in.) make: *** [Modules/signalmodule.o] Error 1 (At one time my libc5 linux system had similar troubles building perl, but perl's Configure get's it right the hard way) ---------------------------------------------------------------------- >Comment By: David Dyck (dcd) Date: 2004-10-18 15:56 Message: Logged In: YES user_id=53855 The line numbers have changed but the error remains in Python-2.4b1 gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -c ./Modules/signalmodule.c -o Modules/signalmodule.o ./Modules/signalmodule.c: In function `initsignal': ./Modules/signalmodule.c:544: `_NSIG' undeclared (first use in this function) ./Modules/signalmodule.c:544: (Each undeclared identifier is reported only once ./Modules/signalmodule.c:544: for each function it appears in.) make: *** [Modules/signalmodule.o] Error 1 ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-17 20:54 Message: Logged In: YES user_id=33168 Is this still a problem w/ 2.4b1? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=881812&group_id=5470 From noreply at sourceforge.net Mon Oct 18 19:39:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 19:39:54 2004 Subject: [ python-Bugs-1049450 ] Solaris: EINTR exception in select/socket calls in telnetlib Message-ID: Bugs item #1049450, was opened at 2004-10-18 17:39 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049450&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Mike Brauwerman (brauwerman) Assigned to: Nobody/Anonymous (nobody) Summary: Solaris: EINTR exception in select/socket calls in telnetlib Initial Comment: On Solaris, calls to select.select() and socket.socket() in telnetlib (and possibly others) often fail due to unhandled EINTR signals from the OS while select() is polling. I think this problem is Solaris-specific since Solaris has interruptible non-restartable sytem calls. This behavior is apparently a known issue with the system API select(); see man -s3c select and http://lists.community.tummy.com/pipermail/frpythoneers/2000-August/000122.html The recommend fix from frpythoneers is to wrap the select (and socket, respectively) calls in a loop: while True: try: select.select(...) break except select.error, v: if v[0] == errno.EINTR: continue else: raise It's probably more appropriate to put the exception-handling *inside* select.select (and socket.socket) but that's beyond my expertise... OS: SunOS 5.9 Generic_112233-11 sun4u sparc SUNW,Sun-Blade-100 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049450&group_id=5470 From noreply at sourceforge.net Mon Oct 18 20:45:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 20:45:38 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 07:04 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Open >Resolution: None Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-10-18 11:45 Message: Logged In: YES user_id=357491 Reopening to deal with the error Tony caught. Tony, can you apply the included patch and verify it works for you okay? I added another if to track back a week if the calculation gets pushed past 366 days. That does fix your error. And if you can think of any other edge cases let me know. I think that should cover everything. And yes, the example for the OP is wrong:: >>> datetime.date(2004, 10, 11).strftime("%Y %U %w") '2004 41 1' ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-17 20:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-17 18:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-17 16:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Mon Oct 18 23:27:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 18 23:27:52 2004 Subject: [ python-Bugs-1048870 ] call arg of lambda not updating Message-ID: Bugs item #1048870, was opened at 2004-10-17 22:55 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 Category: Parser/Compiler Group: Python 2.3 Status: Open Resolution: None Priority: 2 Submitted By: Kevin Quick (kquick) Assigned to: Jeremy Hylton (jhylton) Summary: call arg of lambda not updating Initial Comment: The attachment contains a script intended to do performance testing on various types of dictionary keys. Lambda functions are used to "generate" a sequence of keys with bound arguments to arrays that are modified during execution. Script fails with pop() on empty list error, but examination shows that the lambda function from the *previous* call is being used for the current call; if proper lambda was used, list would not be empty and operation should succeed. Script is large and has been "grown", so it's not very elegant, but either I'm missing somewhere where I'm making a stupid mistake or else the lambda argument to the function call isn't being set properly. Don't be put off by the size of the script; the problem is easily demonstrated and fairly clearly understood in just a few minutes of looking at the generated stack trace relative to the code. As it exists, script will fail immediately due to the bug. If it worked as expected, at least 3-4 lines of output should be generated before encountering new code that hasn't yet been fully debugged. New code could be removed, but it was left in because some of the complexity of core code is due to this latter code, and it doesn't obstruct the primary bug, so it can be ignored unless of interest. python Python 2.3.3 (#1, May 27 2004, 20:44:16) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Thanks for looking at this! ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-18 22:27 Message: Logged In: YES user_id=6656 I *think* what's happening here is that co_firstlineno is not being considered when comparing code objects. Can you try the attached patch? There may well be other fields not being considered. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-18 08:37 Message: Logged In: YES user_id=6133 Thanks for confirming this issue. My position is that while minor, this is still a bug and not a feature. It's good to detect identical lambda bodies and optimize for that case, but since argument binding occurs at declaration time and not execution time for lambdas, the bound arguments should properly be considered part of the body; the two lambda functions in the attached script should be disjoint. >From the practical perspective, the python backtrace was misleading in my debug efforts. Internal optimizations should take second place to accuracy of displayed information. func_code.co_code (and possibly others) could still be the same, but func_code.co_firstlineno (& func_code.co_filename) should be uniquely set. Hmmm. In thinking more about this, it's not that argument binding occurs immediately, but moreso that defaults are supplied for arguments. If the backtrace/debugger points me to a lambda function with entirely different default args than the ones that are applicable to the problem, then that would be wrong, IMHO. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 07:42 Message: Logged In: YES user_id=80475 Under the hood, the compiler is recognizing that the function bodies are identical and it produces just one code object instead of two. Hence, the co_firstline number with always reflect the first time the code is defined. A simple demo shows the id being the same when the bodies are identical: src = """ f1 = lambda x=1: x f2 = lambda x=2: x """ from dis import dis c = compile(src, 'example', 'exec') dis(c) For the OP's code, the effect can be shown by differentiating either lambda body by adding zero to the pop index: nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()+0] I conclude that there is less to the bug than meets the eye and that it may even be considered a feature. Referring to Jeremy for final disposition. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 06:41 Message: Logged In: YES user_id=80475 Confirmed. Also, the behavior persists into Py2.4b1. The error disappears when the lambda is replaced by an equivalent one line def statement: def nextkey2(N=keynums2,K=key_src): return K[N.pop()] The opcodes are the same for both; however, the co_firstlineno attribute is incorrect for the lambda version. The pure python compiler module does not make the same error. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-18 05:48 Message: Logged In: YES user_id=6133 Reopened: Problem not addressed. Please *read* my previous comment. Yes, the script has a bug in it. That's not what's being reported here. The bug is in the backtrace produced by the Python interpreter. See below. Line number reported for lambda function raising exception is line 14 below. Actual line number for lambda generating exception is line 18. $ python lambda_bug.py Traceback (most recent call last): File "lambda_bug.py", line 23, in ? seqtest() File "lambda_bug.py", line 19, in seqtest seq_func1(nextkey2, num_elem+2) File "lambda_bug.py", line 5, in seq_func1 key_genfunc() File "lambda_bug.py", line 14, in nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()] IndexError: pop from empty list ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 05:34 Message: Logged In: YES user_id=80475 Closing as invalid -- this is a bug in the OP's script, not in Python itself. The error is obious when the code is simplified further by in-lining the function. Also, it helps to explicitly set the initial values of variables between the two sections: #!/usr/bin/env python num_elem = 100 key_src = range(num_elem+1) keynums1 = key_src[:] assert key_src[-1] == 100 for _ in xrange(num_elem): keynums1[key_src.pop()] keynums2 = [0] for _ in xrange(102): key_src[keynums2.pop()] I think your coding error was in assuming that keynum2 held a different value before the second half was run. If so, that could have been found by using pdb or by inserting print statements to reveal what is going on. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-18 05:02 Message: Logged In: YES user_id=6133 OK, example script minimized. Lambda arguments are not the problem; thanks for the pointer, but I'm not having trouble with those. I have determined that proper lambda is being used, but not indicated as shown by attached script, which forces an exception to occur in the lambda function but backtrace shows wrong lambda function. Reducing 2nd argument of 2nd call to seq_func1 to 'num_args' instead of 'num_args+2' will not encounter the error, but the nature of the implementation requires that it is using the correct lambda, so it's the backtrace attribution that is apparently wrong. Mea culpa: my original script version did request wrong 'num_args' causing actual bug, but incorrect backtrace led me down an alternate path... I suspected that I had some incorrect code, but I couldn't resolve that against the error report. Still can't. :) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 00:04 Message: Logged In: YES user_id=80475 Please resubmit after boiling this down to the simplest possible thing that doesn't work according to your expectations. Submitting a whole application and playing the "find the bug" game is not good use of developer time. Often, when a submitter starts taking away the unnecessary pieces and isolates the issue, they find that there was a bug in their own code or a bug in their own understanding of how the language works. When it comes to lambdas, the most frequent misunderstanding is knowing that default arguments bind only once as the time of the lambda declaration and that free variables bind whenever the resulting function is invoked. Here's an example of code that looks similar but behaves differently: def once(x): return x def twice(x): return 2*x def thrice(x): return 3*x funcs = [once, twice, thrice] flim = [lambda x:funcs[0](x), lambda x:funcs[1](x), lambda x:funcs[2](x)] flam = [lambda x:f(x) for f in funcs] print flim[0](1), flim[1](1), flim[2](1) print flam[0](1), flam[1](1), flam[2](1) The difference in output is because nested scopes bind names, while argument binding binds values. Hope this helps. If not, please resubmit with the smallest case that demonstrates the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 From noreply at sourceforge.net Tue Oct 19 00:09:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 00:09:09 2004 Subject: [ python-Bugs-1049615 ] smeared title when installing Message-ID: Bugs item #1049615, was opened at 2004-10-18 18:09 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 Category: Installation Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: smeared title when installing Initial Comment: Python 2.4b1 on Windows XP. 1280x800. The "python: for windows" heading looks smeared. Other graphical elements also look somewhat smeared, but it isn't as much of a problem when it isn't text to begin with. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 From noreply at sourceforge.net Tue Oct 19 00:16:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 00:16:08 2004 Subject: [ python-Bugs-1049615 ] smeared title when installing Message-ID: Bugs item #1049615, was opened at 2004-10-18 18:09 Message generated for change (Comment added) made by jimjjewett You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 Category: Installation Group: Python 2.4 Status: Open Resolution: None >Priority: 3 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: smeared title when installing Initial Comment: Python 2.4b1 on Windows XP. 1280x800. The "python: for windows" heading looks smeared. Other graphical elements also look somewhat smeared, but it isn't as much of a problem when it isn't text to begin with. ---------------------------------------------------------------------- >Comment By: Jim Jewett (jimjjewett) Date: 2004-10-18 18:16 Message: Logged In: YES user_id=764593 I used a .bmp to avoid additional info loss. Zipped file is only 20K, but unzips to over a meg. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 From noreply at sourceforge.net Tue Oct 19 00:18:22 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 00:18:25 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-18 03:24 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 00:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 08:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Tue Oct 19 00:29:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 00:29:23 2004 Subject: [ python-Bugs-1048870 ] call arg of lambda not updating Message-ID: Bugs item #1048870, was opened at 2004-10-17 16:55 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 Category: Parser/Compiler Group: Python 2.3 Status: Open Resolution: None Priority: 2 Submitted By: Kevin Quick (kquick) >Assigned to: Michael Hudson (mwh) Summary: call arg of lambda not updating Initial Comment: The attachment contains a script intended to do performance testing on various types of dictionary keys. Lambda functions are used to "generate" a sequence of keys with bound arguments to arrays that are modified during execution. Script fails with pop() on empty list error, but examination shows that the lambda function from the *previous* call is being used for the current call; if proper lambda was used, list would not be empty and operation should succeed. Script is large and has been "grown", so it's not very elegant, but either I'm missing somewhere where I'm making a stupid mistake or else the lambda argument to the function call isn't being set properly. Don't be put off by the size of the script; the problem is easily demonstrated and fairly clearly understood in just a few minutes of looking at the generated stack trace relative to the code. As it exists, script will fail immediately due to the bug. If it worked as expected, at least 3-4 lines of output should be generated before encountering new code that hasn't yet been fully debugged. New code could be removed, but it was left in because some of the complexity of core code is due to this latter code, and it doesn't obstruct the primary bug, so it can be ignored unless of interest. python Python 2.3.3 (#1, May 27 2004, 20:44:16) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Thanks for looking at this! ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 17:29 Message: Logged In: YES user_id=80475 Michael, that is a reasonable solution. Go ahead and apply it along with a test case verifying the object id's are distinct after the change but not before. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-18 16:27 Message: Logged In: YES user_id=6656 I *think* what's happening here is that co_firstlineno is not being considered when comparing code objects. Can you try the attached patch? There may well be other fields not being considered. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-18 02:37 Message: Logged In: YES user_id=6133 Thanks for confirming this issue. My position is that while minor, this is still a bug and not a feature. It's good to detect identical lambda bodies and optimize for that case, but since argument binding occurs at declaration time and not execution time for lambdas, the bound arguments should properly be considered part of the body; the two lambda functions in the attached script should be disjoint. >From the practical perspective, the python backtrace was misleading in my debug efforts. Internal optimizations should take second place to accuracy of displayed information. func_code.co_code (and possibly others) could still be the same, but func_code.co_firstlineno (& func_code.co_filename) should be uniquely set. Hmmm. In thinking more about this, it's not that argument binding occurs immediately, but moreso that defaults are supplied for arguments. If the backtrace/debugger points me to a lambda function with entirely different default args than the ones that are applicable to the problem, then that would be wrong, IMHO. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 01:42 Message: Logged In: YES user_id=80475 Under the hood, the compiler is recognizing that the function bodies are identical and it produces just one code object instead of two. Hence, the co_firstline number with always reflect the first time the code is defined. A simple demo shows the id being the same when the bodies are identical: src = """ f1 = lambda x=1: x f2 = lambda x=2: x """ from dis import dis c = compile(src, 'example', 'exec') dis(c) For the OP's code, the effect can be shown by differentiating either lambda body by adding zero to the pop index: nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()+0] I conclude that there is less to the bug than meets the eye and that it may even be considered a feature. Referring to Jeremy for final disposition. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 00:41 Message: Logged In: YES user_id=80475 Confirmed. Also, the behavior persists into Py2.4b1. The error disappears when the lambda is replaced by an equivalent one line def statement: def nextkey2(N=keynums2,K=key_src): return K[N.pop()] The opcodes are the same for both; however, the co_firstlineno attribute is incorrect for the lambda version. The pure python compiler module does not make the same error. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 23:48 Message: Logged In: YES user_id=6133 Reopened: Problem not addressed. Please *read* my previous comment. Yes, the script has a bug in it. That's not what's being reported here. The bug is in the backtrace produced by the Python interpreter. See below. Line number reported for lambda function raising exception is line 14 below. Actual line number for lambda generating exception is line 18. $ python lambda_bug.py Traceback (most recent call last): File "lambda_bug.py", line 23, in ? seqtest() File "lambda_bug.py", line 19, in seqtest seq_func1(nextkey2, num_elem+2) File "lambda_bug.py", line 5, in seq_func1 key_genfunc() File "lambda_bug.py", line 14, in nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()] IndexError: pop from empty list ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 23:34 Message: Logged In: YES user_id=80475 Closing as invalid -- this is a bug in the OP's script, not in Python itself. The error is obious when the code is simplified further by in-lining the function. Also, it helps to explicitly set the initial values of variables between the two sections: #!/usr/bin/env python num_elem = 100 key_src = range(num_elem+1) keynums1 = key_src[:] assert key_src[-1] == 100 for _ in xrange(num_elem): keynums1[key_src.pop()] keynums2 = [0] for _ in xrange(102): key_src[keynums2.pop()] I think your coding error was in assuming that keynum2 held a different value before the second half was run. If so, that could have been found by using pdb or by inserting print statements to reveal what is going on. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 23:02 Message: Logged In: YES user_id=6133 OK, example script minimized. Lambda arguments are not the problem; thanks for the pointer, but I'm not having trouble with those. I have determined that proper lambda is being used, but not indicated as shown by attached script, which forces an exception to occur in the lambda function but backtrace shows wrong lambda function. Reducing 2nd argument of 2nd call to seq_func1 to 'num_args' instead of 'num_args+2' will not encounter the error, but the nature of the implementation requires that it is using the correct lambda, so it's the backtrace attribution that is apparently wrong. Mea culpa: my original script version did request wrong 'num_args' causing actual bug, but incorrect backtrace led me down an alternate path... I suspected that I had some incorrect code, but I couldn't resolve that against the error report. Still can't. :) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 18:04 Message: Logged In: YES user_id=80475 Please resubmit after boiling this down to the simplest possible thing that doesn't work according to your expectations. Submitting a whole application and playing the "find the bug" game is not good use of developer time. Often, when a submitter starts taking away the unnecessary pieces and isolates the issue, they find that there was a bug in their own code or a bug in their own understanding of how the language works. When it comes to lambdas, the most frequent misunderstanding is knowing that default arguments bind only once as the time of the lambda declaration and that free variables bind whenever the resulting function is invoked. Here's an example of code that looks similar but behaves differently: def once(x): return x def twice(x): return 2*x def thrice(x): return 3*x funcs = [once, twice, thrice] flim = [lambda x:funcs[0](x), lambda x:funcs[1](x), lambda x:funcs[2](x)] flam = [lambda x:f(x) for f in funcs] print flim[0](1), flim[1](1), flim[2](1) print flam[0](1), flam[1](1), flam[2](1) The difference in output is because nested scopes bind names, while argument binding binds values. Hope this helps. If not, please resubmit with the smallest case that demonstrates the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 From noreply at sourceforge.net Tue Oct 19 01:09:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 01:09:28 2004 Subject: [ python-Bugs-1049615 ] smeared title when installing Message-ID: Bugs item #1049615, was opened at 2004-10-18 22:09 Message generated for change (Comment added) made by brauwerman You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 Category: Installation Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: smeared title when installing Initial Comment: Python 2.4b1 on Windows XP. 1280x800. The "python: for windows" heading looks smeared. Other graphical elements also look somewhat smeared, but it isn't as much of a problem when it isn't text to begin with. ---------------------------------------------------------------------- Comment By: Mike Brauwerman (brauwerman) Date: 2004-10-18 23:09 Message: Logged In: YES user_id=71983 The "smearing" is the effect of a small bitmap picture, with aliased text, blown up (stretched) to fit into a large space. The graphic has fat pixels, which look a little funny, but they look better if you squint :-) ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-10-18 22:16 Message: Logged In: YES user_id=764593 I used a .bmp to avoid additional info loss. Zipped file is only 20K, but unzips to over a meg. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 From noreply at sourceforge.net Tue Oct 19 01:24:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 01:24:18 2004 Subject: [ python-Bugs-1048870 ] call arg of lambda not updating Message-ID: Bugs item #1048870, was opened at 2004-10-17 14:55 Message generated for change (Comment added) made by kquick You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 Category: Parser/Compiler Group: Python 2.3 Status: Open Resolution: None Priority: 2 Submitted By: Kevin Quick (kquick) Assigned to: Michael Hudson (mwh) Summary: call arg of lambda not updating Initial Comment: The attachment contains a script intended to do performance testing on various types of dictionary keys. Lambda functions are used to "generate" a sequence of keys with bound arguments to arrays that are modified during execution. Script fails with pop() on empty list error, but examination shows that the lambda function from the *previous* call is being used for the current call; if proper lambda was used, list would not be empty and operation should succeed. Script is large and has been "grown", so it's not very elegant, but either I'm missing somewhere where I'm making a stupid mistake or else the lambda argument to the function call isn't being set properly. Don't be put off by the size of the script; the problem is easily demonstrated and fairly clearly understood in just a few minutes of looking at the generated stack trace relative to the code. As it exists, script will fail immediately due to the bug. If it worked as expected, at least 3-4 lines of output should be generated before encountering new code that hasn't yet been fully debugged. New code could be removed, but it was left in because some of the complexity of core code is due to this latter code, and it doesn't obstruct the primary bug, so it can be ignored unless of interest. python Python 2.3.3 (#1, May 27 2004, 20:44:16) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Thanks for looking at this! ---------------------------------------------------------------------- >Comment By: Kevin Quick (kquick) Date: 2004-10-18 16:24 Message: Logged In: YES user_id=6133 Confirmed that the patch does indeed address the problem. Also verified that a lambda with the same code appearing at the same line in a different file is handled separately and distinctly. Thanks! ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 15:29 Message: Logged In: YES user_id=80475 Michael, that is a reasonable solution. Go ahead and apply it along with a test case verifying the object id's are distinct after the change but not before. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-18 14:27 Message: Logged In: YES user_id=6656 I *think* what's happening here is that co_firstlineno is not being considered when comparing code objects. Can you try the attached patch? There may well be other fields not being considered. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-18 00:37 Message: Logged In: YES user_id=6133 Thanks for confirming this issue. My position is that while minor, this is still a bug and not a feature. It's good to detect identical lambda bodies and optimize for that case, but since argument binding occurs at declaration time and not execution time for lambdas, the bound arguments should properly be considered part of the body; the two lambda functions in the attached script should be disjoint. >From the practical perspective, the python backtrace was misleading in my debug efforts. Internal optimizations should take second place to accuracy of displayed information. func_code.co_code (and possibly others) could still be the same, but func_code.co_firstlineno (& func_code.co_filename) should be uniquely set. Hmmm. In thinking more about this, it's not that argument binding occurs immediately, but moreso that defaults are supplied for arguments. If the backtrace/debugger points me to a lambda function with entirely different default args than the ones that are applicable to the problem, then that would be wrong, IMHO. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 23:42 Message: Logged In: YES user_id=80475 Under the hood, the compiler is recognizing that the function bodies are identical and it produces just one code object instead of two. Hence, the co_firstline number with always reflect the first time the code is defined. A simple demo shows the id being the same when the bodies are identical: src = """ f1 = lambda x=1: x f2 = lambda x=2: x """ from dis import dis c = compile(src, 'example', 'exec') dis(c) For the OP's code, the effect can be shown by differentiating either lambda body by adding zero to the pop index: nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()+0] I conclude that there is less to the bug than meets the eye and that it may even be considered a feature. Referring to Jeremy for final disposition. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 22:41 Message: Logged In: YES user_id=80475 Confirmed. Also, the behavior persists into Py2.4b1. The error disappears when the lambda is replaced by an equivalent one line def statement: def nextkey2(N=keynums2,K=key_src): return K[N.pop()] The opcodes are the same for both; however, the co_firstlineno attribute is incorrect for the lambda version. The pure python compiler module does not make the same error. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 21:48 Message: Logged In: YES user_id=6133 Reopened: Problem not addressed. Please *read* my previous comment. Yes, the script has a bug in it. That's not what's being reported here. The bug is in the backtrace produced by the Python interpreter. See below. Line number reported for lambda function raising exception is line 14 below. Actual line number for lambda generating exception is line 18. $ python lambda_bug.py Traceback (most recent call last): File "lambda_bug.py", line 23, in ? seqtest() File "lambda_bug.py", line 19, in seqtest seq_func1(nextkey2, num_elem+2) File "lambda_bug.py", line 5, in seq_func1 key_genfunc() File "lambda_bug.py", line 14, in nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()] IndexError: pop from empty list ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 21:34 Message: Logged In: YES user_id=80475 Closing as invalid -- this is a bug in the OP's script, not in Python itself. The error is obious when the code is simplified further by in-lining the function. Also, it helps to explicitly set the initial values of variables between the two sections: #!/usr/bin/env python num_elem = 100 key_src = range(num_elem+1) keynums1 = key_src[:] assert key_src[-1] == 100 for _ in xrange(num_elem): keynums1[key_src.pop()] keynums2 = [0] for _ in xrange(102): key_src[keynums2.pop()] I think your coding error was in assuming that keynum2 held a different value before the second half was run. If so, that could have been found by using pdb or by inserting print statements to reveal what is going on. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 21:02 Message: Logged In: YES user_id=6133 OK, example script minimized. Lambda arguments are not the problem; thanks for the pointer, but I'm not having trouble with those. I have determined that proper lambda is being used, but not indicated as shown by attached script, which forces an exception to occur in the lambda function but backtrace shows wrong lambda function. Reducing 2nd argument of 2nd call to seq_func1 to 'num_args' instead of 'num_args+2' will not encounter the error, but the nature of the implementation requires that it is using the correct lambda, so it's the backtrace attribution that is apparently wrong. Mea culpa: my original script version did request wrong 'num_args' causing actual bug, but incorrect backtrace led me down an alternate path... I suspected that I had some incorrect code, but I couldn't resolve that against the error report. Still can't. :) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 16:04 Message: Logged In: YES user_id=80475 Please resubmit after boiling this down to the simplest possible thing that doesn't work according to your expectations. Submitting a whole application and playing the "find the bug" game is not good use of developer time. Often, when a submitter starts taking away the unnecessary pieces and isolates the issue, they find that there was a bug in their own code or a bug in their own understanding of how the language works. When it comes to lambdas, the most frequent misunderstanding is knowing that default arguments bind only once as the time of the lambda declaration and that free variables bind whenever the resulting function is invoked. Here's an example of code that looks similar but behaves differently: def once(x): return x def twice(x): return 2*x def thrice(x): return 3*x funcs = [once, twice, thrice] flim = [lambda x:funcs[0](x), lambda x:funcs[1](x), lambda x:funcs[2](x)] flam = [lambda x:f(x) for f in funcs] print flim[0](1), flim[1](1), flim[2](1) print flam[0](1), flam[1](1), flam[2](1) The difference in output is because nested scopes bind names, while argument binding binds values. Hope this helps. If not, please resubmit with the smallest case that demonstrates the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 From noreply at sourceforge.net Tue Oct 19 01:32:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 01:32:49 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 18:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 02:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Tue Oct 19 01:42:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 01:42:28 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:42 Message: Logged In: YES user_id=6380 (Actually, os.walk has an onerror argument, it just uses a different convention and default, so we could thread this through easily.) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 18:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 02:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Tue Oct 19 03:29:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 03:29:38 2004 Subject: [ python-Bugs-854823 ] Python-2.3.3c1, Solaris 2.7: socketmodule does not compile Message-ID: Bugs item #854823, was opened at 2003-12-05 11:01 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854823&group_id=5470 Category: Build Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Falko Sauermann (fsm2761) Assigned to: Nobody/Anonymous (nobody) Summary: Python-2.3.3c1, Solaris 2.7: socketmodule does not compile Initial Comment: Compiling Python-2.3.3c1 on Solaris 2.7 with gcc 2.95.3 gave me two errors: socketmodule.c:2975: `AF_INET6' undeclared (first use in this function) socketmodule.c:3019: `INET_ADDRSTRLEN' undeclared (first use in this function) The first problem was already reported for Solaris 2.7 with SunPro cc (see item 814613) but is also true for Solaris 2.7 with gcc. The real problem is that AF_INET6 is used when ENABLE_IPV6 is undefined. I believe this must fail for any system where IPV6 is not avalilable. The second problem was already reported for Irix (see item 818490) but is also true for Solaris 2.7. The solution for Irix is also valid for Solaris. If I change socketmodule.c line 204 to: #if (defined(__sgi) || defined(sun)) && !defined(INET_ADDRSTRLEN) the error is gone. ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-18 21:29 Message: Logged In: YES user_id=33168 Does this problem still exist in 2.3.4 or 2.4b1? Can this be closed? ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-17 22:37 Message: Logged In: YES user_id=33168 Does this problem still exist in 2.3.4 or 2.4b1? Can this be closed? ---------------------------------------------------------------------- Comment By: Stephan A. Terre (sfiedler) Date: 2004-02-26 12:23 Message: Logged In: YES user_id=246063 I think it's worth noting that the preprocessor directive near line 2975 is #ifndef , whereas all other preprocessor directives in the file that refer to the IPV6 symbol are #ifdef . In other words, it looks like just that one directive is a typo. Changing #ifndef to #ifdef fixes it for me. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854823&group_id=5470 From noreply at sourceforge.net Tue Oct 19 04:53:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 04:53:25 2004 Subject: [ python-Bugs-881812 ] 2.4a0 build fails in Modules/signalmodule.c Message-ID: Bugs item #881812, was opened at 2004-01-21 18:16 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=881812&group_id=5470 Category: Build Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Dyck (dcd) Assigned to: Nobody/Anonymous (nobody) Summary: 2.4a0 build fails in Modules/signalmodule.c Initial Comment: PYTHON_VERSION, 2.4a0 (Jan 21 CVS) gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -c ./Modules/signalmodule.c -o Modules/signalmodule.o ./Modules/signalmodule.c: In function `signal_handler': ./Modules/signalmodule.c:141: warning: implicit declaration of function `siginterrupt' ./Modules/signalmodule.c: In function `initsignal': ./Modules/signalmodule.c:550: `_NSIG' undeclared (first use in this function) ./Modules/signalmodule.c:550: (Each undeclared identifier is reported only once ./Modules/signalmodule.c:550: for each function it appears in.) make: *** [Modules/signalmodule.o] Error 1 (At one time my libc5 linux system had similar troubles building perl, but perl's Configure get's it right the hard way) ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-18 22:53 Message: Logged In: YES user_id=33168 That's odd. Line 544 is using SIGRTMAX. I presume SIGRTMAX is defined as _NSIG on your system? I don't know why it would do that. Can you try to produce a patch? What type of platform is this (beyond libc5 based)? ---------------------------------------------------------------------- Comment By: David Dyck (dcd) Date: 2004-10-18 11:56 Message: Logged In: YES user_id=53855 The line numbers have changed but the error remains in Python-2.4b1 gcc -fno-strict-aliasing -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I./Include -DPy_BUILD_CORE -c ./Modules/signalmodule.c -o Modules/signalmodule.o ./Modules/signalmodule.c: In function `initsignal': ./Modules/signalmodule.c:544: `_NSIG' undeclared (first use in this function) ./Modules/signalmodule.c:544: (Each undeclared identifier is reported only once ./Modules/signalmodule.c:544: for each function it appears in.) make: *** [Modules/signalmodule.o] Error 1 ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-17 16:54 Message: Logged In: YES user_id=33168 Is this still a problem w/ 2.4b1? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=881812&group_id=5470 From noreply at sourceforge.net Tue Oct 19 09:31:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 09:37:48 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-13 03:04 Message generated for change (Comment added) made by anadelonbrin You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-19 20:31 Message: Logged In: YES user_id=552329 Patch works for that case, yes. However... [from python-dev] > the test case I was using that triggered > the need for that is 1906-12-31 > which is a Monday but changes what week > it is based on whether you use U or W. > which makes no sense since both U and W > should consider it the same week. > Had the same result for 1917-12-31. Something's still not right here (this is with the patch): >>> time.strptime("1917 51 1", "%Y %U %w") (1917, 12, 17, 0, 0, 0, 0, 351, -1) >>> time.strptime("1917 52 1", "%Y %U %w") (1917, 12, 31, 0, 0, 0, 0, 365, -1) 1917 both started and ended on a Monday, so there are 53 U weeks, right? (where the last 'week' is only one day). So 31/12/1917 should be U=53, W=52 ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-19 07:45 Message: Logged In: YES user_id=357491 Reopening to deal with the error Tony caught. Tony, can you apply the included patch and verify it works for you okay? I added another if to track back a week if the calculation gets pushed past 366 days. That does fix your error. And if you can think of any other edge cases let me know. I think that should cover everything. And yes, the example for the OP is wrong:: >>> datetime.date(2004, 10, 11).strftime("%Y %U %w") '2004 41 1' ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-18 16:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 14:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 12:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 14:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 14:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 14:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Tue Oct 19 09:57:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 09:57:52 2004 Subject: [ python-Bugs-1049816 ] test_socket PORT conflict with boa-constructor Message-ID: Bugs item #1049816, was opened at 2004-10-19 10:57 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049816&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Simon Dahlbacka (sdahlbac) Assigned to: Nobody/Anonymous (nobody) Summary: test_socket PORT conflict with boa-constructor Initial Comment: trying to run the test_socket.py either through regrtest.py or directly when boa-constructor is running (v. 0.3.1) makes boa constructor complain about file "Michael Gilfix was here\n" not found, and the test hangs indefinitely. When boa is not running the test works fine. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049816&group_id=5470 From noreply at sourceforge.net Tue Oct 19 10:12:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 10:12:51 2004 Subject: [ python-Bugs-1049826 ] zlib module documentation wrong version Message-ID: Bugs item #1049826, was opened at 2004-10-19 08:12 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049826&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Wolfgang Langner (tds33) Assigned to: Nobody/Anonymous (nobody) Summary: zlib module documentation wrong version Initial Comment: The documentation of the zlib module is a little bit out of date: Version 1.1.3 is the most recent version as of September 2000. This is wrong, the most recent version of zlib is: 1.2.1 Change it to: Version 1.2.1 is the most recent version as of October 2004. source: http://www.python.org/dev/doc/devel/lib/module-zlib.html I hope python 2.4 Release will use this version. (it is out since end 2003) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049826&group_id=5470 From noreply at sourceforge.net Tue Oct 19 10:57:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 10:57:46 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-18 03:24 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 10:57 Message: Logged In: YES user_id=469548 A new patch is attached. I factored out the handle_call_error method so exception handling is more local. Also, errors in os.walk were already being handled using the _raise_err/except OSError combination, but we now do it using the onerror argument to os.walk. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 01:42 Message: Logged In: YES user_id=6380 (Actually, os.walk has an onerror argument, it just uses a different convention and default, so we could thread this through easily.) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 01:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 00:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 08:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Tue Oct 19 11:08:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 11:09:08 2004 Subject: [ python-Bugs-1030629 ] PyOS_InputHook broken Message-ID: Bugs item #1030629, was opened at 2004-09-19 19:27 Message generated for change (Comment added) made by mdehoon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Closed Resolution: Fixed Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Michael Hudson (mwh) Summary: PyOS_InputHook broken Initial Comment: PyOS_InputHook is a pointer to a function to be called periodically when Python is idle. It is used, for example, to get messages delivered to graphics windows. In Python 2.3.4, calling PyOS_InputHook was handled by readline via rl_event_hook = PyOS_InputHook; ... p = readline(prompt); The readline library takes care of calling rl_event_hook (and hence PyOS_InputHook) ten times per second while Python idles at the call to readline(prompt). In Python 2.4a3, the call to readline is replaced by a call to readline_until_enter_or_signal. Here, the "select" function is used: has_input = select(fileno(rl_instream) + 1, &selectset, NULL, NULL, NULL); if(has_input > 0) { rl_callback_read_char(); } Now Python idles at "select", but PyOS_InputHook is not being called. Instead, PyOS_InputHook gets called via rl_event_hook at rl_callback_read_char. This means that PyOS_InputHook is called once for every character the user types, and is not called at all if the user is not typing something. ---------------------------------------------------------------------- >Comment By: Michiel de Hoon (mdehoon) Date: 2004-10-19 18:08 Message: Logged In: YES user_id=488897 Sorry for my late reply, I was out of town last week. I tested the updated readline.c (using Python 2.4b1) and found no problems on Cygwin, Linux, and Mac OS X. On Windows (using the installer from the Python website), PyOS_InputHook is still being called once for every command by the user. However, this was also the case in older versions of Python, and by looking at the source it seems that this problem is in Parser/myreadline.c instead of Modules/readline.c (I submitted a separate patch for that). Anyway, Modules/readline.c seems to be working fine. Thanks for helping me out. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 22:47 Message: Logged In: YES user_id=6656 Ok, I made a couple of trivial changes, including moving the rl_event_hook assignment. Seems to work for me. Modules/readline.c revision 2.78. Thanks for the patch! ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2004-10-07 21:21 Message: Logged In: YES user_id=488897 That is right. I didn't think about that. You might consider assigning to rl_event_hook inside the second readline_until_enter_or_signal (the one if not defined(HAVE_RL_CALLBACK) && defined(HAVE_SELECT)), because it is only needed there. Though it probably wouldn't hurt to set rl_event_hook for both cases. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 21:10 Message: Logged In: YES user_id=6656 Oops, didn't see the patch. While I'm updating my checkout and testing the patch, my first thought is that you probably still want to assign to rl_event_hook, just in case we're *not* using the callback interface to readline (old versions of readline for example). ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2004-10-07 21:08 Message: Logged In: YES user_id=488897 I submitted a patch a couple of weeks ago. Let me know if you can't find it on sourceforge. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 20:20 Message: Logged In: YES user_id=6656 er, ping? i'm willing to believe this is a fairly bad bug, but i could do with some encouragement to fix it :-) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-20 03:35 Message: Logged In: YES user_id=6656 Well, this is probably my fault. However, I'm about to go on holiday for a week and thinking about readline is definitely not part of my plans :-) Can you work up a patch for this? It shouldn't be terribly hard -- a non-NULL fifth argument to select seems in order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 From noreply at sourceforge.net Tue Oct 19 11:20:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 11:20:16 2004 Subject: [ python-Bugs-1030629 ] PyOS_InputHook broken Message-ID: Bugs item #1030629, was opened at 2004-09-19 11:27 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Closed Resolution: Fixed Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Michael Hudson (mwh) Summary: PyOS_InputHook broken Initial Comment: PyOS_InputHook is a pointer to a function to be called periodically when Python is idle. It is used, for example, to get messages delivered to graphics windows. In Python 2.3.4, calling PyOS_InputHook was handled by readline via rl_event_hook = PyOS_InputHook; ... p = readline(prompt); The readline library takes care of calling rl_event_hook (and hence PyOS_InputHook) ten times per second while Python idles at the call to readline(prompt). In Python 2.4a3, the call to readline is replaced by a call to readline_until_enter_or_signal. Here, the "select" function is used: has_input = select(fileno(rl_instream) + 1, &selectset, NULL, NULL, NULL); if(has_input > 0) { rl_callback_read_char(); } Now Python idles at "select", but PyOS_InputHook is not being called. Instead, PyOS_InputHook gets called via rl_event_hook at rl_callback_read_char. This means that PyOS_InputHook is called once for every character the user types, and is not called at all if the user is not typing something. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-19 10:20 Message: Logged In: YES user_id=6656 Glad to help. Windows is *so* not my problem :) Readline isn't used there, anyway (except for cygwin). ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2004-10-19 10:08 Message: Logged In: YES user_id=488897 Sorry for my late reply, I was out of town last week. I tested the updated readline.c (using Python 2.4b1) and found no problems on Cygwin, Linux, and Mac OS X. On Windows (using the installer from the Python website), PyOS_InputHook is still being called once for every command by the user. However, this was also the case in older versions of Python, and by looking at the source it seems that this problem is in Parser/myreadline.c instead of Modules/readline.c (I submitted a separate patch for that). Anyway, Modules/readline.c seems to be working fine. Thanks for helping me out. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 14:47 Message: Logged In: YES user_id=6656 Ok, I made a couple of trivial changes, including moving the rl_event_hook assignment. Seems to work for me. Modules/readline.c revision 2.78. Thanks for the patch! ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2004-10-07 13:21 Message: Logged In: YES user_id=488897 That is right. I didn't think about that. You might consider assigning to rl_event_hook inside the second readline_until_enter_or_signal (the one if not defined(HAVE_RL_CALLBACK) && defined(HAVE_SELECT)), because it is only needed there. Though it probably wouldn't hurt to set rl_event_hook for both cases. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 13:10 Message: Logged In: YES user_id=6656 Oops, didn't see the patch. While I'm updating my checkout and testing the patch, my first thought is that you probably still want to assign to rl_event_hook, just in case we're *not* using the callback interface to readline (old versions of readline for example). ---------------------------------------------------------------------- Comment By: Michiel de Hoon (mdehoon) Date: 2004-10-07 13:08 Message: Logged In: YES user_id=488897 I submitted a patch a couple of weeks ago. Let me know if you can't find it on sourceforge. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-07 12:20 Message: Logged In: YES user_id=6656 er, ping? i'm willing to believe this is a fairly bad bug, but i could do with some encouragement to fix it :-) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-19 19:35 Message: Logged In: YES user_id=6656 Well, this is probably my fault. However, I'm about to go on holiday for a week and thinking about readline is definitely not part of my plans :-) Can you work up a patch for this? It shouldn't be terribly hard -- a non-NULL fifth argument to select seems in order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030629&group_id=5470 From noreply at sourceforge.net Tue Oct 19 11:57:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 12:03:05 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-13 00:04 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 19:57 Message: Logged In: YES user_id=1038590 The calculation of 'preceeding_days' looks incorrect. It assumes that the week starts on day 0 - it needs to adjust for when "week_of year_start" is 6. ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-19 17:31 Message: Logged In: YES user_id=552329 Patch works for that case, yes. However... [from python-dev] > the test case I was using that triggered > the need for that is 1906-12-31 > which is a Monday but changes what week > it is based on whether you use U or W. > which makes no sense since both U and W > should consider it the same week. > Had the same result for 1917-12-31. Something's still not right here (this is with the patch): >>> time.strptime("1917 51 1", "%Y %U %w") (1917, 12, 17, 0, 0, 0, 0, 351, -1) >>> time.strptime("1917 52 1", "%Y %U %w") (1917, 12, 31, 0, 0, 0, 0, 365, -1) 1917 both started and ended on a Monday, so there are 53 U weeks, right? (where the last 'week' is only one day). So 31/12/1917 should be U=53, W=52 ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-19 04:45 Message: Logged In: YES user_id=357491 Reopening to deal with the error Tony caught. Tony, can you apply the included patch and verify it works for you okay? I added another if to track back a week if the calculation gets pushed past 366 days. That does fix your error. And if you can think of any other edge cases let me know. I think that should cover everything. And yes, the example for the OP is wrong:: >>> datetime.date(2004, 10, 11).strftime("%Y %U %w") '2004 41 1' ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-18 13:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 11:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 09:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Tue Oct 19 12:03:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 12:03:14 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-13 00:04 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:03 Message: Logged In: YES user_id=1038590 Scratch the last comment - I missed the wee_of_year adjustment at the top of the function. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 19:57 Message: Logged In: YES user_id=1038590 The calculation of 'preceeding_days' looks incorrect. It assumes that the week starts on day 0 - it needs to adjust for when "week_of year_start" is 6. ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-19 17:31 Message: Logged In: YES user_id=552329 Patch works for that case, yes. However... [from python-dev] > the test case I was using that triggered > the need for that is 1906-12-31 > which is a Monday but changes what week > it is based on whether you use U or W. > which makes no sense since both U and W > should consider it the same week. > Had the same result for 1917-12-31. Something's still not right here (this is with the patch): >>> time.strptime("1917 51 1", "%Y %U %w") (1917, 12, 17, 0, 0, 0, 0, 351, -1) >>> time.strptime("1917 52 1", "%Y %U %w") (1917, 12, 31, 0, 0, 0, 0, 365, -1) 1917 both started and ended on a Monday, so there are 53 U weeks, right? (where the last 'week' is only one day). So 31/12/1917 should be U=53, W=52 ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-19 04:45 Message: Logged In: YES user_id=357491 Reopening to deal with the error Tony caught. Tony, can you apply the included patch and verify it works for you okay? I added another if to track back a week if the calculation gets pushed past 366 days. That does fix your error. And if you can think of any other edge cases let me know. I think that should cover everything. And yes, the example for the OP is wrong:: >>> datetime.date(2004, 10, 11).strftime("%Y %U %w") '2004 41 1' ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-18 13:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 11:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 09:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Tue Oct 19 12:39:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 12:39:31 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-13 00:04 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:39 Message: Logged In: YES user_id=1038590 Taking out the "add one to the week" condition, I get the following for 1906 and 1917: ./python -c "import time; print time.strptime('1906 53 1', '%Y %W %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1906 53 1', '%Y %U %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %W %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %U %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) So, I'm more than a little curious about the details of the "bug" that was being fixed. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:03 Message: Logged In: YES user_id=1038590 Scratch the last comment - I missed the wee_of_year adjustment at the top of the function. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 19:57 Message: Logged In: YES user_id=1038590 The calculation of 'preceeding_days' looks incorrect. It assumes that the week starts on day 0 - it needs to adjust for when "week_of year_start" is 6. ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-19 17:31 Message: Logged In: YES user_id=552329 Patch works for that case, yes. However... [from python-dev] > the test case I was using that triggered > the need for that is 1906-12-31 > which is a Monday but changes what week > it is based on whether you use U or W. > which makes no sense since both U and W > should consider it the same week. > Had the same result for 1917-12-31. Something's still not right here (this is with the patch): >>> time.strptime("1917 51 1", "%Y %U %w") (1917, 12, 17, 0, 0, 0, 0, 351, -1) >>> time.strptime("1917 52 1", "%Y %U %w") (1917, 12, 31, 0, 0, 0, 0, 365, -1) 1917 both started and ended on a Monday, so there are 53 U weeks, right? (where the last 'week' is only one day). So 31/12/1917 should be U=53, W=52 ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-19 04:45 Message: Logged In: YES user_id=357491 Reopening to deal with the error Tony caught. Tony, can you apply the included patch and verify it works for you okay? I added another if to track back a week if the calculation gets pushed past 366 days. That does fix your error. And if you can think of any other edge cases let me know. I think that should cover everything. And yes, the example for the OP is wrong:: >>> datetime.date(2004, 10, 11).strftime("%Y %U %w") '2004 41 1' ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-18 13:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 11:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 09:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Tue Oct 19 12:42:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 12:42:12 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-13 00:04 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:42 Message: Logged In: YES user_id=1038590 I forgot to add that I'm on Linux (Fedora Core 3 Test 1 + misc updates) ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:39 Message: Logged In: YES user_id=1038590 Taking out the "add one to the week" condition, I get the following for 1906 and 1917: ./python -c "import time; print time.strptime('1906 53 1', '%Y %W %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1906 53 1', '%Y %U %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %W %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %U %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) So, I'm more than a little curious about the details of the "bug" that was being fixed. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:03 Message: Logged In: YES user_id=1038590 Scratch the last comment - I missed the wee_of_year adjustment at the top of the function. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 19:57 Message: Logged In: YES user_id=1038590 The calculation of 'preceeding_days' looks incorrect. It assumes that the week starts on day 0 - it needs to adjust for when "week_of year_start" is 6. ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-19 17:31 Message: Logged In: YES user_id=552329 Patch works for that case, yes. However... [from python-dev] > the test case I was using that triggered > the need for that is 1906-12-31 > which is a Monday but changes what week > it is based on whether you use U or W. > which makes no sense since both U and W > should consider it the same week. > Had the same result for 1917-12-31. Something's still not right here (this is with the patch): >>> time.strptime("1917 51 1", "%Y %U %w") (1917, 12, 17, 0, 0, 0, 0, 351, -1) >>> time.strptime("1917 52 1", "%Y %U %w") (1917, 12, 31, 0, 0, 0, 0, 365, -1) 1917 both started and ended on a Monday, so there are 53 U weeks, right? (where the last 'week' is only one day). So 31/12/1917 should be U=53, W=52 ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-19 04:45 Message: Logged In: YES user_id=357491 Reopening to deal with the error Tony caught. Tony, can you apply the included patch and verify it works for you okay? I added another if to track back a week if the calculation gets pushed past 366 days. That does fix your error. And if you can think of any other edge cases let me know. I think that should cover everything. And yes, the example for the OP is wrong:: >>> datetime.date(2004, 10, 11).strftime("%Y %U %w") '2004 41 1' ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-18 13:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 11:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 09:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Tue Oct 19 12:50:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 12:53:28 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-13 00:04 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:50 Message: Logged In: YES user_id=1038590 Any bug is in datetime.date, not strptime. I tried datetime.date(1917, 12, 31).strftime("%Y %U %w") and saw Brett's incorrect calculation of the week. ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %W %w')" 1917 53 1 ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %U %w')" 1917 52 1 December 31 should be week 53 under %U as well. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:42 Message: Logged In: YES user_id=1038590 I forgot to add that I'm on Linux (Fedora Core 3 Test 1 + misc updates) ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:39 Message: Logged In: YES user_id=1038590 Taking out the "add one to the week" condition, I get the following for 1906 and 1917: ./python -c "import time; print time.strptime('1906 53 1', '%Y %W %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1906 53 1', '%Y %U %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %W %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %U %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) So, I'm more than a little curious about the details of the "bug" that was being fixed. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:03 Message: Logged In: YES user_id=1038590 Scratch the last comment - I missed the wee_of_year adjustment at the top of the function. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 19:57 Message: Logged In: YES user_id=1038590 The calculation of 'preceeding_days' looks incorrect. It assumes that the week starts on day 0 - it needs to adjust for when "week_of year_start" is 6. ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-19 17:31 Message: Logged In: YES user_id=552329 Patch works for that case, yes. However... [from python-dev] > the test case I was using that triggered > the need for that is 1906-12-31 > which is a Monday but changes what week > it is based on whether you use U or W. > which makes no sense since both U and W > should consider it the same week. > Had the same result for 1917-12-31. Something's still not right here (this is with the patch): >>> time.strptime("1917 51 1", "%Y %U %w") (1917, 12, 17, 0, 0, 0, 0, 351, -1) >>> time.strptime("1917 52 1", "%Y %U %w") (1917, 12, 31, 0, 0, 0, 0, 365, -1) 1917 both started and ended on a Monday, so there are 53 U weeks, right? (where the last 'week' is only one day). So 31/12/1917 should be U=53, W=52 ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-19 04:45 Message: Logged In: YES user_id=357491 Reopening to deal with the error Tony caught. Tony, can you apply the included patch and verify it works for you okay? I added another if to track back a week if the calculation gets pushed past 366 days. That does fix your error. And if you can think of any other edge cases let me know. I think that should cover everything. And yes, the example for the OP is wrong:: >>> datetime.date(2004, 10, 11).strftime("%Y %U %w") '2004 41 1' ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-18 13:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 11:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 09:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Tue Oct 19 14:18:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 14:19:04 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 23:04 Message generated for change (Comment added) made by quiver You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-19 21:18 Message: Logged In: YES user_id=671362 > December 31 should be week 53 under %U as well. I doubt it. Year 1917 begins on Monday and ends on Monday. So "%U"(Sunday as the first day of the week) should return 52 and "W"(Monday as the first day of the week) should return 53. $ cal 1 1917 January 1917 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 $ cal 12 1917 December 1917 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 19:50 Message: Logged In: YES user_id=1038590 Any bug is in datetime.date, not strptime. I tried datetime.date(1917, 12, 31).strftime("%Y %U %w") and saw Brett's incorrect calculation of the week. ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %W %w')" 1917 53 1 ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %U %w')" 1917 52 1 December 31 should be week 53 under %U as well. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 19:42 Message: Logged In: YES user_id=1038590 I forgot to add that I'm on Linux (Fedora Core 3 Test 1 + misc updates) ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 19:39 Message: Logged In: YES user_id=1038590 Taking out the "add one to the week" condition, I get the following for 1906 and 1917: ./python -c "import time; print time.strptime('1906 53 1', '%Y %W %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1906 53 1', '%Y %U %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %W %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %U %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) So, I'm more than a little curious about the details of the "bug" that was being fixed. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 19:03 Message: Logged In: YES user_id=1038590 Scratch the last comment - I missed the wee_of_year adjustment at the top of the function. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 18:57 Message: Logged In: YES user_id=1038590 The calculation of 'preceeding_days' looks incorrect. It assumes that the week starts on day 0 - it needs to adjust for when "week_of year_start" is 6. ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-19 16:31 Message: Logged In: YES user_id=552329 Patch works for that case, yes. However... [from python-dev] > the test case I was using that triggered > the need for that is 1906-12-31 > which is a Monday but changes what week > it is based on whether you use U or W. > which makes no sense since both U and W > should consider it the same week. > Had the same result for 1917-12-31. Something's still not right here (this is with the patch): >>> time.strptime("1917 51 1", "%Y %U %w") (1917, 12, 17, 0, 0, 0, 0, 351, -1) >>> time.strptime("1917 52 1", "%Y %U %w") (1917, 12, 31, 0, 0, 0, 0, 365, -1) 1917 both started and ended on a Monday, so there are 53 U weeks, right? (where the last 'week' is only one day). So 31/12/1917 should be U=53, W=52 ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-19 03:45 Message: Logged In: YES user_id=357491 Reopening to deal with the error Tony caught. Tony, can you apply the included patch and verify it works for you okay? I added another if to track back a week if the calculation gets pushed past 366 days. That does fix your error. And if you can think of any other edge cases let me know. I think that should cover everything. And yes, the example for the OP is wrong:: >>> datetime.date(2004, 10, 11).strftime("%Y %U %w") '2004 41 1' ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-18 12:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 10:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 08:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 10:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 10:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 10:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Tue Oct 19 15:18:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 15:18:36 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-13 00:04 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 23:18 Message: Logged In: YES user_id=1038590 Under %W, the first week contains a full 7 days, but it only contains 6 days under %U. Either way, we end up with a week 53 - it contains 1 day for %W, but 2 days for %U. In both cases, December 31st is the Monday of week 53. Unless I'm misunderstanding how this is meant to work, and Week 1 is always the first full week of the year, with an optional Week 0 before it (which would always have fewer than 7 days, and doesn't exist at all if the year and the week start on the same day). If I *am* misunderstanding, then that's the bug in strptime - it currently works in accordance with my understanding. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-19 22:18 Message: Logged In: YES user_id=671362 > December 31 should be week 53 under %U as well. I doubt it. Year 1917 begins on Monday and ends on Monday. So "%U"(Sunday as the first day of the week) should return 52 and "W"(Monday as the first day of the week) should return 53. $ cal 1 1917 January 1917 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 $ cal 12 1917 December 1917 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:50 Message: Logged In: YES user_id=1038590 Any bug is in datetime.date, not strptime. I tried datetime.date(1917, 12, 31).strftime("%Y %U %w") and saw Brett's incorrect calculation of the week. ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %W %w')" 1917 53 1 ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %U %w')" 1917 52 1 December 31 should be week 53 under %U as well. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:42 Message: Logged In: YES user_id=1038590 I forgot to add that I'm on Linux (Fedora Core 3 Test 1 + misc updates) ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:39 Message: Logged In: YES user_id=1038590 Taking out the "add one to the week" condition, I get the following for 1906 and 1917: ./python -c "import time; print time.strptime('1906 53 1', '%Y %W %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1906 53 1', '%Y %U %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %W %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %U %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) So, I'm more than a little curious about the details of the "bug" that was being fixed. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 20:03 Message: Logged In: YES user_id=1038590 Scratch the last comment - I missed the wee_of_year adjustment at the top of the function. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 19:57 Message: Logged In: YES user_id=1038590 The calculation of 'preceeding_days' looks incorrect. It assumes that the week starts on day 0 - it needs to adjust for when "week_of year_start" is 6. ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-19 17:31 Message: Logged In: YES user_id=552329 Patch works for that case, yes. However... [from python-dev] > the test case I was using that triggered > the need for that is 1906-12-31 > which is a Monday but changes what week > it is based on whether you use U or W. > which makes no sense since both U and W > should consider it the same week. > Had the same result for 1917-12-31. Something's still not right here (this is with the patch): >>> time.strptime("1917 51 1", "%Y %U %w") (1917, 12, 17, 0, 0, 0, 0, 351, -1) >>> time.strptime("1917 52 1", "%Y %U %w") (1917, 12, 31, 0, 0, 0, 0, 365, -1) 1917 both started and ended on a Monday, so there are 53 U weeks, right? (where the last 'week' is only one day). So 31/12/1917 should be U=53, W=52 ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-19 04:45 Message: Logged In: YES user_id=357491 Reopening to deal with the error Tony caught. Tony, can you apply the included patch and verify it works for you okay? I added another if to track back a week if the calculation gets pushed past 366 days. That does fix your error. And if you can think of any other edge cases let me know. I think that should cover everything. And yes, the example for the OP is wrong:: >>> datetime.date(2004, 10, 11).strftime("%Y %U %w") '2004 41 1' ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-18 13:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 11:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 09:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-13 11:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Tue Oct 19 17:18:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 17:18:44 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 11:18 Message: Logged In: YES user_id=6380 This looks fine, and can be checked in. There's one minor remaining issue, but I don't see how to fix it without changing the signature for the onerror function that os.walk() takes, and that's a no-no: when the problem is a directory permission preventing os.listdir() to work, the path passed to the rmtree caller's onerror function is always the toplevel path, rather than the directory that couldn't be listed. The real path is included in the error message. (Hmm, perhaps you could get it out of there? I think it's an attribute of the exception object; but be careful, it may not exist.) It's unfortunate that the error argument to the onerror function is a sys.exc_info() triple rather than an exception object (calling sys.exc_info() is expensive), but that's an API choice from the past that we can't change until Python 3.0. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 04:57 Message: Logged In: YES user_id=469548 A new patch is attached. I factored out the handle_call_error method so exception handling is more local. Also, errors in os.walk were already being handled using the _raise_err/except OSError combination, but we now do it using the onerror argument to os.walk. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:42 Message: Logged In: YES user_id=6380 (Actually, os.walk has an onerror argument, it just uses a different convention and default, so we could thread this through easily.) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 18:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 02:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Tue Oct 19 20:58:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 20:58:18 2004 Subject: [ python-Bugs-1049826 ] zlib module documentation wrong version Message-ID: Bugs item #1049826, was opened at 2004-10-19 04:12 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049826&group_id=5470 Category: Documentation Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Wolfgang Langner (tds33) >Assigned to: A.M. Kuchling (akuchling) Summary: zlib module documentation wrong version Initial Comment: The documentation of the zlib module is a little bit out of date: Version 1.1.3 is the most recent version as of September 2000. This is wrong, the most recent version of zlib is: 1.2.1 Change it to: Version 1.2.1 is the most recent version as of October 2004. source: http://www.python.org/dev/doc/devel/lib/module-zlib.html I hope python 2.4 Release will use this version. (it is out since end 2003) ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-10-19 14:58 Message: Logged In: YES user_id=11375 Fixed in the text for both 2.4 and 2.3-maint; thanks for pointing this out. Note that on Unix platforms, Python will be compiled to use whatever version of zlib is on the system; it doesn't include its own copy. I think the Windows binary version of 2.3.4 used zlib1.1.4; I don't know what the binary of 2.4 uses. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049826&group_id=5470 From noreply at sourceforge.net Tue Oct 19 21:00:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 21:00:32 2004 Subject: [ python-Bugs-1048820 ] Only "Overwrite" mode possible with curses.textpad.Textbox Message-ID: Bugs item #1048820, was opened at 2004-10-17 15:58 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048820&group_id=5470 >Category: Python Library >Group: Python 2.5 Status: Open Resolution: None Priority: 5 Submitted By: Stefan Heimann (stefanheimann) >Assigned to: A.M. Kuchling (akuchling) Summary: Only "Overwrite" mode possible with curses.textpad.Textbox Initial Comment: When entering a character in a curses.textpad.Textbox, the character under the cursor is deleted instead of being shifted when position. I attached a patch which implements the "Insert" mode. Maybe you should make it customizable wheter the Textbox should use the old "Overwrite" or the new "Insert" mode. I'm using python 2.3.3 ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-10-19 15:00 Message: Logged In: YES user_id=11375 This would be a new feature, so it's getting marked for 2.5. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048820&group_id=5470 From noreply at sourceforge.net Tue Oct 19 21:07:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 21:07:23 2004 Subject: [ python-Bugs-1048816 ] C-k does not work for curses.textpad.Textbox Message-ID: Bugs item #1048816, was opened at 2004-10-17 15:51 Message generated for change (Settings changed) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048816&group_id=5470 >Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Stefan Heimann (stefanheimann) >Assigned to: A.M. Kuchling (akuchling) Summary: C-k does not work for curses.textpad.Textbox Initial Comment: The bug can be reproduced by starting the textpad module directly (by executing 'python textpad.py' inside the curses directory). The you should type: 1 2 3 C-a C-k You would expected an empty textbox (because you first enter the string '123', the go the the beginning of the line and then delete until the end of the line). But the textbox contains the text '12'. My Python version is 2.3.3, a patch is attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048816&group_id=5470 From noreply at sourceforge.net Tue Oct 19 21:20:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 21:20:50 2004 Subject: [ python-Bugs-1049826 ] zlib module documentation wrong version Message-ID: Bugs item #1049826, was opened at 2004-10-19 17:12 Message generated for change (Comment added) made by quiver You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049826&group_id=5470 Category: Documentation Group: None Status: Closed Resolution: Fixed Priority: 5 Submitted By: Wolfgang Langner (tds33) Assigned to: A.M. Kuchling (akuchling) Summary: zlib module documentation wrong version Initial Comment: The documentation of the zlib module is a little bit out of date: Version 1.1.3 is the most recent version as of September 2000. This is wrong, the most recent version of zlib is: 1.2.1 Change it to: Version 1.2.1 is the most recent version as of October 2004. source: http://www.python.org/dev/doc/devel/lib/module-zlib.html I hope python 2.4 Release will use this version. (it is out since end 2003) ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-20 04:20 Message: Logged In: YES user_id=671362 > I think the Windows binary version of 2.3.4 used zlib1.1.4; correct. > I don't know what the binary of 2.4 uses. >>> import zlib >>> zlib.ZLIB_VERSION '1.2.1' ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-10-20 03:58 Message: Logged In: YES user_id=11375 Fixed in the text for both 2.4 and 2.3-maint; thanks for pointing this out. Note that on Unix platforms, Python will be compiled to use whatever version of zlib is on the system; it doesn't include its own copy. I think the Windows binary version of 2.3.4 used zlib1.1.4; I don't know what the binary of 2.4 uses. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049826&group_id=5470 From noreply at sourceforge.net Tue Oct 19 21:24:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 21:24:26 2004 Subject: [ python-Bugs-1048900 ] Registry not updated by 2.4b1 installer Message-ID: Bugs item #1048900, was opened at 2004-10-18 01:00 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048900&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: B Sizer (kylotan) Assigned to: Nobody/Anonymous (nobody) Summary: Registry not updated by 2.4b1 installer Initial Comment: Running the python-2.4b1.msi installer on Win98 SE appeared to proceeded as expected. However, the registry appears not to have been updated and any add-on Python library installers are still asking to be put in my old Python 2.3 directory, effectively preventing me from using those libraries with 2.4b, or any scripts that rely on them. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-19 21:24 Message: Logged In: YES user_id=21627 I cannot reproduce this on XPSP2. Can you please use regedit.exe to find out a) whether HKEY_LOCAL_MACHINE\Software\Python exists, and b) if so, what its subkeys are? In particular, is anything about 2.4 present? If installers choose 2.3, it might be that they have extension modules which need to be recompiled for 2.4, in which case the installer specifically looks for a 2.3 installation, rather than the latest installation. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048900&group_id=5470 From noreply at sourceforge.net Tue Oct 19 21:30:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 21:31:02 2004 Subject: [ python-Bugs-1049003 ] Install-time module compilation fails with 2.4b1 Message-ID: Bugs item #1049003, was opened at 2004-10-18 06:45 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049003&group_id=5470 Category: Installation Group: Python 2.4 Status: Open Resolution: None >Priority: 9 Submitted By: Tony Meyer (anadelonbrin) >Assigned to: Martin v. L?wis (loewis) Summary: Install-time module compilation fails with 2.4b1 Initial Comment: If I tick the "compile modules" box in the advanced options of the 2.4b1 msi installer, it dies part way through this (it scrolls too fast for me to see exactly where, though). The installer then gives a dialog with: """ There is a problem with this Windows Installer package. A program run as part of the setup did not finish as expected. Contact your support personnel or package vendor. """ And goes to a "install did not work" final box, rather than the regular 'all good' one. This happens with Win2K Pro and WinXP Pro SP1. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049003&group_id=5470 From noreply at sourceforge.net Tue Oct 19 21:31:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 21:31:32 2004 Subject: [ python-Bugs-1048900 ] Registry not updated by 2.4b1 installer Message-ID: Bugs item #1048900, was opened at 2004-10-18 01:00 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048900&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: B Sizer (kylotan) >Assigned to: Martin v. L?wis (loewis) Summary: Registry not updated by 2.4b1 installer Initial Comment: Running the python-2.4b1.msi installer on Win98 SE appeared to proceeded as expected. However, the registry appears not to have been updated and any add-on Python library installers are still asking to be put in my old Python 2.3 directory, effectively preventing me from using those libraries with 2.4b, or any scripts that rely on them. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-19 21:24 Message: Logged In: YES user_id=21627 I cannot reproduce this on XPSP2. Can you please use regedit.exe to find out a) whether HKEY_LOCAL_MACHINE\Software\Python exists, and b) if so, what its subkeys are? In particular, is anything about 2.4 present? If installers choose 2.3, it might be that they have extension modules which need to be recompiled for 2.4, in which case the installer specifically looks for a 2.3 installation, rather than the latest installation. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048900&group_id=5470 From noreply at sourceforge.net Tue Oct 19 21:30:02 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 21:31:39 2004 Subject: [ python-Bugs-1049615 ] smeared title when installing Message-ID: Bugs item #1049615, was opened at 2004-10-19 00:09 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 Category: Installation Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: smeared title when installing Initial Comment: Python 2.4b1 on Windows XP. 1280x800. The "python: for windows" heading looks smeared. Other graphical elements also look somewhat smeared, but it isn't as much of a problem when it isn't text to begin with. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-19 21:30 Message: Logged In: YES user_id=21627 I don't know what to do about this. Any suggestion appreciated; without suggestions, this likely is what 2.4 will look like. ---------------------------------------------------------------------- Comment By: Mike Brauwerman (brauwerman) Date: 2004-10-19 01:09 Message: Logged In: YES user_id=71983 The "smearing" is the effect of a small bitmap picture, with aliased text, blown up (stretched) to fit into a large space. The graphic has fat pixels, which look a little funny, but they look better if you squint :-) ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-10-19 00:16 Message: Logged In: YES user_id=764593 I used a .bmp to avoid additional info loss. Zipped file is only 20K, but unzips to over a meg. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 From noreply at sourceforge.net Tue Oct 19 21:35:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 21:35:53 2004 Subject: [ python-Bugs-1048681 ] Add pythonX.dll to KnownDlls Message-ID: Bugs item #1048681, was opened at 2004-10-17 16:29 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048681&group_id=5470 Category: Installation Group: Feature Request >Status: Closed >Resolution: Rejected Priority: 5 Submitted By: Miki Tebeka (tebeka) Assigned to: Nobody/Anonymous (nobody) Summary: Add pythonX.dll to KnownDlls Initial Comment: For Windows systems, when installing pythonX.dll to system32 it should be added to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\KnownDLLs This will speed up the initial loading of python. Many applications the runs a lot of python processes (such as cgi ...) will run much faster. See http://support.microsoft.com/kb/q164501/ for more details. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-19 21:35 Message: Logged In: YES user_id=21627 This change is not desirable. It will cause \windows\system32.dll\python24.dll to be used even if the application comes with its own, potentially modified version of python24.dll. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048681&group_id=5470 From noreply at sourceforge.net Tue Oct 19 21:37:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 21:37:47 2004 Subject: [ python-Bugs-1048900 ] Registry not updated by 2.4b1 installer Message-ID: Bugs item #1048900, was opened at 2004-10-17 23:00 Message generated for change (Comment added) made by kylotan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048900&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: B Sizer (kylotan) >Assigned to: Nobody/Anonymous (nobody) Summary: Registry not updated by 2.4b1 installer Initial Comment: Running the python-2.4b1.msi installer on Win98 SE appeared to proceeded as expected. However, the registry appears not to have been updated and any add-on Python library installers are still asking to be put in my old Python 2.3 directory, effectively preventing me from using those libraries with 2.4b, or any scripts that rely on them. ---------------------------------------------------------------------- >Comment By: B Sizer (kylotan) Date: 2004-10-19 19:37 Message: Logged In: YES user_id=206950 Ok, maybe the summary is misleading. The HKEY_LOCAL_MACHINE\Software\Python subtree appears to exist, with PythonCore and Pythonwin subtrees under it. Under PythonCore is a 2.3 subtree and a 2.4 subtree. This implies the installations work side-by-side. I found nothing explaining whether to uninstall a previous version or not in the docs on Python.org, or whether it is possible to upgrade a previous version by installing into the old directory, etc. In particular http://www.python.org/2.4/msi.html avoids this issue. I had guessed that maybe the move to some sort of official packaging system was because it would enable these automatic updates, somewhat akin to RPM, but obviously not. I can't comment on how the installers for add-on libraries on Win32 are made (although it seems to use a standard package), only that I get no option to install them into the 2.4 directory, at least given the 3 I've just tried. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-19 19:24 Message: Logged In: YES user_id=21627 I cannot reproduce this on XPSP2. Can you please use regedit.exe to find out a) whether HKEY_LOCAL_MACHINE\Software\Python exists, and b) if so, what its subkeys are? In particular, is anything about 2.4 present? If installers choose 2.3, it might be that they have extension modules which need to be recompiled for 2.4, in which case the installer specifically looks for a 2.3 installation, rather than the latest installation. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048900&group_id=5470 From noreply at sourceforge.net Tue Oct 19 21:39:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 21:39:19 2004 Subject: [ python-Bugs-1045893 ] warning '"_POSIX_C_SOURCE" redefined' when include Python.h Message-ID: Bugs item #1045893, was opened at 2004-10-13 07:06 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045893&group_id=5470 Category: Extension Modules Group: Python 2.3 >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Eugene Sizikov (eugene_beast) Assigned to: Nobody/Anonymous (nobody) Summary: warning '"_POSIX_C_SOURCE" redefined' when include Python.h Initial Comment: """ In file included from /usr/include/python2.3/Python.h:8, from script.h:21, from server.cpp:23: /usr/include/python2.3/pyconfig.h:847:1: warning: "_POSIX_C_SOURCE" redefined In file included from /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/i386-redhat-linux/bits/os_defines.h:39, from /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/i386-redhat-linux/bits/c++config.h:35, from /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/iostream:44, from server.cpp:18: /usr/include/features.h:131:1: warning: this is the location of the previous definition ""' The above message is shown whenever program (extending or embedding python with C/C++) using Python.h was compilled. OS: Fedora Core 2 compiller: GCC 3.4.0 libc: glibc-2.3 Python: 2.3.3 ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-19 21:39 Message: Logged In: YES user_id=21627 This is a bug in your program. See http://docs.python.org/api/includes.html ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045893&group_id=5470 From noreply at sourceforge.net Tue Oct 19 21:50:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 21:50:45 2004 Subject: [ python-Bugs-1048816 ] C-k does not work for curses.textpad.Textbox Message-ID: Bugs item #1048816, was opened at 2004-10-17 15:51 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048816&group_id=5470 Category: Python Library Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Stefan Heimann (stefanheimann) Assigned to: A.M. Kuchling (akuchling) Summary: C-k does not work for curses.textpad.Textbox Initial Comment: The bug can be reproduced by starting the textpad module directly (by executing 'python textpad.py' inside the curses directory). The you should type: 1 2 3 C-a C-k You would expected an empty textbox (because you first enter the string '123', the go the the beginning of the line and then delete until the end of the line). But the textbox contains the text '12'. My Python version is 2.3.3, a patch is attached. ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-10-19 15:50 Message: Logged In: YES user_id=11375 Well spotted; thanks! I've applied your patch to 2.4-CVS and to the 2.3-maint branch, so the fix should be in Python 2.3.5, whenever it gets released. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048816&group_id=5470 From noreply at sourceforge.net Tue Oct 19 21:51:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 21:51:36 2004 Subject: [ python-Bugs-1050268 ] rfc822.parseaddr is broken, breaks sendmail call in smtplib Message-ID: Bugs item #1050268, was opened at 2004-10-19 12:51 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050268&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Scott Dossey (sdossey) Assigned to: Nobody/Anonymous (nobody) Summary: rfc822.parseaddr is broken, breaks sendmail call in smtplib Initial Comment: THe following email address is legal according to RFC: <"\quoted string\ somebody"@somedomain.com"> I've got a python mail handling back end server that handles mail coming in from Postfix. Postfix properly accepts mail of this type, but when it goes to relay this through my Python server it fails. The problem is inside smtplib.py inside "quoteaddr". Here's a source code snippet: def quoteaddr(addr) """Quote a subset of the email addresses defined by RFC 821. Should be able to handle anything rfc822.parseaddr can handle. """ m = (None, None) try: m=rfc822.parseaddr(addr)[1] except AttributeError: pass if m == (None, None): # Indicates parse failure or AttributeError #something weird here.. punt -ddm return "<%s>" % addr Basically quoteaddr ends up wrapping whatever parseaddr returns to it in brackets and sending that out on the wire for the RCPT TO command. however, if I call rfc822.parseaddr it does bizarre things to email addresses. For instance the following calls all yield the same thing (some not surprisingly): rfc822.parseaddr('""test" test"@test.com') rfc822.parseaddr('"\test\ test"@test.com') rfc822.parseaddr('"\"test\" test"@test.com') rfc822.parseaddr('"\\test\\ test"@test/com') the above all yield: ('', '""test" test"@test.com') rfc822.parseaddr('"\\"test\\" test"@test/com') yields the VERY surprising result: ('', '"\"test\\" test"@test.com') I submitted this as a new bug report even though there are two similar bugs regarding parseAddr because it is a slightly separate issue. -Scott Dossey ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050268&group_id=5470 From noreply at sourceforge.net Tue Oct 19 22:10:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 22:10:16 2004 Subject: [ python-Bugs-1003471 ] Python 1.5.2 security vulnerability still present in 2.3.4 Message-ID: Bugs item #1003471, was opened at 2004-08-04 14:42 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1003471&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Kirby Kuehl (vacuum) Assigned to: Nobody/Anonymous (nobody) Summary: Python 1.5.2 security vulnerability still present in 2.3.4 Initial Comment: First off, I realize that Python 1.5.2 is old, I am just reporting it to be thorough. I was doing a security audit of an application that used this old version of python and found the following bug in the joinpath function. #0 0xff133bf0 in strncpy () from /usr/lib/libc.so.1 (gdb) bt #0 0xff133bf0 in strncpy () from /usr/lib/libc.so.1 (gdb) bt #0 0xff133bf0 in strncpy () from /usr/lib/libc.so.1 #1 0x000304e0 in joinpath (buffer=0x83528 'A' ..., stuff=0x84140 'A' ...) at ./getpath.c:255 #2 0x00030604 in search_for_prefix (argv0_path=0xffbff530 "/opt/OPSW/bin", home=0xff3a0840 'A' ...) at ./getpath.c:300 #3 0x00030a48 in calculate_path () at ./getpath.c:481 #4 0x00030e6c in Py_GetProgramFullPath () at ./getpath.c:634 #5 0x0002dcac in _PySys_Init () at sysmodule.c:413 #6 0x0002b414 in Py_Initialize () at pythonrun.c:142 #7 0x0001755c in Py_Main (argc=1, argv=0xffbffcdc) at main.c:245 #8 0x000171f0 in main (argc=1, argv=0xffbffcdc) at python.c:12 #8 0x000171f0 in main (argc=1, argv=0xffbffcdc) at python.c:12 (gdb) frame 1 #1 0x000304e0 in joinpath (buffer=0x83528 'A' ..., stuff=0x84140 'A' ...) at ./getpath.c:255 255 ./getpath.c: No such file or directory. in ./getpath.c snippet from Python-1.5.2/Modules/getpath.c : static void joinpath(buffer, stuff) char *buffer; char *stuff; { int n, k; if (stuff[0] == SEP) n = 0; else { n = strlen(buffer); if (n > 0 && buffer[n-1] != SEP && n < MAXPATHLEN) buffer[n++] = SEP; } k = strlen(stuff); if (n + k > MAXPATHLEN) k = MAXPATHLEN - n; strncpy(buffer+n, stuff, k); buffer[n+k] = '\0'; } further examining the backtrace: (gdb) print n $1 = 4999 (gdb) print k $2 = -3975 (gdb) print buffer $4 = 0x83528 'A' ... (gdb) print stuff $5 = 0x84140 'A' ... if (n + k > MAXPATHLEN) /* NOTE: MAXPATHLEN is 1024 */ k = MAXPATHLEN - n; /* NOTE: here k is 1024 - 4999 which is the -3975 */ Which of course crashes in strncpy(buffer+n, stuff, k); Thanks, Kirby Kuehl ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-10-19 16:10 Message: Logged In: YES user_id=11375 The URL mentioned for sharefuzz seems to be out of date; the correct location is now http://sourceforge.net/projects/sharefuzz/ . ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-09 14:43 Message: Logged In: YES user_id=31435 vacuum, are you saying that sharefuzz provokes an actual bug in Python 2.4a2? In Python 2.3.4? ---------------------------------------------------------------------- Comment By: Kirby Kuehl (vacuum) Date: 2004-08-09 14:26 Message: Logged In: YES user_id=116409 Use the sharefuzz utility that I referenced in another comment. Then call any function that will invoke the joinpath function call. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-09 13:56 Message: Logged In: YES user_id=31435 The patch didn't intend to fix a bug that's known to exist. Python 2.0 presumably fixed the bugs here, but in a shaky way, relying on dozens of distinct call sites to establish an undocumented precondition. joinpath() suffers a buffer overrun if the precondition isn't met. The patch verifies that the precondition *is* met, killing the Python process if it's not. That should never happen, but joinpath() is called on so many distinct code paths that "eyeball analysis" is inadequate. To test the patch, change any of joinpath's call sites to violate the precondition, and see that Python dies then. The OP suggested a driver to provoke an envar attack, in the 2004-08-07 14:15 comment. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-08-09 13:39 Message: Logged In: YES user_id=11375 How would I go about testing the patch? Where can I find code that tickles the bug? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 21:01 Message: Logged In: YES user_id=31435 I checked in the attached patch too. If there's ever another release in the 2.3 line, these patches would make decent backport candidates. ---------------------------------------------------------------------- Comment By: Kirby Kuehl (vacuum) Date: 2004-08-07 18:19 Message: Logged In: YES user_id=116409 As tim_one poin ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 15:15 Message: Logged In: YES user_id=31435 Unassigned myself, since I did all I can here. Someone on Linux should test the patch I attached. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 15:13 Message: Logged In: YES user_id=31435 I checked in the change to PC/getpathp.c, which is used on Windows. I'm attaching a patch to Modules/getpath.c, which isn't used on Windows (so I can't test it). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 14:44 Message: Logged In: YES user_id=31435 I'm going to add the panic-check I suggested -- this code is too complicated to have any confidence in "eyeball analysis" over time. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 14:39 Message: Logged In: YES user_id=31435 Yup, and rev 1.32 intended to plug the envar attack. ---------------------------------------------------------------------- Comment By: Kirby Kuehl (vacuum) Date: 2004-08-07 14:15 Message: Logged In: YES user_id=116409 by exporting environment variables. Check sharefuzz which can be found here: http://www.atstake.com/research/tools/index.html#vulnerability_scanning ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 12:43 Message: Logged In: YES user_id=31435 Pretty compilcated internally. A very similar joinpath still exists. While it's not documented or checked, joinpath's code clearly *assumes* strlen(buffer) <= MAXPATHLEN on entry. But it's called from 25 places, and it's not immediately obvious that all call sites guarantee this on all paths. Rev 1.32 of getpath (for Python 2.0) *intended* to fix buffer overflow problems, mostly by changing unsafe strcpy() calls to strncpy() calls. This is delicate, though. I'd be a lot happier if joinpath verified that n <= MAXPATHLEN on entry, and called Py_FatalError() if not so (converting a buffer overrun into a bug report). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-08-07 11:55 Message: Logged In: YES user_id=6656 I'm not sure I understand. How do you get n to be so huge? At any rate, I don't think the bug is present in modern Python, but until I understand how this one was triggered, I'm not prepared to be sure... ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1003471&group_id=5470 From noreply at sourceforge.net Tue Oct 19 22:10:08 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 22:10:17 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 07:04 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-10-19 13:10 Message: Logged In: YES user_id=357491 Nick is seeing things the way I thought they are supposed to be; how can ending on a Monday make any difference for %U or %W? Both start at after the beginning of the week, right? Well, as George was pointing out, it shifts everything by a full week depending on the directive when the year starts on a Monday. So under %U week 1 doesn't start until the first Sunday, while under %W week 1 starts on Jan 1. So the issue is when a year starts and end on a Monday, we are looking at Monday, but %U is used. So the new patch attempts to handle this case. I am now testing against the following dates: test_helper((1901, 1, 3), "week 0") test_helper((1901, 1, 8), "common case") test_helper((1901, 1, 13), "day on Sunday") test_helper((1901, 1, 14), "day on Monday") test_helper((1905, 1, 1), "Jan 1 on Sunday") test_helper((1906, 1, 1), "Jan 1 on Monday") test_helper((1906, 1, 7), "first Sunday in a year starting on Monday") test_helper((1905, 12, 31), "Dec 31 on Sunday") test_helper((1906, 12, 31), "Dec 31 on Monday") test_helper((2008, 12, 29), "Monday in the last week of the year") test_helper((2008, 12, 22), "Monday in the second-to-last week of the " "year") test_helper((1978, 10, 23), "randomly chosen date") test_helper((2004, 12, 18), "randomly chosen date") test_helper((1978, 10, 23), "year starting and ending on Monday while " "date not on Sunday or Monday") test_helper((1917, 12, 17), "year starting and ending on Monday with " "a Monday not at the beginning or end " "of the year") test_helper((1917, 12, 31), "Dec 31 on Monday with year starting and " "ending on Monday") ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 06:18 Message: Logged In: YES user_id=1038590 Under %W, the first week contains a full 7 days, but it only contains 6 days under %U. Either way, we end up with a week 53 - it contains 1 day for %W, but 2 days for %U. In both cases, December 31st is the Monday of week 53. Unless I'm misunderstanding how this is meant to work, and Week 1 is always the first full week of the year, with an optional Week 0 before it (which would always have fewer than 7 days, and doesn't exist at all if the year and the week start on the same day). If I *am* misunderstanding, then that's the bug in strptime - it currently works in accordance with my understanding. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-19 05:18 Message: Logged In: YES user_id=671362 > December 31 should be week 53 under %U as well. I doubt it. Year 1917 begins on Monday and ends on Monday. So "%U"(Sunday as the first day of the week) should return 52 and "W"(Monday as the first day of the week) should return 53. $ cal 1 1917 January 1917 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 $ cal 12 1917 December 1917 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:50 Message: Logged In: YES user_id=1038590 Any bug is in datetime.date, not strptime. I tried datetime.date(1917, 12, 31).strftime("%Y %U %w") and saw Brett's incorrect calculation of the week. ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %W %w')" 1917 53 1 ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %U %w')" 1917 52 1 December 31 should be week 53 under %U as well. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:42 Message: Logged In: YES user_id=1038590 I forgot to add that I'm on Linux (Fedora Core 3 Test 1 + misc updates) ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:39 Message: Logged In: YES user_id=1038590 Taking out the "add one to the week" condition, I get the following for 1906 and 1917: ./python -c "import time; print time.strptime('1906 53 1', '%Y %W %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1906 53 1', '%Y %U %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %W %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %U %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) So, I'm more than a little curious about the details of the "bug" that was being fixed. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:03 Message: Logged In: YES user_id=1038590 Scratch the last comment - I missed the wee_of_year adjustment at the top of the function. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 02:57 Message: Logged In: YES user_id=1038590 The calculation of 'preceeding_days' looks incorrect. It assumes that the week starts on day 0 - it needs to adjust for when "week_of year_start" is 6. ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-19 00:31 Message: Logged In: YES user_id=552329 Patch works for that case, yes. However... [from python-dev] > the test case I was using that triggered > the need for that is 1906-12-31 > which is a Monday but changes what week > it is based on whether you use U or W. > which makes no sense since both U and W > should consider it the same week. > Had the same result for 1917-12-31. Something's still not right here (this is with the patch): >>> time.strptime("1917 51 1", "%Y %U %w") (1917, 12, 17, 0, 0, 0, 0, 351, -1) >>> time.strptime("1917 52 1", "%Y %U %w") (1917, 12, 31, 0, 0, 0, 0, 365, -1) 1917 both started and ended on a Monday, so there are 53 U weeks, right? (where the last 'week' is only one day). So 31/12/1917 should be U=53, W=52 ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 11:45 Message: Logged In: YES user_id=357491 Reopening to deal with the error Tony caught. Tony, can you apply the included patch and verify it works for you okay? I added another if to track back a week if the calculation gets pushed past 366 days. That does fix your error. And if you can think of any other edge cases let me know. I think that should cover everything. And yes, the example for the OP is wrong:: >>> datetime.date(2004, 10, 11).strftime("%Y %U %w") '2004 41 1' ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-17 20:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-17 18:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-17 16:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Tue Oct 19 22:18:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 22:18:44 2004 Subject: [ python-Bugs-1003471 ] Python 1.5.2 security vulnerability still present in 2.3.4 Message-ID: Bugs item #1003471, was opened at 2004-08-04 14:42 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1003471&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Kirby Kuehl (vacuum) Assigned to: Nobody/Anonymous (nobody) Summary: Python 1.5.2 security vulnerability still present in 2.3.4 Initial Comment: First off, I realize that Python 1.5.2 is old, I am just reporting it to be thorough. I was doing a security audit of an application that used this old version of python and found the following bug in the joinpath function. #0 0xff133bf0 in strncpy () from /usr/lib/libc.so.1 (gdb) bt #0 0xff133bf0 in strncpy () from /usr/lib/libc.so.1 (gdb) bt #0 0xff133bf0 in strncpy () from /usr/lib/libc.so.1 #1 0x000304e0 in joinpath (buffer=0x83528 'A' ..., stuff=0x84140 'A' ...) at ./getpath.c:255 #2 0x00030604 in search_for_prefix (argv0_path=0xffbff530 "/opt/OPSW/bin", home=0xff3a0840 'A' ...) at ./getpath.c:300 #3 0x00030a48 in calculate_path () at ./getpath.c:481 #4 0x00030e6c in Py_GetProgramFullPath () at ./getpath.c:634 #5 0x0002dcac in _PySys_Init () at sysmodule.c:413 #6 0x0002b414 in Py_Initialize () at pythonrun.c:142 #7 0x0001755c in Py_Main (argc=1, argv=0xffbffcdc) at main.c:245 #8 0x000171f0 in main (argc=1, argv=0xffbffcdc) at python.c:12 #8 0x000171f0 in main (argc=1, argv=0xffbffcdc) at python.c:12 (gdb) frame 1 #1 0x000304e0 in joinpath (buffer=0x83528 'A' ..., stuff=0x84140 'A' ...) at ./getpath.c:255 255 ./getpath.c: No such file or directory. in ./getpath.c snippet from Python-1.5.2/Modules/getpath.c : static void joinpath(buffer, stuff) char *buffer; char *stuff; { int n, k; if (stuff[0] == SEP) n = 0; else { n = strlen(buffer); if (n > 0 && buffer[n-1] != SEP && n < MAXPATHLEN) buffer[n++] = SEP; } k = strlen(stuff); if (n + k > MAXPATHLEN) k = MAXPATHLEN - n; strncpy(buffer+n, stuff, k); buffer[n+k] = '\0'; } further examining the backtrace: (gdb) print n $1 = 4999 (gdb) print k $2 = -3975 (gdb) print buffer $4 = 0x83528 'A' ... (gdb) print stuff $5 = 0x84140 'A' ... if (n + k > MAXPATHLEN) /* NOTE: MAXPATHLEN is 1024 */ k = MAXPATHLEN - n; /* NOTE: here k is 1024 - 4999 which is the -3975 */ Which of course crashes in strncpy(buffer+n, stuff, k); Thanks, Kirby Kuehl ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-10-19 16:18 Message: Logged In: YES user_id=11375 I haven't figured out how to get sharefuzz to work yet, but did notice one OS/2 specific overflow; around line 3250 of posixmodule.c, it defines char args[1024], and then does a strcpy from the COMSPEC env. var. into args. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-10-19 16:10 Message: Logged In: YES user_id=11375 The URL mentioned for sharefuzz seems to be out of date; the correct location is now http://sourceforge.net/projects/sharefuzz/ . ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-09 14:43 Message: Logged In: YES user_id=31435 vacuum, are you saying that sharefuzz provokes an actual bug in Python 2.4a2? In Python 2.3.4? ---------------------------------------------------------------------- Comment By: Kirby Kuehl (vacuum) Date: 2004-08-09 14:26 Message: Logged In: YES user_id=116409 Use the sharefuzz utility that I referenced in another comment. Then call any function that will invoke the joinpath function call. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-09 13:56 Message: Logged In: YES user_id=31435 The patch didn't intend to fix a bug that's known to exist. Python 2.0 presumably fixed the bugs here, but in a shaky way, relying on dozens of distinct call sites to establish an undocumented precondition. joinpath() suffers a buffer overrun if the precondition isn't met. The patch verifies that the precondition *is* met, killing the Python process if it's not. That should never happen, but joinpath() is called on so many distinct code paths that "eyeball analysis" is inadequate. To test the patch, change any of joinpath's call sites to violate the precondition, and see that Python dies then. The OP suggested a driver to provoke an envar attack, in the 2004-08-07 14:15 comment. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-08-09 13:39 Message: Logged In: YES user_id=11375 How would I go about testing the patch? Where can I find code that tickles the bug? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 21:01 Message: Logged In: YES user_id=31435 I checked in the attached patch too. If there's ever another release in the 2.3 line, these patches would make decent backport candidates. ---------------------------------------------------------------------- Comment By: Kirby Kuehl (vacuum) Date: 2004-08-07 18:19 Message: Logged In: YES user_id=116409 As tim_one poin ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 15:15 Message: Logged In: YES user_id=31435 Unassigned myself, since I did all I can here. Someone on Linux should test the patch I attached. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 15:13 Message: Logged In: YES user_id=31435 I checked in the change to PC/getpathp.c, which is used on Windows. I'm attaching a patch to Modules/getpath.c, which isn't used on Windows (so I can't test it). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 14:44 Message: Logged In: YES user_id=31435 I'm going to add the panic-check I suggested -- this code is too complicated to have any confidence in "eyeball analysis" over time. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 14:39 Message: Logged In: YES user_id=31435 Yup, and rev 1.32 intended to plug the envar attack. ---------------------------------------------------------------------- Comment By: Kirby Kuehl (vacuum) Date: 2004-08-07 14:15 Message: Logged In: YES user_id=116409 by exporting environment variables. Check sharefuzz which can be found here: http://www.atstake.com/research/tools/index.html#vulnerability_scanning ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 12:43 Message: Logged In: YES user_id=31435 Pretty compilcated internally. A very similar joinpath still exists. While it's not documented or checked, joinpath's code clearly *assumes* strlen(buffer) <= MAXPATHLEN on entry. But it's called from 25 places, and it's not immediately obvious that all call sites guarantee this on all paths. Rev 1.32 of getpath (for Python 2.0) *intended* to fix buffer overflow problems, mostly by changing unsafe strcpy() calls to strncpy() calls. This is delicate, though. I'd be a lot happier if joinpath verified that n <= MAXPATHLEN on entry, and called Py_FatalError() if not so (converting a buffer overrun into a bug report). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-08-07 11:55 Message: Logged In: YES user_id=6656 I'm not sure I understand. How do you get n to be so huge? At any rate, I don't think the bug is present in modern Python, but until I understand how this one was triggered, I'm not prepared to be sure... ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1003471&group_id=5470 From noreply at sourceforge.net Tue Oct 19 22:19:34 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 22:19:42 2004 Subject: [ python-Bugs-1048900 ] Registry not updated by 2.4b1 installer Message-ID: Bugs item #1048900, was opened at 2004-10-18 01:00 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048900&group_id=5470 Category: Windows >Group: Not a Bug >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: B Sizer (kylotan) Assigned to: Nobody/Anonymous (nobody) Summary: Registry not updated by 2.4b1 installer Initial Comment: Running the python-2.4b1.msi installer on Win98 SE appeared to proceeded as expected. However, the registry appears not to have been updated and any add-on Python library installers are still asking to be put in my old Python 2.3 directory, effectively preventing me from using those libraries with 2.4b, or any scripts that rely on them. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-19 22:19 Message: Logged In: YES user_id=21627 It appears that that everything works as it should. Side-by-side installation of different x.y versions is supported; different x.y.z versions cannot simultaneously be installed (IOW, you can have 2.3 and 2.4 on the same machine just fine). It further appears that the only indication of a problem that you have is that the third-party library doesn't offer installation for 2.4. This may or may not be correct, depending on that library - you should bring this up with the authors of that library. Closing this as not-a-bug. ---------------------------------------------------------------------- Comment By: B Sizer (kylotan) Date: 2004-10-19 21:37 Message: Logged In: YES user_id=206950 Ok, maybe the summary is misleading. The HKEY_LOCAL_MACHINE\Software\Python subtree appears to exist, with PythonCore and Pythonwin subtrees under it. Under PythonCore is a 2.3 subtree and a 2.4 subtree. This implies the installations work side-by-side. I found nothing explaining whether to uninstall a previous version or not in the docs on Python.org, or whether it is possible to upgrade a previous version by installing into the old directory, etc. In particular http://www.python.org/2.4/msi.html avoids this issue. I had guessed that maybe the move to some sort of official packaging system was because it would enable these automatic updates, somewhat akin to RPM, but obviously not. I can't comment on how the installers for add-on libraries on Win32 are made (although it seems to use a standard package), only that I get no option to install them into the 2.4 directory, at least given the 3 I've just tried. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-19 21:24 Message: Logged In: YES user_id=21627 I cannot reproduce this on XPSP2. Can you please use regedit.exe to find out a) whether HKEY_LOCAL_MACHINE\Software\Python exists, and b) if so, what its subkeys are? In particular, is anything about 2.4 present? If installers choose 2.3, it might be that they have extension modules which need to be recompiled for 2.4, in which case the installer specifically looks for a 2.3 installation, rather than the latest installation. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048900&group_id=5470 From noreply at sourceforge.net Tue Oct 19 22:41:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 19 22:41:27 2004 Subject: [ python-Bugs-1003471 ] Python 1.5.2 security vulnerability still present in 2.3.4 Message-ID: Bugs item #1003471, was opened at 2004-08-04 14:42 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1003471&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Kirby Kuehl (vacuum) Assigned to: Nobody/Anonymous (nobody) Summary: Python 1.5.2 security vulnerability still present in 2.3.4 Initial Comment: First off, I realize that Python 1.5.2 is old, I am just reporting it to be thorough. I was doing a security audit of an application that used this old version of python and found the following bug in the joinpath function. #0 0xff133bf0 in strncpy () from /usr/lib/libc.so.1 (gdb) bt #0 0xff133bf0 in strncpy () from /usr/lib/libc.so.1 (gdb) bt #0 0xff133bf0 in strncpy () from /usr/lib/libc.so.1 #1 0x000304e0 in joinpath (buffer=0x83528 'A' ..., stuff=0x84140 'A' ...) at ./getpath.c:255 #2 0x00030604 in search_for_prefix (argv0_path=0xffbff530 "/opt/OPSW/bin", home=0xff3a0840 'A' ...) at ./getpath.c:300 #3 0x00030a48 in calculate_path () at ./getpath.c:481 #4 0x00030e6c in Py_GetProgramFullPath () at ./getpath.c:634 #5 0x0002dcac in _PySys_Init () at sysmodule.c:413 #6 0x0002b414 in Py_Initialize () at pythonrun.c:142 #7 0x0001755c in Py_Main (argc=1, argv=0xffbffcdc) at main.c:245 #8 0x000171f0 in main (argc=1, argv=0xffbffcdc) at python.c:12 #8 0x000171f0 in main (argc=1, argv=0xffbffcdc) at python.c:12 (gdb) frame 1 #1 0x000304e0 in joinpath (buffer=0x83528 'A' ..., stuff=0x84140 'A' ...) at ./getpath.c:255 255 ./getpath.c: No such file or directory. in ./getpath.c snippet from Python-1.5.2/Modules/getpath.c : static void joinpath(buffer, stuff) char *buffer; char *stuff; { int n, k; if (stuff[0] == SEP) n = 0; else { n = strlen(buffer); if (n > 0 && buffer[n-1] != SEP && n < MAXPATHLEN) buffer[n++] = SEP; } k = strlen(stuff); if (n + k > MAXPATHLEN) k = MAXPATHLEN - n; strncpy(buffer+n, stuff, k); buffer[n+k] = '\0'; } further examining the backtrace: (gdb) print n $1 = 4999 (gdb) print k $2 = -3975 (gdb) print buffer $4 = 0x83528 'A' ... (gdb) print stuff $5 = 0x84140 'A' ... if (n + k > MAXPATHLEN) /* NOTE: MAXPATHLEN is 1024 */ k = MAXPATHLEN - n; /* NOTE: here k is 1024 - 4999 which is the -3975 */ Which of course crashes in strncpy(buffer+n, stuff, k); Thanks, Kirby Kuehl ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-10-19 16:41 Message: Logged In: YES user_id=11375 OK; I think I've got it working now, and neither 2.4CVS or 2.3-maint run into any problems with the sharefuzz library in use. I'm trying to pull the r234 tag, but SF CVS is being really slow; I'll try it tomorrow. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-10-19 16:18 Message: Logged In: YES user_id=11375 I haven't figured out how to get sharefuzz to work yet, but did notice one OS/2 specific overflow; around line 3250 of posixmodule.c, it defines char args[1024], and then does a strcpy from the COMSPEC env. var. into args. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-10-19 16:10 Message: Logged In: YES user_id=11375 The URL mentioned for sharefuzz seems to be out of date; the correct location is now http://sourceforge.net/projects/sharefuzz/ . ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-09 14:43 Message: Logged In: YES user_id=31435 vacuum, are you saying that sharefuzz provokes an actual bug in Python 2.4a2? In Python 2.3.4? ---------------------------------------------------------------------- Comment By: Kirby Kuehl (vacuum) Date: 2004-08-09 14:26 Message: Logged In: YES user_id=116409 Use the sharefuzz utility that I referenced in another comment. Then call any function that will invoke the joinpath function call. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-09 13:56 Message: Logged In: YES user_id=31435 The patch didn't intend to fix a bug that's known to exist. Python 2.0 presumably fixed the bugs here, but in a shaky way, relying on dozens of distinct call sites to establish an undocumented precondition. joinpath() suffers a buffer overrun if the precondition isn't met. The patch verifies that the precondition *is* met, killing the Python process if it's not. That should never happen, but joinpath() is called on so many distinct code paths that "eyeball analysis" is inadequate. To test the patch, change any of joinpath's call sites to violate the precondition, and see that Python dies then. The OP suggested a driver to provoke an envar attack, in the 2004-08-07 14:15 comment. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-08-09 13:39 Message: Logged In: YES user_id=11375 How would I go about testing the patch? Where can I find code that tickles the bug? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 21:01 Message: Logged In: YES user_id=31435 I checked in the attached patch too. If there's ever another release in the 2.3 line, these patches would make decent backport candidates. ---------------------------------------------------------------------- Comment By: Kirby Kuehl (vacuum) Date: 2004-08-07 18:19 Message: Logged In: YES user_id=116409 As tim_one poin ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 15:15 Message: Logged In: YES user_id=31435 Unassigned myself, since I did all I can here. Someone on Linux should test the patch I attached. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 15:13 Message: Logged In: YES user_id=31435 I checked in the change to PC/getpathp.c, which is used on Windows. I'm attaching a patch to Modules/getpath.c, which isn't used on Windows (so I can't test it). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 14:44 Message: Logged In: YES user_id=31435 I'm going to add the panic-check I suggested -- this code is too complicated to have any confidence in "eyeball analysis" over time. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 14:39 Message: Logged In: YES user_id=31435 Yup, and rev 1.32 intended to plug the envar attack. ---------------------------------------------------------------------- Comment By: Kirby Kuehl (vacuum) Date: 2004-08-07 14:15 Message: Logged In: YES user_id=116409 by exporting environment variables. Check sharefuzz which can be found here: http://www.atstake.com/research/tools/index.html#vulnerability_scanning ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 12:43 Message: Logged In: YES user_id=31435 Pretty compilcated internally. A very similar joinpath still exists. While it's not documented or checked, joinpath's code clearly *assumes* strlen(buffer) <= MAXPATHLEN on entry. But it's called from 25 places, and it's not immediately obvious that all call sites guarantee this on all paths. Rev 1.32 of getpath (for Python 2.0) *intended* to fix buffer overflow problems, mostly by changing unsafe strcpy() calls to strncpy() calls. This is delicate, though. I'd be a lot happier if joinpath verified that n <= MAXPATHLEN on entry, and called Py_FatalError() if not so (converting a buffer overrun into a bug report). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-08-07 11:55 Message: Logged In: YES user_id=6656 I'm not sure I understand. How do you get n to be so huge? At any rate, I don't think the bug is present in modern Python, but until I understand how this one was triggered, I'm not prepared to be sure... ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1003471&group_id=5470 From noreply at sourceforge.net Wed Oct 20 02:00:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 02:00:24 2004 Subject: [ python-Bugs-1045893 ] warning '"_POSIX_C_SOURCE" redefined' when include Python.h Message-ID: Bugs item #1045893, was opened at 2004-10-13 14:06 Message generated for change (Comment added) made by eugene_beast You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045893&group_id=5470 Category: Extension Modules Group: Python 2.3 Status: Closed Resolution: Invalid Priority: 5 Submitted By: Eugene Sizikov (eugene_beast) Assigned to: Nobody/Anonymous (nobody) Summary: warning '"_POSIX_C_SOURCE" redefined' when include Python.h Initial Comment: """ In file included from /usr/include/python2.3/Python.h:8, from script.h:21, from server.cpp:23: /usr/include/python2.3/pyconfig.h:847:1: warning: "_POSIX_C_SOURCE" redefined In file included from /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/i386-redhat-linux/bits/os_defines.h:39, from /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/i386-redhat-linux/bits/c++config.h:35, from /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/iostream:44, from server.cpp:18: /usr/include/features.h:131:1: warning: this is the location of the previous definition ""' The above message is shown whenever program (extending or embedding python with C/C++) using Python.h was compilled. OS: Fedora Core 2 compiller: GCC 3.4.0 libc: glibc-2.3 Python: 2.3.3 ---------------------------------------------------------------------- >Comment By: Eugene Sizikov (eugene_beast) Date: 2004-10-20 09:00 Message: Logged In: YES user_id=126917 But why not made it more compliant in Python.h : #ifndef _POSIX_C_SOURCE #define _POSIX_C_SOURCE #endif ??? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-20 04:39 Message: Logged In: YES user_id=21627 This is a bug in your program. See http://docs.python.org/api/includes.html ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045893&group_id=5470 From noreply at sourceforge.net Wed Oct 20 02:23:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 02:28:52 2004 Subject: [ python-Bugs-1049615 ] smeared title when installing Message-ID: Bugs item #1049615, was opened at 2004-10-18 18:09 Message generated for change (Comment added) made by jimjjewett You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 Category: Installation Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: smeared title when installing Initial Comment: Python 2.4b1 on Windows XP. 1280x800. The "python: for windows" heading looks smeared. Other graphical elements also look somewhat smeared, but it isn't as much of a problem when it isn't text to begin with. ---------------------------------------------------------------------- >Comment By: Jim Jewett (jimjjewett) Date: 2004-10-19 20:23 Message: Logged In: YES user_id=764593 Would it be possible to recreate the original picture (or at least parts of it) at the full size, instead of stretching? If the picture parts have to be stretched, that isn't such a problem, but it is better if the button frames aren't stretched -- and particularly the text. Even just replacing the "windows: python" with blank space and text (or redoing that one portion at full size -- maybe not aliased?) would help. Is this the highest-level source? If so, what size would you like it stretched to, and I'll see if I can help. http://cvs.sourceforge.net/viewcvs.py/*checkout*/python/ python/dist/src/PCbuild/installer.bmp?rev=1.2.18.2 ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-19 15:30 Message: Logged In: YES user_id=21627 I don't know what to do about this. Any suggestion appreciated; without suggestions, this likely is what 2.4 will look like. ---------------------------------------------------------------------- Comment By: Mike Brauwerman (brauwerman) Date: 2004-10-18 19:09 Message: Logged In: YES user_id=71983 The "smearing" is the effect of a small bitmap picture, with aliased text, blown up (stretched) to fit into a large space. The graphic has fat pixels, which look a little funny, but they look better if you squint :-) ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-10-18 18:16 Message: Logged In: YES user_id=764593 I used a .bmp to avoid additional info loss. Zipped file is only 20K, but unzips to over a meg. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 From noreply at sourceforge.net Wed Oct 20 03:01:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 03:02:08 2004 Subject: [ python-Bugs-924333 ] Build issues (lib path) on linux2-x86_64 Message-ID: Bugs item #924333, was opened at 2004-03-26 20:12 Message generated for change (Comment added) made by rthalley You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924333&group_id=5470 Category: Build Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Christopher Petrilli (petrilli) Assigned to: Nobody/Anonymous (nobody) Summary: Build issues (lib path) on linux2-x86_64 Initial Comment: The current CVS HEAD won't build "out of the box" on a Fedora x86_64 box, nor on any other Linux AMD64 platform. This is due to the fact that the libraries for many things are not in /usr/lib, but /usr/lib64, and the current 'autoconf' script, as well as the current setup.py do not look there. When added, they detect, build. ---------------------------------------------------------------------- Comment By: Bob Halley (rthalley) Date: 2004-10-19 18:01 Message: Logged In: YES user_id=671513 Python 2.4b1 still doesn't build "out of the box" on a Fedora Core 2 x86_64 box. The patch that was submitted got munged when applied; '/usr/lib/lib64' was added instead of the correct value of '/usr/lib64'. I will submit a new patch once I track down 3 'unexpected skips' in the regression tests. /Bob P.S. I'm very happy with 2.4 so far; good work all! ---------------------------------------------------------------------- Comment By: Mihai Ibanescu (misa) Date: 2004-04-08 10:05 Message: Logged In: YES user_id=205865 have you tried to port the patches from python 2.3.3 to HEAD? From your bug I understand they seem to work - do you have them? ---------------------------------------------------------------------- Comment By: Christopher Petrilli (petrilli) Date: 2004-03-30 11:46 Message: Logged In: YES user_id=38410 See patch #926209 ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-03-29 14:19 Message: Logged In: YES user_id=21627 Can you submit a patch? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924333&group_id=5470 From noreply at sourceforge.net Wed Oct 20 03:11:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 03:11:23 2004 Subject: [ python-Bugs-854823 ] Python-2.3.3c1, Solaris 2.7: socketmodule does not compile Message-ID: Bugs item #854823, was opened at 2003-12-05 09:01 Message generated for change (Comment added) made by sfiedler You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854823&group_id=5470 Category: Build Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Falko Sauermann (fsm2761) Assigned to: Nobody/Anonymous (nobody) Summary: Python-2.3.3c1, Solaris 2.7: socketmodule does not compile Initial Comment: Compiling Python-2.3.3c1 on Solaris 2.7 with gcc 2.95.3 gave me two errors: socketmodule.c:2975: `AF_INET6' undeclared (first use in this function) socketmodule.c:3019: `INET_ADDRSTRLEN' undeclared (first use in this function) The first problem was already reported for Solaris 2.7 with SunPro cc (see item 814613) but is also true for Solaris 2.7 with gcc. The real problem is that AF_INET6 is used when ENABLE_IPV6 is undefined. I believe this must fail for any system where IPV6 is not avalilable. The second problem was already reported for Irix (see item 818490) but is also true for Solaris 2.7. The solution for Irix is also valid for Solaris. If I change socketmodule.c line 204 to: #if (defined(__sgi) || defined(sun)) && !defined(INET_ADDRSTRLEN) the error is gone. ---------------------------------------------------------------------- Comment By: Stephan A. Terre (sfiedler) Date: 2004-10-19 18:11 Message: Logged In: YES user_id=246063 I built and installed both 2.3.3 and 2.3.4, and I get the same error that I was before. It actually occurs in the install phase, when building socketmodule.c. This is on Solaris 2.7 with gcc 2.95.2. I've run out of time tonight to finish building 2.4b1, but I'll do that tomorrow and see what happens. I'm happy to provide the complete or partial build logs if that would help. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-18 18:29 Message: Logged In: YES user_id=33168 Does this problem still exist in 2.3.4 or 2.4b1? Can this be closed? ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-17 19:37 Message: Logged In: YES user_id=33168 Does this problem still exist in 2.3.4 or 2.4b1? Can this be closed? ---------------------------------------------------------------------- Comment By: Stephan A. Terre (sfiedler) Date: 2004-02-26 10:23 Message: Logged In: YES user_id=246063 I think it's worth noting that the preprocessor directive near line 2975 is #ifndef , whereas all other preprocessor directives in the file that refer to the IPV6 symbol are #ifdef . In other words, it looks like just that one directive is a typo. Changing #ifndef to #ifdef fixes it for me. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854823&group_id=5470 From noreply at sourceforge.net Wed Oct 20 06:54:44 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 06:54:54 2004 Subject: [ python-Bugs-924333 ] Build issues (lib path) on linux2-x86_64 Message-ID: Bugs item #924333, was opened at 2004-03-26 20:12 Message generated for change (Comment added) made by rthalley You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924333&group_id=5470 Category: Build Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Christopher Petrilli (petrilli) Assigned to: Nobody/Anonymous (nobody) Summary: Build issues (lib path) on linux2-x86_64 Initial Comment: The current CVS HEAD won't build "out of the box" on a Fedora x86_64 box, nor on any other Linux AMD64 platform. This is due to the fact that the libraries for many things are not in /usr/lib, but /usr/lib64, and the current 'autoconf' script, as well as the current setup.py do not look there. When added, they detect, build. ---------------------------------------------------------------------- Comment By: Bob Halley (rthalley) Date: 2004-10-19 21:54 Message: Logged In: YES user_id=671513 My patch is #1050475 ---------------------------------------------------------------------- Comment By: Bob Halley (rthalley) Date: 2004-10-19 18:01 Message: Logged In: YES user_id=671513 Python 2.4b1 still doesn't build "out of the box" on a Fedora Core 2 x86_64 box. The patch that was submitted got munged when applied; '/usr/lib/lib64' was added instead of the correct value of '/usr/lib64'. I will submit a new patch once I track down 3 'unexpected skips' in the regression tests. /Bob P.S. I'm very happy with 2.4 so far; good work all! ---------------------------------------------------------------------- Comment By: Mihai Ibanescu (misa) Date: 2004-04-08 10:05 Message: Logged In: YES user_id=205865 have you tried to port the patches from python 2.3.3 to HEAD? From your bug I understand they seem to work - do you have them? ---------------------------------------------------------------------- Comment By: Christopher Petrilli (petrilli) Date: 2004-03-30 11:46 Message: Logged In: YES user_id=38410 See patch #926209 ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-03-29 14:19 Message: Logged In: YES user_id=21627 Can you submit a patch? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924333&group_id=5470 From noreply at sourceforge.net Wed Oct 20 07:36:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 07:36:14 2004 Subject: [ python-Bugs-1050486 ] 2.4b1 Win32 installer eliminates 2.3 assoc Message-ID: Bugs item #1050486, was opened at 2004-10-20 01:36 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050486&group_id=5470 Category: Installation Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Mike C. Fletcher (mcfletch) Assigned to: Nobody/Anonymous (nobody) Summary: 2.4b1 Win32 installer eliminates 2.3 assoc Initial Comment: Running a Python 2.4b1 MSI install (in this case installing over a 2.4a1 install) and specifying "don't register extensions" de-registers the default 2.3 "run" action for .py files. Possibly installing over the a1 release confused the installer and it deregistered the 2.3 default action thinking it was deregistering the previously installed 2.4 action? This is seen on Win2k sp4, btw. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050486&group_id=5470 From noreply at sourceforge.net Wed Oct 20 08:26:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 08:26:28 2004 Subject: [ python-Bugs-1045893 ] warning '"_POSIX_C_SOURCE" redefined' when include Python.h Message-ID: Bugs item #1045893, was opened at 2004-10-13 07:06 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045893&group_id=5470 Category: Extension Modules Group: Python 2.3 Status: Closed Resolution: Invalid Priority: 5 Submitted By: Eugene Sizikov (eugene_beast) Assigned to: Nobody/Anonymous (nobody) Summary: warning '"_POSIX_C_SOURCE" redefined' when include Python.h Initial Comment: """ In file included from /usr/include/python2.3/Python.h:8, from script.h:21, from server.cpp:23: /usr/include/python2.3/pyconfig.h:847:1: warning: "_POSIX_C_SOURCE" redefined In file included from /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/i386-redhat-linux/bits/os_defines.h:39, from /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/i386-redhat-linux/bits/c++config.h:35, from /usr/lib/gcc/i386-redhat-linux/3.4.0/../../../../include/c++/3.4.0/iostream:44, from server.cpp:18: /usr/include/features.h:131:1: warning: this is the location of the previous definition ""' The above message is shown whenever program (extending or embedding python with C/C++) using Python.h was compilled. OS: Fedora Core 2 compiller: GCC 3.4.0 libc: glibc-2.3 Python: 2.3.3 ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-20 08:26 Message: Logged In: YES user_id=21627 Python itself and Python extensions must be compiled with precisely the same settings, so taking some other setting of _POSIX_C_SOURCE may cause binary incompatibility. If you include Python.h first, as you should, _POSIX_C_SOURCE will not be defined, so the #ifndef should not have any effect except in broken extension modules. ---------------------------------------------------------------------- Comment By: Eugene Sizikov (eugene_beast) Date: 2004-10-20 02:00 Message: Logged In: YES user_id=126917 But why not made it more compliant in Python.h : #ifndef _POSIX_C_SOURCE #define _POSIX_C_SOURCE #endif ??? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-19 21:39 Message: Logged In: YES user_id=21627 This is a bug in your program. See http://docs.python.org/api/includes.html ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045893&group_id=5470 From noreply at sourceforge.net Wed Oct 20 08:27:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 08:27:43 2004 Subject: [ python-Bugs-1050486 ] 2.4b1 Win32 installer eliminates 2.3 assoc Message-ID: Bugs item #1050486, was opened at 2004-10-20 07:36 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050486&group_id=5470 Category: Installation Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Mike C. Fletcher (mcfletch) Assigned to: Nobody/Anonymous (nobody) Summary: 2.4b1 Win32 installer eliminates 2.3 assoc Initial Comment: Running a Python 2.4b1 MSI install (in this case installing over a 2.4a1 install) and specifying "don't register extensions" de-registers the default 2.3 "run" action for .py files. Possibly installing over the a1 release confused the installer and it deregistered the 2.3 default action thinking it was deregistering the previously installed 2.4 action? This is seen on Win2k sp4, btw. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-20 08:27 Message: Logged In: YES user_id=21627 Did the 2.4a1 installation have extensions registered? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050486&group_id=5470 From noreply at sourceforge.net Wed Oct 20 08:32:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 08:32:20 2004 Subject: [ python-Bugs-1049615 ] smeared title when installing Message-ID: Bugs item #1049615, was opened at 2004-10-19 00:09 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 Category: Installation Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: smeared title when installing Initial Comment: Python 2.4b1 on Windows XP. 1280x800. The "python: for windows" heading looks smeared. Other graphical elements also look somewhat smeared, but it isn't as much of a problem when it isn't text to begin with. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-20 08:32 Message: Logged In: YES user_id=21627 We would need to ask the original author of that picture. I have not been able to contact him; please try yourself. Using PCbuild/installer.bmp might indeed be an option. I could try to arrange the dialogs so that it won't need scaling. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-10-20 02:23 Message: Logged In: YES user_id=764593 Would it be possible to recreate the original picture (or at least parts of it) at the full size, instead of stretching? If the picture parts have to be stretched, that isn't such a problem, but it is better if the button frames aren't stretched -- and particularly the text. Even just replacing the "windows: python" with blank space and text (or redoing that one portion at full size -- maybe not aliased?) would help. Is this the highest-level source? If so, what size would you like it stretched to, and I'll see if I can help. http://cvs.sourceforge.net/viewcvs.py/*checkout*/python/ python/dist/src/PCbuild/installer.bmp?rev=1.2.18.2 ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-19 21:30 Message: Logged In: YES user_id=21627 I don't know what to do about this. Any suggestion appreciated; without suggestions, this likely is what 2.4 will look like. ---------------------------------------------------------------------- Comment By: Mike Brauwerman (brauwerman) Date: 2004-10-19 01:09 Message: Logged In: YES user_id=71983 The "smearing" is the effect of a small bitmap picture, with aliased text, blown up (stretched) to fit into a large space. The graphic has fat pixels, which look a little funny, but they look better if you squint :-) ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-10-19 00:16 Message: Logged In: YES user_id=764593 I used a .bmp to avoid additional info loss. Zipped file is only 20K, but unzips to over a meg. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 From noreply at sourceforge.net Wed Oct 20 08:35:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 08:35:26 2004 Subject: [ python-Bugs-1049615 ] smeared title when installing Message-ID: Bugs item #1049615, was opened at 2004-10-19 00:09 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 Category: Installation Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: smeared title when installing Initial Comment: Python 2.4b1 on Windows XP. 1280x800. The "python: for windows" heading looks smeared. Other graphical elements also look somewhat smeared, but it isn't as much of a problem when it isn't text to begin with. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-20 08:35 Message: Logged In: YES user_id=21627 Scratch the remark on PCbuild/installer.bmp. I somehow thought that 1.2.18.2 has a different resolution than the HEAD version, which it doesn't. I don't understand the phrase "highest-level source". If you are asking whether this is the primary source: for us, it is. Erik van Blokland has some mechanism to generate that, but we don't. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-20 08:32 Message: Logged In: YES user_id=21627 We would need to ask the original author of that picture. I have not been able to contact him; please try yourself. Using PCbuild/installer.bmp might indeed be an option. I could try to arrange the dialogs so that it won't need scaling. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-10-20 02:23 Message: Logged In: YES user_id=764593 Would it be possible to recreate the original picture (or at least parts of it) at the full size, instead of stretching? If the picture parts have to be stretched, that isn't such a problem, but it is better if the button frames aren't stretched -- and particularly the text. Even just replacing the "windows: python" with blank space and text (or redoing that one portion at full size -- maybe not aliased?) would help. Is this the highest-level source? If so, what size would you like it stretched to, and I'll see if I can help. http://cvs.sourceforge.net/viewcvs.py/*checkout*/python/ python/dist/src/PCbuild/installer.bmp?rev=1.2.18.2 ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-19 21:30 Message: Logged In: YES user_id=21627 I don't know what to do about this. Any suggestion appreciated; without suggestions, this likely is what 2.4 will look like. ---------------------------------------------------------------------- Comment By: Mike Brauwerman (brauwerman) Date: 2004-10-19 01:09 Message: Logged In: YES user_id=71983 The "smearing" is the effect of a small bitmap picture, with aliased text, blown up (stretched) to fit into a large space. The graphic has fat pixels, which look a little funny, but they look better if you squint :-) ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-10-19 00:16 Message: Logged In: YES user_id=764593 I used a .bmp to avoid additional info loss. Zipped file is only 20K, but unzips to over a meg. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1049615&group_id=5470 From noreply at sourceforge.net Wed Oct 20 10:22:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 10:22:33 2004 Subject: [ python-Bugs-902037 ] relative on-disk SYSTEM id handling is incorrect Message-ID: Bugs item #902037, was opened at 2004-02-22 03:13 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=902037&group_id=5470 Category: XML Group: Python 2.3 >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: Bob Ippolito (etrepum) Assigned to: Nobody/Anonymous (nobody) Summary: relative on-disk SYSTEM id handling is incorrect Initial Comment: XML.sax.saxutils.prepare_input_source checks os.path.isfile on a relative path without first combining it with the base path. Attached patch is against CVS HEAD (which is also broken, in the exact same way, so I imagine the patch should apply cleanly to 2.3). ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-20 03:22 Message: Logged In: YES user_id=80475 Accepted and applied: Lib/xml/sax/saxutils.py 1.24 ---------------------------------------------------------------------- Comment By: Chuck Rhode (crhode) Date: 2004-03-02 12:42 Message: Logged In: YES user_id=988879 See also Bugs 788931 and 616431. -ccr- ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=902037&group_id=5470 From noreply at sourceforge.net Wed Oct 20 10:29:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 10:29:05 2004 Subject: [ python-Bugs-1010196 ] xml.dom documentation omits hasAttribute, hasAttributeNS Message-ID: Bugs item #1010196, was opened at 2004-08-16 13:12 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010196&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Mike Brown (mike_j_brown) Assigned to: Nobody/Anonymous (nobody) Summary: xml.dom documentation omits hasAttribute, hasAttributeNS Initial Comment: http://www.python.org/doc/2.3.4/lib/dom-element- objects.html does not mention hasAttribute() and hasAttributeNS(), which are DOM L2 Core methods present on Element objects. They should be mentioned not just for completeness, but also so that people are not tempted to use getAttribute() and getAttributeNS() to test for attribute existence. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-20 03:29 Message: Logged In: YES user_id=80475 Can you draft a patch? For the LaTeX markup, just mimic the surrounding text. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010196&group_id=5470 From noreply at sourceforge.net Wed Oct 20 13:21:44 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 13:21:50 2004 Subject: [ python-Bugs-1017553 ] filemode() in tarfile.py makes wrong file mode strings Message-ID: Bugs item #1017553, was opened at 2004-08-27 08:29 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017553&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Peter Loje Hansen (pingupeter) Assigned to: Nobody/Anonymous (nobody) Summary: filemode() in tarfile.py makes wrong file mode strings Initial Comment: filemode(07111) produces the string "---x--x--x" in stead of the correct output "---s--s--t" This small test script provides a correct filemode function: ======================================== #!/usr/bin/env python #--------------------------------------------------------- # Bits used in the mode field, values in octal. #--------------------------------------------------------- S_IFLNK = 0120000 # symbolic link S_IFREG = 0100000 # regular file S_IFBLK = 0060000 # block device S_IFDIR = 0040000 # directory S_IFCHR = 0020000 # character device S_IFIFO = 0010000 # fifo TSUID = 04000 # set UID on execution TSGID = 02000 # set GID on execution TSVTX = 01000 # reserved TUREAD = 0400 # read by owner TUWRITE = 0200 # write by owner TUEXEC = 0100 # execute/search by owner TGREAD = 0040 # read by group TGWRITE = 0020 # write by group TGEXEC = 0010 # execute/search by group TOREAD = 0004 # read by other TOWRITE = 0002 # write by other TOEXEC = 0001 # execute/search by other filemode_table = ( (S_IFLNK, "l", S_IFREG, "-", S_IFBLK, "b", S_IFDIR, "d", S_IFCHR, "c", S_IFIFO, "p"), (TUREAD, "r"), (TUWRITE, "w"), (TUEXEC, "x", TSUID, "S", TUEXEC|TSUID, "s"), (TGREAD, "r"), (TGWRITE, "w"), (TGEXEC, "x", TSGID, "S", TGEXEC|TSGID, "s"), (TOREAD, "r"), (TOWRITE, "w"), (TOEXEC, "x", TSVTX, "T", TOEXEC|TSVTX, "t")) def filemode(mode): """Convert a file's mode to a string of the form -rwxrwxrwx. Used by TarFile.list() """ s = "" for t in filemode_table: p = "-" for i in range(0, len(t), 2): if mode & t[i] == t[i]: p = t[i+1] s += p return s def filemode_old(mode): """Convert a file's mode to a string of the form -rwxrwxrwx. Used by TarFile.list() """ s = "" for t in filemode_table: while True: if mode & t[0] == t[0]: s += t[1] elif len(t) > 2: t = t[2:] continue else: s += "-" break return s def main(): print "filemode_old:\tfilemode:" for p in ( 00777 | 00000, 00777 | 01000, 00777 | 04000, 00777 | 07000, 00000 | 07000, 00111 | 07000 ): print "%s\t%s" % (filemode_old(p), filemode(p)) if __name__ == "__main__": main() ======================================== Regards Peter Loje Hansen ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-10-20 07:21 Message: Logged In: YES user_id=11375 Patch #1043972 is a fix for this bug. ---------------------------------------------------------------------- Comment By: Peter Loje Hansen (pingupeter) Date: 2004-08-27 08:40 Message: Logged In: YES user_id=1112055 Sorry for the broken formatting. I have uploaded the test script in stead. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017553&group_id=5470 From noreply at sourceforge.net Wed Oct 20 13:24:08 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 13:24:15 2004 Subject: [ python-Bugs-1017553 ] filemode() in tarfile.py makes wrong file mode strings Message-ID: Bugs item #1017553, was opened at 2004-08-27 08:29 Message generated for change (Settings changed) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017553&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Peter Loje Hansen (pingupeter) >Assigned to: A.M. Kuchling (akuchling) Summary: filemode() in tarfile.py makes wrong file mode strings Initial Comment: filemode(07111) produces the string "---x--x--x" in stead of the correct output "---s--s--t" This small test script provides a correct filemode function: ======================================== #!/usr/bin/env python #--------------------------------------------------------- # Bits used in the mode field, values in octal. #--------------------------------------------------------- S_IFLNK = 0120000 # symbolic link S_IFREG = 0100000 # regular file S_IFBLK = 0060000 # block device S_IFDIR = 0040000 # directory S_IFCHR = 0020000 # character device S_IFIFO = 0010000 # fifo TSUID = 04000 # set UID on execution TSGID = 02000 # set GID on execution TSVTX = 01000 # reserved TUREAD = 0400 # read by owner TUWRITE = 0200 # write by owner TUEXEC = 0100 # execute/search by owner TGREAD = 0040 # read by group TGWRITE = 0020 # write by group TGEXEC = 0010 # execute/search by group TOREAD = 0004 # read by other TOWRITE = 0002 # write by other TOEXEC = 0001 # execute/search by other filemode_table = ( (S_IFLNK, "l", S_IFREG, "-", S_IFBLK, "b", S_IFDIR, "d", S_IFCHR, "c", S_IFIFO, "p"), (TUREAD, "r"), (TUWRITE, "w"), (TUEXEC, "x", TSUID, "S", TUEXEC|TSUID, "s"), (TGREAD, "r"), (TGWRITE, "w"), (TGEXEC, "x", TSGID, "S", TGEXEC|TSGID, "s"), (TOREAD, "r"), (TOWRITE, "w"), (TOEXEC, "x", TSVTX, "T", TOEXEC|TSVTX, "t")) def filemode(mode): """Convert a file's mode to a string of the form -rwxrwxrwx. Used by TarFile.list() """ s = "" for t in filemode_table: p = "-" for i in range(0, len(t), 2): if mode & t[i] == t[i]: p = t[i+1] s += p return s def filemode_old(mode): """Convert a file's mode to a string of the form -rwxrwxrwx. Used by TarFile.list() """ s = "" for t in filemode_table: while True: if mode & t[0] == t[0]: s += t[1] elif len(t) > 2: t = t[2:] continue else: s += "-" break return s def main(): print "filemode_old:\tfilemode:" for p in ( 00777 | 00000, 00777 | 01000, 00777 | 04000, 00777 | 07000, 00000 | 07000, 00111 | 07000 ): print "%s\t%s" % (filemode_old(p), filemode(p)) if __name__ == "__main__": main() ======================================== Regards Peter Loje Hansen ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-10-20 07:21 Message: Logged In: YES user_id=11375 Patch #1043972 is a fix for this bug. ---------------------------------------------------------------------- Comment By: Peter Loje Hansen (pingupeter) Date: 2004-08-27 08:40 Message: Logged In: YES user_id=1112055 Sorry for the broken formatting. I have uploaded the test script in stead. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017553&group_id=5470 From noreply at sourceforge.net Wed Oct 20 13:57:22 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 13:57:27 2004 Subject: [ python-Bugs-1017553 ] filemode() in tarfile.py makes wrong file mode strings Message-ID: Bugs item #1017553, was opened at 2004-08-27 08:29 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017553&group_id=5470 Category: Python Library Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Peter Loje Hansen (pingupeter) Assigned to: A.M. Kuchling (akuchling) Summary: filemode() in tarfile.py makes wrong file mode strings Initial Comment: filemode(07111) produces the string "---x--x--x" in stead of the correct output "---s--s--t" This small test script provides a correct filemode function: ======================================== #!/usr/bin/env python #--------------------------------------------------------- # Bits used in the mode field, values in octal. #--------------------------------------------------------- S_IFLNK = 0120000 # symbolic link S_IFREG = 0100000 # regular file S_IFBLK = 0060000 # block device S_IFDIR = 0040000 # directory S_IFCHR = 0020000 # character device S_IFIFO = 0010000 # fifo TSUID = 04000 # set UID on execution TSGID = 02000 # set GID on execution TSVTX = 01000 # reserved TUREAD = 0400 # read by owner TUWRITE = 0200 # write by owner TUEXEC = 0100 # execute/search by owner TGREAD = 0040 # read by group TGWRITE = 0020 # write by group TGEXEC = 0010 # execute/search by group TOREAD = 0004 # read by other TOWRITE = 0002 # write by other TOEXEC = 0001 # execute/search by other filemode_table = ( (S_IFLNK, "l", S_IFREG, "-", S_IFBLK, "b", S_IFDIR, "d", S_IFCHR, "c", S_IFIFO, "p"), (TUREAD, "r"), (TUWRITE, "w"), (TUEXEC, "x", TSUID, "S", TUEXEC|TSUID, "s"), (TGREAD, "r"), (TGWRITE, "w"), (TGEXEC, "x", TSGID, "S", TGEXEC|TSGID, "s"), (TOREAD, "r"), (TOWRITE, "w"), (TOEXEC, "x", TSVTX, "T", TOEXEC|TSVTX, "t")) def filemode(mode): """Convert a file's mode to a string of the form -rwxrwxrwx. Used by TarFile.list() """ s = "" for t in filemode_table: p = "-" for i in range(0, len(t), 2): if mode & t[i] == t[i]: p = t[i+1] s += p return s def filemode_old(mode): """Convert a file's mode to a string of the form -rwxrwxrwx. Used by TarFile.list() """ s = "" for t in filemode_table: while True: if mode & t[0] == t[0]: s += t[1] elif len(t) > 2: t = t[2:] continue else: s += "-" break return s def main(): print "filemode_old:\tfilemode:" for p in ( 00777 | 00000, 00777 | 01000, 00777 | 04000, 00777 | 07000, 00000 | 07000, 00111 | 07000 ): print "%s\t%s" % (filemode_old(p), filemode(p)) if __name__ == "__main__": main() ======================================== Regards Peter Loje Hansen ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-10-20 07:57 Message: Logged In: YES user_id=11375 Fixed by committing patch #1043972. Thanks for your bug report! ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-10-20 07:21 Message: Logged In: YES user_id=11375 Patch #1043972 is a fix for this bug. ---------------------------------------------------------------------- Comment By: Peter Loje Hansen (pingupeter) Date: 2004-08-27 08:40 Message: Logged In: YES user_id=1112055 Sorry for the broken formatting. I have uploaded the test script in stead. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017553&group_id=5470 From noreply at sourceforge.net Wed Oct 20 14:03:13 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 14:03:20 2004 Subject: [ python-Bugs-1003471 ] Python 1.5.2 security vulnerability still present in 2.3.4 Message-ID: Bugs item #1003471, was opened at 2004-08-04 14:42 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1003471&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Kirby Kuehl (vacuum) >Assigned to: Andrew I MacIntyre (aimacintyre) Summary: Python 1.5.2 security vulnerability still present in 2.3.4 Initial Comment: First off, I realize that Python 1.5.2 is old, I am just reporting it to be thorough. I was doing a security audit of an application that used this old version of python and found the following bug in the joinpath function. #0 0xff133bf0 in strncpy () from /usr/lib/libc.so.1 (gdb) bt #0 0xff133bf0 in strncpy () from /usr/lib/libc.so.1 (gdb) bt #0 0xff133bf0 in strncpy () from /usr/lib/libc.so.1 #1 0x000304e0 in joinpath (buffer=0x83528 'A' ..., stuff=0x84140 'A' ...) at ./getpath.c:255 #2 0x00030604 in search_for_prefix (argv0_path=0xffbff530 "/opt/OPSW/bin", home=0xff3a0840 'A' ...) at ./getpath.c:300 #3 0x00030a48 in calculate_path () at ./getpath.c:481 #4 0x00030e6c in Py_GetProgramFullPath () at ./getpath.c:634 #5 0x0002dcac in _PySys_Init () at sysmodule.c:413 #6 0x0002b414 in Py_Initialize () at pythonrun.c:142 #7 0x0001755c in Py_Main (argc=1, argv=0xffbffcdc) at main.c:245 #8 0x000171f0 in main (argc=1, argv=0xffbffcdc) at python.c:12 #8 0x000171f0 in main (argc=1, argv=0xffbffcdc) at python.c:12 (gdb) frame 1 #1 0x000304e0 in joinpath (buffer=0x83528 'A' ..., stuff=0x84140 'A' ...) at ./getpath.c:255 255 ./getpath.c: No such file or directory. in ./getpath.c snippet from Python-1.5.2/Modules/getpath.c : static void joinpath(buffer, stuff) char *buffer; char *stuff; { int n, k; if (stuff[0] == SEP) n = 0; else { n = strlen(buffer); if (n > 0 && buffer[n-1] != SEP && n < MAXPATHLEN) buffer[n++] = SEP; } k = strlen(stuff); if (n + k > MAXPATHLEN) k = MAXPATHLEN - n; strncpy(buffer+n, stuff, k); buffer[n+k] = '\0'; } further examining the backtrace: (gdb) print n $1 = 4999 (gdb) print k $2 = -3975 (gdb) print buffer $4 = 0x83528 'A' ... (gdb) print stuff $5 = 0x84140 'A' ... if (n + k > MAXPATHLEN) /* NOTE: MAXPATHLEN is 1024 */ k = MAXPATHLEN - n; /* NOTE: here k is 1024 - 4999 which is the -3975 */ Which of course crashes in strncpy(buffer+n, stuff, k); Thanks, Kirby Kuehl ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-10-20 08:03 Message: Logged In: YES user_id=11375 CVS finally came through, and r234 doesn't seem to report any problems; I put a debug print into joinpath() so I'm sure that it was actually being invoked. So I think this bug no longer applies, in the absence of a transcript demonstrating an actual problem. I'll reassign this bug to Andrew MacIntyre so that he knows about the OS/2 overflow and can fix it before his next OS/2 release. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-10-19 16:41 Message: Logged In: YES user_id=11375 OK; I think I've got it working now, and neither 2.4CVS or 2.3-maint run into any problems with the sharefuzz library in use. I'm trying to pull the r234 tag, but SF CVS is being really slow; I'll try it tomorrow. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-10-19 16:18 Message: Logged In: YES user_id=11375 I haven't figured out how to get sharefuzz to work yet, but did notice one OS/2 specific overflow; around line 3250 of posixmodule.c, it defines char args[1024], and then does a strcpy from the COMSPEC env. var. into args. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-10-19 16:10 Message: Logged In: YES user_id=11375 The URL mentioned for sharefuzz seems to be out of date; the correct location is now http://sourceforge.net/projects/sharefuzz/ . ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-09 14:43 Message: Logged In: YES user_id=31435 vacuum, are you saying that sharefuzz provokes an actual bug in Python 2.4a2? In Python 2.3.4? ---------------------------------------------------------------------- Comment By: Kirby Kuehl (vacuum) Date: 2004-08-09 14:26 Message: Logged In: YES user_id=116409 Use the sharefuzz utility that I referenced in another comment. Then call any function that will invoke the joinpath function call. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-09 13:56 Message: Logged In: YES user_id=31435 The patch didn't intend to fix a bug that's known to exist. Python 2.0 presumably fixed the bugs here, but in a shaky way, relying on dozens of distinct call sites to establish an undocumented precondition. joinpath() suffers a buffer overrun if the precondition isn't met. The patch verifies that the precondition *is* met, killing the Python process if it's not. That should never happen, but joinpath() is called on so many distinct code paths that "eyeball analysis" is inadequate. To test the patch, change any of joinpath's call sites to violate the precondition, and see that Python dies then. The OP suggested a driver to provoke an envar attack, in the 2004-08-07 14:15 comment. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-08-09 13:39 Message: Logged In: YES user_id=11375 How would I go about testing the patch? Where can I find code that tickles the bug? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 21:01 Message: Logged In: YES user_id=31435 I checked in the attached patch too. If there's ever another release in the 2.3 line, these patches would make decent backport candidates. ---------------------------------------------------------------------- Comment By: Kirby Kuehl (vacuum) Date: 2004-08-07 18:19 Message: Logged In: YES user_id=116409 As tim_one poin ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 15:15 Message: Logged In: YES user_id=31435 Unassigned myself, since I did all I can here. Someone on Linux should test the patch I attached. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 15:13 Message: Logged In: YES user_id=31435 I checked in the change to PC/getpathp.c, which is used on Windows. I'm attaching a patch to Modules/getpath.c, which isn't used on Windows (so I can't test it). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 14:44 Message: Logged In: YES user_id=31435 I'm going to add the panic-check I suggested -- this code is too complicated to have any confidence in "eyeball analysis" over time. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 14:39 Message: Logged In: YES user_id=31435 Yup, and rev 1.32 intended to plug the envar attack. ---------------------------------------------------------------------- Comment By: Kirby Kuehl (vacuum) Date: 2004-08-07 14:15 Message: Logged In: YES user_id=116409 by exporting environment variables. Check sharefuzz which can be found here: http://www.atstake.com/research/tools/index.html#vulnerability_scanning ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-08-07 12:43 Message: Logged In: YES user_id=31435 Pretty compilcated internally. A very similar joinpath still exists. While it's not documented or checked, joinpath's code clearly *assumes* strlen(buffer) <= MAXPATHLEN on entry. But it's called from 25 places, and it's not immediately obvious that all call sites guarantee this on all paths. Rev 1.32 of getpath (for Python 2.0) *intended* to fix buffer overflow problems, mostly by changing unsafe strcpy() calls to strncpy() calls. This is delicate, though. I'd be a lot happier if joinpath verified that n <= MAXPATHLEN on entry, and called Py_FatalError() if not so (converting a buffer overrun into a bug report). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-08-07 11:55 Message: Logged In: YES user_id=6656 I'm not sure I understand. How do you get n to be so huge? At any rate, I don't think the bug is present in modern Python, but until I understand how this one was triggered, I'm not prepared to be sure... ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1003471&group_id=5470 From noreply at sourceforge.net Wed Oct 20 14:43:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 14:43:56 2004 Subject: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 Message-ID: Bugs item #1048808, was opened at 2004-10-17 21:24 Message generated for change (Settings changed) made by effbot You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Shalabh Chaturvedi (shalabh) >Assigned to: Peter ?strand (astrand) Summary: test_subprocess 2.4b1 fails on FreeBSD 5.2 Initial Comment: Platform: FreeBSD 5.2.1-RELEASE Python version: 2.4b1 After untarring I just did 'make' and then 'make test'. make test passes all except following (output of make test): ---- test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/usr/home/shalabh/down/Python-2.4b1/Lib/test/test_subprocess.py", line 432, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '5' != '3' ---- Also, make test ends with: ---- *** Error code 1 Stop in /usr/home/shalabh/down/Python-2.4b1. ---- Don't know if above is expected when a test fails. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-17 23:06 Message: Logged In: YES user_id=30293 Here's some more info. On my FreeBSD box, for both 2.3.4 and 2.4b1: >>> import os >>> os.dup(0) 5 But on my Win XP box for 2.3.3: >>> import os >>> os.dup(0) 3 I am guessing this may be relevant. Someone more knowledgable can comment. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 From noreply at sourceforge.net Wed Oct 20 15:06:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 15:07:03 2004 Subject: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 Message-ID: Bugs item #1048808, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by astrand You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Shalabh Chaturvedi (shalabh) Assigned to: Peter ?strand (astrand) Summary: test_subprocess 2.4b1 fails on FreeBSD 5.2 Initial Comment: Platform: FreeBSD 5.2.1-RELEASE Python version: 2.4b1 After untarring I just did 'make' and then 'make test'. make test passes all except following (output of make test): ---- test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/usr/home/shalabh/down/Python-2.4b1/Lib/test/test_subprocess.py", line 432, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '5' != '3' ---- Also, make test ends with: ---- *** Error code 1 Stop in /usr/home/shalabh/down/Python-2.4b1. ---- Don't know if above is expected when a test fails. ---------------------------------------------------------------------- >Comment By: Peter ?strand (astrand) Date: 2004-10-20 15:06 Message: Logged In: YES user_id=344921 >>> import os >>> os.dup(0) 5 I think this is the core of the problem. The test_close_fds test works like this: All file descriptors in the forked child (except 0,1,2) are closed. Then the Python binary is executed via execvp(). A small test program is passed to the Python binary via the -c command line option. If the OS and subprocess module works correctly, we can be sure of that by the time of the execve() system call, only file descriptors (0,1,2) are open (well, the errpipe as well, but let's leave that out for now). But, by the time the Python binary starts executing the small program, all sorts of things may have happened. I'm not really sure we can trust Python not to open files during startup. For example, if we have a PYTHONSTARTUP file, that open file will have a file descriptor, perhaps 3. On one hand, this bug could indicate a bug in the Python interpreter itself: perhaps a file descriptor leak. On the other hand, this test might be a bit too unsafe. So probably, this test should be removed. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-17 23:06 Message: Logged In: YES user_id=30293 Here's some more info. On my FreeBSD box, for both 2.3.4 and 2.4b1: >>> import os >>> os.dup(0) 5 But on my Win XP box for 2.3.3: >>> import os >>> os.dup(0) 3 I am guessing this may be relevant. Someone more knowledgable can comment. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 From noreply at sourceforge.net Wed Oct 20 17:48:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 17:48:21 2004 Subject: [ python-Bugs-1050828 ] reindent.py strips execute permission on Unix Message-ID: Bugs item #1050828, was opened at 2004-10-20 11:48 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Ripton (dripton) Assigned to: Nobody/Anonymous (nobody) Summary: reindent.py strips execute permission on Unix Initial Comment: Tools/scripts/reindent.py strips execute permission from its target file. If the tool detects that changes need to be made, then the target file is renamed to its original name + '.bak', then a new file is created with the original file's name and its whitespace-modified contents. No manipulation of the new file's permissions is done, so it ends up with the default for the user's umask. This makes reindent.py annoying to run in an automated fashion, if you rely on execute permission. The fix is trivial, except for portability. Here's the Posix-only version: + if os.name == "posix": + mode = os.stat(bak).st_mode + os.chmod(file, mode) Patch available upon request. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 From noreply at sourceforge.net Wed Oct 20 19:23:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 19:23:57 2004 Subject: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 Message-ID: Bugs item #1048808, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by effbot You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Shalabh Chaturvedi (shalabh) Assigned to: Peter ?strand (astrand) Summary: test_subprocess 2.4b1 fails on FreeBSD 5.2 Initial Comment: Platform: FreeBSD 5.2.1-RELEASE Python version: 2.4b1 After untarring I just did 'make' and then 'make test'. make test passes all except following (output of make test): ---- test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/usr/home/shalabh/down/Python-2.4b1/Lib/test/test_subprocess.py", line 432, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '5' != '3' ---- Also, make test ends with: ---- *** Error code 1 Stop in /usr/home/shalabh/down/Python-2.4b1. ---- Don't know if above is expected when a test fails. ---------------------------------------------------------------------- >Comment By: Fredrik Lundh (effbot) Date: 2004-10-20 19:23 Message: Logged In: YES user_id=38376 Just curious, but what happens if you start Python using the - S option: $ python -S >>> import os >>> os.dup(0) ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-20 15:06 Message: Logged In: YES user_id=344921 >>> import os >>> os.dup(0) 5 I think this is the core of the problem. The test_close_fds test works like this: All file descriptors in the forked child (except 0,1,2) are closed. Then the Python binary is executed via execvp(). A small test program is passed to the Python binary via the -c command line option. If the OS and subprocess module works correctly, we can be sure of that by the time of the execve() system call, only file descriptors (0,1,2) are open (well, the errpipe as well, but let's leave that out for now). But, by the time the Python binary starts executing the small program, all sorts of things may have happened. I'm not really sure we can trust Python not to open files during startup. For example, if we have a PYTHONSTARTUP file, that open file will have a file descriptor, perhaps 3. On one hand, this bug could indicate a bug in the Python interpreter itself: perhaps a file descriptor leak. On the other hand, this test might be a bit too unsafe. So probably, this test should be removed. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-17 23:06 Message: Logged In: YES user_id=30293 Here's some more info. On my FreeBSD box, for both 2.3.4 and 2.4b1: >>> import os >>> os.dup(0) 5 But on my Win XP box for 2.3.3: >>> import os >>> os.dup(0) 3 I am guessing this may be relevant. Someone more knowledgable can comment. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 From noreply at sourceforge.net Wed Oct 20 19:38:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 19:38:09 2004 Subject: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 Message-ID: Bugs item #1048808, was opened at 2004-10-17 12:24 Message generated for change (Comment added) made by shalabh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Shalabh Chaturvedi (shalabh) Assigned to: Peter ?strand (astrand) Summary: test_subprocess 2.4b1 fails on FreeBSD 5.2 Initial Comment: Platform: FreeBSD 5.2.1-RELEASE Python version: 2.4b1 After untarring I just did 'make' and then 'make test'. make test passes all except following (output of make test): ---- test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/usr/home/shalabh/down/Python-2.4b1/Lib/test/test_subprocess.py", line 432, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '5' != '3' ---- Also, make test ends with: ---- *** Error code 1 Stop in /usr/home/shalabh/down/Python-2.4b1. ---- Don't know if above is expected when a test fails. ---------------------------------------------------------------------- >Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-20 10:38 Message: Logged In: YES user_id=30293 Same: $ python -S Python 2.3.4 (#2, Oct 15 2004, 13:44:35) [GCC 3.3.3 [FreeBSD] 20031106] on freebsd5 >>> import os >>> os.dup(0) 5 The above behaviour is also seen with 2.4b1 using -S. ---------------------------------------------------------------------- Comment By: Fredrik Lundh (effbot) Date: 2004-10-20 10:23 Message: Logged In: YES user_id=38376 Just curious, but what happens if you start Python using the - S option: $ python -S >>> import os >>> os.dup(0) ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-20 06:06 Message: Logged In: YES user_id=344921 >>> import os >>> os.dup(0) 5 I think this is the core of the problem. The test_close_fds test works like this: All file descriptors in the forked child (except 0,1,2) are closed. Then the Python binary is executed via execvp(). A small test program is passed to the Python binary via the -c command line option. If the OS and subprocess module works correctly, we can be sure of that by the time of the execve() system call, only file descriptors (0,1,2) are open (well, the errpipe as well, but let's leave that out for now). But, by the time the Python binary starts executing the small program, all sorts of things may have happened. I'm not really sure we can trust Python not to open files during startup. For example, if we have a PYTHONSTARTUP file, that open file will have a file descriptor, perhaps 3. On one hand, this bug could indicate a bug in the Python interpreter itself: perhaps a file descriptor leak. On the other hand, this test might be a bit too unsafe. So probably, this test should be removed. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-17 14:06 Message: Logged In: YES user_id=30293 Here's some more info. On my FreeBSD box, for both 2.3.4 and 2.4b1: >>> import os >>> os.dup(0) 5 But on my Win XP box for 2.3.3: >>> import os >>> os.dup(0) 3 I am guessing this may be relevant. Someone more knowledgable can comment. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 From noreply at sourceforge.net Wed Oct 20 19:52:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 19:52:17 2004 Subject: [ python-Bugs-854823 ] Python-2.3.3c1, Solaris 2.7: socketmodule does not compile Message-ID: Bugs item #854823, was opened at 2003-12-05 09:01 Message generated for change (Comment added) made by sfiedler You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854823&group_id=5470 Category: Build Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Falko Sauermann (fsm2761) Assigned to: Nobody/Anonymous (nobody) Summary: Python-2.3.3c1, Solaris 2.7: socketmodule does not compile Initial Comment: Compiling Python-2.3.3c1 on Solaris 2.7 with gcc 2.95.3 gave me two errors: socketmodule.c:2975: `AF_INET6' undeclared (first use in this function) socketmodule.c:3019: `INET_ADDRSTRLEN' undeclared (first use in this function) The first problem was already reported for Solaris 2.7 with SunPro cc (see item 814613) but is also true for Solaris 2.7 with gcc. The real problem is that AF_INET6 is used when ENABLE_IPV6 is undefined. I believe this must fail for any system where IPV6 is not avalilable. The second problem was already reported for Irix (see item 818490) but is also true for Solaris 2.7. The solution for Irix is also valid for Solaris. If I change socketmodule.c line 204 to: #if (defined(__sgi) || defined(sun)) && !defined(INET_ADDRSTRLEN) the error is gone. ---------------------------------------------------------------------- Comment By: Stephan A. Terre (sfiedler) Date: 2004-10-20 10:52 Message: Logged In: YES user_id=246063 Okay, here are my results: Python 2.3.3 and 2.3.4 both give errors about both AF_INET6 and INET_ADDRSTRLEN being undeclared. The line numbers are different, but everything else is the same. The specific errors from the 2.3.4 install are: socketmodule.c:2979: `AF_INET6' undeclared (first use in this function) socketmodule.c:3023: `INET_ADDRSTRLEN' undeclared (first use in this function) Python 2.4b1 only gives an error about INET_ADDRSTRLEN: socketmodule.c:3350: `INET_ADDRSTRLEN' undeclared (first use in this function) The string INET_ADDRSTRLEN did not appear anywhere in the files under /usr/include on this system. In the 2.4b1 tree, if I just #define INET_ADDRSTRLEN to 16, as hinted by Lib/plat-sunos5/IN.py, then socketmodule.c compiles with only minor warnings. I don't know the right way to get this information into the source file though, sorry. Also, I'm not sure what a good test would be to determine whether this value is actually correct. Because a fresh build of both 2.3.4 and 2.4b1 does not create the _socket module, I believe this bug should not yet be closed. ---------------------------------------------------------------------- Comment By: Stephan A. Terre (sfiedler) Date: 2004-10-19 18:11 Message: Logged In: YES user_id=246063 I built and installed both 2.3.3 and 2.3.4, and I get the same error that I was before. It actually occurs in the install phase, when building socketmodule.c. This is on Solaris 2.7 with gcc 2.95.2. I've run out of time tonight to finish building 2.4b1, but I'll do that tomorrow and see what happens. I'm happy to provide the complete or partial build logs if that would help. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-18 18:29 Message: Logged In: YES user_id=33168 Does this problem still exist in 2.3.4 or 2.4b1? Can this be closed? ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2004-10-17 19:37 Message: Logged In: YES user_id=33168 Does this problem still exist in 2.3.4 or 2.4b1? Can this be closed? ---------------------------------------------------------------------- Comment By: Stephan A. Terre (sfiedler) Date: 2004-02-26 10:23 Message: Logged In: YES user_id=246063 I think it's worth noting that the preprocessor directive near line 2975 is #ifndef , whereas all other preprocessor directives in the file that refer to the IPV6 symbol are #ifdef . In other words, it looks like just that one directive is a typo. Changing #ifndef to #ifdef fixes it for me. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854823&group_id=5470 From noreply at sourceforge.net Wed Oct 20 20:26:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 20:26:37 2004 Subject: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 Message-ID: Bugs item #1048808, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by astrand You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Shalabh Chaturvedi (shalabh) Assigned to: Peter ?strand (astrand) Summary: test_subprocess 2.4b1 fails on FreeBSD 5.2 Initial Comment: Platform: FreeBSD 5.2.1-RELEASE Python version: 2.4b1 After untarring I just did 'make' and then 'make test'. make test passes all except following (output of make test): ---- test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/usr/home/shalabh/down/Python-2.4b1/Lib/test/test_subprocess.py", line 432, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '5' != '3' ---- Also, make test ends with: ---- *** Error code 1 Stop in /usr/home/shalabh/down/Python-2.4b1. ---- Don't know if above is expected when a test fails. ---------------------------------------------------------------------- >Comment By: Peter ?strand (astrand) Date: 2004-10-20 20:26 Message: Logged In: YES user_id=344921 Do you have lsof on your system? In that case, you could try this: >>> import os, subprocess >>> subprocess.call(["lsof", "-p", str(os.getpid())]) That should give you a hint of what fd 3 and 4 really are. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-20 19:38 Message: Logged In: YES user_id=30293 Same: $ python -S Python 2.3.4 (#2, Oct 15 2004, 13:44:35) [GCC 3.3.3 [FreeBSD] 20031106] on freebsd5 >>> import os >>> os.dup(0) 5 The above behaviour is also seen with 2.4b1 using -S. ---------------------------------------------------------------------- Comment By: Fredrik Lundh (effbot) Date: 2004-10-20 19:23 Message: Logged In: YES user_id=38376 Just curious, but what happens if you start Python using the - S option: $ python -S >>> import os >>> os.dup(0) ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-20 15:06 Message: Logged In: YES user_id=344921 >>> import os >>> os.dup(0) 5 I think this is the core of the problem. The test_close_fds test works like this: All file descriptors in the forked child (except 0,1,2) are closed. Then the Python binary is executed via execvp(). A small test program is passed to the Python binary via the -c command line option. If the OS and subprocess module works correctly, we can be sure of that by the time of the execve() system call, only file descriptors (0,1,2) are open (well, the errpipe as well, but let's leave that out for now). But, by the time the Python binary starts executing the small program, all sorts of things may have happened. I'm not really sure we can trust Python not to open files during startup. For example, if we have a PYTHONSTARTUP file, that open file will have a file descriptor, perhaps 3. On one hand, this bug could indicate a bug in the Python interpreter itself: perhaps a file descriptor leak. On the other hand, this test might be a bit too unsafe. So probably, this test should be removed. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-17 23:06 Message: Logged In: YES user_id=30293 Here's some more info. On my FreeBSD box, for both 2.3.4 and 2.4b1: >>> import os >>> os.dup(0) 5 But on my Win XP box for 2.3.3: >>> import os >>> os.dup(0) 3 I am guessing this may be relevant. Someone more knowledgable can comment. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 From noreply at sourceforge.net Wed Oct 20 20:38:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 20:38:44 2004 Subject: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 Message-ID: Bugs item #1048808, was opened at 2004-10-17 12:24 Message generated for change (Comment added) made by shalabh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Shalabh Chaturvedi (shalabh) Assigned to: Peter ?strand (astrand) Summary: test_subprocess 2.4b1 fails on FreeBSD 5.2 Initial Comment: Platform: FreeBSD 5.2.1-RELEASE Python version: 2.4b1 After untarring I just did 'make' and then 'make test'. make test passes all except following (output of make test): ---- test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/usr/home/shalabh/down/Python-2.4b1/Lib/test/test_subprocess.py", line 432, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '5' != '3' ---- Also, make test ends with: ---- *** Error code 1 Stop in /usr/home/shalabh/down/Python-2.4b1. ---- Don't know if above is expected when a test fails. ---------------------------------------------------------------------- >Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-20 11:38 Message: Logged In: YES user_id=30293 They are pipes: Python 2.4b1 (#1, Oct 17 2004, 12:04:08) [GCC 3.3.3 [FreeBSD] 20031106] on freebsd5 Type "help", "copyright", "credits" or "license" for more information. >>> import os, subprocess >>> subprocess.call(["lsof", "-p", str(os.getpid())]) COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME python 1609 shalabh 0u VCHR 5,1 0t15826 114 /dev/ttyp1 python 1609 shalabh 1u VCHR 5,1 0t15826 114 /dev/ttyp1 python 1609 shalabh 2u VCHR 5,1 0t15826 114 /dev/ttyp1 python 1609 shalabh 3u PIPE 0xc541acc4 16384 ->0xc541ad70 python 1609 shalabh 4u PIPE 0xc541ad70 16384 ->0xc541acc4 0 ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-20 11:26 Message: Logged In: YES user_id=344921 Do you have lsof on your system? In that case, you could try this: >>> import os, subprocess >>> subprocess.call(["lsof", "-p", str(os.getpid())]) That should give you a hint of what fd 3 and 4 really are. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-20 10:38 Message: Logged In: YES user_id=30293 Same: $ python -S Python 2.3.4 (#2, Oct 15 2004, 13:44:35) [GCC 3.3.3 [FreeBSD] 20031106] on freebsd5 >>> import os >>> os.dup(0) 5 The above behaviour is also seen with 2.4b1 using -S. ---------------------------------------------------------------------- Comment By: Fredrik Lundh (effbot) Date: 2004-10-20 10:23 Message: Logged In: YES user_id=38376 Just curious, but what happens if you start Python using the - S option: $ python -S >>> import os >>> os.dup(0) ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-20 06:06 Message: Logged In: YES user_id=344921 >>> import os >>> os.dup(0) 5 I think this is the core of the problem. The test_close_fds test works like this: All file descriptors in the forked child (except 0,1,2) are closed. Then the Python binary is executed via execvp(). A small test program is passed to the Python binary via the -c command line option. If the OS and subprocess module works correctly, we can be sure of that by the time of the execve() system call, only file descriptors (0,1,2) are open (well, the errpipe as well, but let's leave that out for now). But, by the time the Python binary starts executing the small program, all sorts of things may have happened. I'm not really sure we can trust Python not to open files during startup. For example, if we have a PYTHONSTARTUP file, that open file will have a file descriptor, perhaps 3. On one hand, this bug could indicate a bug in the Python interpreter itself: perhaps a file descriptor leak. On the other hand, this test might be a bit too unsafe. So probably, this test should be removed. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-17 14:06 Message: Logged In: YES user_id=30293 Here's some more info. On my FreeBSD box, for both 2.3.4 and 2.4b1: >>> import os >>> os.dup(0) 5 But on my Win XP box for 2.3.3: >>> import os >>> os.dup(0) 3 I am guessing this may be relevant. Someone more knowledgable can comment. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 From noreply at sourceforge.net Wed Oct 20 21:49:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 21:49:58 2004 Subject: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 Message-ID: Bugs item #1048808, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by astrand You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Shalabh Chaturvedi (shalabh) Assigned to: Peter ?strand (astrand) Summary: test_subprocess 2.4b1 fails on FreeBSD 5.2 Initial Comment: Platform: FreeBSD 5.2.1-RELEASE Python version: 2.4b1 After untarring I just did 'make' and then 'make test'. make test passes all except following (output of make test): ---- test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/usr/home/shalabh/down/Python-2.4b1/Lib/test/test_subprocess.py", line 432, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '5' != '3' ---- Also, make test ends with: ---- *** Error code 1 Stop in /usr/home/shalabh/down/Python-2.4b1. ---- Don't know if above is expected when a test fails. ---------------------------------------------------------------------- >Comment By: Peter ?strand (astrand) Date: 2004-10-20 21:49 Message: Logged In: YES user_id=344921 Strange. You could check if any other processes uses these pipes with: lsof | egrep "0xc541acc4|0xc541ad70" ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-20 20:38 Message: Logged In: YES user_id=30293 They are pipes: Python 2.4b1 (#1, Oct 17 2004, 12:04:08) [GCC 3.3.3 [FreeBSD] 20031106] on freebsd5 Type "help", "copyright", "credits" or "license" for more information. >>> import os, subprocess >>> subprocess.call(["lsof", "-p", str(os.getpid())]) COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME python 1609 shalabh 0u VCHR 5,1 0t15826 114 /dev/ttyp1 python 1609 shalabh 1u VCHR 5,1 0t15826 114 /dev/ttyp1 python 1609 shalabh 2u VCHR 5,1 0t15826 114 /dev/ttyp1 python 1609 shalabh 3u PIPE 0xc541acc4 16384 ->0xc541ad70 python 1609 shalabh 4u PIPE 0xc541ad70 16384 ->0xc541acc4 0 ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-20 20:26 Message: Logged In: YES user_id=344921 Do you have lsof on your system? In that case, you could try this: >>> import os, subprocess >>> subprocess.call(["lsof", "-p", str(os.getpid())]) That should give you a hint of what fd 3 and 4 really are. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-20 19:38 Message: Logged In: YES user_id=30293 Same: $ python -S Python 2.3.4 (#2, Oct 15 2004, 13:44:35) [GCC 3.3.3 [FreeBSD] 20031106] on freebsd5 >>> import os >>> os.dup(0) 5 The above behaviour is also seen with 2.4b1 using -S. ---------------------------------------------------------------------- Comment By: Fredrik Lundh (effbot) Date: 2004-10-20 19:23 Message: Logged In: YES user_id=38376 Just curious, but what happens if you start Python using the - S option: $ python -S >>> import os >>> os.dup(0) ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-20 15:06 Message: Logged In: YES user_id=344921 >>> import os >>> os.dup(0) 5 I think this is the core of the problem. The test_close_fds test works like this: All file descriptors in the forked child (except 0,1,2) are closed. Then the Python binary is executed via execvp(). A small test program is passed to the Python binary via the -c command line option. If the OS and subprocess module works correctly, we can be sure of that by the time of the execve() system call, only file descriptors (0,1,2) are open (well, the errpipe as well, but let's leave that out for now). But, by the time the Python binary starts executing the small program, all sorts of things may have happened. I'm not really sure we can trust Python not to open files during startup. For example, if we have a PYTHONSTARTUP file, that open file will have a file descriptor, perhaps 3. On one hand, this bug could indicate a bug in the Python interpreter itself: perhaps a file descriptor leak. On the other hand, this test might be a bit too unsafe. So probably, this test should be removed. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-17 23:06 Message: Logged In: YES user_id=30293 Here's some more info. On my FreeBSD box, for both 2.3.4 and 2.4b1: >>> import os >>> os.dup(0) 5 But on my Win XP box for 2.3.3: >>> import os >>> os.dup(0) 3 I am guessing this may be relevant. Someone more knowledgable can comment. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 From noreply at sourceforge.net Wed Oct 20 22:08:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 22:08:15 2004 Subject: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 Message-ID: Bugs item #1048808, was opened at 2004-10-17 12:24 Message generated for change (Comment added) made by shalabh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 7 Submitted By: Shalabh Chaturvedi (shalabh) Assigned to: Peter ?strand (astrand) Summary: test_subprocess 2.4b1 fails on FreeBSD 5.2 Initial Comment: Platform: FreeBSD 5.2.1-RELEASE Python version: 2.4b1 After untarring I just did 'make' and then 'make test'. make test passes all except following (output of make test): ---- test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/usr/home/shalabh/down/Python-2.4b1/Lib/test/test_subprocess.py", line 432, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '5' != '3' ---- Also, make test ends with: ---- *** Error code 1 Stop in /usr/home/shalabh/down/Python-2.4b1. ---- Don't know if above is expected when a test fails. ---------------------------------------------------------------------- >Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-20 13:08 Message: Logged In: YES user_id=30293 No other processes use these. I checked the output of subprocess.call(["lsof"]) from within python. Observe they refer to each other, though: python 878 shalabh 3u PIPE 0xc5193c18 16384 ->0xc5193cc4 ^^^^^^^^^^ ********** python 878 shalabh 4u PIPE 0xc5193cc4 16384 ->0xc5193c18 ********** ^^^^^^^^^^ ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-20 12:49 Message: Logged In: YES user_id=344921 Strange. You could check if any other processes uses these pipes with: lsof | egrep "0xc541acc4|0xc541ad70" ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-20 11:38 Message: Logged In: YES user_id=30293 They are pipes: Python 2.4b1 (#1, Oct 17 2004, 12:04:08) [GCC 3.3.3 [FreeBSD] 20031106] on freebsd5 Type "help", "copyright", "credits" or "license" for more information. >>> import os, subprocess >>> subprocess.call(["lsof", "-p", str(os.getpid())]) COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME python 1609 shalabh 0u VCHR 5,1 0t15826 114 /dev/ttyp1 python 1609 shalabh 1u VCHR 5,1 0t15826 114 /dev/ttyp1 python 1609 shalabh 2u VCHR 5,1 0t15826 114 /dev/ttyp1 python 1609 shalabh 3u PIPE 0xc541acc4 16384 ->0xc541ad70 python 1609 shalabh 4u PIPE 0xc541ad70 16384 ->0xc541acc4 0 ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-20 11:26 Message: Logged In: YES user_id=344921 Do you have lsof on your system? In that case, you could try this: >>> import os, subprocess >>> subprocess.call(["lsof", "-p", str(os.getpid())]) That should give you a hint of what fd 3 and 4 really are. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-20 10:38 Message: Logged In: YES user_id=30293 Same: $ python -S Python 2.3.4 (#2, Oct 15 2004, 13:44:35) [GCC 3.3.3 [FreeBSD] 20031106] on freebsd5 >>> import os >>> os.dup(0) 5 The above behaviour is also seen with 2.4b1 using -S. ---------------------------------------------------------------------- Comment By: Fredrik Lundh (effbot) Date: 2004-10-20 10:23 Message: Logged In: YES user_id=38376 Just curious, but what happens if you start Python using the - S option: $ python -S >>> import os >>> os.dup(0) ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-20 06:06 Message: Logged In: YES user_id=344921 >>> import os >>> os.dup(0) 5 I think this is the core of the problem. The test_close_fds test works like this: All file descriptors in the forked child (except 0,1,2) are closed. Then the Python binary is executed via execvp(). A small test program is passed to the Python binary via the -c command line option. If the OS and subprocess module works correctly, we can be sure of that by the time of the execve() system call, only file descriptors (0,1,2) are open (well, the errpipe as well, but let's leave that out for now). But, by the time the Python binary starts executing the small program, all sorts of things may have happened. I'm not really sure we can trust Python not to open files during startup. For example, if we have a PYTHONSTARTUP file, that open file will have a file descriptor, perhaps 3. On one hand, this bug could indicate a bug in the Python interpreter itself: perhaps a file descriptor leak. On the other hand, this test might be a bit too unsafe. So probably, this test should be removed. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-17 14:06 Message: Logged In: YES user_id=30293 Here's some more info. On my FreeBSD box, for both 2.3.4 and 2.4b1: >>> import os >>> os.dup(0) 5 But on my Win XP box for 2.3.3: >>> import os >>> os.dup(0) 3 I am guessing this may be relevant. Someone more knowledgable can comment. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 From noreply at sourceforge.net Wed Oct 20 22:30:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 20 22:30:26 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-18 03:24 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-20 22:30 Message: Logged In: YES user_id=469548 With 'be careful, it may not exist.', do you mean rmtree can't rely on exc.filename being set? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 17:18 Message: Logged In: YES user_id=6380 This looks fine, and can be checked in. There's one minor remaining issue, but I don't see how to fix it without changing the signature for the onerror function that os.walk() takes, and that's a no-no: when the problem is a directory permission preventing os.listdir() to work, the path passed to the rmtree caller's onerror function is always the toplevel path, rather than the directory that couldn't be listed. The real path is included in the error message. (Hmm, perhaps you could get it out of there? I think it's an attribute of the exception object; but be careful, it may not exist.) It's unfortunate that the error argument to the onerror function is a sys.exc_info() triple rather than an exception object (calling sys.exc_info() is expensive), but that's an API choice from the past that we can't change until Python 3.0. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 10:57 Message: Logged In: YES user_id=469548 A new patch is attached. I factored out the handle_call_error method so exception handling is more local. Also, errors in os.walk were already being handled using the _raise_err/except OSError combination, but we now do it using the onerror argument to os.walk. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 01:42 Message: Logged In: YES user_id=6380 (Actually, os.walk has an onerror argument, it just uses a different convention and default, so we could thread this through easily.) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 01:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 00:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 08:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Thu Oct 21 05:02:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 21 05:03:06 2004 Subject: [ python-Bugs-1029047 ] No command line args when script run without python.exe Message-ID: Bugs item #1029047, was opened at 2004-09-16 05:07 Message generated for change (Comment added) made by facundobatista You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1029047&group_id=5470 Category: Windows Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Kerim Borchaev (warkid) Assigned to: Nobody/Anonymous (nobody) Summary: No command line args when script run without python.exe Initial Comment: In "FolderOptions/File types" for "py" files in action "open" "Application used to perform action" reads: "C:\Python24\python.exe" "%1" This results in such behaviour: ----------- D:\temp>copy con test.py import sys print sys.version print sys.argv[1] ^Z 1 file(s) copied. D:\temp>python.exe test.py hello 2.4a3 (#56, Sep 2 2004, 20:50:21) [MSC v.1310 32 bit (Intel)] hello D:\temp>test.py hello 2.4a3 (#56, Sep 2 2004, 20:50:21) [MSC v.1310 32 bit (Intel)] Traceback (most recent call last): File "D:\temp\test.py", line 3, in ? print sys.argv[1] IndexError: list index out of range ----------- Python2.3 had this string for "Application used...": "C:\Python24\python.exe" "%1" %* ---------------------------------------------------------------------- >Comment By: Facundo Batista (facundobatista) Date: 2004-10-21 00:02 Message: Logged In: YES user_id=752496 As is well pointed by quiver, this issue was fixed through the bug #1022010. Furthermore, I checked in my system (win2k sp4, Py2.4b1) and is working correctly (changed the file association, including now the %*). . Facundo ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-09-16 05:47 Message: Logged In: YES user_id=671362 This is a duplicate of bug #1022010 http://www.python.org/sf/1022010 Try a new installer from http://www.dcl.hpi.uni-potsdam.de/home/loewis/python- 2.4a3.3.msi P.S. Is the latest MSI installer uploaded to www.python.org? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1029047&group_id=5470 From noreply at sourceforge.net Thu Oct 21 05:15:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 21 05:15:58 2004 Subject: [ python-Bugs-1029561 ] test_pep277 fails Message-ID: Bugs item #1029561, was opened at 2004-09-16 20:24 Message generated for change (Comment added) made by facundobatista You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1029561&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Marek Baczynski (imbaczek) Assigned to: Nobody/Anonymous (nobody) Summary: test_pep277 fails Initial Comment: Running regrtest.py: test_pep277 test test_pep277 produced unexpected output: ********************************************************************** *** mismatch between line 3 of expected output and line 3 of actual output: - ['???', '???', '??????', '????????????', '????G\xdf', 'Ge??-sa?', 'Gr\xfc\xdf-Gott', 'abc', 'ascii'] ? ^ ------------ + ['???', '???', '????-???', '??????', '????????????', '?????\xdf', 'Gr\xfc\xdf-Gott', 'abc', 'ascii'] ? ++++++++++++ ^ ********************************************************************** But when running just test_pep277.py: test_directory (__main__.UnicodeFileTests) ... u'\xdf-\u66e8\u66e9\u66eb' ok test_failures (__main__.UnicodeFileTests) ... ok test_listdir (__main__.UnicodeFileTests) ... ['???', '???', '????-???', '??????', '????????????', '?????\xdf', 'Gr\xfc\xdf-Gott', 'abc', 'ascii'] [u'Gr\xfc\xdf-Gott', u'abc', u'ascii', u'\u0393\u03b5\u03b9\u03ac-\u03c3\u03b1\u03c2', u'\u0417\u0434\u0440\u0430\u0432\u0441\u0442\u0432\u0443\u0439\u0442\u0435', u'\u05d4\u05e9\u05e7\u05e6\u05e5\u05e1', u'\u306b\u307d\u3093', u'\u66e8\u05e9\u3093\u0434\u0393\xdf', u'\u66e8\u66e9\u66eb'] ok test_open (__main__.UnicodeFileTests) ... ok test_rename (__main__.UnicodeFileTests) ... ok ---------------------------------------------------------------------- Ran 5 tests in 0.109s OK ---------------------------------------------------------------------- >Comment By: Facundo Batista (facundobatista) Date: 2004-10-21 00:15 Message: Logged In: YES user_id=752496 The tests (through regrtest.py and alone) pass OK to me. Using (Win2k SP4), Python 2.4b1 (#57, Oct 15 2004, 15:23:38) [MSC v.1310 32 bit (Intel)] on win32, and: >>> import locale >>> locale.getlocale() ['Spanish_Argentina', '1252'] . Facundo ---------------------------------------------------------------------- Comment By: Miki Tebeka (tebeka) Date: 2004-10-17 13:06 Message: Logged In: YES user_id=358087 Fails for me on 2.4b1 on winXP ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-09-17 06:55 Message: Logged In: YES user_id=671362 Same for me. Fail for regrtest.py, but OK for test_277.py in isolation. (Python 2.4a3, Windows 2K SP4, locale:Japan) But my output is a bit different from yours. test test_pep277 produced unexpected output: ********************************************** ************************ *** mismatch between line 3 of expected output and line 3 of actual output: - ['???', '???', '??????', '????????????', '????G\xdf', 'Ge??- sa?', 'Gr\xfc\xdf-Gott', 'abc', 'ascii'] + ['??????', '??\x82\xf1\x84t\x83\xa1s', '?\x9eG?', 'Grus- Gott', 'abc', 'ascii', '\x82\xc9\x82\xdb\x82\xf1', '\x83\xa1\x83 \xc3\x83\xc7?-\x83\xd0\x83\xbf?', '\x84H\x84t\x84\x82 \x84p\x84r\x84\x83\x84\x84\x84r\x84\x85\x84z\x84\x84\x84u'] ********************************************** ************************ This reminds me of the next thread: * More joy with test_strptime http://mail.python.org/pipermail/python-dev/2004- July/046046.html ---------------------------------------------------------------------- Comment By: Marek Baczynski (imbaczek) Date: 2004-09-16 20:25 Message: Logged In: YES user_id=838849 Forgot: Windows XP SP1, Python 2.4a3 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1029561&group_id=5470 From noreply at sourceforge.net Thu Oct 21 06:42:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 21 06:42:09 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-21 00:42 Message: Logged In: YES user_id=6380 Yes. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-20 16:30 Message: Logged In: YES user_id=469548 With 'be careful, it may not exist.', do you mean rmtree can't rely on exc.filename being set? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 11:18 Message: Logged In: YES user_id=6380 This looks fine, and can be checked in. There's one minor remaining issue, but I don't see how to fix it without changing the signature for the onerror function that os.walk() takes, and that's a no-no: when the problem is a directory permission preventing os.listdir() to work, the path passed to the rmtree caller's onerror function is always the toplevel path, rather than the directory that couldn't be listed. The real path is included in the error message. (Hmm, perhaps you could get it out of there? I think it's an attribute of the exception object; but be careful, it may not exist.) It's unfortunate that the error argument to the onerror function is a sys.exc_info() triple rather than an exception object (calling sys.exc_info() is expensive), but that's an API choice from the past that we can't change until Python 3.0. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 04:57 Message: Logged In: YES user_id=469548 A new patch is attached. I factored out the handle_call_error method so exception handling is more local. Also, errors in os.walk were already being handled using the _raise_err/except OSError combination, but we now do it using the onerror argument to os.walk. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:42 Message: Logged In: YES user_id=6380 (Actually, os.walk has an onerror argument, it just uses a different convention and default, so we could thread this through easily.) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 18:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 02:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Thu Oct 21 10:40:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 21 10:40:10 2004 Subject: [ python-Bugs-1010196 ] xml.dom documentation omits hasAttribute, hasAttributeNS Message-ID: Bugs item #1010196, was opened at 2004-08-16 11:12 Message generated for change (Comment added) made by mike_j_brown You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010196&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Mike Brown (mike_j_brown) Assigned to: Nobody/Anonymous (nobody) Summary: xml.dom documentation omits hasAttribute, hasAttributeNS Initial Comment: http://www.python.org/doc/2.3.4/lib/dom-element- objects.html does not mention hasAttribute() and hasAttributeNS(), which are DOM L2 Core methods present on Element objects. They should be mentioned not just for completeness, but also so that people are not tempted to use getAttribute() and getAttributeNS() to test for attribute existence. ---------------------------------------------------------------------- >Comment By: Mike Brown (mike_j_brown) Date: 2004-10-21 01:40 Message: Logged In: YES user_id=371366 OK, submitted as patch # 1051321 https://sourceforge.net/tracker/index.php?func=detail&aid=1051321&group_id=5470&atid=305470 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-20 01:29 Message: Logged In: YES user_id=80475 Can you draft a patch? For the LaTeX markup, just mimic the surrounding text. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1010196&group_id=5470 From noreply at sourceforge.net Thu Oct 21 19:24:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 21 19:24:38 2004 Subject: [ python-Bugs-1051638 ] incorrect traceback filename from pyc Message-ID: Bugs item #1051638, was opened at 2004-10-21 10:24 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1051638&group_id=5470 Category: Parser/Compiler Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Kevin Quick (kquick) Assigned to: Nobody/Anonymous (nobody) Summary: incorrect traceback filename from pyc Initial Comment: The .pyc file apparently caches the entire path of the source file during compilation, causing it to report improper path information if the resulting code is moved and an exception occurs. $ python Python 2.3.3 (#1, Oct 18 2004, 16:10:24) [GCC 3.3.4 20040623 (Gentoo Linux 3.3.4-r1, ssp-3.3.2-2, pie-8.7.6)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1051638&group_id=5470 From noreply at sourceforge.net Thu Oct 21 21:29:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 21 21:29:45 2004 Subject: [ python-Bugs-1048808 ] test_subprocess 2.4b1 fails on FreeBSD 5.2 Message-ID: Bugs item #1048808, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by astrand You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 Category: Python Library Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 7 Submitted By: Shalabh Chaturvedi (shalabh) Assigned to: Peter ?strand (astrand) Summary: test_subprocess 2.4b1 fails on FreeBSD 5.2 Initial Comment: Platform: FreeBSD 5.2.1-RELEASE Python version: 2.4b1 After untarring I just did 'make' and then 'make test'. make test passes all except following (output of make test): ---- test_subprocess this bit of output is from a test of stdout in a different process ... test test_subprocess failed -- Traceback (most recent call last): File "/usr/home/shalabh/down/Python-2.4b1/Lib/test/test_subprocess.py", line 432, in test_close_fds self.assertEqual(p.stdout.read(), "3") AssertionError: '5' != '3' ---- Also, make test ends with: ---- *** Error code 1 Stop in /usr/home/shalabh/down/Python-2.4b1. ---- Don't know if above is expected when a test fails. ---------------------------------------------------------------------- >Comment By: Peter ?strand (astrand) Date: 2004-10-21 21:29 Message: Logged In: YES user_id=344921 I've removed this test now. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-20 22:08 Message: Logged In: YES user_id=30293 No other processes use these. I checked the output of subprocess.call(["lsof"]) from within python. Observe they refer to each other, though: python 878 shalabh 3u PIPE 0xc5193c18 16384 ->0xc5193cc4 ^^^^^^^^^^ ********** python 878 shalabh 4u PIPE 0xc5193cc4 16384 ->0xc5193c18 ********** ^^^^^^^^^^ ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-20 21:49 Message: Logged In: YES user_id=344921 Strange. You could check if any other processes uses these pipes with: lsof | egrep "0xc541acc4|0xc541ad70" ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-20 20:38 Message: Logged In: YES user_id=30293 They are pipes: Python 2.4b1 (#1, Oct 17 2004, 12:04:08) [GCC 3.3.3 [FreeBSD] 20031106] on freebsd5 Type "help", "copyright", "credits" or "license" for more information. >>> import os, subprocess >>> subprocess.call(["lsof", "-p", str(os.getpid())]) COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME python 1609 shalabh 0u VCHR 5,1 0t15826 114 /dev/ttyp1 python 1609 shalabh 1u VCHR 5,1 0t15826 114 /dev/ttyp1 python 1609 shalabh 2u VCHR 5,1 0t15826 114 /dev/ttyp1 python 1609 shalabh 3u PIPE 0xc541acc4 16384 ->0xc541ad70 python 1609 shalabh 4u PIPE 0xc541ad70 16384 ->0xc541acc4 0 ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-20 20:26 Message: Logged In: YES user_id=344921 Do you have lsof on your system? In that case, you could try this: >>> import os, subprocess >>> subprocess.call(["lsof", "-p", str(os.getpid())]) That should give you a hint of what fd 3 and 4 really are. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-20 19:38 Message: Logged In: YES user_id=30293 Same: $ python -S Python 2.3.4 (#2, Oct 15 2004, 13:44:35) [GCC 3.3.3 [FreeBSD] 20031106] on freebsd5 >>> import os >>> os.dup(0) 5 The above behaviour is also seen with 2.4b1 using -S. ---------------------------------------------------------------------- Comment By: Fredrik Lundh (effbot) Date: 2004-10-20 19:23 Message: Logged In: YES user_id=38376 Just curious, but what happens if you start Python using the - S option: $ python -S >>> import os >>> os.dup(0) ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-20 15:06 Message: Logged In: YES user_id=344921 >>> import os >>> os.dup(0) 5 I think this is the core of the problem. The test_close_fds test works like this: All file descriptors in the forked child (except 0,1,2) are closed. Then the Python binary is executed via execvp(). A small test program is passed to the Python binary via the -c command line option. If the OS and subprocess module works correctly, we can be sure of that by the time of the execve() system call, only file descriptors (0,1,2) are open (well, the errpipe as well, but let's leave that out for now). But, by the time the Python binary starts executing the small program, all sorts of things may have happened. I'm not really sure we can trust Python not to open files during startup. For example, if we have a PYTHONSTARTUP file, that open file will have a file descriptor, perhaps 3. On one hand, this bug could indicate a bug in the Python interpreter itself: perhaps a file descriptor leak. On the other hand, this test might be a bit too unsafe. So probably, this test should be removed. ---------------------------------------------------------------------- Comment By: Shalabh Chaturvedi (shalabh) Date: 2004-10-17 23:06 Message: Logged In: YES user_id=30293 Here's some more info. On my FreeBSD box, for both 2.3.4 and 2.4b1: >>> import os >>> os.dup(0) 5 But on my Win XP box for 2.3.3: >>> import os >>> os.dup(0) 3 I am guessing this may be relevant. Someone more knowledgable can comment. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048808&group_id=5470 From noreply at sourceforge.net Fri Oct 22 01:02:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 22 01:02:08 2004 Subject: [ python-Bugs-1051840 ] HTMLParser doesn't treat endtags in blah blah """ m = MyHandler() m.feed(s) This will raise an exception. I fixed the bug by changing the parse_endtag function on line 318 of HTMLParser to the following: def parse_endtag(self, i): rawdata = self.rawdata assert rawdata[i:i+2] == " if not match: return -1 j = match.end() match = endtagfind.match(rawdata, i) # if not match: self.error("bad end tag: %s" % `rawdata[i:j]`) tag = match.group(1) #START BUGFIX if self.interesting == interesting_cdata: #we're in of of the CDATA_CONTENT_ELEMENTS if tag == self.lasttag and tag in self.CDATA_CONTENT_ELEMENTS: #its the end of the CDATA_CONTENT_ELEMENTS tag we are in. self.handle_endtag(tag.lower()) self.clear_cdata_mode()#backto normal mode else: #we're inside the CDATA_CONTENT_ELEMENTS tag still. throw the tag to handle_data instead. self.handle_data(match.group()) else: #we're not in a CDATA_CONTENT_ELEMENTS tag. standard ending: self.handle_endtag(tag.lower()) return j ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1051840&group_id=5470 From noreply at sourceforge.net Fri Oct 22 01:04:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 22 01:04:54 2004 Subject: [ python-Bugs-1051840 ] HTMLParser doesn't treat endtags in blah blah """ m = MyHandler() m.feed(s) This will raise an exception. I fixed the bug by changing the parse_endtag function on line 318 of HTMLParser to the following: def parse_endtag(self, i): rawdata = self.rawdata assert rawdata[i:i+2] == " if not match: return -1 j = match.end() match = endtagfind.match(rawdata, i) # if not match: self.error("bad end tag: %s" % `rawdata[i:j]`) tag = match.group(1) #START BUGFIX if self.interesting == interesting_cdata: #we're in of of the CDATA_CONTENT_ELEMENTS if tag == self.lasttag and tag in self.CDATA_CONTENT_ELEMENTS: #its the end of the CDATA_CONTENT_ELEMENTS tag we are in. self.handle_endtag(tag.lower()) self.clear_cdata_mode()#backto normal mode else: #we're inside the CDATA_CONTENT_ELEMENTS tag still. throw the tag to handle_data instead. self.handle_data(match.group()) else: #we're not in a CDATA_CONTENT_ELEMENTS tag. standard ending: self.handle_endtag(tag.lower()) return j ---------------------------------------------------------------------- >Comment By: Luke Bradley (neptune235) Date: 2004-10-21 16:04 Message: Logged In: YES user_id=178561 oops, I didn't know this would remove indentation. Let me attach a file. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1051840&group_id=5470 From noreply at sourceforge.net Fri Oct 22 10:18:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 22 10:18:27 2004 Subject: [ python-Bugs-1052036 ] Appending non-empty element to a non-empty list yields None Message-ID: Bugs item #1052036, was opened at 2004-10-22 08:18 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052036&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Peer Janssen (peerjanssen) Assigned to: Nobody/Anonymous (nobody) Summary: Appending non-empty element to a non-empty list yields None Initial Comment: Appending a non-empty element to a non-empty list from a dictionary results in a list of None Scope: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 and Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] on win32 yield identical results. Description: The code (which is supposed to be used in a function for deleting duplicate files...) contains this line: print "Warning!",(md5,len),'*',MainDic[(md5,len)],'*',name,'*',MainDic[(md5,len)].append(name) which prints the following result: Warning! ('5e011eda91336d21d3bfc5d5a686bb44\n', '7217') * ['D:\_Arbeit\_In\_Roaming\_Telework\www.ivc.ca\part9.html', 'D:\_Arbeit\_I n\_Roaming\_Telework\www.ivc.ca\part9.html.primary'] * D:\_Arbeit\_In\_Roaming\_Telework\www.ivc.ca\part9.html.primary * None How can this be? The tuple is not empty, the entry for it in the dictionary exists, it is a list of strings, name is not empty and a string, but appending name to the list from the dictionary yields "None". Am I missing something, or is this really a bug? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052036&group_id=5470 From noreply at sourceforge.net Fri Oct 22 10:28:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 22 10:28:19 2004 Subject: [ python-Bugs-1052036 ] Appending non-empty element to a non-empty list yields None Message-ID: Bugs item #1052036, was opened at 2004-10-22 08:18 Message generated for change (Settings changed) made by peerjanssen You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052036&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None >Priority: 7 Submitted By: Peer Janssen (peerjanssen) Assigned to: Nobody/Anonymous (nobody) Summary: Appending non-empty element to a non-empty list yields None Initial Comment: Appending a non-empty element to a non-empty list from a dictionary results in a list of None Scope: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 and Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] on win32 yield identical results. Description: The code (which is supposed to be used in a function for deleting duplicate files...) contains this line: print "Warning!",(md5,len),'*',MainDic[(md5,len)],'*',name,'*',MainDic[(md5,len)].append(name) which prints the following result: Warning! ('5e011eda91336d21d3bfc5d5a686bb44\n', '7217') * ['D:\_Arbeit\_In\_Roaming\_Telework\www.ivc.ca\part9.html', 'D:\_Arbeit\_I n\_Roaming\_Telework\www.ivc.ca\part9.html.primary'] * D:\_Arbeit\_In\_Roaming\_Telework\www.ivc.ca\part9.html.primary * None How can this be? The tuple is not empty, the entry for it in the dictionary exists, it is a list of strings, name is not empty and a string, but appending name to the list from the dictionary yields "None". Am I missing something, or is this really a bug? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052036&group_id=5470 From noreply at sourceforge.net Fri Oct 22 10:58:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 22 10:58:37 2004 Subject: [ python-Bugs-1052036 ] Appending non-empty element to a non-empty list yields None Message-ID: Bugs item #1052036, was opened at 2004-10-22 08:18 Message generated for change (Comment added) made by peerjanssen You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052036&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 >Status: Closed >Resolution: Invalid Priority: 7 Submitted By: Peer Janssen (peerjanssen) Assigned to: Nobody/Anonymous (nobody) Summary: Appending non-empty element to a non-empty list yields None Initial Comment: Appending a non-empty element to a non-empty list from a dictionary results in a list of None Scope: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 and Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] on win32 yield identical results. Description: The code (which is supposed to be used in a function for deleting duplicate files...) contains this line: print "Warning!",(md5,len),'*',MainDic[(md5,len)],'*',name,'*',MainDic[(md5,len)].append(name) which prints the following result: Warning! ('5e011eda91336d21d3bfc5d5a686bb44\n', '7217') * ['D:\_Arbeit\_In\_Roaming\_Telework\www.ivc.ca\part9.html', 'D:\_Arbeit\_I n\_Roaming\_Telework\www.ivc.ca\part9.html.primary'] * D:\_Arbeit\_In\_Roaming\_Telework\www.ivc.ca\part9.html.primary * None How can this be? The tuple is not empty, the entry for it in the dictionary exists, it is a list of strings, name is not empty and a string, but appending name to the list from the dictionary yields "None". Am I missing something, or is this really a bug? ---------------------------------------------------------------------- >Comment By: Peer Janssen (peerjanssen) Date: 2004-10-22 08:58 Message: Logged In: YES user_id=896722 Ok, I missed something: >>> x=['a'] >>> print x.append('b') None >>> print x ['a', 'b'] So this is not a bug. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052036&group_id=5470 From noreply at sourceforge.net Fri Oct 22 12:11:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 22 12:11:21 2004 Subject: [ python-Bugs-1052098 ] Seting defaultencoding through an environment variable Message-ID: Bugs item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052098&group_id=5470 Category: Python Library Group: Feature Request Status: Open Resolution: None Priority: 5 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052098&group_id=5470 From noreply at sourceforge.net Fri Oct 22 12:12:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 22 12:12:30 2004 Subject: [ python-Bugs-1052098 ] Seting defaultencoding through an environment variable Message-ID: Bugs item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Settings changed) made by diedrich You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052098&group_id=5470 Category: Python Library Group: Feature Request Status: Open Resolution: None >Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052098&group_id=5470 From noreply at sourceforge.net Fri Oct 22 13:52:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 22 13:52:39 2004 Subject: [ python-Bugs-1045509 ] Problems with os.system and ulimit -f Message-ID: Bugs item #1045509, was opened at 2004-10-12 12:34 Message generated for change (Comment added) made by kowaltowski You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045509&group_id=5470 Category: Python Interpreter Core >Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Tomasz Kowaltowski (kowaltowski) Assigned to: Nobody/Anonymous (nobody) Summary: Problems with os.system and ulimit -f Initial Comment: Python version (running under Fedora Core 2 Linux): Python 2.3.3 (#1, May 7 2004, 10:31:40) [GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2 --------------------------------------------------------------------------- I found a problem while executing the 'ulimit -f' bash command within the 'os.system' function. According to the documentation this function should behave exactly like the stdlib function 'system'. However it does not happen as illustrated by the minimal Python and C examples: testsystem.py and testsystem.c (see attached zipped file). In these examples 'largefile' is a compiled C program which writes an endless file into the stdout (also attached). The C program testsystem.c works as expected and prints the following output: command: ulimit -f 10; largefile > xxx; result = 153 The Python program testsystem.py **does not stop**; if interrupted by Ctrl-C it prints: command: ulimit -f 10; largefile > xxx; result = 0 In both cases the output file 'xxx' has 10240 bytes, ie, 10 blocks as limited by 'ulimit'. It is interesting though that the command 'ulimit -t 1' (CPU time) produces correct results under both Python and C versions, ie, interrupts the execution and prints: command: ulimit -t 1; largefile > xxx; result = 137 ---------------------------------------------------------------------- >Comment By: Tomasz Kowaltowski (kowaltowski) Date: 2004-10-22 09:52 Message: Logged In: YES user_id=185428 I also tested the new version "Python 2.4b1" -- the problem still occurs :-(. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045509&group_id=5470 From noreply at sourceforge.net Fri Oct 22 16:31:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 22 16:31:52 2004 Subject: [ python-Feature Requests-1052242 ] Attempt to run all atexit handlers Message-ID: Feature Requests item #1052242, was opened at 2004-10-22 06:31 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052242&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Greg Chapman (glchapman) Assigned to: Nobody/Anonymous (nobody) Summary: Attempt to run all atexit handlers Initial Comment: Currently, if an atexit handler raises an exception during _run_exitfuncs, none of the remaining handlers in the _exithandlers list gets run. I would prefer it if _run_exitfuncs did something like: while _exithandlers: func, targs, kargs = _exithandlers.pop() try: func(*targs, **kargs) except SystemExit: raise except: import sys, traceback print >> sys.stderr, "Error in sys.exitfunc:" traceback.print_exc() The SystemExit except clause is there because it looks like (given its special treatment in call_sys_exitfunc) SystemExit is envisioned as a way of intentionally breaking out of the exithandlers loop. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052242&group_id=5470 From noreply at sourceforge.net Fri Oct 22 17:14:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 22 17:14:03 2004 Subject: [ python-Bugs-1052274 ] page on python api down Message-ID: Bugs item #1052274, was opened at 2004-10-22 11:14 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052274&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: mperpick (mperpick) Assigned to: Nobody/Anonymous (nobody) Summary: page on python api down Initial Comment: the following page is down.... http://www.hig.se/~jackson/manuals/python- lib/Installing-your-CGI-script-on-a-Unix-system.html thanks. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052274&group_id=5470 From noreply at sourceforge.net Fri Oct 22 20:02:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 22 20:02:37 2004 Subject: [ python-Bugs-1051840 ] HTMLParser doesn't treat endtags in blah blah """ m = MyHandler() m.feed(s) This will raise an exception. I fixed the bug by changing the parse_endtag function on line 318 of HTMLParser to the following: def parse_endtag(self, i): rawdata = self.rawdata assert rawdata[i:i+2] == " if not match: return -1 j = match.end() match = endtagfind.match(rawdata, i) # if not match: self.error("bad end tag: %s" % `rawdata[i:j]`) tag = match.group(1) #START BUGFIX if self.interesting == interesting_cdata: #we're in of of the CDATA_CONTENT_ELEMENTS if tag == self.lasttag and tag in self.CDATA_CONTENT_ELEMENTS: #its the end of the CDATA_CONTENT_ELEMENTS tag we are in. self.handle_endtag(tag.lower()) self.clear_cdata_mode()#backto normal mode else: #we're inside the CDATA_CONTENT_ELEMENTS tag still. throw the tag to handle_data instead. self.handle_data(match.group()) else: #we're not in a CDATA_CONTENT_ELEMENTS tag. standard ending: self.handle_endtag(tag.lower()) return j ---------------------------------------------------------------------- Comment By: Richard Brodie (leogah) Date: 2004-10-22 18:02 Message: Logged In: YES user_id=356893 Although a fix may be worthwhile, as this happens a lot in practice, HTMLParser is following the letter of the law in throwing exceptions on pages that aren't strictly valid. http://www.w3.org/TR/html4/appendix/notes.html#notes- specifying-data ---------------------------------------------------------------------- Comment By: Luke Bradley (neptune235) Date: 2004-10-21 23:04 Message: Logged In: YES user_id=178561 oops, I didn't know this would remove indentation. Let me attach a file. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1051840&group_id=5470 From noreply at sourceforge.net Fri Oct 22 22:11:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 22 22:11:20 2004 Subject: [ python-Bugs-962442 ] Konqueror can't render docs because of malformed HTML Message-ID: Bugs item #962442, was opened at 2004-05-28 20:39 Message generated for change (Comment added) made by leogah You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962442&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Klaus-J Wolf (yanestra) Assigned to: Nobody/Anonymous (nobody) Summary: Konqueror can't render docs because of malformed HTML Initial Comment: At least on page Python-Docs-2.3.3/lib/re-syntax.html Konqueror doesn't render the
definition part correctly (the
part is not being displayed). The reason appears to be malformed HTML, as to cite Stephan Kulow of KDE Project: "No, this is not correct HTML and you might want to report it to the python maintainers. I wonder why they use
(which is correct HTML) when they put a

in there (which is not correct). Send your snippet through tidy to see how it looks as correct HTML (and as such it's rendered correctly by konqueror)" You might want to visit KDE's bugzilla: http://bugs.kde.org/show_bug.cgi?id=82335 ---------------------------------------------------------------------- Comment By: Richard Brodie (leogah) Date: 2004-10-22 20:11 Message: Logged In: YES user_id=356893 Fixing up the HTML by hand didn't correct the rendering in Konqueror, and the related KDE bug is now marked as fixed in the KDE CVS repository. Bug 899423 covers the LINK issue. ---------------------------------------------------------------------- Comment By: Felix Wiemann (felixwiemann) Date: 2004-07-28 15:20 Message: Logged In: YES user_id=1014490 In fact, the HTML code is not valid. See, e.g., http://validator.w3.org/check?uri=http://www.python.org/dev/doc/devel/lib/boolean.html&charset=%28detect+automatically%29&doctype=Inline&ss=1&sp=1&outline=1&No200=1&verbose=1 And the link-tag which links to the next page appears twice. E.g., there is twice on the page http://www.python.org/dev/doc/devel/lib/boolean.html This is very annoying when using the link toolbar for Mozilla Firefox. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962442&group_id=5470 From noreply at sourceforge.net Fri Oct 22 23:50:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 22 23:50:27 2004 Subject: [ python-Bugs-1052503 ] pdb runcall should accept keyword arguments Message-ID: Bugs item #1052503, was opened at 2004-10-22 19:50 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052503&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Leonardo Rochael Almeida (rochael) Assigned to: Nobody/Anonymous (nobody) Summary: pdb runcall should accept keyword arguments Initial Comment: pdb.runcall(), pdb.Pdb.runcall() and bdb.Bdb.runcall() could all accept keyword arguments to pass them to the debugged callable. The implementation seems trivial, just add "**kw" on the signatures and invocations all the way thru the actual call to the callable. I verified this on Python 2.3.4. No idea if the bug is also on 2.4 betas ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052503&group_id=5470 From noreply at sourceforge.net Sat Oct 23 01:52:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 01:56:17 2004 Subject: [ python-Bugs-1051840 ] HTMLParser doesn't treat endtags in blah blah """ m = MyHandler() m.feed(s) This will raise an exception. I fixed the bug by changing the parse_endtag function on line 318 of HTMLParser to the following: def parse_endtag(self, i): rawdata = self.rawdata assert rawdata[i:i+2] == " if not match: return -1 j = match.end() match = endtagfind.match(rawdata, i) # if not match: self.error("bad end tag: %s" % `rawdata[i:j]`) tag = match.group(1) #START BUGFIX if self.interesting == interesting_cdata: #we're in of of the CDATA_CONTENT_ELEMENTS if tag == self.lasttag and tag in self.CDATA_CONTENT_ELEMENTS: #its the end of the CDATA_CONTENT_ELEMENTS tag we are in. self.handle_endtag(tag.lower()) self.clear_cdata_mode()#backto normal mode else: #we're inside the CDATA_CONTENT_ELEMENTS tag still. throw the tag to handle_data instead. self.handle_data(match.group()) else: #we're not in a CDATA_CONTENT_ELEMENTS tag. standard ending: self.handle_endtag(tag.lower()) return j ---------------------------------------------------------------------- >Comment By: Luke Bradley (neptune235) Date: 2004-10-22 16:52 Message: Logged In: YES user_id=178561 Although a fix may be worthwhile, as this happens a lot in practice, HTMLParser is following the letter of the law in throwing exceptions on pages that aren't strictly valid. http://www.w3.org/TR/html4/appendix/notes.html#notes- specifying-data Well you're right, I'll be damned! Hmm. I want to use HTMLParser to access other people's pages on the net, and I can't fix their bad HTML. The problem here is I'm not sure how to handle this at the level of my Handler, without inadvertantly changing thier javascript by doing something like: handle_data("") in the handle_entag event. Which lowercases the tag. Is there a way to access the literal string of the endtag in my handler I wonder? If not, it might be useful to add some functionality to HTMLParser that allows us to handle invalid HTML at the level of our handler without sacrificing HTMLParsers commitment to standards compliance. ---------------------------------------------------------------------- Comment By: Richard Brodie (leogah) Date: 2004-10-22 11:02 Message: Logged In: YES user_id=356893 Although a fix may be worthwhile, as this happens a lot in practice, HTMLParser is following the letter of the law in throwing exceptions on pages that aren't strictly valid. http://www.w3.org/TR/html4/appendix/notes.html#notes- specifying-data ---------------------------------------------------------------------- Comment By: Luke Bradley (neptune235) Date: 2004-10-21 16:04 Message: Logged In: YES user_id=178561 oops, I didn't know this would remove indentation. Let me attach a file. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1051840&group_id=5470 From noreply at sourceforge.net Sat Oct 23 05:01:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 05:01:18 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-18 03:24 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-23 05:01 Message: Logged In: YES user_id=469548 You know what: I think we should remove the os.listdir error handling: * The docs only specify "errors resulting from *failed removals*" as being handled. If they didn't we should worry about OSErrors from the os.path.* calls in os.walk() too. * The os.listdir error handling hasn't been in a final release, so no backwards compatibility problems yet. * We can't get the path right for os.listdir. * The problem also existed before the rewrite using os.walk(), so reverting to that is not a solution. Patch (the last?) is attached. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-21 06:42 Message: Logged In: YES user_id=6380 Yes. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-20 22:30 Message: Logged In: YES user_id=469548 With 'be careful, it may not exist.', do you mean rmtree can't rely on exc.filename being set? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 17:18 Message: Logged In: YES user_id=6380 This looks fine, and can be checked in. There's one minor remaining issue, but I don't see how to fix it without changing the signature for the onerror function that os.walk() takes, and that's a no-no: when the problem is a directory permission preventing os.listdir() to work, the path passed to the rmtree caller's onerror function is always the toplevel path, rather than the directory that couldn't be listed. The real path is included in the error message. (Hmm, perhaps you could get it out of there? I think it's an attribute of the exception object; but be careful, it may not exist.) It's unfortunate that the error argument to the onerror function is a sys.exc_info() triple rather than an exception object (calling sys.exc_info() is expensive), but that's an API choice from the past that we can't change until Python 3.0. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 10:57 Message: Logged In: YES user_id=469548 A new patch is attached. I factored out the handle_call_error method so exception handling is more local. Also, errors in os.walk were already being handled using the _raise_err/except OSError combination, but we now do it using the onerror argument to os.walk. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 01:42 Message: Logged In: YES user_id=6380 (Actually, os.walk has an onerror argument, it just uses a different convention and default, so we could thread this through easily.) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 01:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 00:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 08:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Sat Oct 23 09:03:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 09:03:11 2004 Subject: [ python-Bugs-548176 ] urlparse doesn't handle host?bla Message-ID: Bugs item #548176, was opened at 2002-04-24 17:36 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=548176&group_id=5470 Category: Python Library Group: Python 2.2 Status: Open Resolution: None Priority: 5 Submitted By: Markus Demleitner (msdemlei) Assigned to: Nobody/Anonymous (nobody) Summary: urlparse doesn't handle host?bla Initial Comment: The urlparse module (at least in 2.2 and 2.1, Linux) doesn't handle URLs of the form http://www.maerkischeallgemeine.de?loc_id=49 correctly -- everything up to the 9 ends up in the host. I didn't check the RFC, but in the real world URLs like this do show up. urlparse works fine when there's a trailing slash on the host name: http://www.maerkischeallgemeine.de/?loc_id=49 Example: <pre> >>> import urlparse >>> urlparse.urlparse("http://www.maerkischeallgemeine.de/?loc_id=49") ('http', 'www.maerkischeallgemeine.de', '/', '', 'loc_id=49', '') >>> urlparse.urlparse("http://www.maerkischeallgemeine.de?loc_id=49") ('http', 'www.maerkischeallgemeine.de?loc_id=49', '', '', '', '') </pre> This has serious implications for urllib, since urllib.urlopen will fail for URLs like the second one, and with a pretty mysterious exception ("host not found") at that. ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-23 09:03 Message: Logged In: YES user_id=469548 Somehow I think I'm missing something. Please check my line of reasoning: 1. http://foo?bar=baz is a legal URL. 2. urlparse's 'Network location' should be the same as from rfc2396. 3. Inside an unescaped '?' is not allowed. Rather: is terminated by the '?'. 4. Currently the 'network location' for http://foo?bar=baz would be 'foo?bar=baz. 5. If 'network location' should be the same as , it should also be terminated by the '?'. So shouldn't urlparse.urlsplit('http://foo?bar=baz') return ('http', 'foo', '', '', 'bar=baz', ''), as patch 712317 implements? ---------------------------------------------------------------------- Comment By: Mike Rovner (mrovner) Date: 2004-01-27 02:13 Message: Logged In: YES user_id=162094 According to RFC2396 (ftp://ftp.isi.edu/in-notes/rfc2396.txt) absoluteURI (part 3 URI Syntactic Components) can be: """ ://? each of which, except , may be absent from a particular URI. """ Later on (3.2): """ The authority component is preceded by a double slash "//" and is terminated by the next slash "/", question-mark "?", or by the end of the URI. """ So URL "http://server?query" is perfectly legal and shall be allowed and patch 712317 rejected. ---------------------------------------------------------------------- Comment By: Steven Taschuk (staschuk) Date: 2003-03-30 22:19 Message: Logged In: YES user_id=666873 For comparison, RFC 1738 section 3.3: An HTTP URL takes the form: http://<host>:<port>/<path>?<searchpart> [...] If neither <path> nor <searchpart> is present, the "/" may also be omitted. ... which does not outright say the '/' may *not* be omitted if <path> is absent but <searchpart> is present (though imho that's implied). But even if the / may not be omitted in this case, ? is not allowed in the authority component under either RFC 2396 or RFC 1738, so urlparse should either treat it as a delimiter or reject the URL as malformed. The principle of being lenient in what you accept favours the former. I've just submitted a patch (712317) for this. ---------------------------------------------------------------------- Comment By: Jeff Epler (jepler) Date: 2002-11-17 17:56 Message: Logged In: YES user_id=2772 This actually appears to be permitted by RFC2396 [http://www.ietf.org/rfc/rfc2396.txt]. See section 3.2: 3.2. Authority Component Many URI schemes include a top hierarchical element for a naming authority, such that the namespace defined by the remainder of the URI is governed by that authority. This authority component is typically defined by an Internet-based server or a scheme-specific registry of naming authorities. authority = server | reg_name The authority component is preceded by a double slash "//" and is terminated by the next slash "/", question-mark "?", or by the end of the URI. Within the authority component, the characters ";", ":", "@", "?", and "/" are reserved. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=548176&group_id=5470 From noreply at sourceforge.net Sat Oct 23 09:44:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 09:44:57 2004 Subject: [ python-Bugs-548176 ] urlparse doesn't handle host?bla Message-ID: Bugs item #548176, was opened at 2002-04-24 08:36 Message generated for change (Comment added) made by mrovner You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=548176&group_id=5470 Category: Python Library Group: Python 2.2 Status: Open Resolution: None Priority: 5 Submitted By: Markus Demleitner (msdemlei) Assigned to: Nobody/Anonymous (nobody) Summary: urlparse doesn't handle host?bla Initial Comment: The urlparse module (at least in 2.2 and 2.1, Linux) doesn't handle URLs of the form http://www.maerkischeallgemeine.de?loc_id=49 correctly -- everything up to the 9 ends up in the host. I didn't check the RFC, but in the real world URLs like this do show up. urlparse works fine when there's a trailing slash on the host name: http://www.maerkischeallgemeine.de/?loc_id=49 Example: <pre> >>> import urlparse >>> urlparse.urlparse("http://www.maerkischeallgemeine.de/?loc_id=49") ('http', 'www.maerkischeallgemeine.de', '/', '', 'loc_id=49', '') >>> urlparse.urlparse("http://www.maerkischeallgemeine.de?loc_id=49") ('http', 'www.maerkischeallgemeine.de?loc_id=49', '', '', '', '') </pre> This has serious implications for urllib, since urllib.urlopen will fail for URLs like the second one, and with a pretty mysterious exception ("host not found") at that. ---------------------------------------------------------------------- Comment By: Mike Rovner (mrovner) Date: 2004-10-23 00:44 Message: Logged In: YES user_id=162094 I'm sorry, I misunderstood the patch. If it accepts such URL and split it at '?', it's perfectly fine. It shall not reject such URL as malformed. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-23 00:03 Message: Logged In: YES user_id=469548 Somehow I think I'm missing something. Please check my line of reasoning: 1. http://foo?bar=baz is a legal URL. 2. urlparse's 'Network location' should be the same as from rfc2396. 3. Inside an unescaped '?' is not allowed. Rather: is terminated by the '?'. 4. Currently the 'network location' for http://foo?bar=baz would be 'foo?bar=baz. 5. If 'network location' should be the same as , it should also be terminated by the '?'. So shouldn't urlparse.urlsplit('http://foo?bar=baz') return ('http', 'foo', '', '', 'bar=baz', ''), as patch 712317 implements? ---------------------------------------------------------------------- Comment By: Mike Rovner (mrovner) Date: 2004-01-26 17:13 Message: Logged In: YES user_id=162094 According to RFC2396 (ftp://ftp.isi.edu/in-notes/rfc2396.txt) absoluteURI (part 3 URI Syntactic Components) can be: """ ://? each of which, except , may be absent from a particular URI. """ Later on (3.2): """ The authority component is preceded by a double slash "//" and is terminated by the next slash "/", question-mark "?", or by the end of the URI. """ So URL "http://server?query" is perfectly legal and shall be allowed and patch 712317 rejected. ---------------------------------------------------------------------- Comment By: Steven Taschuk (staschuk) Date: 2003-03-30 12:19 Message: Logged In: YES user_id=666873 For comparison, RFC 1738 section 3.3: An HTTP URL takes the form: http://<host>:<port>/<path>?<searchpart> [...] If neither <path> nor <searchpart> is present, the "/" may also be omitted. ... which does not outright say the '/' may *not* be omitted if <path> is absent but <searchpart> is present (though imho that's implied). But even if the / may not be omitted in this case, ? is not allowed in the authority component under either RFC 2396 or RFC 1738, so urlparse should either treat it as a delimiter or reject the URL as malformed. The principle of being lenient in what you accept favours the former. I've just submitted a patch (712317) for this. ---------------------------------------------------------------------- Comment By: Jeff Epler (jepler) Date: 2002-11-17 08:56 Message: Logged In: YES user_id=2772 This actually appears to be permitted by RFC2396 [http://www.ietf.org/rfc/rfc2396.txt]. See section 3.2: 3.2. Authority Component Many URI schemes include a top hierarchical element for a naming authority, such that the namespace defined by the remainder of the URI is governed by that authority. This authority component is typically defined by an Internet-based server or a scheme-specific registry of naming authorities. authority = server | reg_name The authority component is preceded by a double slash "//" and is terminated by the next slash "/", question-mark "?", or by the end of the URI. Within the authority component, the characters ";", ":", "@", "?", and "/" are reserved. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=548176&group_id=5470 From noreply at sourceforge.net Sat Oct 23 10:31:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 10:31:25 2004 Subject: [ python-Bugs-1052274 ] page on python api down Message-ID: Bugs item #1052274, was opened at 2004-10-22 17:14 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052274&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: mperpick (mperpick) Assigned to: Nobody/Anonymous (nobody) Summary: page on python api down Initial Comment: the following page is down.... http://www.hig.se/~jackson/manuals/python- lib/Installing-your-CGI-script-on-a-Unix-system.html thanks. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:31 Message: Logged In: YES user_id=21627 We are not maintaining hig.se, please contact the operator of that server. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052274&group_id=5470 From noreply at sourceforge.net Sat Oct 23 10:32:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 10:32:59 2004 Subject: [ python-Feature Requests-1052098 ] Seting defaultencoding through an environment variable Message-ID: Feature Requests item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 >Category: Python Library >Group: None Status: Open Resolution: None Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:32 Message: Logged In: YES user_id=21627 Why do you want this change? In the long run, Python will drop the notion of a default encoding; code should be changed to avoid relying on the default encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 From noreply at sourceforge.net Sat Oct 23 11:08:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 11:08:56 2004 Subject: [ python-Feature Requests-1052098 ] Seting defaultencoding through an environment variable Message-ID: Feature Requests item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Comment added) made by diedrich You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- >Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 11:08 Message: Logged In: YES user_id=442849 Even if the notion of default encoding were dropped from Python, my application will need a global variable to have one encoding that is used most of the time. That`s what I`m using the default encoding for right now. Also, at least one of the 3d party modules I'm using does the same and it makes tons of sense. Changing the default encoding from within my application doesn't work, because sys.setdefaultencoding() gets removed (which makes sense, too, of course). So I need a custom sitecustomize.py for every project which may or may not be adhered to, depending on the PYTHONPATH variable. That`s why a PYTHON_DEFAULT_ENCODING variable would be very usefull to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:32 Message: Logged In: YES user_id=21627 Why do you want this change? In the long run, Python will drop the notion of a default encoding; code should be changed to avoid relying on the default encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 From noreply at sourceforge.net Sat Oct 23 11:52:09 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 11:52:15 2004 Subject: [ python-Feature Requests-1052098 ] Seting defaultencoding through an environment variable Message-ID: Feature Requests item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 11:52 Message: Logged In: YES user_id=21627 If your application just needs a global setting, it can come up with its own environment variable, right? Just set MY_DEFAULT_ENCODING, and do encoding=os.environ["MY_DEFAULT_ENCODING"] Alternatively, try using locale.getpreferredencoding(); this may allow to avoid an additional environment variable altogether. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 11:08 Message: Logged In: YES user_id=442849 Even if the notion of default encoding were dropped from Python, my application will need a global variable to have one encoding that is used most of the time. That`s what I`m using the default encoding for right now. Also, at least one of the 3d party modules I'm using does the same and it makes tons of sense. Changing the default encoding from within my application doesn't work, because sys.setdefaultencoding() gets removed (which makes sense, too, of course). So I need a custom sitecustomize.py for every project which may or may not be adhered to, depending on the PYTHONPATH variable. That`s why a PYTHON_DEFAULT_ENCODING variable would be very usefull to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:32 Message: Logged In: YES user_id=21627 Why do you want this change? In the long run, Python will drop the notion of a default encoding; code should be changed to avoid relying on the default encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 From noreply at sourceforge.net Sat Oct 23 12:05:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 12:05:26 2004 Subject: [ python-Feature Requests-1052098 ] Seting defaultencoding through an environment variable Message-ID: Feature Requests item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Comment added) made by diedrich You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- >Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:05 Message: Logged In: YES user_id=442849 Yes, you are right. I could also replace the global unicode() function with my own version that adheres to my global encoding settings. But that doesn't seem the right thing to do. Anyway, this is not so much a technical decision but one that touches design. Anyway, it's just a small thing I tought usefull. If the defaultencoding goes away altogether, I'll have to make something up on my own anyway. Thanks for your comments! ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 11:52 Message: Logged In: YES user_id=21627 If your application just needs a global setting, it can come up with its own environment variable, right? Just set MY_DEFAULT_ENCODING, and do encoding=os.environ["MY_DEFAULT_ENCODING"] Alternatively, try using locale.getpreferredencoding(); this may allow to avoid an additional environment variable altogether. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 11:08 Message: Logged In: YES user_id=442849 Even if the notion of default encoding were dropped from Python, my application will need a global variable to have one encoding that is used most of the time. That`s what I`m using the default encoding for right now. Also, at least one of the 3d party modules I'm using does the same and it makes tons of sense. Changing the default encoding from within my application doesn't work, because sys.setdefaultencoding() gets removed (which makes sense, too, of course). So I need a custom sitecustomize.py for every project which may or may not be adhered to, depending on the PYTHONPATH variable. That`s why a PYTHON_DEFAULT_ENCODING variable would be very usefull to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:32 Message: Logged In: YES user_id=21627 Why do you want this change? In the long run, Python will drop the notion of a default encoding; code should be changed to avoid relying on the default encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 From noreply at sourceforge.net Sat Oct 23 12:09:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 12:09:44 2004 Subject: [ python-Feature Requests-1052098 ] Seting defaultencoding through an environment variable Message-ID: Feature Requests item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 Category: Python Library Group: None >Status: Closed >Resolution: Rejected Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:09 Message: Logged In: YES user_id=21627 I take it that you are withdrawing your feature request, then? Closing it as rejected; if you still think this should be done, please indicate so. It would be good if you then could give some reasoning beyond "it would be useful" ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:05 Message: Logged In: YES user_id=442849 Yes, you are right. I could also replace the global unicode() function with my own version that adheres to my global encoding settings. But that doesn't seem the right thing to do. Anyway, this is not so much a technical decision but one that touches design. Anyway, it's just a small thing I tought usefull. If the defaultencoding goes away altogether, I'll have to make something up on my own anyway. Thanks for your comments! ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 11:52 Message: Logged In: YES user_id=21627 If your application just needs a global setting, it can come up with its own environment variable, right? Just set MY_DEFAULT_ENCODING, and do encoding=os.environ["MY_DEFAULT_ENCODING"] Alternatively, try using locale.getpreferredencoding(); this may allow to avoid an additional environment variable altogether. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 11:08 Message: Logged In: YES user_id=442849 Even if the notion of default encoding were dropped from Python, my application will need a global variable to have one encoding that is used most of the time. That`s what I`m using the default encoding for right now. Also, at least one of the 3d party modules I'm using does the same and it makes tons of sense. Changing the default encoding from within my application doesn't work, because sys.setdefaultencoding() gets removed (which makes sense, too, of course). So I need a custom sitecustomize.py for every project which may or may not be adhered to, depending on the PYTHONPATH variable. That`s why a PYTHON_DEFAULT_ENCODING variable would be very usefull to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:32 Message: Logged In: YES user_id=21627 Why do you want this change? In the long run, Python will drop the notion of a default encoding; code should be changed to avoid relying on the default encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 From noreply at sourceforge.net Sat Oct 23 12:40:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 12:40:47 2004 Subject: [ python-Feature Requests-1052098 ] Seting defaultencoding through an environment variable Message-ID: Feature Requests item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Comment added) made by diedrich You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 Category: Python Library Group: None Status: Closed Resolution: Rejected Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- >Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:40 Message: Logged In: YES user_id=442849 Well... what more than useful is it supposed to be? Earh-shaking? Spiritually rewarding? ;-) As long as there is a default encoding I'd like to be able to set it in a simpler way than I can now. And I'd like to be able to set it once, at interpreter startup. An environment variable seems like a reasonable way to do that. If the notion of default encoding is removed from Python's library the suggested modification is going to go away as well. If you want to reject the feature request because the notion of default encoding is depricated, than so be it. From my point of view the suggested modification is something I'd like to see. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:09 Message: Logged In: YES user_id=21627 I take it that you are withdrawing your feature request, then? Closing it as rejected; if you still think this should be done, please indicate so. It would be good if you then could give some reasoning beyond "it would be useful" ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:05 Message: Logged In: YES user_id=442849 Yes, you are right. I could also replace the global unicode() function with my own version that adheres to my global encoding settings. But that doesn't seem the right thing to do. Anyway, this is not so much a technical decision but one that touches design. Anyway, it's just a small thing I tought usefull. If the defaultencoding goes away altogether, I'll have to make something up on my own anyway. Thanks for your comments! ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 11:52 Message: Logged In: YES user_id=21627 If your application just needs a global setting, it can come up with its own environment variable, right? Just set MY_DEFAULT_ENCODING, and do encoding=os.environ["MY_DEFAULT_ENCODING"] Alternatively, try using locale.getpreferredencoding(); this may allow to avoid an additional environment variable altogether. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 11:08 Message: Logged In: YES user_id=442849 Even if the notion of default encoding were dropped from Python, my application will need a global variable to have one encoding that is used most of the time. That`s what I`m using the default encoding for right now. Also, at least one of the 3d party modules I'm using does the same and it makes tons of sense. Changing the default encoding from within my application doesn't work, because sys.setdefaultencoding() gets removed (which makes sense, too, of course). So I need a custom sitecustomize.py for every project which may or may not be adhered to, depending on the PYTHONPATH variable. That`s why a PYTHON_DEFAULT_ENCODING variable would be very usefull to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:32 Message: Logged In: YES user_id=21627 Why do you want this change? In the long run, Python will drop the notion of a default encoding; code should be changed to avoid relying on the default encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 From noreply at sourceforge.net Sat Oct 23 12:55:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 12:55:35 2004 Subject: [ python-Feature Requests-1052098 ] Seting defaultencoding through an environment variable Message-ID: Feature Requests item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 Category: Python Library Group: None >Status: Open Resolution: Rejected Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:55 Message: Logged In: YES user_id=21627 No, I'm asking *why* you consider it useful. For example a) here is this and that problem, and there is absolutely no way to express this in Python today. If that feature is added, I can write it in the following way. b) here is this and that problem. Today, I have to write it in this way. With the feature added, I can write it in that way. That way is better than this way because of such and such reason. IOW, I would like to see what you are using the default encoding *for* that makes it desirable to set it through an environment variable. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:40 Message: Logged In: YES user_id=442849 Well... what more than useful is it supposed to be? Earh-shaking? Spiritually rewarding? ;-) As long as there is a default encoding I'd like to be able to set it in a simpler way than I can now. And I'd like to be able to set it once, at interpreter startup. An environment variable seems like a reasonable way to do that. If the notion of default encoding is removed from Python's library the suggested modification is going to go away as well. If you want to reject the feature request because the notion of default encoding is depricated, than so be it. From my point of view the suggested modification is something I'd like to see. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:09 Message: Logged In: YES user_id=21627 I take it that you are withdrawing your feature request, then? Closing it as rejected; if you still think this should be done, please indicate so. It would be good if you then could give some reasoning beyond "it would be useful" ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:05 Message: Logged In: YES user_id=442849 Yes, you are right. I could also replace the global unicode() function with my own version that adheres to my global encoding settings. But that doesn't seem the right thing to do. Anyway, this is not so much a technical decision but one that touches design. Anyway, it's just a small thing I tought usefull. If the defaultencoding goes away altogether, I'll have to make something up on my own anyway. Thanks for your comments! ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 11:52 Message: Logged In: YES user_id=21627 If your application just needs a global setting, it can come up with its own environment variable, right? Just set MY_DEFAULT_ENCODING, and do encoding=os.environ["MY_DEFAULT_ENCODING"] Alternatively, try using locale.getpreferredencoding(); this may allow to avoid an additional environment variable altogether. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 11:08 Message: Logged In: YES user_id=442849 Even if the notion of default encoding were dropped from Python, my application will need a global variable to have one encoding that is used most of the time. That`s what I`m using the default encoding for right now. Also, at least one of the 3d party modules I'm using does the same and it makes tons of sense. Changing the default encoding from within my application doesn't work, because sys.setdefaultencoding() gets removed (which makes sense, too, of course). So I need a custom sitecustomize.py for every project which may or may not be adhered to, depending on the PYTHONPATH variable. That`s why a PYTHON_DEFAULT_ENCODING variable would be very usefull to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:32 Message: Logged In: YES user_id=21627 Why do you want this change? In the long run, Python will drop the notion of a default encoding; code should be changed to avoid relying on the default encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 From noreply at sourceforge.net Sat Oct 23 13:30:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 13:30:16 2004 Subject: [ python-Feature Requests-1052098 ] Seting defaultencoding through an environment variable Message-ID: Feature Requests item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Comment added) made by diedrich You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 Category: Python Library Group: None Status: Open Resolution: Rejected Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- >Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 13:30 Message: Logged In: YES user_id=442849 Ah.. ok.. some examples: I'm writing Web-Applications using CGI or FastCGI. The webserver will pass user input as plain strings but within my program I want to use plain strings as little as possible to avoid all kinds of problems with diacritical characters (foremost umlauts and the EUR symbol). The websites the CGI scripts are used with usually have one charset for all pages. But some sites use iso-8859-1 and some use utf-8 and I don't want to modify my scripts to fit the charset of the site. Rather I'd like to use Apache's SetEnv directive to tell my scripts what charset their input is going to be in. Yes, I know, there are other ways to handle this :-) I keep using XIST for html output (http://www.livinglogic.de/Python). XIST tries to avoid using the default encoding, probably in the light of it being depricated. But it does not avoid it completely. There are a number of subtle cases in which it still depends on it and in all these cases I run into run-time errors which can be avoided in a snap by setting the default encoding to what my actual default encoding is. I use XIST with Zope and FastCGI, running several instances of the same interpreter on the same machine, with different default encodings each (mostly utf-8, but some iso-8859-1, which I can't just convert). My own OSS project orm (http://t4w.de/orm) carefully seperates the database's and the application's charset but uses Python's default encoding as the default for each and every setting. Those settings need to be set explicitly if the default encoding does not match the encoding actually used, cluttering the sourcecode and making maintaince more difficult. I use orm in connection with all the above cases. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:55 Message: Logged In: YES user_id=21627 No, I'm asking *why* you consider it useful. For example a) here is this and that problem, and there is absolutely no way to express this in Python today. If that feature is added, I can write it in the following way. b) here is this and that problem. Today, I have to write it in this way. With the feature added, I can write it in that way. That way is better than this way because of such and such reason. IOW, I would like to see what you are using the default encoding *for* that makes it desirable to set it through an environment variable. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:40 Message: Logged In: YES user_id=442849 Well... what more than useful is it supposed to be? Earh-shaking? Spiritually rewarding? ;-) As long as there is a default encoding I'd like to be able to set it in a simpler way than I can now. And I'd like to be able to set it once, at interpreter startup. An environment variable seems like a reasonable way to do that. If the notion of default encoding is removed from Python's library the suggested modification is going to go away as well. If you want to reject the feature request because the notion of default encoding is depricated, than so be it. From my point of view the suggested modification is something I'd like to see. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:09 Message: Logged In: YES user_id=21627 I take it that you are withdrawing your feature request, then? Closing it as rejected; if you still think this should be done, please indicate so. It would be good if you then could give some reasoning beyond "it would be useful" ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:05 Message: Logged In: YES user_id=442849 Yes, you are right. I could also replace the global unicode() function with my own version that adheres to my global encoding settings. But that doesn't seem the right thing to do. Anyway, this is not so much a technical decision but one that touches design. Anyway, it's just a small thing I tought usefull. If the defaultencoding goes away altogether, I'll have to make something up on my own anyway. Thanks for your comments! ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 11:52 Message: Logged In: YES user_id=21627 If your application just needs a global setting, it can come up with its own environment variable, right? Just set MY_DEFAULT_ENCODING, and do encoding=os.environ["MY_DEFAULT_ENCODING"] Alternatively, try using locale.getpreferredencoding(); this may allow to avoid an additional environment variable altogether. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 11:08 Message: Logged In: YES user_id=442849 Even if the notion of default encoding were dropped from Python, my application will need a global variable to have one encoding that is used most of the time. That`s what I`m using the default encoding for right now. Also, at least one of the 3d party modules I'm using does the same and it makes tons of sense. Changing the default encoding from within my application doesn't work, because sys.setdefaultencoding() gets removed (which makes sense, too, of course). So I need a custom sitecustomize.py for every project which may or may not be adhered to, depending on the PYTHONPATH variable. That`s why a PYTHON_DEFAULT_ENCODING variable would be very usefull to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:32 Message: Logged In: YES user_id=21627 Why do you want this change? In the long run, Python will drop the notion of a default encoding; code should be changed to avoid relying on the default encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 From noreply at sourceforge.net Sat Oct 23 13:50:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 13:50:11 2004 Subject: [ python-Feature Requests-1052098 ] Seting defaultencoding through an environment variable Message-ID: Feature Requests item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 Category: Python Library Group: None Status: Open Resolution: Rejected Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 13:50 Message: Logged In: YES user_id=21627 I see. For the XIST case, it probably would be best to correct XIST, to always use explicit encodings. Alternatively, would it be possible to pass byte strings instead of Unicode objects to XIST in the cases which XIST doesn't handle correctly? If not, why not? If you cannot fix XIST, here is a work-around: import sys reload(sys) sys.setdefaultencoding("desired encoding") This is somewhat of a hack, but using a hack is ok if you are looking for a work-around for a problem that really should be fixed in the long run, anyway. I don't understand the orm example. What is "each and every setting"? Grepping the source of orm 1.0.1, the word "setting" occurs only in two places, both apparently irrelevant. If you are talking about some sort of configuration file - wouldn't it be good to create a library for the configuration file, and make only that library be aware of the encoding of the configuration file? ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 13:30 Message: Logged In: YES user_id=442849 Ah.. ok.. some examples: I'm writing Web-Applications using CGI or FastCGI. The webserver will pass user input as plain strings but within my program I want to use plain strings as little as possible to avoid all kinds of problems with diacritical characters (foremost umlauts and the EUR symbol). The websites the CGI scripts are used with usually have one charset for all pages. But some sites use iso-8859-1 and some use utf-8 and I don't want to modify my scripts to fit the charset of the site. Rather I'd like to use Apache's SetEnv directive to tell my scripts what charset their input is going to be in. Yes, I know, there are other ways to handle this :-) I keep using XIST for html output (http://www.livinglogic.de/Python). XIST tries to avoid using the default encoding, probably in the light of it being depricated. But it does not avoid it completely. There are a number of subtle cases in which it still depends on it and in all these cases I run into run-time errors which can be avoided in a snap by setting the default encoding to what my actual default encoding is. I use XIST with Zope and FastCGI, running several instances of the same interpreter on the same machine, with different default encodings each (mostly utf-8, but some iso-8859-1, which I can't just convert). My own OSS project orm (http://t4w.de/orm) carefully seperates the database's and the application's charset but uses Python's default encoding as the default for each and every setting. Those settings need to be set explicitly if the default encoding does not match the encoding actually used, cluttering the sourcecode and making maintaince more difficult. I use orm in connection with all the above cases. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:55 Message: Logged In: YES user_id=21627 No, I'm asking *why* you consider it useful. For example a) here is this and that problem, and there is absolutely no way to express this in Python today. If that feature is added, I can write it in the following way. b) here is this and that problem. Today, I have to write it in this way. With the feature added, I can write it in that way. That way is better than this way because of such and such reason. IOW, I would like to see what you are using the default encoding *for* that makes it desirable to set it through an environment variable. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:40 Message: Logged In: YES user_id=442849 Well... what more than useful is it supposed to be? Earh-shaking? Spiritually rewarding? ;-) As long as there is a default encoding I'd like to be able to set it in a simpler way than I can now. And I'd like to be able to set it once, at interpreter startup. An environment variable seems like a reasonable way to do that. If the notion of default encoding is removed from Python's library the suggested modification is going to go away as well. If you want to reject the feature request because the notion of default encoding is depricated, than so be it. From my point of view the suggested modification is something I'd like to see. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:09 Message: Logged In: YES user_id=21627 I take it that you are withdrawing your feature request, then? Closing it as rejected; if you still think this should be done, please indicate so. It would be good if you then could give some reasoning beyond "it would be useful" ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:05 Message: Logged In: YES user_id=442849 Yes, you are right. I could also replace the global unicode() function with my own version that adheres to my global encoding settings. But that doesn't seem the right thing to do. Anyway, this is not so much a technical decision but one that touches design. Anyway, it's just a small thing I tought usefull. If the defaultencoding goes away altogether, I'll have to make something up on my own anyway. Thanks for your comments! ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 11:52 Message: Logged In: YES user_id=21627 If your application just needs a global setting, it can come up with its own environment variable, right? Just set MY_DEFAULT_ENCODING, and do encoding=os.environ["MY_DEFAULT_ENCODING"] Alternatively, try using locale.getpreferredencoding(); this may allow to avoid an additional environment variable altogether. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 11:08 Message: Logged In: YES user_id=442849 Even if the notion of default encoding were dropped from Python, my application will need a global variable to have one encoding that is used most of the time. That`s what I`m using the default encoding for right now. Also, at least one of the 3d party modules I'm using does the same and it makes tons of sense. Changing the default encoding from within my application doesn't work, because sys.setdefaultencoding() gets removed (which makes sense, too, of course). So I need a custom sitecustomize.py for every project which may or may not be adhered to, depending on the PYTHONPATH variable. That`s why a PYTHON_DEFAULT_ENCODING variable would be very usefull to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:32 Message: Logged In: YES user_id=21627 Why do you want this change? In the long run, Python will drop the notion of a default encoding; code should be changed to avoid relying on the default encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 From noreply at sourceforge.net Sat Oct 23 13:54:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 13:54:20 2004 Subject: [ python-Feature Requests-1031288 ] Update unicodedata to version 4.0.1 Message-ID: Feature Requests item #1031288, was opened at 2004-09-20 17:37 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1031288&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 5 Submitted By: Oliver Horn (ohorn) Assigned to: Nobody/Anonymous (nobody) Summary: Update unicodedata to version 4.0.1 Initial Comment: The unicodedata library (shipped with Python 2.3.4 and 2.4a3) is still version 3.2.0 albeit version 4.0.1 is available from unicode.org. Is it possible to update 4.0.1, at least for Python 2.4? ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 13:54 Message: Logged In: YES user_id=21627 For Python 2.4, this is not possible, as the beta has already been published. For Python 2.5, it might be possible, but is difficult. Python relies on the unicodedata 3.2.0, as the IDNA RFCs mandate that Unicode 3.2 is used to implement IDNA. So any integration of 4.0.1 must a) still maintain access to the 3.2.0 data b) change all code that relies on 3.2.0 data to refer to these data explicitly c) not simply double the amount of data, but somehow allow for a differential representation. This is very difficult to implement, and hence hasn't been implemented for 2.4 (and might not be implemented for 2.5). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1031288&group_id=5470 From noreply at sourceforge.net Sat Oct 23 14:06:34 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 14:06:40 2004 Subject: [ python-Feature Requests-1052098 ] Seting defaultencoding through an environment variable Message-ID: Feature Requests item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Comment added) made by diedrich You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 Category: Python Library Group: None Status: Open Resolution: Rejected Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- >Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 14:06 Message: Logged In: YES user_id=442849 The reload(sys) bid is a good idea, thanks. Anyway, I can't fix XIST, because it it complicated as heck and I don't really have time to do that. Also I'm not sure, it this is really due to a fault in XIST or some side-effect I can't fathom. ORM is not touched by the encoding problem itself, it's in the data model modules. Each and every Unicode column needs a appEncoding= parameter on definition. That's not a technical problem, though, it's just plain ugly. (This might go away in the mid-run, because I'll re-implement parts of orm and I can put unicode support on the todo list. But until then...;-) ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 13:50 Message: Logged In: YES user_id=21627 I see. For the XIST case, it probably would be best to correct XIST, to always use explicit encodings. Alternatively, would it be possible to pass byte strings instead of Unicode objects to XIST in the cases which XIST doesn't handle correctly? If not, why not? If you cannot fix XIST, here is a work-around: import sys reload(sys) sys.setdefaultencoding("desired encoding") This is somewhat of a hack, but using a hack is ok if you are looking for a work-around for a problem that really should be fixed in the long run, anyway. I don't understand the orm example. What is "each and every setting"? Grepping the source of orm 1.0.1, the word "setting" occurs only in two places, both apparently irrelevant. If you are talking about some sort of configuration file - wouldn't it be good to create a library for the configuration file, and make only that library be aware of the encoding of the configuration file? ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 13:30 Message: Logged In: YES user_id=442849 Ah.. ok.. some examples: I'm writing Web-Applications using CGI or FastCGI. The webserver will pass user input as plain strings but within my program I want to use plain strings as little as possible to avoid all kinds of problems with diacritical characters (foremost umlauts and the EUR symbol). The websites the CGI scripts are used with usually have one charset for all pages. But some sites use iso-8859-1 and some use utf-8 and I don't want to modify my scripts to fit the charset of the site. Rather I'd like to use Apache's SetEnv directive to tell my scripts what charset their input is going to be in. Yes, I know, there are other ways to handle this :-) I keep using XIST for html output (http://www.livinglogic.de/Python). XIST tries to avoid using the default encoding, probably in the light of it being depricated. But it does not avoid it completely. There are a number of subtle cases in which it still depends on it and in all these cases I run into run-time errors which can be avoided in a snap by setting the default encoding to what my actual default encoding is. I use XIST with Zope and FastCGI, running several instances of the same interpreter on the same machine, with different default encodings each (mostly utf-8, but some iso-8859-1, which I can't just convert). My own OSS project orm (http://t4w.de/orm) carefully seperates the database's and the application's charset but uses Python's default encoding as the default for each and every setting. Those settings need to be set explicitly if the default encoding does not match the encoding actually used, cluttering the sourcecode and making maintaince more difficult. I use orm in connection with all the above cases. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:55 Message: Logged In: YES user_id=21627 No, I'm asking *why* you consider it useful. For example a) here is this and that problem, and there is absolutely no way to express this in Python today. If that feature is added, I can write it in the following way. b) here is this and that problem. Today, I have to write it in this way. With the feature added, I can write it in that way. That way is better than this way because of such and such reason. IOW, I would like to see what you are using the default encoding *for* that makes it desirable to set it through an environment variable. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:40 Message: Logged In: YES user_id=442849 Well... what more than useful is it supposed to be? Earh-shaking? Spiritually rewarding? ;-) As long as there is a default encoding I'd like to be able to set it in a simpler way than I can now. And I'd like to be able to set it once, at interpreter startup. An environment variable seems like a reasonable way to do that. If the notion of default encoding is removed from Python's library the suggested modification is going to go away as well. If you want to reject the feature request because the notion of default encoding is depricated, than so be it. From my point of view the suggested modification is something I'd like to see. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:09 Message: Logged In: YES user_id=21627 I take it that you are withdrawing your feature request, then? Closing it as rejected; if you still think this should be done, please indicate so. It would be good if you then could give some reasoning beyond "it would be useful" ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:05 Message: Logged In: YES user_id=442849 Yes, you are right. I could also replace the global unicode() function with my own version that adheres to my global encoding settings. But that doesn't seem the right thing to do. Anyway, this is not so much a technical decision but one that touches design. Anyway, it's just a small thing I tought usefull. If the defaultencoding goes away altogether, I'll have to make something up on my own anyway. Thanks for your comments! ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 11:52 Message: Logged In: YES user_id=21627 If your application just needs a global setting, it can come up with its own environment variable, right? Just set MY_DEFAULT_ENCODING, and do encoding=os.environ["MY_DEFAULT_ENCODING"] Alternatively, try using locale.getpreferredencoding(); this may allow to avoid an additional environment variable altogether. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 11:08 Message: Logged In: YES user_id=442849 Even if the notion of default encoding were dropped from Python, my application will need a global variable to have one encoding that is used most of the time. That`s what I`m using the default encoding for right now. Also, at least one of the 3d party modules I'm using does the same and it makes tons of sense. Changing the default encoding from within my application doesn't work, because sys.setdefaultencoding() gets removed (which makes sense, too, of course). So I need a custom sitecustomize.py for every project which may or may not be adhered to, depending on the PYTHONPATH variable. That`s why a PYTHON_DEFAULT_ENCODING variable would be very usefull to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:32 Message: Logged In: YES user_id=21627 Why do you want this change? In the long run, Python will drop the notion of a default encoding; code should be changed to avoid relying on the default encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 From noreply at sourceforge.net Sat Oct 23 14:25:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 14:25:18 2004 Subject: [ python-Feature Requests-1052098 ] Seting defaultencoding through an environment variable Message-ID: Feature Requests item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 Category: Python Library Group: None Status: Open Resolution: Rejected Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 14:25 Message: Logged In: YES user_id=21627 If all you want is to specify the appEncoding through an environment variable: this should be really easy. Instead of class Unicode(varchar): def __init__(self, columnName=None, appEncoding="iso-8859-1"): datatype.__init__(self, columnName) self._appEncoding = appEncoding do this instead class Unicode(varchar): def __init__(self, columnName=None, appEncoding=os.environ.get("ORM_ENCODING", "iso-8859-1")): datatype.__init__(self, columnName) self._appEncoding = appEncoding Then set ORM_ENCODING as you'ld like. No other changes to your source are required. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 14:06 Message: Logged In: YES user_id=442849 The reload(sys) bid is a good idea, thanks. Anyway, I can't fix XIST, because it it complicated as heck and I don't really have time to do that. Also I'm not sure, it this is really due to a fault in XIST or some side-effect I can't fathom. ORM is not touched by the encoding problem itself, it's in the data model modules. Each and every Unicode column needs a appEncoding= parameter on definition. That's not a technical problem, though, it's just plain ugly. (This might go away in the mid-run, because I'll re-implement parts of orm and I can put unicode support on the todo list. But until then...;-) ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 13:50 Message: Logged In: YES user_id=21627 I see. For the XIST case, it probably would be best to correct XIST, to always use explicit encodings. Alternatively, would it be possible to pass byte strings instead of Unicode objects to XIST in the cases which XIST doesn't handle correctly? If not, why not? If you cannot fix XIST, here is a work-around: import sys reload(sys) sys.setdefaultencoding("desired encoding") This is somewhat of a hack, but using a hack is ok if you are looking for a work-around for a problem that really should be fixed in the long run, anyway. I don't understand the orm example. What is "each and every setting"? Grepping the source of orm 1.0.1, the word "setting" occurs only in two places, both apparently irrelevant. If you are talking about some sort of configuration file - wouldn't it be good to create a library for the configuration file, and make only that library be aware of the encoding of the configuration file? ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 13:30 Message: Logged In: YES user_id=442849 Ah.. ok.. some examples: I'm writing Web-Applications using CGI or FastCGI. The webserver will pass user input as plain strings but within my program I want to use plain strings as little as possible to avoid all kinds of problems with diacritical characters (foremost umlauts and the EUR symbol). The websites the CGI scripts are used with usually have one charset for all pages. But some sites use iso-8859-1 and some use utf-8 and I don't want to modify my scripts to fit the charset of the site. Rather I'd like to use Apache's SetEnv directive to tell my scripts what charset their input is going to be in. Yes, I know, there are other ways to handle this :-) I keep using XIST for html output (http://www.livinglogic.de/Python). XIST tries to avoid using the default encoding, probably in the light of it being depricated. But it does not avoid it completely. There are a number of subtle cases in which it still depends on it and in all these cases I run into run-time errors which can be avoided in a snap by setting the default encoding to what my actual default encoding is. I use XIST with Zope and FastCGI, running several instances of the same interpreter on the same machine, with different default encodings each (mostly utf-8, but some iso-8859-1, which I can't just convert). My own OSS project orm (http://t4w.de/orm) carefully seperates the database's and the application's charset but uses Python's default encoding as the default for each and every setting. Those settings need to be set explicitly if the default encoding does not match the encoding actually used, cluttering the sourcecode and making maintaince more difficult. I use orm in connection with all the above cases. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:55 Message: Logged In: YES user_id=21627 No, I'm asking *why* you consider it useful. For example a) here is this and that problem, and there is absolutely no way to express this in Python today. If that feature is added, I can write it in the following way. b) here is this and that problem. Today, I have to write it in this way. With the feature added, I can write it in that way. That way is better than this way because of such and such reason. IOW, I would like to see what you are using the default encoding *for* that makes it desirable to set it through an environment variable. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:40 Message: Logged In: YES user_id=442849 Well... what more than useful is it supposed to be? Earh-shaking? Spiritually rewarding? ;-) As long as there is a default encoding I'd like to be able to set it in a simpler way than I can now. And I'd like to be able to set it once, at interpreter startup. An environment variable seems like a reasonable way to do that. If the notion of default encoding is removed from Python's library the suggested modification is going to go away as well. If you want to reject the feature request because the notion of default encoding is depricated, than so be it. From my point of view the suggested modification is something I'd like to see. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:09 Message: Logged In: YES user_id=21627 I take it that you are withdrawing your feature request, then? Closing it as rejected; if you still think this should be done, please indicate so. It would be good if you then could give some reasoning beyond "it would be useful" ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:05 Message: Logged In: YES user_id=442849 Yes, you are right. I could also replace the global unicode() function with my own version that adheres to my global encoding settings. But that doesn't seem the right thing to do. Anyway, this is not so much a technical decision but one that touches design. Anyway, it's just a small thing I tought usefull. If the defaultencoding goes away altogether, I'll have to make something up on my own anyway. Thanks for your comments! ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 11:52 Message: Logged In: YES user_id=21627 If your application just needs a global setting, it can come up with its own environment variable, right? Just set MY_DEFAULT_ENCODING, and do encoding=os.environ["MY_DEFAULT_ENCODING"] Alternatively, try using locale.getpreferredencoding(); this may allow to avoid an additional environment variable altogether. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 11:08 Message: Logged In: YES user_id=442849 Even if the notion of default encoding were dropped from Python, my application will need a global variable to have one encoding that is used most of the time. That`s what I`m using the default encoding for right now. Also, at least one of the 3d party modules I'm using does the same and it makes tons of sense. Changing the default encoding from within my application doesn't work, because sys.setdefaultencoding() gets removed (which makes sense, too, of course). So I need a custom sitecustomize.py for every project which may or may not be adhered to, depending on the PYTHONPATH variable. That`s why a PYTHON_DEFAULT_ENCODING variable would be very usefull to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:32 Message: Logged In: YES user_id=21627 Why do you want this change? In the long run, Python will drop the notion of a default encoding; code should be changed to avoid relying on the default encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 From noreply at sourceforge.net Sat Oct 23 14:41:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 14:41:06 2004 Subject: [ python-Feature Requests-1052098 ] Seting defaultencoding through an environment variable Message-ID: Feature Requests item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Comment added) made by diedrich You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 Category: Python Library Group: None Status: Open Resolution: Rejected Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- >Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 14:41 Message: Logged In: YES user_id=442849 Yes, that's a way to go. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 14:25 Message: Logged In: YES user_id=21627 If all you want is to specify the appEncoding through an environment variable: this should be really easy. Instead of class Unicode(varchar): def __init__(self, columnName=None, appEncoding="iso-8859-1"): datatype.__init__(self, columnName) self._appEncoding = appEncoding do this instead class Unicode(varchar): def __init__(self, columnName=None, appEncoding=os.environ.get("ORM_ENCODING", "iso-8859-1")): datatype.__init__(self, columnName) self._appEncoding = appEncoding Then set ORM_ENCODING as you'ld like. No other changes to your source are required. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 14:06 Message: Logged In: YES user_id=442849 The reload(sys) bid is a good idea, thanks. Anyway, I can't fix XIST, because it it complicated as heck and I don't really have time to do that. Also I'm not sure, it this is really due to a fault in XIST or some side-effect I can't fathom. ORM is not touched by the encoding problem itself, it's in the data model modules. Each and every Unicode column needs a appEncoding= parameter on definition. That's not a technical problem, though, it's just plain ugly. (This might go away in the mid-run, because I'll re-implement parts of orm and I can put unicode support on the todo list. But until then...;-) ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 13:50 Message: Logged In: YES user_id=21627 I see. For the XIST case, it probably would be best to correct XIST, to always use explicit encodings. Alternatively, would it be possible to pass byte strings instead of Unicode objects to XIST in the cases which XIST doesn't handle correctly? If not, why not? If you cannot fix XIST, here is a work-around: import sys reload(sys) sys.setdefaultencoding("desired encoding") This is somewhat of a hack, but using a hack is ok if you are looking for a work-around for a problem that really should be fixed in the long run, anyway. I don't understand the orm example. What is "each and every setting"? Grepping the source of orm 1.0.1, the word "setting" occurs only in two places, both apparently irrelevant. If you are talking about some sort of configuration file - wouldn't it be good to create a library for the configuration file, and make only that library be aware of the encoding of the configuration file? ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 13:30 Message: Logged In: YES user_id=442849 Ah.. ok.. some examples: I'm writing Web-Applications using CGI or FastCGI. The webserver will pass user input as plain strings but within my program I want to use plain strings as little as possible to avoid all kinds of problems with diacritical characters (foremost umlauts and the EUR symbol). The websites the CGI scripts are used with usually have one charset for all pages. But some sites use iso-8859-1 and some use utf-8 and I don't want to modify my scripts to fit the charset of the site. Rather I'd like to use Apache's SetEnv directive to tell my scripts what charset their input is going to be in. Yes, I know, there are other ways to handle this :-) I keep using XIST for html output (http://www.livinglogic.de/Python). XIST tries to avoid using the default encoding, probably in the light of it being depricated. But it does not avoid it completely. There are a number of subtle cases in which it still depends on it and in all these cases I run into run-time errors which can be avoided in a snap by setting the default encoding to what my actual default encoding is. I use XIST with Zope and FastCGI, running several instances of the same interpreter on the same machine, with different default encodings each (mostly utf-8, but some iso-8859-1, which I can't just convert). My own OSS project orm (http://t4w.de/orm) carefully seperates the database's and the application's charset but uses Python's default encoding as the default for each and every setting. Those settings need to be set explicitly if the default encoding does not match the encoding actually used, cluttering the sourcecode and making maintaince more difficult. I use orm in connection with all the above cases. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:55 Message: Logged In: YES user_id=21627 No, I'm asking *why* you consider it useful. For example a) here is this and that problem, and there is absolutely no way to express this in Python today. If that feature is added, I can write it in the following way. b) here is this and that problem. Today, I have to write it in this way. With the feature added, I can write it in that way. That way is better than this way because of such and such reason. IOW, I would like to see what you are using the default encoding *for* that makes it desirable to set it through an environment variable. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:40 Message: Logged In: YES user_id=442849 Well... what more than useful is it supposed to be? Earh-shaking? Spiritually rewarding? ;-) As long as there is a default encoding I'd like to be able to set it in a simpler way than I can now. And I'd like to be able to set it once, at interpreter startup. An environment variable seems like a reasonable way to do that. If the notion of default encoding is removed from Python's library the suggested modification is going to go away as well. If you want to reject the feature request because the notion of default encoding is depricated, than so be it. From my point of view the suggested modification is something I'd like to see. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:09 Message: Logged In: YES user_id=21627 I take it that you are withdrawing your feature request, then? Closing it as rejected; if you still think this should be done, please indicate so. It would be good if you then could give some reasoning beyond "it would be useful" ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:05 Message: Logged In: YES user_id=442849 Yes, you are right. I could also replace the global unicode() function with my own version that adheres to my global encoding settings. But that doesn't seem the right thing to do. Anyway, this is not so much a technical decision but one that touches design. Anyway, it's just a small thing I tought usefull. If the defaultencoding goes away altogether, I'll have to make something up on my own anyway. Thanks for your comments! ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 11:52 Message: Logged In: YES user_id=21627 If your application just needs a global setting, it can come up with its own environment variable, right? Just set MY_DEFAULT_ENCODING, and do encoding=os.environ["MY_DEFAULT_ENCODING"] Alternatively, try using locale.getpreferredencoding(); this may allow to avoid an additional environment variable altogether. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 11:08 Message: Logged In: YES user_id=442849 Even if the notion of default encoding were dropped from Python, my application will need a global variable to have one encoding that is used most of the time. That`s what I`m using the default encoding for right now. Also, at least one of the 3d party modules I'm using does the same and it makes tons of sense. Changing the default encoding from within my application doesn't work, because sys.setdefaultencoding() gets removed (which makes sense, too, of course). So I need a custom sitecustomize.py for every project which may or may not be adhered to, depending on the PYTHONPATH variable. That`s why a PYTHON_DEFAULT_ENCODING variable would be very usefull to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:32 Message: Logged In: YES user_id=21627 Why do you want this change? In the long run, Python will drop the notion of a default encoding; code should be changed to avoid relying on the default encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 From noreply at sourceforge.net Sat Oct 23 15:52:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 15:53:05 2004 Subject: [ python-Bugs-896330 ] pyconfig.h is not placed in --includedir Message-ID: Bugs item #896330, was opened at 2004-02-13 09:24 Message generated for change (Comment added) made by morth You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=896330&group_id=5470 Category: Installation Group: Python 2.2.3 Status: Open Resolution: None Priority: 5 Submitted By: G?ran Uddeborg (goeran) Assigned to: Martin v. L?wis (loewis) Summary: pyconfig.h is not placed in --includedir Initial Comment: When configuring Python with an explicit --includedir most of the header files do turn up in this directory. But the pyconfig.h is still placed in $(prefix)/include. I assume it is a bug, and this file too should be placed according to --includedir. ---------------------------------------------------------------------- Comment By: Pelle Johansson (morth) Date: 2004-10-23 15:52 Message: Logged In: YES user_id=180081 Actually, pyconfig.h is placed in $(exec-prefix)/include. This is fine as the file is architecture dependant. However the other header files use a local #include "pyconfig.h" which will not match if it's not the same dir. If you like me want a common directory for all archindependant files and then separate for each arch it causes additional headache, since there's no easy way for me to symlink from the arch include/ dir to the archindependant, as they have a common subdir (python2.3/ in my case). Currently it seems I'll have to symlink each header file individually from the archindependent dir to the arch ones. Ideally pyconfig.h would be put directly in $(exec-prefix)/include/ and be refered to as . I could then just symlink the subdir. Whether to honor includedir or not I don't know, I'll leave that problem to the project. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=896330&group_id=5470 From noreply at sourceforge.net Sat Oct 23 19:25:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 19:25:28 2004 Subject: [ python-Bugs-1052827 ] filelist.findall should use os.lstat Message-ID: Bugs item #1052827, was opened at 2004-10-23 10:25 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052827&group_id=5470 Category: Distutils Group: None Status: Open Resolution: None Priority: 5 Submitted By: Geoffrey T. Dairiki (dairiki) Assigned to: Nobody/Anonymous (nobody) Summary: filelist.findall should use os.lstat Initial Comment: On unix platforms, if one has a dangling symlink anywhere in ones source directory tree (and a MANIFEST.in file), 'setup.py sdist' will fail. ('OSError: [Errno 2] No such file or directory' on the dangling symlink.) A fix for this is for filefind.findall to use os.lstat() instead of os.stat() (when lstat is available.) (Another possible fix would be to catch the OSError(ENOENT) exception and either ignore it or render it as a warning.) Appendix A Details on how to reproduce the bug: In your top-level source directory (the one with a setup.py): 1. Create a dangling symlink. ('ln -s foo bar', assuming the file 'foo' doesn't exist.) 2. Do a 'touch MANIFEST.in' 3. Run 'python setup.py sdist' The command should fail with: 'error: bar: No such file or directory' ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052827&group_id=5470 From noreply at sourceforge.net Sat Oct 23 19:56:57 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 19:57:00 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-23 13:56 Message: Logged In: YES user_id=6380 I disagree. The docs are wrong because a failing listdir wasn't anticipated at first when this function was written. A failing listdir is a common failure case; I don't see how the os.path calls in os.walk could ever fail (since they all catch os.error). ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-22 23:01 Message: Logged In: YES user_id=469548 You know what: I think we should remove the os.listdir error handling: * The docs only specify "errors resulting from *failed removals*" as being handled. If they didn't we should worry about OSErrors from the os.path.* calls in os.walk() too. * The os.listdir error handling hasn't been in a final release, so no backwards compatibility problems yet. * We can't get the path right for os.listdir. * The problem also existed before the rewrite using os.walk(), so reverting to that is not a solution. Patch (the last?) is attached. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-21 00:42 Message: Logged In: YES user_id=6380 Yes. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-20 16:30 Message: Logged In: YES user_id=469548 With 'be careful, it may not exist.', do you mean rmtree can't rely on exc.filename being set? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 11:18 Message: Logged In: YES user_id=6380 This looks fine, and can be checked in. There's one minor remaining issue, but I don't see how to fix it without changing the signature for the onerror function that os.walk() takes, and that's a no-no: when the problem is a directory permission preventing os.listdir() to work, the path passed to the rmtree caller's onerror function is always the toplevel path, rather than the directory that couldn't be listed. The real path is included in the error message. (Hmm, perhaps you could get it out of there? I think it's an attribute of the exception object; but be careful, it may not exist.) It's unfortunate that the error argument to the onerror function is a sys.exc_info() triple rather than an exception object (calling sys.exc_info() is expensive), but that's an API choice from the past that we can't change until Python 3.0. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 04:57 Message: Logged In: YES user_id=469548 A new patch is attached. I factored out the handle_call_error method so exception handling is more local. Also, errors in os.walk were already being handled using the _raise_err/except OSError combination, but we now do it using the onerror argument to os.walk. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:42 Message: Logged In: YES user_id=6380 (Actually, os.walk has an onerror argument, it just uses a different convention and default, so we could thread this through easily.) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 18:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 02:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Sat Oct 23 20:25:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 23 20:25:21 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-23 14:25 Message: Logged In: YES user_id=6380 BTW, if you want the correct path in the onerror calls for os.listdir, your best bet is not to use os.walk but to write something similar that suits our purpose. After you remove the features we don't use, it's really only a few lines of code. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-23 13:56 Message: Logged In: YES user_id=6380 I disagree. The docs are wrong because a failing listdir wasn't anticipated at first when this function was written. A failing listdir is a common failure case; I don't see how the os.path calls in os.walk could ever fail (since they all catch os.error). ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-22 23:01 Message: Logged In: YES user_id=469548 You know what: I think we should remove the os.listdir error handling: * The docs only specify "errors resulting from *failed removals*" as being handled. If they didn't we should worry about OSErrors from the os.path.* calls in os.walk() too. * The os.listdir error handling hasn't been in a final release, so no backwards compatibility problems yet. * We can't get the path right for os.listdir. * The problem also existed before the rewrite using os.walk(), so reverting to that is not a solution. Patch (the last?) is attached. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-21 00:42 Message: Logged In: YES user_id=6380 Yes. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-20 16:30 Message: Logged In: YES user_id=469548 With 'be careful, it may not exist.', do you mean rmtree can't rely on exc.filename being set? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 11:18 Message: Logged In: YES user_id=6380 This looks fine, and can be checked in. There's one minor remaining issue, but I don't see how to fix it without changing the signature for the onerror function that os.walk() takes, and that's a no-no: when the problem is a directory permission preventing os.listdir() to work, the path passed to the rmtree caller's onerror function is always the toplevel path, rather than the directory that couldn't be listed. The real path is included in the error message. (Hmm, perhaps you could get it out of there? I think it's an attribute of the exception object; but be careful, it may not exist.) It's unfortunate that the error argument to the onerror function is a sys.exc_info() triple rather than an exception object (calling sys.exc_info() is expensive), but that's an API choice from the past that we can't change until Python 3.0. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 04:57 Message: Logged In: YES user_id=469548 A new patch is attached. I factored out the handle_call_error method so exception handling is more local. Also, errors in os.walk were already being handled using the _raise_err/except OSError combination, but we now do it using the onerror argument to os.walk. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:42 Message: Logged In: YES user_id=6380 (Actually, os.walk has an onerror argument, it just uses a different convention and default, so we could thread this through easily.) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 18:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 02:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Sun Oct 24 02:11:13 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 24 02:11:21 2004 Subject: [ python-Bugs-1048870 ] call arg of lambda not updating Message-ID: Bugs item #1048870, was opened at 2004-10-17 16:55 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 Category: Parser/Compiler Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 2 Submitted By: Kevin Quick (kquick) Assigned to: Michael Hudson (mwh) Summary: call arg of lambda not updating Initial Comment: The attachment contains a script intended to do performance testing on various types of dictionary keys. Lambda functions are used to "generate" a sequence of keys with bound arguments to arrays that are modified during execution. Script fails with pop() on empty list error, but examination shows that the lambda function from the *previous* call is being used for the current call; if proper lambda was used, list would not be empty and operation should succeed. Script is large and has been "grown", so it's not very elegant, but either I'm missing somewhere where I'm making a stupid mistake or else the lambda argument to the function call isn't being set properly. Don't be put off by the size of the script; the problem is easily demonstrated and fairly clearly understood in just a few minutes of looking at the generated stack trace relative to the code. As it exists, script will fail immediately due to the bug. If it worked as expected, at least 3-4 lines of output should be generated before encountering new code that hasn't yet been fully debugged. New code could be removed, but it was left in because some of the complexity of core code is due to this latter code, and it doesn't obstruct the primary bug, so it can be ignored unless of interest. python Python 2.3.3 (#1, May 27 2004, 20:44:16) [GCC 3.3.2 20031218 (Gentoo Linux 3.3.2-r5, propolice-3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> Thanks for looking at this! ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-23 19:11 Message: Logged In: YES user_id=80475 Fixed. See: Python/compile.c 2.331 ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-18 18:24 Message: Logged In: YES user_id=6133 Confirmed that the patch does indeed address the problem. Also verified that a lambda with the same code appearing at the same line in a different file is handled separately and distinctly. Thanks! ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 17:29 Message: Logged In: YES user_id=80475 Michael, that is a reasonable solution. Go ahead and apply it along with a test case verifying the object id's are distinct after the change but not before. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-18 16:27 Message: Logged In: YES user_id=6656 I *think* what's happening here is that co_firstlineno is not being considered when comparing code objects. Can you try the attached patch? There may well be other fields not being considered. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-18 02:37 Message: Logged In: YES user_id=6133 Thanks for confirming this issue. My position is that while minor, this is still a bug and not a feature. It's good to detect identical lambda bodies and optimize for that case, but since argument binding occurs at declaration time and not execution time for lambdas, the bound arguments should properly be considered part of the body; the two lambda functions in the attached script should be disjoint. >From the practical perspective, the python backtrace was misleading in my debug efforts. Internal optimizations should take second place to accuracy of displayed information. func_code.co_code (and possibly others) could still be the same, but func_code.co_firstlineno (& func_code.co_filename) should be uniquely set. Hmmm. In thinking more about this, it's not that argument binding occurs immediately, but moreso that defaults are supplied for arguments. If the backtrace/debugger points me to a lambda function with entirely different default args than the ones that are applicable to the problem, then that would be wrong, IMHO. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 01:42 Message: Logged In: YES user_id=80475 Under the hood, the compiler is recognizing that the function bodies are identical and it produces just one code object instead of two. Hence, the co_firstline number with always reflect the first time the code is defined. A simple demo shows the id being the same when the bodies are identical: src = """ f1 = lambda x=1: x f2 = lambda x=2: x """ from dis import dis c = compile(src, 'example', 'exec') dis(c) For the OP's code, the effect can be shown by differentiating either lambda body by adding zero to the pop index: nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()+0] I conclude that there is less to the bug than meets the eye and that it may even be considered a feature. Referring to Jeremy for final disposition. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-18 00:41 Message: Logged In: YES user_id=80475 Confirmed. Also, the behavior persists into Py2.4b1. The error disappears when the lambda is replaced by an equivalent one line def statement: def nextkey2(N=keynums2,K=key_src): return K[N.pop()] The opcodes are the same for both; however, the co_firstlineno attribute is incorrect for the lambda version. The pure python compiler module does not make the same error. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 23:48 Message: Logged In: YES user_id=6133 Reopened: Problem not addressed. Please *read* my previous comment. Yes, the script has a bug in it. That's not what's being reported here. The bug is in the backtrace produced by the Python interpreter. See below. Line number reported for lambda function raising exception is line 14 below. Actual line number for lambda generating exception is line 18. $ python lambda_bug.py Traceback (most recent call last): File "lambda_bug.py", line 23, in ? seqtest() File "lambda_bug.py", line 19, in seqtest seq_func1(nextkey2, num_elem+2) File "lambda_bug.py", line 5, in seq_func1 key_genfunc() File "lambda_bug.py", line 14, in nextkey1 = lambda N=keynums1,K=key_src: K[N.pop()] IndexError: pop from empty list ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 23:34 Message: Logged In: YES user_id=80475 Closing as invalid -- this is a bug in the OP's script, not in Python itself. The error is obious when the code is simplified further by in-lining the function. Also, it helps to explicitly set the initial values of variables between the two sections: #!/usr/bin/env python num_elem = 100 key_src = range(num_elem+1) keynums1 = key_src[:] assert key_src[-1] == 100 for _ in xrange(num_elem): keynums1[key_src.pop()] keynums2 = [0] for _ in xrange(102): key_src[keynums2.pop()] I think your coding error was in assuming that keynum2 held a different value before the second half was run. If so, that could have been found by using pdb or by inserting print statements to reveal what is going on. ---------------------------------------------------------------------- Comment By: Kevin Quick (kquick) Date: 2004-10-17 23:02 Message: Logged In: YES user_id=6133 OK, example script minimized. Lambda arguments are not the problem; thanks for the pointer, but I'm not having trouble with those. I have determined that proper lambda is being used, but not indicated as shown by attached script, which forces an exception to occur in the lambda function but backtrace shows wrong lambda function. Reducing 2nd argument of 2nd call to seq_func1 to 'num_args' instead of 'num_args+2' will not encounter the error, but the nature of the implementation requires that it is using the correct lambda, so it's the backtrace attribution that is apparently wrong. Mea culpa: my original script version did request wrong 'num_args' causing actual bug, but incorrect backtrace led me down an alternate path... I suspected that I had some incorrect code, but I couldn't resolve that against the error report. Still can't. :) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-17 18:04 Message: Logged In: YES user_id=80475 Please resubmit after boiling this down to the simplest possible thing that doesn't work according to your expectations. Submitting a whole application and playing the "find the bug" game is not good use of developer time. Often, when a submitter starts taking away the unnecessary pieces and isolates the issue, they find that there was a bug in their own code or a bug in their own understanding of how the language works. When it comes to lambdas, the most frequent misunderstanding is knowing that default arguments bind only once as the time of the lambda declaration and that free variables bind whenever the resulting function is invoked. Here's an example of code that looks similar but behaves differently: def once(x): return x def twice(x): return 2*x def thrice(x): return 3*x funcs = [once, twice, thrice] flim = [lambda x:funcs[0](x), lambda x:funcs[1](x), lambda x:funcs[2](x)] flam = [lambda x:f(x) for f in funcs] print flim[0](1), flim[1](1), flim[2](1) print flam[0](1), flam[1](1), flam[2](1) The difference in output is because nested scopes bind names, while argument binding binds values. Hope this helps. If not, please resubmit with the smallest case that demonstrates the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048870&group_id=5470 From noreply at sourceforge.net Sun Oct 24 02:35:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 24 02:35:51 2004 Subject: [ python-Bugs-1047540 ] Turtle.py hangs Idle Message-ID: Bugs item #1047540, was opened at 2004-10-14 22:51 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047540&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Taro Ogawa (tso) Assigned to: Nobody/Anonymous (nobody) Summary: Turtle.py hangs Idle Initial Comment: 1 line fix: After line 336's: root.destroy() insert root.quit() [Since IDLE is Tkinter based, the implict quit can't happen immediately. Worse, the shell has difficulty resetting and using a ^C can leave a zombie process] -T. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-23 19:35 Message: Logged In: YES user_id=80475 What version of Python are you using? I cannot reproduce this on Py2.3 or on Py2.4b1. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047540&group_id=5470 From noreply at sourceforge.net Sun Oct 24 02:36:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 24 02:36:23 2004 Subject: [ python-Bugs-1052503 ] pdb runcall should accept keyword arguments Message-ID: Bugs item #1052503, was opened at 2004-10-22 16:50 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052503&group_id=5470 Category: Python Library Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Leonardo Rochael Almeida (rochael) Assigned to: Nobody/Anonymous (nobody) Summary: pdb runcall should accept keyword arguments Initial Comment: pdb.runcall(), pdb.Pdb.runcall() and bdb.Bdb.runcall() could all accept keyword arguments to pass them to the debugged callable. The implementation seems trivial, just add "**kw" on the signatures and invocations all the way thru the actual call to the callable. I verified this on Python 2.3.4. No idea if the bug is also on 2.4 betas ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-23 19:36 Message: Logged In: YES user_id=80475 Fixed. See: Lib/pdb.py 1.72 Lib/bdb.py 1.46 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052503&group_id=5470 From noreply at sourceforge.net Sun Oct 24 02:40:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 24 02:40:34 2004 Subject: [ python-Bugs-1051638 ] incorrect traceback filename from pyc Message-ID: Bugs item #1051638, was opened at 2004-10-21 12:24 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1051638&group_id=5470 Category: Parser/Compiler Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Kevin Quick (kquick) Assigned to: Nobody/Anonymous (nobody) Summary: incorrect traceback filename from pyc Initial Comment: The .pyc file apparently caches the entire path of the source file during compilation, causing it to report improper path information if the resulting code is moved and an exception occurs. $ python Python 2.3.3 (#1, Oct 18 2004, 16:10:24) [GCC 3.3.4 20040623 (Gentoo Linux 3.3.4-r1, ssp-3.3.2-2, pie-8.7.6)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-23 19:40 Message: Logged In: YES user_id=80475 IMO, this is not a bug. A pyc file has no way of knowing the future locations of the source that originally created it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1051638&group_id=5470 From noreply at sourceforge.net Sun Oct 24 02:46:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 24 02:46:37 2004 Subject: [ python-Bugs-1051840 ] HTMLParser doesn't treat endtags in blah blah """ m = MyHandler() m.feed(s) This will raise an exception. I fixed the bug by changing the parse_endtag function on line 318 of HTMLParser to the following: def parse_endtag(self, i): rawdata = self.rawdata assert rawdata[i:i+2] == " if not match: return -1 j = match.end() match = endtagfind.match(rawdata, i) # if not match: self.error("bad end tag: %s" % `rawdata[i:j]`) tag = match.group(1) #START BUGFIX if self.interesting == interesting_cdata: #we're in of of the CDATA_CONTENT_ELEMENTS if tag == self.lasttag and tag in self.CDATA_CONTENT_ELEMENTS: #its the end of the CDATA_CONTENT_ELEMENTS tag we are in. self.handle_endtag(tag.lower()) self.clear_cdata_mode()#backto normal mode else: #we're inside the CDATA_CONTENT_ELEMENTS tag still. throw the tag to handle_data instead. self.handle_data(match.group()) else: #we're not in a CDATA_CONTENT_ELEMENTS tag. standard ending: self.handle_endtag(tag.lower()) return j ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-23 19:46 Message: Logged In: YES user_id=80475 Fred, what do you think? ---------------------------------------------------------------------- Comment By: Luke Bradley (neptune235) Date: 2004-10-22 18:52 Message: Logged In: YES user_id=178561 Although a fix may be worthwhile, as this happens a lot in practice, HTMLParser is following the letter of the law in throwing exceptions on pages that aren't strictly valid. http://www.w3.org/TR/html4/appendix/notes.html#notes- specifying-data Well you're right, I'll be damned! Hmm. I want to use HTMLParser to access other people's pages on the net, and I can't fix their bad HTML. The problem here is I'm not sure how to handle this at the level of my Handler, without inadvertantly changing thier javascript by doing something like: handle_data("") in the handle_entag event. Which lowercases the tag. Is there a way to access the literal string of the endtag in my handler I wonder? If not, it might be useful to add some functionality to HTMLParser that allows us to handle invalid HTML at the level of our handler without sacrificing HTMLParsers commitment to standards compliance. ---------------------------------------------------------------------- Comment By: Richard Brodie (leogah) Date: 2004-10-22 13:02 Message: Logged In: YES user_id=356893 Although a fix may be worthwhile, as this happens a lot in practice, HTMLParser is following the letter of the law in throwing exceptions on pages that aren't strictly valid. http://www.w3.org/TR/html4/appendix/notes.html#notes- specifying-data ---------------------------------------------------------------------- Comment By: Luke Bradley (neptune235) Date: 2004-10-21 18:04 Message: Logged In: YES user_id=178561 oops, I didn't know this would remove indentation. Let me attach a file. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1051840&group_id=5470 From noreply at sourceforge.net Sun Oct 24 02:48:02 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 24 02:48:05 2004 Subject: [ python-Bugs-1050268 ] rfc822.parseaddr is broken, breaks sendmail call in smtplib Message-ID: Bugs item #1050268, was opened at 2004-10-19 14:51 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050268&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Scott Dossey (sdossey) >Assigned to: Barry A. Warsaw (bwarsaw) Summary: rfc822.parseaddr is broken, breaks sendmail call in smtplib Initial Comment: THe following email address is legal according to RFC: <"\quoted string\ somebody"@somedomain.com"> I've got a python mail handling back end server that handles mail coming in from Postfix. Postfix properly accepts mail of this type, but when it goes to relay this through my Python server it fails. The problem is inside smtplib.py inside "quoteaddr". Here's a source code snippet: def quoteaddr(addr) """Quote a subset of the email addresses defined by RFC 821. Should be able to handle anything rfc822.parseaddr can handle. """ m = (None, None) try: m=rfc822.parseaddr(addr)[1] except AttributeError: pass if m == (None, None): # Indicates parse failure or AttributeError #something weird here.. punt -ddm return "<%s>" % addr Basically quoteaddr ends up wrapping whatever parseaddr returns to it in brackets and sending that out on the wire for the RCPT TO command. however, if I call rfc822.parseaddr it does bizarre things to email addresses. For instance the following calls all yield the same thing (some not surprisingly): rfc822.parseaddr('""test" test"@test.com') rfc822.parseaddr('"\test\ test"@test.com') rfc822.parseaddr('"\"test\" test"@test.com') rfc822.parseaddr('"\\test\\ test"@test/com') the above all yield: ('', '""test" test"@test.com') rfc822.parseaddr('"\\"test\\" test"@test/com') yields the VERY surprising result: ('', '"\"test\\" test"@test.com') I submitted this as a new bug report even though there are two similar bugs regarding parseAddr because it is a slightly separate issue. -Scott Dossey ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050268&group_id=5470 From noreply at sourceforge.net Sun Oct 24 02:52:02 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 24 02:52:05 2004 Subject: [ python-Bugs-1017329 ] email.Message does not allow iteration Message-ID: Bugs item #1017329, was opened at 2004-08-26 21:11 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017329&group_id=5470 Category: Extension Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Paul McGuire (ptmcg) >Assigned to: Barry A. Warsaw (bwarsaw) Summary: email.Message does not allow iteration Initial Comment: >From comp.lang.python: "Roman Suzi" wrote in message news:mailman.2348.1093434827.5135.python- list@python.org... > > In Python2.3.4: > > >>> em = email.message_from_file(open ('MAILMESSAGE')) > >>> for i in em: > ... print i > ... > Traceback (most recent call last): > File "", line 1, in ? > File "/usr/lib/python2.3/email/Message.py", > line > 304, in __getitem__ > File "/usr/lib/python2.3/email/Message.py", > line > 370, in get > AttributeError: 'int' object has no attribute 'lower' In this example, em is an email.Message object. The Message acts as a pseudo-dictionary, containing a list of e-mail fields, organized by name and contents. The problem occurs because __getitem__ assumes that the provided index is a name into a dictionary. For programmer convenience, keys in the Message are case- insensitive, so __getitem__ starts by calling lower() on the provided name. Apparently, the attempt to iterate over Message successively calls __getitem__ with integers from 0 to len(em). Integers don't like having lower() called, though. To fix this problem: - have __getitem__ use the input arg as a name if it is of type string; otherwise, just use the provided arg as an index to the underlying self.headers list (arg could be either an integer or a slice, so don't just test for integer!) - make a similar change to __delitem__ - it too assumes that the incoming arg is a string, naming a header field to be deleted; must test first to see if arg is an integer or slice - add __iter__ method, which returns iter(self.headers) -- Paul ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017329&group_id=5470 From noreply at sourceforge.net Sun Oct 24 02:52:34 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 24 02:52:37 2004 Subject: [ python-Bugs-1030118 ] email.Utils not mentioned Message-ID: Bugs item #1030118, was opened at 2004-09-17 16:40 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030118&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jeff Blaine (jblaine) >Assigned to: Barry A. Warsaw (bwarsaw) Summary: email.Utils not mentioned Initial Comment: I read the following and had to track down why 'email.make_msgid()' was failing. Turns out I need to import email.Utils of course, but the name of the module to import is not even mentioned in the docs. It states they come 'with the email package' only. ... 12.2.9 Miscellaneous utilities There are several useful utilities provided with the email package. quote(str) Return a new string with backslashes in str replaced by two backslashes, and double quotes replaced by backslash-double quote. .... ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1030118&group_id=5470 From noreply at sourceforge.net Sun Oct 24 02:56:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 24 02:56:22 2004 Subject: [ python-Bugs-1052242 ] Attempt to run all atexit handlers Message-ID: Bugs item #1052242, was opened at 2004-10-22 09:31 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 >Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Greg Chapman (glchapman) Assigned to: Nobody/Anonymous (nobody) Summary: Attempt to run all atexit handlers Initial Comment: Currently, if an atexit handler raises an exception during _run_exitfuncs, none of the remaining handlers in the _exithandlers list gets run. I would prefer it if _run_exitfuncs did something like: while _exithandlers: func, targs, kargs = _exithandlers.pop() try: func(*targs, **kargs) except SystemExit: raise except: import sys, traceback print >> sys.stderr, "Error in sys.exitfunc:" traceback.print_exc() The SystemExit except clause is there because it looks like (given its special treatment in call_sys_exitfunc) SystemExit is envisioned as a way of intentionally breaking out of the exithandlers loop. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-23 19:56 Message: Logged In: YES user_id=80475 This could be considered a bug. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 From noreply at sourceforge.net Sun Oct 24 19:21:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 24 19:22:54 2004 Subject: [ python-Bugs-1052242 ] Attempt to run all atexit handlers Message-ID: Bugs item #1052242, was opened at 2004-10-22 09:31 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Greg Chapman (glchapman) >Assigned to: Skip Montanaro (montanaro) Summary: Attempt to run all atexit handlers Initial Comment: Currently, if an atexit handler raises an exception during _run_exitfuncs, none of the remaining handlers in the _exithandlers list gets run. I would prefer it if _run_exitfuncs did something like: while _exithandlers: func, targs, kargs = _exithandlers.pop() try: func(*targs, **kargs) except SystemExit: raise except: import sys, traceback print >> sys.stderr, "Error in sys.exitfunc:" traceback.print_exc() The SystemExit except clause is there because it looks like (given its special treatment in call_sys_exitfunc) SystemExit is envisioned as a way of intentionally breaking out of the exithandlers loop. ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2004-10-24 12:21 Message: Logged In: YES user_id=44345 OTOH, since the ordering of the exit handlers is not guaranteed, exiting after one exception is also reasonable behavior. Attached is a diff that does two things: adds a set_run_all function and converts the functionality into an ExitHandler class to keep from all the different bits of global state. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-23 19:56 Message: Logged In: YES user_id=80475 This could be considered a bug. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 From noreply at sourceforge.net Sun Oct 24 23:20:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 24 23:20:11 2004 Subject: [ python-Bugs-1050828 ] reindent.py strips execute permission on Unix Message-ID: Bugs item #1050828, was opened at 2004-10-20 12:48 Message generated for change (Comment added) made by facundobatista You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Ripton (dripton) >Assigned to: Tim Peters (tim_one) Summary: reindent.py strips execute permission on Unix Initial Comment: Tools/scripts/reindent.py strips execute permission from its target file. If the tool detects that changes need to be made, then the target file is renamed to its original name + '.bak', then a new file is created with the original file's name and its whitespace-modified contents. No manipulation of the new file's permissions is done, so it ends up with the default for the user's umask. This makes reindent.py annoying to run in an automated fashion, if you rely on execute permission. The fix is trivial, except for portability. Here's the Posix-only version: + if os.name == "posix": + mode = os.stat(bak).st_mode + os.chmod(file, mode) Patch available upon request. ---------------------------------------------------------------------- >Comment By: Facundo Batista (facundobatista) Date: 2004-10-24 18:20 Message: Logged In: YES user_id=752496 Instead of renaming the file and creating a new one, just copied the original and that's all (the target file is open with write mode, the content replaced, but the permissions stay). A patch is attached. Tim, I'm assigning this to you to doublecheck (not found unit tests for this script). . Facundo ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 From noreply at sourceforge.net Mon Oct 25 02:46:13 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 02:46:19 2004 Subject: [ python-Bugs-1047540 ] Turtle.py hangs Idle Message-ID: Bugs item #1047540, was opened at 2004-10-15 13:51 Message generated for change (Comment added) made by tso You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047540&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Taro Ogawa (tso) Assigned to: Nobody/Anonymous (nobody) Summary: Turtle.py hangs Idle Initial Comment: 1 line fix: After line 336's: root.destroy() insert root.quit() [Since IDLE is Tkinter based, the implict quit can't happen immediately. Worse, the shell has difficulty resetting and using a ^C can leave a zombie process] -T. ---------------------------------------------------------------------- >Comment By: Taro Ogawa (tso) Date: 2004-10-25 10:46 Message: Logged In: YES user_id=121025 Sorry - should have included this: NT4, Python 2.3.3 [This isn't an isolated case - I've had problems with other programs behaving similarly on my XP box (using 2.3.4), eg the pygame demos]. Windows only thing...? -T. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-24 10:35 Message: Logged In: YES user_id=80475 What version of Python are you using? I cannot reproduce this on Py2.3 or on Py2.4b1. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1047540&group_id=5470 From noreply at sourceforge.net Mon Oct 25 08:41:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 08:41:21 2004 Subject: [ python-Bugs-1053539 ] time module returns wrong timezone under windows Message-ID: Bugs item #1053539, was opened at 2004-10-25 06:41 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 Category: Python Library Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: panjunyong (panjunyong) Assigned to: Nobody/Anonymous (nobody) Summary: time module returns wrong timezone under windows Initial Comment: I have tested windows xp/2000 Chinese version with python 2.3.3 and python 2.4b1. My timezone in windows is GMT+8. But: >>> import time >>> time.timezone -28800 -28800 means GMT-8, but not GMT+8. This cause time.time() time.gmtime() return wrong values(16 hours late). I have no such problem under linux. BTW, I found the issue: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 So I think maybe this problem is sensitive with my Chinese platform too: >>> time.tzname ('\xd6\xd0\xb9\xfa\xb1\xea\xd7\xbc\xca\xb1\xbc\xe4', '\xd6\xd0\xb9\xfa\xb1\xea\x d7\xbc\xca\xb1\xbc\xe4') ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 From noreply at sourceforge.net Mon Oct 25 10:50:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 10:51:00 2004 Subject: [ python-Feature Requests-1031288 ] Update unicodedata to version 4.0.1 Message-ID: Feature Requests item #1031288, was opened at 2004-09-20 17:37 Message generated for change (Comment added) made by lemburg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1031288&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 5 Submitted By: Oliver Horn (ohorn) Assigned to: Nobody/Anonymous (nobody) Summary: Update unicodedata to version 4.0.1 Initial Comment: The unicodedata library (shipped with Python 2.3.4 and 2.4a3) is still version 3.2.0 albeit version 4.0.1 is available from unicode.org. Is it possible to update 4.0.1, at least for Python 2.4? ---------------------------------------------------------------------- >Comment By: M.-A. Lemburg (lemburg) Date: 2004-10-25 10:50 Message: Logged In: YES user_id=38388 While Python 2.4 is not possible anymore, we should consider this change for Python 2.5. I don't think we should stick to version 3.2 forever just because some RFC editors got their text wrong. Besides, changes to the Unicode database are usually backwards compatible for the vast majority of code points. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 13:54 Message: Logged In: YES user_id=21627 For Python 2.4, this is not possible, as the beta has already been published. For Python 2.5, it might be possible, but is difficult. Python relies on the unicodedata 3.2.0, as the IDNA RFCs mandate that Unicode 3.2 is used to implement IDNA. So any integration of 4.0.1 must a) still maintain access to the 3.2.0 data b) change all code that relies on 3.2.0 data to refer to these data explicitly c) not simply double the amount of data, but somehow allow for a differential representation. This is very difficult to implement, and hence hasn't been implemented for 2.4 (and might not be implemented for 2.5). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1031288&group_id=5470 From noreply at sourceforge.net Mon Oct 25 11:06:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 11:06:46 2004 Subject: [ python-Bugs-1053604 ] Unjustified SyntaxWarning Message-ID: Bugs item #1053604, was opened at 2004-10-25 11:06 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053604&group_id=5470 Category: Parser/Compiler Group: None Status: Open Resolution: None Priority: 5 Submitted By: Markus Gritsch (markus_gritsch) Assigned to: Nobody/Anonymous (nobody) Summary: Unjustified SyntaxWarning Initial Comment: The following short script produces a "SyntaxWarning: name 'cPickle' is assigned to before global declaration", which is IMO not correct. mode = 'pickle' def main(): if mode == 'pickle': global cPickle import cPickle elif mode == 'socket': global cPickle, socket import cPickle, socket elif mode == 'sqlite': global sqlite import sqlite else: print 'Warning: Unknown mode.' def test(): print cPickle.__doc__ if __name__ == '__main__': main() test() ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053604&group_id=5470 From noreply at sourceforge.net Mon Oct 25 12:09:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 12:09:26 2004 Subject: [ python-Bugs-1053629 ] Why is this expression == False? Message-ID: Bugs item #1053629, was opened at 2004-10-25 10:09 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053629&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Alexander R?dseth (alexanro) Assigned to: Nobody/Anonymous (nobody) Summary: Why is this expression == False? Initial Comment: I am not sure if this is a bug or not, but I'm very curious of the reason for the following result: >>> from math import * >>> theta = pi >>> e**(complex(0,1)*theta)==cos(theta)+complex(0,1)*sin(theta) False This is supposed to be True, unless I've made some typo or made some other mistake. It's a version of Euler's equation. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053629&group_id=5470 From noreply at sourceforge.net Mon Oct 25 12:33:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 12:33:09 2004 Subject: [ python-Feature Requests-1052098 ] Seting defaultencoding through an environment variable Message-ID: Feature Requests item #1052098, was opened at 2004-10-22 12:11 Message generated for change (Comment added) made by doerwalter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 Category: Python Library Group: None Status: Open Resolution: Rejected Priority: 1 Submitted By: Diedrich Vorberg (diedrich) Assigned to: Nobody/Anonymous (nobody) Summary: Seting defaultencoding through an environment variable Initial Comment: Hallo, I'd love to be able to set Python's default encoding on a per instance basis through an environment variable like PYTHON_DEFAULT_ENCODING or something. That would require a trivial modification to site.py. Diedrich ---------------------------------------------------------------------- >Comment By: Walter D?rwald (doerwalter) Date: 2004-10-25 12:33 Message: Logged In: YES user_id=89016 There's only one spot in XIST where the default encoding is relevant: When you pass a str object to one of the node constructors. This str object is converted to unicode using the default encoding. So html.p("?") won't work, unless you change the default encoding, but html.p(unicode("?", "iso- 8859-1")) will, and so will html.p(u"?") with a proper PEP 263 encoding declaration at the beginning of the file. If this doesn't fix you problem, you can mail me directly. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 14:41 Message: Logged In: YES user_id=442849 Yes, that's a way to go. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 14:25 Message: Logged In: YES user_id=21627 If all you want is to specify the appEncoding through an environment variable: this should be really easy. Instead of class Unicode(varchar): def __init__(self, columnName=None, appEncoding="iso-8859-1"): datatype.__init__(self, columnName) self._appEncoding = appEncoding do this instead class Unicode(varchar): def __init__(self, columnName=None, appEncoding=os.environ.get("ORM_ENCODING", "iso-8859-1")): datatype.__init__(self, columnName) self._appEncoding = appEncoding Then set ORM_ENCODING as you'ld like. No other changes to your source are required. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 14:06 Message: Logged In: YES user_id=442849 The reload(sys) bid is a good idea, thanks. Anyway, I can't fix XIST, because it it complicated as heck and I don't really have time to do that. Also I'm not sure, it this is really due to a fault in XIST or some side-effect I can't fathom. ORM is not touched by the encoding problem itself, it's in the data model modules. Each and every Unicode column needs a appEncoding= parameter on definition. That's not a technical problem, though, it's just plain ugly. (This might go away in the mid-run, because I'll re-implement parts of orm and I can put unicode support on the todo list. But until then...;-) ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 13:50 Message: Logged In: YES user_id=21627 I see. For the XIST case, it probably would be best to correct XIST, to always use explicit encodings. Alternatively, would it be possible to pass byte strings instead of Unicode objects to XIST in the cases which XIST doesn't handle correctly? If not, why not? If you cannot fix XIST, here is a work-around: import sys reload(sys) sys.setdefaultencoding("desired encoding") This is somewhat of a hack, but using a hack is ok if you are looking for a work-around for a problem that really should be fixed in the long run, anyway. I don't understand the orm example. What is "each and every setting"? Grepping the source of orm 1.0.1, the word "setting" occurs only in two places, both apparently irrelevant. If you are talking about some sort of configuration file - wouldn't it be good to create a library for the configuration file, and make only that library be aware of the encoding of the configuration file? ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 13:30 Message: Logged In: YES user_id=442849 Ah.. ok.. some examples: I'm writing Web-Applications using CGI or FastCGI. The webserver will pass user input as plain strings but within my program I want to use plain strings as little as possible to avoid all kinds of problems with diacritical characters (foremost umlauts and the EUR symbol). The websites the CGI scripts are used with usually have one charset for all pages. But some sites use iso-8859-1 and some use utf-8 and I don't want to modify my scripts to fit the charset of the site. Rather I'd like to use Apache's SetEnv directive to tell my scripts what charset their input is going to be in. Yes, I know, there are other ways to handle this :-) I keep using XIST for html output (http://www.livinglogic.de/Python). XIST tries to avoid using the default encoding, probably in the light of it being depricated. But it does not avoid it completely. There are a number of subtle cases in which it still depends on it and in all these cases I run into run-time errors which can be avoided in a snap by setting the default encoding to what my actual default encoding is. I use XIST with Zope and FastCGI, running several instances of the same interpreter on the same machine, with different default encodings each (mostly utf-8, but some iso-8859-1, which I can't just convert). My own OSS project orm (http://t4w.de/orm) carefully seperates the database's and the application's charset but uses Python's default encoding as the default for each and every setting. Those settings need to be set explicitly if the default encoding does not match the encoding actually used, cluttering the sourcecode and making maintaince more difficult. I use orm in connection with all the above cases. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:55 Message: Logged In: YES user_id=21627 No, I'm asking *why* you consider it useful. For example a) here is this and that problem, and there is absolutely no way to express this in Python today. If that feature is added, I can write it in the following way. b) here is this and that problem. Today, I have to write it in this way. With the feature added, I can write it in that way. That way is better than this way because of such and such reason. IOW, I would like to see what you are using the default encoding *for* that makes it desirable to set it through an environment variable. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:40 Message: Logged In: YES user_id=442849 Well... what more than useful is it supposed to be? Earh-shaking? Spiritually rewarding? ;-) As long as there is a default encoding I'd like to be able to set it in a simpler way than I can now. And I'd like to be able to set it once, at interpreter startup. An environment variable seems like a reasonable way to do that. If the notion of default encoding is removed from Python's library the suggested modification is going to go away as well. If you want to reject the feature request because the notion of default encoding is depricated, than so be it. From my point of view the suggested modification is something I'd like to see. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 12:09 Message: Logged In: YES user_id=21627 I take it that you are withdrawing your feature request, then? Closing it as rejected; if you still think this should be done, please indicate so. It would be good if you then could give some reasoning beyond "it would be useful" ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 12:05 Message: Logged In: YES user_id=442849 Yes, you are right. I could also replace the global unicode() function with my own version that adheres to my global encoding settings. But that doesn't seem the right thing to do. Anyway, this is not so much a technical decision but one that touches design. Anyway, it's just a small thing I tought usefull. If the defaultencoding goes away altogether, I'll have to make something up on my own anyway. Thanks for your comments! ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 11:52 Message: Logged In: YES user_id=21627 If your application just needs a global setting, it can come up with its own environment variable, right? Just set MY_DEFAULT_ENCODING, and do encoding=os.environ["MY_DEFAULT_ENCODING"] Alternatively, try using locale.getpreferredencoding(); this may allow to avoid an additional environment variable altogether. ---------------------------------------------------------------------- Comment By: Diedrich Vorberg (diedrich) Date: 2004-10-23 11:08 Message: Logged In: YES user_id=442849 Even if the notion of default encoding were dropped from Python, my application will need a global variable to have one encoding that is used most of the time. That`s what I`m using the default encoding for right now. Also, at least one of the 3d party modules I'm using does the same and it makes tons of sense. Changing the default encoding from within my application doesn't work, because sys.setdefaultencoding() gets removed (which makes sense, too, of course). So I need a custom sitecustomize.py for every project which may or may not be adhered to, depending on the PYTHONPATH variable. That`s why a PYTHON_DEFAULT_ENCODING variable would be very usefull to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-23 10:32 Message: Logged In: YES user_id=21627 Why do you want this change? In the long run, Python will drop the notion of a default encoding; code should be changed to avoid relying on the default encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=1052098&group_id=5470 From noreply at sourceforge.net Mon Oct 25 12:35:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 12:35:55 2004 Subject: [ python-Bugs-1053629 ] Why is this expression == False? Message-ID: Bugs item #1053629, was opened at 2004-10-25 12:09 Message generated for change (Comment added) made by doerwalter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053629&group_id=5470 Category: None Group: Python 2.3 >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Alexander R?dseth (alexanro) Assigned to: Nobody/Anonymous (nobody) Summary: Why is this expression == False? Initial Comment: I am not sure if this is a bug or not, but I'm very curious of the reason for the following result: >>> from math import * >>> theta = pi >>> e**(complex(0,1)*theta)==cos(theta)+complex(0,1)*sin(theta) False This is supposed to be True, unless I've made some typo or made some other mistake. It's a version of Euler's equation. ---------------------------------------------------------------------- >Comment By: Walter D?rwald (doerwalter) Date: 2004-10-25 12:35 Message: Logged In: YES user_id=89016 This is due to rounding errors: >>> from math import * *>>> e**(1j*pi) (-1+1.2246063538223773e-16j) *>>> cos(pi)+1j*sin(pi) (-1+1.2246063538223773e-16j) Closing the bug report ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053629&group_id=5470 From noreply at sourceforge.net Mon Oct 25 13:09:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 13:10:02 2004 Subject: [ python-Bugs-1053629 ] Why is this expression == False? Message-ID: Bugs item #1053629, was opened at 2004-10-25 11:09 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053629&group_id=5470 Category: None Group: Python 2.3 Status: Closed Resolution: Invalid Priority: 5 Submitted By: Alexander R?dseth (alexanro) Assigned to: Nobody/Anonymous (nobody) Summary: Why is this expression == False? Initial Comment: I am not sure if this is a bug or not, but I'm very curious of the reason for the following result: >>> from math import * >>> theta = pi >>> e**(complex(0,1)*theta)==cos(theta)+complex(0,1)*sin(theta) False This is supposed to be True, unless I've made some typo or made some other mistake. It's a version of Euler's equation. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-25 12:09 Message: Logged In: YES user_id=6656 Heh. Walter, the two numbers you show are in fact equal. Also, the expression is True for me. I wonder what platform Alexander is on -- but not very much, attempting to verify equalities involving transcedental quantities using finite precision arithmetic is always going to be risky. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-10-25 11:35 Message: Logged In: YES user_id=89016 This is due to rounding errors: >>> from math import * *>>> e**(1j*pi) (-1+1.2246063538223773e-16j) *>>> cos(pi)+1j*sin(pi) (-1+1.2246063538223773e-16j) Closing the bug report ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053629&group_id=5470 From noreply at sourceforge.net Mon Oct 25 14:31:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 14:31:45 2004 Subject: [ python-Bugs-1053687 ] PyOS_InputHook not called in subprocess Message-ID: Bugs item #1053687, was opened at 2004-10-25 21:31 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053687&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Michiel de Hoon (mdehoon) Assigned to: Nobody/Anonymous (nobody) Summary: PyOS_InputHook not called in subprocess Initial Comment: PyOS_InputHook is a variable points to a user-defined function (e.g. in an extension module) that is to be called periodically when Python is idle (e.g. waiting for user input). It is used for example by Tkinter to get messages delivered to the graphics window. For Python run from the command prompt, PyOS_InputHook (if not NULL) is called ten times per second. In IDLE, when a subprocess is being used (so the default behavior), the main process and the subprocess each have their own PyOS_InputHook. The problem is that the subprocess (the one that runs the user's commands) does not call PyOS_InputHook. So if a user (in IDLE) imports an extension module that sets PyOS_InputHook, the extension module will not run correctly as PyOS_InputHook is being ignored. The solution to this problem is probably quite simple, maybe just a line if (PyOS_InputHook) PyOS_InputHook(); but I am not familiar enough with threads to be able to figure out what the best location for such a line would be. I tried putting it inside the loop in PyEval_EvalFrame in Python/ceval.c, which solved the problem on some platforms but not on others. --Michiel. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053687&group_id=5470 From noreply at sourceforge.net Mon Oct 25 15:01:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 15:01:22 2004 Subject: [ python-Bugs-1053539 ] time module returns wrong timezone under windows Message-ID: Bugs item #1053539, was opened at 2004-10-25 15:41 Message generated for change (Comment added) made by quiver You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 Category: Python Library Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: panjunyong (panjunyong) Assigned to: Nobody/Anonymous (nobody) Summary: time module returns wrong timezone under windows Initial Comment: I have tested windows xp/2000 Chinese version with python 2.3.3 and python 2.4b1. My timezone in windows is GMT+8. But: >>> import time >>> time.timezone -28800 -28800 means GMT-8, but not GMT+8. This cause time.time() time.gmtime() return wrong values(16 hours late). I have no such problem under linux. BTW, I found the issue: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 So I think maybe this problem is sensitive with my Chinese platform too: >>> time.tzname ('\xd6\xd0\xb9\xfa\xb1\xea\xd7\xbc\xca\xb1\xbc\xe4', '\xd6\xd0\xb9\xfa\xb1\xea\x d7\xbc\xca\xb1\xbc\xe4') ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 22:01 Message: Logged In: YES user_id=671362 > -28800 means GMT-8 No, look at the doc more carefully. www.python.org/doc/current/lib/module-time.html negative in most of Western Europe, positive in the US, zero in the UK If I remember correctly, China is on the opposite side of the world from the US. :-) > time.time() time.gmtime() return wrong values(16 hours > late). Reverify the time-zone with your Windows machine. BTW, "Most-of-Western-Europe" doesn't seem clear to me at all. What's wrong with: "negative in Asia, positive in the US, zero in the UK." ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 From noreply at sourceforge.net Mon Oct 25 15:26:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 15:27:00 2004 Subject: [ python-Bugs-1053539 ] time module returns wrong timezone under windows Message-ID: Bugs item #1053539, was opened at 2004-10-25 06:41 Message generated for change (Comment added) made by panjunyong You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 Category: Python Library Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: panjunyong (panjunyong) Assigned to: Nobody/Anonymous (nobody) Summary: time module returns wrong timezone under windows Initial Comment: I have tested windows xp/2000 Chinese version with python 2.3.3 and python 2.4b1. My timezone in windows is GMT+8. But: >>> import time >>> time.timezone -28800 -28800 means GMT-8, but not GMT+8. This cause time.time() time.gmtime() return wrong values(16 hours late). I have no such problem under linux. BTW, I found the issue: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 So I think maybe this problem is sensitive with my Chinese platform too: >>> time.tzname ('\xd6\xd0\xb9\xfa\xb1\xea\xd7\xbc\xca\xb1\xbc\xe4', '\xd6\xd0\xb9\xfa\xb1\xea\x d7\xbc\xca\xb1\xbc\xe4') ---------------------------------------------------------------------- >Comment By: panjunyong (panjunyong) Date: 2004-10-25 13:26 Message: Logged In: YES user_id=601368 I am sure with my windows timezone setting. The time.tzname is Chinese, which means 'Chinese Standard Time'. And China is in GMT+8 zone. Everything is ok under linux: >>> import time >>> time.tzname ('GMT+8', 'GMT+8') >>> time.timezone 28800 ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 13:01 Message: Logged In: YES user_id=671362 > -28800 means GMT-8 No, look at the doc more carefully. www.python.org/doc/current/lib/module-time.html negative in most of Western Europe, positive in the US, zero in the UK If I remember correctly, China is on the opposite side of the world from the US. :-) > time.time() time.gmtime() return wrong values(16 hours > late). Reverify the time-zone with your Windows machine. BTW, "Most-of-Western-Europe" doesn't seem clear to me at all. What's wrong with: "negative in Asia, positive in the US, zero in the UK." ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 From noreply at sourceforge.net Mon Oct 25 15:56:02 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 15:56:10 2004 Subject: [ python-Bugs-1053539 ] time module returns wrong timezone under windows Message-ID: Bugs item #1053539, was opened at 2004-10-25 06:41 Message generated for change (Comment added) made by panjunyong You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 Category: Python Library Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: panjunyong (panjunyong) Assigned to: Nobody/Anonymous (nobody) Summary: time module returns wrong timezone under windows Initial Comment: I have tested windows xp/2000 Chinese version with python 2.3.3 and python 2.4b1. My timezone in windows is GMT+8. But: >>> import time >>> time.timezone -28800 -28800 means GMT-8, but not GMT+8. This cause time.time() time.gmtime() return wrong values(16 hours late). I have no such problem under linux. BTW, I found the issue: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 So I think maybe this problem is sensitive with my Chinese platform too: >>> time.tzname ('\xd6\xd0\xb9\xfa\xb1\xea\xd7\xbc\xca\xb1\xbc\xe4', '\xd6\xd0\xb9\xfa\xb1\xea\x d7\xbc\xca\xb1\xbc\xe4') ---------------------------------------------------------------------- >Comment By: panjunyong (panjunyong) Date: 2004-10-25 13:56 Message: Logged In: YES user_id=601368 Sorry, time should be right under Windows(my mistake :-( ). Then since the result is quite different, maybe time is wrong under Linux? windows: >>> import time >>> time.timezone -28800 >>> time.localtime() (2004, 10, 25, 21, 28, 24, 0, 299, 0) >>> time.gmtime() (2004, 10, 25, 13, 28, 29, 0, 299, 0) And Linux: >>> import time >>> time.timezone 28800 >>> time.tzname ('GMT+8', 'GMT+8') >>> time.localtime() (2004, 10, 25, 21, 28, 6, 0, 299, 0) >>> time.gmtime() (2004, 10, 26, 5, 28, 10, 1, 300, 0) ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 13:26 Message: Logged In: YES user_id=601368 I am sure with my windows timezone setting. The time.tzname is Chinese, which means 'Chinese Standard Time'. And China is in GMT+8 zone. Everything is ok under linux: >>> import time >>> time.tzname ('GMT+8', 'GMT+8') >>> time.timezone 28800 ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 13:01 Message: Logged In: YES user_id=671362 > -28800 means GMT-8 No, look at the doc more carefully. www.python.org/doc/current/lib/module-time.html negative in most of Western Europe, positive in the US, zero in the UK If I remember correctly, China is on the opposite side of the world from the US. :-) > time.time() time.gmtime() return wrong values(16 hours > late). Reverify the time-zone with your Windows machine. BTW, "Most-of-Western-Europe" doesn't seem clear to me at all. What's wrong with: "negative in Asia, positive in the US, zero in the UK." ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 From noreply at sourceforge.net Mon Oct 25 16:13:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 16:13:19 2004 Subject: [ python-Bugs-1053539 ] time module returns wrong timezone under windows Message-ID: Bugs item #1053539, was opened at 2004-10-25 06:41 Message generated for change (Comment added) made by panjunyong You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 Category: Python Library Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: panjunyong (panjunyong) Assigned to: Nobody/Anonymous (nobody) Summary: time module returns wrong timezone under windows Initial Comment: I have tested windows xp/2000 Chinese version with python 2.3.3 and python 2.4b1. My timezone in windows is GMT+8. But: >>> import time >>> time.timezone -28800 -28800 means GMT-8, but not GMT+8. This cause time.time() time.gmtime() return wrong values(16 hours late). I have no such problem under linux. BTW, I found the issue: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 So I think maybe this problem is sensitive with my Chinese platform too: >>> time.tzname ('\xd6\xd0\xb9\xfa\xb1\xea\xd7\xbc\xca\xb1\xbc\xe4', '\xd6\xd0\xb9\xfa\xb1\xea\x d7\xbc\xca\xb1\xbc\xe4') ---------------------------------------------------------------------- >Comment By: panjunyong (panjunyong) Date: 2004-10-25 14:13 Message: Logged In: YES user_id=601368 Oh, is this a linux bug? See these two commands under linux: # date Mon Oct 25 22:06:58 GMT+8 2004 # date -u Tue Oct 26 06:06:59 UTC 2004 the UTC time should be wrong. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 13:56 Message: Logged In: YES user_id=601368 Sorry, time should be right under Windows(my mistake :-( ). Then since the result is quite different, maybe time is wrong under Linux? windows: >>> import time >>> time.timezone -28800 >>> time.localtime() (2004, 10, 25, 21, 28, 24, 0, 299, 0) >>> time.gmtime() (2004, 10, 25, 13, 28, 29, 0, 299, 0) And Linux: >>> import time >>> time.timezone 28800 >>> time.tzname ('GMT+8', 'GMT+8') >>> time.localtime() (2004, 10, 25, 21, 28, 6, 0, 299, 0) >>> time.gmtime() (2004, 10, 26, 5, 28, 10, 1, 300, 0) ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 13:26 Message: Logged In: YES user_id=601368 I am sure with my windows timezone setting. The time.tzname is Chinese, which means 'Chinese Standard Time'. And China is in GMT+8 zone. Everything is ok under linux: >>> import time >>> time.tzname ('GMT+8', 'GMT+8') >>> time.timezone 28800 ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 13:01 Message: Logged In: YES user_id=671362 > -28800 means GMT-8 No, look at the doc more carefully. www.python.org/doc/current/lib/module-time.html negative in most of Western Europe, positive in the US, zero in the UK If I remember correctly, China is on the opposite side of the world from the US. :-) > time.time() time.gmtime() return wrong values(16 hours > late). Reverify the time-zone with your Windows machine. BTW, "Most-of-Western-Europe" doesn't seem clear to me at all. What's wrong with: "negative in Asia, positive in the US, zero in the UK." ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 From noreply at sourceforge.net Mon Oct 25 16:26:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 16:27:00 2004 Subject: [ python-Bugs-1053539 ] time module returns wrong timezone under windows Message-ID: Bugs item #1053539, was opened at 2004-10-25 15:41 Message generated for change (Comment added) made by quiver You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 Category: Python Library Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: panjunyong (panjunyong) Assigned to: Nobody/Anonymous (nobody) Summary: time module returns wrong timezone under windows Initial Comment: I have tested windows xp/2000 Chinese version with python 2.3.3 and python 2.4b1. My timezone in windows is GMT+8. But: >>> import time >>> time.timezone -28800 -28800 means GMT-8, but not GMT+8. This cause time.time() time.gmtime() return wrong values(16 hours late). I have no such problem under linux. BTW, I found the issue: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 So I think maybe this problem is sensitive with my Chinese platform too: >>> time.tzname ('\xd6\xd0\xb9\xfa\xb1\xea\xd7\xbc\xca\xb1\xbc\xe4', '\xd6\xd0\xb9\xfa\xb1\xea\x d7\xbc\xca\xb1\xbc\xe4') ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 23:26 Message: Logged In: YES user_id=671362 > Oh, is this a linux bug? Looks like. On my box, $ date Mon Oct 25 23:20:26 JST 2004 $ date -u Mon Oct 25 14:20:27 UTF 2004 Just out of curiosity, what's your /proc/version? You just say Linux, and don't give any further info. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 23:13 Message: Logged In: YES user_id=601368 Oh, is this a linux bug? See these two commands under linux: # date Mon Oct 25 22:06:58 GMT+8 2004 # date -u Tue Oct 26 06:06:59 UTC 2004 the UTC time should be wrong. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 22:56 Message: Logged In: YES user_id=601368 Sorry, time should be right under Windows(my mistake :-( ). Then since the result is quite different, maybe time is wrong under Linux? windows: >>> import time >>> time.timezone -28800 >>> time.localtime() (2004, 10, 25, 21, 28, 24, 0, 299, 0) >>> time.gmtime() (2004, 10, 25, 13, 28, 29, 0, 299, 0) And Linux: >>> import time >>> time.timezone 28800 >>> time.tzname ('GMT+8', 'GMT+8') >>> time.localtime() (2004, 10, 25, 21, 28, 6, 0, 299, 0) >>> time.gmtime() (2004, 10, 26, 5, 28, 10, 1, 300, 0) ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 22:26 Message: Logged In: YES user_id=601368 I am sure with my windows timezone setting. The time.tzname is Chinese, which means 'Chinese Standard Time'. And China is in GMT+8 zone. Everything is ok under linux: >>> import time >>> time.tzname ('GMT+8', 'GMT+8') >>> time.timezone 28800 ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 22:01 Message: Logged In: YES user_id=671362 > -28800 means GMT-8 No, look at the doc more carefully. www.python.org/doc/current/lib/module-time.html negative in most of Western Europe, positive in the US, zero in the UK If I remember correctly, China is on the opposite side of the world from the US. :-) > time.time() time.gmtime() return wrong values(16 hours > late). Reverify the time-zone with your Windows machine. BTW, "Most-of-Western-Europe" doesn't seem clear to me at all. What's wrong with: "negative in Asia, positive in the US, zero in the UK." ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 From noreply at sourceforge.net Mon Oct 25 16:39:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 16:39:23 2004 Subject: [ python-Bugs-1053539 ] time module returns wrong timezone under windows Message-ID: Bugs item #1053539, was opened at 2004-10-25 06:41 Message generated for change (Comment added) made by panjunyong You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 Category: Python Library Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: panjunyong (panjunyong) Assigned to: Nobody/Anonymous (nobody) Summary: time module returns wrong timezone under windows Initial Comment: I have tested windows xp/2000 Chinese version with python 2.3.3 and python 2.4b1. My timezone in windows is GMT+8. But: >>> import time >>> time.timezone -28800 -28800 means GMT-8, but not GMT+8. This cause time.time() time.gmtime() return wrong values(16 hours late). I have no such problem under linux. BTW, I found the issue: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 So I think maybe this problem is sensitive with my Chinese platform too: >>> time.tzname ('\xd6\xd0\xb9\xfa\xb1\xea\xd7\xbc\xca\xb1\xbc\xe4', '\xd6\xd0\xb9\xfa\xb1\xea\x d7\xbc\xca\xb1\xbc\xe4') ---------------------------------------------------------------------- >Comment By: panjunyong (panjunyong) Date: 2004-10-25 14:39 Message: Logged In: YES user_id=601368 sorry, here is my linux /pro/version: # more /proc/version Linux version 2.4.25-1 (gcc version 3.3.3 (Debian )) #1 SMP Wed Mar 10 13:10:05 GMT+8 2004 and more about date: # date +%Z GMT+8 # date +%z -0800 ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 14:26 Message: Logged In: YES user_id=671362 > Oh, is this a linux bug? Looks like. On my box, $ date Mon Oct 25 23:20:26 JST 2004 $ date -u Mon Oct 25 14:20:27 UTF 2004 Just out of curiosity, what's your /proc/version? You just say Linux, and don't give any further info. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 14:13 Message: Logged In: YES user_id=601368 Oh, is this a linux bug? See these two commands under linux: # date Mon Oct 25 22:06:58 GMT+8 2004 # date -u Tue Oct 26 06:06:59 UTC 2004 the UTC time should be wrong. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 13:56 Message: Logged In: YES user_id=601368 Sorry, time should be right under Windows(my mistake :-( ). Then since the result is quite different, maybe time is wrong under Linux? windows: >>> import time >>> time.timezone -28800 >>> time.localtime() (2004, 10, 25, 21, 28, 24, 0, 299, 0) >>> time.gmtime() (2004, 10, 25, 13, 28, 29, 0, 299, 0) And Linux: >>> import time >>> time.timezone 28800 >>> time.tzname ('GMT+8', 'GMT+8') >>> time.localtime() (2004, 10, 25, 21, 28, 6, 0, 299, 0) >>> time.gmtime() (2004, 10, 26, 5, 28, 10, 1, 300, 0) ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 13:26 Message: Logged In: YES user_id=601368 I am sure with my windows timezone setting. The time.tzname is Chinese, which means 'Chinese Standard Time'. And China is in GMT+8 zone. Everything is ok under linux: >>> import time >>> time.tzname ('GMT+8', 'GMT+8') >>> time.timezone 28800 ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 13:01 Message: Logged In: YES user_id=671362 > -28800 means GMT-8 No, look at the doc more carefully. www.python.org/doc/current/lib/module-time.html negative in most of Western Europe, positive in the US, zero in the UK If I remember correctly, China is on the opposite side of the world from the US. :-) > time.time() time.gmtime() return wrong values(16 hours > late). Reverify the time-zone with your Windows machine. BTW, "Most-of-Western-Europe" doesn't seem clear to me at all. What's wrong with: "negative in Asia, positive in the US, zero in the UK." ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 From noreply at sourceforge.net Mon Oct 25 16:53:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 16:53:32 2004 Subject: [ python-Bugs-1053539 ] time module returns wrong timezone under windows Message-ID: Bugs item #1053539, was opened at 2004-10-25 06:41 Message generated for change (Comment added) made by panjunyong You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 Category: Python Library Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: panjunyong (panjunyong) Assigned to: Nobody/Anonymous (nobody) Summary: time module returns wrong timezone under windows Initial Comment: I have tested windows xp/2000 Chinese version with python 2.3.3 and python 2.4b1. My timezone in windows is GMT+8. But: >>> import time >>> time.timezone -28800 -28800 means GMT-8, but not GMT+8. This cause time.time() time.gmtime() return wrong values(16 hours late). I have no such problem under linux. BTW, I found the issue: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 So I think maybe this problem is sensitive with my Chinese platform too: >>> time.tzname ('\xd6\xd0\xb9\xfa\xb1\xea\xd7\xbc\xca\xb1\xbc\xe4', '\xd6\xd0\xb9\xfa\xb1\xea\x d7\xbc\xca\xb1\xbc\xe4') ---------------------------------------------------------------------- >Comment By: panjunyong (panjunyong) Date: 2004-10-25 14:53 Message: Logged In: YES user_id=601368 Oh, I should set my linux time zone to GMT-8, which represent the zone +0800, but not GMT+8. This is quite strange. I learn it from this: http://lists.debian.org/debian-chinese-gb/2002/07/msg00063.html Thanks for your kind. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 14:39 Message: Logged In: YES user_id=601368 sorry, here is my linux /pro/version: # more /proc/version Linux version 2.4.25-1 (gcc version 3.3.3 (Debian )) #1 SMP Wed Mar 10 13:10:05 GMT+8 2004 and more about date: # date +%Z GMT+8 # date +%z -0800 ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 14:26 Message: Logged In: YES user_id=671362 > Oh, is this a linux bug? Looks like. On my box, $ date Mon Oct 25 23:20:26 JST 2004 $ date -u Mon Oct 25 14:20:27 UTF 2004 Just out of curiosity, what's your /proc/version? You just say Linux, and don't give any further info. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 14:13 Message: Logged In: YES user_id=601368 Oh, is this a linux bug? See these two commands under linux: # date Mon Oct 25 22:06:58 GMT+8 2004 # date -u Tue Oct 26 06:06:59 UTC 2004 the UTC time should be wrong. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 13:56 Message: Logged In: YES user_id=601368 Sorry, time should be right under Windows(my mistake :-( ). Then since the result is quite different, maybe time is wrong under Linux? windows: >>> import time >>> time.timezone -28800 >>> time.localtime() (2004, 10, 25, 21, 28, 24, 0, 299, 0) >>> time.gmtime() (2004, 10, 25, 13, 28, 29, 0, 299, 0) And Linux: >>> import time >>> time.timezone 28800 >>> time.tzname ('GMT+8', 'GMT+8') >>> time.localtime() (2004, 10, 25, 21, 28, 6, 0, 299, 0) >>> time.gmtime() (2004, 10, 26, 5, 28, 10, 1, 300, 0) ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 13:26 Message: Logged In: YES user_id=601368 I am sure with my windows timezone setting. The time.tzname is Chinese, which means 'Chinese Standard Time'. And China is in GMT+8 zone. Everything is ok under linux: >>> import time >>> time.tzname ('GMT+8', 'GMT+8') >>> time.timezone 28800 ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 13:01 Message: Logged In: YES user_id=671362 > -28800 means GMT-8 No, look at the doc more carefully. www.python.org/doc/current/lib/module-time.html negative in most of Western Europe, positive in the US, zero in the UK If I remember correctly, China is on the opposite side of the world from the US. :-) > time.time() time.gmtime() return wrong values(16 hours > late). Reverify the time-zone with your Windows machine. BTW, "Most-of-Western-Europe" doesn't seem clear to me at all. What's wrong with: "negative in Asia, positive in the US, zero in the UK." ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 From noreply at sourceforge.net Mon Oct 25 16:55:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 16:55:25 2004 Subject: [ python-Bugs-1053539 ] time module returns wrong timezone under windows Message-ID: Bugs item #1053539, was opened at 2004-10-25 02:41 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 Category: Python Library >Group: Not a Bug >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: panjunyong (panjunyong) Assigned to: Nobody/Anonymous (nobody) Summary: time module returns wrong timezone under windows Initial Comment: I have tested windows xp/2000 Chinese version with python 2.3.3 and python 2.4b1. My timezone in windows is GMT+8. But: >>> import time >>> time.timezone -28800 -28800 means GMT-8, but not GMT+8. This cause time.time() time.gmtime() return wrong values(16 hours late). I have no such problem under linux. BTW, I found the issue: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 So I think maybe this problem is sensitive with my Chinese platform too: >>> time.tzname ('\xd6\xd0\xb9\xfa\xb1\xea\xd7\xbc\xca\xb1\xbc\xe4', '\xd6\xd0\xb9\xfa\xb1\xea\x d7\xbc\xca\xb1\xbc\xe4') ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-25 10:55 Message: Logged In: YES user_id=31435 Closing as invalid, since there doesn't appear to be a Python bug here. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 10:53 Message: Logged In: YES user_id=601368 Oh, I should set my linux time zone to GMT-8, which represent the zone +0800, but not GMT+8. This is quite strange. I learn it from this: http://lists.debian.org/debian-chinese-gb/2002/07/msg00063.html Thanks for your kind. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 10:39 Message: Logged In: YES user_id=601368 sorry, here is my linux /pro/version: # more /proc/version Linux version 2.4.25-1 (gcc version 3.3.3 (Debian )) #1 SMP Wed Mar 10 13:10:05 GMT+8 2004 and more about date: # date +%Z GMT+8 # date +%z -0800 ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 10:26 Message: Logged In: YES user_id=671362 > Oh, is this a linux bug? Looks like. On my box, $ date Mon Oct 25 23:20:26 JST 2004 $ date -u Mon Oct 25 14:20:27 UTF 2004 Just out of curiosity, what's your /proc/version? You just say Linux, and don't give any further info. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 10:13 Message: Logged In: YES user_id=601368 Oh, is this a linux bug? See these two commands under linux: # date Mon Oct 25 22:06:58 GMT+8 2004 # date -u Tue Oct 26 06:06:59 UTC 2004 the UTC time should be wrong. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 09:56 Message: Logged In: YES user_id=601368 Sorry, time should be right under Windows(my mistake :-( ). Then since the result is quite different, maybe time is wrong under Linux? windows: >>> import time >>> time.timezone -28800 >>> time.localtime() (2004, 10, 25, 21, 28, 24, 0, 299, 0) >>> time.gmtime() (2004, 10, 25, 13, 28, 29, 0, 299, 0) And Linux: >>> import time >>> time.timezone 28800 >>> time.tzname ('GMT+8', 'GMT+8') >>> time.localtime() (2004, 10, 25, 21, 28, 6, 0, 299, 0) >>> time.gmtime() (2004, 10, 26, 5, 28, 10, 1, 300, 0) ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 09:26 Message: Logged In: YES user_id=601368 I am sure with my windows timezone setting. The time.tzname is Chinese, which means 'Chinese Standard Time'. And China is in GMT+8 zone. Everything is ok under linux: >>> import time >>> time.tzname ('GMT+8', 'GMT+8') >>> time.timezone 28800 ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 09:01 Message: Logged In: YES user_id=671362 > -28800 means GMT-8 No, look at the doc more carefully. www.python.org/doc/current/lib/module-time.html negative in most of Western Europe, positive in the US, zero in the UK If I remember correctly, China is on the opposite side of the world from the US. :-) > time.time() time.gmtime() return wrong values(16 hours > late). Reverify the time-zone with your Windows machine. BTW, "Most-of-Western-Europe" doesn't seem clear to me at all. What's wrong with: "negative in Asia, positive in the US, zero in the UK." ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 From noreply at sourceforge.net Mon Oct 25 16:59:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 16:59:53 2004 Subject: [ python-Bugs-1053539 ] time module returns wrong timezone under windows Message-ID: Bugs item #1053539, was opened at 2004-10-25 06:41 Message generated for change (Settings changed) made by panjunyong You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 Category: Python Library >Group: Platform-specific >Status: Open >Resolution: Rejected Priority: 5 Submitted By: panjunyong (panjunyong) Assigned to: Nobody/Anonymous (nobody) Summary: time module returns wrong timezone under windows Initial Comment: I have tested windows xp/2000 Chinese version with python 2.3.3 and python 2.4b1. My timezone in windows is GMT+8. But: >>> import time >>> time.timezone -28800 -28800 means GMT-8, but not GMT+8. This cause time.time() time.gmtime() return wrong values(16 hours late). I have no such problem under linux. BTW, I found the issue: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1039270&group_id=5470 So I think maybe this problem is sensitive with my Chinese platform too: >>> time.tzname ('\xd6\xd0\xb9\xfa\xb1\xea\xd7\xbc\xca\xb1\xbc\xe4', '\xd6\xd0\xb9\xfa\xb1\xea\x d7\xbc\xca\xb1\xbc\xe4') ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 14:55 Message: Logged In: YES user_id=31435 Closing as invalid, since there doesn't appear to be a Python bug here. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 14:53 Message: Logged In: YES user_id=601368 Oh, I should set my linux time zone to GMT-8, which represent the zone +0800, but not GMT+8. This is quite strange. I learn it from this: http://lists.debian.org/debian-chinese-gb/2002/07/msg00063.html Thanks for your kind. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 14:39 Message: Logged In: YES user_id=601368 sorry, here is my linux /pro/version: # more /proc/version Linux version 2.4.25-1 (gcc version 3.3.3 (Debian )) #1 SMP Wed Mar 10 13:10:05 GMT+8 2004 and more about date: # date +%Z GMT+8 # date +%z -0800 ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 14:26 Message: Logged In: YES user_id=671362 > Oh, is this a linux bug? Looks like. On my box, $ date Mon Oct 25 23:20:26 JST 2004 $ date -u Mon Oct 25 14:20:27 UTF 2004 Just out of curiosity, what's your /proc/version? You just say Linux, and don't give any further info. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 14:13 Message: Logged In: YES user_id=601368 Oh, is this a linux bug? See these two commands under linux: # date Mon Oct 25 22:06:58 GMT+8 2004 # date -u Tue Oct 26 06:06:59 UTC 2004 the UTC time should be wrong. ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 13:56 Message: Logged In: YES user_id=601368 Sorry, time should be right under Windows(my mistake :-( ). Then since the result is quite different, maybe time is wrong under Linux? windows: >>> import time >>> time.timezone -28800 >>> time.localtime() (2004, 10, 25, 21, 28, 24, 0, 299, 0) >>> time.gmtime() (2004, 10, 25, 13, 28, 29, 0, 299, 0) And Linux: >>> import time >>> time.timezone 28800 >>> time.tzname ('GMT+8', 'GMT+8') >>> time.localtime() (2004, 10, 25, 21, 28, 6, 0, 299, 0) >>> time.gmtime() (2004, 10, 26, 5, 28, 10, 1, 300, 0) ---------------------------------------------------------------------- Comment By: panjunyong (panjunyong) Date: 2004-10-25 13:26 Message: Logged In: YES user_id=601368 I am sure with my windows timezone setting. The time.tzname is Chinese, which means 'Chinese Standard Time'. And China is in GMT+8 zone. Everything is ok under linux: >>> import time >>> time.tzname ('GMT+8', 'GMT+8') >>> time.timezone 28800 ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-25 13:01 Message: Logged In: YES user_id=671362 > -28800 means GMT-8 No, look at the doc more carefully. www.python.org/doc/current/lib/module-time.html negative in most of Western Europe, positive in the US, zero in the UK If I remember correctly, China is on the opposite side of the world from the US. :-) > time.time() time.gmtime() return wrong values(16 hours > late). Reverify the time-zone with your Windows machine. BTW, "Most-of-Western-Europe" doesn't seem clear to me at all. What's wrong with: "negative in Asia, positive in the US, zero in the UK." ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053539&group_id=5470 From noreply at sourceforge.net Mon Oct 25 17:20:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 17:21:03 2004 Subject: [ python-Bugs-1053819 ] Segfault in tuple_of_constants Message-ID: Bugs item #1053819, was opened at 2004-10-25 16:20 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053819&group_id=5470 Category: Parser/Compiler Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Duncan Grisby (dgrisby) Assigned to: Nobody/Anonymous (nobody) Summary: Segfault in tuple_of_constants Initial Comment: 2.4 beta 1 segfaults when compiling a file. The same file works fine on 2.3 and earlier. I have tracked the problem down to the tuple_of_constants function, called by optimize_code. Using valgrind (having turned off pymalloc), I see that the consts list seems to be uninitialised. See the following trace: $ valgrind --tool=memcheck --db-attach=yes ~/inst/bin/python killme.py ==13759== Memcheck, a memory error detector for x86-linux. ==13759== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward et al. ==13759== Using valgrind-2.2.0, a program supervision framework for x86-linux. ==13759== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward et al. ==13759== For more details, rerun with: -v ==13759== ==13759== Invalid read of size 4 ==13759== at 0x80B3BF5: tuple_of_constants (compile.c:422) ==13759== by 0x80B433B: optimize_code (compile.c:607) ==13759== by 0x80BD3F2: jcompile (compile.c:4988) ==13759== by 0x80BD0B2: PyNode_CompileFlags (compile.c:4894) ==13759== Address 0x1BF100A0 is 12 bytes after a block of size 36 alloc'd ==13759== at 0x1B904A80: malloc (vg_replace_malloc.c:131) ==13759== by 0x80D8A4C: _PyObject_GC_Malloc (gcmodule.c:1183) ==13759== by 0x80D8B31: _PyObject_GC_NewVar (gcmodule.c:1214) ==13759== by 0x80854FF: PyTuple_New (tupleobject.c:68) ==13759== ==13759== ---- Attach to debugger ? --- [Return/N/n/Y/y/C/c] ---- y starting debugger ==13759== starting debugger with cmd: /usr/bin/gdb -nw /proc/13761/fd/821 13761 GNU gdb Red Hat Linux (6.0post-0.20040223.19rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1". Attaching to program: /proc/13761/fd/821, process 13761 0x080b3bf5 in tuple_of_constants (codestr=0x1bf1ff90 "d", n=1, consts=0x1badc9c4) at Python/compile.c:422 422 constant = PyList_GET_ITEM(consts, arg); (gdb) bt #0 0x080b3bf5 in tuple_of_constants (codestr=0x1bf1ff90 "d", n=1, consts=0x1badc9c4) at Python/compile.c:422 #1 0x080b433c in optimize_code (code=0x1becb3e8, consts=0x1badc9c4, names=0x1bf1ed5c, lineno_obj=0x1be7cd90) at Python/compile.c:607 #2 0x080bd3f3 in jcompile (n=0x1bae6b10, filename=0x52bfea51 "killme.py", base=0x0, flags=0x52bfe808) at Python/compile.c:4988 #3 0x080bd0b3 in PyNode_CompileFlags (n=0x1bae6b10, filename=0x52bfea51 "killme.py", flags=0x52bfe808) at Python/compile.c:4894 #4 0x080d269a in run_node (n=0x1bae6b10, filename=0x52bfea51 "killme.py", globals=0x1b99a9ac, locals=0x1b99a9ac, flags=0x52bfe808) at Python/pythonrun.c:1260 #5 0x080d2678 in run_err_node (n=0x1bae6b10, filename=0x52bfea51 "killme.py", globals=0x1b99a9ac, locals=0x1b99a9ac, flags=0x52bfe808) at Python/pythonrun.c:1251 #6 0x080d2647 in PyRun_FileExFlags (fp=0x1b95b028, filename=0x52bfea51 "killme.py", start=257, globals=0x1b99a9ac, locals=0x1b99a9ac, closeit=1, flags=0x52bfe808) at Python/pythonrun.c:1242 #7 0x080d1752 in PyRun_SimpleFileExFlags (fp=0x1b95b028, filename=0x52bfea51 "killme.py", closeit=1, flags=0x52bfe808) at Python/pythonrun.c:859 #8 0x080d10d4 in PyRun_AnyFileExFlags (fp=0x1b95b028, filename=0x52bfea51 "killme.py", closeit=1, flags=0x52bfe808) at Python/pythonrun.c:663 #9 0x08055a01 in Py_Main (argc=2, argv=0x52bfe8f4) at Modules/main.c:484 #10 0x08054fc6 in main (argc=2, argv=0x52bfe8f4) at Modules/python.c:23 I have attached the killme.py file. It's rather ugly since it's code generated by omniORB's IDL compiler. I've made it as minimal as I can by cutting things out. If I cut any more out, the segfault no longer occurs. Don't worry that the various things it tries to import are not available for you -- it segfaults before it tries to actually execute anything. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053819&group_id=5470 From noreply at sourceforge.net Mon Oct 25 20:36:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 20:36:15 2004 Subject: [ python-Bugs-1053956 ] help() nor in tutorial index Message-ID: Bugs item #1053956, was opened at 2004-10-25 14:36 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053956&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Nobody/Anonymous (nobody) Summary: help() nor in tutorial index Initial Comment: According to Laura Creighton, help() is not in the tutorial index. If that's true, I think it ought to be fixed. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053956&group_id=5470 From noreply at sourceforge.net Mon Oct 25 22:27:22 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 22:27:25 2004 Subject: [ python-Bugs-1054041 ] Python doesn't exit with proper resultcode on SIGINT Message-ID: Bugs item #1054041, was opened at 2004-10-25 16:27 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054041&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 5 Submitted By: James Y Knight (foom) Assigned to: Nobody/Anonymous (nobody) Summary: Python doesn't exit with proper resultcode on SIGINT Initial Comment: If you kill a python process with SIGINT (e.g. control-c), it catches the signal, and raises a KeyboardInterrupt. If the KeyboardInterrupt propagates to the top level, python exits. However, it exits with a result of 1, not a result of SIGINT. This means that if you run python in a shell script, the script will not properly exit on C-c. When exiting because of a KeyboardInterrupt, python ought to reraise the SIGINT, as follows, so that the exit code is correct for the caller: signal(SIGINT, SIG_DFL); kill(getpid(), SIGINT); See also http://www.cons.org/cracauer/sigint.html for a more detailed discussion on the topic. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054041&group_id=5470 From noreply at sourceforge.net Mon Oct 25 23:59:22 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Oct 25 23:59:32 2004 Subject: [ python-Bugs-1053604 ] Unjustified SyntaxWarning Message-ID: Bugs item #1053604, was opened at 2004-10-25 05:06 Message generated for change (Comment added) made by jimjjewett You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053604&group_id=5470 Category: Parser/Compiler Group: None Status: Open Resolution: None Priority: 5 Submitted By: Markus Gritsch (markus_gritsch) Assigned to: Nobody/Anonymous (nobody) Summary: Unjustified SyntaxWarning Initial Comment: The following short script produces a "SyntaxWarning: name 'cPickle' is assigned to before global declaration", which is IMO not correct. mode = 'pickle' def main(): if mode == 'pickle': global cPickle import cPickle elif mode == 'socket': global cPickle, socket import cPickle, socket elif mode == 'sqlite': global sqlite import sqlite else: print 'Warning: Unknown mode.' def test(): print cPickle.__doc__ if __name__ == '__main__': main() test() ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-10-25 17:59 Message: Logged In: YES user_id=764593 The indenting was lost. If I guess correctly, cPickle is global for modes pickle and socket, but not otherwise. The problem is that it imports cpickle (for mode pickle) before running the global cpickle (for mode socket). It would be nice if the interpreter were smart enough to realize that the paths are exclusive, but I'm inclined to call it a feature request rather than a bug. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053604&group_id=5470 From noreply at sourceforge.net Tue Oct 26 00:32:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 00:32:09 2004 Subject: [ python-Bugs-1054139 ] serious string hashing error in 2.4b1 Message-ID: Bugs item #1054139, was opened at 2004-10-25 15:32 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Bob Halley (rthalley) Assigned to: Nobody/Anonymous (nobody) Summary: serious string hashing error in 2.4b1 Initial Comment: There is a serious hashing error in 2.4b1. I don't know if the error is confined to 64-bit systems, or is a general problem. The program attached to this report produces, as expected, this output when run with python 2.3.3: Python: 20303f0 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == -2784798555566127274 hash("DNSS") == 5809125768486327656 hash("DNSSE") == 5232635463381381892 hash("DNSSEC") == -7602892900506903802 When run with 2.4b1, I get the following output: Python: 20400b1 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == 8704052292078464 hash("DNSS") == 8704052292078464 hash("DNSSE") == 8704052292078464 hash("DNSSEC") == 8704052292078464 Traceback (most recent call last): File "foo.py", line 22, in ? assert hb == ha, 'hashes do not match!' AssertionError: hashes do not match! The way I discovered this was that dnspython's regression suite started failing because the string 'DNSSEC' constructed character-by-character was not being found in a dictionary which had a 'DNSSEC' key. I have not yet found the underlying bug; I was focussing on getting the info needed to demonstrate the bug first, since it's so serious. If I make any progress fixing it, I'll send a patch. /Bob ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 From noreply at sourceforge.net Tue Oct 26 01:08:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 01:08:15 2004 Subject: [ python-Bugs-1054139 ] serious string hashing error in 2.4b1 Message-ID: Bugs item #1054139, was opened at 2004-10-25 18:32 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None >Priority: 9 Submitted By: Bob Halley (rthalley) Assigned to: Nobody/Anonymous (nobody) Summary: serious string hashing error in 2.4b1 Initial Comment: There is a serious hashing error in 2.4b1. I don't know if the error is confined to 64-bit systems, or is a general problem. The program attached to this report produces, as expected, this output when run with python 2.3.3: Python: 20303f0 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == -2784798555566127274 hash("DNSS") == 5809125768486327656 hash("DNSSE") == 5232635463381381892 hash("DNSSEC") == -7602892900506903802 When run with 2.4b1, I get the following output: Python: 20400b1 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == 8704052292078464 hash("DNSS") == 8704052292078464 hash("DNSSE") == 8704052292078464 hash("DNSSEC") == 8704052292078464 Traceback (most recent call last): File "foo.py", line 22, in ? assert hb == ha, 'hashes do not match!' AssertionError: hashes do not match! The way I discovered this was that dnspython's regression suite started failing because the string 'DNSSEC' constructed character-by-character was not being found in a dictionary which had a 'DNSSEC' key. I have not yet found the underlying bug; I was focussing on getting the info needed to demonstrate the bug first, since it's so serious. If I make any progress fixing it, I'll send a patch. /Bob ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-25 19:08 Message: Logged In: YES user_id=31435 Boosted to highest priority. I don't get exactly the same numbers on my 32-bit box, but the problem exists there too. Suspect it has to do with new optimizations for string "+=". ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 From noreply at sourceforge.net Tue Oct 26 01:15:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 01:15:33 2004 Subject: [ python-Bugs-1054139 ] serious string hashing error in 2.4b1 Message-ID: Bugs item #1054139, was opened at 2004-10-25 18:32 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 9 Submitted By: Bob Halley (rthalley) Assigned to: Nobody/Anonymous (nobody) Summary: serious string hashing error in 2.4b1 Initial Comment: There is a serious hashing error in 2.4b1. I don't know if the error is confined to 64-bit systems, or is a general problem. The program attached to this report produces, as expected, this output when run with python 2.3.3: Python: 20303f0 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == -2784798555566127274 hash("DNSS") == 5809125768486327656 hash("DNSSE") == 5232635463381381892 hash("DNSSEC") == -7602892900506903802 When run with 2.4b1, I get the following output: Python: 20400b1 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == 8704052292078464 hash("DNSS") == 8704052292078464 hash("DNSSE") == 8704052292078464 hash("DNSSEC") == 8704052292078464 Traceback (most recent call last): File "foo.py", line 22, in ? assert hb == ha, 'hashes do not match!' AssertionError: hashes do not match! The way I discovered this was that dnspython's regression suite started failing because the string 'DNSSEC' constructed character-by-character was not being found in a dictionary which had a 'DNSSEC' key. I have not yet found the underlying bug; I was focussing on getting the info needed to demonstrate the bug first, since it's so serious. If I make any progress fixing it, I'll send a patch. /Bob ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-25 19:15 Message: Logged In: YES user_id=31435 FYI, I think ceval.c's string_concatenate() needs to reset ob_shash to -1 after it extends a string in-place. Alas, I can't make time to test that now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 19:08 Message: Logged In: YES user_id=31435 Boosted to highest priority. I don't get exactly the same numbers on my 32-bit box, but the problem exists there too. Suspect it has to do with new optimizations for string "+=". ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 From noreply at sourceforge.net Tue Oct 26 01:26:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 01:26:39 2004 Subject: [ python-Bugs-1054139 ] serious string hashing error in 2.4b1 Message-ID: Bugs item #1054139, was opened at 2004-10-25 15:32 Message generated for change (Comment added) made by rthalley You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 9 Submitted By: Bob Halley (rthalley) Assigned to: Nobody/Anonymous (nobody) Summary: serious string hashing error in 2.4b1 Initial Comment: There is a serious hashing error in 2.4b1. I don't know if the error is confined to 64-bit systems, or is a general problem. The program attached to this report produces, as expected, this output when run with python 2.3.3: Python: 20303f0 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == -2784798555566127274 hash("DNSS") == 5809125768486327656 hash("DNSSE") == 5232635463381381892 hash("DNSSEC") == -7602892900506903802 When run with 2.4b1, I get the following output: Python: 20400b1 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == 8704052292078464 hash("DNSS") == 8704052292078464 hash("DNSSE") == 8704052292078464 hash("DNSSEC") == 8704052292078464 Traceback (most recent call last): File "foo.py", line 22, in ? assert hb == ha, 'hashes do not match!' AssertionError: hashes do not match! The way I discovered this was that dnspython's regression suite started failing because the string 'DNSSEC' constructed character-by-character was not being found in a dictionary which had a 'DNSSEC' key. I have not yet found the underlying bug; I was focussing on getting the info needed to demonstrate the bug first, since it's so serious. If I make any progress fixing it, I'll send a patch. /Bob ---------------------------------------------------------------------- >Comment By: Bob Halley (rthalley) Date: 2004-10-25 16:26 Message: Logged In: YES user_id=671513 Patch #1054197 fixes this. The problem was that ceval.c wasn't invalidating the cached value. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 16:15 Message: Logged In: YES user_id=31435 FYI, I think ceval.c's string_concatenate() needs to reset ob_shash to -1 after it extends a string in-place. Alas, I can't make time to test that now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 16:08 Message: Logged In: YES user_id=31435 Boosted to highest priority. I don't get exactly the same numbers on my 32-bit box, but the problem exists there too. Suspect it has to do with new optimizations for string "+=". ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 From noreply at sourceforge.net Tue Oct 26 02:10:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 02:11:17 2004 Subject: [ python-Bugs-1054139 ] serious string hashing error in 2.4b1 Message-ID: Bugs item #1054139, was opened at 2004-10-25 17:32 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 9 Submitted By: Bob Halley (rthalley) >Assigned to: Raymond Hettinger (rhettinger) Summary: serious string hashing error in 2.4b1 Initial Comment: There is a serious hashing error in 2.4b1. I don't know if the error is confined to 64-bit systems, or is a general problem. The program attached to this report produces, as expected, this output when run with python 2.3.3: Python: 20303f0 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == -2784798555566127274 hash("DNSS") == 5809125768486327656 hash("DNSSE") == 5232635463381381892 hash("DNSSEC") == -7602892900506903802 When run with 2.4b1, I get the following output: Python: 20400b1 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == 8704052292078464 hash("DNSS") == 8704052292078464 hash("DNSSE") == 8704052292078464 hash("DNSSEC") == 8704052292078464 Traceback (most recent call last): File "foo.py", line 22, in ? assert hb == ha, 'hashes do not match!' AssertionError: hashes do not match! The way I discovered this was that dnspython's regression suite started failing because the string 'DNSSEC' constructed character-by-character was not being found in a dictionary which had a 'DNSSEC' key. I have not yet found the underlying bug; I was focussing on getting the info needed to demonstrate the bug first, since it's so serious. If I make any progress fixing it, I'll send a patch. /Bob ---------------------------------------------------------------------- Comment By: Bob Halley (rthalley) Date: 2004-10-25 18:26 Message: Logged In: YES user_id=671513 Patch #1054197 fixes this. The problem was that ceval.c wasn't invalidating the cached value. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 18:15 Message: Logged In: YES user_id=31435 FYI, I think ceval.c's string_concatenate() needs to reset ob_shash to -1 after it extends a string in-place. Alas, I can't make time to test that now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 18:08 Message: Logged In: YES user_id=31435 Boosted to highest priority. I don't get exactly the same numbers on my 32-bit box, but the problem exists there too. Suspect it has to do with new optimizations for string "+=". ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 From noreply at sourceforge.net Tue Oct 26 03:50:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 03:50:30 2004 Subject: [ python-Bugs-1053604 ] Unjustified SyntaxWarning Message-ID: Bugs item #1053604, was opened at 2004-10-25 04:06 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053604&group_id=5470 Category: Parser/Compiler Group: None >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Markus Gritsch (markus_gritsch) >Assigned to: Raymond Hettinger (rhettinger) Summary: Unjustified SyntaxWarning Initial Comment: The following short script produces a "SyntaxWarning: name 'cPickle' is assigned to before global declaration", which is IMO not correct. mode = 'pickle' def main(): if mode == 'pickle': global cPickle import cPickle elif mode == 'socket': global cPickle, socket import cPickle, socket elif mode == 'sqlite': global sqlite import sqlite else: print 'Warning: Unknown mode.' def test(): print cPickle.__doc__ if __name__ == '__main__': main() test() ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-25 20:50 Message: Logged In: YES user_id=80475 Sorry, this isn't a bug. Per the language reference, the global statement is not executed like other python statements. Instead, it is a parser directive. Hence, it doesn't matter whether the globals are wrapped inside if blocks. The presence of the first occurrence of cPickle is sufficient to define so that the subsequently parsed (not executed) global directive cannot work. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-10-25 16:59 Message: Logged In: YES user_id=764593 The indenting was lost. If I guess correctly, cPickle is global for modes pickle and socket, but not otherwise. The problem is that it imports cpickle (for mode pickle) before running the global cpickle (for mode socket). It would be nice if the interpreter were smart enough to realize that the paths are exclusive, but I'm inclined to call it a feature request rather than a bug. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053604&group_id=5470 From noreply at sourceforge.net Tue Oct 26 03:53:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 03:54:01 2004 Subject: [ python-Bugs-1054139 ] serious string hashing error in 2.4b1 Message-ID: Bugs item #1054139, was opened at 2004-10-25 17:32 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 9 Submitted By: Bob Halley (rthalley) Assigned to: Raymond Hettinger (rhettinger) Summary: serious string hashing error in 2.4b1 Initial Comment: There is a serious hashing error in 2.4b1. I don't know if the error is confined to 64-bit systems, or is a general problem. The program attached to this report produces, as expected, this output when run with python 2.3.3: Python: 20303f0 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == -2784798555566127274 hash("DNSS") == 5809125768486327656 hash("DNSSE") == 5232635463381381892 hash("DNSSEC") == -7602892900506903802 When run with 2.4b1, I get the following output: Python: 20400b1 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == 8704052292078464 hash("DNSS") == 8704052292078464 hash("DNSSE") == 8704052292078464 hash("DNSSEC") == 8704052292078464 Traceback (most recent call last): File "foo.py", line 22, in ? assert hb == ha, 'hashes do not match!' AssertionError: hashes do not match! The way I discovered this was that dnspython's regression suite started failing because the string 'DNSSEC' constructed character-by-character was not being found in a dictionary which had a 'DNSSEC' key. I have not yet found the underlying bug; I was focussing on getting the info needed to demonstrate the bug first, since it's so serious. If I make any progress fixing it, I'll send a patch. /Bob ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-25 20:53 Message: Logged In: YES user_id=80475 Fixed. See: Objects/stringobject.c 2.226 Lib/test/string_tests.py 1.42 Thanks for the highly specific, timely bug report. ---------------------------------------------------------------------- Comment By: Bob Halley (rthalley) Date: 2004-10-25 18:26 Message: Logged In: YES user_id=671513 Patch #1054197 fixes this. The problem was that ceval.c wasn't invalidating the cached value. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 18:15 Message: Logged In: YES user_id=31435 FYI, I think ceval.c's string_concatenate() needs to reset ob_shash to -1 after it extends a string in-place. Alas, I can't make time to test that now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 18:08 Message: Logged In: YES user_id=31435 Boosted to highest priority. I don't get exactly the same numbers on my 32-bit box, but the problem exists there too. Suspect it has to do with new optimizations for string "+=". ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 From noreply at sourceforge.net Tue Oct 26 04:50:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 04:51:15 2004 Subject: [ python-Bugs-1053819 ] Segfault in tuple_of_constants Message-ID: Bugs item #1053819, was opened at 2004-10-25 11:20 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053819&group_id=5470 Category: Parser/Compiler Group: Python 2.4 Status: Open Resolution: None >Priority: 9 Submitted By: Duncan Grisby (dgrisby) >Assigned to: Raymond Hettinger (rhettinger) Summary: Segfault in tuple_of_constants Initial Comment: 2.4 beta 1 segfaults when compiling a file. The same file works fine on 2.3 and earlier. I have tracked the problem down to the tuple_of_constants function, called by optimize_code. Using valgrind (having turned off pymalloc), I see that the consts list seems to be uninitialised. See the following trace: $ valgrind --tool=memcheck --db-attach=yes ~/inst/bin/python killme.py ==13759== Memcheck, a memory error detector for x86-linux. ==13759== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward et al. ==13759== Using valgrind-2.2.0, a program supervision framework for x86-linux. ==13759== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward et al. ==13759== For more details, rerun with: -v ==13759== ==13759== Invalid read of size 4 ==13759== at 0x80B3BF5: tuple_of_constants (compile.c:422) ==13759== by 0x80B433B: optimize_code (compile.c:607) ==13759== by 0x80BD3F2: jcompile (compile.c:4988) ==13759== by 0x80BD0B2: PyNode_CompileFlags (compile.c:4894) ==13759== Address 0x1BF100A0 is 12 bytes after a block of size 36 alloc'd ==13759== at 0x1B904A80: malloc (vg_replace_malloc.c:131) ==13759== by 0x80D8A4C: _PyObject_GC_Malloc (gcmodule.c:1183) ==13759== by 0x80D8B31: _PyObject_GC_NewVar (gcmodule.c:1214) ==13759== by 0x80854FF: PyTuple_New (tupleobject.c:68) ==13759== ==13759== ---- Attach to debugger ? --- [Return/N/n/Y/y/C/c] ---- y starting debugger ==13759== starting debugger with cmd: /usr/bin/gdb -nw /proc/13761/fd/821 13761 GNU gdb Red Hat Linux (6.0post-0.20040223.19rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1". Attaching to program: /proc/13761/fd/821, process 13761 0x080b3bf5 in tuple_of_constants (codestr=0x1bf1ff90 "d", n=1, consts=0x1badc9c4) at Python/compile.c:422 422 constant = PyList_GET_ITEM(consts, arg); (gdb) bt #0 0x080b3bf5 in tuple_of_constants (codestr=0x1bf1ff90 "d", n=1, consts=0x1badc9c4) at Python/compile.c:422 #1 0x080b433c in optimize_code (code=0x1becb3e8, consts=0x1badc9c4, names=0x1bf1ed5c, lineno_obj=0x1be7cd90) at Python/compile.c:607 #2 0x080bd3f3 in jcompile (n=0x1bae6b10, filename=0x52bfea51 "killme.py", base=0x0, flags=0x52bfe808) at Python/compile.c:4988 #3 0x080bd0b3 in PyNode_CompileFlags (n=0x1bae6b10, filename=0x52bfea51 "killme.py", flags=0x52bfe808) at Python/compile.c:4894 #4 0x080d269a in run_node (n=0x1bae6b10, filename=0x52bfea51 "killme.py", globals=0x1b99a9ac, locals=0x1b99a9ac, flags=0x52bfe808) at Python/pythonrun.c:1260 #5 0x080d2678 in run_err_node (n=0x1bae6b10, filename=0x52bfea51 "killme.py", globals=0x1b99a9ac, locals=0x1b99a9ac, flags=0x52bfe808) at Python/pythonrun.c:1251 #6 0x080d2647 in PyRun_FileExFlags (fp=0x1b95b028, filename=0x52bfea51 "killme.py", start=257, globals=0x1b99a9ac, locals=0x1b99a9ac, closeit=1, flags=0x52bfe808) at Python/pythonrun.c:1242 #7 0x080d1752 in PyRun_SimpleFileExFlags (fp=0x1b95b028, filename=0x52bfea51 "killme.py", closeit=1, flags=0x52bfe808) at Python/pythonrun.c:859 #8 0x080d10d4 in PyRun_AnyFileExFlags (fp=0x1b95b028, filename=0x52bfea51 "killme.py", closeit=1, flags=0x52bfe808) at Python/pythonrun.c:663 #9 0x08055a01 in Py_Main (argc=2, argv=0x52bfe8f4) at Modules/main.c:484 #10 0x08054fc6 in main (argc=2, argv=0x52bfe8f4) at Modules/python.c:23 I have attached the killme.py file. It's rather ugly since it's code generated by omniORB's IDL compiler. I've made it as minimal as I can by cutting things out. If I cut any more out, the segfault no longer occurs. Don't worry that the various things it tries to import are not available for you -- it segfaults before it tries to actually execute anything. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-25 22:50 Message: Logged In: YES user_id=31435 Confirmed the segfault with current Python CVS. Boosted priority to the maximum. Assigned to Raymond, since he probably knows most about this. Dies in tuple_of_constants(), n=1, i=0, on the incref here: for (i=0 ; i Bugs item #1050828, was opened at 2004-10-20 12:48 Message generated for change (Comment added) made by facundobatista You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Ripton (dripton) Assigned to: Tim Peters (tim_one) Summary: reindent.py strips execute permission on Unix Initial Comment: Tools/scripts/reindent.py strips execute permission from its target file. If the tool detects that changes need to be made, then the target file is renamed to its original name + '.bak', then a new file is created with the original file's name and its whitespace-modified contents. No manipulation of the new file's permissions is done, so it ends up with the default for the user's umask. This makes reindent.py annoying to run in an automated fashion, if you rely on execute permission. The fix is trivial, except for portability. Here's the Posix-only version: + if os.name == "posix": + mode = os.stat(bak).st_mode + os.chmod(file, mode) Patch available upon request. ---------------------------------------------------------------------- >Comment By: Facundo Batista (facundobatista) Date: 2004-10-25 23:57 Message: Logged In: YES user_id=752496 Attaching the file. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-24 18:20 Message: Logged In: YES user_id=752496 Instead of renaming the file and creating a new one, just copied the original and that's all (the target file is open with write mode, the content replaced, but the permissions stay). A patch is attached. Tim, I'm assigning this to you to doublecheck (not found unit tests for this script). . Facundo ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 From noreply at sourceforge.net Tue Oct 26 05:01:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 05:01:26 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 07:04 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 4 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-10-25 20:01 Message: Logged In: YES user_id=357491 OK, unless someone speaks up about this patch I will apply it sometime this week (soonest being 2004-10-26 in the evening PDT). ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-19 13:10 Message: Logged In: YES user_id=357491 Nick is seeing things the way I thought they are supposed to be; how can ending on a Monday make any difference for %U or %W? Both start at after the beginning of the week, right? Well, as George was pointing out, it shifts everything by a full week depending on the directive when the year starts on a Monday. So under %U week 1 doesn't start until the first Sunday, while under %W week 1 starts on Jan 1. So the issue is when a year starts and end on a Monday, we are looking at Monday, but %U is used. So the new patch attempts to handle this case. I am now testing against the following dates: test_helper((1901, 1, 3), "week 0") test_helper((1901, 1, 8), "common case") test_helper((1901, 1, 13), "day on Sunday") test_helper((1901, 1, 14), "day on Monday") test_helper((1905, 1, 1), "Jan 1 on Sunday") test_helper((1906, 1, 1), "Jan 1 on Monday") test_helper((1906, 1, 7), "first Sunday in a year starting on Monday") test_helper((1905, 12, 31), "Dec 31 on Sunday") test_helper((1906, 12, 31), "Dec 31 on Monday") test_helper((2008, 12, 29), "Monday in the last week of the year") test_helper((2008, 12, 22), "Monday in the second-to-last week of the " "year") test_helper((1978, 10, 23), "randomly chosen date") test_helper((2004, 12, 18), "randomly chosen date") test_helper((1978, 10, 23), "year starting and ending on Monday while " "date not on Sunday or Monday") test_helper((1917, 12, 17), "year starting and ending on Monday with " "a Monday not at the beginning or end " "of the year") test_helper((1917, 12, 31), "Dec 31 on Monday with year starting and " "ending on Monday") ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 06:18 Message: Logged In: YES user_id=1038590 Under %W, the first week contains a full 7 days, but it only contains 6 days under %U. Either way, we end up with a week 53 - it contains 1 day for %W, but 2 days for %U. In both cases, December 31st is the Monday of week 53. Unless I'm misunderstanding how this is meant to work, and Week 1 is always the first full week of the year, with an optional Week 0 before it (which would always have fewer than 7 days, and doesn't exist at all if the year and the week start on the same day). If I *am* misunderstanding, then that's the bug in strptime - it currently works in accordance with my understanding. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-19 05:18 Message: Logged In: YES user_id=671362 > December 31 should be week 53 under %U as well. I doubt it. Year 1917 begins on Monday and ends on Monday. So "%U"(Sunday as the first day of the week) should return 52 and "W"(Monday as the first day of the week) should return 53. $ cal 1 1917 January 1917 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 $ cal 12 1917 December 1917 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:50 Message: Logged In: YES user_id=1038590 Any bug is in datetime.date, not strptime. I tried datetime.date(1917, 12, 31).strftime("%Y %U %w") and saw Brett's incorrect calculation of the week. ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %W %w')" 1917 53 1 ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %U %w')" 1917 52 1 December 31 should be week 53 under %U as well. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:42 Message: Logged In: YES user_id=1038590 I forgot to add that I'm on Linux (Fedora Core 3 Test 1 + misc updates) ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:39 Message: Logged In: YES user_id=1038590 Taking out the "add one to the week" condition, I get the following for 1906 and 1917: ./python -c "import time; print time.strptime('1906 53 1', '%Y %W %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1906 53 1', '%Y %U %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %W %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %U %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) So, I'm more than a little curious about the details of the "bug" that was being fixed. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:03 Message: Logged In: YES user_id=1038590 Scratch the last comment - I missed the wee_of_year adjustment at the top of the function. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 02:57 Message: Logged In: YES user_id=1038590 The calculation of 'preceeding_days' looks incorrect. It assumes that the week starts on day 0 - it needs to adjust for when "week_of year_start" is 6. ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-19 00:31 Message: Logged In: YES user_id=552329 Patch works for that case, yes. However... [from python-dev] > the test case I was using that triggered > the need for that is 1906-12-31 > which is a Monday but changes what week > it is based on whether you use U or W. > which makes no sense since both U and W > should consider it the same week. > Had the same result for 1917-12-31. Something's still not right here (this is with the patch): >>> time.strptime("1917 51 1", "%Y %U %w") (1917, 12, 17, 0, 0, 0, 0, 351, -1) >>> time.strptime("1917 52 1", "%Y %U %w") (1917, 12, 31, 0, 0, 0, 0, 365, -1) 1917 both started and ended on a Monday, so there are 53 U weeks, right? (where the last 'week' is only one day). So 31/12/1917 should be U=53, W=52 ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 11:45 Message: Logged In: YES user_id=357491 Reopening to deal with the error Tony caught. Tony, can you apply the included patch and verify it works for you okay? I added another if to track back a week if the calculation gets pushed past 366 days. That does fix your error. And if you can think of any other edge cases let me know. I think that should cover everything. And yes, the example for the OP is wrong:: >>> datetime.date(2004, 10, 11).strftime("%Y %U %w") '2004 41 1' ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-17 20:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-17 18:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-17 16:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Tue Oct 26 05:19:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 05:19:47 2004 Subject: [ python-Bugs-1050828 ] reindent.py strips execute permission on Unix Message-ID: Bugs item #1050828, was opened at 2004-10-20 11:48 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Ripton (dripton) >Assigned to: Facundo Batista (facundobatista) Summary: reindent.py strips execute permission on Unix Initial Comment: Tools/scripts/reindent.py strips execute permission from its target file. If the tool detects that changes need to be made, then the target file is renamed to its original name + '.bak', then a new file is created with the original file's name and its whitespace-modified contents. No manipulation of the new file's permissions is done, so it ends up with the default for the user's umask. This makes reindent.py annoying to run in an automated fashion, if you rely on execute permission. The fix is trivial, except for portability. Here's the Posix-only version: + if os.name == "posix": + mode = os.stat(bak).st_mode + os.chmod(file, mode) Patch available upon request. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-25 23:19 Message: Logged In: YES user_id=31435 Sorry, I don't normally run on Unix, so I have no way to tell whether this fixes whatever the complaint is. Offhand it doesn't seem right to create a .bak file with default permissions either. For example, if only the user had permission to get at the original, letting the .bak file have broader permissions seems wrong too. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-25 22:57 Message: Logged In: YES user_id=752496 Attaching the file. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-24 17:20 Message: Logged In: YES user_id=752496 Instead of renaming the file and creating a new one, just copied the original and that's all (the target file is open with write mode, the content replaced, but the permissions stay). A patch is attached. Tim, I'm assigning this to you to doublecheck (not found unit tests for this script). . Facundo ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 From noreply at sourceforge.net Tue Oct 26 05:54:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 05:54:31 2004 Subject: [ python-Bugs-1053956 ] help() nor in tutorial index Message-ID: Bugs item #1053956, was opened at 2004-10-25 13:36 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053956&group_id=5470 Category: Documentation Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Nobody/Anonymous (nobody) Summary: help() nor in tutorial index Initial Comment: According to Laura Creighton, help() is not in the tutorial index. If that's true, I think it ought to be fixed. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-25 22:54 Message: Logged In: YES user_id=80475 Fixed. See Doc/tut/tut.tex 1.256. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053956&group_id=5470 From noreply at sourceforge.net Tue Oct 26 06:13:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 06:13:42 2004 Subject: [ python-Bugs-834351 ] Mouse wheel crashes program Message-ID: Bugs item #834351, was opened at 2003-11-02 13:37 Message generated for change (Comment added) made by johnfouhy You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=834351&group_id=5470 Category: Tkinter Group: Python 2.2.3 Status: Open Resolution: None Priority: 5 Submitted By: Gary Richardson (garyrxx) Assigned to: Martin v. L?wis (loewis) Summary: Mouse wheel crashes program Initial Comment: The following program (by Michael Peuser) crashes as soon as the mouse wheel is moved. See my post to c.l.p. on Oct 29. Gary Richardson #------------------------- from Tkinter import * def _onMouseWheel(event): print event root = Tk() root.bind('<MouseWheel>',_onMouseWheel) root.mainloop() ---------------------------------------------------------------------- Comment By: John Fouhy (johnfouhy) Date: 2004-10-26 17:13 Message: Logged In: YES user_id=1084032 I can reproduce this bug. I am running Python 2.3.4 on WinXP SP2. tk84.dll is version 8.4.3; tcl84.dll is also 8.4.3. The mousewheel works correctly with a Tkinter.Text object with no additional bindings. ---------------------------------------------------------------------- Comment By: Jon Ashley (ash101) Date: 2003-12-05 06:02 Message: Logged In: YES user_id=923903 Happens for me too, on Win2K with python 2.3.2. The fault is in TCL84.DLL at an offset of 1002a58c, if that means anything to anyone. ---------------------------------------------------------------------- Comment By: Gary Richardson (garyrxx) Date: 2003-11-03 15:42 Message: Logged In: YES user_id=899035 I'm using Win98SE. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-11-03 07:37 Message: Logged In: YES user_id=21627 What operating system are you using? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=834351&group_id=5470 From noreply at sourceforge.net Tue Oct 26 07:48:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 07:48:45 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 07:04 Message generated for change (Settings changed) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None >Priority: 7 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-25 20:01 Message: Logged In: YES user_id=357491 OK, unless someone speaks up about this patch I will apply it sometime this week (soonest being 2004-10-26 in the evening PDT). ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-19 13:10 Message: Logged In: YES user_id=357491 Nick is seeing things the way I thought they are supposed to be; how can ending on a Monday make any difference for %U or %W? Both start at after the beginning of the week, right? Well, as George was pointing out, it shifts everything by a full week depending on the directive when the year starts on a Monday. So under %U week 1 doesn't start until the first Sunday, while under %W week 1 starts on Jan 1. So the issue is when a year starts and end on a Monday, we are looking at Monday, but %U is used. So the new patch attempts to handle this case. I am now testing against the following dates: test_helper((1901, 1, 3), "week 0") test_helper((1901, 1, 8), "common case") test_helper((1901, 1, 13), "day on Sunday") test_helper((1901, 1, 14), "day on Monday") test_helper((1905, 1, 1), "Jan 1 on Sunday") test_helper((1906, 1, 1), "Jan 1 on Monday") test_helper((1906, 1, 7), "first Sunday in a year starting on Monday") test_helper((1905, 12, 31), "Dec 31 on Sunday") test_helper((1906, 12, 31), "Dec 31 on Monday") test_helper((2008, 12, 29), "Monday in the last week of the year") test_helper((2008, 12, 22), "Monday in the second-to-last week of the " "year") test_helper((1978, 10, 23), "randomly chosen date") test_helper((2004, 12, 18), "randomly chosen date") test_helper((1978, 10, 23), "year starting and ending on Monday while " "date not on Sunday or Monday") test_helper((1917, 12, 17), "year starting and ending on Monday with " "a Monday not at the beginning or end " "of the year") test_helper((1917, 12, 31), "Dec 31 on Monday with year starting and " "ending on Monday") ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 06:18 Message: Logged In: YES user_id=1038590 Under %W, the first week contains a full 7 days, but it only contains 6 days under %U. Either way, we end up with a week 53 - it contains 1 day for %W, but 2 days for %U. In both cases, December 31st is the Monday of week 53. Unless I'm misunderstanding how this is meant to work, and Week 1 is always the first full week of the year, with an optional Week 0 before it (which would always have fewer than 7 days, and doesn't exist at all if the year and the week start on the same day). If I *am* misunderstanding, then that's the bug in strptime - it currently works in accordance with my understanding. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-19 05:18 Message: Logged In: YES user_id=671362 > December 31 should be week 53 under %U as well. I doubt it. Year 1917 begins on Monday and ends on Monday. So "%U"(Sunday as the first day of the week) should return 52 and "W"(Monday as the first day of the week) should return 53. $ cal 1 1917 January 1917 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 $ cal 12 1917 December 1917 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:50 Message: Logged In: YES user_id=1038590 Any bug is in datetime.date, not strptime. I tried datetime.date(1917, 12, 31).strftime("%Y %U %w") and saw Brett's incorrect calculation of the week. ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %W %w')" 1917 53 1 ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %U %w')" 1917 52 1 December 31 should be week 53 under %U as well. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:42 Message: Logged In: YES user_id=1038590 I forgot to add that I'm on Linux (Fedora Core 3 Test 1 + misc updates) ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:39 Message: Logged In: YES user_id=1038590 Taking out the "add one to the week" condition, I get the following for 1906 and 1917: ./python -c "import time; print time.strptime('1906 53 1', '%Y %W %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1906 53 1', '%Y %U %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %W %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %U %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) So, I'm more than a little curious about the details of the "bug" that was being fixed. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:03 Message: Logged In: YES user_id=1038590 Scratch the last comment - I missed the wee_of_year adjustment at the top of the function. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 02:57 Message: Logged In: YES user_id=1038590 The calculation of 'preceeding_days' looks incorrect. It assumes that the week starts on day 0 - it needs to adjust for when "week_of year_start" is 6. ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-19 00:31 Message: Logged In: YES user_id=552329 Patch works for that case, yes. However... [from python-dev] > the test case I was using that triggered > the need for that is 1906-12-31 > which is a Monday but changes what week > it is based on whether you use U or W. > which makes no sense since both U and W > should consider it the same week. > Had the same result for 1917-12-31. Something's still not right here (this is with the patch): >>> time.strptime("1917 51 1", "%Y %U %w") (1917, 12, 17, 0, 0, 0, 0, 351, -1) >>> time.strptime("1917 52 1", "%Y %U %w") (1917, 12, 31, 0, 0, 0, 0, 365, -1) 1917 both started and ended on a Monday, so there are 53 U weeks, right? (where the last 'week' is only one day). So 31/12/1917 should be U=53, W=52 ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 11:45 Message: Logged In: YES user_id=357491 Reopening to deal with the error Tony caught. Tony, can you apply the included patch and verify it works for you okay? I added another if to track back a week if the calculation gets pushed past 366 days. That does fix your error. And if you can think of any other edge cases let me know. I think that should cover everything. And yes, the example for the OP is wrong:: >>> datetime.date(2004, 10, 11).strftime("%Y %U %w") '2004 41 1' ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-17 20:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-17 18:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-17 16:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Tue Oct 26 09:54:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 09:54:55 2004 Subject: [ python-Bugs-1053819 ] Segfault in tuple_of_constants Message-ID: Bugs item #1053819, was opened at 2004-10-25 10:20 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053819&group_id=5470 Category: Parser/Compiler Group: Python 2.4 Status: Open Resolution: None Priority: 9 Submitted By: Duncan Grisby (dgrisby) Assigned to: Raymond Hettinger (rhettinger) Summary: Segfault in tuple_of_constants Initial Comment: 2.4 beta 1 segfaults when compiling a file. The same file works fine on 2.3 and earlier. I have tracked the problem down to the tuple_of_constants function, called by optimize_code. Using valgrind (having turned off pymalloc), I see that the consts list seems to be uninitialised. See the following trace: $ valgrind --tool=memcheck --db-attach=yes ~/inst/bin/python killme.py ==13759== Memcheck, a memory error detector for x86-linux. ==13759== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward et al. ==13759== Using valgrind-2.2.0, a program supervision framework for x86-linux. ==13759== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward et al. ==13759== For more details, rerun with: -v ==13759== ==13759== Invalid read of size 4 ==13759== at 0x80B3BF5: tuple_of_constants (compile.c:422) ==13759== by 0x80B433B: optimize_code (compile.c:607) ==13759== by 0x80BD3F2: jcompile (compile.c:4988) ==13759== by 0x80BD0B2: PyNode_CompileFlags (compile.c:4894) ==13759== Address 0x1BF100A0 is 12 bytes after a block of size 36 alloc'd ==13759== at 0x1B904A80: malloc (vg_replace_malloc.c:131) ==13759== by 0x80D8A4C: _PyObject_GC_Malloc (gcmodule.c:1183) ==13759== by 0x80D8B31: _PyObject_GC_NewVar (gcmodule.c:1214) ==13759== by 0x80854FF: PyTuple_New (tupleobject.c:68) ==13759== ==13759== ---- Attach to debugger ? --- [Return/N/n/Y/y/C/c] ---- y starting debugger ==13759== starting debugger with cmd: /usr/bin/gdb -nw /proc/13761/fd/821 13761 GNU gdb Red Hat Linux (6.0post-0.20040223.19rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1". Attaching to program: /proc/13761/fd/821, process 13761 0x080b3bf5 in tuple_of_constants (codestr=0x1bf1ff90 "d", n=1, consts=0x1badc9c4) at Python/compile.c:422 422 constant = PyList_GET_ITEM(consts, arg); (gdb) bt #0 0x080b3bf5 in tuple_of_constants (codestr=0x1bf1ff90 "d", n=1, consts=0x1badc9c4) at Python/compile.c:422 #1 0x080b433c in optimize_code (code=0x1becb3e8, consts=0x1badc9c4, names=0x1bf1ed5c, lineno_obj=0x1be7cd90) at Python/compile.c:607 #2 0x080bd3f3 in jcompile (n=0x1bae6b10, filename=0x52bfea51 "killme.py", base=0x0, flags=0x52bfe808) at Python/compile.c:4988 #3 0x080bd0b3 in PyNode_CompileFlags (n=0x1bae6b10, filename=0x52bfea51 "killme.py", flags=0x52bfe808) at Python/compile.c:4894 #4 0x080d269a in run_node (n=0x1bae6b10, filename=0x52bfea51 "killme.py", globals=0x1b99a9ac, locals=0x1b99a9ac, flags=0x52bfe808) at Python/pythonrun.c:1260 #5 0x080d2678 in run_err_node (n=0x1bae6b10, filename=0x52bfea51 "killme.py", globals=0x1b99a9ac, locals=0x1b99a9ac, flags=0x52bfe808) at Python/pythonrun.c:1251 #6 0x080d2647 in PyRun_FileExFlags (fp=0x1b95b028, filename=0x52bfea51 "killme.py", start=257, globals=0x1b99a9ac, locals=0x1b99a9ac, closeit=1, flags=0x52bfe808) at Python/pythonrun.c:1242 #7 0x080d1752 in PyRun_SimpleFileExFlags (fp=0x1b95b028, filename=0x52bfea51 "killme.py", closeit=1, flags=0x52bfe808) at Python/pythonrun.c:859 #8 0x080d10d4 in PyRun_AnyFileExFlags (fp=0x1b95b028, filename=0x52bfea51 "killme.py", closeit=1, flags=0x52bfe808) at Python/pythonrun.c:663 #9 0x08055a01 in Py_Main (argc=2, argv=0x52bfe8f4) at Modules/main.c:484 #10 0x08054fc6 in main (argc=2, argv=0x52bfe8f4) at Modules/python.c:23 I have attached the killme.py file. It's rather ugly since it's code generated by omniORB's IDL compiler. I've made it as minimal as I can by cutting things out. If I cut any more out, the segfault no longer occurs. Don't worry that the various things it tries to import are not available for you -- it segfaults before it tries to actually execute anything. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-26 02:54 Message: Logged In: YES user_id=80475 The bug is intrinsic to the current approach which misindentifies certain code sequences as looking like a tuple of constants. Attaching a simplified example. Unless I find a straight-forward fix in the next day or so, will have to rip out the tuple of constant code entirely. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 21:50 Message: Logged In: YES user_id=31435 Confirmed the segfault with current Python CVS. Boosted priority to the maximum. Assigned to Raymond, since he probably knows most about this. Dies in tuple_of_constants(), n=1, i=0, on the incref here: for (i=0 ; i Bugs item #1053819, was opened at 2004-10-25 10:20 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053819&group_id=5470 Category: Parser/Compiler Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 9 Submitted By: Duncan Grisby (dgrisby) Assigned to: Raymond Hettinger (rhettinger) Summary: Segfault in tuple_of_constants Initial Comment: 2.4 beta 1 segfaults when compiling a file. The same file works fine on 2.3 and earlier. I have tracked the problem down to the tuple_of_constants function, called by optimize_code. Using valgrind (having turned off pymalloc), I see that the consts list seems to be uninitialised. See the following trace: $ valgrind --tool=memcheck --db-attach=yes ~/inst/bin/python killme.py ==13759== Memcheck, a memory error detector for x86-linux. ==13759== Copyright (C) 2002-2004, and GNU GPL'd, by Julian Seward et al. ==13759== Using valgrind-2.2.0, a program supervision framework for x86-linux. ==13759== Copyright (C) 2000-2004, and GNU GPL'd, by Julian Seward et al. ==13759== For more details, rerun with: -v ==13759== ==13759== Invalid read of size 4 ==13759== at 0x80B3BF5: tuple_of_constants (compile.c:422) ==13759== by 0x80B433B: optimize_code (compile.c:607) ==13759== by 0x80BD3F2: jcompile (compile.c:4988) ==13759== by 0x80BD0B2: PyNode_CompileFlags (compile.c:4894) ==13759== Address 0x1BF100A0 is 12 bytes after a block of size 36 alloc'd ==13759== at 0x1B904A80: malloc (vg_replace_malloc.c:131) ==13759== by 0x80D8A4C: _PyObject_GC_Malloc (gcmodule.c:1183) ==13759== by 0x80D8B31: _PyObject_GC_NewVar (gcmodule.c:1214) ==13759== by 0x80854FF: PyTuple_New (tupleobject.c:68) ==13759== ==13759== ---- Attach to debugger ? --- [Return/N/n/Y/y/C/c] ---- y starting debugger ==13759== starting debugger with cmd: /usr/bin/gdb -nw /proc/13761/fd/821 13761 GNU gdb Red Hat Linux (6.0post-0.20040223.19rh) Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1". Attaching to program: /proc/13761/fd/821, process 13761 0x080b3bf5 in tuple_of_constants (codestr=0x1bf1ff90 "d", n=1, consts=0x1badc9c4) at Python/compile.c:422 422 constant = PyList_GET_ITEM(consts, arg); (gdb) bt #0 0x080b3bf5 in tuple_of_constants (codestr=0x1bf1ff90 "d", n=1, consts=0x1badc9c4) at Python/compile.c:422 #1 0x080b433c in optimize_code (code=0x1becb3e8, consts=0x1badc9c4, names=0x1bf1ed5c, lineno_obj=0x1be7cd90) at Python/compile.c:607 #2 0x080bd3f3 in jcompile (n=0x1bae6b10, filename=0x52bfea51 "killme.py", base=0x0, flags=0x52bfe808) at Python/compile.c:4988 #3 0x080bd0b3 in PyNode_CompileFlags (n=0x1bae6b10, filename=0x52bfea51 "killme.py", flags=0x52bfe808) at Python/compile.c:4894 #4 0x080d269a in run_node (n=0x1bae6b10, filename=0x52bfea51 "killme.py", globals=0x1b99a9ac, locals=0x1b99a9ac, flags=0x52bfe808) at Python/pythonrun.c:1260 #5 0x080d2678 in run_err_node (n=0x1bae6b10, filename=0x52bfea51 "killme.py", globals=0x1b99a9ac, locals=0x1b99a9ac, flags=0x52bfe808) at Python/pythonrun.c:1251 #6 0x080d2647 in PyRun_FileExFlags (fp=0x1b95b028, filename=0x52bfea51 "killme.py", start=257, globals=0x1b99a9ac, locals=0x1b99a9ac, closeit=1, flags=0x52bfe808) at Python/pythonrun.c:1242 #7 0x080d1752 in PyRun_SimpleFileExFlags (fp=0x1b95b028, filename=0x52bfea51 "killme.py", closeit=1, flags=0x52bfe808) at Python/pythonrun.c:859 #8 0x080d10d4 in PyRun_AnyFileExFlags (fp=0x1b95b028, filename=0x52bfea51 "killme.py", closeit=1, flags=0x52bfe808) at Python/pythonrun.c:663 #9 0x08055a01 in Py_Main (argc=2, argv=0x52bfe8f4) at Modules/main.c:484 #10 0x08054fc6 in main (argc=2, argv=0x52bfe8f4) at Modules/python.c:23 I have attached the killme.py file. It's rather ugly since it's code generated by omniORB's IDL compiler. I've made it as minimal as I can by cutting things out. If I cut any more out, the segfault no longer occurs. Don't worry that the various things it tries to import are not available for you -- it segfaults before it tries to actually execute anything. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-26 03:59 Message: Logged In: YES user_id=80475 Found a clean solution by counting consecutive runs of LOAD_CONST. See: Python/compile.c 2.332 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-26 02:54 Message: Logged In: YES user_id=80475 The bug is intrinsic to the current approach which misindentifies certain code sequences as looking like a tuple of constants. Attaching a simplified example. Unless I find a straight-forward fix in the next day or so, will have to rip out the tuple of constant code entirely. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 21:50 Message: Logged In: YES user_id=31435 Confirmed the segfault with current Python CVS. Boosted priority to the maximum. Assigned to Raymond, since he probably knows most about this. Dies in tuple_of_constants(), n=1, i=0, on the incref here: for (i=0 ; i Bugs item #1054564, was opened at 2004-10-26 12:55 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054564&group_id=5470 Category: Regular Expressions Group: Python 2.2.3 Status: Open Resolution: None Priority: 5 Submitted By: Rob (rwhent) Assigned to: Fredrik Lundh (effbot) Summary: RE '*.?' cores if len of found string exceeds 10000 Initial Comment: Whilst parsing some extremely long strings I found that the re.match causes segmentation faults on Solaris 2.8 when strings being matched contain '*.?' and the contents of the regex which matches this part of the regex exceeds 10000 chars (actually it seemed to be exactly at 8192 chars) This is the regex used: if re.match('^.*?\[\s*[A-Za-z_0-9]+\s*\].*',string): This regex looks for '[alphaNum_]' present in a large string When it failed the string was 8192 chars long with no matching '[alphaNum_]' present. If I reduce the length of the string below 8192 it works ok. This is a major issue to my application as some string to be parsed are very large. I saw some discussion on another bulletin board with a similar issue ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054564&group_id=5470 From noreply at sourceforge.net Tue Oct 26 15:20:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 15:21:09 2004 Subject: [ python-Bugs-1054564 ] RE '*.?' cores if len of found string exceeds 10000 Message-ID: Bugs item #1054564, was opened at 2004-10-26 14:55 Message generated for change (Comment added) made by effbot You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054564&group_id=5470 Category: Regular Expressions Group: Python 2.2.3 Status: Open >Resolution: Works For Me Priority: 5 Submitted By: Rob (rwhent) Assigned to: Fredrik Lundh (effbot) Summary: RE '*.?' cores if len of found string exceeds 10000 Initial Comment: Whilst parsing some extremely long strings I found that the re.match causes segmentation faults on Solaris 2.8 when strings being matched contain '*.?' and the contents of the regex which matches this part of the regex exceeds 10000 chars (actually it seemed to be exactly at 8192 chars) This is the regex used: if re.match('^.*?\[\s*[A-Za-z_0-9]+\s*\].*',string): This regex looks for '[alphaNum_]' present in a large string When it failed the string was 8192 chars long with no matching '[alphaNum_]' present. If I reduce the length of the string below 8192 it works ok. This is a major issue to my application as some string to be parsed are very large. I saw some discussion on another bulletin board with a similar issue ---------------------------------------------------------------------- Comment By: Fredrik Lundh (effbot) Date: 2004-10-26 15:20 Message: Logged In: YES user_id=38376 The max recursion limit problem in the re module is well-known. Until this limitation in the implementation is removed, to work around it check http://www.python.org/dev/doc/devel/lib/module-re.html http://python/org/sf/493252 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054564&group_id=5470 From noreply at sourceforge.net Tue Oct 26 15:24:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 15:24:28 2004 Subject: [ python-Bugs-1054564 ] RE '*.?' cores if len of found string exceeds 10000 Message-ID: Bugs item #1054564, was opened at 2004-10-26 14:55 Message generated for change (Comment added) made by effbot You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054564&group_id=5470 Category: Regular Expressions Group: Python 2.2.3 Status: Open Resolution: Works For Me Priority: 5 Submitted By: Rob (rwhent) Assigned to: Fredrik Lundh (effbot) Summary: RE '*.?' cores if len of found string exceeds 10000 Initial Comment: Whilst parsing some extremely long strings I found that the re.match causes segmentation faults on Solaris 2.8 when strings being matched contain '*.?' and the contents of the regex which matches this part of the regex exceeds 10000 chars (actually it seemed to be exactly at 8192 chars) This is the regex used: if re.match('^.*?\[\s*[A-Za-z_0-9]+\s*\].*',string): This regex looks for '[alphaNum_]' present in a large string When it failed the string was 8192 chars long with no matching '[alphaNum_]' present. If I reduce the length of the string below 8192 it works ok. This is a major issue to my application as some string to be parsed are very large. I saw some discussion on another bulletin board with a similar issue ---------------------------------------------------------------------- >Comment By: Fredrik Lundh (effbot) Date: 2004-10-26 15:24 Message: Logged In: YES user_id=38376 btw, if you're searching for things, why not use the "search" method? if re.search('\[\s*[A-Za-z_0-9]+\s*\]', string): (also, "[A-Za-z_0-9]" is better spelled "\w") ---------------------------------------------------------------------- Comment By: Fredrik Lundh (effbot) Date: 2004-10-26 15:20 Message: Logged In: YES user_id=38376 The max recursion limit problem in the re module is well-known. Until this limitation in the implementation is removed, to work around it check http://www.python.org/dev/doc/devel/lib/module-re.html http://python/org/sf/493252 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054564&group_id=5470 From noreply at sourceforge.net Tue Oct 26 15:26:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 15:27:03 2004 Subject: [ python-Bugs-1054139 ] serious string hashing error in 2.4b1 Message-ID: Bugs item #1054139, was opened at 2004-10-25 22:32 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Closed Resolution: Fixed Priority: 9 Submitted By: Bob Halley (rthalley) Assigned to: Raymond Hettinger (rhettinger) Summary: serious string hashing error in 2.4b1 Initial Comment: There is a serious hashing error in 2.4b1. I don't know if the error is confined to 64-bit systems, or is a general problem. The program attached to this report produces, as expected, this output when run with python 2.3.3: Python: 20303f0 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == -2784798555566127274 hash("DNSS") == 5809125768486327656 hash("DNSSE") == 5232635463381381892 hash("DNSSEC") == -7602892900506903802 When run with 2.4b1, I get the following output: Python: 20400b1 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == 8704052292078464 hash("DNSS") == 8704052292078464 hash("DNSSE") == 8704052292078464 hash("DNSSEC") == 8704052292078464 Traceback (most recent call last): File "foo.py", line 22, in ? assert hb == ha, 'hashes do not match!' AssertionError: hashes do not match! The way I discovered this was that dnspython's regression suite started failing because the string 'DNSSEC' constructed character-by-character was not being found in a dictionary which had a 'DNSSEC' key. I have not yet found the underlying bug; I was focussing on getting the info needed to demonstrate the bug first, since it's so serious. If I make any progress fixing it, I'll send a patch. /Bob ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-10-26 13:26 Message: Logged In: YES user_id=4771 Mea culpa. Not only premature optimizations but even the ones that are apparently reasonably small-reaching are the root of... etc. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-26 01:53 Message: Logged In: YES user_id=80475 Fixed. See: Objects/stringobject.c 2.226 Lib/test/string_tests.py 1.42 Thanks for the highly specific, timely bug report. ---------------------------------------------------------------------- Comment By: Bob Halley (rthalley) Date: 2004-10-25 23:26 Message: Logged In: YES user_id=671513 Patch #1054197 fixes this. The problem was that ceval.c wasn't invalidating the cached value. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 23:15 Message: Logged In: YES user_id=31435 FYI, I think ceval.c's string_concatenate() needs to reset ob_shash to -1 after it extends a string in-place. Alas, I can't make time to test that now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 23:08 Message: Logged In: YES user_id=31435 Boosted to highest priority. I don't get exactly the same numbers on my 32-bit box, but the problem exists there too. Suspect it has to do with new optimizations for string "+=". ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 From noreply at sourceforge.net Tue Oct 26 16:17:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 16:17:55 2004 Subject: [ python-Bugs-1054615 ] New failure in debug build Message-ID: Bugs item #1054615, was opened at 2004-10-26 10:17 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: New failure in debug build Initial Comment: This is current CVS Python. Sorry, no idea what caused this; it started within the last day (or two at the most): C:\Code\python\PCbuild>python_d ..\lib\test\test_traceb ack.py test_bug737473 (__main__.TracebackCases) ... ERROR test_caret (__main__.TracebackCases) ... ok test_nocaret (__main__.TracebackCases) ... ok ========================================== ============================ ERROR: test_bug737473 (__main__.TracebackCases) ------------------------------------------------------ ---------------- Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call ------------------------------------------------------ ---------------- Ran 3 tests in 0.078s FAILED (errors=1) Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 90, in ? test_main() File "..\lib\test\test_traceback.py", line 86, in test_main run_unittest(TracebackCases) File "C:\Code\python\lib\test\test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "C:\Code\python\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call [12133 refs] C:\Code\python\PCbuild> ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 From noreply at sourceforge.net Tue Oct 26 16:19:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 16:19:55 2004 Subject: [ python-Bugs-1054615 ] New failure in debug build Message-ID: Bugs item #1054615, was opened at 2004-10-26 10:17 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: New failure in debug build Initial Comment: This is current CVS Python. Sorry, no idea what caused this; it started within the last day (or two at the most): C:\Code\python\PCbuild>python_d ..\lib\test\test_traceb ack.py test_bug737473 (__main__.TracebackCases) ... ERROR test_caret (__main__.TracebackCases) ... ok test_nocaret (__main__.TracebackCases) ... ok ========================================== ============================ ERROR: test_bug737473 (__main__.TracebackCases) ------------------------------------------------------ ---------------- Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call ------------------------------------------------------ ---------------- Ran 3 tests in 0.078s FAILED (errors=1) Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 90, in ? test_main() File "..\lib\test\test_traceback.py", line 86, in test_main run_unittest(TracebackCases) File "C:\Code\python\lib\test\test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "C:\Code\python\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call [12133 refs] C:\Code\python\PCbuild> ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-26 10:19 Message: Logged In: YES user_id=31435 Noting that I also see this failure in a release build. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 From noreply at sourceforge.net Tue Oct 26 16:51:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 16:51:23 2004 Subject: [ python-Bugs-1054615 ] New failure in debug build Message-ID: Bugs item #1054615, was opened at 2004-10-26 15:17 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: New failure in debug build Initial Comment: This is current CVS Python. Sorry, no idea what caused this; it started within the last day (or two at the most): C:\Code\python\PCbuild>python_d ..\lib\test\test_traceb ack.py test_bug737473 (__main__.TracebackCases) ... ERROR test_caret (__main__.TracebackCases) ... ok test_nocaret (__main__.TracebackCases) ... ok ========================================== ============================ ERROR: test_bug737473 (__main__.TracebackCases) ------------------------------------------------------ ---------------- Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call ------------------------------------------------------ ---------------- Ran 3 tests in 0.078s FAILED (errors=1) Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 90, in ? test_main() File "..\lib\test\test_traceback.py", line 86, in test_main run_unittest(TracebackCases) File "C:\Code\python\lib\test\test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "C:\Code\python\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call [12133 refs] C:\Code\python\PCbuild> ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-10-26 15:51 Message: Logged In: YES user_id=6656 "Works for me" (linux). Something Windows specific? The obviously suspicious recent change is patch #737473 which perky checked in yesterday -- but that was a pure Python change, so there must be something more to it than that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 15:19 Message: Logged In: YES user_id=31435 Noting that I also see this failure in a release build. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 From noreply at sourceforge.net Tue Oct 26 17:14:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 17:15:02 2004 Subject: [ python-Bugs-1054615 ] New failure in debug build Message-ID: Bugs item #1054615, was opened at 2004-10-26 10:17 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: New failure in debug build Initial Comment: This is current CVS Python. Sorry, no idea what caused this; it started within the last day (or two at the most): C:\Code\python\PCbuild>python_d ..\lib\test\test_traceb ack.py test_bug737473 (__main__.TracebackCases) ... ERROR test_caret (__main__.TracebackCases) ... ok test_nocaret (__main__.TracebackCases) ... ok ========================================== ============================ ERROR: test_bug737473 (__main__.TracebackCases) ------------------------------------------------------ ---------------- Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call ------------------------------------------------------ ---------------- Ran 3 tests in 0.078s FAILED (errors=1) Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 90, in ? test_main() File "..\lib\test\test_traceback.py", line 86, in test_main run_unittest(TracebackCases) File "C:\Code\python\lib\test\test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "C:\Code\python\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call [12133 refs] C:\Code\python\PCbuild> ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-26 11:14 Message: Logged In: YES user_id=31435 Sorry to hear that -- I can't make time to dig. I blew away everything and recompiled from scratch, and the problem didn't go away. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-26 10:51 Message: Logged In: YES user_id=6656 "Works for me" (linux). Something Windows specific? The obviously suspicious recent change is patch #737473 which perky checked in yesterday -- but that was a pure Python change, so there must be something more to it than that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 10:19 Message: Logged In: YES user_id=31435 Noting that I also see this failure in a release build. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 From noreply at sourceforge.net Tue Oct 26 18:08:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 18:08:46 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-26 12:08 Message: Logged In: YES user_id=6380 How about this no-nonsense version? (rmtree.txt upload) The only thing you could quibble about is what to do when os.lstat fails. I chose backward compatibility. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-23 14:25 Message: Logged In: YES user_id=6380 BTW, if you want the correct path in the onerror calls for os.listdir, your best bet is not to use os.walk but to write something similar that suits our purpose. After you remove the features we don't use, it's really only a few lines of code. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-23 13:56 Message: Logged In: YES user_id=6380 I disagree. The docs are wrong because a failing listdir wasn't anticipated at first when this function was written. A failing listdir is a common failure case; I don't see how the os.path calls in os.walk could ever fail (since they all catch os.error). ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-22 23:01 Message: Logged In: YES user_id=469548 You know what: I think we should remove the os.listdir error handling: * The docs only specify "errors resulting from *failed removals*" as being handled. If they didn't we should worry about OSErrors from the os.path.* calls in os.walk() too. * The os.listdir error handling hasn't been in a final release, so no backwards compatibility problems yet. * We can't get the path right for os.listdir. * The problem also existed before the rewrite using os.walk(), so reverting to that is not a solution. Patch (the last?) is attached. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-21 00:42 Message: Logged In: YES user_id=6380 Yes. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-20 16:30 Message: Logged In: YES user_id=469548 With 'be careful, it may not exist.', do you mean rmtree can't rely on exc.filename being set? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 11:18 Message: Logged In: YES user_id=6380 This looks fine, and can be checked in. There's one minor remaining issue, but I don't see how to fix it without changing the signature for the onerror function that os.walk() takes, and that's a no-no: when the problem is a directory permission preventing os.listdir() to work, the path passed to the rmtree caller's onerror function is always the toplevel path, rather than the directory that couldn't be listed. The real path is included in the error message. (Hmm, perhaps you could get it out of there? I think it's an attribute of the exception object; but be careful, it may not exist.) It's unfortunate that the error argument to the onerror function is a sys.exc_info() triple rather than an exception object (calling sys.exc_info() is expensive), but that's an API choice from the past that we can't change until Python 3.0. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 04:57 Message: Logged In: YES user_id=469548 A new patch is attached. I factored out the handle_call_error method so exception handling is more local. Also, errors in os.walk were already being handled using the _raise_err/except OSError combination, but we now do it using the onerror argument to os.walk. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:42 Message: Logged In: YES user_id=6380 (Actually, os.walk has an onerror argument, it just uses a different convention and default, so we could thread this through easily.) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 18:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 02:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Tue Oct 26 21:44:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue Oct 26 21:44:28 2004 Subject: [ python-Bugs-1054821 ] difflib HtmlDiff() extra space on inserted 1 character lines Message-ID: Bugs item #1054821, was opened at 2004-10-26 14:44 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054821&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Dan Gass (dmgass) Assigned to: Nobody/Anonymous (nobody) Summary: difflib HtmlDiff() extra space on inserted 1 character lines Initial Comment: difflib.py -- rev 1.27 class - HtmlDiff method - _make_line line numbers 1370 thru 1371 was: if len(text) <= 1: text = ' '+text should be: if not text: text = ' ' This code was intended to show an "add" highlight in the case of inserted blank lines. It was originally coded to look for lines of length 1 or less to accomodate lines where the newline character is not stripped off. The undesired effect is that lines containing only one character (such as a open or close {} for a 'c' function) that are in column zero get represented with a space infront of it. Since newlines are stripped before this code is executed (a code change done after this logic was written) the above fix eliminates the undesired effect without sacrificing anything. The test for this module will not break as a result of this change. If it is desired, I can modify the test to cover this code. It would involve adding a blank line and possibly a one character line to the test input and then rebaselining the output. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054821&group_id=5470 From noreply at sourceforge.net Wed Oct 27 01:42:13 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 01:42:17 2004 Subject: [ python-Bugs-1054139 ] serious string hashing error in 2.4b1 Message-ID: Bugs item #1054139, was opened at 2004-10-25 17:32 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Closed Resolution: Fixed Priority: 9 Submitted By: Bob Halley (rthalley) Assigned to: Raymond Hettinger (rhettinger) Summary: serious string hashing error in 2.4b1 Initial Comment: There is a serious hashing error in 2.4b1. I don't know if the error is confined to 64-bit systems, or is a general problem. The program attached to this report produces, as expected, this output when run with python 2.3.3: Python: 20303f0 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == -2784798555566127274 hash("DNSS") == 5809125768486327656 hash("DNSSE") == 5232635463381381892 hash("DNSSEC") == -7602892900506903802 When run with 2.4b1, I get the following output: Python: 20400b1 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == 8704052292078464 hash("DNSS") == 8704052292078464 hash("DNSSE") == 8704052292078464 hash("DNSSEC") == 8704052292078464 Traceback (most recent call last): File "foo.py", line 22, in ? assert hb == ha, 'hashes do not match!' AssertionError: hashes do not match! The way I discovered this was that dnspython's regression suite started failing because the string 'DNSSEC' constructed character-by-character was not being found in a dictionary which had a 'DNSSEC' key. I have not yet found the underlying bug; I was focussing on getting the info needed to demonstrate the bug first, since it's so serious. If I make any progress fixing it, I'll send a patch. /Bob ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-26 18:42 Message: Logged In: YES user_id=80475 IMO, it was my fault. I reviewed, tested and applied the patch. Also, I don't think optimization evils have much to do with it. It was an oversight that could affect any patch. Fortunately, a good beta process helped catch it quickly. There may even be a silver lining. By invalidating the cache in the resize call, we may have fixed other possibly broken code and will have prevented other errors in the future. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-10-26 08:26 Message: Logged In: YES user_id=4771 Mea culpa. Not only premature optimizations but even the ones that are apparently reasonably small-reaching are the root of... etc. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-25 20:53 Message: Logged In: YES user_id=80475 Fixed. See: Objects/stringobject.c 2.226 Lib/test/string_tests.py 1.42 Thanks for the highly specific, timely bug report. ---------------------------------------------------------------------- Comment By: Bob Halley (rthalley) Date: 2004-10-25 18:26 Message: Logged In: YES user_id=671513 Patch #1054197 fixes this. The problem was that ceval.c wasn't invalidating the cached value. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 18:15 Message: Logged In: YES user_id=31435 FYI, I think ceval.c's string_concatenate() needs to reset ob_shash to -1 after it extends a string in-place. Alas, I can't make time to test that now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 18:08 Message: Logged In: YES user_id=31435 Boosted to highest priority. I don't get exactly the same numbers on my 32-bit box, but the problem exists there too. Suspect it has to do with new optimizations for string "+=". ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 From noreply at sourceforge.net Wed Oct 27 01:58:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 01:58:12 2004 Subject: [ python-Bugs-1054943 ] Python may contain NFC/NFKC bug per Unicode PRI #29 Message-ID: Bugs item #1054943, was opened at 2004-10-26 23:58 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054943&group_id=5470 Category: Unicode Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Rick McGowan (rick_mcgowan) Assigned to: M.-A. Lemburg (lemburg) Summary: Python may contain NFC/NFKC bug per Unicode PRI #29 Initial Comment: The Unicode Technical Committee posted Public Review Issue #29, describing a bug in the documentation of NFC and NFKC in the text of UAX #15 Unicode Normalization Forms. I have examined unicodedata.c in the Python implementation (2.3.4) and it appears the implementation of normalization in Python 2.3.4 may have the bug therein described. Please see the description of the bug and the textual fix that is being made to UAX #15, at the URL: http://www.unicode.org/review/pr-29.html The bug is in the definition of rule D2, affecting the characters "blocked" during re-composition. You may contact me by e-mail, or fill out the Unicode.org error reporting form if you have any questions or concerns. Since Python uses Unicode internally, it may also be wise to have someone from the Python development community on the Unicode Consortium's notification list to receive immediate notifications of public review issues, bugs, and other announcements affecting implementation of the standard. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054943&group_id=5470 From noreply at sourceforge.net Wed Oct 27 02:44:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 02:44:49 2004 Subject: [ python-Bugs-1050828 ] reindent.py strips execute permission on Unix Message-ID: Bugs item #1050828, was opened at 2004-10-20 12:48 Message generated for change (Comment added) made by facundobatista You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Ripton (dripton) Assigned to: Facundo Batista (facundobatista) Summary: reindent.py strips execute permission on Unix Initial Comment: Tools/scripts/reindent.py strips execute permission from its target file. If the tool detects that changes need to be made, then the target file is renamed to its original name + '.bak', then a new file is created with the original file's name and its whitespace-modified contents. No manipulation of the new file's permissions is done, so it ends up with the default for the user's umask. This makes reindent.py annoying to run in an automated fashion, if you rely on execute permission. The fix is trivial, except for portability. Here's the Posix-only version: + if os.name == "posix": + mode = os.stat(bak).st_mode + os.chmod(file, mode) Patch available upon request. ---------------------------------------------------------------------- >Comment By: Facundo Batista (facundobatista) Date: 2004-10-26 21:44 Message: Logged In: YES user_id=752496 This fixes the problem: the original file metadata (permisions, etc) stays unchanged. The issue with the metadata of the bak file is none: the new file has the user defaults permissions, group and owner. In the previous version, the original file left with those defaults permissions, so security is not getting worse. The only left issue is, if the user wants to recover the data from the bak file, he would need to change its metadata (this is minor, if something goes wrong and the bak is needed, it's likely that the user will need a manual approach). My primary concern is that I don't want to commit the changes now that we're beta without testing the changes in other platforms (however, I'll ask this on python-dev). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 00:19 Message: Logged In: YES user_id=31435 Sorry, I don't normally run on Unix, so I have no way to tell whether this fixes whatever the complaint is. Offhand it doesn't seem right to create a .bak file with default permissions either. For example, if only the user had permission to get at the original, letting the .bak file have broader permissions seems wrong too. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-25 23:57 Message: Logged In: YES user_id=752496 Attaching the file. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-24 18:20 Message: Logged In: YES user_id=752496 Instead of renaming the file and creating a new one, just copied the original and that's all (the target file is open with write mode, the content replaced, but the permissions stay). A patch is attached. Tim, I'm assigning this to you to doublecheck (not found unit tests for this script). . Facundo ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 From noreply at sourceforge.net Wed Oct 27 02:51:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 02:51:14 2004 Subject: [ python-Bugs-1054139 ] serious string hashing error in 2.4b1 Message-ID: Bugs item #1054139, was opened at 2004-10-25 18:32 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Closed Resolution: Fixed Priority: 9 Submitted By: Bob Halley (rthalley) Assigned to: Raymond Hettinger (rhettinger) Summary: serious string hashing error in 2.4b1 Initial Comment: There is a serious hashing error in 2.4b1. I don't know if the error is confined to 64-bit systems, or is a general problem. The program attached to this report produces, as expected, this output when run with python 2.3.3: Python: 20303f0 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == -2784798555566127274 hash("DNSS") == 5809125768486327656 hash("DNSSE") == 5232635463381381892 hash("DNSSEC") == -7602892900506903802 When run with 2.4b1, I get the following output: Python: 20400b1 System: Linux localhost 2.6.8-1.521 #1 Mon Aug 16 09:01:00 EDT 2004 x86 _64 x86_64 x86_64 GNU/Linux hash("DNSSEC") == -7602892900506903802 hash("D") == 8704026181 hash("DN") == 8704052292078464 hash("DNS") == 8704052292078464 hash("DNSS") == 8704052292078464 hash("DNSSE") == 8704052292078464 hash("DNSSEC") == 8704052292078464 Traceback (most recent call last): File "foo.py", line 22, in ? assert hb == ha, 'hashes do not match!' AssertionError: hashes do not match! The way I discovered this was that dnspython's regression suite started failing because the string 'DNSSEC' constructed character-by-character was not being found in a dictionary which had a 'DNSSEC' key. I have not yet found the underlying bug; I was focussing on getting the info needed to demonstrate the bug first, since it's so serious. If I make any progress fixing it, I'll send a patch. /Bob ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-26 20:51 Message: Logged In: YES user_id=31435 Na, it's optimization, all right -- there's no conceivable reason other than optimization to try to mutate an immutable object, esp. one that's already been visible to user code. But that's OK. The more optimizations we do, the more optimization bugs we'll get. We'd be the only language implementation project in the history of the universe to avoid that otherwise. Don't be so touchy . I'll note that the string object invariants aren't clearly written down in one place. One use for making invariants explicit is as a sanity checklist when doing optimizations. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-26 19:42 Message: Logged In: YES user_id=80475 IMO, it was my fault. I reviewed, tested and applied the patch. Also, I don't think optimization evils have much to do with it. It was an oversight that could affect any patch. Fortunately, a good beta process helped catch it quickly. There may even be a silver lining. By invalidating the cache in the resize call, we may have fixed other possibly broken code and will have prevented other errors in the future. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-10-26 09:26 Message: Logged In: YES user_id=4771 Mea culpa. Not only premature optimizations but even the ones that are apparently reasonably small-reaching are the root of... etc. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-25 21:53 Message: Logged In: YES user_id=80475 Fixed. See: Objects/stringobject.c 2.226 Lib/test/string_tests.py 1.42 Thanks for the highly specific, timely bug report. ---------------------------------------------------------------------- Comment By: Bob Halley (rthalley) Date: 2004-10-25 19:26 Message: Logged In: YES user_id=671513 Patch #1054197 fixes this. The problem was that ceval.c wasn't invalidating the cached value. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 19:15 Message: Logged In: YES user_id=31435 FYI, I think ceval.c's string_concatenate() needs to reset ob_shash to -1 after it extends a string in-place. Alas, I can't make time to test that now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 19:08 Message: Logged In: YES user_id=31435 Boosted to highest priority. I don't get exactly the same numbers on my 32-bit box, but the problem exists there too. Suspect it has to do with new optimizations for string "+=". ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054139&group_id=5470 From noreply at sourceforge.net Wed Oct 27 03:44:34 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 03:44:37 2004 Subject: [ python-Bugs-1054615 ] New failure in debug build Message-ID: Bugs item #1054615, was opened at 2004-10-26 10:17 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None >Priority: 8 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: New failure in debug build Initial Comment: This is current CVS Python. Sorry, no idea what caused this; it started within the last day (or two at the most): C:\Code\python\PCbuild>python_d ..\lib\test\test_traceb ack.py test_bug737473 (__main__.TracebackCases) ... ERROR test_caret (__main__.TracebackCases) ... ok test_nocaret (__main__.TracebackCases) ... ok ========================================== ============================ ERROR: test_bug737473 (__main__.TracebackCases) ------------------------------------------------------ ---------------- Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call ------------------------------------------------------ ---------------- Ran 3 tests in 0.078s FAILED (errors=1) Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 90, in ? test_main() File "..\lib\test\test_traceback.py", line 86, in test_main run_unittest(TracebackCases) File "C:\Code\python\lib\test\test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "C:\Code\python\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call [12133 refs] C:\Code\python\PCbuild> ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-26 21:44 Message: Logged In: YES user_id=31435 Reduced the priority, because test_bug737473() raises the same SystemError under the released Python 2.3.4 on my box -- it's not a new problem. OTOH, we can't release 2.4 with a standard test that fails on a major platform either. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 11:14 Message: Logged In: YES user_id=31435 Sorry to hear that -- I can't make time to dig. I blew away everything and recompiled from scratch, and the problem didn't go away. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-26 10:51 Message: Logged In: YES user_id=6656 "Works for me" (linux). Something Windows specific? The obviously suspicious recent change is patch #737473 which perky checked in yesterday -- but that was a pure Python change, so there must be something more to it than that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 10:19 Message: Logged In: YES user_id=31435 Noting that I also see this failure in a release build. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 From noreply at sourceforge.net Wed Oct 27 03:55:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 03:55:57 2004 Subject: [ python-Bugs-1054615 ] New failure in debug build Message-ID: Bugs item #1054615, was opened at 2004-10-26 23:17 Message generated for change (Comment added) made by perky You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 8 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: New failure in debug build Initial Comment: This is current CVS Python. Sorry, no idea what caused this; it started within the last day (or two at the most): C:\Code\python\PCbuild>python_d ..\lib\test\test_traceb ack.py test_bug737473 (__main__.TracebackCases) ... ERROR test_caret (__main__.TracebackCases) ... ok test_nocaret (__main__.TracebackCases) ... ok ========================================== ============================ ERROR: test_bug737473 (__main__.TracebackCases) ------------------------------------------------------ ---------------- Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call ------------------------------------------------------ ---------------- Ran 3 tests in 0.078s FAILED (errors=1) Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 90, in ? test_main() File "..\lib\test\test_traceback.py", line 86, in test_main run_unittest(TracebackCases) File "C:\Code\python\lib\test\test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "C:\Code\python\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call [12133 refs] C:\Code\python\PCbuild> ---------------------------------------------------------------------- >Comment By: Hye-Shik Chang (perky) Date: 2004-10-27 10:55 Message: Logged In: YES user_id=55188 I failed to reproduce the problem in FreeBSD, Windows XP and Linux. What's your platform testing in? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 10:44 Message: Logged In: YES user_id=31435 Reduced the priority, because test_bug737473() raises the same SystemError under the released Python 2.3.4 on my box -- it's not a new problem. OTOH, we can't release 2.4 with a standard test that fails on a major platform either. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 00:14 Message: Logged In: YES user_id=31435 Sorry to hear that -- I can't make time to dig. I blew away everything and recompiled from scratch, and the problem didn't go away. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-26 23:51 Message: Logged In: YES user_id=6656 "Works for me" (linux). Something Windows specific? The obviously suspicious recent change is patch #737473 which perky checked in yesterday -- but that was a pure Python change, so there must be something more to it than that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 23:19 Message: Logged In: YES user_id=31435 Noting that I also see this failure in a release build. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 From noreply at sourceforge.net Wed Oct 27 04:15:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 04:15:09 2004 Subject: [ python-Bugs-1054615 ] New failure in debug build Message-ID: Bugs item #1054615, was opened at 2004-10-26 10:17 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 8 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: New failure in debug build Initial Comment: This is current CVS Python. Sorry, no idea what caused this; it started within the last day (or two at the most): C:\Code\python\PCbuild>python_d ..\lib\test\test_traceb ack.py test_bug737473 (__main__.TracebackCases) ... ERROR test_caret (__main__.TracebackCases) ... ok test_nocaret (__main__.TracebackCases) ... ok ========================================== ============================ ERROR: test_bug737473 (__main__.TracebackCases) ------------------------------------------------------ ---------------- Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call ------------------------------------------------------ ---------------- Ran 3 tests in 0.078s FAILED (errors=1) Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 90, in ? test_main() File "..\lib\test\test_traceback.py", line 86, in test_main run_unittest(TracebackCases) File "C:\Code\python\lib\test\test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "C:\Code\python\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call [12133 refs] C:\Code\python\PCbuild> ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:15 Message: Logged In: YES user_id=31435 I'm on WinXP Pro SP2. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-10-26 21:55 Message: Logged In: YES user_id=55188 I failed to reproduce the problem in FreeBSD, Windows XP and Linux. What's your platform testing in? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 21:44 Message: Logged In: YES user_id=31435 Reduced the priority, because test_bug737473() raises the same SystemError under the released Python 2.3.4 on my box -- it's not a new problem. OTOH, we can't release 2.4 with a standard test that fails on a major platform either. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 11:14 Message: Logged In: YES user_id=31435 Sorry to hear that -- I can't make time to dig. I blew away everything and recompiled from scratch, and the problem didn't go away. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-26 10:51 Message: Logged In: YES user_id=6656 "Works for me" (linux). Something Windows specific? The obviously suspicious recent change is patch #737473 which perky checked in yesterday -- but that was a pure Python change, so there must be something more to it than that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 10:19 Message: Logged In: YES user_id=31435 Noting that I also see this failure in a release build. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 From noreply at sourceforge.net Wed Oct 27 04:20:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 04:20:16 2004 Subject: [ python-Bugs-1054615 ] New failure in debug build Message-ID: Bugs item #1054615, was opened at 2004-10-26 10:17 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None Priority: 8 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: New failure in debug build Initial Comment: This is current CVS Python. Sorry, no idea what caused this; it started within the last day (or two at the most): C:\Code\python\PCbuild>python_d ..\lib\test\test_traceb ack.py test_bug737473 (__main__.TracebackCases) ... ERROR test_caret (__main__.TracebackCases) ... ok test_nocaret (__main__.TracebackCases) ... ok ========================================== ============================ ERROR: test_bug737473 (__main__.TracebackCases) ------------------------------------------------------ ---------------- Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call ------------------------------------------------------ ---------------- Ran 3 tests in 0.078s FAILED (errors=1) Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 90, in ? test_main() File "..\lib\test\test_traceback.py", line 86, in test_main run_unittest(TracebackCases) File "C:\Code\python\lib\test\test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "C:\Code\python\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call [12133 refs] C:\Code\python\PCbuild> ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:20 Message: Logged In: YES user_id=31435 Looks damn complicated. In import.c, load_source_module() returns NULL if PyOS_GetLastModificationTime() returns -1. But PyOS_GetLastModificationTime() doesn't set any exception when it does return -1. This NULL then propagates all the way back to PyObject_Call(), which produces the SystemError I see. So there's at least a bug in load_source_module() here -- it can't return NULL without setting an exception. I don't yet know why PyOS_GetLastModificationTime() returns -1 in this test case. I suspect it's because the (0,0) passed to utime() doesn't make much sense. I don't get the error if I force it to take the time.sleep() path instead. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:15 Message: Logged In: YES user_id=31435 I'm on WinXP Pro SP2. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-10-26 21:55 Message: Logged In: YES user_id=55188 I failed to reproduce the problem in FreeBSD, Windows XP and Linux. What's your platform testing in? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 21:44 Message: Logged In: YES user_id=31435 Reduced the priority, because test_bug737473() raises the same SystemError under the released Python 2.3.4 on my box -- it's not a new problem. OTOH, we can't release 2.4 with a standard test that fails on a major platform either. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 11:14 Message: Logged In: YES user_id=31435 Sorry to hear that -- I can't make time to dig. I blew away everything and recompiled from scratch, and the problem didn't go away. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-26 10:51 Message: Logged In: YES user_id=6656 "Works for me" (linux). Something Windows specific? The obviously suspicious recent change is patch #737473 which perky checked in yesterday -- but that was a pure Python change, so there must be something more to it than that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 10:19 Message: Logged In: YES user_id=31435 Noting that I also see this failure in a release build. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 From noreply at sourceforge.net Wed Oct 27 04:34:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 04:34:31 2004 Subject: [ python-Bugs-1054615 ] New failure in debug build Message-ID: Bugs item #1054615, was opened at 2004-10-26 10:17 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Open Resolution: None >Priority: 5 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: New failure in debug build Initial Comment: This is current CVS Python. Sorry, no idea what caused this; it started within the last day (or two at the most): C:\Code\python\PCbuild>python_d ..\lib\test\test_traceb ack.py test_bug737473 (__main__.TracebackCases) ... ERROR test_caret (__main__.TracebackCases) ... ok test_nocaret (__main__.TracebackCases) ... ok ========================================== ============================ ERROR: test_bug737473 (__main__.TracebackCases) ------------------------------------------------------ ---------------- Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call ------------------------------------------------------ ---------------- Ran 3 tests in 0.078s FAILED (errors=1) Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 90, in ? test_main() File "..\lib\test\test_traceback.py", line 86, in test_main run_unittest(TracebackCases) File "C:\Code\python\lib\test\test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "C:\Code\python\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call [12133 refs] C:\Code\python\PCbuild> ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:34 Message: Logged In: YES user_id=31435 Reduced priority to 5 (default). Papered over the bug by passing sane values to os.utime() in the new test code. The underlying import.c bug is still there. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:20 Message: Logged In: YES user_id=31435 Looks damn complicated. In import.c, load_source_module() returns NULL if PyOS_GetLastModificationTime() returns -1. But PyOS_GetLastModificationTime() doesn't set any exception when it does return -1. This NULL then propagates all the way back to PyObject_Call(), which produces the SystemError I see. So there's at least a bug in load_source_module() here -- it can't return NULL without setting an exception. I don't yet know why PyOS_GetLastModificationTime() returns -1 in this test case. I suspect it's because the (0,0) passed to utime() doesn't make much sense. I don't get the error if I force it to take the time.sleep() path instead. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:15 Message: Logged In: YES user_id=31435 I'm on WinXP Pro SP2. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-10-26 21:55 Message: Logged In: YES user_id=55188 I failed to reproduce the problem in FreeBSD, Windows XP and Linux. What's your platform testing in? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 21:44 Message: Logged In: YES user_id=31435 Reduced the priority, because test_bug737473() raises the same SystemError under the released Python 2.3.4 on my box -- it's not a new problem. OTOH, we can't release 2.4 with a standard test that fails on a major platform either. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 11:14 Message: Logged In: YES user_id=31435 Sorry to hear that -- I can't make time to dig. I blew away everything and recompiled from scratch, and the problem didn't go away. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-26 10:51 Message: Logged In: YES user_id=6656 "Works for me" (linux). Something Windows specific? The obviously suspicious recent change is patch #737473 which perky checked in yesterday -- but that was a pure Python change, so there must be something more to it than that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 10:19 Message: Logged In: YES user_id=31435 Noting that I also see this failure in a release build. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 From noreply at sourceforge.net Wed Oct 27 04:36:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 04:36:24 2004 Subject: [ python-Bugs-1054615 ] import.c bug when getting mod time fails Message-ID: Bugs item #1054615, was opened at 2004-10-26 10:17 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 Category: Python Interpreter Core >Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) >Summary: import.c bug when getting mod time fails Initial Comment: This is current CVS Python. Sorry, no idea what caused this; it started within the last day (or two at the most): C:\Code\python\PCbuild>python_d ..\lib\test\test_traceb ack.py test_bug737473 (__main__.TracebackCases) ... ERROR test_caret (__main__.TracebackCases) ... ok test_nocaret (__main__.TracebackCases) ... ok ========================================== ============================ ERROR: test_bug737473 (__main__.TracebackCases) ------------------------------------------------------ ---------------- Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call ------------------------------------------------------ ---------------- Ran 3 tests in 0.078s FAILED (errors=1) Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 90, in ? test_main() File "..\lib\test\test_traceback.py", line 86, in test_main run_unittest(TracebackCases) File "C:\Code\python\lib\test\test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "C:\Code\python\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call [12133 refs] C:\Code\python\PCbuild> ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:36 Message: Logged In: YES user_id=31435 Changed the description to match current reality, and changed group to Python 2.3 (the import.c bug is an old bug). I don't intend to work on it. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:34 Message: Logged In: YES user_id=31435 Reduced priority to 5 (default). Papered over the bug by passing sane values to os.utime() in the new test code. The underlying import.c bug is still there. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:20 Message: Logged In: YES user_id=31435 Looks damn complicated. In import.c, load_source_module() returns NULL if PyOS_GetLastModificationTime() returns -1. But PyOS_GetLastModificationTime() doesn't set any exception when it does return -1. This NULL then propagates all the way back to PyObject_Call(), which produces the SystemError I see. So there's at least a bug in load_source_module() here -- it can't return NULL without setting an exception. I don't yet know why PyOS_GetLastModificationTime() returns -1 in this test case. I suspect it's because the (0,0) passed to utime() doesn't make much sense. I don't get the error if I force it to take the time.sleep() path instead. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:15 Message: Logged In: YES user_id=31435 I'm on WinXP Pro SP2. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-10-26 21:55 Message: Logged In: YES user_id=55188 I failed to reproduce the problem in FreeBSD, Windows XP and Linux. What's your platform testing in? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 21:44 Message: Logged In: YES user_id=31435 Reduced the priority, because test_bug737473() raises the same SystemError under the released Python 2.3.4 on my box -- it's not a new problem. OTOH, we can't release 2.4 with a standard test that fails on a major platform either. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 11:14 Message: Logged In: YES user_id=31435 Sorry to hear that -- I can't make time to dig. I blew away everything and recompiled from scratch, and the problem didn't go away. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-26 10:51 Message: Logged In: YES user_id=6656 "Works for me" (linux). Something Windows specific? The obviously suspicious recent change is patch #737473 which perky checked in yesterday -- but that was a pure Python change, so there must be something more to it than that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 10:19 Message: Logged In: YES user_id=31435 Noting that I also see this failure in a release build. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 From noreply at sourceforge.net Wed Oct 27 05:30:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 05:30:57 2004 Subject: [ python-Bugs-1054615 ] import.c bug when getting mod time fails Message-ID: Bugs item #1054615, was opened at 2004-10-26 10:17 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: import.c bug when getting mod time fails Initial Comment: This is current CVS Python. Sorry, no idea what caused this; it started within the last day (or two at the most): C:\Code\python\PCbuild>python_d ..\lib\test\test_traceb ack.py test_bug737473 (__main__.TracebackCases) ... ERROR test_caret (__main__.TracebackCases) ... ok test_nocaret (__main__.TracebackCases) ... ok ========================================== ============================ ERROR: test_bug737473 (__main__.TracebackCases) ------------------------------------------------------ ---------------- Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call ------------------------------------------------------ ---------------- Ran 3 tests in 0.078s FAILED (errors=1) Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 90, in ? test_main() File "..\lib\test\test_traceback.py", line 86, in test_main run_unittest(TracebackCases) File "C:\Code\python\lib\test\test_support.py", line 290, in run_unittest run_suite(suite, testclass) File "C:\Code\python\lib\test\test_support.py", line 275, in run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "..\lib\test\test_traceback.py", line 62, in test_bug737473 import test_bug737473 SystemError: NULL result without error in PyObject_Call [12133 refs] C:\Code\python\PCbuild> ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-26 23:30 Message: Logged In: YES user_id=31435 FYI, any value passed to utime less than 18000 causes the problem on my box, and I bet it's not a coincidence that I'm currently 5 hours (18000 seconds) west of the prime meridian. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:36 Message: Logged In: YES user_id=31435 Changed the description to match current reality, and changed group to Python 2.3 (the import.c bug is an old bug). I don't intend to work on it. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:34 Message: Logged In: YES user_id=31435 Reduced priority to 5 (default). Papered over the bug by passing sane values to os.utime() in the new test code. The underlying import.c bug is still there. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:20 Message: Logged In: YES user_id=31435 Looks damn complicated. In import.c, load_source_module() returns NULL if PyOS_GetLastModificationTime() returns -1. But PyOS_GetLastModificationTime() doesn't set any exception when it does return -1. This NULL then propagates all the way back to PyObject_Call(), which produces the SystemError I see. So there's at least a bug in load_source_module() here -- it can't return NULL without setting an exception. I don't yet know why PyOS_GetLastModificationTime() returns -1 in this test case. I suspect it's because the (0,0) passed to utime() doesn't make much sense. I don't get the error if I force it to take the time.sleep() path instead. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 22:15 Message: Logged In: YES user_id=31435 I'm on WinXP Pro SP2. ---------------------------------------------------------------------- Comment By: Hye-Shik Chang (perky) Date: 2004-10-26 21:55 Message: Logged In: YES user_id=55188 I failed to reproduce the problem in FreeBSD, Windows XP and Linux. What's your platform testing in? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 21:44 Message: Logged In: YES user_id=31435 Reduced the priority, because test_bug737473() raises the same SystemError under the released Python 2.3.4 on my box -- it's not a new problem. OTOH, we can't release 2.4 with a standard test that fails on a major platform either. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 11:14 Message: Logged In: YES user_id=31435 Sorry to hear that -- I can't make time to dig. I blew away everything and recompiled from scratch, and the problem didn't go away. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-26 10:51 Message: Logged In: YES user_id=6656 "Works for me" (linux). Something Windows specific? The obviously suspicious recent change is patch #737473 which perky checked in yesterday -- but that was a pure Python change, so there must be something more to it than that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 10:19 Message: Logged In: YES user_id=31435 Noting that I also see this failure in a release build. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054615&group_id=5470 From noreply at sourceforge.net Wed Oct 27 05:35:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 05:35:46 2004 Subject: [ python-Bugs-1053629 ] Why is this expression == False? Message-ID: Bugs item #1053629, was opened at 2004-10-25 06:09 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053629&group_id=5470 Category: None Group: Python 2.3 Status: Closed Resolution: Invalid Priority: 5 Submitted By: Alexander R?dseth (alexanro) Assigned to: Nobody/Anonymous (nobody) Summary: Why is this expression == False? Initial Comment: I am not sure if this is a bug or not, but I'm very curious of the reason for the following result: >>> from math import * >>> theta = pi >>> e**(complex(0,1)*theta)==cos(theta)+complex(0,1)*sin(theta) False This is supposed to be True, unless I've made some typo or made some other mistake. It's a version of Euler's equation. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-26 23:35 Message: Logged In: YES user_id=31435 Try this one: >>> 37.0 / 3 / 3 * 9 == 37.0 False Worry about transcendentals after making peace with basic float arithmetic . ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-10-25 07:09 Message: Logged In: YES user_id=6656 Heh. Walter, the two numbers you show are in fact equal. Also, the expression is True for me. I wonder what platform Alexander is on -- but not very much, attempting to verify equalities involving transcedental quantities using finite precision arithmetic is always going to be risky. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2004-10-25 06:35 Message: Logged In: YES user_id=89016 This is due to rounding errors: >>> from math import * *>>> e**(1j*pi) (-1+1.2246063538223773e-16j) *>>> cos(pi)+1j*sin(pi) (-1+1.2246063538223773e-16j) Closing the bug report ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1053629&group_id=5470 From noreply at sourceforge.net Wed Oct 27 07:46:08 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 07:46:17 2004 Subject: [ python-Bugs-1052242 ] Attempt to run all atexit handlers Message-ID: Bugs item #1052242, was opened at 2004-10-22 09:31 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Greg Chapman (glchapman) Assigned to: Skip Montanaro (montanaro) Summary: Attempt to run all atexit handlers Initial Comment: Currently, if an atexit handler raises an exception during _run_exitfuncs, none of the remaining handlers in the _exithandlers list gets run. I would prefer it if _run_exitfuncs did something like: while _exithandlers: func, targs, kargs = _exithandlers.pop() try: func(*targs, **kargs) except SystemExit: raise except: import sys, traceback print >> sys.stderr, "Error in sys.exitfunc:" traceback.print_exc() The SystemExit except clause is there because it looks like (given its special treatment in call_sys_exitfunc) SystemExit is envisioned as a way of intentionally breaking out of the exithandlers loop. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-27 00:46 Message: Logged In: YES user_id=80475 I prefer the original proposal that leaves the API untouched. Irrespective of ordering, my thought is that all exit handlers are probably expected to run (needed for important cleanup, resource freeing, etc) and may be independent of one another. So, all of them that can run, probably should. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-10-24 12:21 Message: Logged In: YES user_id=44345 OTOH, since the ordering of the exit handlers is not guaranteed, exiting after one exception is also reasonable behavior. Attached is a diff that does two things: adds a set_run_all function and converts the functionality into an ExitHandler class to keep from all the different bits of global state. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-23 19:56 Message: Logged In: YES user_id=80475 This could be considered a bug. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 From noreply at sourceforge.net Wed Oct 27 11:21:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 11:21:45 2004 Subject: [ python-Bugs-1055168 ] pdb.set_trace should go up one more frame Message-ID: Bugs item #1055168, was opened at 2004-10-27 11:21 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055168&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Johannes Gijsbers (jlgijsbers) Assigned to: Nobody/Anonymous (nobody) Summary: pdb.set_trace should go up one more frame Initial Comment: Calling pdb.set_trace() calls Bdb.set_trace, which makes the debugger enter inside pdb.set_trace (see the attached console session for details). It should enter at the stack frame calling pdb.set_trace(). I can fix this, but I don't mind if someone else does. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055168&group_id=5470 From noreply at sourceforge.net Wed Oct 27 11:57:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 11:57:58 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-18 03:24 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-27 11:57 Message: Logged In: YES user_id=469548 Yes, I think this is the right way to go about it. I'd personally make a couple of changes, but those are up to your discretion: * Factor out the try: except:'s to a nested function. * Use sys.exc_info() instead of the exception for backwards compatibility. * Use 'os.path.isdir(fullname) and not os.path.islink(fullname)' expression instead of the lstat call. I think it more clearly communicates what we want, even though it's more stat calls. Latest version attached (rmtree-refactored.py). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-26 18:08 Message: Logged In: YES user_id=6380 How about this no-nonsense version? (rmtree.txt upload) The only thing you could quibble about is what to do when os.lstat fails. I chose backward compatibility. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-23 20:25 Message: Logged In: YES user_id=6380 BTW, if you want the correct path in the onerror calls for os.listdir, your best bet is not to use os.walk but to write something similar that suits our purpose. After you remove the features we don't use, it's really only a few lines of code. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-23 19:56 Message: Logged In: YES user_id=6380 I disagree. The docs are wrong because a failing listdir wasn't anticipated at first when this function was written. A failing listdir is a common failure case; I don't see how the os.path calls in os.walk could ever fail (since they all catch os.error). ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-23 05:01 Message: Logged In: YES user_id=469548 You know what: I think we should remove the os.listdir error handling: * The docs only specify "errors resulting from *failed removals*" as being handled. If they didn't we should worry about OSErrors from the os.path.* calls in os.walk() too. * The os.listdir error handling hasn't been in a final release, so no backwards compatibility problems yet. * We can't get the path right for os.listdir. * The problem also existed before the rewrite using os.walk(), so reverting to that is not a solution. Patch (the last?) is attached. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-21 06:42 Message: Logged In: YES user_id=6380 Yes. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-20 22:30 Message: Logged In: YES user_id=469548 With 'be careful, it may not exist.', do you mean rmtree can't rely on exc.filename being set? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 17:18 Message: Logged In: YES user_id=6380 This looks fine, and can be checked in. There's one minor remaining issue, but I don't see how to fix it without changing the signature for the onerror function that os.walk() takes, and that's a no-no: when the problem is a directory permission preventing os.listdir() to work, the path passed to the rmtree caller's onerror function is always the toplevel path, rather than the directory that couldn't be listed. The real path is included in the error message. (Hmm, perhaps you could get it out of there? I think it's an attribute of the exception object; but be careful, it may not exist.) It's unfortunate that the error argument to the onerror function is a sys.exc_info() triple rather than an exception object (calling sys.exc_info() is expensive), but that's an API choice from the past that we can't change until Python 3.0. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 10:57 Message: Logged In: YES user_id=469548 A new patch is attached. I factored out the handle_call_error method so exception handling is more local. Also, errors in os.walk were already being handled using the _raise_err/except OSError combination, but we now do it using the onerror argument to os.walk. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 01:42 Message: Logged In: YES user_id=6380 (Actually, os.walk has an onerror argument, it just uses a different convention and default, so we could thread this through easily.) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 01:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 00:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 08:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Wed Oct 27 14:38:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 14:38:38 2004 Subject: [ python-Bugs-1050828 ] reindent.py strips execute permission on Unix Message-ID: Bugs item #1050828, was opened at 2004-10-20 17:48 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Ripton (dripton) Assigned to: Facundo Batista (facundobatista) Summary: reindent.py strips execute permission on Unix Initial Comment: Tools/scripts/reindent.py strips execute permission from its target file. If the tool detects that changes need to be made, then the target file is renamed to its original name + '.bak', then a new file is created with the original file's name and its whitespace-modified contents. No manipulation of the new file's permissions is done, so it ends up with the default for the user's umask. This makes reindent.py annoying to run in an automated fashion, if you rely on execute permission. The fix is trivial, except for portability. Here's the Posix-only version: + if os.name == "posix": + mode = os.stat(bak).st_mode + os.chmod(file, mode) Patch available upon request. ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-27 14:38 Message: Logged In: YES user_id=469548 The patch works for me (Linux 2.6.8-1-686 i686 GNU/Linux) and a coworker (Mac OS X 10.3). ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-27 02:44 Message: Logged In: YES user_id=752496 This fixes the problem: the original file metadata (permisions, etc) stays unchanged. The issue with the metadata of the bak file is none: the new file has the user defaults permissions, group and owner. In the previous version, the original file left with those defaults permissions, so security is not getting worse. The only left issue is, if the user wants to recover the data from the bak file, he would need to change its metadata (this is minor, if something goes wrong and the bak is needed, it's likely that the user will need a manual approach). My primary concern is that I don't want to commit the changes now that we're beta without testing the changes in other platforms (however, I'll ask this on python-dev). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 05:19 Message: Logged In: YES user_id=31435 Sorry, I don't normally run on Unix, so I have no way to tell whether this fixes whatever the complaint is. Offhand it doesn't seem right to create a .bak file with default permissions either. For example, if only the user had permission to get at the original, letting the .bak file have broader permissions seems wrong too. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-26 04:57 Message: Logged In: YES user_id=752496 Attaching the file. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-24 23:20 Message: Logged In: YES user_id=752496 Instead of renaming the file and creating a new one, just copied the original and that's all (the target file is open with write mode, the content replaced, but the permissions stay). A patch is attached. Tim, I'm assigning this to you to doublecheck (not found unit tests for this script). . Facundo ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 From noreply at sourceforge.net Wed Oct 27 16:44:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 16:44:10 2004 Subject: [ python-Bugs-1052242 ] Attempt to run all atexit handlers Message-ID: Bugs item #1052242, was opened at 2004-10-22 09:31 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Greg Chapman (glchapman) Assigned to: Skip Montanaro (montanaro) Summary: Attempt to run all atexit handlers Initial Comment: Currently, if an atexit handler raises an exception during _run_exitfuncs, none of the remaining handlers in the _exithandlers list gets run. I would prefer it if _run_exitfuncs did something like: while _exithandlers: func, targs, kargs = _exithandlers.pop() try: func(*targs, **kargs) except SystemExit: raise except: import sys, traceback print >> sys.stderr, "Error in sys.exitfunc:" traceback.print_exc() The SystemExit except clause is there because it looks like (given its special treatment in call_sys_exitfunc) SystemExit is envisioned as a way of intentionally breaking out of the exithandlers loop. ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2004-10-27 09:44 Message: Logged In: YES user_id=44345 I agree with you in principle, but you'd change behavior and possibly break existing code that relies on the current implicit semantics, hence my suggestion to use a variable to control the behavior, with it set to preserve the current behavior by default. I'll post a request to python-dev soliciting input. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-27 00:46 Message: Logged In: YES user_id=80475 I prefer the original proposal that leaves the API untouched. Irrespective of ordering, my thought is that all exit handlers are probably expected to run (needed for important cleanup, resource freeing, etc) and may be independent of one another. So, all of them that can run, probably should. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-10-24 12:21 Message: Logged In: YES user_id=44345 OTOH, since the ordering of the exit handlers is not guaranteed, exiting after one exception is also reasonable behavior. Attached is a diff that does two things: adds a set_run_all function and converts the functionality into an ExitHandler class to keep from all the different bits of global state. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-23 19:56 Message: Logged In: YES user_id=80475 This could be considered a bug. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 From noreply at sourceforge.net Wed Oct 27 16:45:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 16:45:32 2004 Subject: [ python-Bugs-1050828 ] reindent.py strips execute permission on Unix Message-ID: Bugs item #1050828, was opened at 2004-10-20 11:48 Message generated for change (Comment added) made by dripton You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Ripton (dripton) Assigned to: Facundo Batista (facundobatista) Summary: reindent.py strips execute permission on Unix Initial Comment: Tools/scripts/reindent.py strips execute permission from its target file. If the tool detects that changes need to be made, then the target file is renamed to its original name + '.bak', then a new file is created with the original file's name and its whitespace-modified contents. No manipulation of the new file's permissions is done, so it ends up with the default for the user's umask. This makes reindent.py annoying to run in an automated fashion, if you rely on execute permission. The fix is trivial, except for portability. Here's the Posix-only version: + if os.name == "posix": + mode = os.stat(bak).st_mode + os.chmod(file, mode) Patch available upon request. ---------------------------------------------------------------------- >Comment By: David Ripton (dripton) Date: 2004-10-27 10:45 Message: Logged In: YES user_id=9425 The patch is an improvement (better to have the wrong permissions on the target file than on the backup file), but I think it would be even better if the .bak file had the same permissions as the target file. How about using shutil.copy2 (which preserves permissions, among other things) instead of shutil.copy? Thanks. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-27 08:38 Message: Logged In: YES user_id=469548 The patch works for me (Linux 2.6.8-1-686 i686 GNU/Linux) and a coworker (Mac OS X 10.3). ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-26 20:44 Message: Logged In: YES user_id=752496 This fixes the problem: the original file metadata (permisions, etc) stays unchanged. The issue with the metadata of the bak file is none: the new file has the user defaults permissions, group and owner. In the previous version, the original file left with those defaults permissions, so security is not getting worse. The only left issue is, if the user wants to recover the data from the bak file, he would need to change its metadata (this is minor, if something goes wrong and the bak is needed, it's likely that the user will need a manual approach). My primary concern is that I don't want to commit the changes now that we're beta without testing the changes in other platforms (however, I'll ask this on python-dev). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 23:19 Message: Logged In: YES user_id=31435 Sorry, I don't normally run on Unix, so I have no way to tell whether this fixes whatever the complaint is. Offhand it doesn't seem right to create a .bak file with default permissions either. For example, if only the user had permission to get at the original, letting the .bak file have broader permissions seems wrong too. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-25 22:57 Message: Logged In: YES user_id=752496 Attaching the file. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-24 17:20 Message: Logged In: YES user_id=752496 Instead of renaming the file and creating a new one, just copied the original and that's all (the target file is open with write mode, the content replaced, but the permissions stay). A patch is attached. Tim, I'm assigning this to you to doublecheck (not found unit tests for this script). . Facundo ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 From noreply at sourceforge.net Wed Oct 27 16:47:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 16:48:02 2004 Subject: [ python-Bugs-1050828 ] reindent.py strips execute permission on Unix Message-ID: Bugs item #1050828, was opened at 2004-10-20 11:48 Message generated for change (Comment added) made by dripton You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Ripton (dripton) Assigned to: Facundo Batista (facundobatista) Summary: reindent.py strips execute permission on Unix Initial Comment: Tools/scripts/reindent.py strips execute permission from its target file. If the tool detects that changes need to be made, then the target file is renamed to its original name + '.bak', then a new file is created with the original file's name and its whitespace-modified contents. No manipulation of the new file's permissions is done, so it ends up with the default for the user's umask. This makes reindent.py annoying to run in an automated fashion, if you rely on execute permission. The fix is trivial, except for portability. Here's the Posix-only version: + if os.name == "posix": + mode = os.stat(bak).st_mode + os.chmod(file, mode) Patch available upon request. ---------------------------------------------------------------------- >Comment By: David Ripton (dripton) Date: 2004-10-27 10:47 Message: Logged In: YES user_id=9425 s/wrong/right/ in my last comment. The target is more important than the backup, but we should go for the win/win and give both of them the correct permissions, since it's easy. ---------------------------------------------------------------------- Comment By: David Ripton (dripton) Date: 2004-10-27 10:45 Message: Logged In: YES user_id=9425 The patch is an improvement (better to have the wrong permissions on the target file than on the backup file), but I think it would be even better if the .bak file had the same permissions as the target file. How about using shutil.copy2 (which preserves permissions, among other things) instead of shutil.copy? Thanks. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-27 08:38 Message: Logged In: YES user_id=469548 The patch works for me (Linux 2.6.8-1-686 i686 GNU/Linux) and a coworker (Mac OS X 10.3). ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-26 20:44 Message: Logged In: YES user_id=752496 This fixes the problem: the original file metadata (permisions, etc) stays unchanged. The issue with the metadata of the bak file is none: the new file has the user defaults permissions, group and owner. In the previous version, the original file left with those defaults permissions, so security is not getting worse. The only left issue is, if the user wants to recover the data from the bak file, he would need to change its metadata (this is minor, if something goes wrong and the bak is needed, it's likely that the user will need a manual approach). My primary concern is that I don't want to commit the changes now that we're beta without testing the changes in other platforms (however, I'll ask this on python-dev). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-25 23:19 Message: Logged In: YES user_id=31435 Sorry, I don't normally run on Unix, so I have no way to tell whether this fixes whatever the complaint is. Offhand it doesn't seem right to create a .bak file with default permissions either. For example, if only the user had permission to get at the original, letting the .bak file have broader permissions seems wrong too. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-25 22:57 Message: Logged In: YES user_id=752496 Attaching the file. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-24 17:20 Message: Logged In: YES user_id=752496 Instead of renaming the file and creating a new one, just copied the original and that's all (the target file is open with write mode, the content replaced, but the permissions stay). A patch is attached. Tim, I'm assigning this to you to doublecheck (not found unit tests for this script). . Facundo ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 From noreply at sourceforge.net Wed Oct 27 16:56:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 16:56:54 2004 Subject: [ python-Bugs-1050828 ] reindent.py strips execute permission on Unix Message-ID: Bugs item #1050828, was opened at 2004-10-20 12:48 Message generated for change (Comment added) made by facundobatista You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Ripton (dripton) Assigned to: Facundo Batista (facundobatista) Summary: reindent.py strips execute permission on Unix Initial Comment: Tools/scripts/reindent.py strips execute permission from its target file. If the tool detects that changes need to be made, then the target file is renamed to its original name + '.bak', then a new file is created with the original file's name and its whitespace-modified contents. No manipulation of the new file's permissions is done, so it ends up with the default for the user's umask. This makes reindent.py annoying to run in an automated fashion, if you rely on execute permission. The fix is trivial, except for portability. Here's the Posix-only version: + if os.name == "posix": + mode = os.stat(bak).st_mode + os.chmod(file, mode) Patch available upon request. ---------------------------------------------------------------------- >Comment By: Facundo Batista (facundobatista) Date: 2004-10-27 11:56 Message: Logged In: YES user_id=752496 I thought about it. The problem with this approach is that it also copies the permissions, but not the owner/group ids (you get the defaults there). So, I think is better to leave the user defaults than mix the old permissions with new owner/group (which may lead to a security risk in some combinations). There's a better approach: to copy *all* the file metadata. But don't know how to do it in a portable way... ---------------------------------------------------------------------- Comment By: David Ripton (dripton) Date: 2004-10-27 11:47 Message: Logged In: YES user_id=9425 s/wrong/right/ in my last comment. The target is more important than the backup, but we should go for the win/win and give both of them the correct permissions, since it's easy. ---------------------------------------------------------------------- Comment By: David Ripton (dripton) Date: 2004-10-27 11:45 Message: Logged In: YES user_id=9425 The patch is an improvement (better to have the wrong permissions on the target file than on the backup file), but I think it would be even better if the .bak file had the same permissions as the target file. How about using shutil.copy2 (which preserves permissions, among other things) instead of shutil.copy? Thanks. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-27 09:38 Message: Logged In: YES user_id=469548 The patch works for me (Linux 2.6.8-1-686 i686 GNU/Linux) and a coworker (Mac OS X 10.3). ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-26 21:44 Message: Logged In: YES user_id=752496 This fixes the problem: the original file metadata (permisions, etc) stays unchanged. The issue with the metadata of the bak file is none: the new file has the user defaults permissions, group and owner. In the previous version, the original file left with those defaults permissions, so security is not getting worse. The only left issue is, if the user wants to recover the data from the bak file, he would need to change its metadata (this is minor, if something goes wrong and the bak is needed, it's likely that the user will need a manual approach). My primary concern is that I don't want to commit the changes now that we're beta without testing the changes in other platforms (however, I'll ask this on python-dev). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 00:19 Message: Logged In: YES user_id=31435 Sorry, I don't normally run on Unix, so I have no way to tell whether this fixes whatever the complaint is. Offhand it doesn't seem right to create a .bak file with default permissions either. For example, if only the user had permission to get at the original, letting the .bak file have broader permissions seems wrong too. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-25 23:57 Message: Logged In: YES user_id=752496 Attaching the file. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-24 18:20 Message: Logged In: YES user_id=752496 Instead of renaming the file and creating a new one, just copied the original and that's all (the target file is open with write mode, the content replaced, but the permissions stay). A patch is attached. Tim, I'm assigning this to you to doublecheck (not found unit tests for this script). . Facundo ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 From noreply at sourceforge.net Wed Oct 27 17:29:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 17:29:35 2004 Subject: [ python-Bugs-1050828 ] reindent.py strips execute permission on Unix Message-ID: Bugs item #1050828, was opened at 2004-10-20 17:48 Message generated for change (Comment added) made by sjoerd You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Ripton (dripton) Assigned to: Facundo Batista (facundobatista) Summary: reindent.py strips execute permission on Unix Initial Comment: Tools/scripts/reindent.py strips execute permission from its target file. If the tool detects that changes need to be made, then the target file is renamed to its original name + '.bak', then a new file is created with the original file's name and its whitespace-modified contents. No manipulation of the new file's permissions is done, so it ends up with the default for the user's umask. This makes reindent.py annoying to run in an automated fashion, if you rely on execute permission. The fix is trivial, except for portability. Here's the Posix-only version: + if os.name == "posix": + mode = os.stat(bak).st_mode + os.chmod(file, mode) Patch available upon request. ---------------------------------------------------------------------- >Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-27 17:29 Message: Logged In: YES user_id=43607 Then perhaps it should do in-place replacement, so first *copy* the original to the backup, and then overwrite the original. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-27 16:56 Message: Logged In: YES user_id=752496 I thought about it. The problem with this approach is that it also copies the permissions, but not the owner/group ids (you get the defaults there). So, I think is better to leave the user defaults than mix the old permissions with new owner/group (which may lead to a security risk in some combinations). There's a better approach: to copy *all* the file metadata. But don't know how to do it in a portable way... ---------------------------------------------------------------------- Comment By: David Ripton (dripton) Date: 2004-10-27 16:47 Message: Logged In: YES user_id=9425 s/wrong/right/ in my last comment. The target is more important than the backup, but we should go for the win/win and give both of them the correct permissions, since it's easy. ---------------------------------------------------------------------- Comment By: David Ripton (dripton) Date: 2004-10-27 16:45 Message: Logged In: YES user_id=9425 The patch is an improvement (better to have the wrong permissions on the target file than on the backup file), but I think it would be even better if the .bak file had the same permissions as the target file. How about using shutil.copy2 (which preserves permissions, among other things) instead of shutil.copy? Thanks. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-27 14:38 Message: Logged In: YES user_id=469548 The patch works for me (Linux 2.6.8-1-686 i686 GNU/Linux) and a coworker (Mac OS X 10.3). ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-27 02:44 Message: Logged In: YES user_id=752496 This fixes the problem: the original file metadata (permisions, etc) stays unchanged. The issue with the metadata of the bak file is none: the new file has the user defaults permissions, group and owner. In the previous version, the original file left with those defaults permissions, so security is not getting worse. The only left issue is, if the user wants to recover the data from the bak file, he would need to change its metadata (this is minor, if something goes wrong and the bak is needed, it's likely that the user will need a manual approach). My primary concern is that I don't want to commit the changes now that we're beta without testing the changes in other platforms (however, I'll ask this on python-dev). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 05:19 Message: Logged In: YES user_id=31435 Sorry, I don't normally run on Unix, so I have no way to tell whether this fixes whatever the complaint is. Offhand it doesn't seem right to create a .bak file with default permissions either. For example, if only the user had permission to get at the original, letting the .bak file have broader permissions seems wrong too. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-26 04:57 Message: Logged In: YES user_id=752496 Attaching the file. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-24 23:20 Message: Logged In: YES user_id=752496 Instead of renaming the file and creating a new one, just copied the original and that's all (the target file is open with write mode, the content replaced, but the permissions stay). A patch is attached. Tim, I'm assigning this to you to doublecheck (not found unit tests for this script). . Facundo ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 From noreply at sourceforge.net Wed Oct 27 17:56:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 17:56:43 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Open Resolution: None Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-27 11:56 Message: Logged In: YES user_id=6380 - I don't like the nested function -- Python's function call overhead is significant and I worry about that when deleting massive trees. (I want to be able to "rm -rf /*" efficiently. :-) - The err instance instead of sys.exc_info() was an oversight. The docstring repeats this mistake. - I don't think the lstat() call is unclean, and again I prefer performance. - Here's an additional trick to reduce the code duplication (see upload rmtree-ng.py): At the top of the function put: if ignore_errors: ....def onerror(*args): pass elif onerror is None: ....def onerror(*args): raise Then all the except blocks can be collapsed to just the onerror() call. (I care less about the performance in case there's an error.) ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-27 05:57 Message: Logged In: YES user_id=469548 Yes, I think this is the right way to go about it. I'd personally make a couple of changes, but those are up to your discretion: * Factor out the try: except:'s to a nested function. * Use sys.exc_info() instead of the exception for backwards compatibility. * Use 'os.path.isdir(fullname) and not os.path.islink(fullname)' expression instead of the lstat call. I think it more clearly communicates what we want, even though it's more stat calls. Latest version attached (rmtree-refactored.py). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-26 12:08 Message: Logged In: YES user_id=6380 How about this no-nonsense version? (rmtree.txt upload) The only thing you could quibble about is what to do when os.lstat fails. I chose backward compatibility. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-23 14:25 Message: Logged In: YES user_id=6380 BTW, if you want the correct path in the onerror calls for os.listdir, your best bet is not to use os.walk but to write something similar that suits our purpose. After you remove the features we don't use, it's really only a few lines of code. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-23 13:56 Message: Logged In: YES user_id=6380 I disagree. The docs are wrong because a failing listdir wasn't anticipated at first when this function was written. A failing listdir is a common failure case; I don't see how the os.path calls in os.walk could ever fail (since they all catch os.error). ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-22 23:01 Message: Logged In: YES user_id=469548 You know what: I think we should remove the os.listdir error handling: * The docs only specify "errors resulting from *failed removals*" as being handled. If they didn't we should worry about OSErrors from the os.path.* calls in os.walk() too. * The os.listdir error handling hasn't been in a final release, so no backwards compatibility problems yet. * We can't get the path right for os.listdir. * The problem also existed before the rewrite using os.walk(), so reverting to that is not a solution. Patch (the last?) is attached. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-21 00:42 Message: Logged In: YES user_id=6380 Yes. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-20 16:30 Message: Logged In: YES user_id=469548 With 'be careful, it may not exist.', do you mean rmtree can't rely on exc.filename being set? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 11:18 Message: Logged In: YES user_id=6380 This looks fine, and can be checked in. There's one minor remaining issue, but I don't see how to fix it without changing the signature for the onerror function that os.walk() takes, and that's a no-no: when the problem is a directory permission preventing os.listdir() to work, the path passed to the rmtree caller's onerror function is always the toplevel path, rather than the directory that couldn't be listed. The real path is included in the error message. (Hmm, perhaps you could get it out of there? I think it's an attribute of the exception object; but be careful, it may not exist.) It's unfortunate that the error argument to the onerror function is a sys.exc_info() triple rather than an exception object (calling sys.exc_info() is expensive), but that's an API choice from the past that we can't change until Python 3.0. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 04:57 Message: Logged In: YES user_id=469548 A new patch is attached. I factored out the handle_call_error method so exception handling is more local. Also, errors in os.walk were already being handled using the _raise_err/except OSError combination, but we now do it using the onerror argument to os.walk. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:42 Message: Logged In: YES user_id=6380 (Actually, os.walk has an onerror argument, it just uses a different convention and default, so we could thread this through easily.) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 18:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 02:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Wed Oct 27 18:07:34 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 18:07:42 2004 Subject: [ python-Bugs-1050828 ] reindent.py strips execute permission on Unix Message-ID: Bugs item #1050828, was opened at 2004-10-20 12:48 Message generated for change (Comment added) made by facundobatista You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Ripton (dripton) Assigned to: Facundo Batista (facundobatista) Summary: reindent.py strips execute permission on Unix Initial Comment: Tools/scripts/reindent.py strips execute permission from its target file. If the tool detects that changes need to be made, then the target file is renamed to its original name + '.bak', then a new file is created with the original file's name and its whitespace-modified contents. No manipulation of the new file's permissions is done, so it ends up with the default for the user's umask. This makes reindent.py annoying to run in an automated fashion, if you rely on execute permission. The fix is trivial, except for portability. Here's the Posix-only version: + if os.name == "posix": + mode = os.stat(bak).st_mode + os.chmod(file, mode) Patch available upon request. ---------------------------------------------------------------------- >Comment By: Facundo Batista (facundobatista) Date: 2004-10-27 13:07 Message: Logged In: YES user_id=752496 That is *exactly* what the patch does, but David complained about the metadata of the copy. ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-27 12:29 Message: Logged In: YES user_id=43607 Then perhaps it should do in-place replacement, so first *copy* the original to the backup, and then overwrite the original. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-27 11:56 Message: Logged In: YES user_id=752496 I thought about it. The problem with this approach is that it also copies the permissions, but not the owner/group ids (you get the defaults there). So, I think is better to leave the user defaults than mix the old permissions with new owner/group (which may lead to a security risk in some combinations). There's a better approach: to copy *all* the file metadata. But don't know how to do it in a portable way... ---------------------------------------------------------------------- Comment By: David Ripton (dripton) Date: 2004-10-27 11:47 Message: Logged In: YES user_id=9425 s/wrong/right/ in my last comment. The target is more important than the backup, but we should go for the win/win and give both of them the correct permissions, since it's easy. ---------------------------------------------------------------------- Comment By: David Ripton (dripton) Date: 2004-10-27 11:45 Message: Logged In: YES user_id=9425 The patch is an improvement (better to have the wrong permissions on the target file than on the backup file), but I think it would be even better if the .bak file had the same permissions as the target file. How about using shutil.copy2 (which preserves permissions, among other things) instead of shutil.copy? Thanks. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-27 09:38 Message: Logged In: YES user_id=469548 The patch works for me (Linux 2.6.8-1-686 i686 GNU/Linux) and a coworker (Mac OS X 10.3). ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-26 21:44 Message: Logged In: YES user_id=752496 This fixes the problem: the original file metadata (permisions, etc) stays unchanged. The issue with the metadata of the bak file is none: the new file has the user defaults permissions, group and owner. In the previous version, the original file left with those defaults permissions, so security is not getting worse. The only left issue is, if the user wants to recover the data from the bak file, he would need to change its metadata (this is minor, if something goes wrong and the bak is needed, it's likely that the user will need a manual approach). My primary concern is that I don't want to commit the changes now that we're beta without testing the changes in other platforms (however, I'll ask this on python-dev). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 00:19 Message: Logged In: YES user_id=31435 Sorry, I don't normally run on Unix, so I have no way to tell whether this fixes whatever the complaint is. Offhand it doesn't seem right to create a .bak file with default permissions either. For example, if only the user had permission to get at the original, letting the .bak file have broader permissions seems wrong too. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-25 23:57 Message: Logged In: YES user_id=752496 Attaching the file. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-24 18:20 Message: Logged In: YES user_id=752496 Instead of renaming the file and creating a new one, just copied the original and that's all (the target file is open with write mode, the content replaced, but the permissions stay). A patch is attached. Tim, I'm assigning this to you to doublecheck (not found unit tests for this script). . Facundo ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 From noreply at sourceforge.net Wed Oct 27 19:22:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 19:22:32 2004 Subject: [ python-Bugs-1036626 ] file.next() info hidden Message-ID: Bugs item #1036626, was opened at 2004-09-28 17:42 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1036626&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Nick Jacobson (nickjacobson) Assigned to: Nobody/Anonymous (nobody) Summary: file.next() info hidden Initial Comment: (A.M. Kuchling told me to post this bug report.) In section 2.3.8, "File Objects", of the Python Library Reference, the following section is given for file.next(): << In order to make a for loop the most efficient way of looping over the lines of a file (a very common operation), the next() method uses a hidden read-ahead buffer. As a consequence of using a read-ahead buffer, combining next() with other file methods (like readline()) does not work right. >> This information is necessary for anyone who mixes "for line in file:" with read()/tell()/etc. However, in the statement "for line in file:", next() is called implicitly, and therefore it may not occur to the programmer to find the information there. Therefore the documentation should be reorganized. One solution is to have the read()/tell()/etc. methods refer to the above section. i.e. "Don't use this method with next() or 'for line in file:'; see next()." A second solution is to move the above section to before the methods section (i.e. after the second paragraph after the 2.3.8 header), and to mention 'for line in file:'. i.e. "As a consequence of using a read-ahead buffer, combining next() or 'for line in file:' with other file methods (like readline()) does not work right." (Also, saying "with other file methods (like readline())" is too vague. IMHO it should specify which ones not to use. That's debatable though.) ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-27 12:22 Message: Logged In: YES user_id=80475 IMO, the docs are fine as-is. Sometimes, trying to be over specific makes the docs less usable rather than more. The current wording pretty much covers why the iterator protocol is at odds with the readline() style methods. Of course, this could be argued to death. So, I'll leave the report open so that others can express their take. ---------------------------------------------------------------------- Comment By: engelbert gruber (grubert) Date: 2004-10-05 08:55 Message: Logged In: YES user_id=147070 see also http://www.python.org/sf/645594 for lin in file: file.tell() tells wrong ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1036626&group_id=5470 From noreply at sourceforge.net Wed Oct 27 19:34:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 19:34:47 2004 Subject: [ python-Bugs-1050828 ] reindent.py strips execute permission on Unix Message-ID: Bugs item #1050828, was opened at 2004-10-20 17:48 Message generated for change (Comment added) made by sjoerd You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Ripton (dripton) Assigned to: Facundo Batista (facundobatista) Summary: reindent.py strips execute permission on Unix Initial Comment: Tools/scripts/reindent.py strips execute permission from its target file. If the tool detects that changes need to be made, then the target file is renamed to its original name + '.bak', then a new file is created with the original file's name and its whitespace-modified contents. No manipulation of the new file's permissions is done, so it ends up with the default for the user's umask. This makes reindent.py annoying to run in an automated fashion, if you rely on execute permission. The fix is trivial, except for portability. Here's the Posix-only version: + if os.name == "posix": + mode = os.stat(bak).st_mode + os.chmod(file, mode) Patch available upon request. ---------------------------------------------------------------------- >Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-27 19:34 Message: Logged In: YES user_id=43607 Now that I looked at the patch, I have one concern. I don't mind if the backup copy doesn't have the same permissions as the original, especially not if you can't guarantee that the owner/group doesn't change. However, I do mind if the backup copy has *more* permissions than the original. Consider a file that was only readable by the owner (probably for some good reason), and because of a permissive umask, the backup copy all of a sudden is world-readable. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-27 18:07 Message: Logged In: YES user_id=752496 That is *exactly* what the patch does, but David complained about the metadata of the copy. ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-27 17:29 Message: Logged In: YES user_id=43607 Then perhaps it should do in-place replacement, so first *copy* the original to the backup, and then overwrite the original. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-27 16:56 Message: Logged In: YES user_id=752496 I thought about it. The problem with this approach is that it also copies the permissions, but not the owner/group ids (you get the defaults there). So, I think is better to leave the user defaults than mix the old permissions with new owner/group (which may lead to a security risk in some combinations). There's a better approach: to copy *all* the file metadata. But don't know how to do it in a portable way... ---------------------------------------------------------------------- Comment By: David Ripton (dripton) Date: 2004-10-27 16:47 Message: Logged In: YES user_id=9425 s/wrong/right/ in my last comment. The target is more important than the backup, but we should go for the win/win and give both of them the correct permissions, since it's easy. ---------------------------------------------------------------------- Comment By: David Ripton (dripton) Date: 2004-10-27 16:45 Message: Logged In: YES user_id=9425 The patch is an improvement (better to have the wrong permissions on the target file than on the backup file), but I think it would be even better if the .bak file had the same permissions as the target file. How about using shutil.copy2 (which preserves permissions, among other things) instead of shutil.copy? Thanks. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-27 14:38 Message: Logged In: YES user_id=469548 The patch works for me (Linux 2.6.8-1-686 i686 GNU/Linux) and a coworker (Mac OS X 10.3). ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-27 02:44 Message: Logged In: YES user_id=752496 This fixes the problem: the original file metadata (permisions, etc) stays unchanged. The issue with the metadata of the bak file is none: the new file has the user defaults permissions, group and owner. In the previous version, the original file left with those defaults permissions, so security is not getting worse. The only left issue is, if the user wants to recover the data from the bak file, he would need to change its metadata (this is minor, if something goes wrong and the bak is needed, it's likely that the user will need a manual approach). My primary concern is that I don't want to commit the changes now that we're beta without testing the changes in other platforms (however, I'll ask this on python-dev). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 05:19 Message: Logged In: YES user_id=31435 Sorry, I don't normally run on Unix, so I have no way to tell whether this fixes whatever the complaint is. Offhand it doesn't seem right to create a .bak file with default permissions either. For example, if only the user had permission to get at the original, letting the .bak file have broader permissions seems wrong too. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-26 04:57 Message: Logged In: YES user_id=752496 Attaching the file. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-24 23:20 Message: Logged In: YES user_id=752496 Instead of renaming the file and creating a new one, just copied the original and that's all (the target file is open with write mode, the content replaced, but the permissions stay). A patch is attached. Tim, I'm assigning this to you to doublecheck (not found unit tests for this script). . Facundo ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 From noreply at sourceforge.net Wed Oct 27 20:11:09 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 20:11:37 2004 Subject: [ python-Bugs-1054943 ] Python may contain NFC/NFKC bug per Unicode PRI #29 Message-ID: Bugs item #1054943, was opened at 2004-10-27 01:58 Message generated for change (Comment added) made by lemburg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054943&group_id=5470 Category: Unicode Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Rick McGowan (rick_mcgowan) >Assigned to: Martin v. L?wis (loewis) Summary: Python may contain NFC/NFKC bug per Unicode PRI #29 Initial Comment: The Unicode Technical Committee posted Public Review Issue #29, describing a bug in the documentation of NFC and NFKC in the text of UAX #15 Unicode Normalization Forms. I have examined unicodedata.c in the Python implementation (2.3.4) and it appears the implementation of normalization in Python 2.3.4 may have the bug therein described. Please see the description of the bug and the textual fix that is being made to UAX #15, at the URL: http://www.unicode.org/review/pr-29.html The bug is in the definition of rule D2, affecting the characters "blocked" during re-composition. You may contact me by e-mail, or fill out the Unicode.org error reporting form if you have any questions or concerns. Since Python uses Unicode internally, it may also be wise to have someone from the Python development community on the Unicode Consortium's notification list to receive immediate notifications of public review issues, bugs, and other announcements affecting implementation of the standard. ---------------------------------------------------------------------- >Comment By: M.-A. Lemburg (lemburg) Date: 2004-10-27 20:11 Message: Logged In: YES user_id=38388 Thanks for submitting a bug report. The problem does indeed occur in the Python normalization code: >>> unicodedata.normalize('NFC', u'\u0B47\u0300\u0B3E') u'\u0b4b\u0300' I think the following line in unicodedata.c needs to be changed: if (comb1 && comb == comb1) { /* Character is blocked. */ i1++; continue; } to if (comb && (comb1 == 0 || comb == comb1)) { /* Character is blocked. */ i1++; continue; } Martin, what do you think ? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054943&group_id=5470 From noreply at sourceforge.net Wed Oct 27 20:47:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 20:47:39 2004 Subject: [ python-Bugs-1055540 ] bdist_wininst broken for pure Python distributions Message-ID: Bugs item #1055540, was opened at 2004-10-27 20:47 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055540&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Thomas Heller (theller) Assigned to: Thomas Heller (theller) Summary: bdist_wininst broken for pure Python distributions Initial Comment: Trying to create a pure python distribution windows installer with 2.4b1 bdist_wininst command fails with an assertion. The attached patch seems to fix the problem. Mark, I believe you changed this code, can you proofread the patch, and then assign back to me (or check it in directly)? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055540&group_id=5470 From noreply at sourceforge.net Wed Oct 27 20:48:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 20:48:22 2004 Subject: [ python-Bugs-1055540 ] bdist_wininst broken for pure Python distributions Message-ID: Bugs item #1055540, was opened at 2004-10-27 20:47 Message generated for change (Settings changed) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055540&group_id=5470 >Category: Distutils >Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Thomas Heller (theller) >Assigned to: Mark Hammond (mhammond) Summary: bdist_wininst broken for pure Python distributions Initial Comment: Trying to create a pure python distribution windows installer with 2.4b1 bdist_wininst command fails with an assertion. The attached patch seems to fix the problem. Mark, I believe you changed this code, can you proofread the patch, and then assign back to me (or check it in directly)? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055540&group_id=5470 From noreply at sourceforge.net Wed Oct 27 22:11:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 22:11:37 2004 Subject: [ python-Bugs-1054943 ] Python may contain NFC/NFKC bug per Unicode PRI #29 Message-ID: Bugs item #1054943, was opened at 2004-10-26 23:58 Message generated for change (Comment added) made by rick_mcgowan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054943&group_id=5470 Category: Unicode Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Rick McGowan (rick_mcgowan) Assigned to: Martin v. L?wis (loewis) Summary: Python may contain NFC/NFKC bug per Unicode PRI #29 Initial Comment: The Unicode Technical Committee posted Public Review Issue #29, describing a bug in the documentation of NFC and NFKC in the text of UAX #15 Unicode Normalization Forms. I have examined unicodedata.c in the Python implementation (2.3.4) and it appears the implementation of normalization in Python 2.3.4 may have the bug therein described. Please see the description of the bug and the textual fix that is being made to UAX #15, at the URL: http://www.unicode.org/review/pr-29.html The bug is in the definition of rule D2, affecting the characters "blocked" during re-composition. You may contact me by e-mail, or fill out the Unicode.org error reporting form if you have any questions or concerns. Since Python uses Unicode internally, it may also be wise to have someone from the Python development community on the Unicode Consortium's notification list to receive immediate notifications of public review issues, bugs, and other announcements affecting implementation of the standard. ---------------------------------------------------------------------- >Comment By: Rick McGowan (rick_mcgowan) Date: 2004-10-27 20:11 Message: Logged In: YES user_id=1146994 Thanks all for quick reply. My initial thoughts regarding a fix were as below. The relevant piece of code seems to be in function "nfc_nfkc()" in the file unicodedata.c > if (comb1 && comb == comb1) { > /* Character is blocked. */ > i1++; > continue; > } That should possibly be changed to: > if (comb1 && (comb <= comb1)) { > /* Character is blocked. */ > i1++; > continue; > } because the new spec says "either B is a starter or it has the same or higher combining class as C". ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-10-27 18:11 Message: Logged In: YES user_id=38388 Thanks for submitting a bug report. The problem does indeed occur in the Python normalization code: >>> unicodedata.normalize('NFC', u'\u0B47\u0300\u0B3E') u'\u0b4b\u0300' I think the following line in unicodedata.c needs to be changed: if (comb1 && comb == comb1) { /* Character is blocked. */ i1++; continue; } to if (comb && (comb1 == 0 || comb == comb1)) { /* Character is blocked. */ i1++; continue; } Martin, what do you think ? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054943&group_id=5470 From noreply at sourceforge.net Wed Oct 27 23:55:09 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed Oct 27 23:55:31 2004 Subject: [ python-Bugs-1055540 ] bdist_wininst broken for pure Python distributions Message-ID: Bugs item #1055540, was opened at 2004-10-28 04:47 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055540&group_id=5470 Category: Distutils Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Thomas Heller (theller) Assigned to: Mark Hammond (mhammond) Summary: bdist_wininst broken for pure Python distributions Initial Comment: Trying to create a pure python distribution windows installer with 2.4b1 bdist_wininst command fails with an assertion. The attached patch seems to fix the problem. Mark, I believe you changed this code, can you proofread the patch, and then assign back to me (or check it in directly)? ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-10-28 07:55 Message: Logged In: YES user_id=14198 Checking in bdist_wininst.py; new revision: 1.55; previous revision: 1.54 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055540&group_id=5470 From noreply at sourceforge.net Thu Oct 28 03:58:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 03:58:56 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Tim Peters (tim_one) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Thu Oct 28 04:28:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 04:28:10 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Tim Peters (tim_one) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Thu Oct 28 04:37:08 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 04:37:12 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Tim Peters (tim_one) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Thu Oct 28 06:51:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 06:51:51 2004 Subject: [ python-Bugs-1045381 ] strptime doesn't work with %U Message-ID: Bugs item #1045381, was opened at 2004-10-12 07:04 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 7 Submitted By: Sebastien JUST (sebastienjust) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't work with %U Initial Comment: It seems that strptime() ignores %U. For example when trying to get the first day of the 42th week of year 2004. Please test on the command line : import time time.strftime("%Y-%m-%d",time.strptime("2004 42 1","%Y %U %w")) the result is 2004-01-01 and not 2004-10-11 seems that strptime() is ignoring %U indications. Works fine on Python 2.2, bad on Python 2.3.3 , 2.3.4 and 2.4a1. Tested on Fedora Core 2. Thanks ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-10-27 21:51 Message: Logged In: YES user_id=357491 For Lib/_strptime.py, rev. 1.38 on HEAD and 1.23.4.7 on 2.3 and Lib/test/ test_strptime.py, rev. 1.29 on HEAD and 1.23.4.6 on 2.3 have the fix and extra test cases. Bug fixed and closed... again. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-25 20:01 Message: Logged In: YES user_id=357491 OK, unless someone speaks up about this patch I will apply it sometime this week (soonest being 2004-10-26 in the evening PDT). ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-19 13:10 Message: Logged In: YES user_id=357491 Nick is seeing things the way I thought they are supposed to be; how can ending on a Monday make any difference for %U or %W? Both start at after the beginning of the week, right? Well, as George was pointing out, it shifts everything by a full week depending on the directive when the year starts on a Monday. So under %U week 1 doesn't start until the first Sunday, while under %W week 1 starts on Jan 1. So the issue is when a year starts and end on a Monday, we are looking at Monday, but %U is used. So the new patch attempts to handle this case. I am now testing against the following dates: test_helper((1901, 1, 3), "week 0") test_helper((1901, 1, 8), "common case") test_helper((1901, 1, 13), "day on Sunday") test_helper((1901, 1, 14), "day on Monday") test_helper((1905, 1, 1), "Jan 1 on Sunday") test_helper((1906, 1, 1), "Jan 1 on Monday") test_helper((1906, 1, 7), "first Sunday in a year starting on Monday") test_helper((1905, 12, 31), "Dec 31 on Sunday") test_helper((1906, 12, 31), "Dec 31 on Monday") test_helper((2008, 12, 29), "Monday in the last week of the year") test_helper((2008, 12, 22), "Monday in the second-to-last week of the " "year") test_helper((1978, 10, 23), "randomly chosen date") test_helper((2004, 12, 18), "randomly chosen date") test_helper((1978, 10, 23), "year starting and ending on Monday while " "date not on Sunday or Monday") test_helper((1917, 12, 17), "year starting and ending on Monday with " "a Monday not at the beginning or end " "of the year") test_helper((1917, 12, 31), "Dec 31 on Monday with year starting and " "ending on Monday") ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 06:18 Message: Logged In: YES user_id=1038590 Under %W, the first week contains a full 7 days, but it only contains 6 days under %U. Either way, we end up with a week 53 - it contains 1 day for %W, but 2 days for %U. In both cases, December 31st is the Monday of week 53. Unless I'm misunderstanding how this is meant to work, and Week 1 is always the first full week of the year, with an optional Week 0 before it (which would always have fewer than 7 days, and doesn't exist at all if the year and the week start on the same day). If I *am* misunderstanding, then that's the bug in strptime - it currently works in accordance with my understanding. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2004-10-19 05:18 Message: Logged In: YES user_id=671362 > December 31 should be week 53 under %U as well. I doubt it. Year 1917 begins on Monday and ends on Monday. So "%U"(Sunday as the first day of the week) should return 52 and "W"(Monday as the first day of the week) should return 53. $ cal 1 1917 January 1917 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 $ cal 12 1917 December 1917 Su Mo Tu We Th Fr Sa 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:50 Message: Logged In: YES user_id=1038590 Any bug is in datetime.date, not strptime. I tried datetime.date(1917, 12, 31).strftime("%Y %U %w") and saw Brett's incorrect calculation of the week. ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %W %w')" 1917 53 1 ./python -c "import datetime; print datetime.date(1917, 12, 31).strftime('%Y %U %w')" 1917 52 1 December 31 should be week 53 under %U as well. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:42 Message: Logged In: YES user_id=1038590 I forgot to add that I'm on Linux (Fedora Core 3 Test 1 + misc updates) ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:39 Message: Logged In: YES user_id=1038590 Taking out the "add one to the week" condition, I get the following for 1906 and 1917: ./python -c "import time; print time.strptime('1906 53 1', '%Y %W %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1906 53 1', '%Y %U %w')" (1906, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %W %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) ./python -c "import time; print time.strptime('1917 53 1', '%Y %U %w')" (1917, 12, 31, 0, 0, 0, 0, 365, -1) So, I'm more than a little curious about the details of the "bug" that was being fixed. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 03:03 Message: Logged In: YES user_id=1038590 Scratch the last comment - I missed the wee_of_year adjustment at the top of the function. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-10-19 02:57 Message: Logged In: YES user_id=1038590 The calculation of 'preceeding_days' looks incorrect. It assumes that the week starts on day 0 - it needs to adjust for when "week_of year_start" is 6. ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-19 00:31 Message: Logged In: YES user_id=552329 Patch works for that case, yes. However... [from python-dev] > the test case I was using that triggered > the need for that is 1906-12-31 > which is a Monday but changes what week > it is based on whether you use U or W. > which makes no sense since both U and W > should consider it the same week. > Had the same result for 1917-12-31. Something's still not right here (this is with the patch): >>> time.strptime("1917 51 1", "%Y %U %w") (1917, 12, 17, 0, 0, 0, 0, 351, -1) >>> time.strptime("1917 52 1", "%Y %U %w") (1917, 12, 31, 0, 0, 0, 0, 365, -1) 1917 both started and ended on a Monday, so there are 53 U weeks, right? (where the last 'week' is only one day). So 31/12/1917 should be U=53, W=52 ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-18 11:45 Message: Logged In: YES user_id=357491 Reopening to deal with the error Tony caught. Tony, can you apply the included patch and verify it works for you okay? I added another if to track back a week if the calculation gets pushed past 366 days. That does fix your error. And if you can think of any other edge cases let me know. I think that should cover everything. And yes, the example for the OP is wrong:: >>> datetime.date(2004, 10, 11).strftime("%Y %U %w") '2004 41 1' ---------------------------------------------------------------------- Comment By: Tony Meyer (anadelonbrin) Date: 2004-10-17 20:50 Message: Logged In: YES user_id=552329 FWIW, with the example given the correct answer is 2004-10-18, which CVS now gives, not 2004-10-11. However, this doesn't look right: >>> time.strptime("2008 52 1", "%Y %U %w") (2009, 1, 5, 0, 0, 0, 0, 371, -1) It ought to be >>> time.strptime("2008 52 1", "%Y %U %w") (2008, 12, 29, 0, 0, 0, 0, 364, -1) By my figuring. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-17 18:56 Message: Logged In: YES user_id=357491 In rev. 1.36 in HEAD has the fix as well as rev. 1.23.4.6 for 2.3 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-17 16:10 Message: Logged In: YES user_id=357491 OK, I have the algorithm written. Now I am waiting for python-dev to make a decision on whether this should go into 2.4b2 or wait until 2.5 . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Forgot the link to the glibc page: http://www.gnu.org/software/libc/ manual/html_node/Low-Level-Time-String-Parsing.html#Low- Level%20Time%20String%20Parsing ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:57 Message: Logged In: YES user_id=357491 Well, it looks like glibc 2.3.x doesn't even support %U or %W for strptime(); this might take a while to implement... ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-10-12 18:42 Message: Logged In: YES user_id=357491 Yeah, right now it isn't supported since the calculation, at least when I first implemented strptime() seemed useless in terms of reversing back into a time tuple. Guess there at least one way there is enough info to make it useful. Now I just need to figure out how to make the calculation work. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1045381&group_id=5470 From noreply at sourceforge.net Thu Oct 28 06:59:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 06:59:38 2004 Subject: [ python-Bugs-1055864 ] HTMLParser not compliant to XHTML spec Message-ID: Bugs item #1055864, was opened at 2004-10-27 21:59 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055864&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Luke Bradley (neptune235) Assigned to: Nobody/Anonymous (nobody) Summary: HTMLParser not compliant to XHTML spec Initial Comment: HTMLParser has a problem related to the fact that is doesn't seem to comply to the spec for XHTML. What I am refering to can be read about here: http://www.w3.org/TR/xhtml1/#h-4.8 In a nutshell, HTMLParser doesn't treat data inside 'script' or 'style' elements as #PCDATA, but rather behaves like an HTML 4 parser even for XHTML documents, parsing only end tags. As a result, entity references in javascript are not converted as they should be. XHTML authors writing to spec can expect entities in script sections of XHTML documents to be converted if the script is not explicitly escaped as a CDATA section. which brings up problem two, That sections explicitly escaped as CDATA are also parsed as HTML 4 'script' and 'style' sections...End tags are parsed... My understanding is that this is bad as well: http://www.w3.org/TR/2004/REC-xml-20040204/#dt-cdsection because CDend is the only thing that's supposed to be parsed in a CDATA section for all XML documents? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055864&group_id=5470 From noreply at sourceforge.net Thu Oct 28 10:06:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 10:06:25 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Tim Peters (tim_one) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Thu Oct 28 10:31:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 10:32:15 2004 Subject: [ python-Bugs-1055864 ] HTMLParser not compliant to XHTML spec Message-ID: Bugs item #1055864, was opened at 2004-10-27 21:59 Message generated for change (Comment added) made by neptune235 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055864&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Luke Bradley (neptune235) Assigned to: Nobody/Anonymous (nobody) Summary: HTMLParser not compliant to XHTML spec Initial Comment: HTMLParser has a problem related to the fact that is doesn't seem to comply to the spec for XHTML. What I am refering to can be read about here: http://www.w3.org/TR/xhtml1/#h-4.8 In a nutshell, HTMLParser doesn't treat data inside 'script' or 'style' elements as #PCDATA, but rather behaves like an HTML 4 parser even for XHTML documents, parsing only end tags. As a result, entity references in javascript are not converted as they should be. XHTML authors writing to spec can expect entities in script sections of XHTML documents to be converted if the script is not explicitly escaped as a CDATA section. which brings up problem two, That sections explicitly escaped as CDATA are also parsed as HTML 4 'script' and 'style' sections...End tags are parsed... My understanding is that this is bad as well: http://www.w3.org/TR/2004/REC-xml-20040204/#dt-cdsection because CDend is the only thing that's supposed to be parsed in a CDATA section for all XML documents? ---------------------------------------------------------------------- >Comment By: Luke Bradley (neptune235) Date: 2004-10-28 01:31 Message: Logged In: YES user_id=178561 I also reported bug 1051840. I discovered this when I was looking for a universal way to handle all the wierd things people do with their script sections on HTML/XHTML pages on the net. I've ended up modifying HTMLParser.py so that the HTMLParser class has an extra attribute called last_match, which is the exact string of HTML that whatever handler event is being called for...So that putting: sys.stdout.write(self.last_match) or sys.stdout.write(self.get_last_match()) for every handler event (except handle_data, which can be directly outputted) will output the page exactly as was inputted. This allows me to handle all oddities in people's code at the level of my handler, without changing HTMLParser in any other way... Here's the code, attached. Not that you care, but on the off chance that you guys might want to think about doing something like this....:) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055864&group_id=5470 From noreply at sourceforge.net Thu Oct 28 10:56:44 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 10:56:50 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Tim Peters (tim_one) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Thu Oct 28 13:28:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 13:28:38 2004 Subject: [ python-Bugs-1052242 ] Attempt to run all atexit handlers Message-ID: Bugs item #1052242, was opened at 2004-10-22 09:31 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Greg Chapman (glchapman) Assigned to: Skip Montanaro (montanaro) Summary: Attempt to run all atexit handlers Initial Comment: Currently, if an atexit handler raises an exception during _run_exitfuncs, none of the remaining handlers in the _exithandlers list gets run. I would prefer it if _run_exitfuncs did something like: while _exithandlers: func, targs, kargs = _exithandlers.pop() try: func(*targs, **kargs) except SystemExit: raise except: import sys, traceback print >> sys.stderr, "Error in sys.exitfunc:" traceback.print_exc() The SystemExit except clause is there because it looks like (given its special treatment in call_sys_exitfunc) SystemExit is envisioned as a way of intentionally breaking out of the exithandlers loop. ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2004-10-28 06:28 Message: Logged In: YES user_id=44345 Okay, Guido has pronounced. My idea for preserving compatibility is out. Now, I wonder about Greg's SystemExit behavior... Should we defer it until the loop is complete and reraise it then? ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-10-27 09:44 Message: Logged In: YES user_id=44345 I agree with you in principle, but you'd change behavior and possibly break existing code that relies on the current implicit semantics, hence my suggestion to use a variable to control the behavior, with it set to preserve the current behavior by default. I'll post a request to python-dev soliciting input. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-27 00:46 Message: Logged In: YES user_id=80475 I prefer the original proposal that leaves the API untouched. Irrespective of ordering, my thought is that all exit handlers are probably expected to run (needed for important cleanup, resource freeing, etc) and may be independent of one another. So, all of them that can run, probably should. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-10-24 12:21 Message: Logged In: YES user_id=44345 OTOH, since the ordering of the exit handlers is not guaranteed, exiting after one exception is also reasonable behavior. Attached is a diff that does two things: adds a set_run_all function and converts the functionality into an ExitHandler class to keep from all the different bits of global state. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-23 19:56 Message: Logged In: YES user_id=80475 This could be considered a bug. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 From noreply at sourceforge.net Thu Oct 28 17:24:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 17:24:17 2004 Subject: [ python-Bugs-1052242 ] Attempt to run all atexit handlers Message-ID: Bugs item #1052242, was opened at 2004-10-22 06:31 Message generated for change (Comment added) made by glchapman You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Greg Chapman (glchapman) Assigned to: Skip Montanaro (montanaro) Summary: Attempt to run all atexit handlers Initial Comment: Currently, if an atexit handler raises an exception during _run_exitfuncs, none of the remaining handlers in the _exithandlers list gets run. I would prefer it if _run_exitfuncs did something like: while _exithandlers: func, targs, kargs = _exithandlers.pop() try: func(*targs, **kargs) except SystemExit: raise except: import sys, traceback print >> sys.stderr, "Error in sys.exitfunc:" traceback.print_exc() The SystemExit except clause is there because it looks like (given its special treatment in call_sys_exitfunc) SystemExit is envisioned as a way of intentionally breaking out of the exithandlers loop. ---------------------------------------------------------------------- >Comment By: Greg Chapman (glchapman) Date: 2004-10-28 07:24 Message: Logged In: YES user_id=86307 > Should we defer it until the loop is complete and reraise it > then? +1 ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-10-28 03:28 Message: Logged In: YES user_id=44345 Okay, Guido has pronounced. My idea for preserving compatibility is out. Now, I wonder about Greg's SystemExit behavior... Should we defer it until the loop is complete and reraise it then? ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-10-27 06:44 Message: Logged In: YES user_id=44345 I agree with you in principle, but you'd change behavior and possibly break existing code that relies on the current implicit semantics, hence my suggestion to use a variable to control the behavior, with it set to preserve the current behavior by default. I'll post a request to python-dev soliciting input. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-26 21:46 Message: Logged In: YES user_id=80475 I prefer the original proposal that leaves the API untouched. Irrespective of ordering, my thought is that all exit handlers are probably expected to run (needed for important cleanup, resource freeing, etc) and may be independent of one another. So, all of them that can run, probably should. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-10-24 09:21 Message: Logged In: YES user_id=44345 OTOH, since the ordering of the exit handlers is not guaranteed, exiting after one exception is also reasonable behavior. Attached is a diff that does two things: adds a set_run_all function and converts the functionality into an ExitHandler class to keep from all the different bits of global state. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-23 16:56 Message: Logged In: YES user_id=80475 This could be considered a bug. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 From noreply at sourceforge.net Thu Oct 28 20:08:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 20:08:19 2004 Subject: [ python-Bugs-1056293 ] dir() should only return strings Message-ID: Bugs item #1056293, was opened at 2004-10-28 13:08 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056293&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 5 Submitted By: Skip Montanaro (montanaro) Assigned to: Nobody/Anonymous (nobody) Summary: dir() should only return strings Initial Comment: Calling this a bug in Python is probably a misnomer. Think of it as more of a "request for workaround". Recent versions of SWIG create wrapper classes that contain a Python class as a key in their __dict__'s. Consequently, something like [k for k in dir(obj) if k.startswith("set_")] because there is an element in dir(obj) that's not a string or a unicode object. I realize dir() is mostly a debugging aid, unlikely to be used frequently in production code, but I'd like see the docs updated so that it's guaranteed to only return strings or unicode objects and I'd like its implementation changed to enforce that (and maybe even warn if something else is found). If this is seen as an acceptable change I will work up a patch. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056293&group_id=5470 From noreply at sourceforge.net Thu Oct 28 21:21:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 21:21:46 2004 Subject: [ python-Bugs-1052242 ] Attempt to run all atexit handlers Message-ID: Bugs item #1052242, was opened at 2004-10-22 09:31 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Greg Chapman (glchapman) Assigned to: Skip Montanaro (montanaro) Summary: Attempt to run all atexit handlers Initial Comment: Currently, if an atexit handler raises an exception during _run_exitfuncs, none of the remaining handlers in the _exithandlers list gets run. I would prefer it if _run_exitfuncs did something like: while _exithandlers: func, targs, kargs = _exithandlers.pop() try: func(*targs, **kargs) except SystemExit: raise except: import sys, traceback print >> sys.stderr, "Error in sys.exitfunc:" traceback.print_exc() The SystemExit except clause is there because it looks like (given its special treatment in call_sys_exitfunc) SystemExit is envisioned as a way of intentionally breaking out of the exithandlers loop. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-28 14:21 Message: Logged In: YES user_id=80475 This should probably also be kicked around on python-dev so that we arrive at a clear expression of what it means to raise SystemExit and how that differs from os._exit(). One school of thought is that Greg's initial version is correct -- raising SystemExit means get out as fast as you can but not as abruptly as os._exit() so that buffers can be flushed, etc. ---------------------------------------------------------------------- Comment By: Greg Chapman (glchapman) Date: 2004-10-28 10:24 Message: Logged In: YES user_id=86307 > Should we defer it until the loop is complete and reraise it > then? +1 ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-10-28 06:28 Message: Logged In: YES user_id=44345 Okay, Guido has pronounced. My idea for preserving compatibility is out. Now, I wonder about Greg's SystemExit behavior... Should we defer it until the loop is complete and reraise it then? ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-10-27 09:44 Message: Logged In: YES user_id=44345 I agree with you in principle, but you'd change behavior and possibly break existing code that relies on the current implicit semantics, hence my suggestion to use a variable to control the behavior, with it set to preserve the current behavior by default. I'll post a request to python-dev soliciting input. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-27 00:46 Message: Logged In: YES user_id=80475 I prefer the original proposal that leaves the API untouched. Irrespective of ordering, my thought is that all exit handlers are probably expected to run (needed for important cleanup, resource freeing, etc) and may be independent of one another. So, all of them that can run, probably should. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-10-24 12:21 Message: Logged In: YES user_id=44345 OTOH, since the ordering of the exit handlers is not guaranteed, exiting after one exception is also reasonable behavior. Attached is a diff that does two things: adds a set_run_all function and converts the functionality into an ExitHandler class to keep from all the different bits of global state. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-23 19:56 Message: Logged In: YES user_id=80475 This could be considered a bug. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052242&group_id=5470 From noreply at sourceforge.net Thu Oct 28 21:38:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 21:38:18 2004 Subject: [ python-Bugs-1056293 ] dir() should only return strings Message-ID: Bugs item #1056293, was opened at 2004-10-28 20:08 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056293&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 5 Submitted By: Skip Montanaro (montanaro) Assigned to: Nobody/Anonymous (nobody) Summary: dir() should only return strings Initial Comment: Calling this a bug in Python is probably a misnomer. Think of it as more of a "request for workaround". Recent versions of SWIG create wrapper classes that contain a Python class as a key in their __dict__'s. Consequently, something like [k for k in dir(obj) if k.startswith("set_")] because there is an element in dir(obj) that's not a string or a unicode object. I realize dir() is mostly a debugging aid, unlikely to be used frequently in production code, but I'd like see the docs updated so that it's guaranteed to only return strings or unicode objects and I'd like its implementation changed to enforce that (and maybe even warn if something else is found). If this is seen as an acceptable change I will work up a patch. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-28 21:38 Message: Logged In: YES user_id=21627 It seems that the documentation guarantees that it returns "names" ("the list contains the object's attributes' names, ..."). Most people certainly expect names to be strings (although they might be Unicode objects at some point in the future :-). So I think it is ok to fix dir in this respect. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056293&group_id=5470 From noreply at sourceforge.net Thu Oct 28 21:41:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 21:41:54 2004 Subject: [ python-Bugs-1055864 ] HTMLParser not compliant to XHTML spec Message-ID: Bugs item #1055864, was opened at 2004-10-28 06:59 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055864&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Luke Bradley (neptune235) Assigned to: Nobody/Anonymous (nobody) Summary: HTMLParser not compliant to XHTML spec Initial Comment: HTMLParser has a problem related to the fact that is doesn't seem to comply to the spec for XHTML. What I am refering to can be read about here: http://www.w3.org/TR/xhtml1/#h-4.8 In a nutshell, HTMLParser doesn't treat data inside 'script' or 'style' elements as #PCDATA, but rather behaves like an HTML 4 parser even for XHTML documents, parsing only end tags. As a result, entity references in javascript are not converted as they should be. XHTML authors writing to spec can expect entities in script sections of XHTML documents to be converted if the script is not explicitly escaped as a CDATA section. which brings up problem two, That sections explicitly escaped as CDATA are also parsed as HTML 4 'script' and 'style' sections...End tags are parsed... My understanding is that this is bad as well: http://www.w3.org/TR/2004/REC-xml-20040204/#dt-cdsection because CDend is the only thing that's supposed to be parsed in a CDATA section for all XML documents? ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-10-28 21:41 Message: Logged In: YES user_id=21627 Can you give an example demonstrating this problem, please? A Python script with a small embedded HTML file, and a PASS/FAIL condition would be best. ---------------------------------------------------------------------- Comment By: Luke Bradley (neptune235) Date: 2004-10-28 10:31 Message: Logged In: YES user_id=178561 I also reported bug 1051840. I discovered this when I was looking for a universal way to handle all the wierd things people do with their script sections on HTML/XHTML pages on the net. I've ended up modifying HTMLParser.py so that the HTMLParser class has an extra attribute called last_match, which is the exact string of HTML that whatever handler event is being called for...So that putting: sys.stdout.write(self.last_match) or sys.stdout.write(self.get_last_match()) for every handler event (except handle_data, which can be directly outputted) will output the page exactly as was inputted. This allows me to handle all oddities in people's code at the level of my handler, without changing HTMLParser in any other way... Here's the code, attached. Not that you care, but on the off chance that you guys might want to think about doing something like this....:) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055864&group_id=5470 From noreply at sourceforge.net Thu Oct 28 22:55:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu Oct 28 22:55:17 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Tim Peters (tim_one) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Fri Oct 29 00:23:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 00:25:20 2004 Subject: [ python-Bugs-1055864 ] HTMLParser not compliant to XHTML spec Message-ID: Bugs item #1055864, was opened at 2004-10-27 21:59 Message generated for change (Comment added) made by neptune235 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055864&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Luke Bradley (neptune235) Assigned to: Nobody/Anonymous (nobody) Summary: HTMLParser not compliant to XHTML spec Initial Comment: HTMLParser has a problem related to the fact that is doesn't seem to comply to the spec for XHTML. What I am refering to can be read about here: http://www.w3.org/TR/xhtml1/#h-4.8 In a nutshell, HTMLParser doesn't treat data inside 'script' or 'style' elements as #PCDATA, but rather behaves like an HTML 4 parser even for XHTML documents, parsing only end tags. As a result, entity references in javascript are not converted as they should be. XHTML authors writing to spec can expect entities in script sections of XHTML documents to be converted if the script is not explicitly escaped as a CDATA section. which brings up problem two, That sections explicitly escaped as CDATA are also parsed as HTML 4 'script' and 'style' sections...End tags are parsed... My understanding is that this is bad as well: http://www.w3.org/TR/2004/REC-xml-20040204/#dt-cdsection because CDend is the only thing that's supposed to be parsed in a CDATA section for all XML documents? ---------------------------------------------------------------------- >Comment By: Luke Bradley (neptune235) Date: 2004-10-28 15:23 Message: Logged In: YES user_id=178561 Sure. I'll attach it as a file: tidytest2.py btw: I'm no guru so tell me if I'm misinterpretting the w3c. I'm just trying to use HTMLParser in such a way that it won't mangle anybodies script sections, and I want to have all my bases covered. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-10-28 12:41 Message: Logged In: YES user_id=21627 Can you give an example demonstrating this problem, please? A Python script with a small embedded HTML file, and a PASS/FAIL condition would be best. ---------------------------------------------------------------------- Comment By: Luke Bradley (neptune235) Date: 2004-10-28 01:31 Message: Logged In: YES user_id=178561 I also reported bug 1051840. I discovered this when I was looking for a universal way to handle all the wierd things people do with their script sections on HTML/XHTML pages on the net. I've ended up modifying HTMLParser.py so that the HTMLParser class has an extra attribute called last_match, which is the exact string of HTML that whatever handler event is being called for...So that putting: sys.stdout.write(self.last_match) or sys.stdout.write(self.get_last_match()) for every handler event (except handle_data, which can be directly outputted) will output the page exactly as was inputted. This allows me to handle all oddities in people's code at the level of my handler, without changing HTMLParser in any other way... Here's the code, attached. Not that you care, but on the off chance that you guys might want to think about doing something like this....:) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055864&group_id=5470 From noreply at sourceforge.net Fri Oct 29 00:32:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 00:32:39 2004 Subject: [ python-Bugs-1056441 ] subprocess fails for args="...", executable="..." Message-ID: Bugs item #1056441, was opened at 2004-10-28 15:32 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056441&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Russell Owen (reowen) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess fails for args="...", executable="..." Initial Comment: In Python 2.4b1 I am trying to use the subprocess module and running into problems when args is a string and executable is specified. For example: >>> p = subprocess.Popen( executable = "xpaget", args = "xpaget ds9 mode", shell = True, stdout = subprocess.PIPE, ) results in this mysterious error msg: >>> sh: illegal option -- c and the data in p.stdout is a boatload of help text that strongly suggests xpaget never got the "ds9 mode" command. Piping stdin and stderr make no difference, of course, but omitting the stdout makes one strange difference: I don't see the error message, just the boatload of help text. Removing the executable argument makes it work as expected: >>> p = subprocess.Popen( args = "xpaget ds9 mode", shell = True, stdout = subprocess.PIPE, ) >>> p.stdout.read() 'pointer\n' And the executable argument works fine if I specify the arguments as a list and don't use shell=True: p = subprocess.Popen( executable = "xpaget", args = ["xpaget", "ds9", "mode"], stdout = subprocess.PIPE, ) >>> p.stdout.read() 'pointer\n' xpa and ds9 are free from but I hope think they are not needed to debug this. I saw this problem using a unix installation of Python 2.4b1 on MacOS X 10.3.5. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056441&group_id=5470 From noreply at sourceforge.net Fri Oct 29 02:49:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 02:49:23 2004 Subject: [ python-Bugs-1050828 ] reindent.py strips execute permission on Unix Message-ID: Bugs item #1050828, was opened at 2004-10-20 12:48 Message generated for change (Comment added) made by facundobatista You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: David Ripton (dripton) Assigned to: Facundo Batista (facundobatista) Summary: reindent.py strips execute permission on Unix Initial Comment: Tools/scripts/reindent.py strips execute permission from its target file. If the tool detects that changes need to be made, then the target file is renamed to its original name + '.bak', then a new file is created with the original file's name and its whitespace-modified contents. No manipulation of the new file's permissions is done, so it ends up with the default for the user's umask. This makes reindent.py annoying to run in an automated fashion, if you rely on execute permission. The fix is trivial, except for portability. Here's the Posix-only version: + if os.name == "posix": + mode = os.stat(bak).st_mode + os.chmod(file, mode) Patch available upon request. ---------------------------------------------------------------------- >Comment By: Facundo Batista (facundobatista) Date: 2004-10-28 21:49 Message: Logged In: YES user_id=752496 I could touch the permissions of the copy to be only readable, but with os.chmod() it'll be portable only to Windows and Unix (according to the docs). And in Mac? Take note that in this very moment, the copy is created via the open(..., "w") which leaves the copy with the same permissions that with the patch. We're not creating a problem... maybe just don't solving it! ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-27 14:34 Message: Logged In: YES user_id=43607 Now that I looked at the patch, I have one concern. I don't mind if the backup copy doesn't have the same permissions as the original, especially not if you can't guarantee that the owner/group doesn't change. However, I do mind if the backup copy has *more* permissions than the original. Consider a file that was only readable by the owner (probably for some good reason), and because of a permissive umask, the backup copy all of a sudden is world-readable. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-27 13:07 Message: Logged In: YES user_id=752496 That is *exactly* what the patch does, but David complained about the metadata of the copy. ---------------------------------------------------------------------- Comment By: Sjoerd Mullender (sjoerd) Date: 2004-10-27 12:29 Message: Logged In: YES user_id=43607 Then perhaps it should do in-place replacement, so first *copy* the original to the backup, and then overwrite the original. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-27 11:56 Message: Logged In: YES user_id=752496 I thought about it. The problem with this approach is that it also copies the permissions, but not the owner/group ids (you get the defaults there). So, I think is better to leave the user defaults than mix the old permissions with new owner/group (which may lead to a security risk in some combinations). There's a better approach: to copy *all* the file metadata. But don't know how to do it in a portable way... ---------------------------------------------------------------------- Comment By: David Ripton (dripton) Date: 2004-10-27 11:47 Message: Logged In: YES user_id=9425 s/wrong/right/ in my last comment. The target is more important than the backup, but we should go for the win/win and give both of them the correct permissions, since it's easy. ---------------------------------------------------------------------- Comment By: David Ripton (dripton) Date: 2004-10-27 11:45 Message: Logged In: YES user_id=9425 The patch is an improvement (better to have the wrong permissions on the target file than on the backup file), but I think it would be even better if the .bak file had the same permissions as the target file. How about using shutil.copy2 (which preserves permissions, among other things) instead of shutil.copy? Thanks. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-27 09:38 Message: Logged In: YES user_id=469548 The patch works for me (Linux 2.6.8-1-686 i686 GNU/Linux) and a coworker (Mac OS X 10.3). ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-26 21:44 Message: Logged In: YES user_id=752496 This fixes the problem: the original file metadata (permisions, etc) stays unchanged. The issue with the metadata of the bak file is none: the new file has the user defaults permissions, group and owner. In the previous version, the original file left with those defaults permissions, so security is not getting worse. The only left issue is, if the user wants to recover the data from the bak file, he would need to change its metadata (this is minor, if something goes wrong and the bak is needed, it's likely that the user will need a manual approach). My primary concern is that I don't want to commit the changes now that we're beta without testing the changes in other platforms (however, I'll ask this on python-dev). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-26 00:19 Message: Logged In: YES user_id=31435 Sorry, I don't normally run on Unix, so I have no way to tell whether this fixes whatever the complaint is. Offhand it doesn't seem right to create a .bak file with default permissions either. For example, if only the user had permission to get at the original, letting the .bak file have broader permissions seems wrong too. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-25 23:57 Message: Logged In: YES user_id=752496 Attaching the file. ---------------------------------------------------------------------- Comment By: Facundo Batista (facundobatista) Date: 2004-10-24 18:20 Message: Logged In: YES user_id=752496 Instead of renaming the file and creating a new one, just copied the original and that's all (the target file is open with write mode, the content replaced, but the permissions stay). A patch is attached. Tim, I'm assigning this to you to doublecheck (not found unit tests for this script). . Facundo ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1050828&group_id=5470 From noreply at sourceforge.net Fri Oct 29 02:59:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 02:59:44 2004 Subject: [ python-Bugs-1056483 ] Lambda in list comprehension Message-ID: Bugs item #1056483, was opened at 2004-10-28 20:59 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056483&group_id=5470 Category: Parser/Compiler Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Doug Blank (dsblank) Assigned to: Nobody/Anonymous (nobody) Summary: Lambda in list comprehension Initial Comment: I know that this has been broken for a long time, but I thought that it would be addressed by 2.4. I couldn't find it in the bug database. Here's the problem: $ python2.4 Python 2.4b1 (#1, Oct 25 2004, 01:44:03) [GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> x = [lambda:a for a in range(5)] >>> x[0]() 4 >>> x[1]() 4 The workaround is the old bind-the-arg trick: >>> x = [lambda a=a:a for a in range(5)] >>> x[0]() 0 >>> x[1]() 1 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056483&group_id=5470 From noreply at sourceforge.net Fri Oct 29 03:46:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 03:46:43 2004 Subject: [ python-Bugs-1056498 ] minor error in os.access Message-ID: Bugs item #1056498, was opened at 2004-10-29 01:46 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056498&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: edward (edwardcannon) Assigned to: Nobody/Anonymous (nobody) Summary: minor error in os.access Initial Comment: current description of os.acess is a follows: access(path, mode) Use the real uid/gid to test for access to path. . . . Return 1 if access is allowed, 0 if not. See the Unix man page access(2) for more information. Availability: Unix, Windows. in reality, os.access returns True or False (at least on 2.3.3) this should probably be updated ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056498&group_id=5470 From noreply at sourceforge.net Fri Oct 29 03:50:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 03:50:07 2004 Subject: [ python-Bugs-1056483 ] Lambda in list comprehension Message-ID: Bugs item #1056483, was opened at 2004-10-28 19:59 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056483&group_id=5470 Category: Parser/Compiler Group: Python 2.4 >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Doug Blank (dsblank) Assigned to: Nobody/Anonymous (nobody) Summary: Lambda in list comprehension Initial Comment: I know that this has been broken for a long time, but I thought that it would be addressed by 2.4. I couldn't find it in the bug database. Here's the problem: $ python2.4 Python 2.4b1 (#1, Oct 25 2004, 01:44:03) [GCC 3.3.3 20040412 (Red Hat Linux 3.3.3-7)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> x = [lambda:a for a in range(5)] >>> x[0]() 4 >>> x[1]() 4 The workaround is the old bind-the-arg trick: >>> x = [lambda a=a:a for a in range(5)] >>> x[0]() 0 >>> x[1]() 1 ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-28 20:50 Message: Logged In: YES user_id=80475 Sorry, this is not a bug. Someone on comp.lang.python or python-tutor would be happy to explain it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056483&group_id=5470 From noreply at sourceforge.net Fri Oct 29 03:55:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 03:55:46 2004 Subject: [ python-Bugs-1056502 ] ambiguity in os.tmpfile Message-ID: Bugs item #1056502, was opened at 2004-10-29 01:55 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056502&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: edward (edwardcannon) Assigned to: Nobody/Anonymous (nobody) Summary: ambiguity in os.tmpfile Initial Comment: tmpfile documentation states that "Return a new file object opened in update mode ("w+b")." update mode is "a" the specified modes is "write, binary", unclear which is correct ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056502&group_id=5470 From noreply at sourceforge.net Fri Oct 29 04:56:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 04:56:55 2004 Subject: [ python-Bugs-1056515 ] ambiguity in os.tmpfile Message-ID: Bugs item #1056515, was opened at 2004-10-29 02:56 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056515&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: edward (edwardcannon) Assigned to: Nobody/Anonymous (nobody) Summary: ambiguity in os.tmpfile Initial Comment: tmpfile documentation states that "Return a new file object opened in update mode ("w+b")." update mode is "a" the specified modes is "write, binary", unclear which is correct ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056515&group_id=5470 From noreply at sourceforge.net Fri Oct 29 08:54:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 08:54:57 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) >Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Fri Oct 29 11:11:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 11:11:52 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-29 05:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Fri Oct 29 16:56:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 16:56:53 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-28 01:58 Message generated for change (Comment added) made by nascheme You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 14:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 09:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 06:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 20:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 08:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 08:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 02:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 02:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Fri Oct 29 17:18:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 17:18:40 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-29 11:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 10:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 05:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Fri Oct 29 18:01:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 18:01:50 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 11:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 10:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 05:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Fri Oct 29 18:41:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 18:41:24 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 11:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 10:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 05:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Fri Oct 29 19:00:13 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 19:00:29 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-28 01:58 Message generated for change (Comment added) made by nascheme You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 17:00 Message: Logged In: YES user_id=35752 I had to change _PyWeakref_ClearRef() since it was also clearing the weakref list of the trash object. Now it just sets wr_object to Py_None. I also made some serious simplifications to gcmodule by just treating trash weakref objects with callbacks the same as objects with __del__ methods (i.e. move them to the finalizers list and then do the transitive closure of that set). I'm attaching a patch. It's a work in progress. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 16:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 16:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 15:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 14:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 09:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 06:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 20:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 08:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 08:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 02:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 02:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Fri Oct 29 19:03:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 19:03:13 2004 Subject: [ python-Bugs-1017978 ] IDLE DOES NOT START ON WinXP Pro Message-ID: Bugs item #1017978, was opened at 2004-08-27 17:23 Message generated for change (Comment added) made by rireland You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017978&group_id=5470 Category: IDLE Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Snake (gj_carman) Assigned to: Nobody/Anonymous (nobody) Summary: IDLE DOES NOT START ON WinXP Pro Initial Comment: Installed Python 2.3.4 on WinXP Pro on Dell Inspiron. Command line interpreter works fine, but the IDLE GUI will not start. Only occurs on Dell Inspiron; cannot reproduce this on other machines. McAffee firewall software is disabled. Any ideas? ---------------------------------------------------------------------- Comment By: Rich Ireland (rireland) Date: 2004-10-29 10:03 Message: Logged In: YES user_id=302950 I had similar symptoms on a Win2K box. There was a problem with the TCL install - I removed the Ruby-1.6.7 and Python-2.2.1 installs and the problem has gone away. Unfortunately I did not capture the error output. ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2004-09-02 14:37 Message: Logged In: YES user_id=149084 Strange. Although IDLE has been communicating with its subprocess while registering methods, either the socket has gone away or it doesn't have a fileno() method. It always fails at this point? Try adding the following debug code just before line 337 in rpc.py: print>>sys.__stderr__, "*** self.sock: ", self.sock print>>sys.__stderr__, "*** self.sock.fileno(): ", self.sock.fileno() and post the results here. Thanks. ---------------------------------------------------------------------- Comment By: Snake (gj_carman) Date: 2004-09-01 12:09 Message: Logged In: YES user_id=1112369 When clicking the shortcut via the Start menu, the hourglass briefly appears but IDLE does not start. The WinXP personal firewall is not enabled. There is McAfee security software running, but the same behavior occurs when the software is disabled. When running Idle from the commandline, get the following output: C:\Python23\Lib\idlelib>idle.py Traceback (most recent call last): File "C:\Python23\Lib\idlelib\idle.py", line 23, in ? idlelib.PyShell.main() File "C:\Python23\lib\idlelib\PyShell.py", line 1287, in main flist.pyshell = PyShell(flist) File "C:\Python23\lib\idlelib\PyShell.py", line 769, in __init__ self.interp.start_subprocess() File "C:\Python23\lib\idlelib\PyShell.py", line 369, in start_subprocess self.transfer_path() File "C:\Python23\lib\idlelib\PyShell.py", line 438, in transfer_path self.runcommand("""if 1: File "C:\Python23\lib\idlelib\PyShell.py", line 655, in runcommand self.rpcclt.remotequeue("exec", "runcode", (code,), {}) File "C:\Python23\lib\idlelib\rpc.py", line 214, in remotequeue return self.asyncreturn(seq) File "C:\Python23\lib\idlelib\rpc.py", line 238, in asyncreturn response = self.getresponse(seq, wait=0.05) File "C:\Python23\lib\idlelib\rpc.py", line 278, in getresponse response = self._getresponse(myseq, wait) File "C:\Python23\lib\idlelib\rpc.py", line 298, in _getresponse response = self.pollresponse(myseq, wait) File "C:\Python23\lib\idlelib\rpc.py", line 423, in pollresponse message = self.pollmessage(wait) File "C:\Python23\lib\idlelib\rpc.py", line 375, in pollmessage packet = self.pollpacket(wait) File "C:\Python23\lib\idlelib\rpc.py", line 347, in pollpacket if not self.ioready(wait): File "C:\Python23\lib\idlelib\rpc.py", line 337, in ioready r, w, x = select.select([self.sock.fileno()], [], [], wait) select.error: (10022, 'An invalid argument was supplied') ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2004-08-29 15:18 Message: Logged In: YES user_id=149084 More information is needed, what does "will not start" mean? Open a command window, cd to \python23\Lib\idlelib and run ..\..\python .\PyShell.py Post all error message in command window or in the IDLE GUI here. Thanks. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1017978&group_id=5470 From noreply at sourceforge.net Fri Oct 29 19:13:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 19:13:12 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-29 13:13 Message: Logged In: YES user_id=31435 > I had to change _PyWeakref_ClearRef() since it was also > clearing the weakref list of the trash object. That was really its *purpose*. If a trash weakref with a callback isn't removed from the referent's list of weakrefs, then the callback will trigger when PyObject_ClearWeakRefs() is invoked on the referent. The purpose of _PyWeakref_ClearRef() was to ensure that the callback never triggers. > Now it just sets wr_object to Py_None. That won't stop the callback from triggering. It also means (see earlier comment) that PyObject_ClearWeakRefs() will never removed the weakref from the list either, although I'm not sure that does real harm. > I also made some serious simplifications to gcmodule by > just treating trash weakref objects with callbacks the same > as objects with __del__ methods (i.e. move them to the > finalizers list and then do the transitive closure of that set). Does that mean they can end up in gc.garbage too? If so, I don't think that's sellable. See the end of gc_weakref.txt: An alternative would have been to treat objects with callbacks like objects with __del__ methods, refusing to collect them, appending them to gc.garbage instead. That would have been much easier. Jim Fulton gave a strong argument against that (on Python-Dev): ... > I'm attaching a patch. It's a work in progress. Aren't we all . ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 13:00 Message: Logged In: YES user_id=35752 I had to change _PyWeakref_ClearRef() since it was also clearing the weakref list of the trash object. Now it just sets wr_object to Py_None. I also made some serious simplifications to gcmodule by just treating trash weakref objects with callbacks the same as objects with __del__ methods (i.e. move them to the finalizers list and then do the transitive closure of that set). I'm attaching a patch. It's a work in progress. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 11:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 10:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 05:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Fri Oct 29 23:24:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 23:24:32 2004 Subject: [ python-Bugs-1057048 ] subprocess works poorly on Windows with Python 2.3 Message-ID: Bugs item #1057048, was opened at 2004-10-29 14:24 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057048&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Russell Owen (reowen) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess works poorly on Windows with Python 2.3 Initial Comment: My understanding was that subprocess.py was supposed to be backwards compatible with at least Python 2.3 if not 2.2. Wanting subprocess and backwards compatibility, I grabbed the subprocess.py from 2.4b1, changed the import if so that win32api was used (since I had no _subprocess library!) and found several problems, of which this is one (one per bug report): If the executable has a space in its path name (as is standard on Windows) and command-line arguments, things fail miserably: from subprocess import * Popen('"C:\Program Files\ds9\xpaget" ds9 mode') produces an error that C:\Program cannot be run. Clearly the double quoting isn't making it through Popen. Note that this runs just fine if there are no command line arguments, i.e. this works: Popen('"C:\Program Files\ds9\xpaget"') Note that this same bug also exists in os.popen3, so you may have inherited it. This bug does NOT occur on a unix build on MacOS X. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057048&group_id=5470 From noreply at sourceforge.net Fri Oct 29 23:35:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 23:35:27 2004 Subject: [ python-Bugs-1057052 ] subprocess on Windows 2: unexpected failure Message-ID: Bugs item #1057052, was opened at 2004-10-29 14:35 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057052&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Russell Owen (reowen) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess on Windows 2: unexpected failure Initial Comment: My understanding was that subprocess.py was supposed to be backwards compatible with at least Python 2.3 if not 2.2. Wanting subprocess and backwards compatibility, I grabbed the subprocess.py from 2.4b1, changed the import if so that win32api was used (since I had no _subprocess library!) and found several problems, of which this is one (one per bug report): The following code fails when run as a program on Windows Python 2.3. However, the exact same code works perfectly on a unix build of Python 2.4b1 on MacOS X. It also works on Windows if I use an interactive interpreter and type each line in by hand (skipping some irrelevant bits). import subprocess def xpaget(cmd, template="ds9"): fullCmd = '"xpaget" %s %s' % (template, cmd,) p = subprocess.Popen( # executable = _XPADir + "xpaget", # use if bug in subprocess gets fixed args = fullCmd, shell = True, stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE, cwd = _XPADir, ) try: p.stdin.close() errMsg = p.stderr.read() if errMsg: raise RuntimeError('%r failed: %s' % (fullCmd, errMsg)) reply = p.stdin.read() # the code fails here; the next print statement never executes print "xpaget read %r" % reply return reply finally: p.stdout.close() p.stderr.close() ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057052&group_id=5470 From noreply at sourceforge.net Fri Oct 29 23:49:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 23:49:40 2004 Subject: [ python-Bugs-1057061 ] subprocess on Windows: extra windows Message-ID: Bugs item #1057061, was opened at 2004-10-29 14:49 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057061&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Russell Owen (reowen) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess on Windows: extra windows Initial Comment: My understanding was that subprocess.py was supposed to be backwards compatible with at least Python 2.3 if not 2.2. Wanting subprocess and backwards compatibility, I grabbed the subprocess.py from 2.4b1, changed the import if so that win32api was used (since I had no _subprocess library!) and found several problems, of which this is is the mildes: when using subprocess.Popen on Windows I get one or more windows called C:\WINNT\system32\CMD.EXE popping up. Sometimes they stick around, other times they are only around for a short time. Note that I don't see this probem using os.popen3 (on the other hand, os.popen3 has inferior error handling). The following code is one example: from subprocess import * Popen("ds9 -title foo", shell=True, cwd="C:\Program Files\ds9\") ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057061&group_id=5470 From noreply at sourceforge.net Fri Oct 29 23:58:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri Oct 29 23:58:35 2004 Subject: [ python-Bugs-1056441 ] subprocess fails for args="...", executable="..." Message-ID: Bugs item #1056441, was opened at 2004-10-28 15:32 Message generated for change (Comment added) made by reowen You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056441&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Russell Owen (reowen) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess fails for args="...", executable="..." Initial Comment: In Python 2.4b1 I am trying to use the subprocess module and running into problems when args is a string and executable is specified. For example: >>> p = subprocess.Popen( executable = "xpaget", args = "xpaget ds9 mode", shell = True, stdout = subprocess.PIPE, ) results in this mysterious error msg: >>> sh: illegal option -- c and the data in p.stdout is a boatload of help text that strongly suggests xpaget never got the "ds9 mode" command. Piping stdin and stderr make no difference, of course, but omitting the stdout makes one strange difference: I don't see the error message, just the boatload of help text. Removing the executable argument makes it work as expected: >>> p = subprocess.Popen( args = "xpaget ds9 mode", shell = True, stdout = subprocess.PIPE, ) >>> p.stdout.read() 'pointer\n' And the executable argument works fine if I specify the arguments as a list and don't use shell=True: p = subprocess.Popen( executable = "xpaget", args = ["xpaget", "ds9", "mode"], stdout = subprocess.PIPE, ) >>> p.stdout.read() 'pointer\n' xpa and ds9 are free from but I hope think they are not needed to debug this. I saw this problem using a unix installation of Python 2.4b1 on MacOS X 10.3.5. ---------------------------------------------------------------------- >Comment By: Russell Owen (reowen) Date: 2004-10-29 14:58 Message: Logged In: YES user_id=431773 I looked at the code and the problem is here: def _execute_child(self, args, executable, preexec_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite): """Execute program (POSIX version)""" if isinstance(args, types.StringTypes): args = [args] if shell: args = ["/bin/sh", "-c"] + args if executable == None: executable = args[0] You can se that if shell is true (as it must be if one specifies args as a string) and executable is supplied, one ends p with a mess, i.e. the executable ends up trying to execute ["/bin/sh", "-c", args] Fortunately cwd offers a hackaround. Still, I hope this can be either fixed or else can produce a warning. While on that subject, should specifying args as a string without specifying shell=True should produce a warning? Thanks for subprocess. It's a wonderful module. I am really looking forward to using it (once a few kinks get worked out). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056441&group_id=5470 From noreply at sourceforge.net Sat Oct 30 03:02:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 03:02:41 2004 Subject: [ python-Bugs-1056502 ] ambiguity in os.tmpfile Message-ID: Bugs item #1056502, was opened at 2004-10-28 22:55 Message generated for change (Comment added) made by facundobatista You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056502&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Duplicate Priority: 5 Submitted By: edward (edwardcannon) Assigned to: Nobody/Anonymous (nobody) Summary: ambiguity in os.tmpfile Initial Comment: tmpfile documentation states that "Return a new file object opened in update mode ("w+b")." update mode is "a" the specified modes is "write, binary", unclear which is correct ---------------------------------------------------------------------- >Comment By: Facundo Batista (facundobatista) Date: 2004-10-29 22:02 Message: Logged In: YES user_id=752496 Duplicate of bug #1056515. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056502&group_id=5470 From noreply at sourceforge.net Sat Oct 30 05:23:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 05:24:49 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-29 23:23 Message: Logged In: YES user_id=31435 patch2.txt stops abusing weakref `hash`, and seems to have no issues w/ Python's or Zope3's -uall/--all tests, in debug or release builds. weakrefs are handled entirely in move_troublemakers() now; collect() doesn't even have its old wr_callbacks list anymore. If there's a more elegant approach coming, great, but if it's not coming soon I'm going to check this in, to get as much developer test coverage as possible before 2.4.b1. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 13:13 Message: Logged In: YES user_id=31435 > I had to change _PyWeakref_ClearRef() since it was also > clearing the weakref list of the trash object. That was really its *purpose*. If a trash weakref with a callback isn't removed from the referent's list of weakrefs, then the callback will trigger when PyObject_ClearWeakRefs() is invoked on the referent. The purpose of _PyWeakref_ClearRef() was to ensure that the callback never triggers. > Now it just sets wr_object to Py_None. That won't stop the callback from triggering. It also means (see earlier comment) that PyObject_ClearWeakRefs() will never removed the weakref from the list either, although I'm not sure that does real harm. > I also made some serious simplifications to gcmodule by > just treating trash weakref objects with callbacks the same > as objects with __del__ methods (i.e. move them to the > finalizers list and then do the transitive closure of that set). Does that mean they can end up in gc.garbage too? If so, I don't think that's sellable. See the end of gc_weakref.txt: An alternative would have been to treat objects with callbacks like objects with __del__ methods, refusing to collect them, appending them to gc.garbage instead. That would have been much easier. Jim Fulton gave a strong argument against that (on Python-Dev): ... > I'm attaching a patch. It's a work in progress. Aren't we all . ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 13:00 Message: Logged In: YES user_id=35752 I had to change _PyWeakref_ClearRef() since it was also clearing the weakref list of the trash object. Now it just sets wr_object to Py_None. I also made some serious simplifications to gcmodule by just treating trash weakref objects with callbacks the same as objects with __del__ methods (i.e. move them to the finalizers list and then do the transitive closure of that set). I'm attaching a patch. It's a work in progress. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 11:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 10:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 05:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Sat Oct 30 08:01:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 08:01:44 2004 Subject: [ python-Bugs-1051638 ] incorrect traceback filename from pyc Message-ID: Bugs item #1051638, was opened at 2004-10-21 10:24 Message generated for change (Comment added) made by kquick You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1051638&group_id=5470 Category: Parser/Compiler Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Kevin Quick (kquick) Assigned to: Nobody/Anonymous (nobody) Summary: incorrect traceback filename from pyc Initial Comment: The .pyc file apparently caches the entire path of the source file during compilation, causing it to report improper path information if the resulting code is moved and an exception occurs. $ python Python 2.3.3 (#1, Oct 18 2004, 16:10:24) [GCC 3.3.4 20040623 (Gentoo Linux 3.3.4-r1, ssp-3.3.2-2, pie-8.7.6)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> ---------------------------------------------------------------------- >Comment By: Kevin Quick (kquick) Date: 2004-10-29 23:01 Message: Logged In: YES user_id=6133 OK, I've done some more investigation and have some more details and clarification. I've attached another python script that demonstrates these additional clarifications: First, it runs sample scripts in a test directory. The sample scripts generate an exception, print that exception with traceback.print_exc, then raise to let the interpreter print the exception as it fails the script. The control script verifies that the output of traceback.print_exc matches the interpreter's traceback output. So far, so good. Step two demonstrates the first problem: the original directory is simply renamed; the previous location does not exist. The same sample scripts are run, and the output now shows that traceback.print_exc() output is *different* than the interpreter's traceback. Specifically, the interpreter prints the new, correct path, but the traceback.print_exc() prints the old path. Step three demonstrates another problem. The scripts are run from the new directory, but the old directory now exists, along files having the original .py names, but those files are now much shorter; the lines that will be referenced in the traceback do not exist. In the output from this step, note that the output from traceback.print_exc() and the interpreter traceback are again *different* from each other. In this case, both are fooled into reporting file paths in the old directory, but for a non-existent line the interpreter prints a blank line but traceback.print_exc() doesn't print a line at all. And finally, the .py files in the original directory are filled with nonsense (in python terms). This test case shows that the output from traceback. print_exc() and the interpreter *do match*, but both are fooled into showing file paths in the old directory and the nonsense from the new versions of the files. IMHO, I think that the output from the traceback.print_exc() and the interpreter traceback should be the same in *all* scenarios, and I also think that the path used for importing the module should be used in traceback reporting. Rather than encoding the full path of the .py source in the .pyc, the compiler should just encode the name portion, and use the path from the .pyc import and look for the .py in the same directory as the .pyc; if that's garbage, then so be it, but this will 99% of the time be more correct than the current behavior. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-10-23 17:40 Message: Logged In: YES user_id=80475 IMO, this is not a bug. A pyc file has no way of knowing the future locations of the source that originally created it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1051638&group_id=5470 From noreply at sourceforge.net Sat Oct 30 09:55:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 09:55:55 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-30 03:55 Message: Logged In: YES user_id=31435 patch3.txt repairs some typos in comments; semantically, it's the same as patch2.txt. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 23:23 Message: Logged In: YES user_id=31435 patch2.txt stops abusing weakref `hash`, and seems to have no issues w/ Python's or Zope3's -uall/--all tests, in debug or release builds. weakrefs are handled entirely in move_troublemakers() now; collect() doesn't even have its old wr_callbacks list anymore. If there's a more elegant approach coming, great, but if it's not coming soon I'm going to check this in, to get as much developer test coverage as possible before 2.4.b1. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 13:13 Message: Logged In: YES user_id=31435 > I had to change _PyWeakref_ClearRef() since it was also > clearing the weakref list of the trash object. That was really its *purpose*. If a trash weakref with a callback isn't removed from the referent's list of weakrefs, then the callback will trigger when PyObject_ClearWeakRefs() is invoked on the referent. The purpose of _PyWeakref_ClearRef() was to ensure that the callback never triggers. > Now it just sets wr_object to Py_None. That won't stop the callback from triggering. It also means (see earlier comment) that PyObject_ClearWeakRefs() will never removed the weakref from the list either, although I'm not sure that does real harm. > I also made some serious simplifications to gcmodule by > just treating trash weakref objects with callbacks the same > as objects with __del__ methods (i.e. move them to the > finalizers list and then do the transitive closure of that set). Does that mean they can end up in gc.garbage too? If so, I don't think that's sellable. See the end of gc_weakref.txt: An alternative would have been to treat objects with callbacks like objects with __del__ methods, refusing to collect them, appending them to gc.garbage instead. That would have been much easier. Jim Fulton gave a strong argument against that (on Python-Dev): ... > I'm attaching a patch. It's a work in progress. Aren't we all . ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 13:00 Message: Logged In: YES user_id=35752 I had to change _PyWeakref_ClearRef() since it was also clearing the weakref list of the trash object. Now it just sets wr_object to Py_None. I also made some serious simplifications to gcmodule by just treating trash weakref objects with callbacks the same as objects with __del__ methods (i.e. move them to the finalizers list and then do the transitive closure of that set). I'm attaching a patch. It's a work in progress. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 11:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 10:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 05:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Sat Oct 30 17:04:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 17:04:04 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-28 01:58 Message generated for change (Comment added) made by nascheme You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 15:04 Message: Logged In: YES user_id=35752 I think your patch is about as elegant as it can be. The approach looks sound but I believe you forgot to clear the wr_callback pointer. I'm attaching an incremental patch (relative to patch3.txt). Regarding gc_weakref.txt, I think we should add a note at the top explaining the critical rule. I.e. after we can computed the unreachable set (aka trash), it's okay to invoke code that can access non-trash but we cannot allow code to run that can reference trash. There is only one minor issue that I can think of. We decide which callbacks to invoke inside move_troublemakers(). It's possible that unreachable weakrefs become reachable when move_finalizer_reachable() is called (i.e. weakrefs that would be trash except that they got revived because they were referenced by an object with a __del__ method). However, since those weakrefs would have been trash I think it's okay not to invoke their callbacks. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 07:55 Message: Logged In: YES user_id=31435 patch3.txt repairs some typos in comments; semantically, it's the same as patch2.txt. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 03:23 Message: Logged In: YES user_id=31435 patch2.txt stops abusing weakref `hash`, and seems to have no issues w/ Python's or Zope3's -uall/--all tests, in debug or release builds. weakrefs are handled entirely in move_troublemakers() now; collect() doesn't even have its old wr_callbacks list anymore. If there's a more elegant approach coming, great, but if it's not coming soon I'm going to check this in, to get as much developer test coverage as possible before 2.4.b1. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 17:13 Message: Logged In: YES user_id=31435 > I had to change _PyWeakref_ClearRef() since it was also > clearing the weakref list of the trash object. That was really its *purpose*. If a trash weakref with a callback isn't removed from the referent's list of weakrefs, then the callback will trigger when PyObject_ClearWeakRefs() is invoked on the referent. The purpose of _PyWeakref_ClearRef() was to ensure that the callback never triggers. > Now it just sets wr_object to Py_None. That won't stop the callback from triggering. It also means (see earlier comment) that PyObject_ClearWeakRefs() will never removed the weakref from the list either, although I'm not sure that does real harm. > I also made some serious simplifications to gcmodule by > just treating trash weakref objects with callbacks the same > as objects with __del__ methods (i.e. move them to the > finalizers list and then do the transitive closure of that set). Does that mean they can end up in gc.garbage too? If so, I don't think that's sellable. See the end of gc_weakref.txt: An alternative would have been to treat objects with callbacks like objects with __del__ methods, refusing to collect them, appending them to gc.garbage instead. That would have been much easier. Jim Fulton gave a strong argument against that (on Python-Dev): ... > I'm attaching a patch. It's a work in progress. Aren't we all . ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 17:00 Message: Logged In: YES user_id=35752 I had to change _PyWeakref_ClearRef() since it was also clearing the weakref list of the trash object. Now it just sets wr_object to Py_None. I also made some serious simplifications to gcmodule by just treating trash weakref objects with callbacks the same as objects with __del__ methods (i.e. move them to the finalizers list and then do the transitive closure of that set). I'm attaching a patch. It's a work in progress. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 16:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 16:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 15:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 14:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 09:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 06:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 20:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 08:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 08:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 02:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 02:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Sat Oct 30 17:27:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 17:27:31 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-28 01:58 Message generated for change (Comment added) made by nascheme You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 15:27 Message: Logged In: YES user_id=35752 Attaching another patch. This one splits move_troublemakers() into move_finalizers() and handle_weakrefs(). The code is a little easier to understand and arguably more correct. The down side is that it adds another pass over `unreachable`. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 15:04 Message: Logged In: YES user_id=35752 I think your patch is about as elegant as it can be. The approach looks sound but I believe you forgot to clear the wr_callback pointer. I'm attaching an incremental patch (relative to patch3.txt). Regarding gc_weakref.txt, I think we should add a note at the top explaining the critical rule. I.e. after we can computed the unreachable set (aka trash), it's okay to invoke code that can access non-trash but we cannot allow code to run that can reference trash. There is only one minor issue that I can think of. We decide which callbacks to invoke inside move_troublemakers(). It's possible that unreachable weakrefs become reachable when move_finalizer_reachable() is called (i.e. weakrefs that would be trash except that they got revived because they were referenced by an object with a __del__ method). However, since those weakrefs would have been trash I think it's okay not to invoke their callbacks. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 07:55 Message: Logged In: YES user_id=31435 patch3.txt repairs some typos in comments; semantically, it's the same as patch2.txt. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 03:23 Message: Logged In: YES user_id=31435 patch2.txt stops abusing weakref `hash`, and seems to have no issues w/ Python's or Zope3's -uall/--all tests, in debug or release builds. weakrefs are handled entirely in move_troublemakers() now; collect() doesn't even have its old wr_callbacks list anymore. If there's a more elegant approach coming, great, but if it's not coming soon I'm going to check this in, to get as much developer test coverage as possible before 2.4.b1. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 17:13 Message: Logged In: YES user_id=31435 > I had to change _PyWeakref_ClearRef() since it was also > clearing the weakref list of the trash object. That was really its *purpose*. If a trash weakref with a callback isn't removed from the referent's list of weakrefs, then the callback will trigger when PyObject_ClearWeakRefs() is invoked on the referent. The purpose of _PyWeakref_ClearRef() was to ensure that the callback never triggers. > Now it just sets wr_object to Py_None. That won't stop the callback from triggering. It also means (see earlier comment) that PyObject_ClearWeakRefs() will never removed the weakref from the list either, although I'm not sure that does real harm. > I also made some serious simplifications to gcmodule by > just treating trash weakref objects with callbacks the same > as objects with __del__ methods (i.e. move them to the > finalizers list and then do the transitive closure of that set). Does that mean they can end up in gc.garbage too? If so, I don't think that's sellable. See the end of gc_weakref.txt: An alternative would have been to treat objects with callbacks like objects with __del__ methods, refusing to collect them, appending them to gc.garbage instead. That would have been much easier. Jim Fulton gave a strong argument against that (on Python-Dev): ... > I'm attaching a patch. It's a work in progress. Aren't we all . ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 17:00 Message: Logged In: YES user_id=35752 I had to change _PyWeakref_ClearRef() since it was also clearing the weakref list of the trash object. Now it just sets wr_object to Py_None. I also made some serious simplifications to gcmodule by just treating trash weakref objects with callbacks the same as objects with __del__ methods (i.e. move them to the finalizers list and then do the transitive closure of that set). I'm attaching a patch. It's a work in progress. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 16:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 16:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 15:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 14:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 09:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 06:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 20:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 08:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 08:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 02:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 02:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Sat Oct 30 17:44:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 17:45:41 2004 Subject: [ python-Bugs-1054564 ] RE '*.?' cores if len of found string exceeds 10000 Message-ID: Bugs item #1054564, was opened at 2004-10-26 05:55 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054564&group_id=5470 Category: Regular Expressions Group: Python 2.2.3 Status: Open Resolution: Works For Me Priority: 5 Submitted By: Rob (rwhent) Assigned to: Fredrik Lundh (effbot) Summary: RE '*.?' cores if len of found string exceeds 10000 Initial Comment: Whilst parsing some extremely long strings I found that the re.match causes segmentation faults on Solaris 2.8 when strings being matched contain '*.?' and the contents of the regex which matches this part of the regex exceeds 10000 chars (actually it seemed to be exactly at 8192 chars) This is the regex used: if re.match('^.*?\[\s*[A-Za-z_0-9]+\s*\].*',string): This regex looks for '[alphaNum_]' present in a large string When it failed the string was 8192 chars long with no matching '[alphaNum_]' present. If I reduce the length of the string below 8192 it works ok. This is a major issue to my application as some string to be parsed are very large. I saw some discussion on another bulletin board with a similar issue ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-10-30 08:44 Message: Logged In: YES user_id=341410 In the case of this particular search, you could write your own little searcher. The following could likely be done better, but this is a quick 5-minute job that won't core on you unless something is really wrong with Python, and may be a reasonable stopgap until someone re-does the regular expression library. import string def find_thing(s): sp = 0 d = dict.fromkeys(list(string.letters+string.digits+'_')) while sp < len(s): start = None for i in xrange(sp, len(s)): if s[i] == '[': start = i break if start is None: return for i in xrange(start+1, len(s)): if s[i] in d: continue elif s[i] == ']': return s[start:i+1] else: sp = i break It returns None on failure to find, and the string otherwise. ---------------------------------------------------------------------- Comment By: Fredrik Lundh (effbot) Date: 2004-10-26 06:24 Message: Logged In: YES user_id=38376 btw, if you're searching for things, why not use the "search" method? if re.search('\[\s*[A-Za-z_0-9]+\s*\]', string): (also, "[A-Za-z_0-9]" is better spelled "\w") ---------------------------------------------------------------------- Comment By: Fredrik Lundh (effbot) Date: 2004-10-26 06:20 Message: Logged In: YES user_id=38376 The max recursion limit problem in the re module is well-known. Until this limitation in the implementation is removed, to work around it check http://www.python.org/dev/doc/devel/lib/module-re.html http://python/org/sf/493252 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1054564&group_id=5470 From noreply at sourceforge.net Sat Oct 30 17:46:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 17:46:40 2004 Subject: [ python-Bugs-1057052 ] subprocess on Windows 2: unexpected failure Message-ID: Bugs item #1057052, was opened at 2004-10-29 14:35 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057052&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Russell Owen (reowen) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess on Windows 2: unexpected failure Initial Comment: My understanding was that subprocess.py was supposed to be backwards compatible with at least Python 2.3 if not 2.2. Wanting subprocess and backwards compatibility, I grabbed the subprocess.py from 2.4b1, changed the import if so that win32api was used (since I had no _subprocess library!) and found several problems, of which this is one (one per bug report): The following code fails when run as a program on Windows Python 2.3. However, the exact same code works perfectly on a unix build of Python 2.4b1 on MacOS X. It also works on Windows if I use an interactive interpreter and type each line in by hand (skipping some irrelevant bits). import subprocess def xpaget(cmd, template="ds9"): fullCmd = '"xpaget" %s %s' % (template, cmd,) p = subprocess.Popen( # executable = _XPADir + "xpaget", # use if bug in subprocess gets fixed args = fullCmd, shell = True, stdin = subprocess.PIPE, stdout = subprocess.PIPE, stderr = subprocess.PIPE, cwd = _XPADir, ) try: p.stdin.close() errMsg = p.stderr.read() if errMsg: raise RuntimeError('%r failed: %s' % (fullCmd, errMsg)) reply = p.stdin.read() # the code fails here; the next print statement never executes print "xpaget read %r" % reply return reply finally: p.stdout.close() p.stderr.close() ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-10-30 08:46 Message: Logged In: YES user_id=341410 You may want to repost that with indentation (the original bug report always loses indentation). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057052&group_id=5470 From noreply at sourceforge.net Sat Oct 30 17:55:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 17:55:36 2004 Subject: [ python-Bugs-1052036 ] Appending non-empty element to a non-empty list yields None Message-ID: Bugs item #1052036, was opened at 2004-10-22 01:18 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052036&group_id=5470 Category: Python Interpreter Core Group: Python 2.4 Status: Closed Resolution: Invalid Priority: 7 Submitted By: Peer Janssen (peerjanssen) Assigned to: Nobody/Anonymous (nobody) Summary: Appending non-empty element to a non-empty list yields None Initial Comment: Appending a non-empty element to a non-empty list from a dictionary results in a list of None Scope: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 and Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] on win32 yield identical results. Description: The code (which is supposed to be used in a function for deleting duplicate files...) contains this line: print "Warning!",(md5,len),'*',MainDic[(md5,len)],'*',name,'*',MainDic[(md5,len)].append(name) which prints the following result: Warning! ('5e011eda91336d21d3bfc5d5a686bb44\n', '7217') * ['D:\_Arbeit\_In\_Roaming\_Telework\www.ivc.ca\part9.html', 'D:\_Arbeit\_I n\_Roaming\_Telework\www.ivc.ca\part9.html.primary'] * D:\_Arbeit\_In\_Roaming\_Telework\www.ivc.ca\part9.html.primary * None How can this be? The tuple is not empty, the entry for it in the dictionary exists, it is a list of strings, name is not empty and a string, but appending name to the list from the dictionary yields "None". Am I missing something, or is this really a bug? ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-10-30 08:55 Message: Logged In: YES user_id=341410 I'll also mention that you are shadowing the global 'len' and likely global 'md5' with your variable names. Since you seem to be relatively new to Python, I'll just say that such things are generally frowned upon in production Python software, as it tends to confuse others who would read your code. ---------------------------------------------------------------------- Comment By: Peer Janssen (peerjanssen) Date: 2004-10-22 01:58 Message: Logged In: YES user_id=896722 Ok, I missed something: >>> x=['a'] >>> print x.append('b') None >>> print x ['a', 'b'] So this is not a bug. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1052036&group_id=5470 From noreply at sourceforge.net Sat Oct 30 19:20:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 19:20:42 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-30 13:20 Message: Logged In: YES user_id=31435 > The approach looks sound but I believe you forgot to clear > the wr_callback pointer. I'm attaching an incremental patch > (relative to patch3.txt). Some of the comments in that incremental patch are good additions, but I don't agree with the code changes. A weakref object owns a (strong) reference to its wr_callback, and the first pass gives move_troublemakers its own new strong reference to each weakref object. Because of the latter, nothing any callback does can cause any of these weakrefs to go away, so nothing can happen to cause their wr_callback slots to become NULL either. The assertion that wr_callback isn't NULL is appropriate. If you still think it's possible for wr_callback to become NULL, please concoct a test showing it happen. The passes after the first give up move_troublemaker's references to the weakref objects. Then they'll go away, when their refcounts hit 0. Sometimes that happens right away, sometimes not. In either case, it's weakref_dealloc's job to decrement the refcount on w->wr_callback when w goes away, and there's no point (that I can see) to duplicating that logic too inside gcmodule. Note that the comments before _PyWeakref_ClearRef() explain that the weakref's tp_dealloc is expected to finish the job (clear the wr_callback slot) -- that's been true since _PyWeakref_ClearRef() was first introduced. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 11:27 Message: Logged In: YES user_id=35752 Attaching another patch. This one splits move_troublemakers() into move_finalizers() and handle_weakrefs(). The code is a little easier to understand and arguably more correct. The down side is that it adds another pass over `unreachable`. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 11:04 Message: Logged In: YES user_id=35752 I think your patch is about as elegant as it can be. The approach looks sound but I believe you forgot to clear the wr_callback pointer. I'm attaching an incremental patch (relative to patch3.txt). Regarding gc_weakref.txt, I think we should add a note at the top explaining the critical rule. I.e. after we can computed the unreachable set (aka trash), it's okay to invoke code that can access non-trash but we cannot allow code to run that can reference trash. There is only one minor issue that I can think of. We decide which callbacks to invoke inside move_troublemakers(). It's possible that unreachable weakrefs become reachable when move_finalizer_reachable() is called (i.e. weakrefs that would be trash except that they got revived because they were referenced by an object with a __del__ method). However, since those weakrefs would have been trash I think it's okay not to invoke their callbacks. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 03:55 Message: Logged In: YES user_id=31435 patch3.txt repairs some typos in comments; semantically, it's the same as patch2.txt. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 23:23 Message: Logged In: YES user_id=31435 patch2.txt stops abusing weakref `hash`, and seems to have no issues w/ Python's or Zope3's -uall/--all tests, in debug or release builds. weakrefs are handled entirely in move_troublemakers() now; collect() doesn't even have its old wr_callbacks list anymore. If there's a more elegant approach coming, great, but if it's not coming soon I'm going to check this in, to get as much developer test coverage as possible before 2.4.b1. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 13:13 Message: Logged In: YES user_id=31435 > I had to change _PyWeakref_ClearRef() since it was also > clearing the weakref list of the trash object. That was really its *purpose*. If a trash weakref with a callback isn't removed from the referent's list of weakrefs, then the callback will trigger when PyObject_ClearWeakRefs() is invoked on the referent. The purpose of _PyWeakref_ClearRef() was to ensure that the callback never triggers. > Now it just sets wr_object to Py_None. That won't stop the callback from triggering. It also means (see earlier comment) that PyObject_ClearWeakRefs() will never removed the weakref from the list either, although I'm not sure that does real harm. > I also made some serious simplifications to gcmodule by > just treating trash weakref objects with callbacks the same > as objects with __del__ methods (i.e. move them to the > finalizers list and then do the transitive closure of that set). Does that mean they can end up in gc.garbage too? If so, I don't think that's sellable. See the end of gc_weakref.txt: An alternative would have been to treat objects with callbacks like objects with __del__ methods, refusing to collect them, appending them to gc.garbage instead. That would have been much easier. Jim Fulton gave a strong argument against that (on Python-Dev): ... > I'm attaching a patch. It's a work in progress. Aren't we all . ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 13:00 Message: Logged In: YES user_id=35752 I had to change _PyWeakref_ClearRef() since it was also clearing the weakref list of the trash object. Now it just sets wr_object to Py_None. I also made some serious simplifications to gcmodule by just treating trash weakref objects with callbacks the same as objects with __del__ methods (i.e. move them to the finalizers list and then do the transitive closure of that set). I'm attaching a patch. It's a work in progress. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 11:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 10:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 05:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Sat Oct 30 19:35:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 19:35:18 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Neil Schemenauer (nascheme) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-30 13:35 Message: Logged In: YES user_id=31435 > Regarding gc_weakref.txt, I think we should add a note at > the top explaining the critical rule. An excellent idea -- I'll do that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 13:20 Message: Logged In: YES user_id=31435 > The approach looks sound but I believe you forgot to clear > the wr_callback pointer. I'm attaching an incremental patch > (relative to patch3.txt). Some of the comments in that incremental patch are good additions, but I don't agree with the code changes. A weakref object owns a (strong) reference to its wr_callback, and the first pass gives move_troublemakers its own new strong reference to each weakref object. Because of the latter, nothing any callback does can cause any of these weakrefs to go away, so nothing can happen to cause their wr_callback slots to become NULL either. The assertion that wr_callback isn't NULL is appropriate. If you still think it's possible for wr_callback to become NULL, please concoct a test showing it happen. The passes after the first give up move_troublemaker's references to the weakref objects. Then they'll go away, when their refcounts hit 0. Sometimes that happens right away, sometimes not. In either case, it's weakref_dealloc's job to decrement the refcount on w->wr_callback when w goes away, and there's no point (that I can see) to duplicating that logic too inside gcmodule. Note that the comments before _PyWeakref_ClearRef() explain that the weakref's tp_dealloc is expected to finish the job (clear the wr_callback slot) -- that's been true since _PyWeakref_ClearRef() was first introduced. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 11:27 Message: Logged In: YES user_id=35752 Attaching another patch. This one splits move_troublemakers() into move_finalizers() and handle_weakrefs(). The code is a little easier to understand and arguably more correct. The down side is that it adds another pass over `unreachable`. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 11:04 Message: Logged In: YES user_id=35752 I think your patch is about as elegant as it can be. The approach looks sound but I believe you forgot to clear the wr_callback pointer. I'm attaching an incremental patch (relative to patch3.txt). Regarding gc_weakref.txt, I think we should add a note at the top explaining the critical rule. I.e. after we can computed the unreachable set (aka trash), it's okay to invoke code that can access non-trash but we cannot allow code to run that can reference trash. There is only one minor issue that I can think of. We decide which callbacks to invoke inside move_troublemakers(). It's possible that unreachable weakrefs become reachable when move_finalizer_reachable() is called (i.e. weakrefs that would be trash except that they got revived because they were referenced by an object with a __del__ method). However, since those weakrefs would have been trash I think it's okay not to invoke their callbacks. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 03:55 Message: Logged In: YES user_id=31435 patch3.txt repairs some typos in comments; semantically, it's the same as patch2.txt. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 23:23 Message: Logged In: YES user_id=31435 patch2.txt stops abusing weakref `hash`, and seems to have no issues w/ Python's or Zope3's -uall/--all tests, in debug or release builds. weakrefs are handled entirely in move_troublemakers() now; collect() doesn't even have its old wr_callbacks list anymore. If there's a more elegant approach coming, great, but if it's not coming soon I'm going to check this in, to get as much developer test coverage as possible before 2.4.b1. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 13:13 Message: Logged In: YES user_id=31435 > I had to change _PyWeakref_ClearRef() since it was also > clearing the weakref list of the trash object. That was really its *purpose*. If a trash weakref with a callback isn't removed from the referent's list of weakrefs, then the callback will trigger when PyObject_ClearWeakRefs() is invoked on the referent. The purpose of _PyWeakref_ClearRef() was to ensure that the callback never triggers. > Now it just sets wr_object to Py_None. That won't stop the callback from triggering. It also means (see earlier comment) that PyObject_ClearWeakRefs() will never removed the weakref from the list either, although I'm not sure that does real harm. > I also made some serious simplifications to gcmodule by > just treating trash weakref objects with callbacks the same > as objects with __del__ methods (i.e. move them to the > finalizers list and then do the transitive closure of that set). Does that mean they can end up in gc.garbage too? If so, I don't think that's sellable. See the end of gc_weakref.txt: An alternative would have been to treat objects with callbacks like objects with __del__ methods, refusing to collect them, appending them to gc.garbage instead. That would have been much easier. Jim Fulton gave a strong argument against that (on Python-Dev): ... > I'm attaching a patch. It's a work in progress. Aren't we all . ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 13:00 Message: Logged In: YES user_id=35752 I had to change _PyWeakref_ClearRef() since it was also clearing the weakref list of the trash object. Now it just sets wr_object to Py_None. I also made some serious simplifications to gcmodule by just treating trash weakref objects with callbacks the same as objects with __del__ methods (i.e. move them to the finalizers list and then do the transitive closure of that set). I'm attaching a patch. It's a work in progress. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 11:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 10:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 05:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Sat Oct 30 19:51:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 19:51:56 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-28 01:58 Message generated for change (Comment added) made by nascheme You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) >Assigned to: Tim Peters (tim_one) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 17:51 Message: Logged In: YES user_id=35752 Ah,the detail that I was missing was that wr_callback is not invoked if the weakref list has been cleared. I.e., calling _PyWeakref_ClearRef() is enough ensure that the callback is never implicitly envoked. That's cleaner than explicitly clearing the wr_callback pointer and also seems to a behavior that can be depended on (it's pretty fundamental to how weakrefs are implemented). patch3.txt looks like a winner. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 17:35 Message: Logged In: YES user_id=31435 > Regarding gc_weakref.txt, I think we should add a note at > the top explaining the critical rule. An excellent idea -- I'll do that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 17:20 Message: Logged In: YES user_id=31435 > The approach looks sound but I believe you forgot to clear > the wr_callback pointer. I'm attaching an incremental patch > (relative to patch3.txt). Some of the comments in that incremental patch are good additions, but I don't agree with the code changes. A weakref object owns a (strong) reference to its wr_callback, and the first pass gives move_troublemakers its own new strong reference to each weakref object. Because of the latter, nothing any callback does can cause any of these weakrefs to go away, so nothing can happen to cause their wr_callback slots to become NULL either. The assertion that wr_callback isn't NULL is appropriate. If you still think it's possible for wr_callback to become NULL, please concoct a test showing it happen. The passes after the first give up move_troublemaker's references to the weakref objects. Then they'll go away, when their refcounts hit 0. Sometimes that happens right away, sometimes not. In either case, it's weakref_dealloc's job to decrement the refcount on w->wr_callback when w goes away, and there's no point (that I can see) to duplicating that logic too inside gcmodule. Note that the comments before _PyWeakref_ClearRef() explain that the weakref's tp_dealloc is expected to finish the job (clear the wr_callback slot) -- that's been true since _PyWeakref_ClearRef() was first introduced. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 15:27 Message: Logged In: YES user_id=35752 Attaching another patch. This one splits move_troublemakers() into move_finalizers() and handle_weakrefs(). The code is a little easier to understand and arguably more correct. The down side is that it adds another pass over `unreachable`. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 15:04 Message: Logged In: YES user_id=35752 I think your patch is about as elegant as it can be. The approach looks sound but I believe you forgot to clear the wr_callback pointer. I'm attaching an incremental patch (relative to patch3.txt). Regarding gc_weakref.txt, I think we should add a note at the top explaining the critical rule. I.e. after we can computed the unreachable set (aka trash), it's okay to invoke code that can access non-trash but we cannot allow code to run that can reference trash. There is only one minor issue that I can think of. We decide which callbacks to invoke inside move_troublemakers(). It's possible that unreachable weakrefs become reachable when move_finalizer_reachable() is called (i.e. weakrefs that would be trash except that they got revived because they were referenced by an object with a __del__ method). However, since those weakrefs would have been trash I think it's okay not to invoke their callbacks. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 07:55 Message: Logged In: YES user_id=31435 patch3.txt repairs some typos in comments; semantically, it's the same as patch2.txt. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 03:23 Message: Logged In: YES user_id=31435 patch2.txt stops abusing weakref `hash`, and seems to have no issues w/ Python's or Zope3's -uall/--all tests, in debug or release builds. weakrefs are handled entirely in move_troublemakers() now; collect() doesn't even have its old wr_callbacks list anymore. If there's a more elegant approach coming, great, but if it's not coming soon I'm going to check this in, to get as much developer test coverage as possible before 2.4.b1. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 17:13 Message: Logged In: YES user_id=31435 > I had to change _PyWeakref_ClearRef() since it was also > clearing the weakref list of the trash object. That was really its *purpose*. If a trash weakref with a callback isn't removed from the referent's list of weakrefs, then the callback will trigger when PyObject_ClearWeakRefs() is invoked on the referent. The purpose of _PyWeakref_ClearRef() was to ensure that the callback never triggers. > Now it just sets wr_object to Py_None. That won't stop the callback from triggering. It also means (see earlier comment) that PyObject_ClearWeakRefs() will never removed the weakref from the list either, although I'm not sure that does real harm. > I also made some serious simplifications to gcmodule by > just treating trash weakref objects with callbacks the same > as objects with __del__ methods (i.e. move them to the > finalizers list and then do the transitive closure of that set). Does that mean they can end up in gc.garbage too? If so, I don't think that's sellable. See the end of gc_weakref.txt: An alternative would have been to treat objects with callbacks like objects with __del__ methods, refusing to collect them, appending them to gc.garbage instead. That would have been much easier. Jim Fulton gave a strong argument against that (on Python-Dev): ... > I'm attaching a patch. It's a work in progress. Aren't we all . ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 17:00 Message: Logged In: YES user_id=35752 I had to change _PyWeakref_ClearRef() since it was also clearing the weakref list of the trash object. Now it just sets wr_object to Py_None. I also made some serious simplifications to gcmodule by just treating trash weakref objects with callbacks the same as objects with __del__ methods (i.e. move them to the finalizers list and then do the transitive closure of that set). I'm attaching a patch. It's a work in progress. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 16:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 16:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 15:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 14:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 09:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 06:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 20:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 08:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 08:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 02:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 02:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Sat Oct 30 20:28:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 20:28:49 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Tim Peters (tim_one) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-30 14:28 Message: Logged In: YES user_id=31435 > Ah,the detail that I was missing was that wr_callback is not > invoked if the weakref list has been cleared. I.e., calling > _PyWeakref_ClearRef() is enough ensure that the callback is > never implicitly envoked Indeed, ensuring that the callback never triggers by magic is a primary *purpose* of _PyWeakref_ClearRef(). I should add something here: this is so delicate in part because I didn't (during the *last* round of bug fixes) want to allocate any heap space during gc -- if malloc couldn't find more heap space when gc needed it, then the only thing I could do is kill the Python process. So the deeper and somewhat ugly truth here is that gc is (re) using the (already-allocated) weakref object as a container *just* to remember its wr_callback pointer. It would have been easier to understand if, instead, gc could have saved away the wr_callback pointers in its own PyList object, and used weakrefobject.c's clear_weakref() instead. Alas, there's no limit on how big such a list may need to become. As things turned out, it's a very good thing I couldn't do that. Fred subsequently made weakrefs subclassable, and now there are actual uses for passing the weakref object to the callback (e.g., weak value dictionaries use a weakref subclass now, and an instance of this subclass has the associated dict key as an attribute), so now a weakref object has to be kept around as long as its callback may still get invoked. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 13:51 Message: Logged In: YES user_id=35752 Ah,the detail that I was missing was that wr_callback is not invoked if the weakref list has been cleared. I.e., calling _PyWeakref_ClearRef() is enough ensure that the callback is never implicitly envoked. That's cleaner than explicitly clearing the wr_callback pointer and also seems to a behavior that can be depended on (it's pretty fundamental to how weakrefs are implemented). patch3.txt looks like a winner. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 13:35 Message: Logged In: YES user_id=31435 > Regarding gc_weakref.txt, I think we should add a note at > the top explaining the critical rule. An excellent idea -- I'll do that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 13:20 Message: Logged In: YES user_id=31435 > The approach looks sound but I believe you forgot to clear > the wr_callback pointer. I'm attaching an incremental patch > (relative to patch3.txt). Some of the comments in that incremental patch are good additions, but I don't agree with the code changes. A weakref object owns a (strong) reference to its wr_callback, and the first pass gives move_troublemakers its own new strong reference to each weakref object. Because of the latter, nothing any callback does can cause any of these weakrefs to go away, so nothing can happen to cause their wr_callback slots to become NULL either. The assertion that wr_callback isn't NULL is appropriate. If you still think it's possible for wr_callback to become NULL, please concoct a test showing it happen. The passes after the first give up move_troublemaker's references to the weakref objects. Then they'll go away, when their refcounts hit 0. Sometimes that happens right away, sometimes not. In either case, it's weakref_dealloc's job to decrement the refcount on w->wr_callback when w goes away, and there's no point (that I can see) to duplicating that logic too inside gcmodule. Note that the comments before _PyWeakref_ClearRef() explain that the weakref's tp_dealloc is expected to finish the job (clear the wr_callback slot) -- that's been true since _PyWeakref_ClearRef() was first introduced. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 11:27 Message: Logged In: YES user_id=35752 Attaching another patch. This one splits move_troublemakers() into move_finalizers() and handle_weakrefs(). The code is a little easier to understand and arguably more correct. The down side is that it adds another pass over `unreachable`. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 11:04 Message: Logged In: YES user_id=35752 I think your patch is about as elegant as it can be. The approach looks sound but I believe you forgot to clear the wr_callback pointer. I'm attaching an incremental patch (relative to patch3.txt). Regarding gc_weakref.txt, I think we should add a note at the top explaining the critical rule. I.e. after we can computed the unreachable set (aka trash), it's okay to invoke code that can access non-trash but we cannot allow code to run that can reference trash. There is only one minor issue that I can think of. We decide which callbacks to invoke inside move_troublemakers(). It's possible that unreachable weakrefs become reachable when move_finalizer_reachable() is called (i.e. weakrefs that would be trash except that they got revived because they were referenced by an object with a __del__ method). However, since those weakrefs would have been trash I think it's okay not to invoke their callbacks. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 03:55 Message: Logged In: YES user_id=31435 patch3.txt repairs some typos in comments; semantically, it's the same as patch2.txt. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 23:23 Message: Logged In: YES user_id=31435 patch2.txt stops abusing weakref `hash`, and seems to have no issues w/ Python's or Zope3's -uall/--all tests, in debug or release builds. weakrefs are handled entirely in move_troublemakers() now; collect() doesn't even have its old wr_callbacks list anymore. If there's a more elegant approach coming, great, but if it's not coming soon I'm going to check this in, to get as much developer test coverage as possible before 2.4.b1. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 13:13 Message: Logged In: YES user_id=31435 > I had to change _PyWeakref_ClearRef() since it was also > clearing the weakref list of the trash object. That was really its *purpose*. If a trash weakref with a callback isn't removed from the referent's list of weakrefs, then the callback will trigger when PyObject_ClearWeakRefs() is invoked on the referent. The purpose of _PyWeakref_ClearRef() was to ensure that the callback never triggers. > Now it just sets wr_object to Py_None. That won't stop the callback from triggering. It also means (see earlier comment) that PyObject_ClearWeakRefs() will never removed the weakref from the list either, although I'm not sure that does real harm. > I also made some serious simplifications to gcmodule by > just treating trash weakref objects with callbacks the same > as objects with __del__ methods (i.e. move them to the > finalizers list and then do the transitive closure of that set). Does that mean they can end up in gc.garbage too? If so, I don't think that's sellable. See the end of gc_weakref.txt: An alternative would have been to treat objects with callbacks like objects with __del__ methods, refusing to collect them, appending them to gc.garbage instead. That would have been much easier. Jim Fulton gave a strong argument against that (on Python-Dev): ... > I'm attaching a patch. It's a work in progress. Aren't we all . ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 13:00 Message: Logged In: YES user_id=35752 I had to change _PyWeakref_ClearRef() since it was also clearing the weakref list of the trash object. Now it just sets wr_object to Py_None. I also made some serious simplifications to gcmodule by just treating trash weakref objects with callbacks the same as objects with __del__ methods (i.e. move them to the finalizers list and then do the transitive closure of that set). I'm attaching a patch. It's a work in progress. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 11:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 10:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 05:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Sat Oct 30 20:47:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat Oct 30 20:47:56 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Tim Peters (tim_one) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-30 14:47 Message: Logged In: YES user_id=31435 > There is only one minor issue that I can think of. We > decide which callbacks to invoke inside > move_troublemakers(). It's possible that unreachable > weakrefs become reachable when > move_finalizer_reachable() is called (i.e. weakrefs that > would be trash except that they got revived because they > were referenced by an object with a __del__ method). > > However, since those weakrefs would have been trash I > think it's okay not to invoke their callbacks. I sure agree there are grey areas here! Since I hate __del__ methods, I'm afraid I tend to have a "oh, who gives a shit?" attitude toward them. There are also grey areas here outside of cyclic gc. Take a look at subtype_dealloc: that calls PyObject_ClearWeakRefs () before calling a __del__ method, so invokes weakref callbacks even if the object doesn't actually go away (the __del__ method could still resurrect it). Is that right or wrong? Beats me. I need to think more about this. Offhand I'm inclined to agree that your reworking to crawl over unreachable twice is the more principled approach. I only cringe because it *is* twice, and so probably a little slower. Can you write a test where it makes a visible difference? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 14:28 Message: Logged In: YES user_id=31435 > Ah,the detail that I was missing was that wr_callback is not > invoked if the weakref list has been cleared. I.e., calling > _PyWeakref_ClearRef() is enough ensure that the callback is > never implicitly envoked Indeed, ensuring that the callback never triggers by magic is a primary *purpose* of _PyWeakref_ClearRef(). I should add something here: this is so delicate in part because I didn't (during the *last* round of bug fixes) want to allocate any heap space during gc -- if malloc couldn't find more heap space when gc needed it, then the only thing I could do is kill the Python process. So the deeper and somewhat ugly truth here is that gc is (re) using the (already-allocated) weakref object as a container *just* to remember its wr_callback pointer. It would have been easier to understand if, instead, gc could have saved away the wr_callback pointers in its own PyList object, and used weakrefobject.c's clear_weakref() instead. Alas, there's no limit on how big such a list may need to become. As things turned out, it's a very good thing I couldn't do that. Fred subsequently made weakrefs subclassable, and now there are actual uses for passing the weakref object to the callback (e.g., weak value dictionaries use a weakref subclass now, and an instance of this subclass has the associated dict key as an attribute), so now a weakref object has to be kept around as long as its callback may still get invoked. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 13:51 Message: Logged In: YES user_id=35752 Ah,the detail that I was missing was that wr_callback is not invoked if the weakref list has been cleared. I.e., calling _PyWeakref_ClearRef() is enough ensure that the callback is never implicitly envoked. That's cleaner than explicitly clearing the wr_callback pointer and also seems to a behavior that can be depended on (it's pretty fundamental to how weakrefs are implemented). patch3.txt looks like a winner. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 13:35 Message: Logged In: YES user_id=31435 > Regarding gc_weakref.txt, I think we should add a note at > the top explaining the critical rule. An excellent idea -- I'll do that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 13:20 Message: Logged In: YES user_id=31435 > The approach looks sound but I believe you forgot to clear > the wr_callback pointer. I'm attaching an incremental patch > (relative to patch3.txt). Some of the comments in that incremental patch are good additions, but I don't agree with the code changes. A weakref object owns a (strong) reference to its wr_callback, and the first pass gives move_troublemakers its own new strong reference to each weakref object. Because of the latter, nothing any callback does can cause any of these weakrefs to go away, so nothing can happen to cause their wr_callback slots to become NULL either. The assertion that wr_callback isn't NULL is appropriate. If you still think it's possible for wr_callback to become NULL, please concoct a test showing it happen. The passes after the first give up move_troublemaker's references to the weakref objects. Then they'll go away, when their refcounts hit 0. Sometimes that happens right away, sometimes not. In either case, it's weakref_dealloc's job to decrement the refcount on w->wr_callback when w goes away, and there's no point (that I can see) to duplicating that logic too inside gcmodule. Note that the comments before _PyWeakref_ClearRef() explain that the weakref's tp_dealloc is expected to finish the job (clear the wr_callback slot) -- that's been true since _PyWeakref_ClearRef() was first introduced. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 11:27 Message: Logged In: YES user_id=35752 Attaching another patch. This one splits move_troublemakers() into move_finalizers() and handle_weakrefs(). The code is a little easier to understand and arguably more correct. The down side is that it adds another pass over `unreachable`. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 11:04 Message: Logged In: YES user_id=35752 I think your patch is about as elegant as it can be. The approach looks sound but I believe you forgot to clear the wr_callback pointer. I'm attaching an incremental patch (relative to patch3.txt). Regarding gc_weakref.txt, I think we should add a note at the top explaining the critical rule. I.e. after we can computed the unreachable set (aka trash), it's okay to invoke code that can access non-trash but we cannot allow code to run that can reference trash. There is only one minor issue that I can think of. We decide which callbacks to invoke inside move_troublemakers(). It's possible that unreachable weakrefs become reachable when move_finalizer_reachable() is called (i.e. weakrefs that would be trash except that they got revived because they were referenced by an object with a __del__ method). However, since those weakrefs would have been trash I think it's okay not to invoke their callbacks. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 03:55 Message: Logged In: YES user_id=31435 patch3.txt repairs some typos in comments; semantically, it's the same as patch2.txt. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 23:23 Message: Logged In: YES user_id=31435 patch2.txt stops abusing weakref `hash`, and seems to have no issues w/ Python's or Zope3's -uall/--all tests, in debug or release builds. weakrefs are handled entirely in move_troublemakers() now; collect() doesn't even have its old wr_callbacks list anymore. If there's a more elegant approach coming, great, but if it's not coming soon I'm going to check this in, to get as much developer test coverage as possible before 2.4.b1. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 13:13 Message: Logged In: YES user_id=31435 > I had to change _PyWeakref_ClearRef() since it was also > clearing the weakref list of the trash object. That was really its *purpose*. If a trash weakref with a callback isn't removed from the referent's list of weakrefs, then the callback will trigger when PyObject_ClearWeakRefs() is invoked on the referent. The purpose of _PyWeakref_ClearRef() was to ensure that the callback never triggers. > Now it just sets wr_object to Py_None. That won't stop the callback from triggering. It also means (see earlier comment) that PyObject_ClearWeakRefs() will never removed the weakref from the list either, although I'm not sure that does real harm. > I also made some serious simplifications to gcmodule by > just treating trash weakref objects with callbacks the same > as objects with __del__ methods (i.e. move them to the > finalizers list and then do the transitive closure of that set). Does that mean they can end up in gc.garbage too? If so, I don't think that's sellable. See the end of gc_weakref.txt: An alternative would have been to treat objects with callbacks like objects with __del__ methods, refusing to collect them, appending them to gc.garbage instead. That would have been much easier. Jim Fulton gave a strong argument against that (on Python-Dev): ... > I'm attaching a patch. It's a work in progress. Aren't we all . ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 13:00 Message: Logged In: YES user_id=35752 I had to change _PyWeakref_ClearRef() since it was also clearing the weakref list of the trash object. Now it just sets wr_object to Py_None. I also made some serious simplifications to gcmodule by just treating trash weakref objects with callbacks the same as objects with __del__ methods (i.e. move them to the finalizers list and then do the transitive closure of that set). I'm attaching a patch. It's a work in progress. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 11:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 10:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 05:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Sun Oct 31 00:49:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 31 00:49:23 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Tim Peters (tim_one) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-30 18:49 Message: Logged In: YES user_id=31435 patch42.txt is what I'll check in (and backport to 2.3 maint) provided my remaining tests all pass. This does two passes over `unreachable` as in Neil's patch4.txt, and incorporates various improvements to the comments. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 14:47 Message: Logged In: YES user_id=31435 > There is only one minor issue that I can think of. We > decide which callbacks to invoke inside > move_troublemakers(). It's possible that unreachable > weakrefs become reachable when > move_finalizer_reachable() is called (i.e. weakrefs that > would be trash except that they got revived because they > were referenced by an object with a __del__ method). > > However, since those weakrefs would have been trash I > think it's okay not to invoke their callbacks. I sure agree there are grey areas here! Since I hate __del__ methods, I'm afraid I tend to have a "oh, who gives a shit?" attitude toward them. There are also grey areas here outside of cyclic gc. Take a look at subtype_dealloc: that calls PyObject_ClearWeakRefs () before calling a __del__ method, so invokes weakref callbacks even if the object doesn't actually go away (the __del__ method could still resurrect it). Is that right or wrong? Beats me. I need to think more about this. Offhand I'm inclined to agree that your reworking to crawl over unreachable twice is the more principled approach. I only cringe because it *is* twice, and so probably a little slower. Can you write a test where it makes a visible difference? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 14:28 Message: Logged In: YES user_id=31435 > Ah,the detail that I was missing was that wr_callback is not > invoked if the weakref list has been cleared. I.e., calling > _PyWeakref_ClearRef() is enough ensure that the callback is > never implicitly envoked Indeed, ensuring that the callback never triggers by magic is a primary *purpose* of _PyWeakref_ClearRef(). I should add something here: this is so delicate in part because I didn't (during the *last* round of bug fixes) want to allocate any heap space during gc -- if malloc couldn't find more heap space when gc needed it, then the only thing I could do is kill the Python process. So the deeper and somewhat ugly truth here is that gc is (re) using the (already-allocated) weakref object as a container *just* to remember its wr_callback pointer. It would have been easier to understand if, instead, gc could have saved away the wr_callback pointers in its own PyList object, and used weakrefobject.c's clear_weakref() instead. Alas, there's no limit on how big such a list may need to become. As things turned out, it's a very good thing I couldn't do that. Fred subsequently made weakrefs subclassable, and now there are actual uses for passing the weakref object to the callback (e.g., weak value dictionaries use a weakref subclass now, and an instance of this subclass has the associated dict key as an attribute), so now a weakref object has to be kept around as long as its callback may still get invoked. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 13:51 Message: Logged In: YES user_id=35752 Ah,the detail that I was missing was that wr_callback is not invoked if the weakref list has been cleared. I.e., calling _PyWeakref_ClearRef() is enough ensure that the callback is never implicitly envoked. That's cleaner than explicitly clearing the wr_callback pointer and also seems to a behavior that can be depended on (it's pretty fundamental to how weakrefs are implemented). patch3.txt looks like a winner. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 13:35 Message: Logged In: YES user_id=31435 > Regarding gc_weakref.txt, I think we should add a note at > the top explaining the critical rule. An excellent idea -- I'll do that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 13:20 Message: Logged In: YES user_id=31435 > The approach looks sound but I believe you forgot to clear > the wr_callback pointer. I'm attaching an incremental patch > (relative to patch3.txt). Some of the comments in that incremental patch are good additions, but I don't agree with the code changes. A weakref object owns a (strong) reference to its wr_callback, and the first pass gives move_troublemakers its own new strong reference to each weakref object. Because of the latter, nothing any callback does can cause any of these weakrefs to go away, so nothing can happen to cause their wr_callback slots to become NULL either. The assertion that wr_callback isn't NULL is appropriate. If you still think it's possible for wr_callback to become NULL, please concoct a test showing it happen. The passes after the first give up move_troublemaker's references to the weakref objects. Then they'll go away, when their refcounts hit 0. Sometimes that happens right away, sometimes not. In either case, it's weakref_dealloc's job to decrement the refcount on w->wr_callback when w goes away, and there's no point (that I can see) to duplicating that logic too inside gcmodule. Note that the comments before _PyWeakref_ClearRef() explain that the weakref's tp_dealloc is expected to finish the job (clear the wr_callback slot) -- that's been true since _PyWeakref_ClearRef() was first introduced. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 11:27 Message: Logged In: YES user_id=35752 Attaching another patch. This one splits move_troublemakers() into move_finalizers() and handle_weakrefs(). The code is a little easier to understand and arguably more correct. The down side is that it adds another pass over `unreachable`. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 11:04 Message: Logged In: YES user_id=35752 I think your patch is about as elegant as it can be. The approach looks sound but I believe you forgot to clear the wr_callback pointer. I'm attaching an incremental patch (relative to patch3.txt). Regarding gc_weakref.txt, I think we should add a note at the top explaining the critical rule. I.e. after we can computed the unreachable set (aka trash), it's okay to invoke code that can access non-trash but we cannot allow code to run that can reference trash. There is only one minor issue that I can think of. We decide which callbacks to invoke inside move_troublemakers(). It's possible that unreachable weakrefs become reachable when move_finalizer_reachable() is called (i.e. weakrefs that would be trash except that they got revived because they were referenced by an object with a __del__ method). However, since those weakrefs would have been trash I think it's okay not to invoke their callbacks. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 03:55 Message: Logged In: YES user_id=31435 patch3.txt repairs some typos in comments; semantically, it's the same as patch2.txt. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 23:23 Message: Logged In: YES user_id=31435 patch2.txt stops abusing weakref `hash`, and seems to have no issues w/ Python's or Zope3's -uall/--all tests, in debug or release builds. weakrefs are handled entirely in move_troublemakers() now; collect() doesn't even have its old wr_callbacks list anymore. If there's a more elegant approach coming, great, but if it's not coming soon I'm going to check this in, to get as much developer test coverage as possible before 2.4.b1. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 13:13 Message: Logged In: YES user_id=31435 > I had to change _PyWeakref_ClearRef() since it was also > clearing the weakref list of the trash object. That was really its *purpose*. If a trash weakref with a callback isn't removed from the referent's list of weakrefs, then the callback will trigger when PyObject_ClearWeakRefs() is invoked on the referent. The purpose of _PyWeakref_ClearRef() was to ensure that the callback never triggers. > Now it just sets wr_object to Py_None. That won't stop the callback from triggering. It also means (see earlier comment) that PyObject_ClearWeakRefs() will never removed the weakref from the list either, although I'm not sure that does real harm. > I also made some serious simplifications to gcmodule by > just treating trash weakref objects with callbacks the same > as objects with __del__ methods (i.e. move them to the > finalizers list and then do the transitive closure of that set). Does that mean they can end up in gc.garbage too? If so, I don't think that's sellable. See the end of gc_weakref.txt: An alternative would have been to treat objects with callbacks like objects with __del__ methods, refusing to collect them, appending them to gc.garbage instead. That would have been much easier. Jim Fulton gave a strong argument against that (on Python-Dev): ... > I'm attaching a patch. It's a work in progress. Aren't we all . ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 13:00 Message: Logged In: YES user_id=35752 I had to change _PyWeakref_ClearRef() since it was also clearing the weakref list of the trash object. Now it just sets wr_object to Py_None. I also made some serious simplifications to gcmodule by just treating trash weakref objects with callbacks the same as objects with __del__ methods (i.e. move them to the finalizers list and then do the transitive closure of that set). I'm attaching a patch. It's a work in progress. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 11:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 10:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 05:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Sun Oct 31 01:13:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 31 01:14:05 2004 Subject: [ python-Bugs-1055820 ] weakref callback vs gc vs threads Message-ID: Bugs item #1055820, was opened at 2004-10-27 21:58 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 Category: Python Interpreter Core Group: None >Status: Closed >Resolution: Fixed Priority: 9 Submitted By: Tim Peters (tim_one) Assigned to: Tim Peters (tim_one) Summary: weakref callback vs gc vs threads Initial Comment: Oh oh. It's that time of year again. I'm sure the attached (temp2a.py) can be simplified, perhaps down to two objects and one thread. As is, I *think* it demonstrates that invoking a weakref callback can do fatal damage, not necessarily because of what the callback does, but because simply calling it while gc is running can allow other threads to run during gc too, and resurrect a piece of cyclic trash T that's already been tp_clear()'ed, via invoking a still-living weakref to T. If so, this isn't new in 2.4. It's a real problem since temp2a.py is what's left of ZODB 3.4a1 . ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-10-30 19:13 Message: Logged In: YES user_id=31435 Thanks for the eyeballs and brain cells, Neil! patch42 was checked in: Include/weakrefobject.h 1.6 Lib/test/test_gc.py 1.31 Misc/NEWS 1.1179 Modules/gc_weakref.txt 2.2 Modules/gcmodule.c 2.77 Objects/weakrefobject.c 1.20 I'll backport to 2.3 next, and that's it, so closing this report. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 18:49 Message: Logged In: YES user_id=31435 patch42.txt is what I'll check in (and backport to 2.3 maint) provided my remaining tests all pass. This does two passes over `unreachable` as in Neil's patch4.txt, and incorporates various improvements to the comments. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 14:47 Message: Logged In: YES user_id=31435 > There is only one minor issue that I can think of. We > decide which callbacks to invoke inside > move_troublemakers(). It's possible that unreachable > weakrefs become reachable when > move_finalizer_reachable() is called (i.e. weakrefs that > would be trash except that they got revived because they > were referenced by an object with a __del__ method). > > However, since those weakrefs would have been trash I > think it's okay not to invoke their callbacks. I sure agree there are grey areas here! Since I hate __del__ methods, I'm afraid I tend to have a "oh, who gives a shit?" attitude toward them. There are also grey areas here outside of cyclic gc. Take a look at subtype_dealloc: that calls PyObject_ClearWeakRefs () before calling a __del__ method, so invokes weakref callbacks even if the object doesn't actually go away (the __del__ method could still resurrect it). Is that right or wrong? Beats me. I need to think more about this. Offhand I'm inclined to agree that your reworking to crawl over unreachable twice is the more principled approach. I only cringe because it *is* twice, and so probably a little slower. Can you write a test where it makes a visible difference? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 14:28 Message: Logged In: YES user_id=31435 > Ah,the detail that I was missing was that wr_callback is not > invoked if the weakref list has been cleared. I.e., calling > _PyWeakref_ClearRef() is enough ensure that the callback is > never implicitly envoked Indeed, ensuring that the callback never triggers by magic is a primary *purpose* of _PyWeakref_ClearRef(). I should add something here: this is so delicate in part because I didn't (during the *last* round of bug fixes) want to allocate any heap space during gc -- if malloc couldn't find more heap space when gc needed it, then the only thing I could do is kill the Python process. So the deeper and somewhat ugly truth here is that gc is (re) using the (already-allocated) weakref object as a container *just* to remember its wr_callback pointer. It would have been easier to understand if, instead, gc could have saved away the wr_callback pointers in its own PyList object, and used weakrefobject.c's clear_weakref() instead. Alas, there's no limit on how big such a list may need to become. As things turned out, it's a very good thing I couldn't do that. Fred subsequently made weakrefs subclassable, and now there are actual uses for passing the weakref object to the callback (e.g., weak value dictionaries use a weakref subclass now, and an instance of this subclass has the associated dict key as an attribute), so now a weakref object has to be kept around as long as its callback may still get invoked. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 13:51 Message: Logged In: YES user_id=35752 Ah,the detail that I was missing was that wr_callback is not invoked if the weakref list has been cleared. I.e., calling _PyWeakref_ClearRef() is enough ensure that the callback is never implicitly envoked. That's cleaner than explicitly clearing the wr_callback pointer and also seems to a behavior that can be depended on (it's pretty fundamental to how weakrefs are implemented). patch3.txt looks like a winner. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 13:35 Message: Logged In: YES user_id=31435 > Regarding gc_weakref.txt, I think we should add a note at > the top explaining the critical rule. An excellent idea -- I'll do that. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 13:20 Message: Logged In: YES user_id=31435 > The approach looks sound but I believe you forgot to clear > the wr_callback pointer. I'm attaching an incremental patch > (relative to patch3.txt). Some of the comments in that incremental patch are good additions, but I don't agree with the code changes. A weakref object owns a (strong) reference to its wr_callback, and the first pass gives move_troublemakers its own new strong reference to each weakref object. Because of the latter, nothing any callback does can cause any of these weakrefs to go away, so nothing can happen to cause their wr_callback slots to become NULL either. The assertion that wr_callback isn't NULL is appropriate. If you still think it's possible for wr_callback to become NULL, please concoct a test showing it happen. The passes after the first give up move_troublemaker's references to the weakref objects. Then they'll go away, when their refcounts hit 0. Sometimes that happens right away, sometimes not. In either case, it's weakref_dealloc's job to decrement the refcount on w->wr_callback when w goes away, and there's no point (that I can see) to duplicating that logic too inside gcmodule. Note that the comments before _PyWeakref_ClearRef() explain that the weakref's tp_dealloc is expected to finish the job (clear the wr_callback slot) -- that's been true since _PyWeakref_ClearRef() was first introduced. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 11:27 Message: Logged In: YES user_id=35752 Attaching another patch. This one splits move_troublemakers() into move_finalizers() and handle_weakrefs(). The code is a little easier to understand and arguably more correct. The down side is that it adds another pass over `unreachable`. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-30 11:04 Message: Logged In: YES user_id=35752 I think your patch is about as elegant as it can be. The approach looks sound but I believe you forgot to clear the wr_callback pointer. I'm attaching an incremental patch (relative to patch3.txt). Regarding gc_weakref.txt, I think we should add a note at the top explaining the critical rule. I.e. after we can computed the unreachable set (aka trash), it's okay to invoke code that can access non-trash but we cannot allow code to run that can reference trash. There is only one minor issue that I can think of. We decide which callbacks to invoke inside move_troublemakers(). It's possible that unreachable weakrefs become reachable when move_finalizer_reachable() is called (i.e. weakrefs that would be trash except that they got revived because they were referenced by an object with a __del__ method). However, since those weakrefs would have been trash I think it's okay not to invoke their callbacks. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-30 03:55 Message: Logged In: YES user_id=31435 patch3.txt repairs some typos in comments; semantically, it's the same as patch2.txt. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 23:23 Message: Logged In: YES user_id=31435 patch2.txt stops abusing weakref `hash`, and seems to have no issues w/ Python's or Zope3's -uall/--all tests, in debug or release builds. weakrefs are handled entirely in move_troublemakers() now; collect() doesn't even have its old wr_callbacks list anymore. If there's a more elegant approach coming, great, but if it's not coming soon I'm going to check this in, to get as much developer test coverage as possible before 2.4.b1. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 13:13 Message: Logged In: YES user_id=31435 > I had to change _PyWeakref_ClearRef() since it was also > clearing the weakref list of the trash object. That was really its *purpose*. If a trash weakref with a callback isn't removed from the referent's list of weakrefs, then the callback will trigger when PyObject_ClearWeakRefs() is invoked on the referent. The purpose of _PyWeakref_ClearRef() was to ensure that the callback never triggers. > Now it just sets wr_object to Py_None. That won't stop the callback from triggering. It also means (see earlier comment) that PyObject_ClearWeakRefs() will never removed the weakref from the list either, although I'm not sure that does real harm. > I also made some serious simplifications to gcmodule by > just treating trash weakref objects with callbacks the same > as objects with __del__ methods (i.e. move them to the > finalizers list and then do the transitive closure of that set). Does that mean they can end up in gc.garbage too? If so, I don't think that's sellable. See the end of gc_weakref.txt: An alternative would have been to treat objects with callbacks like objects with __del__ methods, refusing to collect them, appending them to gc.garbage instead. That would have been much easier. Jim Fulton gave a strong argument against that (on Python-Dev): ... > I'm attaching a patch. It's a work in progress. Aren't we all . ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 13:00 Message: Logged In: YES user_id=35752 I had to change _PyWeakref_ClearRef() since it was also clearing the weakref list of the trash object. Now it just sets wr_object to Py_None. I also made some serious simplifications to gcmodule by just treating trash weakref objects with callbacks the same as objects with __del__ methods (i.e. move them to the finalizers list and then do the transitive closure of that set). I'm attaching a patch. It's a work in progress. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:41 Message: Logged In: YES user_id=31435 Bingo. This is a bit delicate . It's still necessary to call _PyWeakref_ClearRef() on a trash weakref with a callback, to prevent the callback from ever triggering (that was the key to fixing the previous pile of bugs). For all other weakrefs to trash, I think you're right that just setting wr_object to None is conceptually enough. But ... What I pissed away the last half hour discovering is that if you set wr_object to None *before* calling _PyWeakref_ClearRef(), then the latter doesn't do anything, because clear_weakref() doesn't do anything when wr_object is None. So that leaves me a little worried: if we just set wr_object to None on some weakrefs, then PyObject_ClearWeakrefs() will never (and for the same reason) remove such a weakref from its doubly-linked list either. Doesn't look like the weakref code intended that this be possible, and I'm not yet entirely convinced it can't hurt ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 12:01 Message: Logged In: YES user_id=31435 Hmm. Just about anything fixes the current rash of bugs. It's keeping the old bugs from coming back that's been the hangup -- don't forget to run test_weakref.py too . The last batch of bugs was really much subtler than this batch! test_callback_in_cycle_1 is a bitch -- I think we have to stop its callback from ever getting invoked, not just prevent I.wr() from returning J ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 11:18 Message: Logged In: YES user_id=31435 Excellent! I think you're right about this. It never occurred to me that just setting wr_object to None would be as effective at disabling a weakref as calling clear_weakref(). If that's really so (& I can't see why not offhand), it would be better in oh-so-many ways. ---------------------------------------------------------------------- Comment By: Neil Schemenauer (nascheme) Date: 2004-10-29 10:56 Message: Logged In: YES user_id=35752 I don't see why you need the extra bit. When you find weakly referenced objects, clear the wr_object pointer of the weakref. Move trash weakref objects with callbacks to the wr_callbacks list as we always did. When the trash goes away then PyObject_ClearWeakRefs() will invoke the callbacks normally (we only cleared the wr_object pointer, the backpointer list is still intact). I'm going to see if I can make this work. In the process I will probably discover what I have been missing. :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 05:11 Message: Logged In: YES user_id=31435 I strongly suspect that abusing the weakref struct's `hash` member is responsible for the gazillions of KeyErrors. Dang. I need one lousy bit ... ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-29 02:54 Message: Logged In: YES user_id=31435 Neil, can you make some time to eyeball this (patch.txt)? If not, please assign to Fred. The Python -uall suite passes in debug and release builds. I just ran the -all Zope3 test suite in a debug buld, and that passes as well as it does with Python 2.3.4 on Windows, but I see many thousands of lines like: Exception exceptions.KeyError: in ignored So there's something still wrong here. I'm not sure who's writing those msgs; I expect it's in the guts of gc, when invoking delayed weakref callbacks, and triggered by the weak dictionary implementations (which haven't changed). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 16:55 Message: Logged In: YES user_id=31435 A fix is in progress, as sketched on Python-Dev. I expect it to land Friday (or tonight yet, if I'm lucky). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:56 Message: Logged In: YES user_id=31435 temp2d.py shows that we don't need any weakrefs with callbacks to get in trouble -- a __del__ method can hose us too. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-28 04:06 Message: Logged In: YES user_id=31435 temp2c.py may be as bad as it gets. It shows that the problem can occur on a gc collection that doesn't see *any* object that has a weakref with a callback. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:37 Message: Logged In: YES user_id=31435 Noting that temp2b.py fails in current CVS, 2.3.4, and 2.2.3. That's all the Pythons I have handy right now. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-10-27 22:28 Message: Logged In: YES user_id=31435 Yup, temp2b.py probably shows the same problem, with 2 objects and one thread. This one is definitely more strained, though, since the weakref callback does the damage directly. In temp2a.py, nothing is trying to fool anything, and the only damage done by the wr callbacks there is simply in releasing the GIL (thus allowing other threads to do perfectly ordinary things with weakrefs). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1055820&group_id=5470 From noreply at sourceforge.net Sun Oct 31 02:19:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 31 02:19:37 2004 Subject: [ python-Bugs-1057535 ] add link in time module to calendar.timegm() Message-ID: Bugs item #1057535, was opened at 2004-10-30 21:19 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057535&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jeff Shute (jshute) Assigned to: Nobody/Anonymous (nobody) Summary: add link in time module to calendar.timegm() Initial Comment: There should be a link in the documentation for time.gmtime() to calendar.timegm(), which is its inverse. The timegm() function is really in the wrong module, which makes it very difficult to find. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057535&group_id=5470 From noreply at sourceforge.net Sun Oct 31 07:30:02 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 31 07:30:21 2004 Subject: [ python-Bugs-1056441 ] subprocess fails for args="...", executable="..." Message-ID: Bugs item #1056441, was opened at 2004-10-29 00:32 Message generated for change (Comment added) made by astrand You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056441&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Russell Owen (reowen) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess fails for args="...", executable="..." Initial Comment: In Python 2.4b1 I am trying to use the subprocess module and running into problems when args is a string and executable is specified. For example: >>> p = subprocess.Popen( executable = "xpaget", args = "xpaget ds9 mode", shell = True, stdout = subprocess.PIPE, ) results in this mysterious error msg: >>> sh: illegal option -- c and the data in p.stdout is a boatload of help text that strongly suggests xpaget never got the "ds9 mode" command. Piping stdin and stderr make no difference, of course, but omitting the stdout makes one strange difference: I don't see the error message, just the boatload of help text. Removing the executable argument makes it work as expected: >>> p = subprocess.Popen( args = "xpaget ds9 mode", shell = True, stdout = subprocess.PIPE, ) >>> p.stdout.read() 'pointer\n' And the executable argument works fine if I specify the arguments as a list and don't use shell=True: p = subprocess.Popen( executable = "xpaget", args = ["xpaget", "ds9", "mode"], stdout = subprocess.PIPE, ) >>> p.stdout.read() 'pointer\n' xpa and ds9 are free from but I hope think they are not needed to debug this. I saw this problem using a unix installation of Python 2.4b1 on MacOS X 10.3.5. ---------------------------------------------------------------------- >Comment By: Peter ?strand (astrand) Date: 2004-10-31 07:30 Message: Logged In: YES user_id=344921 Russell: What did you expect? Shells doesn't support that you specify an alternative executable, only the raw system calls do. Currently, if you use shell=True, the executable argument can be used to specify an alternative shell, if you are not happy with the default /bin/sh. I think this is useful. I must admit that this is not very well documented, though. ---------------------------------------------------------------------- Comment By: Russell Owen (reowen) Date: 2004-10-29 23:58 Message: Logged In: YES user_id=431773 I looked at the code and the problem is here: def _execute_child(self, args, executable, preexec_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite): """Execute program (POSIX version)""" if isinstance(args, types.StringTypes): args = [args] if shell: args = ["/bin/sh", "-c"] + args if executable == None: executable = args[0] You can se that if shell is true (as it must be if one specifies args as a string) and executable is supplied, one ends p with a mess, i.e. the executable ends up trying to execute ["/bin/sh", "-c", args] Fortunately cwd offers a hackaround. Still, I hope this can be either fixed or else can produce a warning. While on that subject, should specifying args as a string without specifying shell=True should produce a warning? Thanks for subprocess. It's a wonderful module. I am really looking forward to using it (once a few kinks get worked out). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056441&group_id=5470 From noreply at sourceforge.net Sun Oct 31 07:33:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 31 07:33:28 2004 Subject: [ python-Bugs-1056441 ] subprocess fails for args="...", executable="..." Message-ID: Bugs item #1056441, was opened at 2004-10-29 00:32 Message generated for change (Comment added) made by astrand You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056441&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Russell Owen (reowen) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess fails for args="...", executable="..." Initial Comment: In Python 2.4b1 I am trying to use the subprocess module and running into problems when args is a string and executable is specified. For example: >>> p = subprocess.Popen( executable = "xpaget", args = "xpaget ds9 mode", shell = True, stdout = subprocess.PIPE, ) results in this mysterious error msg: >>> sh: illegal option -- c and the data in p.stdout is a boatload of help text that strongly suggests xpaget never got the "ds9 mode" command. Piping stdin and stderr make no difference, of course, but omitting the stdout makes one strange difference: I don't see the error message, just the boatload of help text. Removing the executable argument makes it work as expected: >>> p = subprocess.Popen( args = "xpaget ds9 mode", shell = True, stdout = subprocess.PIPE, ) >>> p.stdout.read() 'pointer\n' And the executable argument works fine if I specify the arguments as a list and don't use shell=True: p = subprocess.Popen( executable = "xpaget", args = ["xpaget", "ds9", "mode"], stdout = subprocess.PIPE, ) >>> p.stdout.read() 'pointer\n' xpa and ds9 are free from but I hope think they are not needed to debug this. I saw this problem using a unix installation of Python 2.4b1 on MacOS X 10.3.5. ---------------------------------------------------------------------- >Comment By: Peter ?strand (astrand) Date: 2004-10-31 07:33 Message: Logged In: YES user_id=344921 reowen: >While on that subject, should specifying args as a string without >specifying shell=True should produce a warning? No. This is documented: "A string will be treated as a sequence with the string as the only item (the program to execute)." >Thanks for subprocess. It's a wonderful module. Thanks :) ---------------------------------------------------------------------- Comment By: Peter ?strand (astrand) Date: 2004-10-31 07:30 Message: Logged In: YES user_id=344921 Russell: What did you expect? Shells doesn't support that you specify an alternative executable, only the raw system calls do. Currently, if you use shell=True, the executable argument can be used to specify an alternative shell, if you are not happy with the default /bin/sh. I think this is useful. I must admit that this is not very well documented, though. ---------------------------------------------------------------------- Comment By: Russell Owen (reowen) Date: 2004-10-29 23:58 Message: Logged In: YES user_id=431773 I looked at the code and the problem is here: def _execute_child(self, args, executable, preexec_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite): """Execute program (POSIX version)""" if isinstance(args, types.StringTypes): args = [args] if shell: args = ["/bin/sh", "-c"] + args if executable == None: executable = args[0] You can se that if shell is true (as it must be if one specifies args as a string) and executable is supplied, one ends p with a mess, i.e. the executable ends up trying to execute ["/bin/sh", "-c", args] Fortunately cwd offers a hackaround. Still, I hope this can be either fixed or else can produce a warning. While on that subject, should specifying args as a string without specifying shell=True should produce a warning? Thanks for subprocess. It's a wonderful module. I am really looking forward to using it (once a few kinks get worked out). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056441&group_id=5470 From noreply at sourceforge.net Sun Oct 31 08:05:22 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 31 08:05:27 2004 Subject: [ python-Bugs-1057048 ] subprocess works poorly on Windows with Python 2.3 Message-ID: Bugs item #1057048, was opened at 2004-10-29 23:24 Message generated for change (Comment added) made by astrand You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057048&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Russell Owen (reowen) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess works poorly on Windows with Python 2.3 Initial Comment: My understanding was that subprocess.py was supposed to be backwards compatible with at least Python 2.3 if not 2.2. Wanting subprocess and backwards compatibility, I grabbed the subprocess.py from 2.4b1, changed the import if so that win32api was used (since I had no _subprocess library!) and found several problems, of which this is one (one per bug report): If the executable has a space in its path name (as is standard on Windows) and command-line arguments, things fail miserably: from subprocess import * Popen('"C:\Program Files\ds9\xpaget" ds9 mode') produces an error that C:\Program cannot be run. Clearly the double quoting isn't making it through Popen. Note that this runs just fine if there are no command line arguments, i.e. this works: Popen('"C:\Program Files\ds9\xpaget"') Note that this same bug also exists in os.popen3, so you may have inherited it. This bug does NOT occur on a unix build on MacOS X. ---------------------------------------------------------------------- >Comment By: Peter ?strand (astrand) Date: 2004-10-31 08:05 Message: Logged In: YES user_id=344921 Are you running on Windows 9X, or using command.com? If you are using shell=False (as in your example), subprocess doesn't change the argument string in any way: It is passed directly to CreateProcess. Can you confirm that the problem only exists with Python 2.3, and not with 2.4? In that case, it might be a bug in win32api:s CreateProcess. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057048&group_id=5470 From noreply at sourceforge.net Sun Oct 31 08:09:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 31 08:09:05 2004 Subject: [ python-Bugs-1057061 ] subprocess on Windows: extra windows Message-ID: Bugs item #1057061, was opened at 2004-10-29 23:49 Message generated for change (Comment added) made by astrand You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057061&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Russell Owen (reowen) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess on Windows: extra windows Initial Comment: My understanding was that subprocess.py was supposed to be backwards compatible with at least Python 2.3 if not 2.2. Wanting subprocess and backwards compatibility, I grabbed the subprocess.py from 2.4b1, changed the import if so that win32api was used (since I had no _subprocess library!) and found several problems, of which this is is the mildes: when using subprocess.Popen on Windows I get one or more windows called C:\WINNT\system32\CMD.EXE popping up. Sometimes they stick around, other times they are only around for a short time. Note that I don't see this probem using os.popen3 (on the other hand, os.popen3 has inferior error handling). The following code is one example: from subprocess import * Popen("ds9 -title foo", shell=True, cwd="C:\Program Files\ds9\") ---------------------------------------------------------------------- >Comment By: Peter ?strand (astrand) Date: 2004-10-31 08:09 Message: Logged In: YES user_id=344921 Which Windows version are you using? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057061&group_id=5470 From noreply at sourceforge.net Sun Oct 31 13:07:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 31 13:07:05 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-18 03:24 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-31 13:07 Message: Logged In: YES user_id=469548 Ok, checked in rmtree-ng.py version as rev. 1.34 of shutil.py and rev 1.8 of test_shutil.py. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-27 17:56 Message: Logged In: YES user_id=6380 - I don't like the nested function -- Python's function call overhead is significant and I worry about that when deleting massive trees. (I want to be able to "rm -rf /*" efficiently. :-) - The err instance instead of sys.exc_info() was an oversight. The docstring repeats this mistake. - I don't think the lstat() call is unclean, and again I prefer performance. - Here's an additional trick to reduce the code duplication (see upload rmtree-ng.py): At the top of the function put: if ignore_errors: ....def onerror(*args): pass elif onerror is None: ....def onerror(*args): raise Then all the except blocks can be collapsed to just the onerror() call. (I care less about the performance in case there's an error.) ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-27 11:57 Message: Logged In: YES user_id=469548 Yes, I think this is the right way to go about it. I'd personally make a couple of changes, but those are up to your discretion: * Factor out the try: except:'s to a nested function. * Use sys.exc_info() instead of the exception for backwards compatibility. * Use 'os.path.isdir(fullname) and not os.path.islink(fullname)' expression instead of the lstat call. I think it more clearly communicates what we want, even though it's more stat calls. Latest version attached (rmtree-refactored.py). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-26 18:08 Message: Logged In: YES user_id=6380 How about this no-nonsense version? (rmtree.txt upload) The only thing you could quibble about is what to do when os.lstat fails. I chose backward compatibility. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-23 20:25 Message: Logged In: YES user_id=6380 BTW, if you want the correct path in the onerror calls for os.listdir, your best bet is not to use os.walk but to write something similar that suits our purpose. After you remove the features we don't use, it's really only a few lines of code. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-23 19:56 Message: Logged In: YES user_id=6380 I disagree. The docs are wrong because a failing listdir wasn't anticipated at first when this function was written. A failing listdir is a common failure case; I don't see how the os.path calls in os.walk could ever fail (since they all catch os.error). ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-23 05:01 Message: Logged In: YES user_id=469548 You know what: I think we should remove the os.listdir error handling: * The docs only specify "errors resulting from *failed removals*" as being handled. If they didn't we should worry about OSErrors from the os.path.* calls in os.walk() too. * The os.listdir error handling hasn't been in a final release, so no backwards compatibility problems yet. * We can't get the path right for os.listdir. * The problem also existed before the rewrite using os.walk(), so reverting to that is not a solution. Patch (the last?) is attached. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-21 06:42 Message: Logged In: YES user_id=6380 Yes. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-20 22:30 Message: Logged In: YES user_id=469548 With 'be careful, it may not exist.', do you mean rmtree can't rely on exc.filename being set? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 17:18 Message: Logged In: YES user_id=6380 This looks fine, and can be checked in. There's one minor remaining issue, but I don't see how to fix it without changing the signature for the onerror function that os.walk() takes, and that's a no-no: when the problem is a directory permission preventing os.listdir() to work, the path passed to the rmtree caller's onerror function is always the toplevel path, rather than the directory that couldn't be listed. The real path is included in the error message. (Hmm, perhaps you could get it out of there? I think it's an attribute of the exception object; but be careful, it may not exist.) It's unfortunate that the error argument to the onerror function is a sys.exc_info() triple rather than an exception object (calling sys.exc_info() is expensive), but that's an API choice from the past that we can't change until Python 3.0. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 10:57 Message: Logged In: YES user_id=469548 A new patch is attached. I factored out the handle_call_error method so exception handling is more local. Also, errors in os.walk were already being handled using the _raise_err/except OSError combination, but we now do it using the onerror argument to os.walk. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 01:42 Message: Logged In: YES user_id=6380 (Actually, os.walk has an onerror argument, it just uses a different convention and default, so we could thread this through easily.) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 01:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 00:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 08:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From noreply at sourceforge.net Sun Oct 31 13:12:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 31 13:12:46 2004 Subject: [ python-Bugs-1056498 ] minor error in os.access Message-ID: Bugs item #1056498, was opened at 2004-10-29 03:46 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056498&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Out of Date Priority: 5 Submitted By: edward (edwardcannon) Assigned to: Nobody/Anonymous (nobody) Summary: minor error in os.access Initial Comment: current description of os.acess is a follows: access(path, mode) Use the real uid/gid to test for access to path. . . . Return 1 if access is allowed, 0 if not. See the Unix man page access(2) for more information. Availability: Unix, Windows. in reality, os.access returns True or False (at least on 2.3.3) this should probably be updated ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-31 13:12 Message: Logged In: YES user_id=469548 Already fixed in Python 2.4 by patch #859286. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1056498&group_id=5470 From noreply at sourceforge.net Sun Oct 31 13:20:13 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 31 13:20:17 2004 Subject: [ python-Bugs-1057535 ] add link in time module to calendar.timegm() Message-ID: Bugs item #1057535, was opened at 2004-10-31 02:19 Message generated for change (Comment added) made by jlgijsbers You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057535&group_id=5470 Category: Documentation Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Jeff Shute (jshute) Assigned to: Nobody/Anonymous (nobody) Summary: add link in time module to calendar.timegm() Initial Comment: There should be a link in the documentation for time.gmtime() to calendar.timegm(), which is its inverse. The timegm() function is really in the wrong module, which makes it very difficult to find. ---------------------------------------------------------------------- >Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-31 13:20 Message: Logged In: YES user_id=469548 Fixed in rev 1.70 of libtime.tex. Thanks for the report! ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057535&group_id=5470 From noreply at sourceforge.net Sun Oct 31 21:20:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 31 21:20:35 2004 Subject: [ python-Bugs-1057835 ] compiler.transformer, "from module import *" Message-ID: Bugs item #1057835, was opened at 2004-10-31 21:20 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057835&group_id=5470 Category: Parser/Compiler Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Felix Wiemann (felixwiemann) Assigned to: Nobody/Anonymous (nobody) Summary: compiler.transformer, "from module import *" Initial Comment: import compiler.transformer compiler.transformer.parse('from module import *').getChildNodes()[0].getChildNodes()[0].lineno ... is 1 in Python 2.3 but None in Python 2.4b1. lineno is 1 in Python 2.4 too when passing 'from module import something' instead of 'from module import *'. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1057835&group_id=5470 From noreply at sourceforge.net Sun Oct 31 22:57:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun Oct 31 22:57:08 2004 Subject: [ python-Bugs-1048941 ] os.rmtree error handling was always broken Message-ID: Bugs item #1048941, was opened at 2004-10-17 21:24 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 Category: None Group: Python 2.3 Status: Closed Resolution: Fixed Priority: 8 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Johannes Gijsbers (jlgijsbers) Summary: os.rmtree error handling was always broken Initial Comment: I just realized that the error handling in shutil.rmtree has been broken ever since Python 2.3, and the new implementation in 2.4 inherits this bug. The problem is that the try/except is outside the for loop, so that when an error is found, if either ignore_errors or onerror is set, the desired action is taken *once* but then the function returns rather than trying to remove more files (ignoring or reporting subsequent errors) as it should. The sad thing is that the original code by David Ascher (shutil.py rev 1.8) was correct! The bug was introduced in rev 1.27, which claimed to make the implementation agree with the docs. :-( I'm giving this a high priority, hoping that it will be fixed in 2.4b2 (and hopefully backported to 2.3). ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-31 16:57 Message: Logged In: YES user_id=6380 Thanks a lot! I think this qualifies for a backport to 2.3 as well. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-31 07:07 Message: Logged In: YES user_id=469548 Ok, checked in rmtree-ng.py version as rev. 1.34 of shutil.py and rev 1.8 of test_shutil.py. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-27 11:56 Message: Logged In: YES user_id=6380 - I don't like the nested function -- Python's function call overhead is significant and I worry about that when deleting massive trees. (I want to be able to "rm -rf /*" efficiently. :-) - The err instance instead of sys.exc_info() was an oversight. The docstring repeats this mistake. - I don't think the lstat() call is unclean, and again I prefer performance. - Here's an additional trick to reduce the code duplication (see upload rmtree-ng.py): At the top of the function put: if ignore_errors: ....def onerror(*args): pass elif onerror is None: ....def onerror(*args): raise Then all the except blocks can be collapsed to just the onerror() call. (I care less about the performance in case there's an error.) ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-27 05:57 Message: Logged In: YES user_id=469548 Yes, I think this is the right way to go about it. I'd personally make a couple of changes, but those are up to your discretion: * Factor out the try: except:'s to a nested function. * Use sys.exc_info() instead of the exception for backwards compatibility. * Use 'os.path.isdir(fullname) and not os.path.islink(fullname)' expression instead of the lstat call. I think it more clearly communicates what we want, even though it's more stat calls. Latest version attached (rmtree-refactored.py). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-26 12:08 Message: Logged In: YES user_id=6380 How about this no-nonsense version? (rmtree.txt upload) The only thing you could quibble about is what to do when os.lstat fails. I chose backward compatibility. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-23 14:25 Message: Logged In: YES user_id=6380 BTW, if you want the correct path in the onerror calls for os.listdir, your best bet is not to use os.walk but to write something similar that suits our purpose. After you remove the features we don't use, it's really only a few lines of code. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-23 13:56 Message: Logged In: YES user_id=6380 I disagree. The docs are wrong because a failing listdir wasn't anticipated at first when this function was written. A failing listdir is a common failure case; I don't see how the os.path calls in os.walk could ever fail (since they all catch os.error). ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-22 23:01 Message: Logged In: YES user_id=469548 You know what: I think we should remove the os.listdir error handling: * The docs only specify "errors resulting from *failed removals*" as being handled. If they didn't we should worry about OSErrors from the os.path.* calls in os.walk() too. * The os.listdir error handling hasn't been in a final release, so no backwards compatibility problems yet. * We can't get the path right for os.listdir. * The problem also existed before the rewrite using os.walk(), so reverting to that is not a solution. Patch (the last?) is attached. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-21 00:42 Message: Logged In: YES user_id=6380 Yes. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-20 16:30 Message: Logged In: YES user_id=469548 With 'be careful, it may not exist.', do you mean rmtree can't rely on exc.filename being set? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-19 11:18 Message: Logged In: YES user_id=6380 This looks fine, and can be checked in. There's one minor remaining issue, but I don't see how to fix it without changing the signature for the onerror function that os.walk() takes, and that's a no-no: when the problem is a directory permission preventing os.listdir() to work, the path passed to the rmtree caller's onerror function is always the toplevel path, rather than the directory that couldn't be listed. The real path is included in the error message. (Hmm, perhaps you could get it out of there? I think it's an attribute of the exception object; but be careful, it may not exist.) It's unfortunate that the error argument to the onerror function is a sys.exc_info() triple rather than an exception object (calling sys.exc_info() is expensive), but that's an API choice from the past that we can't change until Python 3.0. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-19 04:57 Message: Logged In: YES user_id=469548 A new patch is attached. I factored out the handle_call_error method so exception handling is more local. Also, errors in os.walk were already being handled using the _raise_err/except OSError combination, but we now do it using the onerror argument to os.walk. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:42 Message: Logged In: YES user_id=6380 (Actually, os.walk has an onerror argument, it just uses a different convention and default, so we could thread this through easily.) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-10-18 19:32 Message: Logged In: YES user_id=6380 (I'm sure Barry didn't mind, I just assigned it to him because rev. 1.27 had his name on it. :-) I saw the patch, don't have time to apply it, note that it has three(!) nested try-except blocks; instead, it should really have some smaller ones around just exactly the call that you expect to raise an exception. Exceptions elsewhere should *not* be caught. Also, the raise statement in handle_error() should just be "raise" without parameters, which re-raises the original exception. The exception already contains the filename, usually. Finally: os.walk() seems to suppress errors in os.listdir(). Technically, errors in os.listdir() should be treated the same as errors in os.remove() / os.rmdir(), but that means we can't use os.walk() -- or we'd have to add error handling to it recursively. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 18:18 Message: Logged In: YES user_id=469548 Patch attached. A review would be appreciated, especially as to whether there's a nicer way to handle the problem, but I'll check it in before 2.4b2 anyway. ---------------------------------------------------------------------- Comment By: Johannes Gijsbers (jlgijsbers) Date: 2004-10-18 02:06 Message: Logged In: YES user_id=469548 Barry, hope you don't mind I'll take this one. I can have a fix tonight or tomorrow. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=1048941&group_id=5470 From mhhohn at sbcglobal.net Sat Oct 30 08:09:57 2004 From: mhhohn at sbcglobal.net (Michael Hohn) Date: Mon Jan 3 06:15:53 2005 Subject: AssertionError in pickle's memoize function Message-ID: <4183302F.40203@sbcglobal.net> Hi, here is a code sample that shows the problem I ran into: test.py: ================================= import pickle class aList(list): def __init__(self, arg): # w/o this call, pickle works... list.__init__(self, arg) pass A = aList([1,2]) B = aList([A, 3]) the_data = {'a': A, 'b': B} A._stored_by = the_data pickle.dumps([the_data, B]) # ok pickle.dumps([B, the_data]) # fails ================================= Outputs under: Python 2.3 (#1, Sep 13 2003, 00:49:11) [GCC 3.3 20030304 (Apple Computer, Inc. build 1495)] on darwin 9 scarlet::~:0> python test.py Traceback (most recent call last): File "test.py", line 16, in ? pickle.dumps([B, the_data]) # fails File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p ickle.py", line 1386, in dumps Pickler(file, protocol, bin).dump(obj) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p ickle.py", line 231, in dump self.save(obj) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p ickle.py", line 293, in save f(self, obj) # Call unbound method with explicit self File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p ickle.py", line 614, in save_list self._batch_appends(iter(obj)) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p ickle.py", line 629, in _batch_appends save(x) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p ickle.py", line 338, in save self.save_reduce(obj=obj, *rv) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p ickle.py", line 419, in save_reduce self.memoize(obj) File "/System/Library/Frameworks/Python.framework/Versions/2.3/lib/python2.3/p ickle.py", line 251, in memoize assert id(obj) not in self.memo AssertionError with the same problem under python on linux: Python 2.3 (#1, Jul 31 2003, 14:19:24) [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-113)] on linux2 Traceback (most recent call last): File "", line 1, in ? File "/usr/tmp/python-286703ll", line 1, in ? pickle.dumps([B, the_data]) # fails File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 1386, in dumps Pickler(file, protocol, bin).dump(obj) File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 231, in dump self.save(obj) File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 293, in save f(self, obj) # Call unbound method with explicit self File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 614, in save_list self._batch_appends(iter(obj)) File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 629, in _batch_appends save(x) File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 338, in save self.save_reduce(obj=obj, *rv) File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 419, in save_reduce self.memoize(obj) File "/usr/local_cci/Python-2.3/lib/python2.3/pickle.py", line 251, in memoize assert id(obj) not in self.memo AssertionError From noreply at sourceforge.net Tue Oct 26 09:27:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon Jan 3 06:16:24 2005 Subject: [ python-Bugs-728330 ] Don't define _SGAPI on IRIX Message-ID: Bugs item #728330, was opened at 2003-04-27 02:21 Message generated for change (Comment added) made by rwgk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 Category: Installation Group: Python 2.3 Status: Open Resolution: None Priority: 9 Submitted By: Ralf W. Grosse-Kunstleve (rwgk) Assigned to: Trent Mick (tmick) Summary: Don't define _SGAPI on IRIX Initial Comment: Python release: 2.3b1 Platform: IRIX 6.5 MIPSpro Compilers: Version 7.3.1.2m Commands used: ./configure --prefix=/usr/local_cci/Python-2.3b1t make The socket module fails to compile. The output from the compiler is attached. ---------------------------------------------------------------------- >Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-10-26 00:27 Message: Logged In: YES user_id=71407 How is my 2004-09-09 patch for the main line doing in 2.4b1? Does my 2004-09-14 submission for the release23-maint branch need any tweaking? AFAICT from the viewcvs page at SourceForge it is still not checked in. Thanks! ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-09-14 21:20 Message: Logged In: YES user_id=71407 Hi Trent, > Sorry for the delay. Your final socketmodule changes are > finally checked in. Thanks!!! > Yes, that is the correct tag for the latest Py2.3 code. > So you are going to look into the relevant bits to backport > for Py2.3 compilation on IRIX then? Here is the patch (based on socketmodule.c CVS rev. 1.271.6.8): Index: socketmodule.c ============================================== ===================== RCS file: /cvsroot/python/python/dist/src/Modules/socketmodule.c, v retrieving revision 1.271.6.8 diff -r1.271.6.8 socketmodule.c 194a195 > #undef _SGIAPI /* to avoid warning */ 196a198,206 > #undef _XOPEN_SOURCE > #include > #include > #include > #ifdef _SS_ALIGNSIZE > #define HAVE_GETADDRINFO 1 > #define HAVE_GETNAMEINFO 1 > #endif > 262c272,284 < #include "addrinfo.h" --- > /* include Python's addrinfo.h unless it causes trouble */ > #if defined(__sgi) && _COMPILER_VERSION>700 && defined (_SS_ALIGNSIZE) > /* Do not include addinfo.h on some newer IRIX versions. > * __SS_ALIGNSIZE is defined in sys/socket.h by 6.5.21, > * for example, but not by 6.5.10. > */ > #elif defined(_MSC_VER) && _MSC_VER>1200 > /* Do not include addrinfo.h for MSVC7 or greater. > * addrinfo and EAI_* constants are defined in ws2tcpip.h > */ > #else > # include "addrinfo.h" > #endif Tested on these platforms: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.25 with MIPRpro 7.3.1.3 RedHat 8.0, gcc 3.2 Tru64 UNIX V5.1, C V6.3-028 For your convenience and reference: wget http://cci.lbl.gov/~rwgk/bugs/python/socketmodule_r1_271_6 _8_patched_c Cheers, Ralf ---------------------------------------------------------------------- Comment By: Trent Mick (tmick) Date: 2004-09-13 10:50 Message: Logged In: YES user_id=34892 Ralf, Sorry for the delay. Your final socketmodule changes are finally checked in. Checking in socketmodule.c; /cvsroot/python/python/dist/src/Modules/socketmodule.c,v <-- socketmodule.c new revision: 1.306; previous revision: 1.305 >> Can you take over any relevant 2.3 work? > > Which CVS tag should I be using? release23-maint? Yes, that is the correct tag for the latest Py2.3 code. So you are going to look into the relevant bits to backport for Py2.3 compilation on IRIX then? Cheers, Trent ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-09-09 03:52 Message: Logged In: YES user_id=71407 Sorry for the delay. It took 6+ hours for revision 1.305 to show up in the anonymous cvs. > Do those other patch bits need to go in for IRIX compilation? Absolutely. I've made a new patch again, based on socketmodule.c revision 1.305: Index: socketmodule.c ============================================== ===================== RCS file: /cvsroot/python/python/dist/src/Modules/socketmodule.c, v retrieving revision 1.305 diff -u -r1.305 socketmodule.c --- socketmodule.c 7 Sep 2004 17:48:26 -0000 1.305 +++ socketmodule.c 9 Sep 2004 10:25:29 -0000 @@ -192,8 +192,18 @@ /* XXX Using _SGIAPI is the wrong thing, but I don't know what the right thing is. */ +#undef _SGIAPI /* to avoid warning */ #define _SGIAPI 1 +#undef _XOPEN_SOURCE +#include +#include +#include +#ifdef _SS_ALIGNSIZE +#define HAVE_GETADDRINFO 1 +#define HAVE_GETNAMEINFO 1 +#endif + #define HAVE_INET_PTON #include #endif Tested again on these platforms: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.25 with MIPRpro 7.3.1.3 RedHat 8.0, gcc 3.2 For your convenience and reference: wget http://cci.lbl.gov/~rwgk/bugs/python/socketmodule_r1_305_p atched_c It may be most convenient to wget the complete file as shown, mv, cvs diff. > Can you take over any relevant 2.3 work? Which CVS tag should I be using? release23-maint? ---------------------------------------------------------------------- Comment By: Trent Mick (tmick) Date: 2004-09-07 10:51 Message: Logged In: YES user_id=34892 Now checked in to Python CVS (socketmodule.c revision 1.305) Ralf, Note that I just submitted the #ifdef'ery stuff that I posted and (I now see) not all the stuff that you have in http://cci.lbl.gov/~rwgk/bugs/python/socketmodule_r1_303_patch Do those other patch bits need to go in for IRIX compilation? Can you take over any relevant 2.3 work? ---------------------------------------------------------------------- Comment By: Trent Mick (tmick) Date: 2004-09-07 09:07 Message: Logged In: YES user_id=34892 > The new patch didn't make it into Python 2.4, alpha 3. > Is it still not good? No, I just dropped the ball. I'll look into it right now. ---------------------------------------------------------------------- Comment By: Steven Queen (squeen) Date: 2004-09-07 03:49 Message: Logged In: YES user_id=883896 > #define _BSD_TYPES > You could try adding this to the top of bsddbmodule.c in > Python 2.3.4. I did as you suggested and added the define (before #include "Python.h") at the top of bsdmodule.c and _ssl.c. Both not compile for 2.3.4, with MIPSpro 7.4.2 without error. Thank you again! The last remaining warning reads: building 'bsddb185' extension cc -DNDEBUG -O -OPT:Olimit=0 -I. [...]-o build/temp.irix64-6.5-2.3/bsddbmodule.o ld -shared -all build/temp.irix64-6.5-2.3/bsddbmodule.o [...] -o build/lib.irix64-6.5-2.3/bsddb185.so *** WARNING: renaming "bsddb185" since importing it failed: 14768:./python: rld: Fatal Error: unresolvable symbol in build/lib.irix64-6.5-2.3/bsddb185.so: dbopen ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-09-06 12:03 Message: Logged In: YES user_id=71407 > But it is not compileable in Python 2.3.4. That's expected because I took out this part of my original 2.3.4c1 patch: #if defined(__sgi) && _COMPILER_VERSION>700 && !_SGIAPI #define _BSD_TYPES #endif 2.4 doesn't require this anymore because _BSD_TYPES is already defined in pyconfig.h. For 2.3.4 you have to put it back near the top of socketmodule.c. I'll post a patch against the 2.3 maintenance branch after the 2.4 patch is in the CVS. Holding my breath... ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-09-05 10:39 Message: Logged In: NO The patch worked for python 2.4 a2. But I patched the configured source tree (no gmake distclean, ./configure). But it is not compileable in Python 2.3.4. Syntax error about "struct python.api" and some other. I will post this tomorrow (06.09.04). Environment: SGI Irix IP35, MipsPro cc 7.3.1.3, Irix 6.5.24. --maik./ ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-09-03 22:39 Message: Logged In: YES user_id=71407 The new patch didn't make it into Python 2.4, alpha 3. :( :( :( Is it still not good? ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-08-26 10:03 Message: Logged In: YES user_id=71407 > I still see compile errors in bsddbmodule.c and _ssl.c > during to missing types for u_short, u_int, u_long, etc. Yesterday I noticed that bsddbmodule.c compiles correctly with the latest Python CVS. Maybe this is because the latest pyconfig.h defines this: #define _BSD_TYPES You could try adding this to the top of bsddbmodule.c in Python 2.3.4. ---------------------------------------------------------------------- Comment By: Steven Queen (squeen) Date: 2004-08-26 09:50 Message: Logged In: YES user_id=883896 Oops, previous message should have read: ... applied your patch and socketmodule is NOW working in Python 2.3.4 for me. Sorry. ---------------------------------------------------------------------- Comment By: Steven Queen (squeen) Date: 2004-08-26 09:49 Message: Logged In: YES user_id=883896 I just ran into this same problem using the MIPSpro 7.4.2 compiler under IRIX 6.5.23. I applied your patch and socketmodule is not working in Python 2.3.4 for me. THANK YOU. I still see compile errors in bsddbmodule.c and _ssl.c during to missing types for u_short, u_int, u_long, etc. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-08-25 13:34 Message: Logged In: YES user_id=71407 I forgot to mention that the patch and the patched file are also available here, along with the original code: http://cci.lbl.gov/~rwgk/bugs/python/socketmodule_r1_303_c http://cci.lbl.gov/~rwgk/bugs/python/socketmodule_r1_303_p atch http://cci.lbl.gov/~rwgk/bugs/python/socketmodule_r1_303_p atched_c ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-08-25 13:30 Message: Logged In: YES user_id=71407 I will attach a complete patch for SGI, based on Trent's work. I've tested on these platforms: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.25 with MIPRpro 7.3.1.3 RedHat 8.0, gcc 3.2 Comments: - Trent's fragment has a small error: __SS_ALIGNSIZE, should be _SS_ALIGNSIZE. Other than that it was fine. - I guess the Python 2.3 maintenance branch should also be patched to correct my flaw. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-08-25 12:45 Message: Logged In: YES user_id=21627 tmick, from inspection only, your last patch seems clear and correct - especially as it has comments in it :-) I would leave the OS/2 part out - I somewhat doubt that any compiler on that platform provides EAI_*, and even if it would: addrinfo.h typically ends up empty if the system has "proper" getaddrinfo support, so including it should not hurt even on systems where it isn't used. ---------------------------------------------------------------------- Comment By: Trent Mick (tmick) Date: 2004-08-25 10:30 Message: Logged In: YES user_id=34892 Would ifdef'ery like the following be a little bit more clear and still work? /* include Python's addrinfo.h unless it causes trouble */ #if defined(__sgi) && _COMPILER_VERSION>700 && defined(__SS_ALIGNSIZE) /* Do not include addinfo.h on some newer IRIX versions. * __SS_ALIGNSIZE is defined in sys/socket.h by 6.5.21, * for example, but not by 6.5.10. */ #elif defined(_MSC_VER) && _MSC_VER>1200 /* Do not include addrinfo.h for MSVC7 or greater. * addrinfo and EAI_* constants are defined in ws2tcpip.h */ #else # include "addrinfo.h" #endif ...an additional #elif block might be necessary for OS/2 EMX (but let's figure that out later). Ralf, Would you be able to try a Python build with the latest Python CVS and replace the bare #include "addrinfo.h" on line 260 of socketmodule.c with the above block. Does that work for you? If so then I think this should be a basis for a corrected patch for socketmodule.c. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-08-25 09:59 Message: Logged In: YES user_id=71407 Mick is right. Unfortunately my patch from May had a flaw. It should have been something along the lines of #if !defined(__sgi) || (_COMPILER_VERSION>700 && !defined (_SS_ALIGNSIZE)) # include "addrinfo.h" #endif Sorry for the oversight. ---------------------------------------------------------------------- Comment By: Trent Mick (tmick) Date: 2004-08-25 09:26 Message: Logged In: YES user_id=34892 I am using this block in my builds. I *think* that this preserves the intentions of the IRIX and MSVC6 patches and am less sure about the OS/2 EMX-related part (see the python-dev thread for a reference). #if defined(__sgi) # if _COMPILER_VERSION>700 && !defined(_SS_ALIGNSIZE) /* defined in sys/socket.h */ /* by some newer versions of IRIX */ /* (e.g. not by 6.5.10 but by 6.5.21) */ # include "addrinfo.h" # endif #elif defined(PYOS_OS2) # if defined(PYCC_GCC) # include "addrinfo.h" # endif #elif defined(_MSC_VER) # if _MSC_VER == 1200 # include "addrinfo.h" # endif #else # include "addrinfo.h" #endif ---------------------------------------------------------------------- Comment By: Trent Mick (tmick) Date: 2004-08-25 09:09 Message: Logged In: YES user_id=34892 Related thread on python-dev: http://mail.python.org/pipermail/python-dev/2004-August/048341.html ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-08-24 23:26 Message: Logged In: YES user_id=21627 Indeed, this broke a number of platforms, so I backed it out. ---------------------------------------------------------------------- Comment By: John Ehresman (jpe) Date: 2004-07-10 11:41 Message: Logged In: YES user_id=22785 This seems to break the building of CVS HEAD sources on windows with VC++ 6 because addrinfo.h is not included and HAVE_GETADDRINFO is not defined in my build. It also may break on other platforms where HAVE_GETADDRINFO is not defined. Maybe addrinfo.h should be included if not on sgi. I know that VC++ 6 isn't the primary compiler for Python 2.4, but it works otherwise and it presumably will be the primary compiler for 2.3.5. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-27 03:28 Message: Logged In: YES user_id=6656 Closing! ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-27 00:17 Message: Logged In: YES user_id=29957 There's a note on the 2.3.4 bugs page pointing at this bug. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-26 22:41 Message: Logged In: YES user_id=29957 If the patch is good, I'll make sure it's included in the 2.3.4 'bugs' page. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 21:14 Message: Logged In: YES user_id=71407 > Can you try gmake? Yes, it works with gmake. I've verified that the socket module on the CVS head builds and runs correctly on the three platforms mentioned in my message from 2004-05- 22 01:23. There are not even warnings. > Is IRIX make particularly braindead? I don't think so. It works with most auto-configured packages incl. all earlier versions of Python. But I'll save this for another day when the first 2.4 beta release is out. Thank you for your help! I think this tracker issue should be closed. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 11:19 Message: Logged In: YES user_id=6656 Uh, that's odd. Is IRIX make particularly braindead? Can you try gmake? Maybe we've let something non-portable slip in. At any rate, I think this should be done in a separate report... ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 11:14 Message: Logged In: YES user_id=71407 I checked out the python cvs head from scratch and ran configure in the usual way, with the usual result. However, make doesn't work: % make don't know how to make Parser/printgrammar.o (bu42). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 10:07 Message: Logged In: YES user_id=6656 OK, so I've commited revision 1.289 of Modules/socketmodule.c. Can you check if CVS HEAD works for you now? ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 09:59 Message: Logged In: YES user_id=71407 > Is there any chance of the Right Thing actually happening? The experience of the last 13 months suggests "No." SGI's freeware team has not responded to my report for more than a year. I am absolutely not a SGI or socket expert. I only worked on the patch because I was desperate enough and because Maik had already done the groundwork. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 09:27 Message: Logged In: YES user_id=6656 Is this really The Right Thing? I still see /* XXX Using _SGIAPI is the wrong thing, but I don't know what the right thing is. */ Is there any chance of the Right Thing actually happening? If not, I'll check this into HEAD, and if that seems to work, move it to release23-maint in time for 2.3.5. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-22 01:23 Message: Logged In: YES user_id=71407 Based on Maik's posting from 2004-03-08 08:53 I've worked out a small bug-fix patch relative to Python-2.3.4c1 that leads to successful creation of _socket.so under three different versions of IRIX: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.23 with MIPRpro 7.3.1.3 I've consistently used #if defined(__sgi) && _COMPILER_VERSION>700 to restrict the changes to IRIX. However, to be sure I've also verified that the Python installation is still OK under Redhat 8 and Tru64 Unix 5.1. I will upload the patch. It is also available here: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patch For completeness I've also posted these files: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patched_c http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_original_c I've made an effort to keep the patch minimally intrusive, but it is certainly not "clean." However, the status quo is so bad that it can only get better. We have not been able to switch to Python 2.3 because of the problems with the socket module. It would be fantastic if the bug-fix patch could be included in the 2.3.4 final distribution. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2004-03-08 08:53 Message: Logged In: YES user_id=768223 I got Python 2.3.3 compiled with MipsPro Compilers: Version 7.3.1.3m. The following changes were made to satisfy all needs. Line 206 in Modules/socketmodule.c #define _SGIAPI 1 NEW: #undef _XOPEN_SOURCE NEW: #define HAVE_GETADDRINFO 1 NEW: #define HAVE_GETNAMEINFO 1 #define HAVE_INET_PTON NEW: #include NEW: #include NEW: #include #include Line 269: /* #include "addrinfo.h" */ In addition to this I extend the CFLAGS with -D_BSD_TYPES to satisfy the u_short and friends definition. --maik./ ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-03-03 08:20 Message: Logged In: YES user_id=892601 Got it down to one error: % > ./python Lib/test/test_socket.py Traceback (most recent call last): File "Lib/test/test_socket.py", line 6, in ? import socket File "/usr/local/spider/lib/python2.3/socket.py", line 44, in ? import _socket ImportError: 210454:./python: rld: Fatal Error: unresolvable symbol in /usr/local/spider/lib/python2.3/lib-dynload/_socketmodule.so: _xpg5_accept xpg5_accept does appear in any library on our SGI's that are only 1-2 years old. It does appear in libc in very recent revisions of Irix (> 6.5.19?) Does anyone know how to get Python 2.3 to avoid the use of xpg5_accept? Thanks ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-01-12 10:59 Message: Logged In: YES user_id=892601 Has anyone encountered problems with different versions of IRIX 6.5? I finally got sockets to work in Python2.3.3, but only on the most up-to-date version of IRIX (uname -v = 01100304). On any other machine with a lower version (archetecture doesn't seem to matter), I get messages which all seem to be related to sockets: 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_vsnprintf 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recv 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_connect 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_bind 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_sendto 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_setsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_accept 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recvfrom 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_socket 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockname Upgrading all of our SGIs is not something I have control over. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-22 10:52 Message: Logged In: YES user_id=768223 Sorry if there is something unclear. Redefine means: Set a value which is required to satisfy ifdef-statements in the header files. As you see in the definitions, given in the comment, the value for _NO_XOPEN4 is set as a result of some dependencies automatically defined. A redefinition of _SGIAPI in the socketmodule.c is not sufficient alone. Therefore to satisfy all requirements in the header files, you should set the values for _NO_XOPEN4 and _NO_XOPEN5 also manually. Howto do in socketmodule.c? ... #define _SGIAPI 1 #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 ... If you are familiar with regular expressions, then you will know that _NO_XOPEN[45] is a shorthand for _NO_XOPEN4, _NO_XOPEN5. Sorry if there was something wrong in my comment. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-12-22 07:49 Message: Logged In: YES user_id=892601 Can someone please clarify mherta's comment from 2003-12-16? How should we "redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition"? Should those lines of code be added to socketmodule.c, or are they the source of the problem? Is _NO_XOPEN[45] shorthand for _NO_XOPEN[4] and _NO_XOPEN[5] or are they all different? Thanks. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-16 09:27 Message: Logged In: YES user_id=768223 I've tested the latest Python 2.3.3c1 to compile. The problem reported is still present. To avoid the error messages, about the incompatible types of several netdb-functions (like inet_pton), you should redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition. If there are error messages that types like u_short and u_char are not defined, please add -D_BSD_TYPES in the Makefile (BASECFLAGS is okay). If anybody don't need a executable compiled with MipsPro, gcc from the freeware.sgi.com could be a choice. The problem for this seems in /usr/include/standards.h: #define _NO_XOPEN4 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 >= 500))) #define _NO_XOPEN5 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 < 500))) As the first option is true (_XOPEN_SOURCE is set to 600 in pyconfig.h) the second option will be false. The constant #define NI_NUMERICHOST 0x00000002 /* return numeric form of hostname */ is enveloped in a clause #ifdef _NO_XOPEN4 && _NO_XOPEN5 (/usr/include/netdb.h) which is always false. In the manual pages there is a example which uses the constant NI_NUMERICHOST and compiles without error message. I'm not sure to cut off the file pyconfig.h from compiling the socketmodule.c? Any comments? ---------------------------------------------------------------------- Comment By: Hans Nowak (zephyrfalcon) Date: 2003-10-30 19:04 Message: Logged In: YES user_id=173607 My $0.02... Yesterday I successfully compiled Python 2.3.2, with socket module. I'm using an O2 running IRIX 6.5.11f. I didn't encounter any of the problems described before, and had no problems with _SGIAPI or anything. The reason it didn't compile was that u_long and u_char were not defined in some places; /usr/include/netinet/tcp.h to be precise, and a Python file (getaddrinfo.c). I'm using gcc 3.3 (the one downloaded from freeware.sgi.com). If anyone is interested, drop me a mail at hans at zephyrfalcon.org and I'll put the binary online somewhere. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-24 12:59 Message: Logged In: YES user_id=892601 Here's the output from 'gmake', using MIPSpro Compiler 7.30 on IRIX 6.5 Sorry, I don't know how to attach something using the "Add a Comment" box. Is there another way to post? The patch at www.python.org/2.3.2/patches/sgi-socketmodule.patch was edited into socketmodule.c : #include <netdb.h> #endif /* GCC 3.2 on Irix 6.5 fails to define this variable at all. Based on the above code, I'd say that the SGI headers are just horked. */ #if defined(__sgi) && !defined(INET_ADDRSTRLEN) #define INET_ADDRSTRLEN 16 #endif /* Generic includes */ #include <sys/types.h> #include <signal.h> --------------------------------------------------------------------------- cc -OPT:Olimit=0 -DNDEBUG -n32 -O -I. -I./Include -DPy_BUILD_CORE -c ./Modules/socketmodule.c -o Modules/socketmodule.o cc-1047 cc: WARNING File = ./Modules/socketmodule.c, Line = 195 Macro "_SGIAPI" (declared at line 210 of "/usr/include/standards.h") has an incompatible redefinition. #define _SGIAPI 1 ^ cc-1020 cc: ERROR File = /usr/include/netdb.h, Line = 198 The identifier "socklen_t" is undefined. extern int iruserok_sa(const void *, socklen_t, int, const char *, ^ cc-1047 cc: WARNING File = ./Modules/addrinfo.h, Line = 145 Macro "_SS_ALIGNSIZE" (declared at line 246 of "/usr/include/sys/socket.h") has an incompatible redefinition. #define _SS_ALIGNSIZE (sizeof(PY_LONG_LONG)) ^ cc-1020 cc: ERROR File = ./Modules/socketmodule.c, Line = 822 The identifier "NI_NUMERICHOST" is undefined. NI_NUMERICHOST); ^ cc-1515 cc: ERROR File = ./Modules/socketmodule.c, Line = 3054 A value of type "int" cannot be assigned to an entity of type "const char *". retval = inet_ntop(af, packed, ip, sizeof(ip)); ^ 3 errors detected in the compilation of "./Modules/socketmodule.c". gmake: *** [Modules/socketmodule.o] Error 2 ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2003-10-23 03:25 Message: Logged In: YES user_id=6656 Just to be clear, please "attach" the build rather than pasting it inline. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2003-10-22 19:19 Message: Logged In: YES user_id=29957 Did you try 2.3.2 with the patch from the 2.3.2 build bugs page? If it failed, please post the build log of the failing bit to this bug. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-22 11:26 Message: Logged In: YES user_id=892601 Has there been any progress since August on getting Python-2.3 to compile on IRIX 6.5? I really need to have sockets working to use python. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-08-14 04:37 Message: Logged In: YES user_id=768223 I have the same problem on IRIX 6.5.19m and MIPSpro 7.3.1.3m on a FUEL 600Mhz. The situation seems a little bit strange on this system. But I have luck and get it compiled. As the current implementation is not clean, my resolution also. Which should be changed to get it work. The manpage for gethostbyaddr_r says: #include <sys/types> #include <sys/socket> #include <netinet/in.h> #include <netdb.h> So I included this in the __sgi-clause in socketmodule.c. Looking in the header-files shows, that some variables are defined if #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 are set, before the includes. This defines some more variables, so that pyconfig.h should be changed. #define HAVE_ADDRINFO 1 #define HAVE_SOCKADDR_STORAGE 1 Sure there are warnings about the redefinition of the _(SGIAPI|NO_XOPEN4|NO_XOPEN5) but it compiles and the tests don't fail. Also my applications run without exception or coredump on FUEL, Octane, Octane 2. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-08-01 05:10 Message: Logged In: YES user_id=33168 Ralf, I will try to work on this, but it won't be for a week or two probably. Can you tell me which machines (by name) work and doesn't work. The only IRIX machine I seem to know about it rattler. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-08-01 00:34 Message: Logged In: YES user_id=71407 For the records: The Python2.3 (final) socket module still does not compile under IRIX 6.5.20. It only compiles under 6.5.10. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 11:07 Message: Logged In: YES user_id=31392 I applied a slight variant of Neal's patch and that seemed to fix things. It still seems wrong that we define _SGIAPI, but that will have to wait for 2.3.1. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 06:54 Message: Logged In: YES user_id=31392 I learned a little about inet_ntop() in the last few minutes. It looks like the size_t argument is an older spelling, and the current POSIX says it should be socklen_t. Regardless, it looks like we're using the new definition when IRIX is actually providing the older definition. Neal attached a patch that looks like it should clean up the problem. Have you tried it? I think a better solution would be to teach configure how to find inet_ntop() via netdb.h, but that's probably too hard to get done in the new couple of hours. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 06:42 Message: Logged In: YES user_id=31392 There seem to be two causes of worry. The actual compiler error in your build log and the use of _SGIAPI that Casavanti warns again. It looks like the compiler problems is because our inet_ntop() prototype is wrong. That is Python says the last argument is a socklen_t, but IRIX and Linux man pages both say size_t. Perhaps size_t and socklen_t are different sizes on IRIX, but not on Linux. What happens if you change it to size_t? The problematic code would appear to be these lines from socketmodule.c: #if defined(__sgi)&&_COMPILER_VERSION>700 && !_SGIAPI /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #define _SGIAPI 1 #include "netdb.h" #endif It's the only use the _SGIAPI symbol that Casavant says we shouldn't use. What happens if you remove the use _SGIAPI, i.e. just make it #if defined(__sgi) && _COMPILER_VERSION > 700 /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #include "netdb.h" #endif ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-16 17:27 Message: Logged In: YES user_id=71407 We have confirmed that the socket module in Python 2.3b1 and 2.3b2 fails to compile under these versions of IRIX: 6.5.10, MIPSpro 7.3.1.2m 6.5.19, MIPSpro 7.3.1.3m 6.5.20, MIPSpro 7.3.1.3m (latest available, to my knowledge) If Python 2.3 goes out without a working socket module it will be useless for many people in science where IRIX is still found quite frequently. Maybe even worse, we will also no longer be able to advertise Python as a language that "runs everywhere." ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-15 15:37 Message: Logged In: YES user_id=71407 Date: Tue, 15 Jul 2003 15:06:25 -0500 From: "Brent Casavant" <bcasavan@sgi.com> To: "Ralf W. Grosse-Kunstleve" <rwgk@yahoo.com> Subject: Re: IRIX & Python Ralf, I read through the sourceforge report. I don't have a sourceforge account so I didn't add there, but feel free to distribute my ramblings. First, they should be very careful about what version of the operating system they are using. Real IPv6 support was only added a few releases ago (6.5.20 I believe, though my memory is hazy). Some of the functions/macros corresponding to IPv6 were introduced prior to that point, but implementation may not have been complete. If they have to play tricks with #define values to get a symbol recongnized it most likely means that SGI didn't intend for that symbol to be used yet. I suspect the problems they are seeing are because they are using some version of IRIX 6.5.x that includes inet_ntoa() definitions under _SGIAPI, but for which real support was not yet complete. The namespace of symbols that begin with a single underscore and a capital letter is reserved for the OS vendors, and the fact that someone is trying to manually insert a definition of _SGIAPI into the source should be a big red warning flag. That said, there is a correct method to deal with the new capabilities of IRIX 6.5.x over time. This comes in two parts. A better explanation is in /usr/include/optional_sym.h First, when a new symbol is added to IRIX, the corresponding header file will always have the following line in it: #pragma optional some_new_symbol The presence of this #pragma will cause the linker to not complain if the symbol is missing when the executable is linked. If SGI neglects to do so it is a bug that should be reported to us. The second part is that calls to these "optional" symbols should be protected by a run-time check to see if the symbol is actually present. Just like this: #include <optional_sym.h> #include <header_file_for_some_new_symbol.h> . . . if (_MIPS_SYMBOL_PRESENT(some_new_symbol)) { /* New function is present, use it */ qux = some_new_symbol(foo, bar, baz); } else { /* New function isn't present, do something else */ qux = builtin_some_new_symbol(foo, bar, baz); } This ensures that a single executable will be able to function properly (or at least degrade gracefully) on all version of IRIX 6.5.x. In other words you can compile on 6.5.20 and run on 6.5.8, or vice versa. In particular, this means that compile-time detection of features such as inet_ntoa() is incorrect for IRIX 6.5.x -- these decisions should be made at run-time instead. If the decision is made at compile time the executable may well not execute on an older version of IRIX 6.5.x. Unfortunately GNU configure is not equipped to utilize such advanced binary-compatability mechanisms, so there's no "easy" way to get GNU configure'd software to run across all IRIX 6.5.x releases, short of compiling only on the original non-upgraded version of IRIX 6.5. Hope that helps, Brent Casavant On Tue, 15 Jul 2003, Ralf W. Grosse-Kunstleve wrote: > We are developing a Python-based (www.python.org) application for > IRIX. Until now the Python IRIX port was fully supported, but the last > two beta releases of Python (2.3b1 and 2.3b2) make me increasingly > worried because the crucial socket module (TCP/IP sockets) fails to > build. A while ago I've filed a bug report, but the Python developers > appear to be stuck: > > https://sourceforge.net/tracker/? func=detail&aid=728330&group_id=5470&atid=105470 > > Is there a way to bring this to the attention of developers at SGI to > hopefully get some help? > > Thank you in advance! > Ralf > -- Brent Casavant Silicon Graphics, Inc. Where am I? And what am I IRIX O.S. Engineer http://www.sgi.com/ doing in this handbasket? bcasavan@sgi.com 44.8562N 93.1355W 860F -- Unknown ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-06-30 07:21 Message: Logged In: YES user_id=71407 The socket module still fails to compile with Python 2.3b2, in addition to two other extensions. The full make log is attached. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-21 23:47 Message: Logged In: YES user_id=21627 I think it would be best if detect presence of inet_aton/inet_pton. Is IPv6 support possible on this system? Then it would be best if that was detected, as well. If it is not possible to detect aton/pton/ipv6, what is the problem with not using them? I'd like to have a look at the relevant headers (in particular to understand the duplicate definition of _SGIAPI), indeed. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-05-21 16:35 Message: Logged In: YES user_id=33168 Argh!!! This is a nightmare. The attached hack ^h^h^h patch fixes the problem, but ... _SGIAPI needs to be defined (~192 in socketmodule.c) in order to get the prototypes for inet_aton, inet_pton. But _SGIAPI can't be defined in the standard header files because XOPEN_SOURCE is defined. Martin, do you have any other suggestions? I don't see how to do this right within configure. If you want I can send you the header file(s). ---------------------------------------------------------------------- Comment By: alexandre gillet (gillet) Date: 2003-05-12 10:41 Message: Logged In: YES user_id=150999 I had the same problem compiling socket on Irix. We did find that on our system ENABLE_IPV6 is not define and all the error are cause by that. System that do not enable IPV6 are not well supported with the socketmodule.c code. We were able to compile socket after making few change in the code: I am not sure if I did it right but it compile. I will post my patch but I do not see any place to attach my file example: See line 2794,2800 + #ifdef ENABLE_IPV6 char packed[MAX(sizeof(struct in_addr), sizeof(struct in6_addr))]; + #else + char packed[sizeof(struct in_addr)]; + #endif I am not sure how to post the patch or file. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-05-05 11:44 Message: Logged In: YES user_id=71407 This is not in my area of expertise, but I've tried a couple of (probably stupid) things: #include <standards.h> in socketmodule.c instead of #define _SGIAPI. This resulted in even more errors. ./configure --disable-ipv6 Same errors as before. I am lost here. Sorry. But you still have ssh access to rattler.lbl.gov if that helps. Accounts for other Python developers are a possibility. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-04 05:39 Message: Logged In: YES user_id=21627 Can you analyse these compiler errors in more detail, please? What, in your opinion, is causing these problems, and how would you correct them? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470