From noreply at sourceforge.net Thu Feb 1 00:17:47 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 31 Jan 2007 15:17:47 -0800 Subject: [Patches] [ python-Patches-1638033 ] Add httponly to Cookie module Message-ID: Patches item #1638033, was opened at 2007-01-17 20:07 Message generated for change (Comment added) made by jjlee You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1638033&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Arvin Schnell (arvins) Assigned to: Nobody/Anonymous (nobody) Summary: Add httponly to Cookie module Initial Comment: Add the Microsoft extension httponly to the Cookie module. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2007-01-31 23:17 Message: Logged In: YES user_id=261020 Originator: NO I see. That sounds reasonable, but I won't comment on whether it should be applied since this part of module Cookie didn't really make sense to me in the first place (I explain why in my comment of 2006-12-03 16:49 in http://python.org/sf/1372650). ---------------------------------------------------------------------- Comment By: Arvin Schnell (arvins) Date: 2007-01-30 18:45 Message: Logged In: YES user_id=698939 Originator: YES Anybody who sets a cookie with key="httponly" is likely in trouble. I don't know and can't check how the IE behaves in that case. But disallowing this use shouldn't hurt. Use case: I would like to use the httponly attribute in Django. I think it's also useful for other web-frameworks. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2007-01-30 00:52 Message: Logged In: YES user_id=261020 Originator: NO This is backwards-incompatible, no? The behaviour of Morsel.set() changes (disallowing key="httponly") hence the behaviour of BaseCookie.__setitem__ changes. Do you have a use case? ---------------------------------------------------------------------- Comment By: Arvin Schnell (arvins) Date: 2007-01-19 17:01 Message: Logged In: YES user_id=698939 Originator: YES Sure, I have added some documentation to the patch. File Added: python.diff ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2007-01-19 15:06 Message: Logged In: YES user_id=764593 Originator: NO The documentation change should say what the attribute does. (It requests the the cookie be hidden from javascript, and available only to http requests.) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1638033&group_id=5470 From noreply at sourceforge.net Thu Feb 1 16:50:39 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 01 Feb 2007 07:50:39 -0800 Subject: [Patches] [ python-Patches-1649190 ] Adding support for _Bool to ctypes as c_bool Message-ID: Patches item #1649190, was opened at 2007-01-31 21:04 Message generated for change (Comment added) made by chmod007 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1649190&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: David Remahl (chmod007) Assigned to: Nobody/Anonymous (nobody) Summary: Adding support for _Bool to ctypes as c_bool Initial Comment: Adds support for the C99 _Bool type to ctypes. Requires struct patch that has already been accepted in the python 2.6 train. Updates ctypes C extension, ctypes __init__.py, ctypes documentation and ctypes tests. ---------------------------------------------------------------------- >Comment By: David Remahl (chmod007) Date: 2007-02-01 15:50 Message: Logged In: YES user_id=2135 Originator: YES Should I prepare the patch relative to /ctypes/trunk/ctypes instead? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-01-31 22:05 Message: Logged In: YES user_id=21627 Originator: NO Thomas, can you take a look? I think this needs to take into account Python versions without bool support in the struct module, as the same ctypes code is used with older Python releases as well. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1649190&group_id=5470 From noreply at sourceforge.net Thu Feb 1 21:22:08 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 01 Feb 2007 12:22:08 -0800 Subject: [Patches] [ python-Patches-1649190 ] Adding support for _Bool to ctypes as c_bool Message-ID: Patches item #1649190, was opened at 2007-01-31 22:04 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1649190&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: David Remahl (chmod007) >Assigned to: Thomas Heller (theller) Summary: Adding support for _Bool to ctypes as c_bool Initial Comment: Adds support for the C99 _Bool type to ctypes. Requires struct patch that has already been accepted in the python 2.6 train. Updates ctypes C extension, ctypes __init__.py, ctypes documentation and ctypes tests. ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2007-02-01 21:22 Message: Logged In: YES user_id=11105 Originator: NO David, I encountered these issues with the patch: 1. a typo in Lib/ctypes/test/test_numbers.py: 'bool_type' should be 'bool_types' It is trivial to fix this. 2. ctypes/__init__.py should be cean for 'from ctypes import *', but with your patch it exposes the 'struct_error' symbol. I would recommend to change Modules/_ctypes/cfield.c so that a c_bool type can be created even if HAVE_C99_BOOL is undefined (I think you took that approach in your _Bool struct patch). Then, ctypes/__init__.py would not have to depend on the struct 't' typecode at all. The '_check_size(c_bool, "t")' check can probably be removed. 3. ctypes/test/test_repr.py does not work. Easy to fix this: just add c_bool to the list of types starting at line 5. These changes would also ensure compatibility with previous Python versions, with one exception: A check for the _Bool type must be added to Modules/ctypes/libffi/configure.in. To answer your question: No, it is not needed to prepare a patch relative to ctypes/trunk/ctypes. This repository contains svn:external properties for the 'source' and 'ctypes' directories that point into the 'release25-maint/Modules/_ctypes' and 'release25-maint/Lib/ctypes' directories from Python. ---------------------------------------------------------------------- Comment By: David Remahl (chmod007) Date: 2007-02-01 16:50 Message: Logged In: YES user_id=2135 Originator: YES Should I prepare the patch relative to /ctypes/trunk/ctypes instead? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-01-31 23:05 Message: Logged In: YES user_id=21627 Originator: NO Thomas, can you take a look? I think this needs to take into account Python versions without bool support in the struct module, as the same ctypes code is used with older Python releases as well. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1649190&group_id=5470 From noreply at sourceforge.net Thu Feb 1 22:55:23 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 01 Feb 2007 13:55:23 -0800 Subject: [Patches] [ python-Patches-1650174 ] configHandler support for raw data Message-ID: Patches item #1650174, was opened at 2007-02-01 23:55 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1650174&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Tal Einat (taleinat) Assigned to: Nobody/Anonymous (nobody) Summary: configHandler support for raw data Initial Comment: ConfigParser has support for reading/writing "raw" data, but support for this was simply not implemented in configHandler.py. This patch makes IdleConf.GetOption(...) pass on the "raw" keyword argument to the config handlers, which is all that was missing. This patch was previously posted as part of the Squeezer patch, but this is a very simple change, and it is now required by a new IDLE extension (currently in development.) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1650174&group_id=5470 From noreply at sourceforge.net Thu Feb 1 23:10:38 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 01 Feb 2007 14:10:38 -0800 Subject: [Patches] [ python-Patches-1646432 ] ConfigParser getboolean() consistency Message-ID: Patches item #1646432, was opened at 2007-01-28 16:21 Message generated for change (Settings changed) made by taleinat You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1646432&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None >Priority: 1 Private: No Submitted By: Tal Einat (taleinat) Assigned to: Nobody/Anonymous (nobody) Summary: ConfigParser getboolean() consistency Initial Comment: Minor code change - made getboolean() implementation more consistent with other get...() methods. (i.e. uses _get) (functionality unchanged) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1646432&group_id=5470 From noreply at sourceforge.net Thu Feb 1 23:11:31 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 01 Feb 2007 14:11:31 -0800 Subject: [Patches] [ python-Patches-1650174 ] configHandler support for raw data Message-ID: Patches item #1650174, was opened at 2007-02-01 23:55 Message generated for change (Settings changed) made by taleinat You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1650174&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: None Status: Open Resolution: None >Priority: 3 Private: No Submitted By: Tal Einat (taleinat) Assigned to: Nobody/Anonymous (nobody) Summary: configHandler support for raw data Initial Comment: ConfigParser has support for reading/writing "raw" data, but support for this was simply not implemented in configHandler.py. This patch makes IdleConf.GetOption(...) pass on the "raw" keyword argument to the config handlers, which is all that was missing. This patch was previously posted as part of the Squeezer patch, but this is a very simple change, and it is now required by a new IDLE extension (currently in development.) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1650174&group_id=5470 From noreply at sourceforge.net Fri Feb 2 01:35:40 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 01 Feb 2007 16:35:40 -0800 Subject: [Patches] [ python-Patches-1649190 ] Adding support for _Bool to ctypes as c_bool Message-ID: Patches item #1649190, was opened at 2007-01-31 21:04 Message generated for change (Comment added) made by chmod007 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1649190&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: David Remahl (chmod007) Assigned to: Thomas Heller (theller) Summary: Adding support for _Bool to ctypes as c_bool Initial Comment: Adds support for the C99 _Bool type to ctypes. Requires struct patch that has already been accepted in the python 2.6 train. Updates ctypes C extension, ctypes __init__.py, ctypes documentation and ctypes tests. ---------------------------------------------------------------------- >Comment By: David Remahl (chmod007) Date: 2007-02-02 00:35 Message: Logged In: YES user_id=2135 Originator: YES Thanks for the review, Thomas! I've attached a new patch that addresses the points you bring up (I hope) in the suggested manner. I have not tested it on 2.5, but tests pass on 2.6. File Added: c_bool patch against r53623.diff ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2007-02-01 20:22 Message: Logged In: YES user_id=11105 Originator: NO David, I encountered these issues with the patch: 1. a typo in Lib/ctypes/test/test_numbers.py: 'bool_type' should be 'bool_types' It is trivial to fix this. 2. ctypes/__init__.py should be cean for 'from ctypes import *', but with your patch it exposes the 'struct_error' symbol. I would recommend to change Modules/_ctypes/cfield.c so that a c_bool type can be created even if HAVE_C99_BOOL is undefined (I think you took that approach in your _Bool struct patch). Then, ctypes/__init__.py would not have to depend on the struct 't' typecode at all. The '_check_size(c_bool, "t")' check can probably be removed. 3. ctypes/test/test_repr.py does not work. Easy to fix this: just add c_bool to the list of types starting at line 5. These changes would also ensure compatibility with previous Python versions, with one exception: A check for the _Bool type must be added to Modules/ctypes/libffi/configure.in. To answer your question: No, it is not needed to prepare a patch relative to ctypes/trunk/ctypes. This repository contains svn:external properties for the 'source' and 'ctypes' directories that point into the 'release25-maint/Modules/_ctypes' and 'release25-maint/Lib/ctypes' directories from Python. ---------------------------------------------------------------------- Comment By: David Remahl (chmod007) Date: 2007-02-01 15:50 Message: Logged In: YES user_id=2135 Originator: YES Should I prepare the patch relative to /ctypes/trunk/ctypes instead? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-01-31 22:05 Message: Logged In: YES user_id=21627 Originator: NO Thomas, can you take a look? I think this needs to take into account Python versions without bool support in the struct module, as the same ctypes code is used with older Python releases as well. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1649190&group_id=5470 From noreply at sourceforge.net Fri Feb 2 17:30:31 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 02 Feb 2007 08:30:31 -0800 Subject: [Patches] [ python-Patches-1641544 ] rlcompleter tab completion in pdb Message-ID: Patches item #1641544, was opened at 2007-01-22 11:52 Message generated for change (Comment added) made by stephenemslie You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1641544&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Stephen Emslie (stephenemslie) Assigned to: Nobody/Anonymous (nobody) Summary: rlcompleter tab completion in pdb Initial Comment: By default, Pdb and other instances of Cmd complete names for their commands. However in the context of pdb, I think it is more useful to complete identifiers and keywords in its current scope than to complete names of commands (most of which have single letter abbreviations). I believe this makes pdb a far more usable introspection tool. I have discussed this proposal on the python-ideas list: http://mail.python.org/pipermail/python-ideas/2007-January/000084.html This patch implements the following: - creates an rlcompleter instance on Pdb if readline is available - adds a 'complete' method to the Pdb class. The only difference with rlcompleter's default behaviour is that is also updates rlcompleter's namespace to reflect the current local and global namespace, which is necessary because pdb changes scope as it steps through a program This is a patch against python/Lib/pdb.py rev. 51745 ---------------------------------------------------------------------- >Comment By: Stephen Emslie (stephenemslie) Date: 2007-02-02 16:30 Message: Logged In: YES user_id=1698489 Originator: YES Thanks for your comments, and thanks for pointing out pydb I agree that debugger commands should also belong in the completion namespace. I'll have a look at adding that to the patch. > The second problem I have is that completion is not all that sensitive to the preceding context. The idea is that the namespace that is available to the completer at any time will be the same as the local and global identifiers and keywords that are available in the same scope. I think it makes sense to complete everything that is valid in the current scope because that makes for a more useful introspection tool. Thats what frame.f_locals and frame.f_globals are about (if foo and bar are available in the current scope, and you step into a function outside of that scope then foo and bar will no longer be available in the completer's namespace). I notice this is also used in pydb's complete method under certain circumstances. I'll definitely take a closer look at pydb. I probably wouldn't have needed to write this if I'd known about it before :) ---------------------------------------------------------------------- Comment By: Rocky Bernstein (rockyb) Date: 2007-01-28 02:48 Message: Logged In: YES user_id=158581 Originator: NO I experimented with this a little in the pydb variant (http://bashdb.sf.net/pydb). Some observations. First, one can include the debugger commands into the namespace without too much trouble. See what's checked into CVS for pydb; In particular look at the complete method of pydbbdb. (Personally, I think adding debugger commands to the list of completions is a little more honest.) The second problem I have is that completion is not all that sensitive to the preceding context. If the line begins "step" or "1 + ", is it really correct to list all valid symbols? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1641544&group_id=5470 From noreply at sourceforge.net Sat Feb 3 18:22:15 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 03 Feb 2007 09:22:15 -0800 Subject: [Patches] [ python-Patches-1630118 ] Patch to add tempfile.SpooledTemporaryFile (for #415692) Message-ID: Patches item #1630118, was opened at 2007-01-07 14:36 Message generated for change (Comment added) made by djmitche You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1630118&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Dustin J. Mitchell (djmitche) Assigned to: Nobody/Anonymous (nobody) Summary: Patch to add tempfile.SpooledTemporaryFile (for #415692) Initial Comment: Attached please find a patch that adds a SpooledTemporaryFile class to tempfile, along with the corresponding documentation (optimistically labeling the feature as added in Python 2.5) and some test cases. ---------------------------------------------------------------------- >Comment By: Dustin J. Mitchell (djmitche) Date: 2007-02-03 11:22 Message: Logged In: YES user_id=7446 Originator: YES Sounds good -- here's a new version. File Added: SpooledTemporaryFile.patch ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2007-01-11 01:55 Message: Logged In: YES user_id=4771 Originator: NO Reimplementing the whole file interface as a proxy functions might be the safest route, yes. I realized that the __getattr__() magic is also used to serve at least one special method, namely the __iter__() of the file objects. This only works with old-style classes. In the long-term future, when old-style classes disappear and these classes become new-style, this is likely to introduce a subtle bug. ---------------------------------------------------------------------- Comment By: Dustin J. Mitchell (djmitche) Date: 2007-01-08 09:53 Message: Logged In: YES user_id=7446 Originator: YES I agree it would break in such a situation, but I'm not clear on which direction your bias leads you (specifically, which do we get right -- don't use bound methods, or don't use the __getattr__ magic?). I could fix this by defining "proxy" functions (and some properties) for the whole file interface, rather than just the methods that potentially trigger rollover. That would lose a little efficiency, but mostly only in reading (calling e.g., f.read() will always result in two function applications; in the current model, after the first call it runs at "native" speed). It would also lose forward compatibility if the file protocol changes, although I'm not sure how likely that is. Would you like me to do that? ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2007-01-08 02:26 Message: Logged In: YES user_id=4771 Originator: NO The __getattr__ magic makes the following kind of code fail with SpooledTemporaryFile: f = SpooledTemporaryFile(max_size=something) rd = f.read wr = f.write for x in y: ...use rd(size) and wr(data)... The problem is that the captured 'f.read' method is the one from the StringIO instance, even after the write() rolled the file over to disk. Given that capturing bound methods is a semi-official speed hack advertised in some respected places, we might have to be careful about it. About such matters I am biased towards first getting it right and then getting it fast... Also, Python 2.5 is already out, so this will probably be a 2.6 addition. ---------------------------------------------------------------------- Comment By: Dustin J. Mitchell (djmitche) Date: 2007-01-07 14:37 Message: Logged In: YES user_id=7446 Originator: YES File Added: SpooledTemporaryFile.patch ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1630118&group_id=5470 From noreply at sourceforge.net Mon Feb 5 05:04:21 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 04 Feb 2007 20:04:21 -0800 Subject: [Patches] [ python-Patches-1540869 ] CodeContext visibility Message-ID: Patches item #1540869, was opened at 2006-08-15 15:20 Message generated for change (Comment added) made by kbk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1540869&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: Python 2.6 >Status: Closed >Resolution: Rejected Priority: 5 Private: No Submitted By: Jim Jewett (jimjjewett) Assigned to: Kurt B. Kaiser (kbk) Summary: CodeContext visibility Initial Comment: CodeContext hardcodes several constants; these two in particular make it difficult to visually separate the "context" from the top of the regular code. Ideally, the measurement would change to something like em or ex, but ... this was an improvement. Note that the separation is explicitly there in the code already; it just doesn't work with the current constants. If the Relesae Manager decides to delay it to 2.6 (or backport to 2.4), I won't object. ---------------------------------------------------------------------- >Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-04 23:04 Message: Logged In: YES user_id=149084 Originator: NO I agree with Tal Einat on this. ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2006-12-09 11:51 Message: Logged In: YES user_id=1330769 Originator: NO Increasing the vertical separation is a possibility, but I think this is mostly a matter of taste. Personally, I prefer to differentiate between the code and context with a different backgroud than to have a large blank separator. (Note that the background and foreground colors of the context are configurable.) In any case, just increasing the border ruins the horizontal alignment. This should be done with 'pady' or by adding a padding frame below the context window. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-08-16 13:19 Message: Logged In: YES user_id=764593 ahh ... I hadn't even noticed the (effectively one- character) horizontal change. The change I aimed for (which I thought was the goal of the pad frame) was a slight *vertical* separation between the parts. So definately not until 2.6, and hopefully the eventual replacement will be better than what I posted here. Thank you for the clarification. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-08-16 13:18 Message: Logged In: YES user_id=764593 ahh ... I hadn't even noticed the (effectively one- character) horizontal change. The change I aimed for (which I thought was the goal of the pad frame) was a slight *vertical* separation between the parts. So definately not until 2.6, and hopefully the eventual replacement will be better than what I posted here. Thank you for the clarification. ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2006-08-16 11:40 Message: Logged In: YES user_id=1330769 I originally wrote the code with those constants. I used those constants because I specifically wanted to line up the CodeContext code with the code in the main window, for easier reading. Your patch does the exact opposite - it pushes the CodeContext code way to the right. IMO, since the purpose of CodeContext is to have the current scope easily seen, having the code aligned is vital. You may prefer dis-alignment because it's easier to tell CodeContext apart, but that's just a personal preference, and IMO not something that should be the default behavior of IDLE. Personally, I find that CodeContext's different background color is more than enough to distinguish it from the main editor window. CodeContext's background and foreground colors are configurable (in config-extensions), you can always choose colors you like better. Tal ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1540869&group_id=5470 From noreply at sourceforge.net Mon Feb 5 05:16:24 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 04 Feb 2007 20:16:24 -0800 Subject: [Patches] [ python-Patches-1621265 ] Auto-completion list placement Message-ID: Patches item #1621265, was opened at 2006-12-23 08:16 Message generated for change (Comment added) made by kbk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1621265&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: Python 2.6 Status: Open Resolution: None >Priority: 5 Private: No Submitted By: Tal Einat (taleinat) >Assigned to: Kurt B. Kaiser (kbk) Summary: Auto-completion list placement Initial Comment: The completion list placement logic is currently very primitive. Sometimes the completion list extends outside of the editor window, or even outside the screen so that part of it is not visible. This patch places the completion list inside the text widget if possible, above or below the typed text, as appropriate. It will not hide the currently typed text in any case. End-cases: * If the text widget is not high enough, the completion list is placed below the currently typed text. * If the text widget is not wide enough, the completion list is aligned with the left-most edge of the text widget. In any case the size of the completion list is not altered to fit the text widget, though this would probably be the best solution when the text widget is too small. This means that with this patch, the completion list may extend beyond the text widget if it is small. I have not implemented resizing of the list this since the list's size is currently not set at all - Tk's default is used. Changing it anywhere in the code would require setting it elsewhere, hard-coding a default or making it configurable, etc. Comments please! ---------------------------------------------------------------------- >Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-04 23:16 Message: Logged In: YES user_id=149084 Originator: NO Haven't you introduced a regression on [ 1571112 ] simple moves freeze IDLE by the changes at the beginning of the patch? Also, the assignment to previous_completion was intended to make the code clearer. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1621265&group_id=5470 From noreply at sourceforge.net Mon Feb 5 05:29:00 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 04 Feb 2007 20:29:00 -0800 Subject: [Patches] [ python-Patches-1110205 ] Make Python Editor Useful Without Full IDLE Message-ID: Patches item #1110205, was opened at 2005-01-26 15:54 Message generated for change (Comment added) made by kbk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1110205&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: Python 2.4 >Status: Closed >Resolution: Rejected Priority: 5 Private: No Submitted By: sowjanya (sowjanya) Assigned to: Kurt B. Kaiser (kbk) Summary: Make Python Editor Useful Without Full IDLE Initial Comment: We are using the Filelist class from idlelib to create Python syntax aware source code editor in our applications. In order to be able to embed the code editor window into a Tk graphical user interface and specify the width and height of the editor (in lines and columns) we had to modify the EditorWidow and ScriptBinding classes to accept a 'top' argument (container in which to create the editor), and 'width' and 'height' arguments. We are submitting the patches for your consideration to be included into the next release of idlelib1.2a0. The patches were generated using the following command: diff -c old new ---------------------------------------------------------------------- >Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-04 23:29 Message: Logged In: YES user_id=149084 Originator: NO It seems to me that this further complicates what is an already fairly confusing part of the code. I'd rather not do that. ---------------------------------------------------------------------- Comment By: sowjanya (sowjanya) Date: 2005-03-04 16:38 Message: Logged In: YES user_id=1202212 Yes we are the same people.What i sent supersedes the patch previously sent to you. Thanks Sowjanya ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-03-04 14:06 Message: Logged In: YES user_id=21627 http://sourceforge.net/tracker/index.php?func=detail&aid=796254&group_id=9579&atid=309579 ---------------------------------------------------------------------- Comment By: sowjanya (sowjanya) Date: 2005-03-04 13:18 Message: Logged In: YES user_id=1202212 I couldn't find Idlefork Patch 796254 to make sure whether it is ours. Could you please send me an url for this patch or guide me in searching it. Thanks Sowjanya ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2005-03-03 19:50 Message: Logged In: YES user_id=149084 May I assume that this is a copy of Idlefork Patch 796254? Are you the same people who submitted that anonymous patch? If so, thanks for the context diff. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-03-03 18:03 Message: Logged In: YES user_id=21627 Kurt, can you please take a look? If not, please unassign. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1110205&group_id=5470 From noreply at sourceforge.net Mon Feb 5 05:30:33 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 04 Feb 2007 20:30:33 -0800 Subject: [Patches] [ python-Patches-1196946 ] allow using normal indent width in shell in IDLE Message-ID: Patches item #1196946, was opened at 2005-05-06 19:26 Message generated for change (Comment added) made by kbk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1196946&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: None >Status: Closed >Resolution: Rejected Priority: 5 Private: No Submitted By: Jeff Shute (jshute) Assigned to: Kurt B. Kaiser (kbk) Summary: allow using normal indent width in shell in IDLE Initial Comment: IDLE currently ignores the indent width setting in the main shell window, and forces it to use tab characters, 8 spaces wide. This is ugly, and causes problems when you write code in the shell and then cut-and-paste it into a real editor window. This patch adds an option (off by default) to honour the indent width setting in the PyShell window rather than using tabs. This makes it much nicer to edit commands that use several levels of indenting. The only reason for this to be an option, and for it to be off by default, is that it looks a little weird when the second line of code is indented, but still lines up with the first indented line because the prompt is also 4 characters wide. Fixing it so this would not look weird would be nice, but would require a lot of Tk hacking and might make the gui slower. ---------------------------------------------------------------------- >Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-04 23:30 Message: Logged In: YES user_id=149084 Originator: NO Sorry. Doesn't really solve the problem. ---------------------------------------------------------------------- Comment By: Jeff Shute (jshute) Date: 2005-05-12 00:46 Message: Logged In: YES user_id=39615 Hmm, I had only considered using a continuation of "...". Using spaces would make things look mostly right, and will have odd behaviour only when you cut-and-paste into an editor, since you will get an extra indentation level, but that won't cause errors and is easily fixed. Here is a new patch that implements it this way. I think doing it this way solves the problem well enough that the preference is not necessary. (Or both patches can be applied to get this behaviour and the preference.) ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2005-05-11 19:02 Message: Logged In: YES user_id=764593 Would using a continuation prompt of " "*len(sys.ps1) solve the problem? (standard module) code uses a continuation prompt of "... " which gets the spacing right, though it doesn't do your indents for you. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1196946&group_id=5470 From noreply at sourceforge.net Mon Feb 5 07:03:56 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 04 Feb 2007 22:03:56 -0800 Subject: [Patches] [ python-Patches-1650174 ] configHandler support for raw data Message-ID: Patches item #1650174, was opened at 2007-02-01 16:55 Message generated for change (Comment added) made by kbk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1650174&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: None >Status: Closed >Resolution: Accepted Priority: 3 Private: No Submitted By: Tal Einat (taleinat) >Assigned to: Kurt B. Kaiser (kbk) Summary: configHandler support for raw data Initial Comment: ConfigParser has support for reading/writing "raw" data, but support for this was simply not implemented in configHandler.py. This patch makes IdleConf.GetOption(...) pass on the "raw" keyword argument to the config handlers, which is all that was missing. This patch was previously posted as part of the Squeezer patch, but this is a very simple change, and it is now required by a new IDLE extension (currently in development.) ---------------------------------------------------------------------- >Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-05 01:03 Message: Logged In: YES user_id=149084 Originator: NO rev 53635. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1650174&group_id=5470 From noreply at sourceforge.net Mon Feb 5 07:05:02 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 04 Feb 2007 22:05:02 -0800 Subject: [Patches] [ python-Patches-1529353 ] Squeezer - squeeze large output in the interpreter Message-ID: Patches item #1529353, was opened at 2006-07-26 20:13 Message generated for change (Comment added) made by kbk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1529353&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: None Status: Open Resolution: None Priority: 4 Private: No Submitted By: Tal Einat (taleinat) Assigned to: Nobody/Anonymous (nobody) Summary: Squeezer - squeeze large output in the interpreter Initial Comment: Here is my improved version of Squeezer - the IDLE extension to end all of your oh-no-I-just-printed-War-and-Peace-in-the-interpreter troubles! Changes: * Fixed line counting logic * Line counting is done according to current window width and current tab width * Fixed configHandler to enable reading of raw data from config files (100% backwards compatible) - needed by Squeezer to enable configuration of text viewers in config-extensions I've been working this vesion without a hitch for several weeks, but that's hardly enough testing is it? Attached: * the original Squeezer.py by Noam Raphael * a diff against Noam's original Squeezer.py * a diff of configHandler.py against the current SVN version (revision 46863) * fixed 'middle click to copy' - now works on Windows One minor note - the 'middle click to copy' didn't work on Windows because it simply called selection_own() which doesn't affect the clipboard. I added clipboard_clear() and clipboard_append() and it now works on Windows. This should be tested on Linux, Unix, OSX etc. to check compatibility. For more info on the clipboard issue, see: http://groups.google.com/group/comp.lang.python/browse_thread/thread/647feeef55625733/bc08421d8aea9928?lnk=st&q=&rnum=2#bc08421d8aea9928 ---------------------------------------------------------------------- >Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-05 01:05 Message: Logged In: YES user_id=149084 Originator: NO Delete the patch to configHandler.py, implemented with Patch 1650174. ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2006-07-30 10:20 Message: Logged In: YES user_id=149084 Ah, that's better. Thanks. ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2006-07-30 07:09 Message: Logged In: YES user_id=1330769 Uploaded unified diffs instead of standard diffs. Probably the first diffs had bad newlines because I'm working on Win. Also 2 changes: * minor fix - end case when clicking at beginning of text block * Allow squeezing of stderr (tracebacks...) ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2006-07-26 22:49 Message: Logged In: YES user_id=149084 OK, thanks. Please take a look at the two diffs you uploaded, they seem corrupted. Squeezer.py looks ok. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1529353&group_id=5470 From noreply at sourceforge.net Mon Feb 5 12:45:16 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 05 Feb 2007 03:45:16 -0800 Subject: [Patches] [ python-Patches-1652328 ] stream writing support in wave.py Message-ID: Patches item #1652328, was opened at 2007-02-05 03:45 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1652328&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: tom (tom375) Assigned to: Nobody/Anonymous (nobody) Summary: stream writing support in wave.py Initial Comment: allows wave files to be written to non-seekable file descriptors ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1652328&group_id=5470 From noreply at sourceforge.net Mon Feb 5 13:03:47 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 05 Feb 2007 04:03:47 -0800 Subject: [Patches] [ python-Patches-1621265 ] Auto-completion list placement Message-ID: Patches item #1621265, was opened at 2006-12-23 15:16 Message generated for change (Comment added) made by taleinat You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1621265&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Tal Einat (taleinat) Assigned to: Kurt B. Kaiser (kbk) Summary: Auto-completion list placement Initial Comment: The completion list placement logic is currently very primitive. Sometimes the completion list extends outside of the editor window, or even outside the screen so that part of it is not visible. This patch places the completion list inside the text widget if possible, above or below the typed text, as appropriate. It will not hide the currently typed text in any case. End-cases: * If the text widget is not high enough, the completion list is placed below the currently typed text. * If the text widget is not wide enough, the completion list is aligned with the left-most edge of the text widget. In any case the size of the completion list is not altered to fit the text widget, though this would probably be the best solution when the text widget is too small. This means that with this patch, the completion list may extend beyond the text widget if it is small. I have not implemented resizing of the list this since the list's size is currently not set at all - Tk's default is used. Changing it anywhere in the code would require setting it elsewhere, hard-coding a default or making it configurable, etc. Comments please! ---------------------------------------------------------------------- >Comment By: Tal Einat (taleinat) Date: 2007-02-05 14:03 Message: Logged In: YES user_id=1330769 Originator: YES My mistake - I missed the changes at the top! I meant only to patch the placement mechanism. The newly attached patch includes only the fix for window placement. BTW, that change to the completion logic seems buggy - if a duplicate completion entry is selected, only one more letter (if available) will be added? IMO the current completion logic is weird and non-intuitive. When selecting a completion from the list (in any way) I would expect it all to appear where I am typing, not just some of it... File Added: AutoCompleteWindow.placement.070205.patch ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-05 06:16 Message: Logged In: YES user_id=149084 Originator: NO Haven't you introduced a regression on [ 1571112 ] simple moves freeze IDLE by the changes at the beginning of the patch? Also, the assignment to previous_completion was intended to make the code clearer. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1621265&group_id=5470 From noreply at sourceforge.net Mon Feb 5 21:15:40 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 05 Feb 2007 12:15:40 -0800 Subject: [Patches] [ python-Patches-1652681 ] tarfile append behavior Message-ID: Patches item #1652681, was opened at 2007-02-05 15:15 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1652681&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Witten (herrwitten) Assigned to: Nobody/Anonymous (nobody) Summary: tarfile append behavior Initial Comment: The builtin file object has the following behavior for the appending ('a') mode: 'The file will be created if it doesn't exist when opened for writing or appending' Currently, the file must already exist and it must not be an empty file. This patch achieves the file object's behavior with a small one-line addition. If this patch be used, please add a reference to 'herrwitten' in the log. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1652681&group_id=5470 From noreply at sourceforge.net Mon Feb 5 22:05:48 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 05 Feb 2007 13:05:48 -0800 Subject: [Patches] [ python-Patches-1652681 ] tarfile append behavior Message-ID: Patches item #1652681, was opened at 2007-02-05 20:15 Message generated for change (Settings changed) made by gbrandl You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1652681&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Witten (herrwitten) >Assigned to: Lars Gust?bel (gustaebel) Summary: tarfile append behavior Initial Comment: The builtin file object has the following behavior for the appending ('a') mode: 'The file will be created if it doesn't exist when opened for writing or appending' Currently, the file must already exist and it must not be an empty file. This patch achieves the file object's behavior with a small one-line addition. If this patch be used, please add a reference to 'herrwitten' in the log. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1652681&group_id=5470 From noreply at sourceforge.net Tue Feb 6 00:33:16 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 05 Feb 2007 15:33:16 -0800 Subject: [Patches] [ python-Patches-1652681 ] tarfile append behavior Message-ID: Patches item #1652681, was opened at 2007-02-05 15:15 Message generated for change (Comment added) made by herrwitten You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1652681&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Witten (herrwitten) Assigned to: Lars Gust?bel (gustaebel) Summary: tarfile append behavior Initial Comment: The builtin file object has the following behavior for the appending ('a') mode: 'The file will be created if it doesn't exist when opened for writing or appending' Currently, the file must already exist and it must not be an empty file. This patch achieves the file object's behavior with a small one-line addition. If this patch be used, please add a reference to 'herrwitten' in the log. ---------------------------------------------------------------------- >Comment By: Witten (herrwitten) Date: 2007-02-05 18:33 Message: Logged In: YES user_id=1595909 Originator: YES I fixed a bad patch with some personal information and bad path names by using 'svn diff' instead. File Added: tarfile.py.diff ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1652681&group_id=5470 From noreply at sourceforge.net Tue Feb 6 14:43:54 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 06 Feb 2007 05:43:54 -0800 Subject: [Patches] [ python-Patches-1623563 ] Allow __class __ assignment for classes with __slots__ Message-ID: Patches item #1623563, was opened at 2006-12-28 11:48 Message generated for change (Comment added) made by therve You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1623563&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: TH (therve) Assigned to: Nobody/Anonymous (nobody) Summary: Allow __class __ assignment for classes with __slots__ Initial Comment: I made a modification in typeobject.c to allow __class__ modification for classes with slots. It's basically a change in same_slots_added to count the offset of the slots and check if the names of the slots are the same (in the naive way). I don't check if slots are in a different order, that may be an improve. The patch is against trunk, with some tests. It's my first submission on Python, so every feedback will be welcome :). ---------------------------------------------------------------------- >Comment By: TH (therve) Date: 2007-02-06 13:43 Message: Logged In: YES user_id=1038797 Originator: YES Is there anything I can do to have a resolution on this ? Thanks. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-12-31 00:26 Message: Logged In: YES user_id=764593 Originator: NO Just going through the list doesn't seem so naive to me. If the slots are in a different order, then you would need to move the data around -- which borders on "maybe they ought to have written the translator themselves" ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1623563&group_id=5470 From noreply at sourceforge.net Tue Feb 6 19:46:55 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 06 Feb 2007 10:46:55 -0800 Subject: [Patches] [ python-Patches-1652681 ] tarfile append behavior Message-ID: Patches item #1652681, was opened at 2007-02-05 21:15 Message generated for change (Comment added) made by gustaebel You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1652681&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Witten (herrwitten) Assigned to: Lars Gust?bel (gustaebel) Summary: tarfile append behavior Initial Comment: The builtin file object has the following behavior for the appending ('a') mode: 'The file will be created if it doesn't exist when opened for writing or appending' Currently, the file must already exist and it must not be an empty file. This patch achieves the file object's behavior with a small one-line addition. If this patch be used, please add a reference to 'herrwitten' in the log. ---------------------------------------------------------------------- >Comment By: Lars Gust?bel (gustaebel) Date: 2007-02-06 19:46 Message: Logged In: YES user_id=642936 Originator: NO Thank you for the idea. As your patch only implemented appending to empty files, I added support for creating files if they don't exist. I added a testcase and adjusted the docs as well. Checked in the trunk as rev. 53648, so it will appear in 2.6. ---------------------------------------------------------------------- Comment By: Witten (herrwitten) Date: 2007-02-06 00:33 Message: Logged In: YES user_id=1595909 Originator: YES I fixed a bad patch with some personal information and bad path names by using 'svn diff' instead. File Added: tarfile.py.diff ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1652681&group_id=5470 From noreply at sourceforge.net Tue Feb 6 20:12:07 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 06 Feb 2007 11:12:07 -0800 Subject: [Patches] [ python-Patches-1362975 ] CodeContext - Improved text indentation Message-ID: Patches item #1362975, was opened at 2005-11-21 13:06 Message generated for change (Comment added) made by kbk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1362975&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE >Group: Python 2.6 Status: Closed Resolution: Accepted Priority: 3 Private: No Submitted By: Tal Einat (taleinat) Assigned to: Kurt B. Kaiser (kbk) Summary: CodeContext - Improved text indentation Initial Comment: This is a second patch for the text indentation - the first one was sent by mail and patched directly by KBK. I've touched up the indentation code to be cleaner and more generic. There's no longer need for a pad Frame, the padding is done by the Label widget. More IDLE patches coming - stay tuned ;) ---------------------------------------------------------------------- >Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-06 14:12 Message: Logged In: YES user_id=149084 Originator: NO Rev 53649 Thanks for the revised patch! ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2006-12-17 05:00 Message: Logged In: YES user_id=1330769 Originator: YES The changes were at Kurt's request. No new issues are addressed, it's only a revision of the previous changes. ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2006-12-16 18:40 Message: Logged In: YES user_id=1330769 Originator: YES Revised excessively verbose comments, removed overly defensive try/except blocks, some minor comment and code cleanup. File Added: CodeContext.061217.patch ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-16 18:01 Message: Logged In: YES user_id=21627 Originator: NO Hmm. This patch is already closed. Should the previous checkin be reverted? If this addresses additional issues, please submit it as a new report. ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2006-12-16 17:35 Message: Logged In: YES user_id=1330769 Originator: YES Revised excessively verbose comments, removed overly defensive try/except blocks, some minor comment and code cleanup. File Added: CodeContext.061217.patch ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2006-11-22 09:30 Message: Logged In: YES user_id=1330769 Originator: YES Woohoo! :) ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-11-22 03:51 Message: Logged In: YES user_id=21627 Originator: NO Thanks for the patch. I can't find anything wrong anymore, and agree that the restructuring is better, so I have committed it as r52821. ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2006-11-20 14:06 Message: Logged In: YES user_id=1330769 Originator: YES Seems like a bug in EditorWindow.py to me - shouldn't edtiwin.vbar (the scrollbar) be created inside editwin.text_frame instead of editwin.top? Seems to me that this is the reason text_frame is there in the first place - to encapsulate the Text widget and the scrollbar that accompanies it. Funny that I hadn't noticed the odd placement of the scrollbar... Thanks for the thorough review! I've submitted a revised patch, which fixes the bug in EditorWindow.py. I've also wrapped all widget attribute lookups in try/except blocks to avoid weird Tk behavior. This way future changes in Tk's inner workings will cause the text to be misaligned, instead of crashing IDLE. Lastly, I've added verbose comments where appropriate. Testing on current SVN trunk version of IDLE with only these changes shows no problems, and it works smoothly with my development version of IDLE as well. Let's finally get this piece of code behind us :) ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-11-19 14:18 Message: Logged In: YES user_id=21627 Originator: NO The patch is incorrect: without the patch, the scrollbar for the text window ends below the context window. With the patch, the scrollbar is on the side of the context window also. Tentatively rejecting the patch; if you revise it, please add some comments explaining all the computations. ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2005-11-23 04:04 Message: Logged In: YES user_id=1330769 This patch is an improvement for two reasons: 1. One less TK widget is used 2. Replaces literal values with code that fetches the appropriate values from the editor window's widgets I'll admit it's not that big a deal. If it's really a lot of work, I'll leave it up to you to decide whether it's worth it. ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2005-11-22 20:25 Message: Logged In: YES user_id=149084 At first glance the new code seems harder to understand and is longer. What is the advantage of going through the effort to apply, test, check in, and properly document the change? p.s. use Expand=False ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1362975&group_id=5470 From noreply at sourceforge.net Tue Feb 6 20:23:02 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 06 Feb 2007 11:23:02 -0800 Subject: [Patches] [ python-Patches-1540849 ] except too broad Message-ID: Patches item #1540849, was opened at 2006-08-15 14:55 Message generated for change (Comment added) made by kbk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1540849&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: Python 2.6 >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Jim Jewett (jimjjewett) Assigned to: Kurt B. Kaiser (kbk) Summary: except too broad Initial Comment: IOBinding.open looks for an self.editwin.interp (after having already verified that self.editwin exists), but catches exceptions with a bare except. This narrows it to an AttributeError. (Note that the save method already assumes that missing attributes on an editwin will be represented as AtributeError rather than TypeError.) ---------------------------------------------------------------------- >Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-06 14:23 Message: Logged In: YES user_id=149084 Originator: NO Rev 53650. This wasn't really worth a patch. What other exception could have occurred? (Yes, it's nicer to be narrow :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1540849&group_id=5470 From noreply at sourceforge.net Tue Feb 6 23:02:02 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 06 Feb 2007 14:02:02 -0800 Subject: [Patches] [ python-Patches-1652681 ] tarfile append behavior Message-ID: Patches item #1652681, was opened at 2007-02-05 15:15 Message generated for change (Comment added) made by herrwitten You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1652681&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Closed Resolution: Accepted Priority: 5 Private: No Submitted By: Witten (herrwitten) Assigned to: Lars Gust?bel (gustaebel) Summary: tarfile append behavior Initial Comment: The builtin file object has the following behavior for the appending ('a') mode: 'The file will be created if it doesn't exist when opened for writing or appending' Currently, the file must already exist and it must not be an empty file. This patch achieves the file object's behavior with a small one-line addition. If this patch be used, please add a reference to 'herrwitten' in the log. ---------------------------------------------------------------------- >Comment By: Witten (herrwitten) Date: 2007-02-06 17:02 Message: Logged In: YES user_id=1595909 Originator: YES Thanks! I'll try to be as thorough next time. ---------------------------------------------------------------------- Comment By: Lars Gust?bel (gustaebel) Date: 2007-02-06 13:46 Message: Logged In: YES user_id=642936 Originator: NO Thank you for the idea. As your patch only implemented appending to empty files, I added support for creating files if they don't exist. I added a testcase and adjusted the docs as well. Checked in the trunk as rev. 53648, so it will appear in 2.6. ---------------------------------------------------------------------- Comment By: Witten (herrwitten) Date: 2007-02-05 18:33 Message: Logged In: YES user_id=1595909 Originator: YES I fixed a bad patch with some personal information and bad path names by using 'svn diff' instead. File Added: tarfile.py.diff ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1652681&group_id=5470 From noreply at sourceforge.net Wed Feb 7 04:47:06 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 06 Feb 2007 19:47:06 -0800 Subject: [Patches] [ python-Patches-1621265 ] Auto-completion list placement Message-ID: Patches item #1621265, was opened at 2006-12-23 08:16 Message generated for change (Comment added) made by kbk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1621265&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: Python 2.6 >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Tal Einat (taleinat) Assigned to: Kurt B. Kaiser (kbk) Summary: Auto-completion list placement Initial Comment: The completion list placement logic is currently very primitive. Sometimes the completion list extends outside of the editor window, or even outside the screen so that part of it is not visible. This patch places the completion list inside the text widget if possible, above or below the typed text, as appropriate. It will not hide the currently typed text in any case. End-cases: * If the text widget is not high enough, the completion list is placed below the currently typed text. * If the text widget is not wide enough, the completion list is aligned with the left-most edge of the text widget. In any case the size of the completion list is not altered to fit the text widget, though this would probably be the best solution when the text widget is too small. This means that with this patch, the completion list may extend beyond the text widget if it is small. I have not implemented resizing of the list this since the list's size is currently not set at all - Tk's default is used. Changing it anywhere in the code would require setting it elsewhere, hard-coding a default or making it configurable, etc. Comments please! ---------------------------------------------------------------------- >Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-06 22:47 Message: Logged In: YES user_id=149084 Originator: NO Rev 53653 Thanks for the patch! I agree with your comment about the completion logic. I would have thought that by now someone from Noam's enthusiast group would have sent a patch. If there's only one completion, a tab should complete it. If there is a mouse selection in the ACW, it should be chosen. If an up/down cursor key is used to change the ACW selection, the selection should be chosen. Otherwise, the closest match in the ACW window to what's been typed should be highlighted. I'm looking at it. Any comments on this? The change to the completion logic prevented a loop, but as you say, the logic was buggy to begin with. ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2007-02-05 07:03 Message: Logged In: YES user_id=1330769 Originator: YES My mistake - I missed the changes at the top! I meant only to patch the placement mechanism. The newly attached patch includes only the fix for window placement. BTW, that change to the completion logic seems buggy - if a duplicate completion entry is selected, only one more letter (if available) will be added? IMO the current completion logic is weird and non-intuitive. When selecting a completion from the list (in any way) I would expect it all to appear where I am typing, not just some of it... File Added: AutoCompleteWindow.placement.070205.patch ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-04 23:16 Message: Logged In: YES user_id=149084 Originator: NO Haven't you introduced a regression on [ 1571112 ] simple moves freeze IDLE by the changes at the beginning of the patch? Also, the assignment to previous_completion was intended to make the code clearer. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1621265&group_id=5470 From noreply at sourceforge.net Wed Feb 7 08:26:51 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 06 Feb 2007 23:26:51 -0800 Subject: [Patches] [ python-Patches-1621265 ] Auto-completion list placement Message-ID: Patches item #1621265, was opened at 2006-12-23 15:16 Message generated for change (Comment added) made by taleinat You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1621265&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: Python 2.6 Status: Closed Resolution: Accepted Priority: 5 Private: No Submitted By: Tal Einat (taleinat) Assigned to: Kurt B. Kaiser (kbk) Summary: Auto-completion list placement Initial Comment: The completion list placement logic is currently very primitive. Sometimes the completion list extends outside of the editor window, or even outside the screen so that part of it is not visible. This patch places the completion list inside the text widget if possible, above or below the typed text, as appropriate. It will not hide the currently typed text in any case. End-cases: * If the text widget is not high enough, the completion list is placed below the currently typed text. * If the text widget is not wide enough, the completion list is aligned with the left-most edge of the text widget. In any case the size of the completion list is not altered to fit the text widget, though this would probably be the best solution when the text widget is too small. This means that with this patch, the completion list may extend beyond the text widget if it is small. I have not implemented resizing of the list this since the list's size is currently not set at all - Tk's default is used. Changing it anywhere in the code would require setting it elsewhere, hard-coding a default or making it configurable, etc. Comments please! ---------------------------------------------------------------------- >Comment By: Tal Einat (taleinat) Date: 2007-02-07 09:26 Message: Logged In: YES user_id=1330769 Originator: YES My only comment is that I've already implemented precisely the logic you've described. The changes required are mostly removal of code... I could cough up a patch, sometime about the weekend. (That way we could also move the conversation from here :) ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-07 05:47 Message: Logged In: YES user_id=149084 Originator: NO Rev 53653 Thanks for the patch! I agree with your comment about the completion logic. I would have thought that by now someone from Noam's enthusiast group would have sent a patch. If there's only one completion, a tab should complete it. If there is a mouse selection in the ACW, it should be chosen. If an up/down cursor key is used to change the ACW selection, the selection should be chosen. Otherwise, the closest match in the ACW window to what's been typed should be highlighted. I'm looking at it. Any comments on this? The change to the completion logic prevented a loop, but as you say, the logic was buggy to begin with. ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2007-02-05 14:03 Message: Logged In: YES user_id=1330769 Originator: YES My mistake - I missed the changes at the top! I meant only to patch the placement mechanism. The newly attached patch includes only the fix for window placement. BTW, that change to the completion logic seems buggy - if a duplicate completion entry is selected, only one more letter (if available) will be added? IMO the current completion logic is weird and non-intuitive. When selecting a completion from the list (in any way) I would expect it all to appear where I am typing, not just some of it... File Added: AutoCompleteWindow.placement.070205.patch ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-05 06:16 Message: Logged In: YES user_id=149084 Originator: NO Haven't you introduced a regression on [ 1571112 ] simple moves freeze IDLE by the changes at the beginning of the patch? Also, the assignment to previous_completion was intended to make the code clearer. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1621265&group_id=5470 From noreply at sourceforge.net Wed Feb 7 08:49:03 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 06 Feb 2007 23:49:03 -0800 Subject: [Patches] [ python-Patches-1529142 ] Allowing multiple instances of IDLE with sub-processes Message-ID: Patches item #1529142, was opened at 2006-07-26 20:45 Message generated for change (Comment added) made by taleinat You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1529142&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Tal Einat (taleinat) Assigned to: Kurt B. Kaiser (kbk) Summary: Allowing multiple instances of IDLE with sub-processes Initial Comment: (See patch 1201569 "allow running multiple instances of IDLE" for previous discussion on this topic) To summarize discussion up to this point: There is a problem doing this on Windows since more than one listening socket can be openned on a single port. I suggest we "dodge" the Windows problem for now by having IDLE try a random port every time (like in the IDLEfork patch 661363) - thus collisions will be kept to a minimum. If we choose among over 10,000 ports, most users will never encounter a port collision. And even when a collision does happen, it will probably be detected and properly dealt with - collision non-detection errors are, as Kurt mentioned, erratic. This patch chooses a port from range(49152, 65536), which are the 'dynamic' ports as described by the DCCP (see http://www.iana.org/assignments/port-numbers), and avoids known 'dangerous' ports (used by trojans, worms, etc.) which I gathered by Googling for half an hour. I replaced socket.timeout with select(), since I find its timeout mechanism is more reliable. (Currently, Python's socket's timeout mechanism raises unexpected, unexplained errors on Windows.) I also changed the flow a bit, so if a timeout occurs after a sub-process is spawned, the subprocess is killed. IMO This requires thorough testing. I tested it for half an hour on my WinXP Python2.4.3, and after fixing typos and such, everything seems to work. ---------------------------------------------------------------------- >Comment By: Tal Einat (taleinat) Date: 2007-02-07 09:49 Message: Logged In: YES user_id=1330769 Originator: YES After removing the SO_REUSEADDR flag, this has been working without any pauses, delays or hitches on my WinXP Python2.5 install. I sometimes have as many as 8 separate instances of IDLE running in parallel, with no side effects. About the SO_REUSEADDR flag, I'm wondering if this is still required on Posix systems now that we're choosing random ports. If so, we should choose whether to use this flag based on the OS. Is there any good reason -not- to choose a random port? I can't think of any. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1529142&group_id=5470 From noreply at sourceforge.net Wed Feb 7 09:32:57 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 07 Feb 2007 00:32:57 -0800 Subject: [Patches] [ python-Patches-1633807 ] from __future__ import print_function Message-ID: Patches item #1633807, was opened at 2007-01-12 08:13 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1633807&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Anthony Baxter (anthonybaxter) Assigned to: Nobody/Anonymous (nobody) Summary: from __future__ import print_function Initial Comment: This was done partly as a learning exercise, partly just as a vague idea that might prove to be practical (chatting with Neal at the time, but all blame is with me, not him!) The following adds 'from __future__ import print_function' to 2.x. When this is enabled, 'print' is no longer a statement. Combined with copying bltinmodule.c:builtin_print() from the p3yk trunk, this should give some compatibility options for 2.6 <-> 3.0 Note that for some reason I don't fully understand, this doesn't work in interactive mode. For some reason, in interactive mode, the parser flags get reset for each line. Wah. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-07 09:32 Message: Logged In: YES user_id=21627 Originator: NO rhettinger: I agree with anthonybaxter; implementing this in a module to import is not possible (unless you want *another* module that works like __future__ imports, i.e. where importing it isn't just a plain import - you could then just as well require to put "this is py3k" as the first line of the file). I wish PEP 244 would have been adopted, then you could "directive python3" at the top. As for testing: I don't see it as a problem if the test suite only works with the version where the feature is introduced. See test_future* and test_with. As for PRINT_ITEM: it was always inlined, since day 1. It has grown since, but apparently nobody ever bothered refactoring (on day 1, it was smaller, and the bulk of it was, and still is, in writeobject aka PyFile_WriteObject) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2007-01-30 07:19 Message: Logged In: YES user_id=29957 Originator: YES That only works for a very small number of the changes. I can't see how we could change the parser using a module. from __future__ import foo is the standard way we make forwards-compatible changes to Python. It's been that way for quite a while now - I don't see it as being controversial at all. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2007-01-29 18:07 Message: Logged In: YES user_id=80475 Originator: NO Instead of __future__ imports, it would be better to put all of this Py3.0 stuff in a single compatability module and keep the rest of Py2.x as clean as possible. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2007-01-29 13:37 Message: Logged In: YES user_id=29957 Originator: YES Attached version 3 of the patch. I've added an '#if 0'd warning in ast.c - for instance, when enabled, you get ./setup.py:1336: SyntaxWarning: print no longer a statement in Py3.0 I'll make a new version of a -W py3k patch that enables this as well. I've made the other cleanup suggested by twouters. I'm not clear on the best way to do the tests for this - the from __future__ needs to be at the start of the file. My concern is that anything that tries to compile the whole test file with this under a previous version will choke and die on the print-as-function. Not sure if this is a hugely bad problem or not. Docs will follow once I bother wrapping my head around LaTeX and figuring out the best way to do the docs. I'm guessing we need a note in ref/ref6.tex in the section on the print statement, another bit in the same file in the subsection on Future statements, and something in lib/libbltin.tex. Did I miss anywhere? In current 3.0, the builtin is called Print, not print. Is there a reason for this? Is it just a matter of updating all the tests and ripping out the support for the print statement and the related opcodes? If so, I'll tackle that next. Doing this does mean that the docs and the stdlib and the tests will all need a huge amount of updating, and it will make merging from the trunk to the p3yk branch much more painful. While I'm in the vague area - why is PRINT_ITEM inlined in ceval.c? Couldn't it be punted out to a separate function, making the main switch statement just that little bit smaller? I can't imagine that making 'print' that little tiny bit faster is actually worthwhile, compared to shrinking the main switch statement. except E as V, I'll look at later for a different patch. My tree is already getting quite cluttered already with uncommitted patches :-) File Added: print_function.patch3 ---------------------------------------------------------------------- Comment By: Thomas Wouters (twouters) Date: 2007-01-17 16:58 Message: Logged In: YES user_id=34209 Originator: NO You seem to have '#if 0'ed-out some code related to the with/as-statement warnings; I suggest just removing them. Since you're in this code now, it might make sense to provide a commented out warning about the use of the print statement, so we won't have to figure it out later (in Python 2.9 or when we add -Wp3yk.) It needs a test, and probably a doc change somewhere. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-01-17 16:24 Message: Logged In: YES user_id=6380 Originator: NO I don't think we need to do anything special for exec, as the exec(s, locals, globals) syntax is already (still :-) supported in 2.x with identical semantics as in 3.0. except E as V *syntax* can go in without a future stmt; and (only when that syntax is used) it should also enforce the new semantics (V must be a simple name and is deleted at the end of the except clause). I think Anthony's patch is a great idea, but I'll refrain from reviewing it. I'd say "just do it". :-) ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2007-01-17 08:42 Message: Logged In: YES user_id=33168 Originator: NO Guido, this is the patch I was talking about wrt supporting a print function in 2.6. exec could get similar treatment. You mentioned in mail that things like except E as V: can go in without a future stmt. I agree. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2007-01-12 08:31 Message: Logged In: YES user_id=29957 Originator: YES Updated version of patch - fixes interactive mode, adds builtins.print File Added: print_function.patch ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1633807&group_id=5470 From noreply at sourceforge.net Wed Feb 7 18:58:48 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 07 Feb 2007 09:58:48 -0800 Subject: [Patches] [ python-Patches-1654417 ] operator.c slice functions need to parse ssize_t Message-ID: Patches item #1654417, was opened at 2007-02-07 12:58 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1654417&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Andy Wingo (wingo) Assigned to: Nobody/Anonymous (nobody) Summary: operator.c slice functions need to parse ssize_t Initial Comment: To reproduce, on a system with 64 bits in sys.maxint: $ python x = [3,4,5] del x[1:] x = [3,4,5] from operator import delslice import sys delslice(x, 1, sys.maxint) Traceback (most recent call last): File "", line 1, in OverflowError: signed integer is greater than maximum ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1654417&group_id=5470 From noreply at sourceforge.net Wed Feb 7 21:17:23 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 07 Feb 2007 12:17:23 -0800 Subject: [Patches] [ python-Patches-1615701 ] Creating dicts for dict subclasses Message-ID: Patches item #1615701, was opened at 2006-12-14 08:08 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1615701&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.6 >Status: Closed >Resolution: Fixed Priority: 5 Private: No Submitted By: Walter D?rwald (doerwalter) Assigned to: Raymond Hettinger (rhettinger) Summary: Creating dicts for dict subclasses Initial Comment: This patch changes dictobject.c so that creating dicts from mapping like objects only uses the internal dict functions if the argument is a *real* dict, not a subclass. This means that overwritten keys() and __getitem__() methods are now honored. In addition to that the fallback implementation now tries iterkeys() before trying keys(). It also adds a PyMapping_IterKeys() macro. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2007-02-07 15:17 Message: Logged In: YES user_id=80475 Originator: NO Added PyDict_CheckExact() in revisions 53655 and 53656. A side-effect of this change is to slow-down updates with dict subclasses that are not overriding keys() and __getitem__(). This is especially unfortunate given good existing alternatives and given a lack of real use cases (dict subclasses that aspire to hand-off updates but not use their actual keys and mapped values). Left out the gratuitous expansion of the API which exposed too much of the internal implementation and sought to introduce more implicit behaviors that would better be handled by explictly passing in an iterable of items to d.update(). For example. d.update((k(x), g(x)) for x in myweirdmapping). ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2006-12-20 07:59 Message: Logged In: YES user_id=89016 Originator: YES To clear up some apparent misunderstandings: This patch does *not* advocate that some dict methods should be implemented by calling other dict methods so that dict subclasses only have to overwrite a few methods to gain a completely consistent implementation. This patch only fixes the dict constructor (and update()) and consists of two parts: (1) There are three code paths in dictobject.c::dict_update_common(): (a) if the constructor argument doesn't have a "keys" attribute treat it as a iterable of items; (b) if the argument has a "keys" attribute, but is not a dict (and not an instance of a subclass of dict), use keys() and __getitem__() to make a copy of the mapping-like object. (c) if the argument has a "keys" attribute and is a dict (or an instance of a subclass of dict) bypass any of the overwritten methods that the object might provide and directly use the dict implementation. This patch changes PyDict_Merge() so that code path (b) is used for dict constructor arguments that are subclasses of dict, so that any overwritten methods are honored. (2) This means that now if a subclass of dict is passed to the constructor or update() the code is IMHO more correct (it honors the reimplemenation of the mapping methods), but slower. To reduce the slowdown instead of using kesY() and __getitem__(), iterkeys() and __getitem__() are used. I can't see why the current behaviour should be better: Yes, it is faster, but it is also wrong: Without the patch the behaviour of dict() and dict.update() depends on the fact whether the argument happens to subclass dict or not. If it doesn't all is well: the argument is treated as a mapping (i.e. keys() and __getitem__() are used) otherwise the methods are completely ignored. So can we agree on the fact that (1) is desirable? (At least Guido said as much: http://mail.python.org/pipermail/python-dev/2006-December/070341.html) BTW, I only added PyMapping_Iterkeys() because it mirrors PyMapping_Keys(). ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2006-12-19 19:13 Message: Logged In: YES user_id=80475 Originator: NO Since update already supports (key, item) changes, I do not see that rationale in trying to expand the definition what is dict-like to include a try-this, then try-that approach. This is a little too ad-hoc for too little benefit. Also, I do not see the point of adding PyMapping_Iterkeys to the C API. It affords no advantage over its macro definition (the current one-way-to-do-it). ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2006-12-19 18:00 Message: Logged In: YES user_id=80475 Originator: NO It is also asking for bugs if someone hooks __getitem__ and starts to make possibly invalid assumptions about what other changes occur implicitly. Also, this patch kills the performance of builtin subclasses. If I subclass dict to add a new method, it would suck to have the performance of all of the other methods drop percariously. This patch is somewhat overzealous. It encroaches on the terriority of UserDict.DictMixin which was specifically made for propagating new behaviors. It unnecessarily exposes implementation details. It introduces implicit behaviors that should be done through explicit overrides of methods whose behavior is supposed to change. And, it is on the top of a slippery slope that we don't want to go down (i.e. do we want to guarantee that list.append is implemented in terms of list.extend, etc). Python has no shortage of places where builtin subclasses make direct calls to the underlying C code -- this patch leads to a bottomless pit of changes that kill performance and make implicit side-effects the norm instead of the exception. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-12-19 17:29 Message: Logged In: YES user_id=764593 Originator: NO FWIW, I'm not sure I agree on not specifying which methods call share implementation. If someone hooks __getitem__ but not get, that is just asking for bugs. (The implementation of get may -- but need not -- make its own call to __getitem__, and not everyone will make the same decision.) ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-12-19 17:26 Message: Logged In: YES user_id=764593 Originator: NO As I understand it, the problem is that dict.update is assuming any dict subclass will use the same internal data representation. Restricting the fast path to exactly builtin dicts (not subclasses) fixes the bug, but makes the fallback more frequent. The existing fallback is to call keys(), then iterate over it, retrieving the value for each key. (keys is required for a "minimal mapping" as documented is UserDict, and a few other random places.) The only potential dependency on other methods is his proposed new intermediate path that avoids creating a list of all keys, by using iterkeys if it exists. (I suggested using iteritems to avoid the lookups.) If iter* aren't implemented, the only harm is falling back to the existing fallback of "for k in keys():" ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2006-12-19 16:07 Message: Logged In: YES user_id=80475 Originator: NO I'm -1 on making ANY guarantees about which methods underlie others -- that would constitute new and everlasting guarantees about how mappings are implemented. Subclasses should explicity override/extend the methods withed changed behavior. If that proves non-trivial, then it is likely there should be a has-a relationship instead of an is-a relationship. Also, it is likely that the subclass will have Liskov substitutability violations. Either way, there is probably a design flaw. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2006-12-19 14:23 Message: Logged In: YES user_id=89016 Originator: YES iteritems() has to create a new tuple for each item, so this might be slower. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-12-19 12:50 Message: Logged In: YES user_id=764593 Originator: NO Why are you using iterkeys instead of iteritems? It seems like if they've filled out the interface enough to have iterkeys, they've probably filled it out all the way, and you do need the value as soon as you get the key. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1615701&group_id=5470 From noreply at sourceforge.net Wed Feb 7 22:05:01 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 07 Feb 2007 13:05:01 -0800 Subject: [Patches] [ python-Patches-1397711 ] Fix dict and set docs, re: immutability Message-ID: Patches item #1397711, was opened at 2006-01-05 07:28 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1397711&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Documentation Group: None >Status: Closed >Resolution: Fixed Priority: 5 Private: No Submitted By: Collin Winter (collinwinter) Assigned to: Raymond Hettinger (rhettinger) Summary: Fix dict and set docs, re: immutability Initial Comment: This patch resolves bug #1368768. As mentioned in the bug description, the documentation for sets and dicts incorrectly reports that set members and dict keys must be immutable. This patch corrects this, changing references to immutability to hashability and inserting footnotes describing what is meant by "hashable". The patch is a diff against Doc/lib/libstdtypes.tex from SVN revision 41926. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2007-02-07 16:05 Message: Logged In: YES user_id=80475 Originator: NO Fixed in revisions 53657 and 53658. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2006-07-25 03:23 Message: Logged In: YES user_id=29957 Raymond, this is assigned to you - were you going to check it in? If not, I will do it. ---------------------------------------------------------------------- Comment By: Collin Winter (collinwinter) Date: 2006-01-20 08:07 Message: Logged In: YES user_id=1344176 I've updated the patch to include Jim's version of the footnote. The new version of the patch is against SVN revision 42105. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-01-13 17:09 Message: Logged In: YES user_id=764593 Definately an improvement, but it still may not be quite right in the footnote. You don't have to define a __hash__ method; you just need to avoid redefining it (or __eq__) to something that doesn't work. Here is another stab Hashable objects must define or inherit proper \method{__hash__} and \method{__eq__} methods. Most objects do, but some mutable types -- including lists -- redefine equality in such a way that the hash code would not be stable. Instead, their hash functions return -1 to indicate that the object should not be used as a dict key or set member. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1397711&group_id=5470 From noreply at sourceforge.net Wed Feb 7 22:43:36 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 07 Feb 2007 13:43:36 -0800 Subject: [Patches] [ python-Patches-1443159 ] Remove bad PREDICT in ceval.c Message-ID: Patches item #1443159, was opened at 2006-03-04 13:52 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1443159&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.5 >Status: Closed >Resolution: Invalid Priority: 5 Private: No Submitted By: Collin Winter (collinwinter) Assigned to: Raymond Hettinger (rhettinger) Summary: Remove bad PREDICT in ceval.c Initial Comment: The current FOR_ITER -> UNPACK_SEQUENCE prediction is only right 6.41% of the time (212 million opcodes analysed). This patch removes it. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2006-03-06 19:02 Message: Logged In: YES user_id=80475 Collin, you are welcome to write to me directly to discuss the results of your dynamic analysis. I suspect your results are almost certainly incorrect. The prediction was originally added after a detailed analysis of a great deal of Python code and checking the results of multiple pieces of benchmark code. Please do not arbitrarily start removing these unless you're damned certain the original study was erronenous. Also, I suspect that the given percentage does not reflect the conditional probability of the preceding PREDICT (STORE_FAST). The overwhelming majority of cases needing optimization involve "for x in iterable" where x is a fast local. If that prediction fails, then the next most common case is "for x,y in iterable". The latter is highly likely whenever we know the former has not occured. IOW, the unpack sequence may only come up 6% of the time, but its conditional probability rises above 90% when it follows the other prediction. Also, take some case with dynamic analysis. It is always tricky because it is very difficult to find truly representative code and to not have a single anomalous case run over and over again. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-03-06 18:42 Message: Logged In: YES user_id=33168 Raymond any comments? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1443159&group_id=5470 From noreply at sourceforge.net Thu Feb 8 10:46:01 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 08 Feb 2007 01:46:01 -0800 Subject: [Patches] [ python-Patches-1654974 ] sys.excepthook shows relevant bindings. Message-ID: Patches item #1654974, was opened at 2007-02-08 09:46 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1654974&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nefarious CodeMonkey, Jr. (nejucomo) Assigned to: Nobody/Anonymous (nobody) Summary: sys.excepthook shows relevant bindings. Initial Comment: The attached module provides a suitable replacement for sys.excepthook to handle uncaught exceptions. The output is the same, except after each source line shown, a list of relevant bindings is shown. Here's a quick example from the tail end of a test: File "./test-exprann.py", line 16, in f assert c == 12 # With bindings: # c = 42 AssertionError The bindings shown are the intersection of the code object "co_names" and the non-keyword name tokens in the parsed source line. The goal is to only show bindings relevant to the exception. I hope the utility of this is self-evident to any programmer. I tried testing it with crazy expressions containing nonfree bindings (such as lambda's and list comprehensions), so I think it behaves well in most circumstances. The performance might be bad (it parses each line in the backtrace), but I figure it is worth it for uncaught exceptions. Let me know if you find this tool useful. Thanks, Nejucomo ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1654974&group_id=5470 From noreply at sourceforge.net Thu Feb 8 10:50:36 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 08 Feb 2007 01:50:36 -0800 Subject: [Patches] [ python-Patches-1654974 ] sys.excepthook shows relevant bindings. Message-ID: Patches item #1654974, was opened at 2007-02-08 09:46 Message generated for change (Comment added) made by nejucomo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1654974&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nefarious CodeMonkey, Jr. (nejucomo) Assigned to: Nobody/Anonymous (nobody) Summary: sys.excepthook shows relevant bindings. Initial Comment: The attached module provides a suitable replacement for sys.excepthook to handle uncaught exceptions. The output is the same, except after each source line shown, a list of relevant bindings is shown. Here's a quick example from the tail end of a test: File "./test-exprann.py", line 16, in f assert c == 12 # With bindings: # c = 42 AssertionError The bindings shown are the intersection of the code object "co_names" and the non-keyword name tokens in the parsed source line. The goal is to only show bindings relevant to the exception. I hope the utility of this is self-evident to any programmer. I tried testing it with crazy expressions containing nonfree bindings (such as lambda's and list comprehensions), so I think it behaves well in most circumstances. The performance might be bad (it parses each line in the backtrace), but I figure it is worth it for uncaught exceptions. Let me know if you find this tool useful. Thanks, Nejucomo ---------------------------------------------------------------------- >Comment By: Nefarious CodeMonkey, Jr. (nejucomo) Date: 2007-02-08 09:50 Message: Logged In: YES user_id=44911 Originator: YES BTW, I only tested this on python 2.4. Because it examines the parse tree (very simply), it may not work if the AST format changes. The only parsing it does is search for all NAME tokens which are not keywords, so I assumed this was stable across releases. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1654974&group_id=5470 From vtwxqf at centralcomunicacao.com.br Thu Feb 8 12:35:34 2007 From: vtwxqf at centralcomunicacao.com.br (Anthony) Date: Thu, 8 Feb 2007 12:35:34 +0100 Subject: [Patches] Arise Sir Pete Shelley! Message-ID: <000f01c74b76$6ef03a90$6d76c190@hzfr> At 24, I found the pickings were pretty slim; there just aren't that many girls in my age group on these sites. There's not much to say about Amen really - they were okay, the drummer gave it loads, and they sounded more punky and less thrashy than I expected but in my opinion, they lacked catchy tunes. While it may seem strange for Yahoo! If you want something fancier, then I'd second Brad's recommendation of perlbal. There were four bands on offer for a tenner and it don't get much better than that. Or at least I did and then forgot. Please contact your system administrator to report this fault. I'm envious of the fact that you earned your private in a tailwheel aircraft - the hour and a half or so of Champ and Decathlon time I have was incredibly fun. This is the small downside of building community. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20070208/9ac42117/attachment-0001.html -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 20054 bytes Desc: not available Url : http://mail.python.org/pipermail/patches/attachments/20070208/9ac42117/attachment-0001.gif From noreply at sourceforge.net Thu Feb 8 19:41:25 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 08 Feb 2007 10:41:25 -0800 Subject: [Patches] [ python-Patches-1623563 ] Allow __class __ assignment for classes with __slots__ Message-ID: Patches item #1623563, was opened at 2006-12-28 06:48 Message generated for change (Comment added) made by jimjjewett You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1623563&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: TH (therve) Assigned to: Nobody/Anonymous (nobody) Summary: Allow __class __ assignment for classes with __slots__ Initial Comment: I made a modification in typeobject.c to allow __class__ modification for classes with slots. It's basically a change in same_slots_added to count the offset of the slots and check if the names of the slots are the same (in the naive way). I don't check if slots are in a different order, that may be an improve. The patch is against trunk, with some tests. It's my first submission on Python, so every feedback will be welcome :). ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2007-02-08 13:41 Message: Logged In: YES user_id=764593 Originator: NO Review five other patches. Post the review summaries (and tracker numbers) to the python-dev mailing list. In that same message, ask someone with commit privileges to do the 5:1 deal, pointing at this tracker number. ---------------------------------------------------------------------- Comment By: TH (therve) Date: 2007-02-06 08:43 Message: Logged In: YES user_id=1038797 Originator: YES Is there anything I can do to have a resolution on this ? Thanks. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-12-30 19:26 Message: Logged In: YES user_id=764593 Originator: NO Just going through the list doesn't seem so naive to me. If the slots are in a different order, then you would need to move the data around -- which borders on "maybe they ought to have written the translator themselves" ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1623563&group_id=5470 From noreply at sourceforge.net Fri Feb 9 00:00:21 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 08 Feb 2007 15:00:21 -0800 Subject: [Patches] [ python-Patches-1621265 ] Auto-completion list placement Message-ID: Patches item #1621265, was opened at 2006-12-23 08:16 Message generated for change (Comment added) made by kbk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1621265&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: Python 2.6 Status: Closed Resolution: Accepted Priority: 5 Private: No Submitted By: Tal Einat (taleinat) Assigned to: Kurt B. Kaiser (kbk) Summary: Auto-completion list placement Initial Comment: The completion list placement logic is currently very primitive. Sometimes the completion list extends outside of the editor window, or even outside the screen so that part of it is not visible. This patch places the completion list inside the text widget if possible, above or below the typed text, as appropriate. It will not hide the currently typed text in any case. End-cases: * If the text widget is not high enough, the completion list is placed below the currently typed text. * If the text widget is not wide enough, the completion list is aligned with the left-most edge of the text widget. In any case the size of the completion list is not altered to fit the text widget, though this would probably be the best solution when the text widget is too small. This means that with this patch, the completion list may extend beyond the text widget if it is small. I have not implemented resizing of the list this since the list's size is currently not set at all - Tk's default is used. Changing it anywhere in the code would require setting it elsewhere, hard-coding a default or making it configurable, etc. Comments please! ---------------------------------------------------------------------- >Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-08 18:00 Message: Logged In: YES user_id=149084 Originator: NO Well, I've checked something in, see NEWS.txt. Take a look; I think it should be backported. ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2007-02-07 02:26 Message: Logged In: YES user_id=1330769 Originator: YES My only comment is that I've already implemented precisely the logic you've described. The changes required are mostly removal of code... I could cough up a patch, sometime about the weekend. (That way we could also move the conversation from here :) ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-06 22:47 Message: Logged In: YES user_id=149084 Originator: NO Rev 53653 Thanks for the patch! I agree with your comment about the completion logic. I would have thought that by now someone from Noam's enthusiast group would have sent a patch. If there's only one completion, a tab should complete it. If there is a mouse selection in the ACW, it should be chosen. If an up/down cursor key is used to change the ACW selection, the selection should be chosen. Otherwise, the closest match in the ACW window to what's been typed should be highlighted. I'm looking at it. Any comments on this? The change to the completion logic prevented a loop, but as you say, the logic was buggy to begin with. ---------------------------------------------------------------------- Comment By: Tal Einat (taleinat) Date: 2007-02-05 07:03 Message: Logged In: YES user_id=1330769 Originator: YES My mistake - I missed the changes at the top! I meant only to patch the placement mechanism. The newly attached patch includes only the fix for window placement. BTW, that change to the completion logic seems buggy - if a duplicate completion entry is selected, only one more letter (if available) will be added? IMO the current completion logic is weird and non-intuitive. When selecting a completion from the list (in any way) I would expect it all to appear where I am typing, not just some of it... File Added: AutoCompleteWindow.placement.070205.patch ---------------------------------------------------------------------- Comment By: Kurt B. Kaiser (kbk) Date: 2007-02-04 23:16 Message: Logged In: YES user_id=149084 Originator: NO Haven't you introduced a regression on [ 1571112 ] simple moves freeze IDLE by the changes at the beginning of the patch? Also, the assignment to previous_completion was intended to make the code clearer. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1621265&group_id=5470 From noreply at sourceforge.net Sat Feb 10 19:02:19 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 10 Feb 2007 10:02:19 -0800 Subject: [Patches] [ python-Patches-1607149 ] bug# 1607041: Condition.wait timeout fails on clock change Message-ID: Patches item #1607149, was opened at 2006-12-01 14:32 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607149&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Private: No Submitted By: BC (hashstat) Assigned to: Nobody/Anonymous (nobody) Summary: bug# 1607041: Condition.wait timeout fails on clock change Initial Comment: This patch if for bug# 1607041. If the system clock is adjusted after Condition.wait is called with a timeout, the timeout does not expire as expected. This appears to be due to the threading.Condition class using the system clock to calculate the timeout expiration without taking system clock changes into account. No matter what timeout is used, setting the system clock ahead reduces or eliminates the wait while setting the system clock back increases the wait. So if the clock is set back one hour in the middle of a 1 microsecond wait (c.wait(1)), wait will return in an hour and 1 microsecond rather than after 1 microsecond. This patch modifies the Condition classes wait method to check for variations in the clock between calls to sleep and ajust for abnormalities. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-10 10:02 Message: Logged In: YES user_id=341410 Originator: NO Really, it would be better if time.sleep was based off of a monotonic clock, then everything could use the single working API, and we wouldn't need to monkey-patch everything that uses sleep. Unfortunately, due to the insanity that is the underlying Windows platform, the bios clock is set to local time and is converted to UTC (which is then used natively by NT). There is a work-around involving setting the system clock to UTC and setting a registry setting, but it doesn't always work, can cause 100% CPU usage during DST changes, and doesn't propagate time changes from NT to the underlying clock. http://www.cl.cam.ac.uk/~mgk25/mswish/ut-rtc.html An alternate (or additional option) is to just disable DST changes, manually adjust times as necessary (or using a script that runs only at boot), and never run any software while adjusting the time. http://www.pcreview.co.uk/forums/thread-531048-2.php Considering the recent changes to DST, the latter link is probably more applicable. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-02 23:11 Message: Logged In: YES user_id=21627 Originator: NO It would be better if this was based on a monotonic clock if available on the system (such as the POSIX CLOCK_MONOTONIC argument to clock_gettime). "Guessing" jumps in the time is inherently unreliable; in your code, you won't notice changes that involve less than 10s. Notice that the same problem exists for Thread.join; it would be good if they were fixed together. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607149&group_id=5470 From noreply at sourceforge.net Sat Feb 10 19:57:33 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 10 Feb 2007 10:57:33 -0800 Subject: [Patches] [ python-Patches-723312 ] ability to pass a timeout to underlying socket Message-ID: Patches item #723312, was opened at 2003-04-17 14:03 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=723312&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) >Group: Python 2.6 Status: Open >Resolution: None >Priority: 5 Private: No Submitted By: Matthew Russell (mattruss) Assigned to: Skip Montanaro (montanaro) Summary: ability to pass a timeout to underlying socket Initial Comment: this patch superceeds an earlier one i posted (#714592) - a bit *too* enthusiastic am afraid (sorry!) classes in modules such as httplib, ftpplib do not currently allow an easy way to take advantage of the new socket timeout feature in 2.3. This patch provides that abilty though one additonal class in socket.py ( socket.NetworkConnection ) and accompying test in test_socket.py (basic though the test is :-s ) As an extra benifit, the patch removes duplicate code, as each connect method in the main class of most modules (FTP, HTTPConnection, Telnet, POP3 etc) are copies of each other. The modules that use sockets are: * ftplib * httplib * telnetlib * poplib * urllib * imaplib * nntplib * xmlrpclib Of these I have only been able to easily refactor NetworkConnection into httplib, ftplib, telnetllib, poplib and smtplib. I did look to see if there were any unittests for theese modules in .Lib/test but found none (? - I appologise if there are some, i am new to the library tests) I did however check that the test() [like] methods at the bottom of each of the afore mentioned modules worked. thanks for your advice again Skip :o) Matt ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2007-02-10 12:57 Message: Logged In: YES user_id=44345 Originator: NO Here's an updated patch. Tests pass. There's no documentation for socket.NetworkConnection or for the API changes to smtplib, httplib, telnetlib, poplib or ftplib. File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-04-14 11:09 Message: Logged In: YES user_id=21627 mattruss, are you interested in forward-porting it to 2.5? If that gets done before the beta release, please assign it to me for review. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2005-09-30 22:39 Message: Logged In: YES user_id=29957 Just a quick note - this patch might be accepted for 2.5, but it's not suitable for 2.4.x - once a major release is done, subsequent minor releases (2.3.x, 2.4.x) are bugfix-only. No new features are acceptable. For more on this, see http://www.python.org/peps/pep-0006.html ---------------------------------------------------------------------- Comment By: zeidler (zeidler) Date: 2005-09-28 09:22 Message: Logged In: YES user_id=1082576 perhaps to add a little emphasis: i'd really like to see that patch included into one of the next releases too. i was just going to submit my own, much much simpler version of adding a socket timeout in httplib.py, but matt's patch is a much better way to do this... we're using xml-rpc a lot over here, and the default timeouts are just way too long to keep up a decent user interaction in case of server problems. so, we'd really appreciate this becoming part of the distro! ;) thanks & good job matt, andi ---------------------------------------------------------------------- Comment By: Gr?goire Weber (gregweb) Date: 2005-01-12 07:52 Message: Logged In: YES user_id=812601 Just lobbying: It would be nice to have that on board in futire python versions. Would this be possible in python 2.4.1? I need timeouts for xmlrpclib and subclassed xmlrpclib.Transport for my needs. But nevertheless it would be nice to have that in python 2.4.1. Thanks! Gregoire ---------------------------------------------------------------------- Comment By: Matthew Russell (mattruss) Date: 2003-04-20 18:20 Message: Logged In: YES user_id=737261 Errata: i forgot to mention urlllib2 as modules affected. the line above starting "The modules that use sockets are: " should really read: "classes in the modules stated below would benifit from the ability pass timeouts:" ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=723312&group_id=5470 From noreply at sourceforge.net Sat Feb 10 20:26:17 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 10 Feb 2007 11:26:17 -0800 Subject: [Patches] [ python-Patches-1607149 ] bug# 1607041: Condition.wait timeout fails on clock change Message-ID: Patches item #1607149, was opened at 2006-12-01 23:32 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607149&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Private: No Submitted By: BC (hashstat) Assigned to: Nobody/Anonymous (nobody) Summary: bug# 1607041: Condition.wait timeout fails on clock change Initial Comment: This patch if for bug# 1607041. If the system clock is adjusted after Condition.wait is called with a timeout, the timeout does not expire as expected. This appears to be due to the threading.Condition class using the system clock to calculate the timeout expiration without taking system clock changes into account. No matter what timeout is used, setting the system clock ahead reduces or eliminates the wait while setting the system clock back increases the wait. So if the clock is set back one hour in the middle of a 1 microsecond wait (c.wait(1)), wait will return in an hour and 1 microsecond rather than after 1 microsecond. This patch modifies the Condition classes wait method to check for variations in the clock between calls to sleep and ajust for abnormalities. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-10 20:26 Message: Logged In: YES user_id=21627 Originator: NO DST is irrelevant for this discussion. On WIndows NT+, the system clock (as returned by libc ftime/windows GetSystemTime) is always in UTC (broken down to a Gregorian date). It is thus not affected by DST changes. What does matter for this bug is administrator changes to the system time (like manual changes), and time adjustments due to NTP (although the Windows NTP implementation will perform a non-monototic change only during boot, AFAIK). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-10 19:02 Message: Logged In: YES user_id=341410 Originator: NO Really, it would be better if time.sleep was based off of a monotonic clock, then everything could use the single working API, and we wouldn't need to monkey-patch everything that uses sleep. Unfortunately, due to the insanity that is the underlying Windows platform, the bios clock is set to local time and is converted to UTC (which is then used natively by NT). There is a work-around involving setting the system clock to UTC and setting a registry setting, but it doesn't always work, can cause 100% CPU usage during DST changes, and doesn't propagate time changes from NT to the underlying clock. http://www.cl.cam.ac.uk/~mgk25/mswish/ut-rtc.html An alternate (or additional option) is to just disable DST changes, manually adjust times as necessary (or using a script that runs only at boot), and never run any software while adjusting the time. http://www.pcreview.co.uk/forums/thread-531048-2.php Considering the recent changes to DST, the latter link is probably more applicable. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-03 08:11 Message: Logged In: YES user_id=21627 Originator: NO It would be better if this was based on a monotonic clock if available on the system (such as the POSIX CLOCK_MONOTONIC argument to clock_gettime). "Guessing" jumps in the time is inherently unreliable; in your code, you won't notice changes that involve less than 10s. Notice that the same problem exists for Thread.join; it would be good if they were fixed together. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607149&group_id=5470 From noreply at sourceforge.net Sat Feb 10 20:56:45 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 10 Feb 2007 11:56:45 -0800 Subject: [Patches] [ python-Patches-1615158 ] POSIX capabilities support Message-ID: Patches item #1615158, was opened at 2006-12-13 19:10 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1615158&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Matt Kern (gj0aqzda) Assigned to: Nobody/Anonymous (nobody) Summary: POSIX capabilities support Initial Comment: Attached is a patch which adds POSIX capabilities support. The following API functions are supported: * cap_clear * cap_copy_ext * cap_dup * cap_from_text * cap_get_flag * cap_get_proc * cap_init * cap_set_flag * cap_set_proc * cap_size * cap_to_text The following API function is supported, but is broken with certain versions of libcap (I am running debian testing's libcap1, version 1.10-14, which has an issue; I have reported this upstream): * cap_copy_int The following API functions are in there as stubs, but currently are not compiled. I need access to a machine to test these. I will probably add autoconf tests for availability of these functions in due course: * cap_get_fd * cap_get_file * cap_set_fd * cap_set_file The patch includes diffs to configure. My autoconf is however at a different revision to that used on the python trunk. You may want to re-autoconf configure.in. I've added a few API tests to test_posix.py. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-10 20:56 Message: Logged In: YES user_id=21627 Originator: NO I don't mind the POSIX module getting bigger. In C, these functions are all in a flat namespace, also. I like the view "if it's specified by POSIX, you find it in the POSIX module" (although this specific API was rejected for inclusion into POSIX). The functions are all very small, as the real functionality is in the C library, or even the OS kernel. As for the ifdefery: most of it is straight-forward: functionality is either present or it isn't. It gets messy when it tries to use alternative underlying APIs, e.g. for Win32 or OS/2. If the code is to be refactored, this should be the way to go (i.e. move all Win32 and OS/2 implementations out of the module) As for PyMem_Malloc: I see no need to use that API; it doesn't improve the code to do so, compared to malloc/free. All that matters it is symmetric. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2007-01-30 05:52 Message: Logged In: YES user_id=33168 Originator: NO ISTM that this would be better as a separate module or an optional submodule to posix. The posix module is already 8720 lines. I really don't want it to get bigger, especially when you realize how much #ifdef'ery is in there. Some other things I noticed: You should use PyMem_Malloc instead of a raw malloc (same deal with free). Methods that take no arguments should use METH_NOARGS and then there's no need to call PyArgs_ParseTuple (e.g., posix_cap_get_proc). There definitely shouldn't be any abort()s in there, even if #ifdef'ed out. Is this 64-bit safe? My manpage (gentoo) says this: int cap_set_flag(cap_t cap_p, cap_flag_t flag, int ncap, cap_value_t *caps, cap_flag_value_t value); I see that you are using ints. I don't know if that's correct on a 64-bit platform. If not, you will need to modify the places that ints are used to take longs. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-01-29 20:30 Message: Logged In: YES user_id=21627 Originator: NO The patch cannot go in in its current form (I started applying it, but then found that I just can't do it). It contains conditional, commented out code. Either the code is correct, then it should be added, or it is incorrect, in which case it should be removed entirely. There shouldn't be any work-in-progress code in the Python repository whatsoever. This refers to both the if 0 blocks (which I thought I can safely delete), as well as commented-out entries in CapabilityStateMethods (for which I didn't know what to do). So while you are revising it, I have a few remarks: - you can safely omit the generated configure changes from the patch - I will regenerate them, anyway. - please follow the alphabet in the header files in configure.in (bsdtty.h < capabilities.h) - please don't expose method on objects on which they aren't methods. E.g. cap_clear is available both as a method and a module-level function; that can't be both right (there should be one way to do it) Following the socket API, I think offering these as methods is reasonable - try avoiding the extra copy in copy_ext (copying directly into the string). If you keep malloc calls, don't return NULL without setting a Python exception. - use the "s" format for copy_int and from_text - consider using booleans for [gs]et_flags ---------------------------------------------------------------------- Comment By: Matt Kern (gj0aqzda) Date: 2007-01-29 17:57 Message: Logged In: YES user_id=1667774 Originator: YES No news on these patches in a while. To summarise, the patches are ready to go in. The issues surrounding cap_copy_int(), cap_get_*() and cap_set_*() are pretty minor. The vast majority of uses will be of the cap_get_proc(), cap_set_flag(), cap_set_proc() variety. I am not trying to hassle you; I know you don't have enough time to get through everything. However, I'll hang fire on future development of stuff that I, personally, am not going to use, until I know when/if these patches are going to go in. ---------------------------------------------------------------------- Comment By: Matt Kern (gj0aqzda) Date: 2006-12-19 11:48 Message: Logged In: YES user_id=1667774 Originator: YES I've attached a documentation patch, which should be applied in addition to the base patch. File Added: patch-svn-doc.diff ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2006-12-16 14:25 Message: Logged In: YES user_id=849994 Originator: NO (If you don't want to write LaTeX, it's enough to write the docs in plaintext, there are a few volunteers who will convert it appropriately.) ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-16 13:28 Message: Logged In: YES user_id=21627 Originator: NO Can you please provide documentation changes as well? ---------------------------------------------------------------------- Comment By: Matt Kern (gj0aqzda) Date: 2006-12-13 19:12 Message: Logged In: YES user_id=1667774 Originator: YES I should further add that I have implemented the following API calls as methods of the new CapabilityState object in addition to the standard functions: * cap_clear * cap_copy_ext * cap_dup * cap_get_flag * cap_set_flag * cap_set_proc * cap_size * cap_to_text ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1615158&group_id=5470 From noreply at sourceforge.net Sat Feb 10 21:42:39 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 10 Feb 2007 12:42:39 -0800 Subject: [Patches] [ python-Patches-1607149 ] bug# 1607041: Condition.wait timeout fails on clock change Message-ID: Patches item #1607149, was opened at 2006-12-01 14:32 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607149&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Private: No Submitted By: BC (hashstat) Assigned to: Nobody/Anonymous (nobody) Summary: bug# 1607041: Condition.wait timeout fails on clock change Initial Comment: This patch if for bug# 1607041. If the system clock is adjusted after Condition.wait is called with a timeout, the timeout does not expire as expected. This appears to be due to the threading.Condition class using the system clock to calculate the timeout expiration without taking system clock changes into account. No matter what timeout is used, setting the system clock ahead reduces or eliminates the wait while setting the system clock back increases the wait. So if the clock is set back one hour in the middle of a 1 microsecond wait (c.wait(1)), wait will return in an hour and 1 microsecond rather than after 1 microsecond. This patch modifies the Condition classes wait method to check for variations in the clock between calls to sleep and ajust for abnormalities. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-10 12:42 Message: Logged In: YES user_id=341410 Originator: NO Hrm. I know I've had issues with file modification times "changing" during a DST change on Windows in a Python process, but that may be due to Python only checking the timezone at startup. What you are saying sounds suspiciously like, "don't change the system clock when your program is running", which seems reasonable to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-10 11:26 Message: Logged In: YES user_id=21627 Originator: NO DST is irrelevant for this discussion. On WIndows NT+, the system clock (as returned by libc ftime/windows GetSystemTime) is always in UTC (broken down to a Gregorian date). It is thus not affected by DST changes. What does matter for this bug is administrator changes to the system time (like manual changes), and time adjustments due to NTP (although the Windows NTP implementation will perform a non-monototic change only during boot, AFAIK). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-10 10:02 Message: Logged In: YES user_id=341410 Originator: NO Really, it would be better if time.sleep was based off of a monotonic clock, then everything could use the single working API, and we wouldn't need to monkey-patch everything that uses sleep. Unfortunately, due to the insanity that is the underlying Windows platform, the bios clock is set to local time and is converted to UTC (which is then used natively by NT). There is a work-around involving setting the system clock to UTC and setting a registry setting, but it doesn't always work, can cause 100% CPU usage during DST changes, and doesn't propagate time changes from NT to the underlying clock. http://www.cl.cam.ac.uk/~mgk25/mswish/ut-rtc.html An alternate (or additional option) is to just disable DST changes, manually adjust times as necessary (or using a script that runs only at boot), and never run any software while adjusting the time. http://www.pcreview.co.uk/forums/thread-531048-2.php Considering the recent changes to DST, the latter link is probably more applicable. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-02 23:11 Message: Logged In: YES user_id=21627 Originator: NO It would be better if this was based on a monotonic clock if available on the system (such as the POSIX CLOCK_MONOTONIC argument to clock_gettime). "Guessing" jumps in the time is inherently unreliable; in your code, you won't notice changes that involve less than 10s. Notice that the same problem exists for Thread.join; it would be good if they were fixed together. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607149&group_id=5470 From noreply at sourceforge.net Sun Feb 11 10:03:34 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 01:03:34 -0800 Subject: [Patches] [ python-Patches-1607149 ] bug# 1607041: Condition.wait timeout fails on clock change Message-ID: Patches item #1607149, was opened at 2006-12-01 23:32 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607149&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Private: No Submitted By: BC (hashstat) Assigned to: Nobody/Anonymous (nobody) Summary: bug# 1607041: Condition.wait timeout fails on clock change Initial Comment: This patch if for bug# 1607041. If the system clock is adjusted after Condition.wait is called with a timeout, the timeout does not expire as expected. This appears to be due to the threading.Condition class using the system clock to calculate the timeout expiration without taking system clock changes into account. No matter what timeout is used, setting the system clock ahead reduces or eliminates the wait while setting the system clock back increases the wait. So if the clock is set back one hour in the middle of a 1 microsecond wait (c.wait(1)), wait will return in an hour and 1 microsecond rather than after 1 microsecond. This patch modifies the Condition classes wait method to check for variations in the clock between calls to sleep and ajust for abnormalities. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-11 10:03 Message: Logged In: YES user_id=21627 Originator: NO Indeed, you would have issues with file stamps changing on Windows NT+ if the file system is FAT. However, this is not because the system time moves forth and back, but because FAT time stamps get reinterpreted according to the current time zone, not according to the time zone in force when the file was created (as NT tries to convert the FAT time stamps to UTC). On NTFS, this is not an issue, as NTFS file stamps are in UTC already (units of 100ns since Jan 1, 1601). So indeed, the submitter complains that Condition.wait doesn't time out correctly if the system clock is changed while the wait occurs. I think it is reasonable to make this complaint, however, I don't like to the patch proposed, and I do think this can't really be fixed without the help of the operating system (so I wouldn't mind if it was unfixed on systems that don't help). The point is that .wait() really takes a duration, not a time stamp, for timeout, and we should honor that. The duration should expire after the given amount of physical time has passed, independent of the system clock. As I said, one way to implement this would be to use a monotonic clock if the system has one. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-10 21:42 Message: Logged In: YES user_id=341410 Originator: NO Hrm. I know I've had issues with file modification times "changing" during a DST change on Windows in a Python process, but that may be due to Python only checking the timezone at startup. What you are saying sounds suspiciously like, "don't change the system clock when your program is running", which seems reasonable to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-10 20:26 Message: Logged In: YES user_id=21627 Originator: NO DST is irrelevant for this discussion. On WIndows NT+, the system clock (as returned by libc ftime/windows GetSystemTime) is always in UTC (broken down to a Gregorian date). It is thus not affected by DST changes. What does matter for this bug is administrator changes to the system time (like manual changes), and time adjustments due to NTP (although the Windows NTP implementation will perform a non-monototic change only during boot, AFAIK). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-10 19:02 Message: Logged In: YES user_id=341410 Originator: NO Really, it would be better if time.sleep was based off of a monotonic clock, then everything could use the single working API, and we wouldn't need to monkey-patch everything that uses sleep. Unfortunately, due to the insanity that is the underlying Windows platform, the bios clock is set to local time and is converted to UTC (which is then used natively by NT). There is a work-around involving setting the system clock to UTC and setting a registry setting, but it doesn't always work, can cause 100% CPU usage during DST changes, and doesn't propagate time changes from NT to the underlying clock. http://www.cl.cam.ac.uk/~mgk25/mswish/ut-rtc.html An alternate (or additional option) is to just disable DST changes, manually adjust times as necessary (or using a script that runs only at boot), and never run any software while adjusting the time. http://www.pcreview.co.uk/forums/thread-531048-2.php Considering the recent changes to DST, the latter link is probably more applicable. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-03 08:11 Message: Logged In: YES user_id=21627 Originator: NO It would be better if this was based on a monotonic clock if available on the system (such as the POSIX CLOCK_MONOTONIC argument to clock_gettime). "Guessing" jumps in the time is inherently unreliable; in your code, you won't notice changes that involve less than 10s. Notice that the same problem exists for Thread.join; it would be good if they were fixed together. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607149&group_id=5470 From noreply at sourceforge.net Sun Feb 11 11:11:25 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 02:11:25 -0800 Subject: [Patches] [ python-Patches-1607149 ] bug# 1607041: Condition.wait timeout fails on clock change Message-ID: Patches item #1607149, was opened at 2006-12-01 14:32 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607149&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Private: No Submitted By: BC (hashstat) Assigned to: Nobody/Anonymous (nobody) Summary: bug# 1607041: Condition.wait timeout fails on clock change Initial Comment: This patch if for bug# 1607041. If the system clock is adjusted after Condition.wait is called with a timeout, the timeout does not expire as expected. This appears to be due to the threading.Condition class using the system clock to calculate the timeout expiration without taking system clock changes into account. No matter what timeout is used, setting the system clock ahead reduces or eliminates the wait while setting the system clock back increases the wait. So if the clock is set back one hour in the middle of a 1 microsecond wait (c.wait(1)), wait will return in an hour and 1 microsecond rather than after 1 microsecond. This patch modifies the Condition classes wait method to check for variations in the clock between calls to sleep and ajust for abnormalities. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-11 02:11 Message: Logged In: YES user_id=341410 Originator: NO I've never run FAT in Windows 2000. While I am pretty sure that the underlying platform doesn't have issues, Python keeps its DST offsets when it first starts up, so if you have a Python process that persists across DST changes, the modification time changes, even if the actual file modification time didn't (quitting and restarting Python "fixes" the issue). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-11 01:03 Message: Logged In: YES user_id=21627 Originator: NO Indeed, you would have issues with file stamps changing on Windows NT+ if the file system is FAT. However, this is not because the system time moves forth and back, but because FAT time stamps get reinterpreted according to the current time zone, not according to the time zone in force when the file was created (as NT tries to convert the FAT time stamps to UTC). On NTFS, this is not an issue, as NTFS file stamps are in UTC already (units of 100ns since Jan 1, 1601). So indeed, the submitter complains that Condition.wait doesn't time out correctly if the system clock is changed while the wait occurs. I think it is reasonable to make this complaint, however, I don't like to the patch proposed, and I do think this can't really be fixed without the help of the operating system (so I wouldn't mind if it was unfixed on systems that don't help). The point is that .wait() really takes a duration, not a time stamp, for timeout, and we should honor that. The duration should expire after the given amount of physical time has passed, independent of the system clock. As I said, one way to implement this would be to use a monotonic clock if the system has one. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-10 12:42 Message: Logged In: YES user_id=341410 Originator: NO Hrm. I know I've had issues with file modification times "changing" during a DST change on Windows in a Python process, but that may be due to Python only checking the timezone at startup. What you are saying sounds suspiciously like, "don't change the system clock when your program is running", which seems reasonable to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-10 11:26 Message: Logged In: YES user_id=21627 Originator: NO DST is irrelevant for this discussion. On WIndows NT+, the system clock (as returned by libc ftime/windows GetSystemTime) is always in UTC (broken down to a Gregorian date). It is thus not affected by DST changes. What does matter for this bug is administrator changes to the system time (like manual changes), and time adjustments due to NTP (although the Windows NTP implementation will perform a non-monototic change only during boot, AFAIK). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-10 10:02 Message: Logged In: YES user_id=341410 Originator: NO Really, it would be better if time.sleep was based off of a monotonic clock, then everything could use the single working API, and we wouldn't need to monkey-patch everything that uses sleep. Unfortunately, due to the insanity that is the underlying Windows platform, the bios clock is set to local time and is converted to UTC (which is then used natively by NT). There is a work-around involving setting the system clock to UTC and setting a registry setting, but it doesn't always work, can cause 100% CPU usage during DST changes, and doesn't propagate time changes from NT to the underlying clock. http://www.cl.cam.ac.uk/~mgk25/mswish/ut-rtc.html An alternate (or additional option) is to just disable DST changes, manually adjust times as necessary (or using a script that runs only at boot), and never run any software while adjusting the time. http://www.pcreview.co.uk/forums/thread-531048-2.php Considering the recent changes to DST, the latter link is probably more applicable. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-02 23:11 Message: Logged In: YES user_id=21627 Originator: NO It would be better if this was based on a monotonic clock if available on the system (such as the POSIX CLOCK_MONOTONIC argument to clock_gettime). "Guessing" jumps in the time is inherently unreliable; in your code, you won't notice changes that involve less than 10s. Notice that the same problem exists for Thread.join; it would be good if they were fixed together. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607149&group_id=5470 From noreply at sourceforge.net Sun Feb 11 13:45:38 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 04:45:38 -0800 Subject: [Patches] [ python-Patches-1657276 ] socketmodule fails to build because missing NETLINK_DNRTMSG Message-ID: Patches item #1657276, was opened at 2007-02-11 14:45 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657276&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Build Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Aleksandr Koltsoff (czr_) Assigned to: Nobody/Anonymous (nobody) Summary: socketmodule fails to build because missing NETLINK_DNRTMSG Initial Comment: While building 2.5 on a debian 2.2 (potato) system I encountered a problem in socketmodule.c. The system in question has netlink.h, but a symbol (NETLINK_DNRTMSG) is not (whole module fails to build because of this). The provided patch (unified diff) fixes this by surrounding the line with an ifdef. The respective code also contains other int constants that probably should be protected, but this patch doesn't touch them. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657276&group_id=5470 From noreply at sourceforge.net Sun Feb 11 13:55:35 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 04:55:35 -0800 Subject: [Patches] [ python-Patches-1607149 ] bug# 1607041: Condition.wait timeout fails on clock change Message-ID: Patches item #1607149, was opened at 2006-12-01 23:32 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607149&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Private: No Submitted By: BC (hashstat) Assigned to: Nobody/Anonymous (nobody) Summary: bug# 1607041: Condition.wait timeout fails on clock change Initial Comment: This patch if for bug# 1607041. If the system clock is adjusted after Condition.wait is called with a timeout, the timeout does not expire as expected. This appears to be due to the threading.Condition class using the system clock to calculate the timeout expiration without taking system clock changes into account. No matter what timeout is used, setting the system clock ahead reduces or eliminates the wait while setting the system clock back increases the wait. So if the clock is set back one hour in the middle of a 1 microsecond wait (c.wait(1)), wait will return in an hour and 1 microsecond rather than after 1 microsecond. This patch modifies the Condition classes wait method to check for variations in the clock between calls to sleep and ajust for abnormalities. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-11 13:55 Message: Logged In: YES user_id=21627 Originator: NO Ah. I fixed the "changing stat result problem" in Python 2.5, by not using the C library's stat implementation anymore. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-11 11:11 Message: Logged In: YES user_id=341410 Originator: NO I've never run FAT in Windows 2000. While I am pretty sure that the underlying platform doesn't have issues, Python keeps its DST offsets when it first starts up, so if you have a Python process that persists across DST changes, the modification time changes, even if the actual file modification time didn't (quitting and restarting Python "fixes" the issue). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-11 10:03 Message: Logged In: YES user_id=21627 Originator: NO Indeed, you would have issues with file stamps changing on Windows NT+ if the file system is FAT. However, this is not because the system time moves forth and back, but because FAT time stamps get reinterpreted according to the current time zone, not according to the time zone in force when the file was created (as NT tries to convert the FAT time stamps to UTC). On NTFS, this is not an issue, as NTFS file stamps are in UTC already (units of 100ns since Jan 1, 1601). So indeed, the submitter complains that Condition.wait doesn't time out correctly if the system clock is changed while the wait occurs. I think it is reasonable to make this complaint, however, I don't like to the patch proposed, and I do think this can't really be fixed without the help of the operating system (so I wouldn't mind if it was unfixed on systems that don't help). The point is that .wait() really takes a duration, not a time stamp, for timeout, and we should honor that. The duration should expire after the given amount of physical time has passed, independent of the system clock. As I said, one way to implement this would be to use a monotonic clock if the system has one. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-10 21:42 Message: Logged In: YES user_id=341410 Originator: NO Hrm. I know I've had issues with file modification times "changing" during a DST change on Windows in a Python process, but that may be due to Python only checking the timezone at startup. What you are saying sounds suspiciously like, "don't change the system clock when your program is running", which seems reasonable to me. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-10 20:26 Message: Logged In: YES user_id=21627 Originator: NO DST is irrelevant for this discussion. On WIndows NT+, the system clock (as returned by libc ftime/windows GetSystemTime) is always in UTC (broken down to a Gregorian date). It is thus not affected by DST changes. What does matter for this bug is administrator changes to the system time (like manual changes), and time adjustments due to NTP (although the Windows NTP implementation will perform a non-monototic change only during boot, AFAIK). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-10 19:02 Message: Logged In: YES user_id=341410 Originator: NO Really, it would be better if time.sleep was based off of a monotonic clock, then everything could use the single working API, and we wouldn't need to monkey-patch everything that uses sleep. Unfortunately, due to the insanity that is the underlying Windows platform, the bios clock is set to local time and is converted to UTC (which is then used natively by NT). There is a work-around involving setting the system clock to UTC and setting a registry setting, but it doesn't always work, can cause 100% CPU usage during DST changes, and doesn't propagate time changes from NT to the underlying clock. http://www.cl.cam.ac.uk/~mgk25/mswish/ut-rtc.html An alternate (or additional option) is to just disable DST changes, manually adjust times as necessary (or using a script that runs only at boot), and never run any software while adjusting the time. http://www.pcreview.co.uk/forums/thread-531048-2.php Considering the recent changes to DST, the latter link is probably more applicable. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-03 08:11 Message: Logged In: YES user_id=21627 Originator: NO It would be better if this was based on a monotonic clock if available on the system (such as the POSIX CLOCK_MONOTONIC argument to clock_gettime). "Guessing" jumps in the time is inherently unreliable; in your code, you won't notice changes that involve less than 10s. Notice that the same problem exists for Thread.join; it would be good if they were fixed together. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607149&group_id=5470 From noreply at sourceforge.net Sun Feb 11 17:24:09 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 08:24:09 -0800 Subject: [Patches] [ python-Patches-723312 ] ability to pass a timeout to underlying socket Message-ID: Patches item #723312, was opened at 2003-04-17 14:03 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=723312&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Matthew Russell (mattruss) Assigned to: Skip Montanaro (montanaro) Summary: ability to pass a timeout to underlying socket Initial Comment: this patch superceeds an earlier one i posted (#714592) - a bit *too* enthusiastic am afraid (sorry!) classes in modules such as httplib, ftpplib do not currently allow an easy way to take advantage of the new socket timeout feature in 2.3. This patch provides that abilty though one additonal class in socket.py ( socket.NetworkConnection ) and accompying test in test_socket.py (basic though the test is :-s ) As an extra benifit, the patch removes duplicate code, as each connect method in the main class of most modules (FTP, HTTPConnection, Telnet, POP3 etc) are copies of each other. The modules that use sockets are: * ftplib * httplib * telnetlib * poplib * urllib * imaplib * nntplib * xmlrpclib Of these I have only been able to easily refactor NetworkConnection into httplib, ftplib, telnetllib, poplib and smtplib. I did look to see if there were any unittests for theese modules in .Lib/test but found none (? - I appologise if there are some, i am new to the library tests) I did however check that the test() [like] methods at the bottom of each of the afore mentioned modules worked. thanks for your advice again Skip :o) Matt ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 10:24 Message: Logged In: YES user_id=44345 Originator: NO New patch with minor tweaks. Added doc changes. File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2007-02-10 12:57 Message: Logged In: YES user_id=44345 Originator: NO Here's an updated patch. Tests pass. There's no documentation for socket.NetworkConnection or for the API changes to smtplib, httplib, telnetlib, poplib or ftplib. File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-04-14 11:09 Message: Logged In: YES user_id=21627 mattruss, are you interested in forward-porting it to 2.5? If that gets done before the beta release, please assign it to me for review. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2005-09-30 22:39 Message: Logged In: YES user_id=29957 Just a quick note - this patch might be accepted for 2.5, but it's not suitable for 2.4.x - once a major release is done, subsequent minor releases (2.3.x, 2.4.x) are bugfix-only. No new features are acceptable. For more on this, see http://www.python.org/peps/pep-0006.html ---------------------------------------------------------------------- Comment By: zeidler (zeidler) Date: 2005-09-28 09:22 Message: Logged In: YES user_id=1082576 perhaps to add a little emphasis: i'd really like to see that patch included into one of the next releases too. i was just going to submit my own, much much simpler version of adding a socket timeout in httplib.py, but matt's patch is a much better way to do this... we're using xml-rpc a lot over here, and the default timeouts are just way too long to keep up a decent user interaction in case of server problems. so, we'd really appreciate this becoming part of the distro! ;) thanks & good job matt, andi ---------------------------------------------------------------------- Comment By: Gr?goire Weber (gregweb) Date: 2005-01-12 07:52 Message: Logged In: YES user_id=812601 Just lobbying: It would be nice to have that on board in futire python versions. Would this be possible in python 2.4.1? I need timeouts for xmlrpclib and subclassed xmlrpclib.Transport for my needs. But nevertheless it would be nice to have that in python 2.4.1. Thanks! Gregoire ---------------------------------------------------------------------- Comment By: Matthew Russell (mattruss) Date: 2003-04-20 18:20 Message: Logged In: YES user_id=737261 Errata: i forgot to mention urlllib2 as modules affected. the line above starting "The modules that use sockets are: " should really read: "classes in the modules stated below would benifit from the ability pass timeouts:" ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=723312&group_id=5470 From noreply at sourceforge.net Sun Feb 11 19:11:45 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 10:11:45 -0800 Subject: [Patches] [ python-Patches-723312 ] ability to pass a timeout to underlying socket Message-ID: Patches item #723312, was opened at 2003-04-17 14:03 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=723312&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Matthew Russell (mattruss) Assigned to: Skip Montanaro (montanaro) Summary: ability to pass a timeout to underlying socket Initial Comment: this patch superceeds an earlier one i posted (#714592) - a bit *too* enthusiastic am afraid (sorry!) classes in modules such as httplib, ftpplib do not currently allow an easy way to take advantage of the new socket timeout feature in 2.3. This patch provides that abilty though one additonal class in socket.py ( socket.NetworkConnection ) and accompying test in test_socket.py (basic though the test is :-s ) As an extra benifit, the patch removes duplicate code, as each connect method in the main class of most modules (FTP, HTTPConnection, Telnet, POP3 etc) are copies of each other. The modules that use sockets are: * ftplib * httplib * telnetlib * poplib * urllib * imaplib * nntplib * xmlrpclib Of these I have only been able to easily refactor NetworkConnection into httplib, ftplib, telnetllib, poplib and smtplib. I did look to see if there were any unittests for theese modules in .Lib/test but found none (? - I appologise if there are some, i am new to the library tests) I did however check that the test() [like] methods at the bottom of each of the afore mentioned modules worked. thanks for your advice again Skip :o) Matt ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 12:11 Message: Logged In: YES user_id=44345 Originator: NO *argh* one minor tweak too many... File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 10:24 Message: Logged In: YES user_id=44345 Originator: NO New patch with minor tweaks. Added doc changes. File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2007-02-10 12:57 Message: Logged In: YES user_id=44345 Originator: NO Here's an updated patch. Tests pass. There's no documentation for socket.NetworkConnection or for the API changes to smtplib, httplib, telnetlib, poplib or ftplib. File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-04-14 11:09 Message: Logged In: YES user_id=21627 mattruss, are you interested in forward-porting it to 2.5? If that gets done before the beta release, please assign it to me for review. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2005-09-30 22:39 Message: Logged In: YES user_id=29957 Just a quick note - this patch might be accepted for 2.5, but it's not suitable for 2.4.x - once a major release is done, subsequent minor releases (2.3.x, 2.4.x) are bugfix-only. No new features are acceptable. For more on this, see http://www.python.org/peps/pep-0006.html ---------------------------------------------------------------------- Comment By: zeidler (zeidler) Date: 2005-09-28 09:22 Message: Logged In: YES user_id=1082576 perhaps to add a little emphasis: i'd really like to see that patch included into one of the next releases too. i was just going to submit my own, much much simpler version of adding a socket timeout in httplib.py, but matt's patch is a much better way to do this... we're using xml-rpc a lot over here, and the default timeouts are just way too long to keep up a decent user interaction in case of server problems. so, we'd really appreciate this becoming part of the distro! ;) thanks & good job matt, andi ---------------------------------------------------------------------- Comment By: Gr?goire Weber (gregweb) Date: 2005-01-12 07:52 Message: Logged In: YES user_id=812601 Just lobbying: It would be nice to have that on board in futire python versions. Would this be possible in python 2.4.1? I need timeouts for xmlrpclib and subclassed xmlrpclib.Transport for my needs. But nevertheless it would be nice to have that in python 2.4.1. Thanks! Gregoire ---------------------------------------------------------------------- Comment By: Matthew Russell (mattruss) Date: 2003-04-20 18:20 Message: Logged In: YES user_id=737261 Errata: i forgot to mention urlllib2 as modules affected. the line above starting "The modules that use sockets are: " should really read: "classes in the modules stated below would benifit from the ability pass timeouts:" ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=723312&group_id=5470 From noreply at sourceforge.net Sun Feb 11 19:14:23 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 10:14:23 -0800 Subject: [Patches] [ python-Patches-723312 ] ability to pass a timeout to underlying socket Message-ID: Patches item #723312, was opened at 2003-04-17 14:03 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=723312&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Matthew Russell (mattruss) Assigned to: Skip Montanaro (montanaro) Summary: ability to pass a timeout to underlying socket Initial Comment: this patch superceeds an earlier one i posted (#714592) - a bit *too* enthusiastic am afraid (sorry!) classes in modules such as httplib, ftpplib do not currently allow an easy way to take advantage of the new socket timeout feature in 2.3. This patch provides that abilty though one additonal class in socket.py ( socket.NetworkConnection ) and accompying test in test_socket.py (basic though the test is :-s ) As an extra benifit, the patch removes duplicate code, as each connect method in the main class of most modules (FTP, HTTPConnection, Telnet, POP3 etc) are copies of each other. The modules that use sockets are: * ftplib * httplib * telnetlib * poplib * urllib * imaplib * nntplib * xmlrpclib Of these I have only been able to easily refactor NetworkConnection into httplib, ftplib, telnetllib, poplib and smtplib. I did look to see if there were any unittests for theese modules in .Lib/test but found none (? - I appologise if there are some, i am new to the library tests) I did however check that the test() [like] methods at the bottom of each of the afore mentioned modules worked. thanks for your advice again Skip :o) Matt ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 12:14 Message: Logged In: YES user_id=44345 Originator: NO Jeez, I can be an idiot sometimes... Working on two patches in the same sandbox. File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 12:11 Message: Logged In: YES user_id=44345 Originator: NO *argh* one minor tweak too many... File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 10:24 Message: Logged In: YES user_id=44345 Originator: NO New patch with minor tweaks. Added doc changes. File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2007-02-10 12:57 Message: Logged In: YES user_id=44345 Originator: NO Here's an updated patch. Tests pass. There's no documentation for socket.NetworkConnection or for the API changes to smtplib, httplib, telnetlib, poplib or ftplib. File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-04-14 11:09 Message: Logged In: YES user_id=21627 mattruss, are you interested in forward-porting it to 2.5? If that gets done before the beta release, please assign it to me for review. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2005-09-30 22:39 Message: Logged In: YES user_id=29957 Just a quick note - this patch might be accepted for 2.5, but it's not suitable for 2.4.x - once a major release is done, subsequent minor releases (2.3.x, 2.4.x) are bugfix-only. No new features are acceptable. For more on this, see http://www.python.org/peps/pep-0006.html ---------------------------------------------------------------------- Comment By: zeidler (zeidler) Date: 2005-09-28 09:22 Message: Logged In: YES user_id=1082576 perhaps to add a little emphasis: i'd really like to see that patch included into one of the next releases too. i was just going to submit my own, much much simpler version of adding a socket timeout in httplib.py, but matt's patch is a much better way to do this... we're using xml-rpc a lot over here, and the default timeouts are just way too long to keep up a decent user interaction in case of server problems. so, we'd really appreciate this becoming part of the distro! ;) thanks & good job matt, andi ---------------------------------------------------------------------- Comment By: Gr?goire Weber (gregweb) Date: 2005-01-12 07:52 Message: Logged In: YES user_id=812601 Just lobbying: It would be nice to have that on board in futire python versions. Would this be possible in python 2.4.1? I need timeouts for xmlrpclib and subclassed xmlrpclib.Transport for my needs. But nevertheless it would be nice to have that in python 2.4.1. Thanks! Gregoire ---------------------------------------------------------------------- Comment By: Matthew Russell (mattruss) Date: 2003-04-20 18:20 Message: Logged In: YES user_id=737261 Errata: i forgot to mention urlllib2 as modules affected. the line above starting "The modules that use sockets are: " should really read: "classes in the modules stated below would benifit from the ability pass timeouts:" ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=723312&group_id=5470 From noreply at sourceforge.net Sun Feb 11 19:38:43 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 10:38:43 -0800 Subject: [Patches] [ python-Patches-1571379 ] make trace.py --ignore-dir work Message-ID: Patches item #1571379, was opened at 2006-10-05 08:10 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1571379&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Clinton Roy (clintonroy) Assigned to: Skip Montanaro (montanaro) Summary: make trace.py --ignore-dir work Initial Comment: The co_filename attribute is only ever the relative filename, where as the __file__ value of the frame globals is absolute, which lets the ignore() stuff do its thing. ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 12:38 Message: Logged In: YES user_id=44345 Originator: NO Checked in on SVN head as rev 53735. Checked in on 2.4 branch as rev 53736. Will also backport to 2.5 once I figure out why svn thinks I have a conflict in Misc/NEWS. ---------------------------------------------------------------------- Comment By: Clinton Roy (clintonroy) Date: 2006-10-08 07:02 Message: Logged In: YES user_id=31446 Hi Skip, This patch only addresses --ignore-dir working, I didn't initally look at the other uses. Now that I have, they're all passed to linecache, which itself looks in $prefix for a matching filename, so it only needs the relative filename to be passed. My patch specifically just targets the filename param passed to ignore.names() so that it can actually work. cheers, ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2006-10-07 05:25 Message: Logged In: YES user_id=44345 Without digging too deeply, can you explain why only this one instance of using code.co_filename needs to be changed? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1571379&group_id=5470 From noreply at sourceforge.net Sun Feb 11 19:42:31 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 10:42:31 -0800 Subject: [Patches] [ python-Patches-1571379 ] make trace.py --ignore-dir work Message-ID: Patches item #1571379, was opened at 2006-10-05 08:10 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1571379&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Clinton Roy (clintonroy) Assigned to: Skip Montanaro (montanaro) Summary: make trace.py --ignore-dir work Initial Comment: The co_filename attribute is only ever the relative filename, where as the __file__ value of the frame globals is absolute, which lets the ignore() stuff do its thing. ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 12:42 Message: Logged In: YES user_id=44345 Originator: NO 2.5 versions is rev 53537. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 12:38 Message: Logged In: YES user_id=44345 Originator: NO Checked in on SVN head as rev 53735. Checked in on 2.4 branch as rev 53736. Will also backport to 2.5 once I figure out why svn thinks I have a conflict in Misc/NEWS. ---------------------------------------------------------------------- Comment By: Clinton Roy (clintonroy) Date: 2006-10-08 07:02 Message: Logged In: YES user_id=31446 Hi Skip, This patch only addresses --ignore-dir working, I didn't initally look at the other uses. Now that I have, they're all passed to linecache, which itself looks in $prefix for a matching filename, so it only needs the relative filename to be passed. My patch specifically just targets the filename param passed to ignore.names() so that it can actually work. cheers, ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2006-10-07 05:25 Message: Logged In: YES user_id=44345 Without digging too deeply, can you explain why only this one instance of using code.co_filename needs to be changed? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1571379&group_id=5470 From noreply at sourceforge.net Sun Feb 11 19:59:09 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 10:59:09 -0800 Subject: [Patches] [ python-Patches-1225769 ] Proposal to implement comment rows in csv module Message-ID: Patches item #1225769, was opened at 2005-06-22 14:48 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1225769&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Iain Haslam (iain_haslam) Assigned to: Skip Montanaro (montanaro) Summary: Proposal to implement comment rows in csv module Initial Comment: Sometimes csv files contain comment rows, for temporarily commenting out data or occasionally for documentation. The current csv module has no built-in ability to skip rows; in order to skip all lines beginning with '#', the programmer writes something like: csv_reader = csv.reader(fp) for row in csv_reader: if row[0][0] != '#': #assuming no blank lines print row I propose adding a "commentchar" parameter to the csv parser, so that the above code could be written (more elegantly, in my opinion): csv_reader = csv.reader(fp, commentchar='#') for row in csv_reader: print row This requires only relatively minor changes to the module, and by defaulting to using no comment character, existing code will behave as before. If you are interested, the patch (diffs against current cvs) required for the second example to run are attached. Note that that implementation adds SKIPPED_RECORD as a pseudonym for START_RECORD, because setting status to START_RECORD after skipping a record would cause a blank record to be returned. Altering that behaviour would cause more changes and the patch would be harder to review. I've also held back on updating tests and documentation to reflect this change, pending any support for it. It shoud be irrelevant, but this has been developed on Debian testing against the cvs head of Python. ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 12:59 Message: Logged In: YES user_id=44345 Originator: NO Sorry, I'm coming back to this after a long hiatus... I'm still not inclined to make this change to the C source. I think a) comments in CSV files are pretty rare and that b) implementing this using a file object wrapper would be more flexible. #!/usr/bin/env python import csv import StringIO class CommentedFile: def __init__(self, f, commentstring="#"): self.f = f self.commentstring = commentstring def next(self): line = self.f.next() while line.startswith(self.commentstring): line = self.f.next() return line def __iter__(self): return self f = StringIO.StringIO('''\ "event","performers","start","end","time" # Rachel Sage "","Rachael Sage","2008-01-03","2008-01-03","8:00pm" # Others "","Tung-N-GRoeVE","2008-01-16","2008-01-16","9:30pm-2:00am" "","Bossa Nova Beatniks","2007-11-11","2007-11-11","11:11pm" "","Special Consensus","2006-10-06","2006-10-06","" ''') for row in csv.reader(CommentedFile(f)): print row The output of the above is as expected: ['event', 'performers', 'start', 'end', 'time'] ['', 'Rachael Sage', '2008-01-03', '2008-01-03', '8:00pm'] ['', 'Tung-N-GRoeVE', '2008-01-16', '2008-01-16', '9:30pm-2:00am'] ['', 'Bossa Nova Beatniks', '2007-11-11', '2007-11-11', '11:11pm'] ['', 'Special Consensus', '2006-10-06', '2006-10-06', ''] This has the added benefit that comment lines aren't restricted to single character comment prefixes. On the downside, comment characters appearing at the beginning of a continuation line would trip this up. In practice, I don't think it would be a significant limitation. In almost all cases I suspect CSV files with embedded comments would be manually created and maintained and aren't likely to contain fields with embedded comments. Skip ---------------------------------------------------------------------- Comment By: Iain Haslam (iain_haslam) Date: 2005-06-30 18:40 Message: Logged In: YES user_id=1301296 Here are the documentation and test diffs. I'm glad to hear of the positive feedback. I couldn't find the csv mailing list (I assume it's not public), so didn't see the discussion of round-tripping, but I agree with the implied conclusion that flagging rows as comments would complicate the interface too much. On a related point, I noticed that the csv documentation is filed under "Internet data handling". This seems a little odd - I'd suggest moving it to "Miscellaneous Services" alongside fileinput and ConfigParser. Thanks, Iain. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2005-06-27 12:25 Message: Logged In: YES user_id=44345 Iain - There was some positive response to your patch from the csv mailing list (most notably from one of the authors of the C extension module). Can you provide diffs for the module documentation and necessary test cases to go along with your patch? Also, addressing the issue that CSV files with comments (probably?) won't round-trip would be a good thing to note in the docs. Skip ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2005-06-26 13:02 Message: Logged In: YES user_id=44345 Something else just occurred to me. What about writing csv files with comments? Also, a tweak to the docs would be in order if this is accepted. ---------------------------------------------------------------------- Comment By: Iain Haslam (iain_haslam) Date: 2005-06-26 12:26 Message: Logged In: YES user_id=1301296 > I'm not inclined to clutter the C code with further > complications. Sorry - I haven't been keeping up with the existing complications! Don't forget that one man's clutter is another man's functionality. It doesn't actually require much of a change to the code, although I was slightly suprised to discover that this module was in C in the first place... Basically, I noticed that the csv module has a bias towards Excel-generated csv files, but most of the time I've come across csv files, they were hand-edited, and I've often seen comment fields as described in the submission. My submission was intended in the "batteries included" spirit (I do understand that you stop short of the kitchen sink), and also seemed in-keeping with the 'skipinitialspace' option within the existing csv module. > Why can't you implement this on an as-needed basis > with a file object wrapper [other options] True, I could do any of those things, but it would be simpler / clearer not to have to. Of course, if you took your argument further, you could cut chunks out of several modules; the argument comes down to whether the benefits outweigh the additional complexity. I was suprised to discover the option wasn't already there, but maybe that's just me. In any case, if your vote goes from your apparent -0 to -1, that's your choice, and you're better placed to make it than I am. Cheers, Iain. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2005-06-25 14:39 Message: Logged In: YES user_id=44345 I'm not inclined to clutter the C code with further complications. Why can't you implement this on an as-needed basis with a file object wrapper, a subclass of the csv.reader class, or just continue to use the example in your submission? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1225769&group_id=5470 From noreply at sourceforge.net Sun Feb 11 20:14:29 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 11:14:29 -0800 Subject: [Patches] [ python-Patches-1374063 ] Broader iterable support for xmlrpclib Message-ID: Patches item #1374063, was opened at 2005-12-05 22:14 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1374063&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open >Resolution: None Priority: 5 Private: No Submitted By: Skip Montanaro (montanaro) Assigned to: Skip Montanaro (montanaro) Summary: Broader iterable support for xmlrpclib Initial Comment: The XML-RPC spec supports a fairly limited set of datatypes. Most languages, Python included, contain many more types than XML-RPC. Some types, such as Python's complex number type, have no reasonable analog in XML-RPC. Others, such as unicode objects and array objects do. This patch allows anything that can be converted to a list but that is not otherwise supported directly by the xmlrpclib module already to be marshalled as an XML-RPC array if the allow_iter parameter to the ServerProxy constructor evaluated to true. This includes sets and arrays. Motivation... 1. Python already overloads the XML-RPC array type with both lists and tuples. This just extends that overloading to other currently unsupported Python types which can be converted to lists. Why should lists and tuples have all the fun? 2. Providing transparent conversion to XML-RPC arrays keeps calling code a bit cleaner. One of the attractions of XML-RPC is that the remote procedure call looks identical to a local call. This is especially true in Python because of /F's excellent little _Method class. Clearly as a programmer I could type: import array a = array.array('i', [1, 2,3]) ... from somemodule import somefunction print somefunction(list(a)) but that reveals details of the implementation of somefunction, namely that it can't handle arrays directly, even though in most respects arrays and lists are interchangeable. Attached is a patch for the xmlrpclib library that implements this feature, including minor doc changes and a new test case. ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 13:14 Message: Logged In: YES user_id=44345 Originator: YES Martin, I'm returning to this after a pause as well... Suppose I have a subclass of dict. When dumping with my proposed patch it will get dumped as a list. If it is to be dumped as some lowest common denominator type, it should be a dict I think. For example: #!/usr/bin/env python class D(dict): pass d = D(x=1, y=2, z=3) print d print list(d) Until we address is case (at least) I'm un-accepting the patch... Skip ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-03 06:32 Message: Logged In: YES user_id=21627 Originator: NO The patch is fine. Notice that it is currently out of date, due to inclusion of patch #1070046. This other patch added generic support for objects having an __dict__ attribute, marshaling them struct elements. This is in real conflict to this patch: an object both supporting a list conversion and having a __dict__ attribute could be marshaled either way. My proposal is to resolve this in favor of this patch: If an object has a list conversion, it's most likely meant to be a list. If you agree to this resolution, please update the code and the documentation, and apply this patch. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2005-12-05 22:23 Message: Logged In: YES user_id=44345 Oh, I forgot my original use case. I was constructing a list of musicians from a larger data structure and used a set to guarantee uniqueness during construction. I didn't really care about element ordering. I either had to convert the set to a list when calling the local function that made the RPC call, or modify the local function to always convert that arg to a list. Both alternatives seemed unattractive to me. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1374063&group_id=5470 From noreply at sourceforge.net Mon Feb 12 00:00:43 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 15:00:43 -0800 Subject: [Patches] [ python-Patches-1657573 ] Add syntax for dynamic attribute access Message-ID: Patches item #1657573, was opened at 2007-02-11 23:00 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657573&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Parser/Compiler Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Ben North (bennorth) Assigned to: Nobody/Anonymous (nobody) Summary: Add syntax for dynamic attribute access Initial Comment: Add syntax to allow easier access to attributes when attribute name known only at run-time. E.g., replace setattr(obj, attr_name, getattr(obj, attr_name) + 1) with obj.(attr_name) += 1 For further discussion, see PEP 363 [PROVISIONAL NUMBER] ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657573&group_id=5470 From noreply at sourceforge.net Mon Feb 12 01:15:46 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 16:15:46 -0800 Subject: [Patches] [ python-Patches-1657613 ] documentation for element interface Message-ID: Patches item #1657613, was opened at 2007-02-12 13:15 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657613&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Tim Mitchell (tim-mitchell) Assigned to: Nobody/Anonymous (nobody) Summary: documentation for element interface Initial Comment: I have added a subsection to 8.13 xml.etree.ElementTree documenting the _ElementInterface class which objects returned by the Element and SubElement factory functions emulate. I do not have latex installed on my system so I have not been able to check for compiling errors - sorry. (I don't intend to become a regular contributor to go through the bother of setting it up.) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657613&group_id=5470 From noreply at sourceforge.net Mon Feb 12 01:16:17 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 16:16:17 -0800 Subject: [Patches] [ python-Patches-1657613 ] documentation for element interface Message-ID: Patches item #1657613, was opened at 2007-02-12 13:15 Message generated for change (Settings changed) made by tim-mitchell You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657613&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Tim Mitchell (tim-mitchell) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: documentation for element interface Initial Comment: I have added a subsection to 8.13 xml.etree.ElementTree documenting the _ElementInterface class which objects returned by the Element and SubElement factory functions emulate. I do not have latex installed on my system so I have not been able to check for compiling errors - sorry. (I don't intend to become a regular contributor to go through the bother of setting it up.) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657613&group_id=5470 From noreply at sourceforge.net Mon Feb 12 08:51:06 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 11 Feb 2007 23:51:06 -0800 Subject: [Patches] [ python-Patches-728815 ] test_timeout updates Message-ID: Patches item #728815, was opened at 2003-04-28 05:21 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=728815&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Tests Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Dmitry Vasiliev (hdima) >Assigned to: Nobody/Anonymous (nobody) Summary: test_timeout updates Initial Comment: Changes: - code refactoring; - now we catch only exceptions with proper errno codes (also see bug #708927); - addr_remote changed, ('www.google.com', 80) not always works for me since we use transparent proxy; - attempt to implement testSend(), testSendto(), testSendall(); ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-04-14 11:10 Message: Logged In: YES user_id=21627 rhettinger: would you like to review it now? If not, please unassign. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-30 09:52 Message: Logged In: YES user_id=80475 I'll give it more review. Since Py2.3b2 is out, I'm marking this as a change for Py2.4 and we can apply it in early August. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-30 09:52 Message: Logged In: YES user_id=80475 I'll give it more review. Since Py2.3b2 is out, I'm marking this as a change for Py2.4 and we can apply it in early August. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-30 09:52 Message: Logged In: YES user_id=80475 I'll give it more review. Since Py2.3b2 is out, I'm marking this as a change for Py2.4 and we can apply it in early August. ---------------------------------------------------------------------- Comment By: Walter D?rwald (doerwalter) Date: 2003-06-30 07:37 Message: Logged In: YES user_id=89016 I'm no expert in socket programming, so the final decision whether this is OK is your's. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-06-28 09:09 Message: Logged In: YES user_id=80475 Walter, are you interested in working this patch? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=728815&group_id=5470 From noreply at sourceforge.net Mon Feb 12 10:45:58 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 12 Feb 2007 01:45:58 -0800 Subject: [Patches] [ python-Patches-1647484 ] gzip.GzipFile has no name attribute Message-ID: Patches item #1647484, was opened at 2007-01-29 23:25 Message generated for change (Comment added) made by gustaebel You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1647484&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Lars Gust?bel (gustaebel) Assigned to: Nobody/Anonymous (nobody) Summary: gzip.GzipFile has no name attribute Initial Comment: The gzip.GzipFile object uses a filename instead of a name attribute. This deviates from the standard practice and the interface described in "3.9 File Objects" and seems unnecessary. Attached patch changes this but still leaves the filename attribute as a property that emits a DeprecationWarning. ---------------------------------------------------------------------- >Comment By: Lars Gust?bel (gustaebel) Date: 2007-02-12 10:45 Message: Logged In: YES user_id=642936 Originator: YES Seems that nobody is interested. I will check this in in a few days if nobody objects. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1647484&group_id=5470 From noreply at sourceforge.net Mon Feb 12 13:22:44 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 12 Feb 2007 04:22:44 -0800 Subject: [Patches] [ python-Patches-1463026 ] Bugfix for #847665 (XMLGenerator dies in namespace mode) Message-ID: Patches item #1463026, was opened at 2006-04-02 14:53 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1463026&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: XML Group: Python 2.4 >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Nikolai Grigoriev (ngrig) Assigned to: Nobody/Anonymous (nobody) Summary: Bugfix for #847665 (XMLGenerator dies in namespace mode) Initial Comment: This is a patch for xml.sax.saxutils.XMLGenerator class. It fixes the ancient bug #847665 - XMLGenerator dies in namespace-aware mode. Technically, the patch defines a function _qname() to write QName from (NSURI, local name) pair, and calls this function where appropriate inside startElementNS()/endElementNS(). The patch is applicable to both maintenance branch (release24-maint) and trunk versions: python/branches/release24-maint/Lib/xml/sax/saxutils.py python/trunk/Lib/xmlcore/sax/saxutils.py More details and a test are available in the bug tracker: http://sourceforge.net/tracker/index.php?func=detail&aid=847665&group_id=5470&atid=105470 ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-12 13:22 Message: Logged In: YES user_id=21627 Originator: NO Thanks for the patch. Committed as r53754 and r53755. ---------------------------------------------------------------------- Comment By: Nikolai Grigoriev (ngrig) Date: 2006-04-02 15:13 Message: Logged In: YES user_id=195108 For convenience, I add a simple test for basic cases. This is for Python 2.4 (it imports xml.sax, rather than xmlcore.sax). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1463026&group_id=5470 From noreply at sourceforge.net Mon Feb 12 20:00:00 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 12 Feb 2007 11:00:00 -0800 Subject: [Patches] [ python-Patches-1654974 ] sys.excepthook shows relevant bindings. Message-ID: Patches item #1654974, was opened at 2007-02-08 09:46 Message generated for change (Comment added) made by nejucomo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1654974&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nefarious CodeMonkey, Jr. (nejucomo) Assigned to: Nobody/Anonymous (nobody) Summary: sys.excepthook shows relevant bindings. Initial Comment: The attached module provides a suitable replacement for sys.excepthook to handle uncaught exceptions. The output is the same, except after each source line shown, a list of relevant bindings is shown. Here's a quick example from the tail end of a test: File "./test-exprann.py", line 16, in f assert c == 12 # With bindings: # c = 42 AssertionError The bindings shown are the intersection of the code object "co_names" and the non-keyword name tokens in the parsed source line. The goal is to only show bindings relevant to the exception. I hope the utility of this is self-evident to any programmer. I tried testing it with crazy expressions containing nonfree bindings (such as lambda's and list comprehensions), so I think it behaves well in most circumstances. The performance might be bad (it parses each line in the backtrace), but I figure it is worth it for uncaught exceptions. Let me know if you find this tool useful. Thanks, Nejucomo ---------------------------------------------------------------------- >Comment By: Nefarious CodeMonkey, Jr. (nejucomo) Date: 2007-02-12 19:00 Message: Logged In: YES user_id=44911 Originator: YES I've simplified and modularized the design of this tool. Included is a patch against the 2.6 svn head which adds a module to the stdlib called bindann (for "binding annotation"), and then modifies the traceback module to use it. It also includes a demo script which compares classic and annotated exception tracebacks. The interface to traceback is backwards compatible (it introduces optional arguments defaulting to pre-patch behavior). bindann is much simpler than the previous annex file, because it doesn't try parsing, only tokenizing the source line, to look for name references. Thus it succeeds where annex would fail (for example if the head of a for-loop causes an uncaught exception, the source line will only be the head which is not a complete/parsable expression). Here's the output of the demo: $ ./Demo/newfeatures/demo-bindann.py This demonstrates binding-annoted exception tracebacks. Consider the following function: def f(c): for char in c + " hello": print 'The string contains "%s".' % (char,) -With standard tracebacks, calling f(42) gives: Traceback (most recent call last): File "./Demo/newfeatures/demo-bindann.py", line 17, in main f(42) File "./Demo/newfeatures/demo-bindann.py", line 28, in f for char in c + " hello": TypeError: unsupported operand type(s) for +: 'int' and 'str' -And now with annotated exceptions: Traceback (most recent call last): File "./Demo/newfeatures/demo-bindann.py", line 33, in # With bindings: # main = # Source: main() File "./Demo/newfeatures/demo-bindann.py", line 24, in main # With bindings: # f = # Source: f(42) File "./Demo/newfeatures/demo-bindann.py", line 28, in f # With bindings: c = 42 # char = # Source: for char in c + " hello": TypeError: unsupported operand type(s) for +: 'int' and 'str' File Added: bindann-patch.1 ---------------------------------------------------------------------- Comment By: Nefarious CodeMonkey, Jr. (nejucomo) Date: 2007-02-08 09:50 Message: Logged In: YES user_id=44911 Originator: YES BTW, I only tested this on python 2.4. Because it examines the parse tree (very simply), it may not work if the AST format changes. The only parsing it does is search for all NAME tokens which are not keywords, so I assumed this was stable across releases. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1654974&group_id=5470 From noreply at sourceforge.net Mon Feb 12 20:01:06 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 12 Feb 2007 11:01:06 -0800 Subject: [Patches] [ python-Patches-1654974 ] Binding annotations in tracebacks. Message-ID: Patches item #1654974, was opened at 2007-02-08 09:46 Message generated for change (Settings changed) made by nejucomo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1654974&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nefarious CodeMonkey, Jr. (nejucomo) Assigned to: Nobody/Anonymous (nobody) >Summary: Binding annotations in tracebacks. Initial Comment: The attached module provides a suitable replacement for sys.excepthook to handle uncaught exceptions. The output is the same, except after each source line shown, a list of relevant bindings is shown. Here's a quick example from the tail end of a test: File "./test-exprann.py", line 16, in f assert c == 12 # With bindings: # c = 42 AssertionError The bindings shown are the intersection of the code object "co_names" and the non-keyword name tokens in the parsed source line. The goal is to only show bindings relevant to the exception. I hope the utility of this is self-evident to any programmer. I tried testing it with crazy expressions containing nonfree bindings (such as lambda's and list comprehensions), so I think it behaves well in most circumstances. The performance might be bad (it parses each line in the backtrace), but I figure it is worth it for uncaught exceptions. Let me know if you find this tool useful. Thanks, Nejucomo ---------------------------------------------------------------------- Comment By: Nefarious CodeMonkey, Jr. (nejucomo) Date: 2007-02-12 19:00 Message: Logged In: YES user_id=44911 Originator: YES I've simplified and modularized the design of this tool. Included is a patch against the 2.6 svn head which adds a module to the stdlib called bindann (for "binding annotation"), and then modifies the traceback module to use it. It also includes a demo script which compares classic and annotated exception tracebacks. The interface to traceback is backwards compatible (it introduces optional arguments defaulting to pre-patch behavior). bindann is much simpler than the previous annex file, because it doesn't try parsing, only tokenizing the source line, to look for name references. Thus it succeeds where annex would fail (for example if the head of a for-loop causes an uncaught exception, the source line will only be the head which is not a complete/parsable expression). Here's the output of the demo: $ ./Demo/newfeatures/demo-bindann.py This demonstrates binding-annoted exception tracebacks. Consider the following function: def f(c): for char in c + " hello": print 'The string contains "%s".' % (char,) -With standard tracebacks, calling f(42) gives: Traceback (most recent call last): File "./Demo/newfeatures/demo-bindann.py", line 17, in main f(42) File "./Demo/newfeatures/demo-bindann.py", line 28, in f for char in c + " hello": TypeError: unsupported operand type(s) for +: 'int' and 'str' -And now with annotated exceptions: Traceback (most recent call last): File "./Demo/newfeatures/demo-bindann.py", line 33, in # With bindings: # main = # Source: main() File "./Demo/newfeatures/demo-bindann.py", line 24, in main # With bindings: # f = # Source: f(42) File "./Demo/newfeatures/demo-bindann.py", line 28, in f # With bindings: c = 42 # char = # Source: for char in c + " hello": TypeError: unsupported operand type(s) for +: 'int' and 'str' File Added: bindann-patch.1 ---------------------------------------------------------------------- Comment By: Nefarious CodeMonkey, Jr. (nejucomo) Date: 2007-02-08 09:50 Message: Logged In: YES user_id=44911 Originator: YES BTW, I only tested this on python 2.4. Because it examines the parse tree (very simply), it may not work if the AST format changes. The only parsing it does is search for all NAME tokens which are not keywords, so I assumed this was stable across releases. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1654974&group_id=5470 From noreply at sourceforge.net Tue Feb 13 09:35:39 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 00:35:39 -0800 Subject: [Patches] [ python-Patches-685268 ] imputil must respect __path__ for package submodules Message-ID: Patches item #685268, was opened at 2003-02-12 12:51 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=685268&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.3 >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Soeren Fietzke (sfiet) Assigned to: Nobody/Anonymous (nobody) Summary: imputil must respect __path__ for package submodules Initial Comment: The imputil module does not look at the __path__ member of a package to find sub-modules. To reproduce, create a package with a submodule, i.e. directory structure like this: p1/ __init__.py sub/ spam.py contents of __init__.py: import os __path__.append(os.path.join(__path__[0], 'sub')) import spam contents of spam.py: print 'imported spam' Doing an "import p1" in a fresh Python session will succeed. However, after loading and installing imputil, it will fail: import imputil imputil._test_revamp() import p1 # <-- will not find module spam ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 09:35 Message: Logged In: YES user_id=21627 Originator: NO Thanks for the patch. Committed as r53763 and r53764. ---------------------------------------------------------------------- Comment By: Soeren Fietzke (sfiet) Date: 2003-02-12 13:52 Message: Logged In: YES user_id=710276 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- Comment By: Soeren Fietzke (sfiet) Date: 2003-02-12 12:58 Message: Logged In: YES user_id=710276 Oops, spaces were stripped out, so just to clarify: 'sub' should be a subdirectory of 'p1' in the above example: p1/__init__.py p1/sub/spam.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=685268&group_id=5470 From noreply at sourceforge.net Tue Feb 13 09:59:41 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 00:59:41 -0800 Subject: [Patches] [ python-Patches-813436 ] Scalable zipfile extension Message-ID: Patches item #813436, was opened at 2003-09-27 10:09 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=813436&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Performance Group: Python 2.5 >Status: Closed >Resolution: Rejected Priority: 5 Private: No Submitted By: Marc De Falco (deufeufeu) Assigned to: Nobody/Anonymous (nobody) Summary: Scalable zipfile extension Initial Comment: Playing around with large zipfiles (> 10000 files), I've encountered big loading time, even if after having loaded it I use only 30 files in it. So I've introduced a differed parameter to the Zipfile.__init__ in order to load headers on-demand. As it's not a really good idea to activated it for all zip it defaults to False. I've updated the documentation too. Thx and keep the good work ;) P.S. : Dunno if it can be added to 2.3 or have to be included in 2.4, so I've choosed 2.4 group. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 09:59 Message: Logged In: YES user_id=21627 Originator: NO I'm rejecting the patch, for the following reasons: - I agree with ronaldoussoren that this deferred loading already happens in the 2.5 version (specifically, it happens inside read) - I also agree that making it an optional parameter is unnecessary, it just complicates the interface. I also think the proposed parameter name ('differed') is mis-spelled, and should have been 'deferred' (unless I'm missing a meaning of 'differed') - the implementation of the patch is unacceptable because it duplicates code. As for why this patch is "put on hold": it was not because a rewrite was planned. The patch was contributed in 2003, and jafo's (first) comment was in 2006. The patch was "on hold" because nobody found the time to review it (just like these other 400 or so patches). ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2006-12-22 20:00 Message: Logged In: YES user_id=11375 Originator: NO Patch #1446489, mentioned in Ronald's 2006-05-27 14:22 comment, was committed and is in Python 2.5. Is this patch still relevant? ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2006-12-22 19:58 Message: Logged In: YES user_id=11375 Originator: NO According to http://mail.python.org/pipermail/python-dev/2006-November/069969.html, the author of the zipfile rewrite isn't quite happy with it. It doesn't look like the new module will be API-compatible with zipfile, so I think this patch should still be considered for inclusion. ---------------------------------------------------------------------- Comment By: Ronald Oussoren (ronaldoussoren) Date: 2006-05-27 20:22 Message: Logged In: YES user_id=580910 Patch [1446489 ] zipfile: support for ZIP64 also addresses this as a side- effect of adding support ZIP64 support (for very big zipfiles). BTW. I don't quite understand why this patch is put on hold just because a rewrite of the zipfile module is planned. W.r.t. this patch: why is the on-demand loading optional? Loading the per-file headers when the zipfile is opened is not necessary for normal operation, the current zipfile module is basically doing a full verify of the zipfile on all occassions. This isn't necessary for normal operation and I don't think the infozip tools do this (probably because verification is very expensive). ---------------------------------------------------------------------- Comment By: Sean Reifschneider (jafo) Date: 2006-05-25 16:38 Message: Logged In: YES user_id=81797 Actually, we'll leave it open until the Summer of Code implementation is completed and accepted. Sean ---------------------------------------------------------------------- Comment By: Sean Reifschneider (jafo) Date: 2006-05-25 16:36 Message: Logged In: YES user_id=81797 There is a summer of code project to re-write the zipfile module, so this patch is moot. Sean ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=813436&group_id=5470 From noreply at sourceforge.net Tue Feb 13 10:01:20 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 01:01:20 -0800 Subject: [Patches] [ python-Patches-992750 ] zipfile and big zipped file Message-ID: Patches item #992750, was opened at 2004-07-17 09:45 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=992750&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.3 >Status: Closed >Resolution: Out of Date Priority: 5 Private: No Submitted By: Keyphrene (keyphrene) Assigned to: Nobody/Anonymous (nobody) Summary: zipfile and big zipped file Initial Comment: read method decompress the zipped files, but this method returns bytes, more the file is big, more the bytes array is big. readFile method allows to send the bytes in a file. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 10:01 Message: Logged In: YES user_id=21627 Originator: NO It seems that this patch was superceded by 1121142, so closing it as out-of-date. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-02-11 02:32 Message: Logged In: YES user_id=1115903 Ok, I ended up just starting from scratch and implementing a file-like class - it just implements read([size]) right now - and an instance of this class is returned when you call ZipFile.open. ZipFile.read was modified to use open(), as loewis suggested. I also added some code to test_zipfile.py to check the new open function, and tried using it to extract a 690MB file from a ZIP with copyfileobj. It seems to be working ok. Since I can't get into CVS at the moment, I don't have diff files to post. Even if I did, I don't see an option for me to post them to this patch. Should I post them as a new patch? ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-02-09 23:29 Message: Logged In: YES user_id=1115903 The current behavior of ZipFile.read is an occasional annoyance for me. I will make the changes described in loewis' comments and submit the updated patch, unless somebody is already working on it. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-08-18 15:13 Message: Logged In: YES user_id=21627 The patch is unacceptable in its current form. It duplicates too much code; read and readfile should really share the common code (e.g. by read falling back to readfile). Also, as this really copies the data to fileobj, it probably is better called copyfile, instead of readfile. Finally, I would much prefer if zipfile would have an .open method, returning a file-like object. Then you would have such implementations def read(self, name): return self.open(name, "r").read() def readfile(self, name, dest): import shutil shutil.copyfileobj(self.open(name,"r"), dest) I would then probably leave the last method out, and encourage users to invoke shutil.copyfileobj directly if desired. Are you willing to make changes in this direction? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=992750&group_id=5470 From noreply at sourceforge.net Tue Feb 13 10:14:52 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 01:14:52 -0800 Subject: [Patches] [ python-Patches-1121142 ] ZipFile.open - read-only file-like obj for files in archive Message-ID: Patches item #1121142, was opened at 2005-02-12 01:08 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open >Resolution: Out of Date Priority: 5 Private: No Submitted By: Alan McIntyre (alanmcintyre) Assigned to: Nobody/Anonymous (nobody) Summary: ZipFile.open - read-only file-like obj for files in archive Initial Comment: I originally started working on updating patch 992750, but decided after a short while to just start from scratch, so I'm posting it as a new patch. Sorry if this isn't appropriate. This patch provides a new open() method on ZipFile; this method returns a file-like object for the requested item in the archive. This file-like object only provides a read() method. ZipFile.read was modified to use the new open method (this was suggested by loewis in reference to patch 992750). The patched zipfile.py passed the existing tests in the test_zipfile.py from CVS. New tests were added to verify the operation of the object returned by open(). These tests were modeled after existing tests for ZipFile.read(); two read fixed-size chunks from the file-like object, and two others read random-sized chunks. I have only run the tests on Windows XP, using Python2.4 from the official Windows installer. I will test the patch out on Linux over the weekend. If the patch is accepted I'll also generate and submit patches for the appropriate documentation as well. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 10:14 Message: Logged In: YES user_id=21627 Originator: NO Can you please update the patch to the current subversion trunk? I'd like to apply it, but the code of zipfile has changed so that the patch is currently out-of-date. When redoing it, notice that the read implementation has changed (I couldn't figure out how you moved code around). Please do use the trunk (not Python 2.5), and please submit the patch as a single 'svn diff' output (rather than a tar file containing multiple individual diff files). ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-05-31 04:56 Message: Logged In: YES user_id=1115903 Revision 5 of this patch has been in constant use with Python 2.4.1 in an application at my job for about a month; it seems to be stable and useful in that regard. If anybody has time to review and accept (or reject) it I would appreciate it. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-27 04:23 Message: Logged In: YES user_id=1115903 After testing on my large batch of large Zip files, I made one fix (version 4 of the patch didn't read all the content of large compressed archive items). The current set of changes is attached as zipfile_patch5.tgz. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-13 18:34 Message: Logged In: YES user_id=1115903 I found a problem with my universal newline handling code in readline(); if the first byte of an '\r\n' pair was read from file but the second byte didn't come in on that same read, it resulted in the next line having an inappropriate '\n' prepended to it. zipfile_patch4.tgz has a fix for this included (with everything else, of course). I'm going to test the open() capability in a real application with some reasonably large zip files (containing files up to just short of 2GB) over the next few days, so if any bugs or performance problems show up I may have some more changes. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-13 07:58 Message: Logged In: YES user_id=1115903 Uploaded the third revision of this patch; passes all regression tests against current CVS on WinXP. I think all the issues Martin brought up have been addressed except perhaps for the case of compression rate <1. I will have a look at that when I have time; just wanted to get an update here before the patch started to look abandoned. :) ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-03-14 15:37 Message: Logged In: YES user_id=1115903 Hmm...I could have sworn I did the diff in the correct order. I'll make sure next time. :) Here's my comments on your remarks (in order): - I'm adding support for universal newlines, and will reject all modes that aren't legal combinations of r, U, and b. - I'll see if I can make a Zip file store something with compression < 1, and if I can I'll add a test case for it. - I'll try work a .flush() in there on the compression object and come up with a test case if possible - .read(0) and .readline(0) will both return an empty string with no side-effects, since this seems to be what builtin files do. Right now ZipExtFile uses the ZipFile's file object, so you pretty much have to do whatever you're going to do with the ZipExtFile instance you get back from .open() before you use that ZipFile instance in a way that moves the file pointer around. I'm sure that somebody will eventually try to use the ZipFile in this way, so it appears my options are either to (1) give the ZipExtFile its own file object to use (when possible), or (2) make sure this limitation is documented. #1 feels (to me) to be the "right thing" to do, so that's what I'll try unless there's a good reason I shouldn't. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-03-01 08:59 Message: Logged In: YES user_id=21627 The patch is reversed: usually, diff is invoked as "-c old new". I think it is almost right, but I have a few remarks: - by tradition, open() should have a mode argument, defaulting to 'r'; it would be ok to raise exceptions if it is anything else. However, do consider implementing universal newlines; allowing 'b' as a no-op might also be reasonable. - I wonder what happens if the compression rate is < 1. It would appear that the code might use too few rawbytes. I suggest to recursively invoke read in this case. - I wonder whether it could ever happen that there is still data to uncompress in the zlib object, ie. whether it might be necessary to invoke .flush() after exhausting the rawbytes (and discard the zlib object afterwards) - it appears that atleast the builtin file object implements .read(0) as returning an empty string; the manual says that the entire file is meant only if size is omitted or negative. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-02-27 07:28 Message: Logged In: YES user_id=1115903 zipfile_patch2.tgz: I updated the file-like object to support readline, readlines, __iter__ and next(). Added tests for those new methods. Also added a patch for the documentation. Passes regression tests on 2.5a0 built from CVS HEAD with MSVC .NET on Windows XP. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 From noreply at sourceforge.net Tue Feb 13 10:50:09 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 01:50:09 -0800 Subject: [Patches] [ python-Patches-698833 ] ZipFile - support for file decryption Message-ID: Patches item #698833, was opened at 2003-03-06 17:57 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=698833&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.3 >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Giovanni Bajo (giovannibajo) Assigned to: Nobody/Anonymous (nobody) Summary: ZipFile - support for file decryption Initial Comment: The attached patch adds support for the ZIP file decryption. Right now, only decryption is supported (not encryption), but I will work on this as well if there are no problems with this patch. The ZIP encryption scheme uses 96-bits keys, so there might be some US law annoyances (see http://www.info- zip.org/pub/infozip/FAQ.html#crypto). To me, everything seems legit, but I am not a lawyer. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 10:50 Message: Logged In: YES user_id=21627 Originator: NO Thanks for the patch. Committed as r53765 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=698833&group_id=5470 From noreply at sourceforge.net Tue Feb 13 11:11:23 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 02:11:23 -0800 Subject: [Patches] [ python-Patches-1517891 ] Bug 1514451: zipfile "append" mode should create a file ... Message-ID: Patches item #1517891, was opened at 2006-07-06 04:25 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1517891&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Thomas Lee (krumms) Assigned to: Nobody/Anonymous (nobody) Summary: Bug 1514451: zipfile "append" mode should create a file ... Initial Comment: '[ 1514451 ] zipfile "append" mode should create a file if not present' I thought this was a logical suggestion. Here's a patch, including tests. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 11:11 Message: Logged In: YES user_id=21627 Originator: NO I think there is a documentation patch missing, also: the documentation explicitly mentions that mode 'a' means to append to an *existing* file. I changed the documentation accordingly, and committed the patch as r53766. Thanks for the contribution! ---------------------------------------------------------------------- Comment By: Ronald Oussoren (ronaldoussoren) Date: 2006-07-06 10:42 Message: Logged In: YES user_id=580910 This patch looks sane. What I don't know is if this qualifies as a bugfix or if this is a new feature (however minor). ---------------------------------------------------------------------- Comment By: Thomas Lee (krumms) Date: 2006-07-06 04:28 Message: Logged In: YES user_id=315535 It should be noted that the decision not to modify self.mode (i.e. change it to 'w', which is effectively what we're doing) was a conscious decision: existing code may depend on the mode attribute (however unlikely that might be). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1517891&group_id=5470 From noreply at sourceforge.net Tue Feb 13 11:34:01 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 02:34:01 -0800 Subject: [Patches] [ python-Patches-1658799 ] Handle requests to intern string subtype instances Message-ID: Patches item #1658799, was opened at 2007-02-13 11:34 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1658799&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Hrvoje Nik?i? (hniksic) Assigned to: Nobody/Anonymous (nobody) Summary: Handle requests to intern string subtype instances Initial Comment: This patch implements a small modification of PyString_InternInPlace that allows for safe interning of string subtype instances. The change should be fully backward compatible -- for a rationale and discussion, see: http://mail.python.org/pipermail/python-dev/2007-February/070973.html ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1658799&group_id=5470 From noreply at sourceforge.net Tue Feb 13 13:14:55 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 04:14:55 -0800 Subject: [Patches] [ python-Patches-1657276 ] socketmodule fails to build because missing NETLINK_DNRTMSG Message-ID: Patches item #1657276, was opened at 2007-02-11 13:45 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657276&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Build Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Aleksandr Koltsoff (czr_) Assigned to: Nobody/Anonymous (nobody) Summary: socketmodule fails to build because missing NETLINK_DNRTMSG Initial Comment: While building 2.5 on a debian 2.2 (potato) system I encountered a problem in socketmodule.c. The system in question has netlink.h, but a symbol (NETLINK_DNRTMSG) is not (whole module fails to build because of this). The provided patch (unified diff) fixes this by surrounding the line with an ifdef. The respective code also contains other int constants that probably should be protected, but this patch doesn't touch them. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 13:14 Message: Logged In: YES user_id=21627 Originator: NO Thanks for the patch. Committed as r53769 and r53770. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657276&group_id=5470 From noreply at sourceforge.net Tue Feb 13 13:15:57 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 04:15:57 -0800 Subject: [Patches] [ python-Patches-1657613 ] documentation for element interface Message-ID: Patches item #1657613, was opened at 2007-02-12 01:15 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657613&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Tim Mitchell (tim-mitchell) >Assigned to: Fredrik Lundh (effbot) Summary: documentation for element interface Initial Comment: I have added a subsection to 8.13 xml.etree.ElementTree documenting the _ElementInterface class which objects returned by the Element and SubElement factory functions emulate. I do not have latex installed on my system so I have not been able to check for compiling errors - sorry. (I don't intend to become a regular contributor to go through the bother of setting it up.) ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 13:15 Message: Logged In: YES user_id=21627 Originator: NO Fredrik, can you please take a look? If not, assign it back to Fred. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657613&group_id=5470 From noreply at sourceforge.net Tue Feb 13 13:16:23 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 04:16:23 -0800 Subject: [Patches] [ python-Patches-1657276 ] socketmodule fails to build because missing NETLINK_DNRTMSG Message-ID: Patches item #1657276, was opened at 2007-02-11 13:45 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657276&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Build Group: Python 2.5 >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Aleksandr Koltsoff (czr_) Assigned to: Nobody/Anonymous (nobody) Summary: socketmodule fails to build because missing NETLINK_DNRTMSG Initial Comment: While building 2.5 on a debian 2.2 (potato) system I encountered a problem in socketmodule.c. The system in question has netlink.h, but a symbol (NETLINK_DNRTMSG) is not (whole module fails to build because of this). The provided patch (unified diff) fixes this by surrounding the line with an ifdef. The respective code also contains other int constants that probably should be protected, but this patch doesn't touch them. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 13:14 Message: Logged In: YES user_id=21627 Originator: NO Thanks for the patch. Committed as r53769 and r53770. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657276&group_id=5470 From noreply at sourceforge.net Tue Feb 13 13:21:42 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 04:21:42 -0800 Subject: [Patches] [ python-Patches-1647484 ] gzip.GzipFile has no name attribute Message-ID: Patches item #1647484, was opened at 2007-01-29 23:25 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1647484&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Lars Gust?bel (gustaebel) Assigned to: Nobody/Anonymous (nobody) Summary: gzip.GzipFile has no name attribute Initial Comment: The gzip.GzipFile object uses a filename instead of a name attribute. This deviates from the standard practice and the interface described in "3.9 File Objects" and seems unnecessary. Attached patch changes this but still leaves the filename attribute as a property that emits a DeprecationWarning. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 13:21 Message: Logged In: YES user_id=21627 Originator: NO I think the patch is fine. It is debatable whether this should be backported to 2.5; I think it is better to not backport it. Please also add a test case testing for the presence of the name attribute (feel free to name it test_1647484) ---------------------------------------------------------------------- Comment By: Lars Gust?bel (gustaebel) Date: 2007-02-12 10:45 Message: Logged In: YES user_id=642936 Originator: YES Seems that nobody is interested. I will check this in in a few days if nobody objects. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1647484&group_id=5470 From noreply at sourceforge.net Tue Feb 13 17:36:56 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 08:36:56 -0800 Subject: [Patches] [ python-Patches-1647484 ] gzip.GzipFile has no name attribute Message-ID: Patches item #1647484, was opened at 2007-01-29 23:25 Message generated for change (Comment added) made by gustaebel You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1647484&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Lars Gust?bel (gustaebel) Assigned to: Nobody/Anonymous (nobody) Summary: gzip.GzipFile has no name attribute Initial Comment: The gzip.GzipFile object uses a filename instead of a name attribute. This deviates from the standard practice and the interface described in "3.9 File Objects" and seems unnecessary. Attached patch changes this but still leaves the filename attribute as a property that emits a DeprecationWarning. ---------------------------------------------------------------------- >Comment By: Lars Gust?bel (gustaebel) Date: 2007-02-13 17:36 Message: Logged In: YES user_id=642936 Originator: YES When I wrote the test I saw that in write mode a '.gz' extension is automatically added to the filename if it has none. However this is stripped again when the name is written to the gzip header. I removed this so that the name attribute always reflects the real filesystem name. The filename property imitates the old behaviour. I don't think that this should necessarily be backported, at least I don't need it ;-) Applied in rev. 53772. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 13:21 Message: Logged In: YES user_id=21627 Originator: NO I think the patch is fine. It is debatable whether this should be backported to 2.5; I think it is better to not backport it. Please also add a test case testing for the presence of the name attribute (feel free to name it test_1647484) ---------------------------------------------------------------------- Comment By: Lars Gust?bel (gustaebel) Date: 2007-02-12 10:45 Message: Logged In: YES user_id=642936 Originator: YES Seems that nobody is interested. I will check this in in a few days if nobody objects. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1647484&group_id=5470 From noreply at sourceforge.net Tue Feb 13 22:45:21 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 13:45:21 -0800 Subject: [Patches] [ python-Patches-1659326 ] Minor pasting patch Message-ID: Patches item #1659326, was opened at 2007-02-13 23:45 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1659326&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Tal Einat (taleinat) Assigned to: Nobody/Anonymous (nobody) Summary: Minor pasting patch Initial Comment: When pasting text so that it goes beyond a window's right edge, the window is not scrolled, and the insertion cursor is not visible. Just add self.text.see("insert") before the return in EditorWindow.paste() to fix this. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1659326&group_id=5470 From noreply at sourceforge.net Tue Feb 13 23:36:46 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 14:36:46 -0800 Subject: [Patches] [ python-Patches-1658799 ] Handle requests to intern string subtype instances Message-ID: Patches item #1658799, was opened at 2007-02-13 10:34 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1658799&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Hrvoje Nik?i? (hniksic) Assigned to: Nobody/Anonymous (nobody) Summary: Handle requests to intern string subtype instances Initial Comment: This patch implements a small modification of PyString_InternInPlace that allows for safe interning of string subtype instances. The change should be fully backward compatible -- for a rationale and discussion, see: http://mail.python.org/pipermail/python-dev/2007-February/070973.html ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2007-02-13 22:36 Message: Logged In: YES user_id=4771 Originator: NO I think that this opens an attack (untested, though): it allows a previously-interned string to be removed from the dictionary. This might lead to a crash because the old string still thinks it is interned. Try something along the lines of: s1 = "hel" s1 = intern(s1 + "lo") class S(str): def __hash__(self): return hash(s1) def __eq__(self, other): return other == s1 s = S("world") intern(s) del s1 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1658799&group_id=5470 From noreply at sourceforge.net Wed Feb 14 01:44:45 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 13 Feb 2007 16:44:45 -0800 Subject: [Patches] [ python-Patches-1659410 ] Minor AST tweaks Message-ID: Patches item #1659410, was opened at 2007-02-13 19:44 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1659410&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Parser/Compiler Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Collin Winter (collinwinter) Assigned to: Nobody/Anonymous (nobody) Summary: Minor AST tweaks Initial Comment: This patch implements these changes, as discussed in http://mail.python.org/pipermail/python-dev/2007-February/071006.html and following. """ 1) There are times when the _fields attribute on some AST nodes is None; if this was done to indicate that a given node has no AST-related attributes, it would be much easier if _fields was [] in this case. As it is, I had to special-case "node._fields is None" in the visitor so that I don't accidentally iterate over it. """ asdl_c.py is changed to always set _fields to a tuple. """ 2) {BinOp,AugAssign,BoolOp,etc}.op is an instance of, eg, Add, Sub, Mult, Mod, meaning you have to dispatch based on tests like "isinstance(node.op, x)" or "type(node.op) is x". I would much, much prefer to spell this "node.op is x", ie, use "node.op = Add" rather than the current "node.op = Add()" when constructing the nodes. """ asdl_c.py is changed so that it no longer emits the *_singleton objects and uses the corresponding *_type objects in their place. """ 3) I'd like there to be an Else() node for If.orelse, While.orelse, etc. My motivation is that I need the lineno and col_offset values of the "else" statement for a code-coverage utility; as it is, I have to find the last lineno of the if/while/etc suite and the first lineno of the "else" suite and then search between those two for the "else" statement. """ An Else node is added to the ASDL file, impacting TryExcept, While, For and If nodes. The compiler, symtable and other files are adjusted to deal with these new nodes. The patch is against r53772. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1659410&group_id=5470 From noreply at sourceforge.net Wed Feb 14 09:31:26 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 00:31:26 -0800 Subject: [Patches] [ python-Patches-1658799 ] Handle requests to intern string subtype instances Message-ID: Patches item #1658799, was opened at 2007-02-13 11:34 Message generated for change (Comment added) made by hniksic You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1658799&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Hrvoje Nik?i? (hniksic) Assigned to: Nobody/Anonymous (nobody) Summary: Handle requests to intern string subtype instances Initial Comment: This patch implements a small modification of PyString_InternInPlace that allows for safe interning of string subtype instances. The change should be fully backward compatible -- for a rationale and discussion, see: http://mail.python.org/pipermail/python-dev/2007-February/070973.html ---------------------------------------------------------------------- >Comment By: Hrvoje Nik?i? (hniksic) Date: 2007-02-14 09:31 Message: Logged In: YES user_id=1718107 Originator: YES I don't think an attack is possible. This patch retains the property that only exact strings are interned. If you create a pathological string subtype that hashes like a different string instance (one that has already been interned), all you'll achieve is that "interning" will return the other instance. As far as I can tell, no string is actually removed from the interned dictionary (until it becomes unreachable, that is.) What is the expected result of your test code? I tried it and it ran without error. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2007-02-13 23:36 Message: Logged In: YES user_id=4771 Originator: NO I think that this opens an attack (untested, though): it allows a previously-interned string to be removed from the dictionary. This might lead to a crash because the old string still thinks it is interned. Try something along the lines of: s1 = "hel" s1 = intern(s1 + "lo") class S(str): def __hash__(self): return hash(s1) def __eq__(self, other): return other == s1 s = S("world") intern(s) del s1 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1658799&group_id=5470 From noreply at sourceforge.net Wed Feb 14 10:00:36 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 01:00:36 -0800 Subject: [Patches] [ python-Patches-1659410 ] Minor AST tweaks Message-ID: Patches item #1659410, was opened at 2007-02-14 01:44 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1659410&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Parser/Compiler Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Collin Winter (collinwinter) Assigned to: Nobody/Anonymous (nobody) Summary: Minor AST tweaks Initial Comment: This patch implements these changes, as discussed in http://mail.python.org/pipermail/python-dev/2007-February/071006.html and following. """ 1) There are times when the _fields attribute on some AST nodes is None; if this was done to indicate that a given node has no AST-related attributes, it would be much easier if _fields was [] in this case. As it is, I had to special-case "node._fields is None" in the visitor so that I don't accidentally iterate over it. """ asdl_c.py is changed to always set _fields to a tuple. """ 2) {BinOp,AugAssign,BoolOp,etc}.op is an instance of, eg, Add, Sub, Mult, Mod, meaning you have to dispatch based on tests like "isinstance(node.op, x)" or "type(node.op) is x". I would much, much prefer to spell this "node.op is x", ie, use "node.op = Add" rather than the current "node.op = Add()" when constructing the nodes. """ asdl_c.py is changed so that it no longer emits the *_singleton objects and uses the corresponding *_type objects in their place. """ 3) I'd like there to be an Else() node for If.orelse, While.orelse, etc. My motivation is that I need the lineno and col_offset values of the "else" statement for a code-coverage utility; as it is, I have to find the last lineno of the if/while/etc suite and the first lineno of the "else" suite and then search between those two for the "else" statement. """ An Else node is added to the ASDL file, impacting TryExcept, While, For and If nodes. The compiler, symtable and other files are adjusted to deal with these new nodes. The patch is against r53772. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-14 10:00 Message: Logged In: YES user_id=21627 Originator: NO Please minimize white-space only changes in the patch. It makes it harder to read (symtable.c in particular). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1659410&group_id=5470 From noreply at sourceforge.net Wed Feb 14 10:11:24 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 01:11:24 -0800 Subject: [Patches] [ python-Patches-1500504 ] Alternate RFC 3986 compliant URI parsing module Message-ID: Patches item #1500504, was opened at 2006-06-05 00:50 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1500504&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nick Coghlan (ncoghlan) Assigned to: Nobody/Anonymous (nobody) Summary: Alternate RFC 3986 compliant URI parsing module Initial Comment: Inspired by (and based on) Paul Jimenez's uriparse module (http://python.org/sf/1462525), urischemes tries to put a cleaner interface in front of the URI parsing engine. Most of the module works with a URI subclass of tuple that is always a 5-tuple (scheme, authority, path, query, fragment). The authority component is either None, or a URIAuthority subclass of tuple that is always a 4-tuple (user, password, host, port). The function make_uri will create a URI string from the 5 constituent components of a URI. The components do not need to be strings - if they are not strings, str() will be invoked on them (this allows the URIAuthority tuple subclass to be used transparently instead of a string for the authority component). The result is checked to ensure it is an RFC-compliant URI. The function split_uri accepts a string and returns a URI object with strings as the individual elements. Invoking str() on this object will recreate a URI string using make_uri(). The regex underlying this operation is now broken out and available as module level attributes like URI_PATTERN. The functions split_authority and make_authority are similar, only working solely on the authority component rather than the whole URI. The function parse_uri digs into the internal structure of a URI, also parsing the components. This will replace a non-empty URI authority component string with a URIAuthority tuple subclass. Depending on the scheme, it may also replace other components (e.g. for mailto links, the path is replaced with a (user, host) tuple subclass). The main parsing engine is still URIParser (much the same as Paul's), but the root of the internal parser hierarchy is now SchemeParser. This has two subclasses, URLParser and MailtoParser. The various URL flavours are now different instances of URLParser rather than subclasses. All of the actual parsers are available as module level attributes with the same name as the scheme they parse. Additionally, each parser knows the name of the scheme it is intended to parse. The parse() methods of the individual parsers are now expected to return a URI object (SchemeParser actually takes care of this). The parse() method also takes a dictionary of defaults, which can override the defaults supplied by the parser instance. The unparse() method is gone - instead, the scheme parser should ensure that all components returned are either strings or produce the right thing when __str__ is invoked (e.g. see _MailtoURIPath) The module level 'schemes' attribute is a mapping from scheme names to parsers that is automatically populated with all instances of SchemeParser that are found in the module globals() urljoin has been renamed to join_uri to match the style of the other names in the module. ---------------------------------------------------------------------- >Comment By: Nick Coghlan (ncoghlan) Date: 2007-02-14 19:11 Message: Logged In: YES user_id=1038590 Originator: YES Removed all versions prior to 0.4 ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2006-06-08 22:11 Message: Logged In: YES user_id=1038590 Uploaded version 0.4 This version cleans up the logic in resolve_uripath a bit (use a separate loop to strip the leading dot segments, add comments explaining meaning of if statements when dealing with dot segments). It also exposes EmailPath (along with split_emailpath and join_emailpath) as public objects, rather than treating them as internal to the MailtoSchemeParser. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2006-06-07 01:46 Message: Logged In: YES user_id=1038590 Uploaded version 0.3 which passes all the RFC tests, as well as the failing 4Suite tests Mike sent me based on version 0.1 and 0.2. The last 4suite failure went away when I realised those tests expected to operate in strict mode :) ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2006-06-05 23:53 Message: Logged In: YES user_id=1038590 Updated version attached which addresses some issues raised by Mike Brown in private mail (the difference between a URI and a URI reference and some major differences between URI paths and posix paths). Also settled on split/join for the component separation and recombination operations and made the join methods all take a tuple so that join_x(split_x(uri)) round trips. Based on the terminology in the RFC, the function to combine a URI reference with a base URI is now called "resolve_uriref". ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1500504&group_id=5470 From noreply at sourceforge.net Wed Feb 14 10:46:46 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 01:46:46 -0800 Subject: [Patches] [ python-Patches-1642547 ] Fix error/crash in AST: syntaxerror in complex ifs Message-ID: Patches item #1642547, was opened at 2007-01-23 15:02 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1642547&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None Status: Open >Resolution: Accepted Priority: 9 Private: No Submitted By: Thomas Wouters (twouters) >Assigned to: Thomas Wouters (twouters) Summary: Fix error/crash in AST: syntaxerror in complex ifs Initial Comment: Fix a bug in Python/ast.c, where a particular syntaxerror in an 'if' with one or more 'elif's would be ignored or mishandled: timberwolf:~/python/python/trunk > cat test2.py def bug(): if w: dir()=1 elif v: pass timberwolf:~/python/python/trunk > python2.4 test2.py File "test2.py", line 3 dir()=1 SyntaxError: can't assign to function call timberwolf:~/python/python/trunk > python2.5 test2.py Exception exceptions.SyntaxError: ("can't assign to function call", 3) in 'garbage collection' ignored Fatal Python error: unexpected exception during garbage collection Aborted The actual problem is the lack of error checks on the return values of ast_for_expr() and ast_for_suite, in ast_for_if_stmt. Attached patch fixes. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-14 10:46 Message: Logged In: YES user_id=21627 Originator: NO The patch looks fine to me, please apply (or let me know if you want me to apply it). I think there are a few other places where return values of ast_for_ aren't checked: - ast_for_listcomp, not checking ast_for_expr results - ast_for_withexpr, not checking context_expr ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1642547&group_id=5470 From noreply at sourceforge.net Wed Feb 14 11:10:22 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 02:10:22 -0800 Subject: [Patches] [ python-Patches-1432399 ] Missing HCI sockets in bluetooth code from socketmodule Message-ID: Patches item #1432399, was opened at 2006-02-15 20:26 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1432399&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.4 >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Philippe Biondi (drphil) Assigned to: Nobody/Anonymous (nobody) Summary: Missing HCI sockets in bluetooth code from socketmodule Initial Comment: Bluetooth code in socketmodule does not implement BTPROTO_HCI. This small patch adds the missing code. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-14 11:10 Message: Logged In: YES user_id=21627 Originator: NO Thanks for the patch, I committed it as r53774. I made a few modifications, like adding the return statement and guessing what the NetBSD support could be; I was not able to test any of this. As for documentation and test suite changes: I'm willing to waive them. The bluetooth sockets are not documented at all so far, and testing them would only be possible if you have the hardware. I guess users will tell us if this is correct or whether something is missing. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-03-14 08:45 Message: Logged In: YES user_id=33168 This patch is missing a return 1; in case BTPROTO_HCI: around line 1310. I don't know if there are other problems or if this code is portable. There are no tests or doc updates. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1432399&group_id=5470 From noreply at sourceforge.net Wed Feb 14 11:12:04 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 02:12:04 -0800 Subject: [Patches] [ python-Patches-1434657 ] Patch to support lots of file descriptors Message-ID: Patches item #1434657, was opened at 2006-02-19 17:34 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1434657&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.4 >Status: Closed >Resolution: Rejected Priority: 5 Private: No Submitted By: Sven Berkvens-Matthijsse (svenberkvens) Assigned to: Nobody/Anonymous (nobody) Summary: Patch to support lots of file descriptors Initial Comment: Off-the-shelf Python on FreeBSD (and probably other OS'es as well) has a limit on the number of file descriptors that select can use. This is limited by the size of the fdset structures. On FreeBSD, and probably in many OS'es, a user can set the size of the fdset structures if he defines FD_SETSIZE before he includes sys/types.h. My patch sets the FD_SETSIZE to a substantial number. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-14 11:12 Message: Logged In: YES user_id=21627 Originator: NO Apparently, there is no activity on this patch, so I'm eventually rejecting it. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-02-28 23:48 Message: Logged In: YES user_id=21627 This test is not good enough. On systems where FD_SETSIZE is not "settable", it still is a macro, and setting it will succeed. The problem is that on these systems, select(2) might refuse to process more than FD_SETSIZE file descriptors, and/or fd_set might just buffer-overrun. In these cases, the cure would be worse than the desease: you would be able to pass that many file descriptors to select, but the system would never tell you when they are ready. If you want to use many file descriptors, just use poll(2) instead of select(2). ---------------------------------------------------------------------- Comment By: Sven Berkvens-Matthijsse (svenberkvens) Date: 2006-02-28 09:51 Message: Logged In: YES user_id=175845 Oops, 10000 should obivously match 10240 here... Sven ---------------------------------------------------------------------- Comment By: Sven Berkvens-Matthijsse (svenberkvens) Date: 2006-02-28 09:50 Message: Logged In: YES user_id=175845 Is it perhaps possible to add a test to the configuring process that tests whether something like: #define FD_SETSIZE 10240 #include #include int main(int argc, char **argv) { #if FD_SETSIZE == 10000 return 0; #else return 1; #endif } can be preprocessed, compiled and run correctly? This will most probably fail on systems where FD_SETSIZE is not settable, and succeed on systems where it is. Thanks for responding, Sven ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-02-27 22:54 Message: Logged In: YES user_id=21627 That patch is wrong: you should only change the value of FD_SETSIZE on systems where this actually works. Unfortunately, I cannot think of a method to determine this reliably. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1434657&group_id=5470 From noreply at sourceforge.net Wed Feb 14 11:15:55 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 02:15:55 -0800 Subject: [Patches] [ python-Patches-1126187 ] isapi.samples.advanced.py fix Message-ID: Patches item #1126187, was opened at 2005-02-17 20:43 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1126187&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None >Status: Closed >Resolution: Rejected Priority: 5 Private: No Submitted By: Philippe Kirsanov (phwp) Assigned to: Nobody/Anonymous (nobody) Summary: isapi.samples.advanced.py fix Initial Comment: This fix is to support win32 files in fotmat: \\?\drive:\dir\name.ext It's valid file format in win32. I've got that kind of file name trying to use __file__ when script is used as isapi extension. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-14 11:15 Message: Logged In: YES user_id=21627 Originator: NO It's not clear to me what the intention of this patch is: it is not a unified or context diff, so it doesn't apply to any of the Python source code files. The contributed advanced.py seems to have no relationship with any of the functionality included in Python. So I'm rejecting the patch as invalid. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1126187&group_id=5470 From noreply at sourceforge.net Wed Feb 14 11:17:38 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 02:17:38 -0800 Subject: [Patches] [ python-Patches-1144504 ] Add IEEE Float support to wave.py Message-ID: Patches item #1144504, was opened at 2005-02-19 21:02 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1144504&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.3 >Status: Closed >Resolution: Rejected Priority: 5 Private: No Submitted By: Ben Schwartz (bensatmit_edu) Assigned to: Nobody/Anonymous (nobody) Summary: Add IEEE Float support to wave.py Initial Comment: IEEE Float .wav files have almost identical structure to standard PCM .wav files. Currently, wave.py gives an unknown format error when attempting to read IEEE Float .wav files. This patch causes wave.py to read those files properly. No changes were made to wave_write, only wave_read. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-14 11:17 Message: Logged In: YES user_id=21627 Originator: NO Since there was no follow-up to Andrew's remark that the patch is incorrect, I conclude that the submitter either lost interest in it, or agrees that the patch is indeed incorrect. Rejecting it as invalid. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2005-11-22 19:54 Message: Logged In: YES user_id=11375 Looking at the WAV format description at http://ccrma.stanford.edu/CCRMA/Courses/422/projects/WaveFormat , it looks like the patch isn't correct. There's an ExtraParams field after the header that's only present if the format is not PCM. If this is correct, the patch should also add code to read this field and either store it somewhere or just discard it. ---------------------------------------------------------------------- Comment By: Michael Chermside (mcherm) Date: 2005-02-23 14:11 Message: Logged In: YES user_id=99874 I don't know anything about the .wav format, but the patch is particularly inoffensive... it simply allows a second format constant in the WAVE header without raising an error. I recomend applying the patch as soon as someone else who DOES know the .wav format can confirm that it's being done right. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1144504&group_id=5470 From noreply at sourceforge.net Wed Feb 14 11:27:25 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 02:27:25 -0800 Subject: [Patches] [ python-Patches-1144636 ] Make urllib2.OpenerDirector instances pickle-able Message-ID: Patches item #1144636, was opened at 2005-02-20 03:48 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1144636&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: John J Lee (jjlee) Assigned to: Nobody/Anonymous (nobody) Summary: Make urllib2.OpenerDirector instances pickle-able Initial Comment: urllib2.OpenerDirector would be pickleable if it were not for the fact that a couple of urllib2 handlers (which build_opener() adds by default) have function instance attributes. It's trivial to replace these instance attributes with a method that delegates to the appropriate function, thus allowing pickling. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-14 11:27 Message: Logged In: YES user_id=21627 Originator: NO The patch doesn't fix the problem; the opener remains unpicklable. If you are still interested in this patch, please revise it accordingly; if not, please close it. When you revise it, please include a test case, like this one: def test_1144636(self): # Make sure OpenerDirector instances can be pickled import pickle try: pickle.dumps(urllib2.build_opener()) except pickle.PicklingError: self.fail("cannot pickle OpenerDirector instances") ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2005-03-04 19:46 Message: Logged In: YES user_id=261020 Hmm, maybe this fault (can't pickle) will be fixed without need for this patch in 2.5, due to the change to unbound methods that was discussed on python-dev. I don't understand that issue, so I can't say. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1144636&group_id=5470 From noreply at sourceforge.net Wed Feb 14 11:35:47 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 02:35:47 -0800 Subject: [Patches] [ python-Patches-909005 ] asyncore fixes and improvements Message-ID: Patches item #909005, was opened at 2004-03-03 14:07 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=909005&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Alexey Klimkin (klimkin) Assigned to: A.M. Kuchling (akuchling) Summary: asyncore fixes and improvements Initial Comment: Minor: * 0/1 for boolean values replaced with False/True. * (887279) Added handling of POLLPRI as POLLIN. POLLERR, POLLHUP, POLLNVAL are handled as exception event. handle_expt_event gets recent error from self.socket object and raises socket.error. * Default readable()/writable() returns False. * Added "map" parameter for file_dispatcher. * file_wrapper: removed "return" in close(), recv/read and send/write swapped because of their nature. * mac code for writable() removed. Manual for accept() on mac is similar to the one on linux. * Repeating exception changed from "raise socket.error, why" to raise. * Added connected/accepting/addr reset on close(). Initialization of variables moved to __init__. * close_all() now calls close for dispatcher object, EBADF treated as already closed socket/file. * Added channel id to "unhandled..." messages. Bugs: * Fixed bug (654766,889153): client never gets connected, nor errored. Connecting client gets writable event from select(), however, some client may want always be non writable. Such client may never get connected. The fix adds _readable() - always True for accepting and always False for connecting socket; and _writable() - always False for accepting and always True for connecting socket. This implies, that listening dispatcher's readable() and writable() will never be called. ("man accept" and "man connect" for non-blocking sockets). * Fixed bug: error handling after accept(). It's said, that accept can return EWOULDBLOCK even for readable socket. This mean, that even after handle_accept(), dispatcher's accept() still raise EWOULDBLOCK. New code does accept() itself and stores accepted socket in self.__pending_accept. If there was socket.error, it's treated as EWOULDBLOCK. dispatcher's accept returns self.__pending_accept and resets it to None. Features: * Added pending_read() and pending_write(). The functions helps to use dispatcher over non socket objects with buffering capabilities. In original dispatcher, if socket makes buffered read and some data is in buffer, entering asyncore.poll() doesn't finishes, since there is no data in real file/socket. This feature allow to use SSL socket, since the socket reads data by 16k chunks. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-14 11:35 Message: Logged In: YES user_id=21627 Originator: NO Alexey, are you interested in revising your code until it is approved? Saying "no" is fine; the question then is if there is anybody else interested in working on this patch. If nobody is interested in working on it, it would be best rejected - there is no point in having it listed as open here for years (it would be sad, of course, for the work put into the patch, and the work put into the review). Andrew, do you have more changes from this patch that you consider worth incorporating? ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2007-01-08 21:44 Message: Logged In: YES user_id=410460 Originator: YES 1) The patch was developed not during some academic research - but during of coding true non-blocking client-server applications, capable to run both on Linux and Windows. Original code had a lot of issues with everything: some parts were not truly blocking, not every socket can be passed, issues with high load, etc. 2) We have used medusa for ssl capability in our project. However, it's impossible to get fully non-blocking functionality with original asyncore and original medusa. So functionality was extended to support these features as well. That is what idispatcher for. 3) In the end we have got pretty reliable code, which supports features I described here and has tons of bug and issues fixed. Again, I didn't fix bug for any academic purpose - every fix was driven by real issue we met during development. I don't also think, that these fixes bond to our project too tight - I believe I made them pretty general. 4) It's possible, that some parts can be made better for other application. But if you follow the same path - developing truly non-blocking client-server with medusa's ssl capabilities, - I think, you will end with the same thing. 5) I don't insist on including the patch into the python tree as is. I feel pretty well using modified asyncore in my private library. My intention was to share my experience. Please use, if you need to. 6) The development I mention above was 2004 year. So the patch is not in sync with this reality for 2 years already. Some issues it was solving can be gone already. I also don't know, what is going on with SSL for python - there seems to be new libraries as well. ...so... just use it as you want... or as you don't want ;) ... ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-01-07 05:53 Message: Logged In: YES user_id=341410 Originator: NO In asynchat, the only stuff that should be accepted is the handle_read() changes. The deque removal should be ignored (we have deques since Python 2.4, which are *significantly* faster than lists in nontrivial applications), the iasync_chat stuff, like the idispatcher stuff, seems unnecessary. And that's pretty much it for asynchat. The proposed asynchttp module shouldn't go into the Python standard library until it has lived on its own for a nontrival amount of time in the Cheeseshop and is found to be as good as httplib, urllib, or urllib2. Even then, its inclusion should be questioned, as medusa (the http server based on asyncore) has been around for a decade or more, is used many places, and yet still isn't in the standard library. The asyncoreTest.py needs a bit of work (I notice some incorrect names), but could be used as an addition to the test suite (currently it seems as though only asynchat is tested). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-01-07 05:42 Message: Logged In: YES user_id=341410 Originator: NO Many of the changes in the source provided by klimkin in his most recent revision from February 27, 2005 seek to solve certain problems in an inconsistent or incorrect way. Some of his changes (or variants thereof) are worthwhile. I'll start with my issues with his asyncore changes, then describe what I think should be added from them. For example, in his updated asyncore.py, the list of sockets is first shuffled randomly, then sorted based on priority. Assuming that one ignored priorities for a moment, if there were more sockets than the max sockets for the platform, then due to the limitations of randomness, there would be no guarantees that all sockets would get polled. Say, for example, that one were using windows and were running close to the actual select file handle limit (512 in Python 2.3) with 500 handles, you would skip 436 of the sockets *this pass*. In 10 passes, there would have been 100 sockets that were never polled. In 20 passes, there would still be, on average, 20 that were never polled. So this "randomization" step is the wrong thing to do, unless you actually make multiple select calls for each poll() call. But really, select is limited by 512, and I've run it with 500 without issue. The priority based sorting has much of the same problems, but it is even worse when you have nontrivial numbers of differing priorities, regardless of randomization or not. The max socket limit of 64 on Windows isn't correct. It's been 512 since at least Python 2.3 . And all other platforms being 65536? No. I've had some versions of linux die on me at 512, others at 4096, but all were dog slow beyond 500 or so. It's better to let the underlying system raise an exception for the user when it fails and let them attempt to tune it, rather than forcing a tuning that may not be correct. The "pending read" stuff is also misdirected. Assuming a non-broken async client or server, either should be handling content as it comes it, dispatching as necessary. See asynchat.collect_incoming_data() and asynchat.found_terminator() for examples. The idispatcher stuff seems unnecessary. Generally speaking, it seems to me that there are 3 levels of abstraction going on: 1) handle_*_event(), called by poll, poll2, etc. 2) handle_*(), called by handle_*_event(), user overrides, calls other handle_*() and *() methods 3) *() (aka recv, send, close, etc.), called by handle_*(), generally left alone. Some of your code breaks the abstraction and has items in layer 2 call items in layer 1, which then call items in layer 2 again. This seems unnecessary, and breaks the general downward calling semantic (except in the case of errors returned by layer 3 resulting in layer 2 handle_close() calls, which is the proper method to call). There are, according to my reading of the asyncore portions of your included module, a few things that may be worthy for inclusion into the Python standard library are: * A variant of your changes to close_all(), though it should proceed in closing everything unless a KeyboardInterrupt, SystemExit, or ExitNow exception is raised. Socket errors should be ignored, because we are closing them - we don't care about their error condition. * Checking sockets for socket error via socket.getsockopt() . * A variant of your .close() implementation. * The CONNRESET, etc., stuff in the send() and recv() methods, but not the handle_close_event() replacements, stick with handle_close() . * Checking for KeyboardInterrupt and SystemExit inside the poll functions. * The _closed_socket class and initialization. All but the last of the above, I would consider to be bugfixes, and if others agree that these are reasonable changes, I'll write up a patch against trunk and 2.5 maintenance. The last change, while I think would be nice, probably shouldn't be included in 2.5 maintenance, though I think would be fine for the trunk. ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2005-02-26 22:39 Message: Logged In: YES user_id=410460 Minor improvements: * Added handle_close_event(): calls handle_close(), then closes channel. No need to write self.close() in each handle_close (). * Improved exception handling. KeyboardInterrupt is not blocked. For python exception handle_error_event() is called, which checks for KeyboardInterrupt and closes socket, if handle_error didn't. Bugs: * Calling connect() could raise exception and doesn't hit handle_error(). Now if there was an exception, handle_error_event() is called. Features: * set_timeout(): Sets timeout for dispatcher object, if there was no io for the object, raises ETIMEDOUT, which handled by handle_error_event(). * Fixed issue with Windows - too many descriptors in select(). The list of sockets shuffled and only first asyncore.max_channels used in select(). * Added set_prio(): Sets priority for dispatcher. After shuffle the list of sockets sorted by priority. You may also check asynhttplib - asynchronous version of httplib. ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2004-07-02 15:44 Message: Logged In: YES user_id=410460 In addition to "[ 909005 ] asyncore fixes and improvements" and CVS version "asyncore.py,v 2.51" this patch provides: * Added handling of buffered socket layer (pending_read(), pending_write()). * Added fd number for __repr__. * Initialized self.socket = socket._closedsocket() instead of None for verbose error output (like closed socket.socket). * asyncore and asynchat implements idispatcher and iasync_chat. * Fixed self.addr initialization. * Removed import exceptions. * Don't filter KeyboardInterrupt, just pass through. * Added queue of sockets, solves the problem of select() on too many descriptors. I have run make test in python cvs distrib without problems. Examples of using i* included. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-06-05 19:54 Message: Logged In: YES user_id=11375 I've struggled to get the test suite running without errors on my machine, but have failed. ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2004-03-22 07:15 Message: Logged In: YES user_id=410460 There is no real reason for this change, please undo. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 21:18 Message: Logged In: YES user_id=11375 In your version of file_dispatch.__init__, the .set_file() call is moved earlier; can you say why? ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 21:13 Message: Logged In: YES user_id=11375 Added "map" parameter for file_dispatcher and dispatcher_with_send in CVS HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 21:08 Message: Logged In: YES user_id=11375 Repeating exception changes ('raise socket.error' -> just 'raise') checked into HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 21:02 Message: Logged In: YES user_id=11375 Mac code for writable() removed from HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 21:02 Message: Logged In: YES user_id=11375 Patch to use True/False applied to HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 20:55 Message: Logged In: YES user_id=11375 Fix for bug #887279 applied to HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 20:48 Message: Logged In: YES user_id=11375 The many number of changes in this patch make it difficult to figure out which changes fix which problem. I've created a new directory in CVS, nondist/sandbox/asyncore, that contains copies of the module with these patches applied, and will work on applying changes to the copy in dist/src. ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2004-03-17 08:15 Message: Logged In: YES user_id=410460 Sorry, unfortunately I have lost old patch file. I have atached new one. In addition to fixes, listed above, the patch includes: 1. Fix for operating on uninitialized socket. self.socket now initializes with _closed_socket(), so any operation throws EBADF. 2. Added class idispatcher - base class for dispatcher. The purpose of this class is to allow simple replacement of media(dispatcher interface) in classes, derived from dispatcher class. This is based on 'object'. I have also attached asynchat.diff - example for new-style dispatcher. Old asynchat works as well. ---------------------------------------------------------------------- Comment By: Wummel (calvin) Date: 2004-03-11 16:49 Message: Logged In: YES user_id=9205 There is no file attached! You have to click on the checkbox next to the upload filename. This is a Sourceforge annoyance :( ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=909005&group_id=5470 From noreply at sourceforge.net Wed Feb 14 11:55:22 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 02:55:22 -0800 Subject: [Patches] [ python-Patches-1126187 ] isapi.samples.advanced.py fix Message-ID: Patches item #1126187, was opened at 2005-02-17 20:43 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1126187&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Closed Resolution: Rejected Priority: 5 Private: No Submitted By: Philippe Kirsanov (phwp) Assigned to: Nobody/Anonymous (nobody) Summary: isapi.samples.advanced.py fix Initial Comment: This fix is to support win32 files in fotmat: \\?\drive:\dir\name.ext It's valid file format in win32. I've got that kind of file name trying to use __file__ when script is used as isapi extension. ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2007-02-14 11:55 Message: Logged In: YES user_id=11105 Originator: NO This patch is probably for py2exe. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-14 11:15 Message: Logged In: YES user_id=21627 Originator: NO It's not clear to me what the intention of this patch is: it is not a unified or context diff, so it doesn't apply to any of the Python source code files. The contributed advanced.py seems to have no relationship with any of the functionality included in Python. So I'm rejecting the patch as invalid. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1126187&group_id=5470 From noreply at sourceforge.net Wed Feb 14 13:13:44 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 04:13:44 -0800 Subject: [Patches] [ python-Patches-1658799 ] Handle requests to intern string subtype instances Message-ID: Patches item #1658799, was opened at 2007-02-13 10:34 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1658799&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Hrvoje Nik?i? (hniksic) Assigned to: Nobody/Anonymous (nobody) Summary: Handle requests to intern string subtype instances Initial Comment: This patch implements a small modification of PyString_InternInPlace that allows for safe interning of string subtype instances. The change should be fully backward compatible -- for a rationale and discussion, see: http://mail.python.org/pipermail/python-dev/2007-February/070973.html ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2007-02-14 12:13 Message: Logged In: YES user_id=4771 Originator: NO Ah, the code was the wrong way around. The following causes an Fatal Python error in a debug build: s1 = "hel" s1 = intern(s1 + "lo") class S(str): def __hash__(self): return 0 def __eq__(self, other): return False s = S(s1) s2 = intern(s) del s1 ---------------------------------------------------------------------- Comment By: Hrvoje Nik?i? (hniksic) Date: 2007-02-14 08:31 Message: Logged In: YES user_id=1718107 Originator: YES I don't think an attack is possible. This patch retains the property that only exact strings are interned. If you create a pathological string subtype that hashes like a different string instance (one that has already been interned), all you'll achieve is that "interning" will return the other instance. As far as I can tell, no string is actually removed from the interned dictionary (until it becomes unreachable, that is.) What is the expected result of your test code? I tried it and it ran without error. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2007-02-13 22:36 Message: Logged In: YES user_id=4771 Originator: NO I think that this opens an attack (untested, though): it allows a previously-interned string to be removed from the dictionary. This might lead to a crash because the old string still thinks it is interned. Try something along the lines of: s1 = "hel" s1 = intern(s1 + "lo") class S(str): def __hash__(self): return hash(s1) def __eq__(self, other): return other == s1 s = S("world") intern(s) del s1 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1658799&group_id=5470 From noreply at sourceforge.net Wed Feb 14 13:21:20 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 04:21:20 -0800 Subject: [Patches] [ python-Patches-1658799 ] Handle requests to intern string subtype instances Message-ID: Patches item #1658799, was opened at 2007-02-13 10:34 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1658799&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Hrvoje Nik?i? (hniksic) Assigned to: Nobody/Anonymous (nobody) Summary: Handle requests to intern string subtype instances Initial Comment: This patch implements a small modification of PyString_InternInPlace that allows for safe interning of string subtype instances. The change should be fully backward compatible -- for a rationale and discussion, see: http://mail.python.org/pipermail/python-dev/2007-February/070973.html ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2007-02-14 12:21 Message: Logged In: YES user_id=4771 Originator: NO Btw, any reason why you cannot simply say in your Python program: intern(str(s)) ? ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2007-02-14 12:13 Message: Logged In: YES user_id=4771 Originator: NO Ah, the code was the wrong way around. The following causes an Fatal Python error in a debug build: s1 = "hel" s1 = intern(s1 + "lo") class S(str): def __hash__(self): return 0 def __eq__(self, other): return False s = S(s1) s2 = intern(s) del s1 ---------------------------------------------------------------------- Comment By: Hrvoje Nik?i? (hniksic) Date: 2007-02-14 08:31 Message: Logged In: YES user_id=1718107 Originator: YES I don't think an attack is possible. This patch retains the property that only exact strings are interned. If you create a pathological string subtype that hashes like a different string instance (one that has already been interned), all you'll achieve is that "interning" will return the other instance. As far as I can tell, no string is actually removed from the interned dictionary (until it becomes unreachable, that is.) What is the expected result of your test code? I tried it and it ran without error. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2007-02-13 22:36 Message: Logged In: YES user_id=4771 Originator: NO I think that this opens an attack (untested, though): it allows a previously-interned string to be removed from the dictionary. This might lead to a crash because the old string still thinks it is interned. Try something along the lines of: s1 = "hel" s1 = intern(s1 + "lo") class S(str): def __hash__(self): return hash(s1) def __eq__(self, other): return other == s1 s = S("world") intern(s) del s1 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1658799&group_id=5470 From noreply at sourceforge.net Wed Feb 14 18:13:18 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 09:13:18 -0800 Subject: [Patches] [ python-Patches-1121142 ] ZipFile.open - read-only file-like obj for files in archive Message-ID: Patches item #1121142, was opened at 2005-02-11 19:08 Message generated for change (Comment added) made by alanmcintyre You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: Out of Date Priority: 5 Private: No Submitted By: Alan McIntyre (alanmcintyre) Assigned to: Nobody/Anonymous (nobody) Summary: ZipFile.open - read-only file-like obj for files in archive Initial Comment: I originally started working on updating patch 992750, but decided after a short while to just start from scratch, so I'm posting it as a new patch. Sorry if this isn't appropriate. This patch provides a new open() method on ZipFile; this method returns a file-like object for the requested item in the archive. This file-like object only provides a read() method. ZipFile.read was modified to use the new open method (this was suggested by loewis in reference to patch 992750). The patched zipfile.py passed the existing tests in the test_zipfile.py from CVS. New tests were added to verify the operation of the object returned by open(). These tests were modeled after existing tests for ZipFile.read(); two read fixed-size chunks from the file-like object, and two others read random-sized chunks. I have only run the tests on Windows XP, using Python2.4 from the official Windows installer. I will test the patch out on Linux over the weekend. If the patch is accepted I'll also generate and submit patches for the appropriate documentation as well. ---------------------------------------------------------------------- >Comment By: Alan McIntyre (alanmcintyre) Date: 2007-02-14 12:13 Message: Logged In: YES user_id=1115903 Originator: YES I will see if I can make some progress on this over the weekend. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 04:14 Message: Logged In: YES user_id=21627 Originator: NO Can you please update the patch to the current subversion trunk? I'd like to apply it, but the code of zipfile has changed so that the patch is currently out-of-date. When redoing it, notice that the read implementation has changed (I couldn't figure out how you moved code around). Please do use the trunk (not Python 2.5), and please submit the patch as a single 'svn diff' output (rather than a tar file containing multiple individual diff files). ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-05-30 21:56 Message: Logged In: YES user_id=1115903 Revision 5 of this patch has been in constant use with Python 2.4.1 in an application at my job for about a month; it seems to be stable and useful in that regard. If anybody has time to review and accept (or reject) it I would appreciate it. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-26 21:23 Message: Logged In: YES user_id=1115903 After testing on my large batch of large Zip files, I made one fix (version 4 of the patch didn't read all the content of large compressed archive items). The current set of changes is attached as zipfile_patch5.tgz. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-13 11:34 Message: Logged In: YES user_id=1115903 I found a problem with my universal newline handling code in readline(); if the first byte of an '\r\n' pair was read from file but the second byte didn't come in on that same read, it resulted in the next line having an inappropriate '\n' prepended to it. zipfile_patch4.tgz has a fix for this included (with everything else, of course). I'm going to test the open() capability in a real application with some reasonably large zip files (containing files up to just short of 2GB) over the next few days, so if any bugs or performance problems show up I may have some more changes. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-13 00:58 Message: Logged In: YES user_id=1115903 Uploaded the third revision of this patch; passes all regression tests against current CVS on WinXP. I think all the issues Martin brought up have been addressed except perhaps for the case of compression rate <1. I will have a look at that when I have time; just wanted to get an update here before the patch started to look abandoned. :) ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-03-14 09:37 Message: Logged In: YES user_id=1115903 Hmm...I could have sworn I did the diff in the correct order. I'll make sure next time. :) Here's my comments on your remarks (in order): - I'm adding support for universal newlines, and will reject all modes that aren't legal combinations of r, U, and b. - I'll see if I can make a Zip file store something with compression < 1, and if I can I'll add a test case for it. - I'll try work a .flush() in there on the compression object and come up with a test case if possible - .read(0) and .readline(0) will both return an empty string with no side-effects, since this seems to be what builtin files do. Right now ZipExtFile uses the ZipFile's file object, so you pretty much have to do whatever you're going to do with the ZipExtFile instance you get back from .open() before you use that ZipFile instance in a way that moves the file pointer around. I'm sure that somebody will eventually try to use the ZipFile in this way, so it appears my options are either to (1) give the ZipExtFile its own file object to use (when possible), or (2) make sure this limitation is documented. #1 feels (to me) to be the "right thing" to do, so that's what I'll try unless there's a good reason I shouldn't. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-03-01 02:59 Message: Logged In: YES user_id=21627 The patch is reversed: usually, diff is invoked as "-c old new". I think it is almost right, but I have a few remarks: - by tradition, open() should have a mode argument, defaulting to 'r'; it would be ok to raise exceptions if it is anything else. However, do consider implementing universal newlines; allowing 'b' as a no-op might also be reasonable. - I wonder what happens if the compression rate is < 1. It would appear that the code might use too few rawbytes. I suggest to recursively invoke read in this case. - I wonder whether it could ever happen that there is still data to uncompress in the zlib object, ie. whether it might be necessary to invoke .flush() after exhausting the rawbytes (and discard the zlib object afterwards) - it appears that atleast the builtin file object implements .read(0) as returning an empty string; the manual says that the entire file is meant only if size is omitted or negative. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-02-27 01:28 Message: Logged In: YES user_id=1115903 zipfile_patch2.tgz: I updated the file-like object to support readline, readlines, __iter__ and next(). Added tests for those new methods. Also added a patch for the documentation. Passes regression tests on 2.5a0 built from CVS HEAD with MSVC .NET on Windows XP. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 From noreply at sourceforge.net Wed Feb 14 19:54:47 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 10:54:47 -0800 Subject: [Patches] [ python-Patches-909005 ] asyncore fixes and improvements Message-ID: Patches item #909005, was opened at 2004-03-03 05:07 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=909005&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Alexey Klimkin (klimkin) Assigned to: A.M. Kuchling (akuchling) Summary: asyncore fixes and improvements Initial Comment: Minor: * 0/1 for boolean values replaced with False/True. * (887279) Added handling of POLLPRI as POLLIN. POLLERR, POLLHUP, POLLNVAL are handled as exception event. handle_expt_event gets recent error from self.socket object and raises socket.error. * Default readable()/writable() returns False. * Added "map" parameter for file_dispatcher. * file_wrapper: removed "return" in close(), recv/read and send/write swapped because of their nature. * mac code for writable() removed. Manual for accept() on mac is similar to the one on linux. * Repeating exception changed from "raise socket.error, why" to raise. * Added connected/accepting/addr reset on close(). Initialization of variables moved to __init__. * close_all() now calls close for dispatcher object, EBADF treated as already closed socket/file. * Added channel id to "unhandled..." messages. Bugs: * Fixed bug (654766,889153): client never gets connected, nor errored. Connecting client gets writable event from select(), however, some client may want always be non writable. Such client may never get connected. The fix adds _readable() - always True for accepting and always False for connecting socket; and _writable() - always False for accepting and always True for connecting socket. This implies, that listening dispatcher's readable() and writable() will never be called. ("man accept" and "man connect" for non-blocking sockets). * Fixed bug: error handling after accept(). It's said, that accept can return EWOULDBLOCK even for readable socket. This mean, that even after handle_accept(), dispatcher's accept() still raise EWOULDBLOCK. New code does accept() itself and stores accepted socket in self.__pending_accept. If there was socket.error, it's treated as EWOULDBLOCK. dispatcher's accept returns self.__pending_accept and resets it to None. Features: * Added pending_read() and pending_write(). The functions helps to use dispatcher over non socket objects with buffering capabilities. In original dispatcher, if socket makes buffered read and some data is in buffer, entering asyncore.poll() doesn't finishes, since there is no data in real file/socket. This feature allow to use SSL socket, since the socket reads data by 16k chunks. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-14 10:54 Message: Logged In: YES user_id=341410 Originator: NO If anything is to be included, it should be the tests (though they need to be rewritten a bit). I've been working on and off on a patch that includes other portions that I felt were worthwhile. When I finish the patch, I'll also include a fixed version of the tests. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-14 02:35 Message: Logged In: YES user_id=21627 Originator: NO Alexey, are you interested in revising your code until it is approved? Saying "no" is fine; the question then is if there is anybody else interested in working on this patch. If nobody is interested in working on it, it would be best rejected - there is no point in having it listed as open here for years (it would be sad, of course, for the work put into the patch, and the work put into the review). Andrew, do you have more changes from this patch that you consider worth incorporating? ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2007-01-08 12:44 Message: Logged In: YES user_id=410460 Originator: YES 1) The patch was developed not during some academic research - but during of coding true non-blocking client-server applications, capable to run both on Linux and Windows. Original code had a lot of issues with everything: some parts were not truly blocking, not every socket can be passed, issues with high load, etc. 2) We have used medusa for ssl capability in our project. However, it's impossible to get fully non-blocking functionality with original asyncore and original medusa. So functionality was extended to support these features as well. That is what idispatcher for. 3) In the end we have got pretty reliable code, which supports features I described here and has tons of bug and issues fixed. Again, I didn't fix bug for any academic purpose - every fix was driven by real issue we met during development. I don't also think, that these fixes bond to our project too tight - I believe I made them pretty general. 4) It's possible, that some parts can be made better for other application. But if you follow the same path - developing truly non-blocking client-server with medusa's ssl capabilities, - I think, you will end with the same thing. 5) I don't insist on including the patch into the python tree as is. I feel pretty well using modified asyncore in my private library. My intention was to share my experience. Please use, if you need to. 6) The development I mention above was 2004 year. So the patch is not in sync with this reality for 2 years already. Some issues it was solving can be gone already. I also don't know, what is going on with SSL for python - there seems to be new libraries as well. ...so... just use it as you want... or as you don't want ;) ... ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-01-06 20:53 Message: Logged In: YES user_id=341410 Originator: NO In asynchat, the only stuff that should be accepted is the handle_read() changes. The deque removal should be ignored (we have deques since Python 2.4, which are *significantly* faster than lists in nontrivial applications), the iasync_chat stuff, like the idispatcher stuff, seems unnecessary. And that's pretty much it for asynchat. The proposed asynchttp module shouldn't go into the Python standard library until it has lived on its own for a nontrival amount of time in the Cheeseshop and is found to be as good as httplib, urllib, or urllib2. Even then, its inclusion should be questioned, as medusa (the http server based on asyncore) has been around for a decade or more, is used many places, and yet still isn't in the standard library. The asyncoreTest.py needs a bit of work (I notice some incorrect names), but could be used as an addition to the test suite (currently it seems as though only asynchat is tested). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-01-06 20:42 Message: Logged In: YES user_id=341410 Originator: NO Many of the changes in the source provided by klimkin in his most recent revision from February 27, 2005 seek to solve certain problems in an inconsistent or incorrect way. Some of his changes (or variants thereof) are worthwhile. I'll start with my issues with his asyncore changes, then describe what I think should be added from them. For example, in his updated asyncore.py, the list of sockets is first shuffled randomly, then sorted based on priority. Assuming that one ignored priorities for a moment, if there were more sockets than the max sockets for the platform, then due to the limitations of randomness, there would be no guarantees that all sockets would get polled. Say, for example, that one were using windows and were running close to the actual select file handle limit (512 in Python 2.3) with 500 handles, you would skip 436 of the sockets *this pass*. In 10 passes, there would have been 100 sockets that were never polled. In 20 passes, there would still be, on average, 20 that were never polled. So this "randomization" step is the wrong thing to do, unless you actually make multiple select calls for each poll() call. But really, select is limited by 512, and I've run it with 500 without issue. The priority based sorting has much of the same problems, but it is even worse when you have nontrivial numbers of differing priorities, regardless of randomization or not. The max socket limit of 64 on Windows isn't correct. It's been 512 since at least Python 2.3 . And all other platforms being 65536? No. I've had some versions of linux die on me at 512, others at 4096, but all were dog slow beyond 500 or so. It's better to let the underlying system raise an exception for the user when it fails and let them attempt to tune it, rather than forcing a tuning that may not be correct. The "pending read" stuff is also misdirected. Assuming a non-broken async client or server, either should be handling content as it comes it, dispatching as necessary. See asynchat.collect_incoming_data() and asynchat.found_terminator() for examples. The idispatcher stuff seems unnecessary. Generally speaking, it seems to me that there are 3 levels of abstraction going on: 1) handle_*_event(), called by poll, poll2, etc. 2) handle_*(), called by handle_*_event(), user overrides, calls other handle_*() and *() methods 3) *() (aka recv, send, close, etc.), called by handle_*(), generally left alone. Some of your code breaks the abstraction and has items in layer 2 call items in layer 1, which then call items in layer 2 again. This seems unnecessary, and breaks the general downward calling semantic (except in the case of errors returned by layer 3 resulting in layer 2 handle_close() calls, which is the proper method to call). There are, according to my reading of the asyncore portions of your included module, a few things that may be worthy for inclusion into the Python standard library are: * A variant of your changes to close_all(), though it should proceed in closing everything unless a KeyboardInterrupt, SystemExit, or ExitNow exception is raised. Socket errors should be ignored, because we are closing them - we don't care about their error condition. * Checking sockets for socket error via socket.getsockopt() . * A variant of your .close() implementation. * The CONNRESET, etc., stuff in the send() and recv() methods, but not the handle_close_event() replacements, stick with handle_close() . * Checking for KeyboardInterrupt and SystemExit inside the poll functions. * The _closed_socket class and initialization. All but the last of the above, I would consider to be bugfixes, and if others agree that these are reasonable changes, I'll write up a patch against trunk and 2.5 maintenance. The last change, while I think would be nice, probably shouldn't be included in 2.5 maintenance, though I think would be fine for the trunk. ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2005-02-26 13:39 Message: Logged In: YES user_id=410460 Minor improvements: * Added handle_close_event(): calls handle_close(), then closes channel. No need to write self.close() in each handle_close (). * Improved exception handling. KeyboardInterrupt is not blocked. For python exception handle_error_event() is called, which checks for KeyboardInterrupt and closes socket, if handle_error didn't. Bugs: * Calling connect() could raise exception and doesn't hit handle_error(). Now if there was an exception, handle_error_event() is called. Features: * set_timeout(): Sets timeout for dispatcher object, if there was no io for the object, raises ETIMEDOUT, which handled by handle_error_event(). * Fixed issue with Windows - too many descriptors in select(). The list of sockets shuffled and only first asyncore.max_channels used in select(). * Added set_prio(): Sets priority for dispatcher. After shuffle the list of sockets sorted by priority. You may also check asynhttplib - asynchronous version of httplib. ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2004-07-02 06:44 Message: Logged In: YES user_id=410460 In addition to "[ 909005 ] asyncore fixes and improvements" and CVS version "asyncore.py,v 2.51" this patch provides: * Added handling of buffered socket layer (pending_read(), pending_write()). * Added fd number for __repr__. * Initialized self.socket = socket._closedsocket() instead of None for verbose error output (like closed socket.socket). * asyncore and asynchat implements idispatcher and iasync_chat. * Fixed self.addr initialization. * Removed import exceptions. * Don't filter KeyboardInterrupt, just pass through. * Added queue of sockets, solves the problem of select() on too many descriptors. I have run make test in python cvs distrib without problems. Examples of using i* included. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-06-05 10:54 Message: Logged In: YES user_id=11375 I've struggled to get the test suite running without errors on my machine, but have failed. ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2004-03-21 22:15 Message: Logged In: YES user_id=410460 There is no real reason for this change, please undo. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 12:18 Message: Logged In: YES user_id=11375 In your version of file_dispatch.__init__, the .set_file() call is moved earlier; can you say why? ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 12:13 Message: Logged In: YES user_id=11375 Added "map" parameter for file_dispatcher and dispatcher_with_send in CVS HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 12:08 Message: Logged In: YES user_id=11375 Repeating exception changes ('raise socket.error' -> just 'raise') checked into HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 12:02 Message: Logged In: YES user_id=11375 Mac code for writable() removed from HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 12:02 Message: Logged In: YES user_id=11375 Patch to use True/False applied to HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 11:55 Message: Logged In: YES user_id=11375 Fix for bug #887279 applied to HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 11:48 Message: Logged In: YES user_id=11375 The many number of changes in this patch make it difficult to figure out which changes fix which problem. I've created a new directory in CVS, nondist/sandbox/asyncore, that contains copies of the module with these patches applied, and will work on applying changes to the copy in dist/src. ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2004-03-16 23:15 Message: Logged In: YES user_id=410460 Sorry, unfortunately I have lost old patch file. I have atached new one. In addition to fixes, listed above, the patch includes: 1. Fix for operating on uninitialized socket. self.socket now initializes with _closed_socket(), so any operation throws EBADF. 2. Added class idispatcher - base class for dispatcher. The purpose of this class is to allow simple replacement of media(dispatcher interface) in classes, derived from dispatcher class. This is based on 'object'. I have also attached asynchat.diff - example for new-style dispatcher. Old asynchat works as well. ---------------------------------------------------------------------- Comment By: Wummel (calvin) Date: 2004-03-11 07:49 Message: Logged In: YES user_id=9205 There is no file attached! You have to click on the checkbox next to the upload filename. This is a Sourceforge annoyance :( ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=909005&group_id=5470 From noreply at sourceforge.net Wed Feb 14 22:43:21 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 13:43:21 -0800 Subject: [Patches] [ python-Patches-1144636 ] Make urllib2.OpenerDirector instances pickle-able Message-ID: Patches item #1144636, was opened at 2005-02-20 02:48 Message generated for change (Comment added) made by jjlee You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1144636&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 >Status: Closed Resolution: None Priority: 5 Private: No Submitted By: John J Lee (jjlee) Assigned to: Nobody/Anonymous (nobody) Summary: Make urllib2.OpenerDirector instances pickle-able Initial Comment: urllib2.OpenerDirector would be pickleable if it were not for the fact that a couple of urllib2 handlers (which build_opener() adds by default) have function instance attributes. It's trivial to replace these instance attributes with a method that delegates to the appropriate function, thus allowing pickling. ---------------------------------------------------------------------- >Comment By: John J Lee (jjlee) Date: 2007-02-14 21:43 Message: Logged In: YES user_id=261020 Originator: YES I am no longer convinced that the intended change is an improvement, so I am closing this tracker item. FWIW, mysteriously, your unit test always *passes* for me (both before and after applying the patch). (And before svn updating from roughly rev 53200, your unit test fails for me before applying the patch, and passes for me after applying the patch.) For the record, the reasons I'm no longer convinced that pickle-ability of urllib2.OpenerDirector instances is a good thing are: 1. I don't think a pickle-ability constraint should be imposed on subtypes 2. I'm not convinced now that the ability to pickle these instances is terribly useful (although I did file the patch as a result of a user request) ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-14 10:27 Message: Logged In: YES user_id=21627 Originator: NO The patch doesn't fix the problem; the opener remains unpicklable. If you are still interested in this patch, please revise it accordingly; if not, please close it. When you revise it, please include a test case, like this one: def test_1144636(self): # Make sure OpenerDirector instances can be pickled import pickle try: pickle.dumps(urllib2.build_opener()) except pickle.PicklingError: self.fail("cannot pickle OpenerDirector instances") ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2005-03-04 18:46 Message: Logged In: YES user_id=261020 Hmm, maybe this fault (can't pickle) will be fixed without need for this patch in 2.5, due to the change to unbound methods that was discussed on python-dev. I don't understand that issue, so I can't say. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1144636&group_id=5470 From noreply at sourceforge.net Thu Feb 15 01:17:07 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 16:17:07 -0800 Subject: [Patches] [ python-Patches-1660179 ] functools.compose to chain functions together Message-ID: Patches item #1660179, was opened at 2007-02-14 19:17 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Chris AtLee (catlee) Assigned to: Nobody/Anonymous (nobody) Summary: functools.compose to chain functions together Initial Comment: The motivation for this patch was to be able to chain together operator.itemgetter objects so that I could sort a list of items that have the form ((a,b), c) by the 'a' or 'b' part of the item. Sorting by 'b' can be accomplished with: l.sort(key=compose(itemgetter(1), itemgetter(0))) compose(itemgetter(1), itemgetter(0)) is equivalent to def c(v): return itemgetter(1)(itemgetter(0)(v)) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 From noreply at sourceforge.net Thu Feb 15 02:26:51 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 14 Feb 2007 17:26:51 -0800 Subject: [Patches] [ python-Patches-1660202 ] IDLE doesn't save files on closing Message-ID: Patches item #1660202, was opened at 2007-02-15 02:26 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660202&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Toni (miargep142) Assigned to: Nobody/Anonymous (nobody) Summary: IDLE doesn't save files on closing Initial Comment: There is a bug in the 'Save on Close'-dialog in the idle editor. When clicking the close-button in an edited file, no matter if the user chooses 'yes', 'no' or 'cancel' in the appearing 'Save on Close'-dialog, the effect is the same: the window closes and changes are not written to the edited file. If in idlelib/EditorWindow.py, in function 'close(self)', the line if reply != "cancel": is replaced by if str(reply) != "cancel": the 'yes', 'no' and 'cancel' buttons work as expected. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660202&group_id=5470 From noreply at sourceforge.net Thu Feb 15 10:41:21 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 01:41:21 -0800 Subject: [Patches] [ python-Patches-1448297 ] decorator module Message-ID: Patches item #1448297, was opened at 2006-03-12 14:35 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1448297&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Georg Brandl (gbrandl) Assigned to: Nobody/Anonymous (nobody) Summary: decorator module Initial Comment: This is a first attempt to a decorator standard module containing only @decorator in a C implementation. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 10:41 Message: Logged In: YES user_id=21627 Originator: NO Why does this implementation have to be in C? I can't see any performance reason - this is all import-time stuff, and not computationally expensive, no? I'm otherwise fine with adding it to 2.6 (perhaps with a reference in the Python implementation where the C version is stored -ie in this issue- in case anybody ever wants to do this in C again). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1448297&group_id=5470 From noreply at sourceforge.net Thu Feb 15 10:53:17 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 01:53:17 -0800 Subject: [Patches] [ python-Patches-1397848 ] dictnotes.txt Message-ID: Patches item #1397848, was opened at 2006-01-05 16:59 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1397848&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Documentation Group: None >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Jim Jewett (jimjjewett) Assigned to: Raymond Hettinger (rhettinger) Summary: dictnotes.txt Initial Comment: Please add the reasoning behind no-resize-on-shrinkage. This replacement version does so by adding two additional tunable parameters and one paragraph at the end of the tunable parameters section. It does document that the current CPython does not use (or even directly represent) these parameters, and explains why. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 10:53 Message: Logged In: YES user_id=21627 Originator: NO Thanks for the patch, applied as r53782. I also added a sentence mentioning an algorithm that completely empties a dictionary and sees no resizing while doing so. Please always submit patches as unified diffs; I had to undo two typos that had been fixed since you edited the file. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1397848&group_id=5470 From noreply at sourceforge.net Thu Feb 15 10:55:19 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 01:55:19 -0800 Subject: [Patches] [ python-Patches-1402289 ] Allow mappings as globals (was: Fix dictionary subclass ...) Message-ID: Patches item #1402289, was opened at 2006-01-11 01:24 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1402289&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: crutcher (crutcher_gmail) Assigned to: Raymond Hettinger (rhettinger) Summary: Allow mappings as globals (was: Fix dictionary subclass ...) Initial Comment: There is an inconsistancy in the way that dictionary subclasses behave when they are used as as namespaces in execs. Basically, while python 2.4 permits the usage of dictionary subclasses for local environments, it still bypasses the subclass functions and uses the C API for global environments. The attached patch (and unittest!) addresses this issue. I'm pretty sure we keep the fast path in this. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 10:55 Message: Logged In: YES user_id=21627 Originator: NO Raymond, do you think you can make it this March? If not, please unassign. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-03-30 20:12 Message: Logged In: YES user_id=1424288 Are you going to get to this? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2006-02-21 14:28 Message: Logged In: YES user_id=80475 Crutcher, this looked good on my first read-through. Will go through it in detail sometime in March. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-30 20:10 Message: Logged In: YES user_id=1424288 doh, forgot to check the 'upload' box ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-30 18:08 Message: Logged In: YES user_id=1424288 I have reworked and extended this patch so that arbitrary mappings are permitted for globals, not just dictionary subtypes. This touched a good deal more code. Please use the updated version. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2006-01-25 19:20 Message: Logged In: YES user_id=4771 I will review and check-in your patch, I think it's a good idea despite the added code complexity. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-25 04:38 Message: Logged In: YES user_id=1424288 currently chasing through the PyFrame code and the EXEC_STMT code to make globals generic mappings, will update this patch yet again. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-22 10:19 Message: Logged In: YES user_id=1424288 I've fixed up the exception case, and extended the test case to check for it. Is there anything else I can do to get this in? ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-22 06:11 Message: Logged In: YES user_id=1424288 I'm going to fix the missed exception test this weekend, and try to get an updated patch to you. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2006-01-14 10:54 Message: Logged In: YES user_id=4771 Provided this can be done with no measurable performance hit, I guess that I'm fine with the idea. The patch needs a bit more work, though: I don't see why it should accept dict subclasses as globals but not arbitrary mappings (as it now does for the locals). This is mainly an issue of removing a few checks in various places, like EXEC_STMT and the eval() and execfile() built-ins. There is a missing exception check/clear in the part about LOAD_NAME, after the PyObject_GetItem(f->f_globals, w). A side note: in the current trunk already, LOAD_GLOBAL contains a couple of checks, namely PyString_CheckExact() and hash != -1. We might be able to prove in advance that these two conditions are always true. We could then remove the checks. Not sure the difference measurable, though. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-12 08:50 Message: Logged In: YES user_id=1424288 Well, why fix it for eval but not for exec? I don't think the time hit is noticeable, I ran 'time make test' twice each on the trunk with and without the patch. Here are the results: Trunk: real 9m17.117s user 3m30.930s sys 0m35.417s real 9m9.471s user 3m31.484s sys 0m34.978s Patch: real 9m32.469s user 3m40.134s sys 0m36.140s real 9m6.779s user 3m27.529s sys 0m34.716s ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-01-12 07:11 Message: Logged In: YES user_id=21627 I think the history is this: it originates from python.org/sf/215126, which requested that you can pass dict subtypes to eval. Armin noted that eval will always produce LOAD/STORE_NAME, so just modifying these opcodes is sufficient to fulfil the feature request. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2006-01-12 02:21 Message: Logged In: YES user_id=80475 Okay, I will take it under consideration. Also, I'll try to remember the reason it wasn't done the first time around. No promises though -- by itself consistency is a weak motivation. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-12 00:11 Message: Logged In: YES user_id=1424288 Here's a more interesting example. This works fine, unless the variables fall through to the global dictionary, or some code marks them global. import code class ManagedVariable: def get(self): return None def set(self, value): pass def delete(self): # Return false to stop the delete. return True class ManagedEnvironment(dict): def __setitem__(self, key, value): if self.has_key(key): if isinstance(dict.__getitem__(self, key), ManagedVariable): dict.__getitem__(self, key).set(value) return dict.__setitem__(self, key, value) def __getitem__(self, key): if self.has_key(key): if isinstance(dict.__getitem__(self, key), ManagedVariable): return dict.__getitem__(self, key).get() return dict.__getitem__(self, key) def __delitem__(self, key): if self.has_key(key): if isinstance(dict.__getitem__(self, key), ManagedVariable): if not dict.__getitem__(self, key).delete(): return dict.__delitem__(self, key) class RangedInt(ManagedVariable): def __init__(self, value, (low, high)): self.value = value self.low = low self.high = high def get(self): return self.value def set(self, value): if value < self.low: value = self.low if value > self.high: value = self.high self.value = value class FunctionValue(ManagedVariable): def __init__(self, get_func = None, set_func = None, del_func = None): self.get_func = get_func self.set_func = set_func self.del_func = del_func def get(self): if self.get_func: return self.get_func() return None def set(self, value): if self.set_func: self.set_func(value) def delete(self): if self.del_func: return self.del_func() return True class Constant(ManagedVariable): def __init__(self, value): self.value = value def get(self): return self.value def delete(self): return False import time d = ManagedEnvironment() d['ranged'] = RangedInt(1, (0, 100)) d['time'] = FunctionValue(lambda: time.time()) d['constant'] = Constant(42) code.interact(local=d) ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-11 19:12 Message: Logged In: YES user_id=1424288 Here's an example of a little typed environment. It's not the most robust, but it gets you thinking. import code class TypedDictionary(dict): def __setitem__(self, key, value): if self.has_key(key): t = type(self[key]) if t != type(value): try: value = t(value) except Exception: raise TypeError, \ "illegal assignment to '%s':" \ " %s cannot be coerced to %s" \ % (key, type(value), t) dict.__setitem__(self, key, value) code.interact(local=TypedDictionary()) ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-11 03:16 Message: Logged In: YES user_id=1424288 With the ability to use dictionary subclasses for local and global environments, it becomes very easy to implement execution environments with extended variable semantics for special purposes. This includes driving interactive processes, config file processors, type checking, lazy object construction, and read-only variables. As it currently stands, the semantics are inconsistent. The local case should not have been changed if the global case was not going to be changed. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2006-01-11 03:03 Message: Logged In: YES user_id=80475 Do you have any use cases? AFAICT, there were no unmet needs with locals option. Also, there was a reason that globals weren't included but I remember what it was. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1402289&group_id=5470 From noreply at sourceforge.net Thu Feb 15 10:58:27 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 01:58:27 -0800 Subject: [Patches] [ python-Patches-1660202 ] IDLE doesn't save files on closing Message-ID: Patches item #1660202, was opened at 2007-02-15 01:26 Message generated for change (Comment added) made by gbrandl You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660202&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: IDLE Group: None >Status: Closed >Resolution: Out of Date >Priority: 7 Private: No Submitted By: Toni (miargep142) Assigned to: Nobody/Anonymous (nobody) Summary: IDLE doesn't save files on closing Initial Comment: There is a bug in the 'Save on Close'-dialog in the idle editor. When clicking the close-button in an edited file, no matter if the user chooses 'yes', 'no' or 'cancel' in the appearing 'Save on Close'-dialog, the effect is the same: the window closes and changes are not written to the edited file. If in idlelib/EditorWindow.py, in function 'close(self)', the line if reply != "cancel": is replaced by if str(reply) != "cancel": the 'yes', 'no' and 'cancel' buttons work as expected. ---------------------------------------------------------------------- >Comment By: Georg Brandl (gbrandl) Date: 2007-02-15 09:58 Message: Logged In: YES user_id=849994 Originator: NO This has already been fixed in the SVN trunk, rev. 53375. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660202&group_id=5470 From noreply at sourceforge.net Thu Feb 15 11:03:17 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 02:03:17 -0800 Subject: [Patches] [ python-Patches-1151323 ] New fpconst module Message-ID: Patches item #1151323, was opened at 2005-02-24 21:06 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1151323&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Gregory Warnes (warnes) Assigned to: Nobody/Anonymous (nobody) Summary: New fpconst module Initial Comment: This patch adds the 'fpconst' module, described in PEP754 to the standard python library. It includes both the source code and the documentation. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 11:03 Message: Logged In: YES user_id=21627 Originator: NO This being a PEP, it can't go into Python without the PEP being accepted first, by BDFL pronouncement (see PEP 1). So as long as this hasn't happened, this patch cannot go in. ---------------------------------------------------------------------- Comment By: Gregory Warnes (warnes) Date: 2006-02-21 17:10 Message: Logged In: YES user_id=9316 Yes, there is still interest in getting this into 2.5. ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2006-02-20 13:04 Message: Logged In: YES user_id=849994 Is there an interest of getting this into 2.5? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-07-14 14:27 Message: Logged In: YES user_id=6656 For Python 2.5 you should indeed be able to use the "' without success before resorting to the big_endian_ check... -Greg ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2005-02-25 01:25 Message: Logged In: YES user_id=139309 Is there a reason why you don't use the struct.pack endian markers? You could get rid of all the code that deals with _big_endian if you just prefixed the formats with ! or > ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1151323&group_id=5470 From noreply at sourceforge.net Thu Feb 15 11:14:44 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 02:14:44 -0800 Subject: [Patches] [ python-Patches-1020188 ] Use Py_CLEAR where necessary to avoid crashes Message-ID: Patches item #1020188, was opened at 2004-09-01 07:35 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1020188&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None Status: Open Resolution: Accepted Priority: 5 Private: No Submitted By: Dima Dorfman (ddorfman) Assigned to: Michael Hudson (mwh) Summary: Use Py_CLEAR where necessary to avoid crashes Initial Comment: The Py_CLEAR macro was introduced to make it less tempting to write incorrect code in the form Py_DECREF(self->field); self->field = NULL; because that can expose a half-destroyed object if deallocation can cause self->field to be read. This patch fixes mistakes like this that still exist in the core. Only cases that are potentially dangerous are fixed in this patch. If self->field can only reference a special kind of [builtin] object, then it's just a style bug because we know that the builtin object isn't evil. Likewise if the code is operating on an automatic variable. It might be nice to fix those style bugs anyway, to discourage the broken idiom. Just for kicks, here's a way to use this bug in reversed to crash the interpreter: import array, gc, weakref a = array.array('c') wr = weakref.ref(a, lambda _: gc.get_referents(rev)) rev = reversed(a) del a list(rev) For me, this crashes immediately with a debug build and after a couple tries otherwise. Also attached is clearcand.py to help find these cases. It's not comprehensive, but it's a start. Usage: $ find src -name '*.c' | python clearcand.py | fgrep -- '->' The patch requires SF #1020185 to be applied for genobject.c to compile without warnings. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 11:14 Message: Logged In: YES user_id=21627 Originator: NO What's the status of this? Does anybody want to incorporate the test script? There are still a few cases left (or readded since the patch was originally made), in Modules (_bsddb, _sqlite, sv, _CarbonEvt), and in unicodeobject. mwh, if you lost interest, please unassign. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-09-27 20:50 Message: Logged In: YES user_id=4771 The grep may miss other expressions, like Py_DECREF(items[index]). There is also the dreadful case of macros: Py_DECREF(x) used in a macro which is called with self->something as x. In other words I am not sure the script can replace a (really motivated) programmer's extensive review... ---------------------------------------------------------------------- Comment By: Dima Dorfman (ddorfman) Date: 2004-09-10 16:29 Message: Logged In: YES user_id=908995 Okay, I'll post a cleaned up script this weekend. About your improvements: 1. Py_CLEAR already has the same semantics as a would-be Py_XCLEAR--it doesn't do anything if its argument is null. Py_XDECREF should definitely be checked for, though. 2. Py_DECREF(x) is probably safe; this is what I meant when I talked about operating on automatic variables in the initial comment. The grep for "->" filters out matches that are safe in this manner. It should be noted that this isn't necessarily safe, just that it's almost certainly so. In particular, it's not safe in this case: PyObject *x = ...; /* self is a struct { PyObject **something; ... } */ self->something = &x; Py_DECREF(x); x = NULL; but doing that is just insane. The grep would also miss ((*self).something), but I didn't expect to see any of those. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-10 16:09 Message: Logged In: YES user_id=6656 Yeah, I've come to a similar conclusion (finally got round to thinking about this properly this morning). I think a cleaned up script would be a good idea. Two improvements (which you may well have made already): Py_XDECREF is vulnerable too (do we want Py_XCLEAR??) and Py_DECREF(x); x == NULL is ok AFAICT when x is an identifier. ---------------------------------------------------------------------- Comment By: Dima Dorfman (ddorfman) Date: 2004-09-10 15:52 Message: Logged In: YES user_id=908995 I didn't write specific tests for these since they really would be testing for one particular line of code. It's not really hard to write them, though; I think my sample using reversed can be easily adapted to iterobject and itertools. I'm not sure whether genobject can be exploited like this, and I don't think it's worth finding out just to write a test case. On the other hand, if you (anyone) think it's worth it, I'll clean up clearcand.py not to require a pipeline and it can be added to the toolbox. For that to make sense, though, we should probably fix the style bugs that it catches so it can be reasonable to expect no output. Here are the outstanding matches that dereference something using "->": src/Modules/_bsddb.c 746 'Py_DECREF(self->myenvobj);\nself->myenvobj = NULL;' src/Modules/_bsddb.c 781 'Py_DECREF(self->myenvobj);\nself->myenvobj = NULL;' src/Modules/_bsddb.c 786 'Py_DECREF(self->associateCallback); \nself->associateCallback = NULL;' src/Modules/_bsddb.c 1184 'Py_DECREF(self->associateCallback); \nself->associateCallback = NULL;' src/Modules/svmodule.c 281 'Py_DECREF(self->ob_svideo); \nself->ob_svideo = NULL;' src/Mac/Modules/carbonevt/_CarbonEvtmodule.c 1060 'Py_DECREF(_self->ob_callback);\n_self->ob_callback = NULL;' src/Objects/unicodeobject.c 161 'Py_DECREF(unicode->defenc); \nunicode->defenc = NULL;' src/Objects/unicodeobject.c 250 'Py_DECREF(unicode->defenc); \nunicode->defenc = NULL;' unicode and sv are safe, and bsddb is on my list to look at. It shouldn't be harmful to just change all of those to Py_CLEAR, though. I can cobble up a patch for that, too. Opinions? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-08 19:33 Message: Logged In: YES user_id=80475 FWIW, rather than add specific testcases (closing the barndoor after the horse has escaped), it might be a good idea to fixup the OP's search code and add it to the toolbox. Future errors of this type are better caught through code scans than by testing stuff that already works fine. Of course, if highly motivated, you can do both :-) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-01 14:32 Message: Logged In: YES user_id=6656 Test cases would be nice. If it's easy for someone whose thought about it, that would be great, if not assign to me and I'll get to it... ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-01 09:06 Message: Logged In: YES user_id=80475 Accepted and applied. See: Include/object.h 2.129 Modules/itertoolsmodule.c 1.35 Objects/enumobject.c 1.18 Objects/genobject.c 1.4 Objects/iterobject.c 1.19 Thanks for the patch. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1020188&group_id=5470 From noreply at sourceforge.net Thu Feb 15 11:25:28 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 02:25:28 -0800 Subject: [Patches] [ python-Patches-1660179 ] functools.compose to chain functions together Message-ID: Patches item #1660179, was opened at 2007-02-15 00:17 Message generated for change (Comment added) made by gbrandl You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Chris AtLee (catlee) >Assigned to: Martin v. L?wis (loewis) Summary: functools.compose to chain functions together Initial Comment: The motivation for this patch was to be able to chain together operator.itemgetter objects so that I could sort a list of items that have the form ((a,b), c) by the 'a' or 'b' part of the item. Sorting by 'b' can be accomplished with: l.sort(key=compose(itemgetter(1), itemgetter(0))) compose(itemgetter(1), itemgetter(0)) is equivalent to def c(v): return itemgetter(1)(itemgetter(0)(v)) ---------------------------------------------------------------------- >Comment By: Georg Brandl (gbrandl) Date: 2007-02-15 10:25 Message: Logged In: YES user_id=849994 Originator: NO The patch looks good, is complete (one nit, "Returns" should be "Return" in the doc) and the functionality belongs into functools. What do you think, Martin? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 From noreply at sourceforge.net Thu Feb 15 11:38:14 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 02:38:14 -0800 Subject: [Patches] [ python-Patches-1448297 ] decorator module Message-ID: Patches item #1448297, was opened at 2006-03-12 13:35 Message generated for change (Comment added) made by gbrandl You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1448297&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.5 >Status: Closed >Resolution: Out of Date Priority: 5 Private: No Submitted By: Georg Brandl (gbrandl) Assigned to: Nobody/Anonymous (nobody) Summary: decorator module Initial Comment: This is a first attempt to a decorator standard module containing only @decorator in a C implementation. ---------------------------------------------------------------------- >Comment By: Georg Brandl (gbrandl) Date: 2007-02-15 10:38 Message: Logged In: YES user_id=849994 Originator: YES I think this is now outdated anyway with the inclusion of wraps() in functools -- it solves the same problem in a more general way. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 09:41 Message: Logged In: YES user_id=21627 Originator: NO Why does this implementation have to be in C? I can't see any performance reason - this is all import-time stuff, and not computationally expensive, no? I'm otherwise fine with adding it to 2.6 (perhaps with a reference in the Python implementation where the C version is stored -ie in this issue- in case anybody ever wants to do this in C again). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1448297&group_id=5470 From noreply at sourceforge.net Thu Feb 15 12:29:04 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 03:29:04 -0800 Subject: [Patches] [ python-Patches-1660500 ] Hide iteration variable in list comprehensions Message-ID: Patches item #1660500, was opened at 2007-02-15 21:29 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660500&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Parser/Compiler Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nick Coghlan (ncoghlan) Assigned to: Georg Brandl (gbrandl) Summary: Hide iteration variable in list comprehensions Initial Comment: This patch hides the iteration variable in list comprehensions. It adds new tests (modelled on the generator expression tests) and also removes some del statements from the standard library (where code previously cleaned up its own namespace). The changes to symtable.[ch] are more significant than strictly necessary - I found it necessary to spend some time cleaning up the code in order to understand what was needed for the list comprehension changes. Given that the 2.x and 3.0 compilers have already diverged fairly significantly, I don't believe this will make the process of keeping them in sync any more difficult than it is already. Assigning to Georg for initial review (as his set comprehensions patch provided a great deal of inspiration for this one). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660500&group_id=5470 From noreply at sourceforge.net Thu Feb 15 12:30:20 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 03:30:20 -0800 Subject: [Patches] [ python-Patches-1494140 ] Documentation for new Struct object Message-ID: Patches item #1494140, was opened at 2006-05-24 09:26 Message generated for change (Comment added) made by gbrandl You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1494140&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Documentation Group: Python 2.5 >Status: Closed >Resolution: Accepted Priority: 6 Private: No Submitted By: Bob Ippolito (etrepum) Assigned to: Nobody/Anonymous (nobody) Summary: Documentation for new Struct object Initial Comment: The performance enhancements to the struct module (patch #1493701) are implemented by having a Struct object, which is a compiled structure. This text file documents these new struct objects. ---------------------------------------------------------------------- >Comment By: Georg Brandl (gbrandl) Date: 2007-02-15 11:30 Message: Logged In: YES user_id=849994 Originator: NO I completed the patch and committed it as rev. 53785, 53786 (2.5). ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2007-01-04 06:55 Message: Logged In: YES user_id=33168 Originator: NO Even if this only documents part of the API, it seems like it would be better to get that in and finish it off later. Anyone know what's going on with this? ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2006-10-29 09:28 Message: Logged In: YES user_id=849994 What's the status of this? It should have been in 2.5 final... ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2006-08-02 07:38 Message: Logged In: YES user_id=849994 New/renamed functions need a \versionadded/changed. For StructObjects, I'd suggest a sentence like "Struct objects are new in version 2.5" at the top of the section. There's no explanation how to create a Struct object. The constructor must be explained, preferably on the module overview page. Isn't the type name "Struct"? ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2006-07-30 17:33 Message: Logged In: YES user_id=671362 > Does this patch still need to be updated for pack_to() I suppose so and hence updated my patch. (1) document pack_into(pack_to is renamed to pack_into). (2) document pack_into/pack_from as module functions too(just like re module) As for the function name change, I've already updated "what's new in 2.5" in r50985. I guess the patch is ready to be applied. Reviews are appreciated. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2006-07-29 19:28 Message: Logged In: YES user_id=11375 Does this patch still need to be updated for pack_to(), or can it just be applied? ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2006-07-10 17:26 Message: Logged In: YES user_id=671362 Patch for the TeX style doc. Bob, can you work on updating the main section right after 2.5 b2? ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2006-05-26 13:05 Message: Logged In: YES user_id=139309 We're going to need to revise this patch some more to document the new pack_to function (for Martin Blais' hotbuf work) Additionally we'll probably also want to revise the main struct documentation to talk about bounds checking and avoiding the creation of long objects. ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2006-05-25 14:32 Message: Logged In: YES user_id=139309 That's clearly a typo. I've attached a new version of the patch that removes those two letters. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-05-24 21:03 Message: Logged In: YES user_id=764593 Shouldn't self.size be the number of bytes required to *pack * the structure? The number required to *unpack* seems like it ought to include tuple overhead and such... ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2006-05-24 15:35 Message: Logged In: YES user_id=139309 New patch attached, fixed unpack documentation, added unpack_from method. ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2006-05-24 14:54 Message: Logged In: YES user_id=139309 Hold up on this patch, I need to revise it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1494140&group_id=5470 From noreply at sourceforge.net Thu Feb 15 12:53:54 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 03:53:54 -0800 Subject: [Patches] [ python-Patches-1660500 ] Hide iteration variable in list comprehensions Message-ID: Patches item #1660500, was opened at 2007-02-15 21:29 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660500&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Parser/Compiler Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nick Coghlan (ncoghlan) Assigned to: Georg Brandl (gbrandl) Summary: Hide iteration variable in list comprehensions Initial Comment: This patch hides the iteration variable in list comprehensions. It adds new tests (modelled on the generator expression tests) and also removes some del statements from the standard library (where code previously cleaned up its own namespace). The changes to symtable.[ch] are more significant than strictly necessary - I found it necessary to spend some time cleaning up the code in order to understand what was needed for the list comprehension changes. Given that the 2.x and 3.0 compilers have already diverged fairly significantly, I don't believe this will make the process of keeping them in sync any more difficult than it is already. Assigning to Georg for initial review (as his set comprehensions patch provided a great deal of inspiration for this one). ---------------------------------------------------------------------- >Comment By: Nick Coghlan (ncoghlan) Date: 2007-02-15 21:53 Message: Logged In: YES user_id=1038590 Originator: YES Speed measurements show a significant speed up over trunk & Python 2.4 for module/class level code: (Python 2.4)$ python -m timeit -s "seq=range(1000)" "[[x for x in seq] for y in seq]" 10 loops, best of 3: 239 msec per loop (Python 2.x trunk)$ ./python -m timeit -s "seq=range(1000)" "[[x for x in seq] for y in seq]" 10 loops, best of 3: 193 msec per loop (Python 3000)$ ./python -m timeit -s "seq=range(1000)" "[[x for x in seq] for y in seq]" 10 loops, best of 3: 176 msec per loop This is almost certainly due to the variables and the list object becoming function locals. There is a slowdown inside a function (but we are still faster than Python 2.4): (Python 2.4)$ python -m timeit -s "seq=range(1000)" -s "def f(): return [[x for x in seq] for y in seq]" "f()" 10 loops, best of 3: 259 msec per loop (Python 2.x trunk)$ ./python -m timeit -s "seq=range(1000)" -s "def f(): return [[x for x in seq] for y in seq]" "f()" 10 loops, best of 3: 176 msec per loop (Python 3000)$ ./python -m timeit -s "seq=range(1000)" -s "def f(): return [[x for x in seq] for y in seq]" "f()" 10 loops, best of 3: 185 msec per loop ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660500&group_id=5470 From noreply at sourceforge.net Thu Feb 15 13:13:58 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 04:13:58 -0800 Subject: [Patches] [ python-Patches-1660500 ] Hide iteration variable in list comprehensions Message-ID: Patches item #1660500, was opened at 2007-02-15 21:29 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660500&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Parser/Compiler Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nick Coghlan (ncoghlan) Assigned to: Georg Brandl (gbrandl) Summary: Hide iteration variable in list comprehensions Initial Comment: This patch hides the iteration variable in list comprehensions. It adds new tests (modelled on the generator expression tests) and also removes some del statements from the standard library (where code previously cleaned up its own namespace). The changes to symtable.[ch] are more significant than strictly necessary - I found it necessary to spend some time cleaning up the code in order to understand what was needed for the list comprehension changes. Given that the 2.x and 3.0 compilers have already diverged fairly significantly, I don't believe this will make the process of keeping them in sync any more difficult than it is already. Assigning to Georg for initial review (as his set comprehensions patch provided a great deal of inspiration for this one). ---------------------------------------------------------------------- >Comment By: Nick Coghlan (ncoghlan) Date: 2007-02-15 22:13 Message: Logged In: YES user_id=1038590 Originator: YES For reference, the original set comprehensions patch & the py3k list discussion: http://www.python.org/sf/1548388 http://mail.python.org/pipermail/python-3000/2006-December/005188.html Note that the current patch ended up looking a *lot* like the original one (the main difference specific to list comprehensions is that the temporary list is built in the inner scope and then returned rather than being passed in as an argument. Additionally, the code has been unified only for the symtable stage - the AST and compilation stages still use separate code for listcomps and genexps). It turns out that there are some really curly scoping problems that using a nested function deals with automatically (see the new test_listcomps.py in the patch for examples). Having a more efficient mechanism specifically for 'transient' scopes like this is an interesting idea, but it's far from easy (mainly because the variables in the scope still need to be properly visible in scopes it *contains*). Adding set comprehensions on top of this patch should be pretty straightforward. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2007-02-15 21:53 Message: Logged In: YES user_id=1038590 Originator: YES Speed measurements show a significant speed up over trunk & Python 2.4 for module/class level code: (Python 2.4)$ python -m timeit -s "seq=range(1000)" "[[x for x in seq] for y in seq]" 10 loops, best of 3: 239 msec per loop (Python 2.x trunk)$ ./python -m timeit -s "seq=range(1000)" "[[x for x in seq] for y in seq]" 10 loops, best of 3: 193 msec per loop (Python 3000)$ ./python -m timeit -s "seq=range(1000)" "[[x for x in seq] for y in seq]" 10 loops, best of 3: 176 msec per loop This is almost certainly due to the variables and the list object becoming function locals. There is a slowdown inside a function (but we are still faster than Python 2.4): (Python 2.4)$ python -m timeit -s "seq=range(1000)" -s "def f(): return [[x for x in seq] for y in seq]" "f()" 10 loops, best of 3: 259 msec per loop (Python 2.x trunk)$ ./python -m timeit -s "seq=range(1000)" -s "def f(): return [[x for x in seq] for y in seq]" "f()" 10 loops, best of 3: 176 msec per loop (Python 3000)$ ./python -m timeit -s "seq=range(1000)" -s "def f(): return [[x for x in seq] for y in seq]" "f()" 10 loops, best of 3: 185 msec per loop ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660500&group_id=5470 From noreply at sourceforge.net Thu Feb 15 13:47:11 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 04:47:11 -0800 Subject: [Patches] [ python-Patches-1660179 ] functools.compose to chain functions together Message-ID: Patches item #1660179, was opened at 2007-02-15 10:17 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Chris AtLee (catlee) Assigned to: Martin v. L?wis (loewis) Summary: functools.compose to chain functions together Initial Comment: The motivation for this patch was to be able to chain together operator.itemgetter objects so that I could sort a list of items that have the form ((a,b), c) by the 'a' or 'b' part of the item. Sorting by 'b' can be accomplished with: l.sort(key=compose(itemgetter(1), itemgetter(0))) compose(itemgetter(1), itemgetter(0)) is equivalent to def c(v): return itemgetter(1)(itemgetter(0)(v)) ---------------------------------------------------------------------- >Comment By: Nick Coghlan (ncoghlan) Date: 2007-02-15 22:47 Message: Logged In: YES user_id=1038590 Originator: NO Patch quality also looks good to me. However, my main concerns would be whether or not the use case comes up often enough to justify inclusion in the standard library, and at what point it makes more sense to just use a lambda expression. For example, I think most people would find this easier to understand than the compose based version: l.sort(key=(lambda k: k[0][1])) ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2007-02-15 20:25 Message: Logged In: YES user_id=849994 Originator: NO The patch looks good, is complete (one nit, "Returns" should be "Return" in the doc) and the functionality belongs into functools. What do you think, Martin? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 From noreply at sourceforge.net Thu Feb 15 15:01:50 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 06:01:50 -0800 Subject: [Patches] [ python-Patches-1020188 ] Use Py_CLEAR where necessary to avoid crashes Message-ID: Patches item #1020188, was opened at 2004-09-01 06:35 Message generated for change (Settings changed) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1020188&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None Status: Open Resolution: Accepted Priority: 5 Private: No Submitted By: Dima Dorfman (ddorfman) >Assigned to: Nobody/Anonymous (nobody) Summary: Use Py_CLEAR where necessary to avoid crashes Initial Comment: The Py_CLEAR macro was introduced to make it less tempting to write incorrect code in the form Py_DECREF(self->field); self->field = NULL; because that can expose a half-destroyed object if deallocation can cause self->field to be read. This patch fixes mistakes like this that still exist in the core. Only cases that are potentially dangerous are fixed in this patch. If self->field can only reference a special kind of [builtin] object, then it's just a style bug because we know that the builtin object isn't evil. Likewise if the code is operating on an automatic variable. It might be nice to fix those style bugs anyway, to discourage the broken idiom. Just for kicks, here's a way to use this bug in reversed to crash the interpreter: import array, gc, weakref a = array.array('c') wr = weakref.ref(a, lambda _: gc.get_referents(rev)) rev = reversed(a) del a list(rev) For me, this crashes immediately with a debug build and after a couple tries otherwise. Also attached is clearcand.py to help find these cases. It's not comprehensive, but it's a start. Usage: $ find src -name '*.c' | python clearcand.py | fgrep -- '->' The patch requires SF #1020185 to be applied for genobject.c to compile without warnings. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 10:14 Message: Logged In: YES user_id=21627 Originator: NO What's the status of this? Does anybody want to incorporate the test script? There are still a few cases left (or readded since the patch was originally made), in Modules (_bsddb, _sqlite, sv, _CarbonEvt), and in unicodeobject. mwh, if you lost interest, please unassign. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-09-27 19:50 Message: Logged In: YES user_id=4771 The grep may miss other expressions, like Py_DECREF(items[index]). There is also the dreadful case of macros: Py_DECREF(x) used in a macro which is called with self->something as x. In other words I am not sure the script can replace a (really motivated) programmer's extensive review... ---------------------------------------------------------------------- Comment By: Dima Dorfman (ddorfman) Date: 2004-09-10 15:29 Message: Logged In: YES user_id=908995 Okay, I'll post a cleaned up script this weekend. About your improvements: 1. Py_CLEAR already has the same semantics as a would-be Py_XCLEAR--it doesn't do anything if its argument is null. Py_XDECREF should definitely be checked for, though. 2. Py_DECREF(x) is probably safe; this is what I meant when I talked about operating on automatic variables in the initial comment. The grep for "->" filters out matches that are safe in this manner. It should be noted that this isn't necessarily safe, just that it's almost certainly so. In particular, it's not safe in this case: PyObject *x = ...; /* self is a struct { PyObject **something; ... } */ self->something = &x; Py_DECREF(x); x = NULL; but doing that is just insane. The grep would also miss ((*self).something), but I didn't expect to see any of those. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-10 15:09 Message: Logged In: YES user_id=6656 Yeah, I've come to a similar conclusion (finally got round to thinking about this properly this morning). I think a cleaned up script would be a good idea. Two improvements (which you may well have made already): Py_XDECREF is vulnerable too (do we want Py_XCLEAR??) and Py_DECREF(x); x == NULL is ok AFAICT when x is an identifier. ---------------------------------------------------------------------- Comment By: Dima Dorfman (ddorfman) Date: 2004-09-10 14:52 Message: Logged In: YES user_id=908995 I didn't write specific tests for these since they really would be testing for one particular line of code. It's not really hard to write them, though; I think my sample using reversed can be easily adapted to iterobject and itertools. I'm not sure whether genobject can be exploited like this, and I don't think it's worth finding out just to write a test case. On the other hand, if you (anyone) think it's worth it, I'll clean up clearcand.py not to require a pipeline and it can be added to the toolbox. For that to make sense, though, we should probably fix the style bugs that it catches so it can be reasonable to expect no output. Here are the outstanding matches that dereference something using "->": src/Modules/_bsddb.c 746 'Py_DECREF(self->myenvobj);\nself->myenvobj = NULL;' src/Modules/_bsddb.c 781 'Py_DECREF(self->myenvobj);\nself->myenvobj = NULL;' src/Modules/_bsddb.c 786 'Py_DECREF(self->associateCallback); \nself->associateCallback = NULL;' src/Modules/_bsddb.c 1184 'Py_DECREF(self->associateCallback); \nself->associateCallback = NULL;' src/Modules/svmodule.c 281 'Py_DECREF(self->ob_svideo); \nself->ob_svideo = NULL;' src/Mac/Modules/carbonevt/_CarbonEvtmodule.c 1060 'Py_DECREF(_self->ob_callback);\n_self->ob_callback = NULL;' src/Objects/unicodeobject.c 161 'Py_DECREF(unicode->defenc); \nunicode->defenc = NULL;' src/Objects/unicodeobject.c 250 'Py_DECREF(unicode->defenc); \nunicode->defenc = NULL;' unicode and sv are safe, and bsddb is on my list to look at. It shouldn't be harmful to just change all of those to Py_CLEAR, though. I can cobble up a patch for that, too. Opinions? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-08 18:33 Message: Logged In: YES user_id=80475 FWIW, rather than add specific testcases (closing the barndoor after the horse has escaped), it might be a good idea to fixup the OP's search code and add it to the toolbox. Future errors of this type are better caught through code scans than by testing stuff that already works fine. Of course, if highly motivated, you can do both :-) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-09-01 13:32 Message: Logged In: YES user_id=6656 Test cases would be nice. If it's easy for someone whose thought about it, that would be great, if not assign to me and I'll get to it... ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-01 08:06 Message: Logged In: YES user_id=80475 Accepted and applied. See: Include/object.h 2.129 Modules/itertoolsmodule.c 1.35 Objects/enumobject.c 1.18 Objects/genobject.c 1.4 Objects/iterobject.c 1.19 Thanks for the patch. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1020188&group_id=5470 From noreply at sourceforge.net Thu Feb 15 18:50:56 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 09:50:56 -0800 Subject: [Patches] [ python-Patches-1151323 ] New fpconst module Message-ID: Patches item #1151323, was opened at 2005-02-24 15:06 Message generated for change (Comment added) made by kousu You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1151323&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Gregory Warnes (warnes) Assigned to: Nobody/Anonymous (nobody) Summary: New fpconst module Initial Comment: This patch adds the 'fpconst' module, described in PEP754 to the standard python library. It includes both the source code and the documentation. ---------------------------------------------------------------------- Comment By: Nick Guenther (kousu) Date: 2007-02-15 12:50 Message: Logged In: YES user_id=705725 Originator: NO Wouldn't it be better to include this directly in the language, and have float.isNaN(), float.isInf(), ..., or perhaps globals inNaN(), ..., ? In any case, what would it take to move this forward then? Python is sorely lacking this. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 05:03 Message: Logged In: YES user_id=21627 Originator: NO This being a PEP, it can't go into Python without the PEP being accepted first, by BDFL pronouncement (see PEP 1). So as long as this hasn't happened, this patch cannot go in. ---------------------------------------------------------------------- Comment By: Gregory Warnes (warnes) Date: 2006-02-21 11:10 Message: Logged In: YES user_id=9316 Yes, there is still interest in getting this into 2.5. ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2006-02-20 07:04 Message: Logged In: YES user_id=849994 Is there an interest of getting this into 2.5? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-07-14 08:27 Message: Logged In: YES user_id=6656 For Python 2.5 you should indeed be able to use the "' without success before resorting to the big_endian_ check... -Greg ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2005-02-24 19:25 Message: Logged In: YES user_id=139309 Is there a reason why you don't use the struct.pack endian markers? You could get rid of all the code that deals with _big_endian if you just prefixed the formats with ! or > ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1151323&group_id=5470 From noreply at sourceforge.net Thu Feb 15 20:51:01 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 11:51:01 -0800 Subject: [Patches] [ python-Patches-1660179 ] functools.compose to chain functions together Message-ID: Patches item #1660179, was opened at 2007-02-15 01:17 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Chris AtLee (catlee) Assigned to: Martin v. L?wis (loewis) Summary: functools.compose to chain functions together Initial Comment: The motivation for this patch was to be able to chain together operator.itemgetter objects so that I could sort a list of items that have the form ((a,b), c) by the 'a' or 'b' part of the item. Sorting by 'b' can be accomplished with: l.sort(key=compose(itemgetter(1), itemgetter(0))) compose(itemgetter(1), itemgetter(0)) is equivalent to def c(v): return itemgetter(1)(itemgetter(0)(v)) ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 20:51 Message: Logged In: YES user_id=21627 Originator: NO I guess the same (can use a lambda) could be said about the entire operators module, and the "entire" functools module. I admit that in the specific example given, I would also find a lambda easier to read (in particular because it's more obvious that it is [0][1]). catlee, what do you dislike about the lambda notation? ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2007-02-15 13:47 Message: Logged In: YES user_id=1038590 Originator: NO Patch quality also looks good to me. However, my main concerns would be whether or not the use case comes up often enough to justify inclusion in the standard library, and at what point it makes more sense to just use a lambda expression. For example, I think most people would find this easier to understand than the compose based version: l.sort(key=(lambda k: k[0][1])) ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2007-02-15 11:25 Message: Logged In: YES user_id=849994 Originator: NO The patch looks good, is complete (one nit, "Returns" should be "Return" in the doc) and the functionality belongs into functools. What do you think, Martin? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 From noreply at sourceforge.net Thu Feb 15 21:03:32 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 12:03:32 -0800 Subject: [Patches] [ python-Patches-1660179 ] functools.compose to chain functions together Message-ID: Patches item #1660179, was opened at 2007-02-14 19:17 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Chris AtLee (catlee) Assigned to: Martin v. L?wis (loewis) Summary: functools.compose to chain functions together Initial Comment: The motivation for this patch was to be able to chain together operator.itemgetter objects so that I could sort a list of items that have the form ((a,b), c) by the 'a' or 'b' part of the item. Sorting by 'b' can be accomplished with: l.sort(key=compose(itemgetter(1), itemgetter(0))) compose(itemgetter(1), itemgetter(0)) is equivalent to def c(v): return itemgetter(1)(itemgetter(0)(v)) ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2007-02-15 15:03 Message: Logged In: YES user_id=80475 Originator: NO IMO, the lambda approach is more readable and more applicable to a variety of situations. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 14:51 Message: Logged In: YES user_id=21627 Originator: NO I guess the same (can use a lambda) could be said about the entire operators module, and the "entire" functools module. I admit that in the specific example given, I would also find a lambda easier to read (in particular because it's more obvious that it is [0][1]). catlee, what do you dislike about the lambda notation? ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2007-02-15 07:47 Message: Logged In: YES user_id=1038590 Originator: NO Patch quality also looks good to me. However, my main concerns would be whether or not the use case comes up often enough to justify inclusion in the standard library, and at what point it makes more sense to just use a lambda expression. For example, I think most people would find this easier to understand than the compose based version: l.sort(key=(lambda k: k[0][1])) ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2007-02-15 05:25 Message: Logged In: YES user_id=849994 Originator: NO The patch looks good, is complete (one nit, "Returns" should be "Return" in the doc) and the functionality belongs into functools. What do you think, Martin? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 From noreply at sourceforge.net Thu Feb 15 21:04:43 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 12:04:43 -0800 Subject: [Patches] [ python-Patches-1402289 ] Allow mappings as globals (was: Fix dictionary subclass ...) Message-ID: Patches item #1402289, was opened at 2006-01-10 19:24 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1402289&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) >Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: crutcher (crutcher_gmail) Assigned to: Raymond Hettinger (rhettinger) Summary: Allow mappings as globals (was: Fix dictionary subclass ...) Initial Comment: There is an inconsistancy in the way that dictionary subclasses behave when they are used as as namespaces in execs. Basically, while python 2.4 permits the usage of dictionary subclasses for local environments, it still bypasses the subclass functions and uses the C API for global environments. The attached patch (and unittest!) addresses this issue. I'm pretty sure we keep the fast path in this. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2007-02-15 15:04 Message: Logged In: YES user_id=80475 Originator: NO I do still expect to work on this patch. I'm not sure where the March date is coming from. This is a new feature and accordingly would go into Py2.6. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 04:55 Message: Logged In: YES user_id=21627 Originator: NO Raymond, do you think you can make it this March? If not, please unassign. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-03-30 13:12 Message: Logged In: YES user_id=1424288 Are you going to get to this? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2006-02-21 08:28 Message: Logged In: YES user_id=80475 Crutcher, this looked good on my first read-through. Will go through it in detail sometime in March. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-30 14:10 Message: Logged In: YES user_id=1424288 doh, forgot to check the 'upload' box ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-30 12:08 Message: Logged In: YES user_id=1424288 I have reworked and extended this patch so that arbitrary mappings are permitted for globals, not just dictionary subtypes. This touched a good deal more code. Please use the updated version. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2006-01-25 13:20 Message: Logged In: YES user_id=4771 I will review and check-in your patch, I think it's a good idea despite the added code complexity. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-24 22:38 Message: Logged In: YES user_id=1424288 currently chasing through the PyFrame code and the EXEC_STMT code to make globals generic mappings, will update this patch yet again. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-22 04:19 Message: Logged In: YES user_id=1424288 I've fixed up the exception case, and extended the test case to check for it. Is there anything else I can do to get this in? ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-22 00:11 Message: Logged In: YES user_id=1424288 I'm going to fix the missed exception test this weekend, and try to get an updated patch to you. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2006-01-14 04:54 Message: Logged In: YES user_id=4771 Provided this can be done with no measurable performance hit, I guess that I'm fine with the idea. The patch needs a bit more work, though: I don't see why it should accept dict subclasses as globals but not arbitrary mappings (as it now does for the locals). This is mainly an issue of removing a few checks in various places, like EXEC_STMT and the eval() and execfile() built-ins. There is a missing exception check/clear in the part about LOAD_NAME, after the PyObject_GetItem(f->f_globals, w). A side note: in the current trunk already, LOAD_GLOBAL contains a couple of checks, namely PyString_CheckExact() and hash != -1. We might be able to prove in advance that these two conditions are always true. We could then remove the checks. Not sure the difference measurable, though. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-12 02:50 Message: Logged In: YES user_id=1424288 Well, why fix it for eval but not for exec? I don't think the time hit is noticeable, I ran 'time make test' twice each on the trunk with and without the patch. Here are the results: Trunk: real 9m17.117s user 3m30.930s sys 0m35.417s real 9m9.471s user 3m31.484s sys 0m34.978s Patch: real 9m32.469s user 3m40.134s sys 0m36.140s real 9m6.779s user 3m27.529s sys 0m34.716s ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-01-12 01:11 Message: Logged In: YES user_id=21627 I think the history is this: it originates from python.org/sf/215126, which requested that you can pass dict subtypes to eval. Armin noted that eval will always produce LOAD/STORE_NAME, so just modifying these opcodes is sufficient to fulfil the feature request. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2006-01-11 20:21 Message: Logged In: YES user_id=80475 Okay, I will take it under consideration. Also, I'll try to remember the reason it wasn't done the first time around. No promises though -- by itself consistency is a weak motivation. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-11 18:11 Message: Logged In: YES user_id=1424288 Here's a more interesting example. This works fine, unless the variables fall through to the global dictionary, or some code marks them global. import code class ManagedVariable: def get(self): return None def set(self, value): pass def delete(self): # Return false to stop the delete. return True class ManagedEnvironment(dict): def __setitem__(self, key, value): if self.has_key(key): if isinstance(dict.__getitem__(self, key), ManagedVariable): dict.__getitem__(self, key).set(value) return dict.__setitem__(self, key, value) def __getitem__(self, key): if self.has_key(key): if isinstance(dict.__getitem__(self, key), ManagedVariable): return dict.__getitem__(self, key).get() return dict.__getitem__(self, key) def __delitem__(self, key): if self.has_key(key): if isinstance(dict.__getitem__(self, key), ManagedVariable): if not dict.__getitem__(self, key).delete(): return dict.__delitem__(self, key) class RangedInt(ManagedVariable): def __init__(self, value, (low, high)): self.value = value self.low = low self.high = high def get(self): return self.value def set(self, value): if value < self.low: value = self.low if value > self.high: value = self.high self.value = value class FunctionValue(ManagedVariable): def __init__(self, get_func = None, set_func = None, del_func = None): self.get_func = get_func self.set_func = set_func self.del_func = del_func def get(self): if self.get_func: return self.get_func() return None def set(self, value): if self.set_func: self.set_func(value) def delete(self): if self.del_func: return self.del_func() return True class Constant(ManagedVariable): def __init__(self, value): self.value = value def get(self): return self.value def delete(self): return False import time d = ManagedEnvironment() d['ranged'] = RangedInt(1, (0, 100)) d['time'] = FunctionValue(lambda: time.time()) d['constant'] = Constant(42) code.interact(local=d) ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-11 13:12 Message: Logged In: YES user_id=1424288 Here's an example of a little typed environment. It's not the most robust, but it gets you thinking. import code class TypedDictionary(dict): def __setitem__(self, key, value): if self.has_key(key): t = type(self[key]) if t != type(value): try: value = t(value) except Exception: raise TypeError, \ "illegal assignment to '%s':" \ " %s cannot be coerced to %s" \ % (key, type(value), t) dict.__setitem__(self, key, value) code.interact(local=TypedDictionary()) ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-10 21:16 Message: Logged In: YES user_id=1424288 With the ability to use dictionary subclasses for local and global environments, it becomes very easy to implement execution environments with extended variable semantics for special purposes. This includes driving interactive processes, config file processors, type checking, lazy object construction, and read-only variables. As it currently stands, the semantics are inconsistent. The local case should not have been changed if the global case was not going to be changed. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2006-01-10 21:03 Message: Logged In: YES user_id=80475 Do you have any use cases? AFAICT, there were no unmet needs with locals option. Also, there was a reason that globals weren't included but I remember what it was. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1402289&group_id=5470 From noreply at sourceforge.net Thu Feb 15 21:12:42 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 12:12:42 -0800 Subject: [Patches] [ python-Patches-1660179 ] functools.compose to chain functions together Message-ID: Patches item #1660179, was opened at 2007-02-14 19:17 Message generated for change (Comment added) made by catlee You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Chris AtLee (catlee) Assigned to: Martin v. L?wis (loewis) Summary: functools.compose to chain functions together Initial Comment: The motivation for this patch was to be able to chain together operator.itemgetter objects so that I could sort a list of items that have the form ((a,b), c) by the 'a' or 'b' part of the item. Sorting by 'b' can be accomplished with: l.sort(key=compose(itemgetter(1), itemgetter(0))) compose(itemgetter(1), itemgetter(0)) is equivalent to def c(v): return itemgetter(1)(itemgetter(0)(v)) ---------------------------------------------------------------------- >Comment By: Chris AtLee (catlee) Date: 2007-02-15 15:12 Message: Logged In: YES user_id=186532 Originator: YES lambda k: k[0][1] is fine for this example. I just thought that since operator.itemgetter existed that there should be a way to represent a series of __getitem__ calls in a similar way...and also that a way of composing functions would be generally useful to have in the stdlib. FWIW, I think that the compose function would make it simpler to compose a set of functions that is determined at run-time. It's also my experience that newbies have a hard time with lambda's :) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2007-02-15 15:03 Message: Logged In: YES user_id=80475 Originator: NO IMO, the lambda approach is more readable and more applicable to a variety of situations. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 14:51 Message: Logged In: YES user_id=21627 Originator: NO I guess the same (can use a lambda) could be said about the entire operators module, and the "entire" functools module. I admit that in the specific example given, I would also find a lambda easier to read (in particular because it's more obvious that it is [0][1]). catlee, what do you dislike about the lambda notation? ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2007-02-15 07:47 Message: Logged In: YES user_id=1038590 Originator: NO Patch quality also looks good to me. However, my main concerns would be whether or not the use case comes up often enough to justify inclusion in the standard library, and at what point it makes more sense to just use a lambda expression. For example, I think most people would find this easier to understand than the compose based version: l.sort(key=(lambda k: k[0][1])) ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2007-02-15 05:25 Message: Logged In: YES user_id=849994 Originator: NO The patch looks good, is complete (one nit, "Returns" should be "Return" in the doc) and the functionality belongs into functools. What do you think, Martin? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 From noreply at sourceforge.net Thu Feb 15 21:40:50 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 15 Feb 2007 12:40:50 -0800 Subject: [Patches] [ python-Patches-1609108 ] Docstring support for ctypesgen Message-ID: Patches item #1609108, was opened at 2006-12-05 11:28 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1609108&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Closed Resolution: Invalid Priority: 5 Private: No Submitted By: David Remahl (chmod007) Assigned to: Thomas Heller (theller) Summary: Docstring support for ctypesgen Initial Comment: ctypesgen should generate helpful docstrings for the functions it creates. This patch adds a "-d" switch to xml2py. ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2007-02-15 21:40 Message: Logged In: YES user_id=11105 Originator: NO I would like to note that I finally integrated a patch similar to yours in the ctypeslib project. The switch for the xml2py script is '-c' for 'comments'. The generated comments include the C function signature from the header file, plus the name of the header file itself and the line number where the function definition is. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2006-12-05 18:54 Message: Logged In: YES user_id=11105 Originator: NO Since ctypeslib (which contains the code generator) is a thirdparty product I'm closing this. Bugs and patches for ctypes addon products should be submitted to the ctypes tracker at http://sourceforge.net/tracker/?group_id=71702. However, I'm happy to discuss this patch on the ctypes-users mailing list. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1609108&group_id=5470 From noreply at sourceforge.net Fri Feb 16 13:53:57 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 16 Feb 2007 04:53:57 -0800 Subject: [Patches] [ python-Patches-1660179 ] functools.compose to chain functions together Message-ID: Patches item #1660179, was opened at 2007-02-15 01:17 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 >Status: Closed >Resolution: Rejected Priority: 5 Private: No Submitted By: Chris AtLee (catlee) Assigned to: Martin v. L?wis (loewis) Summary: functools.compose to chain functions together Initial Comment: The motivation for this patch was to be able to chain together operator.itemgetter objects so that I could sort a list of items that have the form ((a,b), c) by the 'a' or 'b' part of the item. Sorting by 'b' can be accomplished with: l.sort(key=compose(itemgetter(1), itemgetter(0))) compose(itemgetter(1), itemgetter(0)) is equivalent to def c(v): return itemgetter(1)(itemgetter(0)(v)) ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-16 13:53 Message: Logged In: YES user_id=21627 Originator: NO I'm rejecting the patch, because YAGNI. I agree that the quality of the patch is good; thanks for submitting it. I encourage you to post it to the Python Cookbook (if it's not there already); perhaps people may come back with an actual use case some day. As for composing functions that are determined dynamically: this can be done already. Assume f and g are variables, you can do lambda arg:f(g(arg)). What you can't do is if you have a variably-sized list of functions to compose, but again, I think YAGNI (if that ever happens, writing a loop is easy enough). Wrt newbies: I agree they have difficulties with lambda, but I think that is because they always have difficulties with first-class and higher-order functions. So they likely have the same problems with itemgetter and attrgetter (which are higher-order), and would have problems with compose. Even though I believe I can deal with callable objects just fine, I had problems understanding the example, because the order of the itemgetters is reversed to the "logical" order - although it is the standard order for composition in mathematics. ---------------------------------------------------------------------- Comment By: Chris AtLee (catlee) Date: 2007-02-15 21:12 Message: Logged In: YES user_id=186532 Originator: YES lambda k: k[0][1] is fine for this example. I just thought that since operator.itemgetter existed that there should be a way to represent a series of __getitem__ calls in a similar way...and also that a way of composing functions would be generally useful to have in the stdlib. FWIW, I think that the compose function would make it simpler to compose a set of functions that is determined at run-time. It's also my experience that newbies have a hard time with lambda's :) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2007-02-15 21:03 Message: Logged In: YES user_id=80475 Originator: NO IMO, the lambda approach is more readable and more applicable to a variety of situations. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 20:51 Message: Logged In: YES user_id=21627 Originator: NO I guess the same (can use a lambda) could be said about the entire operators module, and the "entire" functools module. I admit that in the specific example given, I would also find a lambda easier to read (in particular because it's more obvious that it is [0][1]). catlee, what do you dislike about the lambda notation? ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2007-02-15 13:47 Message: Logged In: YES user_id=1038590 Originator: NO Patch quality also looks good to me. However, my main concerns would be whether or not the use case comes up often enough to justify inclusion in the standard library, and at what point it makes more sense to just use a lambda expression. For example, I think most people would find this easier to understand than the compose based version: l.sort(key=(lambda k: k[0][1])) ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2007-02-15 11:25 Message: Logged In: YES user_id=849994 Originator: NO The patch looks good, is complete (one nit, "Returns" should be "Return" in the doc) and the functionality belongs into functools. What do you think, Martin? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660179&group_id=5470 From noreply at sourceforge.net Fri Feb 16 14:05:02 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 16 Feb 2007 05:05:02 -0800 Subject: [Patches] [ python-Patches-1628205 ] socket.readline() interface doesn't handle EINTR properly Message-ID: Patches item #1628205, was opened at 2007-01-04 22:37 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1628205&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Maxim Sobolev (sobomax) Assigned to: Nobody/Anonymous (nobody) Summary: socket.readline() interface doesn't handle EINTR properly Initial Comment: The socket.readline() interface doesn't handle EINTR properly. Currently, when EINTR received exception is not handled and all data that has been in the buffer is lost. There is no way to recover that data from the code that uses the interface. Correct behaviour would be to catch EINTR and restart recv(). Patch is attached. Following is the real world example of how it affects httplib module: File "/usr/local/lib/python2.4/xmlrpclib.py", line 1096, in __call__ return self.__send(self.__name, args) File "/usr/local/lib/python2.4/xmlrpclib.py", line 1383, in __request verbose=self.__verbose File "/usr/local/lib/python2.4/xmlrpclib.py", line 1131, in request errcode, errmsg, headers = h.getreply() File "/usr/local/lib/python2.4/httplib.py", line 1137, in getreply response = self._conn.getresponse() File "/usr/local/lib/python2.4/httplib.py", line 866, in getresponse response.begin() File "/usr/local/lib/python2.4/httplib.py", line 336, in begin version, status, reason = self._read_status() File "/usr/local/lib/python2.4/httplib.py", line 294, in _read_status line = self.fp.readline() File "/usr/local/lib/python2.4/socket.py", line 325, in readline data = recv(1) error: (4, 'Interrupted system call') -Maxim ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-16 14:05 Message: Logged In: YES user_id=21627 Originator: NO I agree that this should be fixed; I'm not sure I like the proposed fixed, though. It discards the exception and keeps running. What it (IMO) should do instead is abort, then return the data on the next invocation. Of course, this may have problems in itself, since the file descriptor might not report read-ready when passed to select or poll, even though data are available. Please discuss this on python-dev (and elsewhere), and report what recommendations people made. ---------------------------------------------------------------------- Comment By: Maxim Sobolev (sobomax) Date: 2007-01-08 11:51 Message: Logged In: YES user_id=24670 Originator: YES Well, it's not quite correct since for example httplib.py tries to handle EINTR. The fundamental problem with socket.readline() is that it does internal buffering so that getting EINTR results in data being lost. I don't think it has to be fixed in C, since recv() is very low-level interface and it is expected to return EINTR on signal, so that "fixing" it there could possibly break software that relies on this behaviour. And I don't quite buy your reasoning - "since it's broken in few more places let's keep it consistently broken everywhere". To me it sounds like attempt to hide the head in the sand instead of facing the problem at hand. Fixing socket.readline() may be the first step in improvind the library to handle this condition properly. ---------------------------------------------------------------------- Comment By: Oren Tirosh (orenti) Date: 2007-01-07 19:24 Message: Logged In: YES user_id=562624 Originator: NO You may have encountered this on sockets but *all* Python I/O does not handle restart on EINTR. The right place to fix this is probably in C, not the Python library. The places where an I/O operation could be interrupted are practically anywhere the GIL is released. This kind of change is likely to be controversial. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1628205&group_id=5470 From noreply at sourceforge.net Fri Feb 16 20:36:59 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 16 Feb 2007 11:36:59 -0800 Subject: [Patches] [ python-Patches-1661754 ] ftplib passive ftp problem on multihomed clients Message-ID: Patches item #1661754, was opened at 2007-02-16 14:36 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1661754&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Tim Baum (timbaum) Assigned to: Nobody/Anonymous (nobody) Summary: ftplib passive ftp problem on multihomed clients Initial Comment: When using passive FTP on a multihomed client, data connections should originate from the same address as the control connection. Otherwise, the connection can be blocked by stateful firewalls or rejected by the ftp server. The ftplib.py library allows the OS to choose the local address for each passive data connection, so connections fail intermittently if the operating system selects a different local address than used for the initial connection. The attached patch binds the passive-mode data connection to the same local address as the control connection. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1661754&group_id=5470 From noreply at sourceforge.net Sat Feb 17 04:48:43 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 16 Feb 2007 19:48:43 -0800 Subject: [Patches] [ python-Patches-1121142 ] ZipFile.open - read-only file-like obj for files in archive Message-ID: Patches item #1121142, was opened at 2005-02-12 00:08 Message generated for change (Comment added) made by glyf You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: Out of Date Priority: 5 Private: No Submitted By: Alan McIntyre (alanmcintyre) Assigned to: Nobody/Anonymous (nobody) Summary: ZipFile.open - read-only file-like obj for files in archive Initial Comment: I originally started working on updating patch 992750, but decided after a short while to just start from scratch, so I'm posting it as a new patch. Sorry if this isn't appropriate. This patch provides a new open() method on ZipFile; this method returns a file-like object for the requested item in the archive. This file-like object only provides a read() method. ZipFile.read was modified to use the new open method (this was suggested by loewis in reference to patch 992750). The patched zipfile.py passed the existing tests in the test_zipfile.py from CVS. New tests were added to verify the operation of the object returned by open(). These tests were modeled after existing tests for ZipFile.read(); two read fixed-size chunks from the file-like object, and two others read random-sized chunks. I have only run the tests on Windows XP, using Python2.4 from the official Windows installer. I will test the patch out on Linux over the weekend. If the patch is accepted I'll also generate and submit patches for the appropriate documentation as well. ---------------------------------------------------------------------- Comment By: Glyph Lefkowitz (glyf) Date: 2007-02-17 03:48 Message: Logged In: YES user_id=226807 Originator: NO Twisted also contains an implementation of this functionality, available from http://twistedmatrix.com/trac/browser/trunk/twisted/python/zipstream.py As far as I can tell it doesn't have anything to recommend it over the attached patch, however (in fact, the test coverage of the attached patch looks better), but perhaps it has some behavior which might be desirable to steal. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2007-02-14 17:13 Message: Logged In: YES user_id=1115903 Originator: YES I will see if I can make some progress on this over the weekend. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 09:14 Message: Logged In: YES user_id=21627 Originator: NO Can you please update the patch to the current subversion trunk? I'd like to apply it, but the code of zipfile has changed so that the patch is currently out-of-date. When redoing it, notice that the read implementation has changed (I couldn't figure out how you moved code around). Please do use the trunk (not Python 2.5), and please submit the patch as a single 'svn diff' output (rather than a tar file containing multiple individual diff files). ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-05-31 02:56 Message: Logged In: YES user_id=1115903 Revision 5 of this patch has been in constant use with Python 2.4.1 in an application at my job for about a month; it seems to be stable and useful in that regard. If anybody has time to review and accept (or reject) it I would appreciate it. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-27 02:23 Message: Logged In: YES user_id=1115903 After testing on my large batch of large Zip files, I made one fix (version 4 of the patch didn't read all the content of large compressed archive items). The current set of changes is attached as zipfile_patch5.tgz. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-13 16:34 Message: Logged In: YES user_id=1115903 I found a problem with my universal newline handling code in readline(); if the first byte of an '\r\n' pair was read from file but the second byte didn't come in on that same read, it resulted in the next line having an inappropriate '\n' prepended to it. zipfile_patch4.tgz has a fix for this included (with everything else, of course). I'm going to test the open() capability in a real application with some reasonably large zip files (containing files up to just short of 2GB) over the next few days, so if any bugs or performance problems show up I may have some more changes. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-13 05:58 Message: Logged In: YES user_id=1115903 Uploaded the third revision of this patch; passes all regression tests against current CVS on WinXP. I think all the issues Martin brought up have been addressed except perhaps for the case of compression rate <1. I will have a look at that when I have time; just wanted to get an update here before the patch started to look abandoned. :) ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-03-14 14:37 Message: Logged In: YES user_id=1115903 Hmm...I could have sworn I did the diff in the correct order. I'll make sure next time. :) Here's my comments on your remarks (in order): - I'm adding support for universal newlines, and will reject all modes that aren't legal combinations of r, U, and b. - I'll see if I can make a Zip file store something with compression < 1, and if I can I'll add a test case for it. - I'll try work a .flush() in there on the compression object and come up with a test case if possible - .read(0) and .readline(0) will both return an empty string with no side-effects, since this seems to be what builtin files do. Right now ZipExtFile uses the ZipFile's file object, so you pretty much have to do whatever you're going to do with the ZipExtFile instance you get back from .open() before you use that ZipFile instance in a way that moves the file pointer around. I'm sure that somebody will eventually try to use the ZipFile in this way, so it appears my options are either to (1) give the ZipExtFile its own file object to use (when possible), or (2) make sure this limitation is documented. #1 feels (to me) to be the "right thing" to do, so that's what I'll try unless there's a good reason I shouldn't. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-03-01 07:59 Message: Logged In: YES user_id=21627 The patch is reversed: usually, diff is invoked as "-c old new". I think it is almost right, but I have a few remarks: - by tradition, open() should have a mode argument, defaulting to 'r'; it would be ok to raise exceptions if it is anything else. However, do consider implementing universal newlines; allowing 'b' as a no-op might also be reasonable. - I wonder what happens if the compression rate is < 1. It would appear that the code might use too few rawbytes. I suggest to recursively invoke read in this case. - I wonder whether it could ever happen that there is still data to uncompress in the zlib object, ie. whether it might be necessary to invoke .flush() after exhausting the rawbytes (and discard the zlib object afterwards) - it appears that atleast the builtin file object implements .read(0) as returning an empty string; the manual says that the entire file is meant only if size is omitted or negative. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-02-27 06:28 Message: Logged In: YES user_id=1115903 zipfile_patch2.tgz: I updated the file-like object to support readline, readlines, __iter__ and next(). Added tests for those new methods. Also added a patch for the documentation. Passes regression tests on 2.5a0 built from CVS HEAD with MSVC .NET on Windows XP. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 From noreply at sourceforge.net Sat Feb 17 09:17:38 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 17 Feb 2007 00:17:38 -0800 Subject: [Patches] [ python-Patches-1659410 ] Minor AST tweaks Message-ID: Patches item #1659410, was opened at 2007-02-14 01:44 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1659410&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Parser/Compiler Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Collin Winter (collinwinter) Assigned to: Nobody/Anonymous (nobody) Summary: Minor AST tweaks Initial Comment: This patch implements these changes, as discussed in http://mail.python.org/pipermail/python-dev/2007-February/071006.html and following. """ 1) There are times when the _fields attribute on some AST nodes is None; if this was done to indicate that a given node has no AST-related attributes, it would be much easier if _fields was [] in this case. As it is, I had to special-case "node._fields is None" in the visitor so that I don't accidentally iterate over it. """ asdl_c.py is changed to always set _fields to a tuple. """ 2) {BinOp,AugAssign,BoolOp,etc}.op is an instance of, eg, Add, Sub, Mult, Mod, meaning you have to dispatch based on tests like "isinstance(node.op, x)" or "type(node.op) is x". I would much, much prefer to spell this "node.op is x", ie, use "node.op = Add" rather than the current "node.op = Add()" when constructing the nodes. """ asdl_c.py is changed so that it no longer emits the *_singleton objects and uses the corresponding *_type objects in their place. """ 3) I'd like there to be an Else() node for If.orelse, While.orelse, etc. My motivation is that I need the lineno and col_offset values of the "else" statement for a code-coverage utility; as it is, I have to find the last lineno of the if/while/etc suite and the first lineno of the "else" suite and then search between those two for the "else" statement. """ An Else node is added to the ASDL file, impacting TryExcept, While, For and If nodes. The compiler, symtable and other files are adjusted to deal with these new nodes. The patch is against r53772. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-17 09:17 Message: Logged In: YES user_id=21627 Originator: NO Please also update Demo/parser/unparse.py to match your changes. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-14 10:00 Message: Logged In: YES user_id=21627 Originator: NO Please minimize white-space only changes in the patch. It makes it harder to read (symtable.c in particular). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1659410&group_id=5470 From noreply at sourceforge.net Sun Feb 18 09:29:04 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 18 Feb 2007 00:29:04 -0800 Subject: [Patches] [ python-Patches-909005 ] asyncore fixes and improvements Message-ID: Patches item #909005, was opened at 2004-03-03 14:07 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=909005&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Alexey Klimkin (klimkin) >Assigned to: Josiah Carlson (josiahcarlson) Summary: asyncore fixes and improvements Initial Comment: Minor: * 0/1 for boolean values replaced with False/True. * (887279) Added handling of POLLPRI as POLLIN. POLLERR, POLLHUP, POLLNVAL are handled as exception event. handle_expt_event gets recent error from self.socket object and raises socket.error. * Default readable()/writable() returns False. * Added "map" parameter for file_dispatcher. * file_wrapper: removed "return" in close(), recv/read and send/write swapped because of their nature. * mac code for writable() removed. Manual for accept() on mac is similar to the one on linux. * Repeating exception changed from "raise socket.error, why" to raise. * Added connected/accepting/addr reset on close(). Initialization of variables moved to __init__. * close_all() now calls close for dispatcher object, EBADF treated as already closed socket/file. * Added channel id to "unhandled..." messages. Bugs: * Fixed bug (654766,889153): client never gets connected, nor errored. Connecting client gets writable event from select(), however, some client may want always be non writable. Such client may never get connected. The fix adds _readable() - always True for accepting and always False for connecting socket; and _writable() - always False for accepting and always True for connecting socket. This implies, that listening dispatcher's readable() and writable() will never be called. ("man accept" and "man connect" for non-blocking sockets). * Fixed bug: error handling after accept(). It's said, that accept can return EWOULDBLOCK even for readable socket. This mean, that even after handle_accept(), dispatcher's accept() still raise EWOULDBLOCK. New code does accept() itself and stores accepted socket in self.__pending_accept. If there was socket.error, it's treated as EWOULDBLOCK. dispatcher's accept returns self.__pending_accept and resets it to None. Features: * Added pending_read() and pending_write(). The functions helps to use dispatcher over non socket objects with buffering capabilities. In original dispatcher, if socket makes buffered read and some data is in buffer, entering asyncore.poll() doesn't finishes, since there is no data in real file/socket. This feature allow to use SSL socket, since the socket reads data by 16k chunks. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-02-14 19:54 Message: Logged In: YES user_id=341410 Originator: NO If anything is to be included, it should be the tests (though they need to be rewritten a bit). I've been working on and off on a patch that includes other portions that I felt were worthwhile. When I finish the patch, I'll also include a fixed version of the tests. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-14 11:35 Message: Logged In: YES user_id=21627 Originator: NO Alexey, are you interested in revising your code until it is approved? Saying "no" is fine; the question then is if there is anybody else interested in working on this patch. If nobody is interested in working on it, it would be best rejected - there is no point in having it listed as open here for years (it would be sad, of course, for the work put into the patch, and the work put into the review). Andrew, do you have more changes from this patch that you consider worth incorporating? ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2007-01-08 21:44 Message: Logged In: YES user_id=410460 Originator: YES 1) The patch was developed not during some academic research - but during of coding true non-blocking client-server applications, capable to run both on Linux and Windows. Original code had a lot of issues with everything: some parts were not truly blocking, not every socket can be passed, issues with high load, etc. 2) We have used medusa for ssl capability in our project. However, it's impossible to get fully non-blocking functionality with original asyncore and original medusa. So functionality was extended to support these features as well. That is what idispatcher for. 3) In the end we have got pretty reliable code, which supports features I described here and has tons of bug and issues fixed. Again, I didn't fix bug for any academic purpose - every fix was driven by real issue we met during development. I don't also think, that these fixes bond to our project too tight - I believe I made them pretty general. 4) It's possible, that some parts can be made better for other application. But if you follow the same path - developing truly non-blocking client-server with medusa's ssl capabilities, - I think, you will end with the same thing. 5) I don't insist on including the patch into the python tree as is. I feel pretty well using modified asyncore in my private library. My intention was to share my experience. Please use, if you need to. 6) The development I mention above was 2004 year. So the patch is not in sync with this reality for 2 years already. Some issues it was solving can be gone already. I also don't know, what is going on with SSL for python - there seems to be new libraries as well. ...so... just use it as you want... or as you don't want ;) ... ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-01-07 05:53 Message: Logged In: YES user_id=341410 Originator: NO In asynchat, the only stuff that should be accepted is the handle_read() changes. The deque removal should be ignored (we have deques since Python 2.4, which are *significantly* faster than lists in nontrivial applications), the iasync_chat stuff, like the idispatcher stuff, seems unnecessary. And that's pretty much it for asynchat. The proposed asynchttp module shouldn't go into the Python standard library until it has lived on its own for a nontrival amount of time in the Cheeseshop and is found to be as good as httplib, urllib, or urllib2. Even then, its inclusion should be questioned, as medusa (the http server based on asyncore) has been around for a decade or more, is used many places, and yet still isn't in the standard library. The asyncoreTest.py needs a bit of work (I notice some incorrect names), but could be used as an addition to the test suite (currently it seems as though only asynchat is tested). ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-01-07 05:42 Message: Logged In: YES user_id=341410 Originator: NO Many of the changes in the source provided by klimkin in his most recent revision from February 27, 2005 seek to solve certain problems in an inconsistent or incorrect way. Some of his changes (or variants thereof) are worthwhile. I'll start with my issues with his asyncore changes, then describe what I think should be added from them. For example, in his updated asyncore.py, the list of sockets is first shuffled randomly, then sorted based on priority. Assuming that one ignored priorities for a moment, if there were more sockets than the max sockets for the platform, then due to the limitations of randomness, there would be no guarantees that all sockets would get polled. Say, for example, that one were using windows and were running close to the actual select file handle limit (512 in Python 2.3) with 500 handles, you would skip 436 of the sockets *this pass*. In 10 passes, there would have been 100 sockets that were never polled. In 20 passes, there would still be, on average, 20 that were never polled. So this "randomization" step is the wrong thing to do, unless you actually make multiple select calls for each poll() call. But really, select is limited by 512, and I've run it with 500 without issue. The priority based sorting has much of the same problems, but it is even worse when you have nontrivial numbers of differing priorities, regardless of randomization or not. The max socket limit of 64 on Windows isn't correct. It's been 512 since at least Python 2.3 . And all other platforms being 65536? No. I've had some versions of linux die on me at 512, others at 4096, but all were dog slow beyond 500 or so. It's better to let the underlying system raise an exception for the user when it fails and let them attempt to tune it, rather than forcing a tuning that may not be correct. The "pending read" stuff is also misdirected. Assuming a non-broken async client or server, either should be handling content as it comes it, dispatching as necessary. See asynchat.collect_incoming_data() and asynchat.found_terminator() for examples. The idispatcher stuff seems unnecessary. Generally speaking, it seems to me that there are 3 levels of abstraction going on: 1) handle_*_event(), called by poll, poll2, etc. 2) handle_*(), called by handle_*_event(), user overrides, calls other handle_*() and *() methods 3) *() (aka recv, send, close, etc.), called by handle_*(), generally left alone. Some of your code breaks the abstraction and has items in layer 2 call items in layer 1, which then call items in layer 2 again. This seems unnecessary, and breaks the general downward calling semantic (except in the case of errors returned by layer 3 resulting in layer 2 handle_close() calls, which is the proper method to call). There are, according to my reading of the asyncore portions of your included module, a few things that may be worthy for inclusion into the Python standard library are: * A variant of your changes to close_all(), though it should proceed in closing everything unless a KeyboardInterrupt, SystemExit, or ExitNow exception is raised. Socket errors should be ignored, because we are closing them - we don't care about their error condition. * Checking sockets for socket error via socket.getsockopt() . * A variant of your .close() implementation. * The CONNRESET, etc., stuff in the send() and recv() methods, but not the handle_close_event() replacements, stick with handle_close() . * Checking for KeyboardInterrupt and SystemExit inside the poll functions. * The _closed_socket class and initialization. All but the last of the above, I would consider to be bugfixes, and if others agree that these are reasonable changes, I'll write up a patch against trunk and 2.5 maintenance. The last change, while I think would be nice, probably shouldn't be included in 2.5 maintenance, though I think would be fine for the trunk. ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2005-02-26 22:39 Message: Logged In: YES user_id=410460 Minor improvements: * Added handle_close_event(): calls handle_close(), then closes channel. No need to write self.close() in each handle_close (). * Improved exception handling. KeyboardInterrupt is not blocked. For python exception handle_error_event() is called, which checks for KeyboardInterrupt and closes socket, if handle_error didn't. Bugs: * Calling connect() could raise exception and doesn't hit handle_error(). Now if there was an exception, handle_error_event() is called. Features: * set_timeout(): Sets timeout for dispatcher object, if there was no io for the object, raises ETIMEDOUT, which handled by handle_error_event(). * Fixed issue with Windows - too many descriptors in select(). The list of sockets shuffled and only first asyncore.max_channels used in select(). * Added set_prio(): Sets priority for dispatcher. After shuffle the list of sockets sorted by priority. You may also check asynhttplib - asynchronous version of httplib. ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2004-07-02 15:44 Message: Logged In: YES user_id=410460 In addition to "[ 909005 ] asyncore fixes and improvements" and CVS version "asyncore.py,v 2.51" this patch provides: * Added handling of buffered socket layer (pending_read(), pending_write()). * Added fd number for __repr__. * Initialized self.socket = socket._closedsocket() instead of None for verbose error output (like closed socket.socket). * asyncore and asynchat implements idispatcher and iasync_chat. * Fixed self.addr initialization. * Removed import exceptions. * Don't filter KeyboardInterrupt, just pass through. * Added queue of sockets, solves the problem of select() on too many descriptors. I have run make test in python cvs distrib without problems. Examples of using i* included. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-06-05 19:54 Message: Logged In: YES user_id=11375 I've struggled to get the test suite running without errors on my machine, but have failed. ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2004-03-22 07:15 Message: Logged In: YES user_id=410460 There is no real reason for this change, please undo. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 21:18 Message: Logged In: YES user_id=11375 In your version of file_dispatch.__init__, the .set_file() call is moved earlier; can you say why? ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 21:13 Message: Logged In: YES user_id=11375 Added "map" parameter for file_dispatcher and dispatcher_with_send in CVS HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 21:08 Message: Logged In: YES user_id=11375 Repeating exception changes ('raise socket.error' -> just 'raise') checked into HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 21:02 Message: Logged In: YES user_id=11375 Mac code for writable() removed from HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 21:02 Message: Logged In: YES user_id=11375 Patch to use True/False applied to HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 20:55 Message: Logged In: YES user_id=11375 Fix for bug #887279 applied to HEAD. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-03-21 20:48 Message: Logged In: YES user_id=11375 The many number of changes in this patch make it difficult to figure out which changes fix which problem. I've created a new directory in CVS, nondist/sandbox/asyncore, that contains copies of the module with these patches applied, and will work on applying changes to the copy in dist/src. ---------------------------------------------------------------------- Comment By: Alexey Klimkin (klimkin) Date: 2004-03-17 08:15 Message: Logged In: YES user_id=410460 Sorry, unfortunately I have lost old patch file. I have atached new one. In addition to fixes, listed above, the patch includes: 1. Fix for operating on uninitialized socket. self.socket now initializes with _closed_socket(), so any operation throws EBADF. 2. Added class idispatcher - base class for dispatcher. The purpose of this class is to allow simple replacement of media(dispatcher interface) in classes, derived from dispatcher class. This is based on 'object'. I have also attached asynchat.diff - example for new-style dispatcher. Old asynchat works as well. ---------------------------------------------------------------------- Comment By: Wummel (calvin) Date: 2004-03-11 16:49 Message: Logged In: YES user_id=9205 There is no file attached! You have to click on the checkbox next to the upload filename. This is a Sourceforge annoyance :( ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=909005&group_id=5470 From noreply at sourceforge.net Sun Feb 18 09:39:03 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 18 Feb 2007 00:39:03 -0800 Subject: [Patches] [ python-Patches-900744 ] catch invalid chunk length in httplib read routine Message-ID: Patches item #900744, was opened at 2004-02-20 00:14 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=900744&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open >Resolution: Accepted Priority: 5 Private: No Submitted By: Wummel (calvin) Assigned to: Georg Brandl (gbrandl) Summary: catch invalid chunk length in httplib read routine Initial Comment: In HTTPResponse._read_chunked the chunk length is not checked to be a valid integer, and a ValueError will be raised in such a case. The attached patch catches ValueError (which should not normally happen, so this try:except: is reasonably fast), and raises IncompleteRead exception instead. I have no test case for this yet, but am trying to construct one :) ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-18 09:39 Message: Logged In: YES user_id=21627 Originator: NO Georg, the patch is fine for 2.6, please apply (for 2.5, I would be cautious because of the behaviour change). ---------------------------------------------------------------------- Comment By: Wummel (calvin) Date: 2006-02-01 23:16 Message: Logged In: YES user_id=9205 I attached a simple testcase that triggers the bug. IMHO this patch should be applied for Python 2.5. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2005-07-19 03:07 Message: Logged In: YES user_id=80475 Technically, the patch is fine and it is the way the code should have been written in the first place. Please bring-up on python-dev to determine whether it should be done. I can imagine that a fair amount of existing code has through trial and error discovered the ValueError and chosen to catch that. Changing the exception may unnecessarily break people's code. Sometimes we take that step when it needs to be done, but there should be a good pay-off and, in this case, I don't see it. You may also want to mention it on comp.lang.python to see if anyone cares If the patch goes forward, see if you can mock-up a test that triggers the exception so we have a good unittest. In anycase, this should not be backported (we avoid giving people reasons to not upgrade). ---------------------------------------------------------------------- Comment By: Georg Brandl (birkenfeld) Date: 2005-07-18 21:34 Message: Logged In: YES user_id=1188172 Attaching patch which does what agwego said (httplib-chunked.diff). Please review. ---------------------------------------------------------------------- Comment By: agwego (agwego) Date: 2005-02-28 17:53 Message: Logged In: YES user_id=1228982 I've run into this problem in conjunction with mod_python 3.1.4 (and although the problem is caused in mod_python) my python skills aren't up to the task. Which leaves me with fixing httplib. Although the above patch works when it comes to end of file situations, I think it would be better to return what has been consumed so far and leave it at that. A few lines down there's a comment about consuming trailers, this is the case that is tripping up httplib as far as I can tell. This is happening in Python 2.3.4. --- packages/Python-2.3.4/Lib/httplib.py Sun Nov 2 11:51:38 2003 +++ httplib.py Mon Feb 28 11:26:48 2005 @@ -423,7 +423,11 @@ i = line.find(';') if i >= 0: line = line[:i] # strip chunk-extensions - chunk_left = int(line, 16) + try: + chunk_left = int(line, 16) + except ValueError, msg: + self.close() + return value if chunk_left == 0: break if amt is None: ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=900744&group_id=5470 From noreply at sourceforge.net Sun Feb 18 10:06:07 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 18 Feb 2007 01:06:07 -0800 Subject: [Patches] [ python-Patches-1402289 ] Allow mappings as globals (was: Fix dictionary subclass ...) Message-ID: Patches item #1402289, was opened at 2006-01-11 01:24 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1402289&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: crutcher (crutcher_gmail) Assigned to: Raymond Hettinger (rhettinger) Summary: Allow mappings as globals (was: Fix dictionary subclass ...) Initial Comment: There is an inconsistancy in the way that dictionary subclasses behave when they are used as as namespaces in execs. Basically, while python 2.4 permits the usage of dictionary subclasses for local environments, it still bypasses the subclass functions and uses the C API for global environments. The attached patch (and unittest!) addresses this issue. I'm pretty sure we keep the fast path in this. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-18 10:06 Message: Logged In: YES user_id=21627 Originator: NO Raymond, on 2006-02-21, you said "Will go through it in detail sometime in March", hence I suggested that perhaps you could do it March 2007 :-) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2007-02-15 21:04 Message: Logged In: YES user_id=80475 Originator: NO I do still expect to work on this patch. I'm not sure where the March date is coming from. This is a new feature and accordingly would go into Py2.6. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 10:55 Message: Logged In: YES user_id=21627 Originator: NO Raymond, do you think you can make it this March? If not, please unassign. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-03-30 20:12 Message: Logged In: YES user_id=1424288 Are you going to get to this? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2006-02-21 14:28 Message: Logged In: YES user_id=80475 Crutcher, this looked good on my first read-through. Will go through it in detail sometime in March. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-30 20:10 Message: Logged In: YES user_id=1424288 doh, forgot to check the 'upload' box ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-30 18:08 Message: Logged In: YES user_id=1424288 I have reworked and extended this patch so that arbitrary mappings are permitted for globals, not just dictionary subtypes. This touched a good deal more code. Please use the updated version. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2006-01-25 19:20 Message: Logged In: YES user_id=4771 I will review and check-in your patch, I think it's a good idea despite the added code complexity. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-25 04:38 Message: Logged In: YES user_id=1424288 currently chasing through the PyFrame code and the EXEC_STMT code to make globals generic mappings, will update this patch yet again. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-22 10:19 Message: Logged In: YES user_id=1424288 I've fixed up the exception case, and extended the test case to check for it. Is there anything else I can do to get this in? ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-22 06:11 Message: Logged In: YES user_id=1424288 I'm going to fix the missed exception test this weekend, and try to get an updated patch to you. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2006-01-14 10:54 Message: Logged In: YES user_id=4771 Provided this can be done with no measurable performance hit, I guess that I'm fine with the idea. The patch needs a bit more work, though: I don't see why it should accept dict subclasses as globals but not arbitrary mappings (as it now does for the locals). This is mainly an issue of removing a few checks in various places, like EXEC_STMT and the eval() and execfile() built-ins. There is a missing exception check/clear in the part about LOAD_NAME, after the PyObject_GetItem(f->f_globals, w). A side note: in the current trunk already, LOAD_GLOBAL contains a couple of checks, namely PyString_CheckExact() and hash != -1. We might be able to prove in advance that these two conditions are always true. We could then remove the checks. Not sure the difference measurable, though. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-12 08:50 Message: Logged In: YES user_id=1424288 Well, why fix it for eval but not for exec? I don't think the time hit is noticeable, I ran 'time make test' twice each on the trunk with and without the patch. Here are the results: Trunk: real 9m17.117s user 3m30.930s sys 0m35.417s real 9m9.471s user 3m31.484s sys 0m34.978s Patch: real 9m32.469s user 3m40.134s sys 0m36.140s real 9m6.779s user 3m27.529s sys 0m34.716s ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-01-12 07:11 Message: Logged In: YES user_id=21627 I think the history is this: it originates from python.org/sf/215126, which requested that you can pass dict subtypes to eval. Armin noted that eval will always produce LOAD/STORE_NAME, so just modifying these opcodes is sufficient to fulfil the feature request. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2006-01-12 02:21 Message: Logged In: YES user_id=80475 Okay, I will take it under consideration. Also, I'll try to remember the reason it wasn't done the first time around. No promises though -- by itself consistency is a weak motivation. ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-12 00:11 Message: Logged In: YES user_id=1424288 Here's a more interesting example. This works fine, unless the variables fall through to the global dictionary, or some code marks them global. import code class ManagedVariable: def get(self): return None def set(self, value): pass def delete(self): # Return false to stop the delete. return True class ManagedEnvironment(dict): def __setitem__(self, key, value): if self.has_key(key): if isinstance(dict.__getitem__(self, key), ManagedVariable): dict.__getitem__(self, key).set(value) return dict.__setitem__(self, key, value) def __getitem__(self, key): if self.has_key(key): if isinstance(dict.__getitem__(self, key), ManagedVariable): return dict.__getitem__(self, key).get() return dict.__getitem__(self, key) def __delitem__(self, key): if self.has_key(key): if isinstance(dict.__getitem__(self, key), ManagedVariable): if not dict.__getitem__(self, key).delete(): return dict.__delitem__(self, key) class RangedInt(ManagedVariable): def __init__(self, value, (low, high)): self.value = value self.low = low self.high = high def get(self): return self.value def set(self, value): if value < self.low: value = self.low if value > self.high: value = self.high self.value = value class FunctionValue(ManagedVariable): def __init__(self, get_func = None, set_func = None, del_func = None): self.get_func = get_func self.set_func = set_func self.del_func = del_func def get(self): if self.get_func: return self.get_func() return None def set(self, value): if self.set_func: self.set_func(value) def delete(self): if self.del_func: return self.del_func() return True class Constant(ManagedVariable): def __init__(self, value): self.value = value def get(self): return self.value def delete(self): return False import time d = ManagedEnvironment() d['ranged'] = RangedInt(1, (0, 100)) d['time'] = FunctionValue(lambda: time.time()) d['constant'] = Constant(42) code.interact(local=d) ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-11 19:12 Message: Logged In: YES user_id=1424288 Here's an example of a little typed environment. It's not the most robust, but it gets you thinking. import code class TypedDictionary(dict): def __setitem__(self, key, value): if self.has_key(key): t = type(self[key]) if t != type(value): try: value = t(value) except Exception: raise TypeError, \ "illegal assignment to '%s':" \ " %s cannot be coerced to %s" \ % (key, type(value), t) dict.__setitem__(self, key, value) code.interact(local=TypedDictionary()) ---------------------------------------------------------------------- Comment By: crutcher (crutcher_gmail) Date: 2006-01-11 03:16 Message: Logged In: YES user_id=1424288 With the ability to use dictionary subclasses for local and global environments, it becomes very easy to implement execution environments with extended variable semantics for special purposes. This includes driving interactive processes, config file processors, type checking, lazy object construction, and read-only variables. As it currently stands, the semantics are inconsistent. The local case should not have been changed if the global case was not going to be changed. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2006-01-11 03:03 Message: Logged In: YES user_id=80475 Do you have any use cases? AFAICT, there were no unmet needs with locals option. Also, there was a reason that globals weren't included but I remember what it was. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1402289&group_id=5470 From noreply at sourceforge.net Sun Feb 18 10:09:07 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 18 Feb 2007 01:09:07 -0800 Subject: [Patches] [ python-Patches-1151323 ] New fpconst module Message-ID: Patches item #1151323, was opened at 2005-02-24 21:06 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1151323&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Gregory Warnes (warnes) Assigned to: Nobody/Anonymous (nobody) Summary: New fpconst module Initial Comment: This patch adds the 'fpconst' module, described in PEP754 to the standard python library. It includes both the source code and the documentation. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-18 10:09 Message: Logged In: YES user_id=21627 Originator: NO kousu: write a competing PEP, or work with the author of PEP 754 to revise his specification. This tracker is inappropriate to design the feature given that there is a PEP available. Please follow the procedure for Python enhancements. ---------------------------------------------------------------------- Comment By: Nick Guenther (kousu) Date: 2007-02-15 18:50 Message: Logged In: YES user_id=705725 Originator: NO Wouldn't it be better to include this directly in the language, and have float.isNaN(), float.isInf(), ..., or perhaps globals inNaN(), ..., ? In any case, what would it take to move this forward then? Python is sorely lacking this. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-15 11:03 Message: Logged In: YES user_id=21627 Originator: NO This being a PEP, it can't go into Python without the PEP being accepted first, by BDFL pronouncement (see PEP 1). So as long as this hasn't happened, this patch cannot go in. ---------------------------------------------------------------------- Comment By: Gregory Warnes (warnes) Date: 2006-02-21 17:10 Message: Logged In: YES user_id=9316 Yes, there is still interest in getting this into 2.5. ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2006-02-20 13:04 Message: Logged In: YES user_id=849994 Is there an interest of getting this into 2.5? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2005-07-14 14:27 Message: Logged In: YES user_id=6656 For Python 2.5 you should indeed be able to use the "' without success before resorting to the big_endian_ check... -Greg ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2005-02-25 01:25 Message: Logged In: YES user_id=139309 Is there a reason why you don't use the struct.pack endian markers? You could get rid of all the code that deals with _big_endian if you just prefixed the formats with ! or > ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1151323&group_id=5470 From noreply at sourceforge.net Sun Feb 18 10:12:50 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 18 Feb 2007 01:12:50 -0800 Subject: [Patches] [ python-Patches-1648268 ] Parameter list mismatches (portation problem) Message-ID: Patches item #1648268, was opened at 2007-01-30 23:15 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1648268&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. >Category: None >Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: ked-tao (ked-tao) Assigned to: Nobody/Anonymous (nobody) Summary: Parameter list mismatches (portation problem) Initial Comment: On the system I'm porting to(*), an application will trap if the caller does not pass the exact parameter list that the callee requires. This is causing problems running Python. One common instance where this appears to be causing problems is where functions are registered as METH_NOARGS methods. For example, in Obejcts/dictobject.c, dict_popitem() is declared: static PyObject *dict_popitem(dictobject *mp); However, as it is declared in the method array as METH_NOARGS, it will be called by Objects/methodobject.c:PyCFunction_Call() as "(*meth)(self, NULL)" (i.e., an extra NULL parameter is passed for some reason). This will fail on my target system. I've no problem submitting a patch for this (dictobject.c is by no means the only place this is happening - it's just the first one encountered because it's used so much - though some places _do_ correctly declare a second, ignored parameter). However, I'd like to get agreement on the correct form it should be changed to before I put the effort in to produce a patch (it's going to be a fairly tedious process to identify and fix all these). In various modules, the functions are called internally as well as being registered as METH_NOARGS methods. Therefore, the change can either be: static PyObject *foo(PyObject *self) { ... } static PyObject *foo_noargs(PyObject *self, void *noargs_null) { return foo(self); } ... where 'foo' is called internally and 'foo_noargs' is registered as a METH_NOARGS method. or: static PyObject *foo(PyObject *self, void *noargs_null) { ... } ... and any internal calls in the module have to pass a second, NULL, argument in each call. The former favours internal module calls over METH_NOARGS calls, the latter penalises them. Which is preferred? Should this be raised on a different forum? Does anyone care? ;) Thanks, Kev. (*) Details on request. ---------------------------------------------------------------------- Comment By: ked-tao (ked-tao) Date: 2007-02-16 17:46 Message: Logged In: YES user_id=1703158 Originator: YES Hi, I am submitting two patches (both against the 2.5 release sources). One contains a set of changes which have subsequently been compiled by me and used to run lib/python/test/regrtest.py. If the format of the changes themselves is acceptable, then I believe this patch can be applied relatively confidently. I haven't paid too much attention to conditional compilation in those files, but there appears to be little in the areas I've touched. The second contains a set of changes to source files that are not being used at present on my system. Therefore, they _may_ not compile. I have visually checked that all functions whose signature I have changed are not called directly (across all source files) with the old signature and have also checked header file prototypes. However, that doesn't mean I didn't miss something, so this patch should be applied with a little more care. The nature of the fixes themselves are discussed below. ----------------------------------- ==== Fixes to common problems across several files: * Failure to declare second (always NULL) parameter on functions registered as METH_NOARGS methods. - These all now have a second parameter declared as "PyObject *NOARGS_NULL". - I have also changed ones that already declared the parameter as "void *ignored" etc, as I think the name makes it clear why it's there. If the upper-case name is bad style, feel free to change it to something else - as they are all now consistent, that should be a trivial process to change in the patch file before applying it. * PyGetSetDef 'getter' and 'setter' functions not declaring the final 'closure' parameter. - These all now have a final parameter declared as "void *closure". - I have also changed ones that already declared the parameter as "void *context" or "void *ignored" etc, for consistency. * The tp_clear type slot is defined as type 'inquiry' but the return value is ignored and in some instances, not returned at all. This is related to the following thread: http://mail.python.org/pipermail/python-dev/2003-April/034433.html frameobject.c and traceback.c were either missed when those changes were made, or the problems were re-introduced since. - I have changed the functions in those files to return zero. ==== Miscellaneous individual fixes: * Objects/fileobject.c:file_self() is registered both in the "tp_iter" slot and as a METH_NOARGS function. The "tp_iter" slot function is called with one parameter (the object) and the METH_NOARGS function is called with two parameters (the object, a NULL pointer). - Wrapper function file_self_noargs() created which accepts the additional "PyObject *NOARGS_NULL" parameter and just calls the file_self() function. - All other occurences of tp_iter visually checked and appear to be OK. * The datetimemodule.c problem with time_isoformat() being registered as METH_KEYWORDS instead of METH_NOARGS is also fixed here, though I believe that has already been dealt with. ----------------------------------- All in all, that was a pretty tedious process! Hopefully these changes can mostly make it in so I don't have to do it all over again one day ;) Regards, Kev. File Added: untested.diff ---------------------------------------------------------------------- Comment By: ked-tao (ked-tao) Date: 2007-02-16 17:42 Message: Logged In: YES user_id=1703158 Originator: YES File Added: tested.diff ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-06 20:49 Message: Logged In: YES user_id=21627 Originator: NO The current specification says that these should be PyCFunction pointers, see http://docs.python.org/api/common-structs.html My initial implementation of METH_NOARGS had it differently, and nobody ever bothered fixing them all when this was changed. Please do submit a patch to correct all such errors, both in code and documentation. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1648268&group_id=5470 From noreply at sourceforge.net Sun Feb 18 17:44:40 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 18 Feb 2007 08:44:40 -0800 Subject: [Patches] [ python-Patches-1121142 ] ZipFile.open - read-only file-like obj for files in archive Message-ID: Patches item #1121142, was opened at 2005-02-11 19:08 Message generated for change (Comment added) made by alanmcintyre You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: Out of Date Priority: 5 Private: No Submitted By: Alan McIntyre (alanmcintyre) Assigned to: Nobody/Anonymous (nobody) Summary: ZipFile.open - read-only file-like obj for files in archive Initial Comment: I originally started working on updating patch 992750, but decided after a short while to just start from scratch, so I'm posting it as a new patch. Sorry if this isn't appropriate. This patch provides a new open() method on ZipFile; this method returns a file-like object for the requested item in the archive. This file-like object only provides a read() method. ZipFile.read was modified to use the new open method (this was suggested by loewis in reference to patch 992750). The patched zipfile.py passed the existing tests in the test_zipfile.py from CVS. New tests were added to verify the operation of the object returned by open(). These tests were modeled after existing tests for ZipFile.read(); two read fixed-size chunks from the file-like object, and two others read random-sized chunks. I have only run the tests on Windows XP, using Python2.4 from the official Windows installer. I will test the patch out on Linux over the weekend. If the patch is accepted I'll also generate and submit patches for the appropriate documentation as well. ---------------------------------------------------------------------- >Comment By: Alan McIntyre (alanmcintyre) Date: 2007-02-18 11:44 Message: Logged In: YES user_id=1115903 Originator: YES Thanks, glyph; I'll definitely have a look and see what I can steal. :-) ---------------------------------------------------------------------- Comment By: Glyph Lefkowitz (glyf) Date: 2007-02-16 22:48 Message: Logged In: YES user_id=226807 Originator: NO Twisted also contains an implementation of this functionality, available from http://twistedmatrix.com/trac/browser/trunk/twisted/python/zipstream.py As far as I can tell it doesn't have anything to recommend it over the attached patch, however (in fact, the test coverage of the attached patch looks better), but perhaps it has some behavior which might be desirable to steal. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2007-02-14 12:13 Message: Logged In: YES user_id=1115903 Originator: YES I will see if I can make some progress on this over the weekend. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 04:14 Message: Logged In: YES user_id=21627 Originator: NO Can you please update the patch to the current subversion trunk? I'd like to apply it, but the code of zipfile has changed so that the patch is currently out-of-date. When redoing it, notice that the read implementation has changed (I couldn't figure out how you moved code around). Please do use the trunk (not Python 2.5), and please submit the patch as a single 'svn diff' output (rather than a tar file containing multiple individual diff files). ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-05-30 21:56 Message: Logged In: YES user_id=1115903 Revision 5 of this patch has been in constant use with Python 2.4.1 in an application at my job for about a month; it seems to be stable and useful in that regard. If anybody has time to review and accept (or reject) it I would appreciate it. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-26 21:23 Message: Logged In: YES user_id=1115903 After testing on my large batch of large Zip files, I made one fix (version 4 of the patch didn't read all the content of large compressed archive items). The current set of changes is attached as zipfile_patch5.tgz. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-13 11:34 Message: Logged In: YES user_id=1115903 I found a problem with my universal newline handling code in readline(); if the first byte of an '\r\n' pair was read from file but the second byte didn't come in on that same read, it resulted in the next line having an inappropriate '\n' prepended to it. zipfile_patch4.tgz has a fix for this included (with everything else, of course). I'm going to test the open() capability in a real application with some reasonably large zip files (containing files up to just short of 2GB) over the next few days, so if any bugs or performance problems show up I may have some more changes. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-13 00:58 Message: Logged In: YES user_id=1115903 Uploaded the third revision of this patch; passes all regression tests against current CVS on WinXP. I think all the issues Martin brought up have been addressed except perhaps for the case of compression rate <1. I will have a look at that when I have time; just wanted to get an update here before the patch started to look abandoned. :) ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-03-14 09:37 Message: Logged In: YES user_id=1115903 Hmm...I could have sworn I did the diff in the correct order. I'll make sure next time. :) Here's my comments on your remarks (in order): - I'm adding support for universal newlines, and will reject all modes that aren't legal combinations of r, U, and b. - I'll see if I can make a Zip file store something with compression < 1, and if I can I'll add a test case for it. - I'll try work a .flush() in there on the compression object and come up with a test case if possible - .read(0) and .readline(0) will both return an empty string with no side-effects, since this seems to be what builtin files do. Right now ZipExtFile uses the ZipFile's file object, so you pretty much have to do whatever you're going to do with the ZipExtFile instance you get back from .open() before you use that ZipFile instance in a way that moves the file pointer around. I'm sure that somebody will eventually try to use the ZipFile in this way, so it appears my options are either to (1) give the ZipExtFile its own file object to use (when possible), or (2) make sure this limitation is documented. #1 feels (to me) to be the "right thing" to do, so that's what I'll try unless there's a good reason I shouldn't. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-03-01 02:59 Message: Logged In: YES user_id=21627 The patch is reversed: usually, diff is invoked as "-c old new". I think it is almost right, but I have a few remarks: - by tradition, open() should have a mode argument, defaulting to 'r'; it would be ok to raise exceptions if it is anything else. However, do consider implementing universal newlines; allowing 'b' as a no-op might also be reasonable. - I wonder what happens if the compression rate is < 1. It would appear that the code might use too few rawbytes. I suggest to recursively invoke read in this case. - I wonder whether it could ever happen that there is still data to uncompress in the zlib object, ie. whether it might be necessary to invoke .flush() after exhausting the rawbytes (and discard the zlib object afterwards) - it appears that atleast the builtin file object implements .read(0) as returning an empty string; the manual says that the entire file is meant only if size is omitted or negative. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-02-27 01:28 Message: Logged In: YES user_id=1115903 zipfile_patch2.tgz: I updated the file-like object to support readline, readlines, __iter__ and next(). Added tests for those new methods. Also added a patch for the documentation. Passes regression tests on 2.5a0 built from CVS HEAD with MSVC .NET on Windows XP. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 From noreply at sourceforge.net Sun Feb 18 19:41:43 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 18 Feb 2007 10:41:43 -0800 Subject: [Patches] [ python-Patches-1121142 ] ZipFile.open - read-only file-like obj for files in archive Message-ID: Patches item #1121142, was opened at 2005-02-11 19:08 Message generated for change (Comment added) made by alanmcintyre You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: Out of Date Priority: 5 Private: No Submitted By: Alan McIntyre (alanmcintyre) Assigned to: Nobody/Anonymous (nobody) Summary: ZipFile.open - read-only file-like obj for files in archive Initial Comment: I originally started working on updating patch 992750, but decided after a short while to just start from scratch, so I'm posting it as a new patch. Sorry if this isn't appropriate. This patch provides a new open() method on ZipFile; this method returns a file-like object for the requested item in the archive. This file-like object only provides a read() method. ZipFile.read was modified to use the new open method (this was suggested by loewis in reference to patch 992750). The patched zipfile.py passed the existing tests in the test_zipfile.py from CVS. New tests were added to verify the operation of the object returned by open(). These tests were modeled after existing tests for ZipFile.read(); two read fixed-size chunks from the file-like object, and two others read random-sized chunks. I have only run the tests on Windows XP, using Python2.4 from the official Windows installer. I will test the patch out on Linux over the weekend. If the patch is accepted I'll also generate and submit patches for the appropriate documentation as well. ---------------------------------------------------------------------- >Comment By: Alan McIntyre (alanmcintyre) Date: 2007-02-18 13:41 Message: Logged In: YES user_id=1115903 Originator: YES Here is a version of the patch against the current trunk. It passes a "make test" on my Gentoo laptop at the moment, but I think I still need to add some more tests (like reading from encrypted compressed files), and I need to try out the test_zipfile64 stuff. I don't have time today to look through the module Glyph suggested, so it may be that after I do that I'll have some more tweaks to make. I'm pretty sure I'll be able to spend some more time on Wednesday this week. I also included the full contents of zipfile.py and test_zipfile.py just in case that's useful to somebody. File Added: zipfile_patch6.tgz ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2007-02-18 11:44 Message: Logged In: YES user_id=1115903 Originator: YES Thanks, glyph; I'll definitely have a look and see what I can steal. :-) ---------------------------------------------------------------------- Comment By: Glyph Lefkowitz (glyf) Date: 2007-02-16 22:48 Message: Logged In: YES user_id=226807 Originator: NO Twisted also contains an implementation of this functionality, available from http://twistedmatrix.com/trac/browser/trunk/twisted/python/zipstream.py As far as I can tell it doesn't have anything to recommend it over the attached patch, however (in fact, the test coverage of the attached patch looks better), but perhaps it has some behavior which might be desirable to steal. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2007-02-14 12:13 Message: Logged In: YES user_id=1115903 Originator: YES I will see if I can make some progress on this over the weekend. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 04:14 Message: Logged In: YES user_id=21627 Originator: NO Can you please update the patch to the current subversion trunk? I'd like to apply it, but the code of zipfile has changed so that the patch is currently out-of-date. When redoing it, notice that the read implementation has changed (I couldn't figure out how you moved code around). Please do use the trunk (not Python 2.5), and please submit the patch as a single 'svn diff' output (rather than a tar file containing multiple individual diff files). ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-05-30 21:56 Message: Logged In: YES user_id=1115903 Revision 5 of this patch has been in constant use with Python 2.4.1 in an application at my job for about a month; it seems to be stable and useful in that regard. If anybody has time to review and accept (or reject) it I would appreciate it. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-26 21:23 Message: Logged In: YES user_id=1115903 After testing on my large batch of large Zip files, I made one fix (version 4 of the patch didn't read all the content of large compressed archive items). The current set of changes is attached as zipfile_patch5.tgz. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-13 11:34 Message: Logged In: YES user_id=1115903 I found a problem with my universal newline handling code in readline(); if the first byte of an '\r\n' pair was read from file but the second byte didn't come in on that same read, it resulted in the next line having an inappropriate '\n' prepended to it. zipfile_patch4.tgz has a fix for this included (with everything else, of course). I'm going to test the open() capability in a real application with some reasonably large zip files (containing files up to just short of 2GB) over the next few days, so if any bugs or performance problems show up I may have some more changes. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-13 00:58 Message: Logged In: YES user_id=1115903 Uploaded the third revision of this patch; passes all regression tests against current CVS on WinXP. I think all the issues Martin brought up have been addressed except perhaps for the case of compression rate <1. I will have a look at that when I have time; just wanted to get an update here before the patch started to look abandoned. :) ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-03-14 09:37 Message: Logged In: YES user_id=1115903 Hmm...I could have sworn I did the diff in the correct order. I'll make sure next time. :) Here's my comments on your remarks (in order): - I'm adding support for universal newlines, and will reject all modes that aren't legal combinations of r, U, and b. - I'll see if I can make a Zip file store something with compression < 1, and if I can I'll add a test case for it. - I'll try work a .flush() in there on the compression object and come up with a test case if possible - .read(0) and .readline(0) will both return an empty string with no side-effects, since this seems to be what builtin files do. Right now ZipExtFile uses the ZipFile's file object, so you pretty much have to do whatever you're going to do with the ZipExtFile instance you get back from .open() before you use that ZipFile instance in a way that moves the file pointer around. I'm sure that somebody will eventually try to use the ZipFile in this way, so it appears my options are either to (1) give the ZipExtFile its own file object to use (when possible), or (2) make sure this limitation is documented. #1 feels (to me) to be the "right thing" to do, so that's what I'll try unless there's a good reason I shouldn't. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-03-01 02:59 Message: Logged In: YES user_id=21627 The patch is reversed: usually, diff is invoked as "-c old new". I think it is almost right, but I have a few remarks: - by tradition, open() should have a mode argument, defaulting to 'r'; it would be ok to raise exceptions if it is anything else. However, do consider implementing universal newlines; allowing 'b' as a no-op might also be reasonable. - I wonder what happens if the compression rate is < 1. It would appear that the code might use too few rawbytes. I suggest to recursively invoke read in this case. - I wonder whether it could ever happen that there is still data to uncompress in the zlib object, ie. whether it might be necessary to invoke .flush() after exhausting the rawbytes (and discard the zlib object afterwards) - it appears that atleast the builtin file object implements .read(0) as returning an empty string; the manual says that the entire file is meant only if size is omitted or negative. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-02-27 01:28 Message: Logged In: YES user_id=1115903 zipfile_patch2.tgz: I updated the file-like object to support readline, readlines, __iter__ and next(). Added tests for those new methods. Also added a patch for the documentation. Passes regression tests on 2.5a0 built from CVS HEAD with MSVC .NET on Windows XP. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 From noreply at sourceforge.net Mon Feb 19 09:17:30 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 19 Feb 2007 00:17:30 -0800 Subject: [Patches] [ python-Patches-1663226 ] setuptools: avoid sets module for python>2.3 Message-ID: Patches item #1663226, was opened at 2007-02-19 10:17 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1663226&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Distutils and setup.py Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Amit Aronovitch (amitar) Assigned to: Nobody/Anonymous (nobody) Summary: setuptools: avoid sets module for python>2.3 Initial Comment: In Python 2.6 usage of the sets module issues a warning. patched to use builtin frozenset if version>=2.4 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1663226&group_id=5470 From noreply at sourceforge.net Mon Feb 19 09:33:46 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 19 Feb 2007 00:33:46 -0800 Subject: [Patches] [ python-Patches-1663234 ] enable "python -m doctest FILE" to run tests in FILE Message-ID: Patches item #1663234, was opened at 2007-02-19 09:33 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1663234&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Stefan Behnel (scoder) Assigned to: Nobody/Anonymous (nobody) Summary: enable "python -m doctest FILE" to run tests in FILE Initial Comment: This is a small patch that lets doctest.py support being called via "python -m" in a more useful way. Currently, it only runs a selftest. With the patch applied, you can pass one or more filenames, which will let it process these instead. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1663234&group_id=5470 From noreply at sourceforge.net Mon Feb 19 11:22:22 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 19 Feb 2007 02:22:22 -0800 Subject: [Patches] [ python-Patches-1607087 ] popen() slow on AIX due to large FOPEN_MAX value Message-ID: Patches item #1607087, was opened at 2006-12-01 21:05 Message generated for change (Comment added) made by hvbargen You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607087&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: John L. Allen (johnlallen) Assigned to: Nobody/Anonymous (nobody) Summary: popen() slow on AIX due to large FOPEN_MAX value Initial Comment: Experimentation revealed that the os.popen[234]() family of methods was at least 10 times slower on AIX than it was on Solaris. It turns out that this is because of the _run_child method in popen2.py, which has this definition: def _run_child(self, cmd): if isinstance(cmd, basestring): cmd = ['/bin/sh', '-c', cmd] for i in xrange(3, MAXFD): try: os.close(i) except OSError: pass try: os.execvp(cmd[0], cmd) finally: os._exit(1) MAXFD is set as follows at the top of popen2.py: try: MAXFD = os.sysconf('SC_OPEN_MAX') except (AttributeError, ValueError): MAXFD = 256 On AIX, SC_OPEN_MAX is 32767, whereas on Solaris, it is not defined, and we get the default value of 256. So, on AIX the python code for loop is being used to close 32763 file descriptors, but only 253 on Solaris. The slowness of this python loop is the source of the problem. Several solutions are possible. AIX provides a much faster way to close all file descriptors above a given one using the fcntl F_CLOSEM option. In this case, it would be: fcntl(3, F_CLOSEM, 0). Other OSes, like Solaris and BSD, have the closefrom() function instead. I think ideally, we would want to have an os.closefrom() method defined in posixmodule.c, and always available on every OS, and have popen2.py call that instead of doing the loop. The closefrom() function would be defined something like this: -------------------- PyDoc_STRVAR(posix_closefrom__doc__, "closefrom(fd)\n\n\ Close all file descriptors greater than or equal to fd (for low level IO)."); static PyObject * posix_closefrom(PyObject *self, PyObject *args) { int fd, maxfd, res; if (!PyArg_ParseTuple(args, "i:closefrom", &fd)) return NULL; Py_BEGIN_ALLOW_THREADS #ifdef HAVE_CLOSEFROM res = closefrom(fd); #else #ifdef F_CLOSEM res = fcntl(3, F_CLOSEM, 0) #else #if defined( HAVE_SYSCONF ) && defined( _SC_OPEN_MAX ) # define PY_OPEN_MAX sysconf(_SC_OPEN_MAX) #else # ifdef FOPEN_MAX # define PY_OPEN_MAX FOPEN_MAX # else # ifdef OPEN_MAX # define PY_OPEN_MAX OPEN_MAX # else # ifdef _NFILE # define PY_OPEN_MAX _NFILE # else # define PY_OPEN_MAX 256 # endif # endif # endif #endif maxfd = PY_OPEN_MAX; while (fd < maxfd) { res = close(fd); fd++; } #endif #endif Py_END_ALLOW_THREADS if (res < 0) return posix_error(); Py_INCREF(Py_None); return Py_None; } --------------------- While this is probably (close to) the ideal solution (since it would benefit all OSes by avoiding the close loop if possible or if not, moving it to C code), adding os.closefrom() probably needs to be discussed further before being accepted by the Python community. Instead, I will provide a simpler patch that only benefits AIX. It adds the F_CLOSEM attribute to the fcntl class in fcntlmodule.c, if defined, and modifies popen2.py to check for it and use it if possible, instead of doing the close() loop. See the attached patch (against the 2.5 source). I don't believe that any documentation has to change. John Allen ---------------------------------------------------------------------- Comment By: H. von Bargen (hvbargen) Date: 2007-02-19 11:22 Message: Logged In: YES user_id=1008979 Originator: NO I have submitted a new bug and referenced this patch from there. I think subprocess.py should be patched in a similar way. But, subprocess.py uses a "but" argument in _close_fds. I think a correct solution should close all handles from 3 to but-1 in C using the usual "close()" API; and then close all handles from but+1 to MAX_FD by using the optimization with closefrom or fnctl. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607087&group_id=5470 From noreply at sourceforge.net Mon Feb 19 11:33:48 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 19 Feb 2007 02:33:48 -0800 Subject: [Patches] [ python-Patches-1435651 ] Detect incomplete readline implementation Message-ID: Patches item #1435651, was opened at 2006-02-21 05:33 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1435651&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Skip Montanaro (montanaro) Assigned to: Nobody/Anonymous (nobody) Summary: Detect incomplete readline implementation Initial Comment: Here's a patch for setup.py to detect BSD-ish readlines that don't have the full suite of readline/history functions needed by Python (and provided by GNU readline). Skip ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-19 11:33 Message: Logged In: YES user_id=21627 Originator: NO I'm rejecting this patch as out-of-date. r42562 added a similar change, although restricted to OSX. If anybody wants to work on it again, I recommend to add checks for exported functions to configure, then query them in setup.py (doing it in distutils is probably impractical, as you would want to cache the result, just as configure output is cached in config.status). ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2006-02-21 05:50 Message: Logged In: YES user_id=357491 You mentioned on python-dev and in the diff that autoconf could detect this better. Would adding something to configure and having distutils.sysconfig.get_config_var() see if it is set be a better solution? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1435651&group_id=5470 From noreply at sourceforge.net Mon Feb 19 11:37:24 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 19 Feb 2007 02:37:24 -0800 Subject: [Patches] [ python-Patches-1435651 ] Detect incomplete readline implementation Message-ID: Patches item #1435651, was opened at 2006-02-21 05:33 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1435651&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.5 >Status: Closed >Resolution: Rejected Priority: 5 Private: No Submitted By: Skip Montanaro (montanaro) Assigned to: Nobody/Anonymous (nobody) Summary: Detect incomplete readline implementation Initial Comment: Here's a patch for setup.py to detect BSD-ish readlines that don't have the full suite of readline/history functions needed by Python (and provided by GNU readline). Skip ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-19 11:33 Message: Logged In: YES user_id=21627 Originator: NO I'm rejecting this patch as out-of-date. r42562 added a similar change, although restricted to OSX. If anybody wants to work on it again, I recommend to add checks for exported functions to configure, then query them in setup.py (doing it in distutils is probably impractical, as you would want to cache the result, just as configure output is cached in config.status). ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2006-02-21 05:50 Message: Logged In: YES user_id=357491 You mentioned on python-dev and in the diff that autoconf could detect this better. Would adding something to configure and having distutils.sysconfig.get_config_var() see if it is set be a better solution? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1435651&group_id=5470 From noreply at sourceforge.net Mon Feb 19 11:56:32 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 19 Feb 2007 02:56:32 -0800 Subject: [Patches] [ python-Patches-1490190 ] add os.chflags() and os.lchflags() where available Message-ID: Patches item #1490190, was opened at 2006-05-17 13:45 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1490190&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.5 >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: M. Levinson (levinsm) Assigned to: Neal Norwitz (nnorwitz) Summary: add os.chflags() and os.lchflags() where available Initial Comment: The return value from os.stat() includes st_flags on some systems, but currently there's not much that can be done with the value; this patch aims to make st_flags useful by adding some associated constants to stat.py and the corresponding chflags() and lchflags() functions in posixmodule. For completeness, shutil.copystat() is also updated to call os.chflags() where it's available. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-19 11:56 Message: Logged In: YES user_id=21627 Originator: NO Thanks for the patch. Committed as r53826. ---------------------------------------------------------------------- Comment By: M. Levinson (levinsm) Date: 2006-05-19 13:47 Message: Logged In: YES user_id=1522893 In addition to MacOS, chflags(2) is available on FreeBSD, OpenBSD, and NetBSD. The flags in Lib/stat.py are the full set of available values although, as you noted, MacOS hasn't (yet) implemented several of them. Thanks for the comments - I've attached an updated version of the patch incorporating your suggestions. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-05-19 09:31 Message: Logged In: YES user_id=33168 What operating systems is this available on? The only one I've found is OS X. The man page says it's from BSD 4.4. I tried on Linux of various flavors (4+), Solaris, and Tru64. None of them had chflags. I also could only find some of the flags in my sys/stat.h that were added to stat.py. stat.h didn't have UF_NOUNLINK, SF_NOUNLINK, SF_SNAPSHOT. As far as the patch itself, it looks good. There are a couple of changes if this should be accepted though: doc needs \versionadded{2.5}, I would prefer flags as the var name rather than i in posixmodule.c (btw you shouldn't need to init path). Also would need to update Misc/NEWS and ACKS if accepted. ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2006-05-17 16:24 Message: Logged In: YES user_id=849994 Patch looks good. Do we want to include it? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1490190&group_id=5470 From noreply at sourceforge.net Mon Feb 19 13:05:12 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 19 Feb 2007 04:05:12 -0800 Subject: [Patches] [ python-Patches-1663226 ] setuptools: avoid sets module for python>2.3 Message-ID: Patches item #1663226, was opened at 2007-02-19 08:17 Message generated for change (Settings changed) made by gbrandl You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1663226&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Distutils and setup.py Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Amit Aronovitch (amitar) >Assigned to: Phillip J. Eby (pje) Summary: setuptools: avoid sets module for python>2.3 Initial Comment: In Python 2.6 usage of the sets module issues a warning. patched to use builtin frozenset if version>=2.4 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1663226&group_id=5470 From noreply at sourceforge.net Mon Feb 19 13:08:05 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 19 Feb 2007 04:08:05 -0800 Subject: [Patches] [ python-Patches-1663234 ] enable "python -m doctest FILE" to run tests in FILE Message-ID: Patches item #1663234, was opened at 2007-02-19 08:33 Message generated for change (Comment added) made by gbrandl You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1663234&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Stefan Behnel (scoder) Assigned to: Nobody/Anonymous (nobody) Summary: enable "python -m doctest FILE" to run tests in FILE Initial Comment: This is a small patch that lets doctest.py support being called via "python -m" in a more useful way. Currently, it only runs a selftest. With the patch applied, you can pass one or more filenames, which will let it process these instead. ---------------------------------------------------------------------- >Comment By: Georg Brandl (gbrandl) Date: 2007-02-19 12:08 Message: Logged In: YES user_id=849994 Originator: NO Looks good and useful to me. A documentation patch would be needed though. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1663234&group_id=5470 From noreply at sourceforge.net Mon Feb 19 13:10:45 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 19 Feb 2007 04:10:45 -0800 Subject: [Patches] [ python-Patches-900744 ] catch invalid chunk length in httplib read routine Message-ID: Patches item #900744, was opened at 2004-02-19 23:14 Message generated for change (Comment added) made by gbrandl You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=900744&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: Accepted Priority: 5 Private: No Submitted By: Wummel (calvin) Assigned to: Georg Brandl (gbrandl) Summary: catch invalid chunk length in httplib read routine Initial Comment: In HTTPResponse._read_chunked the chunk length is not checked to be a valid integer, and a ValueError will be raised in such a case. The attached patch catches ValueError (which should not normally happen, so this try:except: is reasonably fast), and raises IncompleteRead exception instead. I have no test case for this yet, but am trying to construct one :) ---------------------------------------------------------------------- >Comment By: Georg Brandl (gbrandl) Date: 2007-02-19 12:10 Message: Logged In: YES user_id=849994 Originator: NO Martin: which patch? Mine, which returns what was read so far, or calvin's, which raises IncompleteRead? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-18 08:39 Message: Logged In: YES user_id=21627 Originator: NO Georg, the patch is fine for 2.6, please apply (for 2.5, I would be cautious because of the behaviour change). ---------------------------------------------------------------------- Comment By: Wummel (calvin) Date: 2006-02-01 22:16 Message: Logged In: YES user_id=9205 I attached a simple testcase that triggers the bug. IMHO this patch should be applied for Python 2.5. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2005-07-19 01:07 Message: Logged In: YES user_id=80475 Technically, the patch is fine and it is the way the code should have been written in the first place. Please bring-up on python-dev to determine whether it should be done. I can imagine that a fair amount of existing code has through trial and error discovered the ValueError and chosen to catch that. Changing the exception may unnecessarily break people's code. Sometimes we take that step when it needs to be done, but there should be a good pay-off and, in this case, I don't see it. You may also want to mention it on comp.lang.python to see if anyone cares If the patch goes forward, see if you can mock-up a test that triggers the exception so we have a good unittest. In anycase, this should not be backported (we avoid giving people reasons to not upgrade). ---------------------------------------------------------------------- Comment By: Georg Brandl (birkenfeld) Date: 2005-07-18 19:34 Message: Logged In: YES user_id=1188172 Attaching patch which does what agwego said (httplib-chunked.diff). Please review. ---------------------------------------------------------------------- Comment By: agwego (agwego) Date: 2005-02-28 16:53 Message: Logged In: YES user_id=1228982 I've run into this problem in conjunction with mod_python 3.1.4 (and although the problem is caused in mod_python) my python skills aren't up to the task. Which leaves me with fixing httplib. Although the above patch works when it comes to end of file situations, I think it would be better to return what has been consumed so far and leave it at that. A few lines down there's a comment about consuming trailers, this is the case that is tripping up httplib as far as I can tell. This is happening in Python 2.3.4. --- packages/Python-2.3.4/Lib/httplib.py Sun Nov 2 11:51:38 2003 +++ httplib.py Mon Feb 28 11:26:48 2005 @@ -423,7 +423,11 @@ i = line.find(';') if i >= 0: line = line[:i] # strip chunk-extensions - chunk_left = int(line, 16) + try: + chunk_left = int(line, 16) + except ValueError, msg: + self.close() + return value if chunk_left == 0: break if amt is None: ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=900744&group_id=5470 From noreply at sourceforge.net Mon Feb 19 13:42:09 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 19 Feb 2007 04:42:09 -0800 Subject: [Patches] [ python-Patches-1663234 ] enable "python -m doctest FILE" to run tests in FILE Message-ID: Patches item #1663234, was opened at 2007-02-19 09:33 Message generated for change (Comment added) made by scoder You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1663234&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Stefan Behnel (scoder) Assigned to: Nobody/Anonymous (nobody) Summary: enable "python -m doctest FILE" to run tests in FILE Initial Comment: This is a small patch that lets doctest.py support being called via "python -m" in a more useful way. Currently, it only runs a selftest. With the patch applied, you can pass one or more filenames, which will let it process these instead. ---------------------------------------------------------------------- >Comment By: Stefan Behnel (scoder) Date: 2007-02-19 13:42 Message: Logged In: YES user_id=313935 Originator: YES fair enough File Added: libdoctest.tex-run-module.patch ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2007-02-19 13:08 Message: Logged In: YES user_id=849994 Originator: NO Looks good and useful to me. A documentation patch would be needed though. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1663234&group_id=5470 From noreply at sourceforge.net Mon Feb 19 17:06:00 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 19 Feb 2007 08:06:00 -0800 Subject: [Patches] [ python-Patches-900744 ] catch invalid chunk length in httplib read routine Message-ID: Patches item #900744, was opened at 2004-02-20 00:14 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=900744&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open >Resolution: None Priority: 5 Private: No Submitted By: Wummel (calvin) Assigned to: Georg Brandl (gbrandl) Summary: catch invalid chunk length in httplib read routine Initial Comment: In HTTPResponse._read_chunked the chunk length is not checked to be a valid integer, and a ValueError will be raised in such a case. The attached patch catches ValueError (which should not normally happen, so this try:except: is reasonably fast), and raises IncompleteRead exception instead. I have no test case for this yet, but am trying to construct one :) ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-19 17:05 Message: Logged In: YES user_id=21627 Originator: NO I meant to suggest that your patch (based on agwego's recommendation) should be accepted. Now that you ask, I'm not so sure anymore: errors should never pass silently. So I now think, that it should raise IncompleteRead, based also on the observation that these methods can *already* raise IncompleteRead, and that this already carries the data that we received. OTOH, IIUC, the error also means that we lost protocol synchronisation with the server, so we should in any case close the connection. So merging these changes is probably the right thing :-) Withdrawing my acceptance again, and hoping that somebody can produce a complete patch (which includes the test case in a style that works with regrtest) ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2007-02-19 13:10 Message: Logged In: YES user_id=849994 Originator: NO Martin: which patch? Mine, which returns what was read so far, or calvin's, which raises IncompleteRead? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-18 09:39 Message: Logged In: YES user_id=21627 Originator: NO Georg, the patch is fine for 2.6, please apply (for 2.5, I would be cautious because of the behaviour change). ---------------------------------------------------------------------- Comment By: Wummel (calvin) Date: 2006-02-01 23:16 Message: Logged In: YES user_id=9205 I attached a simple testcase that triggers the bug. IMHO this patch should be applied for Python 2.5. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2005-07-19 03:07 Message: Logged In: YES user_id=80475 Technically, the patch is fine and it is the way the code should have been written in the first place. Please bring-up on python-dev to determine whether it should be done. I can imagine that a fair amount of existing code has through trial and error discovered the ValueError and chosen to catch that. Changing the exception may unnecessarily break people's code. Sometimes we take that step when it needs to be done, but there should be a good pay-off and, in this case, I don't see it. You may also want to mention it on comp.lang.python to see if anyone cares If the patch goes forward, see if you can mock-up a test that triggers the exception so we have a good unittest. In anycase, this should not be backported (we avoid giving people reasons to not upgrade). ---------------------------------------------------------------------- Comment By: Georg Brandl (birkenfeld) Date: 2005-07-18 21:34 Message: Logged In: YES user_id=1188172 Attaching patch which does what agwego said (httplib-chunked.diff). Please review. ---------------------------------------------------------------------- Comment By: agwego (agwego) Date: 2005-02-28 17:53 Message: Logged In: YES user_id=1228982 I've run into this problem in conjunction with mod_python 3.1.4 (and although the problem is caused in mod_python) my python skills aren't up to the task. Which leaves me with fixing httplib. Although the above patch works when it comes to end of file situations, I think it would be better to return what has been consumed so far and leave it at that. A few lines down there's a comment about consuming trailers, this is the case that is tripping up httplib as far as I can tell. This is happening in Python 2.3.4. --- packages/Python-2.3.4/Lib/httplib.py Sun Nov 2 11:51:38 2003 +++ httplib.py Mon Feb 28 11:26:48 2005 @@ -423,7 +423,11 @@ i = line.find(';') if i >= 0: line = line[:i] # strip chunk-extensions - chunk_left = int(line, 16) + try: + chunk_left = int(line, 16) + except ValueError, msg: + self.close() + return value if chunk_left == 0: break if amt is None: ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=900744&group_id=5470 From noreply at sourceforge.net Tue Feb 20 00:04:20 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 19 Feb 2007 15:04:20 -0800 Subject: [Patches] [ python-Patches-1491939 ] Fix for bug #1486663 mutable types check kwargs in tp_new Message-ID: Patches item #1491939, was opened at 2006-05-20 03:17 Message generated for change (Settings changed) made by zseil You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1491939&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None >Status: Closed >Resolution: Out of Date Priority: 5 Private: No Submitted By: ?iga Seilnacht (zseil) Assigned to: Raymond Hettinger (rhettinger) Summary: Fix for bug #1486663 mutable types check kwargs in tp_new Initial Comment: set and deque check that they are not called with keyword arguments in their tp_new method, although they are mutable. This makes them harder to subclass. See the bug report for more details. Patch contains tests and fixes for both of them. ---------------------------------------------------------------------- Comment By: Georg Brandl (gbrandl) Date: 2006-05-29 23:07 Message: Logged In: YES user_id=849994 Raymond, please check. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1491939&group_id=5470 From noreply at sourceforge.net Tue Feb 20 18:33:01 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 20 Feb 2007 09:33:01 -0800 Subject: [Patches] [ python-Patches-1664522 ] Fix for urllib.ftpwrapper.retrfile() and none existing files Message-ID: Patches item #1664522, was opened at 2007-02-20 18:33 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1664522&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Phil Knirsch (pknirsch) Assigned to: Nobody/Anonymous (nobody) Summary: Fix for urllib.ftpwrapper.retrfile() and none existing files Initial Comment: When trying to retrieve a none existing file using the urllib.ftpwrapper.retrfile() method the behaviour is that instead of an error message a valid hook is returned and you will recieve a 0 byte file. The current behaviour tries to emulate what one typically sees with http servers and DirIndexes, which means: 1) Try to RETR the file. 2) If that fails, assume it is a directory and LIST it. Unfortunately it doesn't actually check whether the directory actually exists. The attached patch fixes this by remembering the current directory using the PWD command, then temporarily change to that directory and switch back to the previous working directory if it was successfull. If not we raise an IO error, as the file could neither be opened (RETR) nor was it a directory. That way the behaviour is even closer to what happens with http servers where we get a 404 when we try to access a none existing file or directory. Storing the current directory and switching back to it in case of no error will also put the connection back in the proper state and directory, so no unexpected behaviour happens here. The patch is against the current SVN repository at revision 53833. Read ya, Phil ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1664522&group_id=5470 From noreply at sourceforge.net Wed Feb 21 22:26:27 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 21 Feb 2007 13:26:27 -0800 Subject: [Patches] [ python-Patches-1654974 ] Binding annotations in tracebacks. Message-ID: Patches item #1654974, was opened at 2007-02-08 09:46 Message generated for change (Comment added) made by nejucomo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1654974&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nefarious CodeMonkey, Jr. (nejucomo) Assigned to: Nobody/Anonymous (nobody) Summary: Binding annotations in tracebacks. Initial Comment: The attached module provides a suitable replacement for sys.excepthook to handle uncaught exceptions. The output is the same, except after each source line shown, a list of relevant bindings is shown. Here's a quick example from the tail end of a test: File "./test-exprann.py", line 16, in f assert c == 12 # With bindings: # c = 42 AssertionError The bindings shown are the intersection of the code object "co_names" and the non-keyword name tokens in the parsed source line. The goal is to only show bindings relevant to the exception. I hope the utility of this is self-evident to any programmer. I tried testing it with crazy expressions containing nonfree bindings (such as lambda's and list comprehensions), so I think it behaves well in most circumstances. The performance might be bad (it parses each line in the backtrace), but I figure it is worth it for uncaught exceptions. Let me know if you find this tool useful. Thanks, Nejucomo ---------------------------------------------------------------------- >Comment By: Nefarious CodeMonkey, Jr. (nejucomo) Date: 2007-02-21 21:26 Message: Logged In: YES user_id=44911 Originator: YES This feature has already been present in the stdlib for awhile, hidden in the cgitb module. Enable it like this: import cgitb cgitb.enable(format="text") I was not aware of this, assuming cgitb just formatted typical tracebacks into html. Perhaps a better design of traceback reporting in general is called for? ---------------------------------------------------------------------- Comment By: Nefarious CodeMonkey, Jr. (nejucomo) Date: 2007-02-12 19:00 Message: Logged In: YES user_id=44911 Originator: YES I've simplified and modularized the design of this tool. Included is a patch against the 2.6 svn head which adds a module to the stdlib called bindann (for "binding annotation"), and then modifies the traceback module to use it. It also includes a demo script which compares classic and annotated exception tracebacks. The interface to traceback is backwards compatible (it introduces optional arguments defaulting to pre-patch behavior). bindann is much simpler than the previous annex file, because it doesn't try parsing, only tokenizing the source line, to look for name references. Thus it succeeds where annex would fail (for example if the head of a for-loop causes an uncaught exception, the source line will only be the head which is not a complete/parsable expression). Here's the output of the demo: $ ./Demo/newfeatures/demo-bindann.py This demonstrates binding-annoted exception tracebacks. Consider the following function: def f(c): for char in c + " hello": print 'The string contains "%s".' % (char,) -With standard tracebacks, calling f(42) gives: Traceback (most recent call last): File "./Demo/newfeatures/demo-bindann.py", line 17, in main f(42) File "./Demo/newfeatures/demo-bindann.py", line 28, in f for char in c + " hello": TypeError: unsupported operand type(s) for +: 'int' and 'str' -And now with annotated exceptions: Traceback (most recent call last): File "./Demo/newfeatures/demo-bindann.py", line 33, in # With bindings: # main = # Source: main() File "./Demo/newfeatures/demo-bindann.py", line 24, in main # With bindings: # f = # Source: f(42) File "./Demo/newfeatures/demo-bindann.py", line 28, in f # With bindings: c = 42 # char = # Source: for char in c + " hello": TypeError: unsupported operand type(s) for +: 'int' and 'str' File Added: bindann-patch.1 ---------------------------------------------------------------------- Comment By: Nefarious CodeMonkey, Jr. (nejucomo) Date: 2007-02-08 09:50 Message: Logged In: YES user_id=44911 Originator: YES BTW, I only tested this on python 2.4. Because it examines the parse tree (very simply), it may not work if the AST format changes. The only parsing it does is search for all NAME tokens which are not keywords, so I assumed this was stable across releases. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1654974&group_id=5470 From noreply at sourceforge.net Thu Feb 22 04:23:22 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 21 Feb 2007 19:23:22 -0800 Subject: [Patches] [ python-Patches-988761 ] re.split emptyok flag (fix for #852532) Message-ID: Patches item #988761, was opened at 2004-07-10 22:25 Message generated for change (Comment added) made by mkc You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=988761&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules >Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Mike Coleman (mkc) Assigned to: Fredrik Lundh (effbot) Summary: re.split emptyok flag (fix for #852532) Initial Comment: This patch addresses bug #852532. The underlying problem is that re.split ignores any match it makes that has length zero, which blocks a number of useful possibilities. The attached patch implements a flag 'emptyok', which if set to True, causes re.split to allow zero length matches. My preference would be to just change the behavior of re.split, rather than adding this flag. The old behavior isn't documented (though a couple of cases in test_re.py do depend on it). As a practical matter, though, I realize that there may be some code out there relying on this undocumented behavior. And I'm hoping that this useful feature can be added quickly. Perhaps this new behavior could be made the default in a future version of Python. (Linux 2.6.3 i686) ---------------------------------------------------------------------- >Comment By: Mike Coleman (mkc) Date: 2007-02-21 21:23 Message: Logged In: YES user_id=555 Originator: YES Hello from 2004! This is your long-lost bug in re.split--how's it going? I'm still alive and well. I think everyone pretty much agrees that I really am a bug, and at least one guy still writes code just to work around me every few weeks or so. My attempt to keep a low profile is doing well--I'm not even documented in the library reference. This allows me to meet new Python users on a regular basis (whether they like it or not). Well, that's it for now. If I don't hear from you until then, I'll drop you another line in 2009. (Hey I'm a poet, too!) Regards, bug 852532/patch 988761 ---------------------------------------------------------------------- Comment By: Mike Coleman (mkc) Date: 2006-01-17 16:37 Message: Logged In: YES user_id=555 I think I still agree with my original answer on this (see http://mail.python.org/pipermail/python-dev/2004-August/047321.html). I'm completely worn down on this, though, so I'd happily take any of these options as an improvement over the present situation. ---------------------------------------------------------------------- Comment By: Filip Salomonsson (filips) Date: 2006-01-16 15:56 Message: Logged In: YES user_id=308203 I agree completely that splitting on non-zero matches should be supported - and that the default behavior should change at some point - but I don't think this patch quite covers it. Taking an example from the dev-python thread back in August of 2004 (http://mail.python.org/pipermail/python-dev/2004-August/047272.html): >>> re.split('x*', 'abxxxcdefxxx', emptyok=True) ['', 'a', 'b', '', 'c', 'd', 'e', 'f', '', ''] To me, this means there's an empty string, beginning and ending in pos 0, followed by a zero-width divider also beginning and ending in the same position, followed by an 'a', etc. That seems awkward to me. I think a more intuitive result would be (I'm omitting the emptyok argument in the following examples): >>> re.split('x*', 'abxxxcdefxxx') ['a', 'b', 'c', 'd', 'e', 'f', ''] That is, empty matches cause a split when they are not adjacent to a non-empty match and not at the beginning or the end of the string. Grouping parentheses would, of course, reveal the empty-string boundaries: >>> re.split('(x*)', 'abxxxcdefxxx') ['', 'a', '', 'b', 'xxx', '', 'c', '', 'd', '', 'e', '', 'f', 'xxx', ''] Using the same approach, these results would also seem perfectly reasonable to me: >>> re.split('(?m)$', 'foo\nbar\nbaz') ['foo', '\nbar', '\nbaz'] >>> re.split('(?m)^', 'foo\nbar\nbaz') ['foo\n', 'bar\n', 'baz'] Splitting a one-character string should be possible only if the pattern matches that character: >>> re.split('\w*', 'a') ['', ''] >>> re.split('\d*', 'a') ['a'] ---------------------------------------------------------------------- Comment By: Mike Coleman (mkc) Date: 2005-12-09 11:04 Message: Logged In: YES user_id=555 This patch seems to have been stalled now for over a year. Could it be applied? Or, alternatively, could someone provide some sort of reason why it shouldn't be? Thanks. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-09-05 17:53 Message: Logged In: YES user_id=80475 Fred, what do you think of the proposal. Are the backwards compatability issues worth it? ---------------------------------------------------------------------- Comment By: Mike Coleman (mkc) Date: 2004-09-03 15:15 Message: Logged In: YES user_id=555 Apparently this patch is stalled, but I'd like to get it in, in some form, for 2.4. The only question, as far as I know, is whether empty matches following non-empty matches "count" or not (they do in the original patch). If I make a patch with the "doesn't count" behavior, could we apply that right away? I'd rather get either behavior in for 2.4 than wait for 2.5... ---------------------------------------------------------------------- Comment By: Mike Coleman (mkc) Date: 2004-07-28 11:23 Message: Logged In: YES user_id=555 I picked through CVS, python-dev and google and came up with this. The current behavior was present way back in the earliest regsub.py in CVS (dated Sep 1992); subsequent implementation seem to be mirroring this behavior. The CVS comment back in 1992 described split as modeled on nawk. A check of nawk(1) confirms that nawk only splits on non-null matches. Perl (circa 5.6) on the other hand, appears to split the way this patch does (though I wasn't aware of that when I wrote the patch) so that might argue in the other direction. I would note, too, that re.findall and re.finditer tend in this direction ("Empty matches are included in the result unless they touch the beginning of another match."). The python-dev archive doesn't seem to go back far enough to be relevant and I'm not sure how to search it. General googling (python "re.split" empty match) found a few hits. Probably the most relevant is Tim Peters saying "Python won't change here (IMO)" and giving the example that he also gives in a comment to bug #852532 (which this patch addresses). He also wonders in his comment about the possibility of a "design constraint", but I think this patch addresses that concern. As far as I can tell, the current behavior was a design decision made over 10 years ago, between two alternatives that probably didn't matter much at the time. Skipping empty matches probably seemed harmless before lookahead/lookbehind assertions. Now, though, the current behavior seems like a significant hindrance. Furthermore, it ought to be pretty trivial to modify any existing patterns to get the old behavior, should that be desired (e.g., use 'x+' instead of 'x*'). (I didn't notice that re.findall doc when I originally wrote this patch. Perhaps the doc in the patch should be slightly modified to help emphasize the similarity between how re.findall and re.split handle empty matches.) ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-07-27 09:08 Message: Logged In: YES user_id=11375 Overall I like the patch and wouldn't mind seeing the change become the default behaviour. However, I'm nervous about possibly not understanding the reason the prohibition on zero-length matches was added in the first place. Can you please do some research in the CVS logs and python-dev archives to figure out why the limitation was implemented in the first place? ---------------------------------------------------------------------- Comment By: Chris King (colander_man) Date: 2004-07-21 07:46 Message: Logged In: YES user_id=573252 Practical example where the current behaviour produces undesirable results (splitting on character transitions): >>> import re >>> re.split(r'(?<=[A-Z])(?=[^a-z])','SOMEstring') ['SOMEstring'] # desired is ['SOME','string'] ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=988761&group_id=5470 From noreply at sourceforge.net Fri Feb 23 01:18:27 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Thu, 22 Feb 2007 16:18:27 -0800 Subject: [Patches] [ python-Patches-1666625 ] Allow specifying headers for MIME parts Message-ID: Patches item #1666625, was opened at 2007-02-23 01:18 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1666625&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Private: No Submitted By: J?rg Sonnenberger (jsonn) Assigned to: Nobody/Anonymous (nobody) Summary: Allow specifying headers for MIME parts Initial Comment: The patch is needed to allow specification e.g. of content-transfer-encoding. I couldn't find a way with the existing infrastructure to do that. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1666625&group_id=5470 From noreply at sourceforge.net Fri Feb 23 20:28:35 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 23 Feb 2007 11:28:35 -0800 Subject: [Patches] [ python-Patches-1663226 ] setuptools: avoid sets module for python>2.3 Message-ID: Patches item #1663226, was opened at 2007-02-19 08:17 Message generated for change (Comment added) made by pje You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1663226&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Distutils and setup.py Group: None >Status: Closed >Resolution: Fixed Priority: 5 Private: No Submitted By: Amit Aronovitch (amitar) Assigned to: Phillip J. Eby (pje) Summary: setuptools: avoid sets module for python>2.3 Initial Comment: In Python 2.6 usage of the sets module issues a warning. patched to use builtin frozenset if version>=2.4 ---------------------------------------------------------------------- >Comment By: Phillip J. Eby (pje) Date: 2007-02-23 19:28 Message: Logged In: YES user_id=56214 Originator: NO I've fixed the problem for setuptools 0.6c6, but did not use the attached patch, as it was incomplete and relies on Python version checking rather than Python feature checking. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1663226&group_id=5470 From noreply at sourceforge.net Fri Feb 23 23:07:51 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 23 Feb 2007 14:07:51 -0800 Subject: [Patches] [ python-Patches-1183712 ] package_data chops off first char of default package Message-ID: Patches item #1183712, was opened at 2005-04-15 12:34 Message generated for change (Comment added) made by pje You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1183712&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Distutils and setup.py Group: Python 2.4 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Wummel (calvin) Assigned to: Nobody/Anonymous (nobody) Summary: package_data chops off first char of default package Initial Comment: If the package name is an empty string (ie the default package), all package_data files have the first character chopped off. Attached is a test package pytest.tar.gz where running python2.4 setup.py build_py produces this error: running build_py creating build creating build/lib copying __init__.py -> build/lib error: can't copy 'ATA': doesn't exist or not a regular file Also attached is a fix proposal, though I have tested this only against the test package. ---------------------------------------------------------------------- >Comment By: Phillip J. Eby (pje) Date: 2007-02-23 22:07 Message: Logged In: YES user_id=56214 Originator: NO I concur with Martin: setup() should disallow an empty string as a package, because an empty string is not a valid Python package name. Regarding Jim's comment that the stripping should be smarter, I'd like to point out that package_dirs must only contain platform-independent directory names, and if anything else is included, it will produce an error when the build_py command is initialized, as 'convert_path' is called on the paths in question. Thus, they can never contain absolute paths or end in a trailing /. Thus, the only problem here is that packages is allowed to contain empty strings, which it makes no sense to include. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-08-31 13:26 Message: Logged In: YES user_id=764593 I think the patch is still missing a case or two. plen represents the length of the path prefix to ignore. Today's code computes this as (len(src_dir) + 1) where the +1 is for the "/" added by os.path.join(). As you found, src_dir won't add anything (including the "/") if src_dir is empty. But it also won't add the "/" if src_dir already ends in "/ ", and it won't even add the src_dir if the path is already absolute. I'm not certain that either of these two cases can occur, but it would be safer to assume they can. My suggestion is that the stripping be smarter -- change """ # Strip directory from globbed filenames filenames = [ file[plen:] for file in self.find_data_files(package, src_dir) ] """ to """ # Strip directory from globbed filenames filenames = [ filetail(name, src_dir) for name in self.find_data_files(package, src_dir) ] """ where filetail is a helper function defined as """ def filetail(name, strip_path): if name.startswith(strip_path): kill=len(strip_path) if name[kill] == "/": kill +=1 name=name[kill:] return name """ with tests """ >>> filetail("asdf/bdededf", "asdf") 'bdededf' >>> filetail("asdf/bdededf", "asdf/") 'bdededf' """ ---------------------------------------------------------------------- Comment By: Wummel (calvin) Date: 2006-05-22 20:13 Message: Logged In: YES user_id=9205 I found it in another Python program (don't remember which though). So I did not think of this as an undocumented feature. I tried it and it worked (except the data file stuff :). The patch should not break any currently working setup.py installation, since src_dir is only empty when using '' (empty string) as package name. Perhaps a cleaner approach would be to forbid an empty package name instead of silently accepting it? I am not sure. At least the documentation should mention that empty package names are not allowed. ---------------------------------------------------------------------- Comment By: Martin v. L??wis (loewis) Date: 2006-04-15 08:23 Message: Logged In: YES user_id=21627 Why are you using an empty name as the package name? There is no default package in Python, so this shouldn't work at all. ---------------------------------------------------------------------- Comment By: Herv? Cauwelier (hcauwelier) Date: 2005-10-05 11:03 Message: Logged In: YES user_id=1216236 The patch worked well for me, thanks for it! ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1183712&group_id=5470 From noreply at sourceforge.net Sat Feb 24 00:25:33 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 23 Feb 2007 15:25:33 -0800 Subject: [Patches] [ python-Patches-1667546 ] Time zone-capable variant of time.localtime Message-ID: Patches item #1667546, was opened at 2007-02-24 00:25 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667546&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Paul Boddie (pboddie) Assigned to: Nobody/Anonymous (nobody) Summary: Time zone-capable variant of time.localtime Initial Comment: Patch related to #1493676: "time.strftime() %z error" This provides a localtime_tz function whose return value is the usual localtime time tuple with an additional field reflecting the underlying tm_gmtoff data. Various internal function signatures are modified to support the flow of time zone information, with the gettmarg most noticably changed (probably quite inelegantly - I don't do Python core development). This patch is against the Python 2.4.4 release sources. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667546&group_id=5470 From noreply at sourceforge.net Sat Feb 24 01:02:53 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Fri, 23 Feb 2007 16:02:53 -0800 Subject: [Patches] [ python-Patches-762963 ] timemodule.c: Python loses current timezone Message-ID: Patches item #762963, was opened at 2003-06-30 04:18 Message generated for change (Comment added) made by pboddie You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=762963&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Doug Quale (quale) Assigned to: Martin v. L?wis (loewis) Summary: timemodule.c: Python loses current timezone Initial Comment: Python routines in timemodule.c sometimes force glibc to use GMT instead of the correct timezone. I think this is a longstanding bug, but I have only looked at Python 2.2 and 2.33b1. This bug has been reported before (510218) but it was misdiagnosed and closed. It also came up recently in on the python-list (http://aspn.activestate.com/ASPN/Mail/Message/python-list/1564384) where it was also misdiagnosed. Standard C and Python use a struct tm with 9 values. BSD influenced C libraries like glibc have an extra field (tm_gmtoff) that keeps the offset from UTC. BSD-style struct tm values know the correct timezone associated with the time, but Python and Standard C struct tm values don't so they can only assume the current timezone. Ideally Python would always treat stuct tm-style time tuples as being associated with the current timezone. Unfortunately in many cases Python forces glibc to use GMT rather than the current timezone. You can see the problem most graphically with the %z format in strftime(). In the transcript Python incorrectly gives the timezone offset as +0000 even though it gets the timezone name (CDT) correct: $ python2.3 Python 2.3b1+ (#2, Jun 4 2003, 03:03:32) [GCC 3.3 (Debian)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import time >>> now = time.localtime() >>> print now (2003, 6, 29, 20, 3, 52, 6, 180, 1) >>> RFC822 = '%a, %d %b %Y %T %z' >>> print time.strftime(RFC822, now) Sun, 29 Jun 2003 20:03:52 +0000 >>> print time.strftime(RFC822) Sun, 29 Jun 2003 20:05:27 -0500 >>> print time.strftime('%Z', now) CDT The correct timezone is CDT and the correct offset is -0500. Notice that when time.strftime() computes the current time it it gets the correct timezone offset, but when the struct tm tuple is passed in it thinks the timezone offset is 0. (glibc does know that the correct timezone name is CDT even when passed a bad struct tm value, but that's not surprising since the timezone name is not stored in struct tm and it is not computed from timezone offset.) The problem is in the gettmargs() function in timemodule.c. When getmargs() parses a Python tm tuple it leaves the tm_gmtoff field zeroed. This specifically tells glibc that the timezone offset is 0, which is wrong for most people. (BSD libc may also be affected.) This problem does not occur when time_strfrtime() gets the current time itself since that codepath uses a struct tm value directly from the libc localtime(), leaving the tm_gmtoff field intact. Fortunately the fix is almost trivial. A call to mktime() will normalize the fields in the struct tm value, including the tm_gmtoff field. I conditionalized the patch only on HAVE_MKTIME. I'm pretty sure there's an autoconfigure test for tm_gmtoff and it would probably be better to use that. ---------------------------------------------------------------------- Comment By: Paul Boddie (pboddie) Date: 2007-02-24 01:02 Message: Logged In: YES user_id=226443 Originator: NO I've just tried this patch against Python 2.4.4, changing the define tested to HAVE_TM_ZONE (in line with the autoconf test AC_STRUCT_TIMEZONE), but it didn't seem to produce the desired result (despite activating the new code in calls to gettmargs): Python 2.4.4 (#1, Feb 23 2007, 12:37:26) [GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2.1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import time >>> lt = time.localtime() >>> time.strftime("%Y-%m-%d %H:%M:%S %z", lt) '2007-02-24 00:45:23 +0000' The %z output should be '+0100'. According to the mktime man page, the time zone should be set "as though mktime() called tzset()", but a simple C test program revealed that either the tm_gmtoff field remains unset or is set to zero (which is not appropriate on my system). In other words, mktime does not miraculously restore the time zone information prior to the structure initialisation in gettmargs, at least on my system. ---------------------------------------------------------------------- Comment By: Doug Quale (quale) Date: 2003-08-21 18:57 Message: Logged In: YES user_id=812266 I have attached a little unittest with two tests showing the problem. I stole some code from Lib/test/test_time.py for the test that checks behavior in the US Eastern timezone. That test won't be run if tzset() isn't available, but this is OK since the only libc's that will show this problem are BSD-inspired and will have tzset(). ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-17 23:54 Message: Logged In: YES user_id=80475 Can you attach a unittest that fails before the patch and succeeds afterward? ---------------------------------------------------------------------- Comment By: Doug Quale (quale) Date: 2003-08-09 19:16 Message: Logged In: YES user_id=812266 Martin is right. The call to mktime() in my patch overwrites the tm_isdst field. This field is in the Python time tuple and is correctly set by the code immediately above. I put the call to mktime in the wrong place. Instead of going at the end of the routine it belongs immediately after the memset used to zero the structure. Sorry about this botch. I have attached a corrected patch. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-08-09 15:00 Message: Logged In: YES user_id=21627 The patch is wrong. It changes the behaviour of time.asctime(time.gmtime(time.time())) which it shouldn't do. The problem is real, though, and might need to be solved by exposing the tm_gmtoff field where available. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-07-12 01:52 Message: Logged In: YES user_id=80475 Martin, can you diagnose whether this should be closed, applied, or deferred? ---------------------------------------------------------------------- Comment By: Doug Quale (quale) Date: 2003-07-10 20:14 Message: Logged In: YES user_id=812266 The problem isn't in strftime. The problem is in gettmargs() in timemodule.c. Python assumes that broken down time tuples are in the local timezone. The gettmargs() routine in timemodule.c is bugged on GNU libc and possibly other BSD-inspired C libraries. gettmargs() is supposed to take a Python broken down time tuple and convert it to a C struct tm. The Python time tuple is assumed to be in the local time zone, so the struct tm should be in the local timezone also. In glibc, struct tm has timezone fields so each struct tm knows its own timezone. The gettmargs() routine never fills in these extra fields so it always creates a struct tm in GMT. The appropriate behavior would be to set tm_gmtoff to the local timezone offset. My patch fixes gettmargs() to create struct tm's in the local timezone for C libraries that have the tm_gmtoff field in struct tm. As to the docs issue, the Python docs say that other formats may be supported than the ones listed. In reality, strftime() is passed to the underlying C library so the format codes supported are whatever the C library supports. The doc statement "Additional directives may be supported on certain platforms, but only the ones listed here have a meaning standardized by ANSI C" is wrong, or at least not up to date. C99 specifies several strftime format codes that are not listed including %z. I think Tim Smith also mentions this in a Python list posting from earlier this year. In the Python time module, the docs say strftime('format') is the same as strftime('format', localtime()). This is simply broken right now on glibc as has been reported by more than one person: >>> strftime('%z') '-0500' >>> strftime('%z', localtime()) '+0000' This is wrong. Unsupported format specifiers do not have this effect, for example: >>> strftime('%L') '%L' >>> strftime('%L', localtime()) '%L' This behavior is correct. A final note on the patch: I should have looked closer at the timemodule.c source. It already uses the appropriate #ifdef in other places. Instead of #ifdef HAVE_MKTIME my patch should be conditionalized #ifdef HAVE_STRUCT_TM_TM_ZONE. It's kind of amusing to write up this long of a justification for what is essentially a 3 line patch. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-07-10 00:36 Message: Logged In: YES user_id=357491 So the problem is with the %z directive not reporting the proper timezone offset, correct? If you look at http://www.python.org/ dev/doc/devel/lib/module-time.html , though, you will notice that %T is not supported as a directive for strftime or strptime nor has it ever been supported. Same goes for %T although it works in your example. Since the docs do not say that these directives are supported I am closing this patch since it would require altering both strftime and stprtime to support %z on all platforms which this patch does not. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=762963&group_id=5470 From noreply at sourceforge.net Sat Feb 24 16:19:05 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 24 Feb 2007 07:19:05 -0800 Subject: [Patches] [ python-Patches-1667860 ] urllib2 raises an UnboundLocalError if "auth-int" is the qop Message-ID: Patches item #1667860, was opened at 2007-02-24 15:19 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667860&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Atul Varma (varmaa) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2 raises an UnboundLocalError if "auth-int" is the qop Initial Comment: If a proxy server is connected to that specifies the "auth-int" quality of protection (qop) code--or any qop code aside from "auth", actually--urllib2 raises an UnboundLocalError exception. While this patch doesn't implement auth-int, it does modify the behavior of urllib2 such that it raises a URLError with the reason "qop '%s' is not supported', where %s is the name of the qop code. Two unit tests that test urllib2's functionality with an in-process proxy can be found at: http://varmaa.googlepages.com/Urllib2ProxyTests.py I will try to attach this file to this patch ticket if I can. I am also interested in integrating this unit testing suite with the Python standard lib unit tests, and am willing to work with you if this is desirable. One of the unit tests in this suite succeeds with the standard version of urllib2.py, but the other fails unless this patch is applied. This patch (and the unit test suite) applies to Python 2.4 and 2.5. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667860&group_id=5470 From noreply at sourceforge.net Sat Feb 24 16:20:19 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 24 Feb 2007 07:20:19 -0800 Subject: [Patches] [ python-Patches-1667860 ] urllib2 raises an UnboundLocalError if "auth-int" is the qop Message-ID: Patches item #1667860, was opened at 2007-02-24 15:19 Message generated for change (Comment added) made by varmaa You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667860&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Atul Varma (varmaa) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2 raises an UnboundLocalError if "auth-int" is the qop Initial Comment: If a proxy server is connected to that specifies the "auth-int" quality of protection (qop) code--or any qop code aside from "auth", actually--urllib2 raises an UnboundLocalError exception. While this patch doesn't implement auth-int, it does modify the behavior of urllib2 such that it raises a URLError with the reason "qop '%s' is not supported', where %s is the name of the qop code. Two unit tests that test urllib2's functionality with an in-process proxy can be found at: http://varmaa.googlepages.com/Urllib2ProxyTests.py I will try to attach this file to this patch ticket if I can. I am also interested in integrating this unit testing suite with the Python standard lib unit tests, and am willing to work with you if this is desirable. One of the unit tests in this suite succeeds with the standard version of urllib2.py, but the other fails unless this patch is applied. This patch (and the unit test suite) applies to Python 2.4 and 2.5. ---------------------------------------------------------------------- >Comment By: Atul Varma (varmaa) Date: 2007-02-24 15:20 Message: Logged In: YES user_id=863202 Originator: YES File Added: Urllib2ProxyTests.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667860&group_id=5470 From noreply at sourceforge.net Sat Feb 24 17:19:34 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 24 Feb 2007 08:19:34 -0800 Subject: [Patches] [ python-Patches-1648268 ] Parameter list mismatches (portation problem) Message-ID: Patches item #1648268, was opened at 2007-01-30 14:15 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1648268&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: ked-tao (ked-tao) Assigned to: Nobody/Anonymous (nobody) Summary: Parameter list mismatches (portation problem) Initial Comment: On the system I'm porting to(*), an application will trap if the caller does not pass the exact parameter list that the callee requires. This is causing problems running Python. One common instance where this appears to be causing problems is where functions are registered as METH_NOARGS methods. For example, in Obejcts/dictobject.c, dict_popitem() is declared: static PyObject *dict_popitem(dictobject *mp); However, as it is declared in the method array as METH_NOARGS, it will be called by Objects/methodobject.c:PyCFunction_Call() as "(*meth)(self, NULL)" (i.e., an extra NULL parameter is passed for some reason). This will fail on my target system. I've no problem submitting a patch for this (dictobject.c is by no means the only place this is happening - it's just the first one encountered because it's used so much - though some places _do_ correctly declare a second, ignored parameter). However, I'd like to get agreement on the correct form it should be changed to before I put the effort in to produce a patch (it's going to be a fairly tedious process to identify and fix all these). In various modules, the functions are called internally as well as being registered as METH_NOARGS methods. Therefore, the change can either be: static PyObject *foo(PyObject *self) { ... } static PyObject *foo_noargs(PyObject *self, void *noargs_null) { return foo(self); } ... where 'foo' is called internally and 'foo_noargs' is registered as a METH_NOARGS method. or: static PyObject *foo(PyObject *self, void *noargs_null) { ... } ... and any internal calls in the module have to pass a second, NULL, argument in each call. The former favours internal module calls over METH_NOARGS calls, the latter penalises them. Which is preferred? Should this be raised on a different forum? Does anyone care? ;) Thanks, Kev. (*) Details on request. ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-24 08:19 Message: Logged In: YES user_id=33168 Originator: NO Kev, I would be interested to know the platform this was a problem on. I haven't looked at the patch much (a little of tested.diff), primarily Martin's msg on python-dev. I'm in favor of fixing this in concept. I tend to agree with Thomas that the parameter name in ALL_CAPS seems a bit annoying. I don't have a better suggestion and would rather see the patch applied with ALL_CAPS than not applied. I would also like to remove the casts to PyCFunction for all the functions that are stored in the various static structures. That will help ensure we don't copy bad examples and propagate the problem in the future. ---------------------------------------------------------------------- Comment By: ked-tao (ked-tao) Date: 2007-02-16 08:46 Message: Logged In: YES user_id=1703158 Originator: YES Hi, I am submitting two patches (both against the 2.5 release sources). One contains a set of changes which have subsequently been compiled by me and used to run lib/python/test/regrtest.py. If the format of the changes themselves is acceptable, then I believe this patch can be applied relatively confidently. I haven't paid too much attention to conditional compilation in those files, but there appears to be little in the areas I've touched. The second contains a set of changes to source files that are not being used at present on my system. Therefore, they _may_ not compile. I have visually checked that all functions whose signature I have changed are not called directly (across all source files) with the old signature and have also checked header file prototypes. However, that doesn't mean I didn't miss something, so this patch should be applied with a little more care. The nature of the fixes themselves are discussed below. ----------------------------------- ==== Fixes to common problems across several files: * Failure to declare second (always NULL) parameter on functions registered as METH_NOARGS methods. - These all now have a second parameter declared as "PyObject *NOARGS_NULL". - I have also changed ones that already declared the parameter as "void *ignored" etc, as I think the name makes it clear why it's there. If the upper-case name is bad style, feel free to change it to something else - as they are all now consistent, that should be a trivial process to change in the patch file before applying it. * PyGetSetDef 'getter' and 'setter' functions not declaring the final 'closure' parameter. - These all now have a final parameter declared as "void *closure". - I have also changed ones that already declared the parameter as "void *context" or "void *ignored" etc, for consistency. * The tp_clear type slot is defined as type 'inquiry' but the return value is ignored and in some instances, not returned at all. This is related to the following thread: http://mail.python.org/pipermail/python-dev/2003-April/034433.html frameobject.c and traceback.c were either missed when those changes were made, or the problems were re-introduced since. - I have changed the functions in those files to return zero. ==== Miscellaneous individual fixes: * Objects/fileobject.c:file_self() is registered both in the "tp_iter" slot and as a METH_NOARGS function. The "tp_iter" slot function is called with one parameter (the object) and the METH_NOARGS function is called with two parameters (the object, a NULL pointer). - Wrapper function file_self_noargs() created which accepts the additional "PyObject *NOARGS_NULL" parameter and just calls the file_self() function. - All other occurences of tp_iter visually checked and appear to be OK. * The datetimemodule.c problem with time_isoformat() being registered as METH_KEYWORDS instead of METH_NOARGS is also fixed here, though I believe that has already been dealt with. ----------------------------------- All in all, that was a pretty tedious process! Hopefully these changes can mostly make it in so I don't have to do it all over again one day ;) Regards, Kev. File Added: untested.diff ---------------------------------------------------------------------- Comment By: ked-tao (ked-tao) Date: 2007-02-16 08:42 Message: Logged In: YES user_id=1703158 Originator: YES File Added: tested.diff ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-06 11:49 Message: Logged In: YES user_id=21627 Originator: NO The current specification says that these should be PyCFunction pointers, see http://docs.python.org/api/common-structs.html My initial implementation of METH_NOARGS had it differently, and nobody ever bothered fixing them all when this was changed. Please do submit a patch to correct all such errors, both in code and documentation. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1648268&group_id=5470 From noreply at sourceforge.net Sun Feb 25 00:20:57 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 24 Feb 2007 15:20:57 -0800 Subject: [Patches] [ python-Patches-1668100 ] urllib2.urlopen() raises OSError instead of URLError Message-ID: Patches item #1668100, was opened at 2007-02-24 16:20 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668100&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Jerry Seutter (jseutter) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2.urlopen() raises OSError instead of URLError Initial Comment: See bug 1591774. The base problem is that when passing a file:// url into urllib2.urlopen() for a file that doesn't exist, the os.stat() call fails with an OSError exception which is not caught. The documentation for urllib2 states that urlopen() raises a URLError upon error. This bug can also be triggered when fetching a http:// url. A misconfigured webserver can cause urllib2.urlopen to use the FileHandler (see bug report). This patch catches OSError in the FileHandler class and raises it as a URLError. Unit test included. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668100&group_id=5470 From noreply at sourceforge.net Sun Feb 25 01:53:42 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 24 Feb 2007 16:53:42 -0800 Subject: [Patches] [ python-Patches-1668132 ] Fix for 767111, 'AttributeError thrown by urllib.open_http' Message-ID: Patches item #1668132, was opened at 2007-02-25 00:53 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668132&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Atul Varma (varmaa) Assigned to: Nobody/Anonymous (nobody) Summary: Fix for 767111, 'AttributeError thrown by urllib.open_http' Initial Comment: This is a fix for bug 767111: http://sourceforge.net/tracker/index.php?func=detail&aid=767111&group_id=5470&atid=105470 I am also attaching a unit test that reliably reproduces the bug using an in-process web server. I didn't want to modify httplib because it would mean altering its interface such that the HTTP.getfile() method would never return None, and that could break existing code. So instead I modified urllib at a point that it appears to assume that HTTP.getfile() will always return a file object. I hope this is a reasonable place to fix the problem, and I don't think it can break any existing code. I also ran the existing "test/test_urllib.py" and "test/test_urllibnet.py" test suites and they both work fine with this patch. Looking at the code, I also believe that a "parallel bug" exists with urllib.open_https(), and I fixed that as well (though I don't have a unit test to prove this). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668132&group_id=5470 From noreply at sourceforge.net Sun Feb 25 01:56:18 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sat, 24 Feb 2007 16:56:18 -0800 Subject: [Patches] [ python-Patches-1668132 ] Fix for 767111, 'AttributeError thrown by urllib.open_http' Message-ID: Patches item #1668132, was opened at 2007-02-25 00:53 Message generated for change (Comment added) made by varmaa You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668132&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Atul Varma (varmaa) Assigned to: Nobody/Anonymous (nobody) Summary: Fix for 767111, 'AttributeError thrown by urllib.open_http' Initial Comment: This is a fix for bug 767111: http://sourceforge.net/tracker/index.php?func=detail&aid=767111&group_id=5470&atid=105470 I am also attaching a unit test that reliably reproduces the bug using an in-process web server. I didn't want to modify httplib because it would mean altering its interface such that the HTTP.getfile() method would never return None, and that could break existing code. So instead I modified urllib at a point that it appears to assume that HTTP.getfile() will always return a file object. I hope this is a reasonable place to fix the problem, and I don't think it can break any existing code. I also ran the existing "test/test_urllib.py" and "test/test_urllibnet.py" test suites and they both work fine with this patch. Looking at the code, I also believe that a "parallel bug" exists with urllib.open_https(), and I fixed that as well (though I don't have a unit test to prove this). ---------------------------------------------------------------------- >Comment By: Atul Varma (varmaa) Date: 2007-02-25 00:56 Message: Logged In: YES user_id=863202 Originator: YES File Added: UrllibBug767111Tester.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668132&group_id=5470 From noreply at sourceforge.net Sun Feb 25 17:43:25 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 08:43:25 -0800 Subject: [Patches] [ python-Patches-723312 ] ability to pass a timeout to underlying socket Message-ID: Patches item #723312, was opened at 2003-04-17 20:03 Message generated for change (Comment added) made by jjlee You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=723312&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Matthew Russell (mattruss) Assigned to: Skip Montanaro (montanaro) Summary: ability to pass a timeout to underlying socket Initial Comment: this patch superceeds an earlier one i posted (#714592) - a bit *too* enthusiastic am afraid (sorry!) classes in modules such as httplib, ftpplib do not currently allow an easy way to take advantage of the new socket timeout feature in 2.3. This patch provides that abilty though one additonal class in socket.py ( socket.NetworkConnection ) and accompying test in test_socket.py (basic though the test is :-s ) As an extra benifit, the patch removes duplicate code, as each connect method in the main class of most modules (FTP, HTTPConnection, Telnet, POP3 etc) are copies of each other. The modules that use sockets are: * ftplib * httplib * telnetlib * poplib * urllib * imaplib * nntplib * xmlrpclib Of these I have only been able to easily refactor NetworkConnection into httplib, ftplib, telnetllib, poplib and smtplib. I did look to see if there were any unittests for theese modules in .Lib/test but found none (? - I appologise if there are some, i am new to the library tests) I did however check that the test() [like] methods at the bottom of each of the afore mentioned modules worked. thanks for your advice again Skip :o) Matt ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2007-02-25 16:43 Message: Logged In: YES user_id=261020 Originator: NO Looks good. I can't write tests right now, but just reading the code: I don't see a path in socket.NetworkConnection.connect() in which "if sock is None" is ever true. And what happens if creating a new socket object raises an exception (e.g. if too many file descriptors are open)? Then socket may be undefined, and will certainly not be None. I think it needs a sock = None before the loop, and also in the except: suite immediately before the continue statement. Some debug output in httplib has been lost (and maybe in the other modules). This is really very useful, so I think it should be preserved. In ftplib, FTP.makeport() should also use socket.NetworkConnection . PEP 8 nits: Use of thisStyleOfName, e.g. networkConnection Use of two-argument raise, e.g. raise socket.error, msg -- replace with raise socket.error(msg) ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 18:14 Message: Logged In: YES user_id=44345 Originator: NO Jeez, I can be an idiot sometimes... Working on two patches in the same sandbox. File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 18:11 Message: Logged In: YES user_id=44345 Originator: NO *argh* one minor tweak too many... File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2007-02-11 16:24 Message: Logged In: YES user_id=44345 Originator: NO New patch with minor tweaks. Added doc changes. File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2007-02-10 18:57 Message: Logged In: YES user_id=44345 Originator: NO Here's an updated patch. Tests pass. There's no documentation for socket.NetworkConnection or for the API changes to smtplib, httplib, telnetlib, poplib or ftplib. File Added: timeout.diff ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-04-14 17:09 Message: Logged In: YES user_id=21627 mattruss, are you interested in forward-porting it to 2.5? If that gets done before the beta release, please assign it to me for review. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2005-10-01 04:39 Message: Logged In: YES user_id=29957 Just a quick note - this patch might be accepted for 2.5, but it's not suitable for 2.4.x - once a major release is done, subsequent minor releases (2.3.x, 2.4.x) are bugfix-only. No new features are acceptable. For more on this, see http://www.python.org/peps/pep-0006.html ---------------------------------------------------------------------- Comment By: zeidler (zeidler) Date: 2005-09-28 15:22 Message: Logged In: YES user_id=1082576 perhaps to add a little emphasis: i'd really like to see that patch included into one of the next releases too. i was just going to submit my own, much much simpler version of adding a socket timeout in httplib.py, but matt's patch is a much better way to do this... we're using xml-rpc a lot over here, and the default timeouts are just way too long to keep up a decent user interaction in case of server problems. so, we'd really appreciate this becoming part of the distro! ;) thanks & good job matt, andi ---------------------------------------------------------------------- Comment By: Gr?goire Weber (gregweb) Date: 2005-01-12 13:52 Message: Logged In: YES user_id=812601 Just lobbying: It would be nice to have that on board in futire python versions. Would this be possible in python 2.4.1? I need timeouts for xmlrpclib and subclassed xmlrpclib.Transport for my needs. But nevertheless it would be nice to have that in python 2.4.1. Thanks! Gregoire ---------------------------------------------------------------------- Comment By: Matthew Russell (mattruss) Date: 2003-04-21 00:20 Message: Logged In: YES user_id=737261 Errata: i forgot to mention urlllib2 as modules affected. the line above starting "The modules that use sockets are: " should really read: "classes in the modules stated below would benifit from the ability pass timeouts:" ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=723312&group_id=5470 From noreply at sourceforge.net Sun Feb 25 18:01:00 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 09:01:00 -0800 Subject: [Patches] [ python-Patches-1668100 ] urllib2.urlopen() raises OSError instead of URLError Message-ID: Patches item #1668100, was opened at 2007-02-24 23:20 Message generated for change (Comment added) made by jjlee You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668100&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Jerry Seutter (jseutter) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2.urlopen() raises OSError instead of URLError Initial Comment: See bug 1591774. The base problem is that when passing a file:// url into urllib2.urlopen() for a file that doesn't exist, the os.stat() call fails with an OSError exception which is not caught. The documentation for urllib2 states that urlopen() raises a URLError upon error. This bug can also be triggered when fetching a http:// url. A misconfigured webserver can cause urllib2.urlopen to use the FileHandler (see bug report). This patch catches OSError in the FileHandler class and raises it as a URLError. Unit test included. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2007-02-25 17:01 Message: Logged In: YES user_id=261020 Originator: NO I recall that when I wrote the urllib2 tests there were a fair few cases similar to this -- indeed, this particular one is already marked in the tests with an "XXXX" in test_file(). What do other people think about these -- should they be fixed, or the docs changed? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668100&group_id=5470 From noreply at sourceforge.net Sun Feb 25 18:29:23 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 09:29:23 -0800 Subject: [Patches] [ python-Patches-1657613 ] documentation for element interface Message-ID: Patches item #1657613, was opened at 2007-02-11 19:15 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657613&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Tim Mitchell (tim-mitchell) Assigned to: Fredrik Lundh (effbot) Summary: documentation for element interface Initial Comment: I have added a subsection to 8.13 xml.etree.ElementTree documenting the _ElementInterface class which objects returned by the Element and SubElement factory functions emulate. I do not have latex installed on my system so I have not been able to check for compiling errors - sorry. (I don't intend to become a regular contributor to go through the bother of setting it up.) ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2007-02-25 12:29 Message: Logged In: YES user_id=3066 Originator: NO The LaTeX formats without error, but the style appears a bit heavy. In particular, the use of the {datadescni} environment to present return value information is inappropriate: that should be presented a normal sentence in the method description. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 07:15 Message: Logged In: YES user_id=21627 Originator: NO Fredrik, can you please take a look? If not, assign it back to Fred. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657613&group_id=5470 From noreply at sourceforge.net Sun Feb 25 18:37:24 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 09:37:24 -0800 Subject: [Patches] [ python-Patches-1667860 ] urllib2 raises an UnboundLocalError if "auth-int" is the qop Message-ID: Patches item #1667860, was opened at 2007-02-24 15:19 Message generated for change (Comment added) made by jjlee You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667860&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Atul Varma (varmaa) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2 raises an UnboundLocalError if "auth-int" is the qop Initial Comment: If a proxy server is connected to that specifies the "auth-int" quality of protection (qop) code--or any qop code aside from "auth", actually--urllib2 raises an UnboundLocalError exception. While this patch doesn't implement auth-int, it does modify the behavior of urllib2 such that it raises a URLError with the reason "qop '%s' is not supported', where %s is the name of the qop code. Two unit tests that test urllib2's functionality with an in-process proxy can be found at: http://varmaa.googlepages.com/Urllib2ProxyTests.py I will try to attach this file to this patch ticket if I can. I am also interested in integrating this unit testing suite with the Python standard lib unit tests, and am willing to work with you if this is desirable. One of the unit tests in this suite succeeds with the standard version of urllib2.py, but the other fails unless this patch is applied. This patch (and the unit test suite) applies to Python 2.4 and 2.5. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2007-02-25 17:37 Message: Logged In: YES user_id=261020 Originator: NO This looks useful. It would be nice to test failure cases too (e.g. wrong password, .add_password() not called, ...). I think if this goes into a new file (which might be reasonable), it should be named something like test_urllib2_localnet.py , to allow for adding more loopback tests, which will not always involve proxy handling. I suppose it should NOT depend on the network resource flag (Lib/test/regrtest.py -u network) since it's only localhost. Style (see PEP 8): Don't put whitespace inside brackets -- e.g. bad: fn( arg ) good: fn(arg) Use this_kind_of_name or thiskind if short, not thisKind. Imports and classes etc. don't need comments to label them as such. Probably is useful to label the BaseHTTPServer code though (maybe '# Local proxy test infrastructure'). See e.g. end of Lib/test/test_urllib2net.py for appropriate stdlib test boilerplate. Your patch has inconsistent line endings, BTW. ---------------------------------------------------------------------- Comment By: Atul Varma (varmaa) Date: 2007-02-24 15:20 Message: Logged In: YES user_id=863202 Originator: YES File Added: Urllib2ProxyTests.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667860&group_id=5470 From noreply at sourceforge.net Sun Feb 25 18:46:28 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 09:46:28 -0800 Subject: [Patches] [ python-Patches-1616125 ] Cached globals+builtins lookup optimization Message-ID: Patches item #1616125, was opened at 2006-12-15 00:44 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1616125&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Performance Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Andrea Griffini (ag6502) Assigned to: Nobody/Anonymous (nobody) Summary: Cached globals+builtins lookup optimization Initial Comment: This patch implements a speed optimization by introducing a timestamp on dictionaries and by caching lookups of constants so that lookup doesn't need to be repeated if the dictionary didn't change. Currently the patch implements the cached lookup only for the LOAD_GLOBAL opcode and stores the cache as two extra member in the code object. I'm going to investigate if LOAD_ATTR in the special case of a module is worth caching too. A cache entry for LOAD_GLOBAL is defined by two timestamps (Py_ssize_t) and a *borrowed* pointer to a PyObject. The use of a borrowed pointer is safe because the value is used only if the dictionaries didn't change, so the result of lookup must still be alive because the reference in the dictionary that returned the cached value it is still in place. The assumptions are: - a LOAD_GLOBAL opcode always looks for the same symbol (i.e. co_names never changes) - the dictionary being searched (globals and builtins) are mostly constant On my system I got a speedup of 8%-10% on a real program (a tcp/ip server that handles an in-memory database of python objects) on the checkpoint load + rollforward steps (a few millions LOAD_GLOBAL operations in total) and the varying percentage depends on which compiler options are used to build python. On another little test program that calls a function def f(x): return math.sin(x) + math.cos(x) + abs(x) one million times the speedup with the default makefile is about 7%. To enable the patch the makefile should define the CACHED_LOOKUPS symbol. The change to dictobject also exports the timestamp, at the python level; other changes are instead invisible at the python level. I'm not sure at all exporting dict.timestamp() is a good idea however... Note that after applying the patch you may find an error in userdict if you simply use "make" (there's probably some dependency problem somewhere). After make clean + make the test suite runs fine on my system. I'm interested in hearing if also other platforms do actually see a real performance improvement. Andrea ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2007-02-25 17:46 Message: Logged In: YES user_id=4771 Originator: NO In the patch, the initial and the reset value of dict_timestamp_value are 0xFFFFFFFFU - 10000. I guess that it's a value you used while debugging the clearing code, and that it should now be 1 instead? ---------------------------------------------------------------------- Comment By: Andrea Griffini (ag6502) Date: 2006-12-30 12:47 Message: Logged In: YES user_id=1668946 Originator: YES I added the two missing parts and the patch should be now functionally complete; now there co_names reordering pass during compilation so that all names used in LOAD_GLOBAL/LOAD_ATTR are moved at the beginning of the tuple, this allows the lookup cache to be allocated only for the needed size. The reordering is done *before* actually assembling basic blocks to avoid to deal with the change of opcode size after reordering. Also now the timestamp is reset when rolling over to 0 and a sweep is done on all dictionaries and all code objects to clear cached lookups; I didn't implemented it in the gc because: 1) I'm not sure if all code objects and dictionaries are listed (how do I find code objects ?), 2) to do the traversing the code must be in gcmodule, or a richer interface must be exported, 3) what if garbage collecting is disabled ? The extra cost for dictionary creation is normally (i.e. when a dictionary is recovered from the free pool) just setting the timestamp to 1, the extra cost for dictionary update is higher but around the "noise" of timeit.Timer on my system. The memory cost is three more 32-bit words for every dict and 4 + n*3 more words for every code object where n is the number of elements of co_names that are used in LOAD_GLOBAL/LOAD_ATTR opcodes. The patch as it stands don't need to change marshalling of compiled code (the loaded code objects will simply be "unoptimized" so n will be 1+max(oparg) for all arguments of LOAD_GLOBAL/LOAD_ATTR - wasting some cache slots). The patch itself doesn't do anything unless the symbol CACHED_LOOKUPS is defined (this will cache LOAD_GLOBALs) or both CACHED_LOOKUPS and CACHED_MODULE_LOOKUPS are defined (this will cache both LOAD_GLOBALs and LOAD_ATTRs when the searched object is a module). This allows to have this optimization as an optional build option (does this make sense ?). File Added: cached_lookups_12.patch ---------------------------------------------------------------------- Comment By: Andrea Griffini (ag6502) Date: 2006-12-23 22:52 Message: Logged In: YES user_id=1668946 Originator: YES I was already changing the code in the direction you're pointing out. I attached the current version (still not complete!) of the patch; more specifically: - Cached lookups are now stored in an array of structures instead than in two separate arrays - Macros are used for handling of cache entries - The timestamp is now unsigned and there is optional support for 64-bit timestamps - Support for stalling timestamping at 32 bit wrap point - Removed the LOAD_FAST_HACK (was just an experiment with oprofile; was left there by mistake) - Fixed memory allocation failures handling in code object creation - Removed exporting of timestamp value at the python level - Used as cache size the largest LOAD_ATTR/LOAD_GLOBAL entry found Still missing are the sorting of co_names to pack cache slots and the cache reset at gc time if the dictionary timestamp is stalled. The cached lookup can be used at two levels: -DCACHED_LOOKUPS enables caching of LOAD_GLOBAL results -DCACHED_MODULE_LOOKUPS (in addition to -DCACHED_LOOKUPS) also caches the result of a lookup in a module. Speed results are sometimes very interesting and sometimes strange; I found measurable differences just moving around statements between logically equivalent places after checking what gcc was doing with register allocation (probably those places were indeed not equivalent if taking in account aliasing that isn't going to happen but that a c compiler must assume as possible). I have no idea if the speedups I've measured are better or worse on other processors/architectures. File Added: cached_lookups_10.patch ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-23 03:33 Message: Logged In: YES user_id=33168 Originator: NO I only reviewed the code in the patch. I have not thought conceptually about this, whether it's a good idea, how to break it, etc. I also did not look to verify all the necessary dict methods were updated to add TOUCHED. Have you profiled the slow down to dicts in normal cases? Things like: d = {} d[0] = 0 etc? All dict operations are going to be a tiny bit slower due to an additional assignment. It's probably not measurable, but you should still verify this. What is LOAD_FAST_HACK in ceval.c? It doesn't seem necessary for this patch. Please revert any whitespace/formatting changes that aren't part of the patch (there are a couple in dictobject.c and codeobject.c). It seems like a little structure would be better than parallel arrays. This would reduce some calculations and reduce the number of mallocs. Also there would only be a single assignment in the eval loop and the names would probably wind up being more clear. Would it be better to use PyObject_Malloc rather than PyMem_Malloc? It should be faster for sizes <= 256. (Despite the name, it doesn't have to handle PyObjects.) Why not use a size_t instead of Py_ssize_t for the timestamp? This is conceptually unsigned, no? Use the macro version PyTuple_GET_SIZE(names); in codeobject.c. It will be a little faster. If the cache allocs fail when allocating a code object, ceval will crash due to not checking for NULL. As for the timestamp, there should probably be a macro to access the field for a dict. This macro should be used in ceval.c/codeobject.c for getting/setting it. When returning the timestamp in dictobject.c you shouldn't cast it to a long. Use the appropriate method such as PyInt_FromSsize_t. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-16 22:57 Message: Logged In: YES user_id=21627 Originator: NO I think restricting yourself to 32-bit counters is fine (it's also an issue of memory overhead as you have one of these per access to a global, and per dictionary); you have to deal with the overflow, though. I suggest to use an unsigned value, e.g. size_t. Overflowing should be dealt with through visiting all objects, as you say. It could either happen right when it overflows, or it could get integrated into GC, and only reset when the GC runs. You'd then need to account for those periods when it already has overflowed, but GC hasn't run yet; in these periods, counting should be "on hold". ---------------------------------------------------------------------- Comment By: Andrea Griffini (ag6502) Date: 2006-12-16 17:13 Message: Logged In: YES user_id=1668946 Originator: YES I am experimenting with long long timestamps (based on "unsigned long long" if available or explicitly on two "unsigned long" otherwise). An alternative to slowing down lookups by using 64 bits on 32-bit computers is just using a single 32-bit counter and trigger a "cache flush" when the timestamp rolls over by visiting all live dictionaries and code objects. This would be IMO a *very* infrequent operation (for example on my P4/2.4GHz just doing nothing else but dict updates getting to 2**32 updates would require 14 minutes). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-16 11:01 Message: Logged In: YES user_id=21627 Originator: NO How does that deal with timestamp wraparound? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1616125&group_id=5470 From noreply at sourceforge.net Sun Feb 25 18:47:29 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 09:47:29 -0800 Subject: [Patches] [ python-Patches-1668482 ] don't use '-' and '_' in mkstemp Message-ID: Patches item #1668482, was opened at 2007-02-25 18:47 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668482&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Arvin Schnell (arvins) Assigned to: Nobody/Anonymous (nobody) Summary: don't use '-' and '_' in mkstemp Initial Comment: I suggest to not use '-' and '_' in the filename generated by mkstemp. Without a prefix you end up with filename that start with '-' witch are clumsy to delete in the shell. glibc's mkstemp also uses only alphanumerical characters. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668482&group_id=5470 From noreply at sourceforge.net Sun Feb 25 18:58:51 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 09:58:51 -0800 Subject: [Patches] [ python-Patches-1657613 ] documentation for element interface Message-ID: Patches item #1657613, was opened at 2007-02-11 19:15 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657613&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Documentation Group: None >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Tim Mitchell (tim-mitchell) Assigned to: Fredrik Lundh (effbot) Summary: documentation for element interface Initial Comment: I have added a subsection to 8.13 xml.etree.ElementTree documenting the _ElementInterface class which objects returned by the Element and SubElement factory functions emulate. I do not have latex installed on my system so I have not been able to check for compiling errors - sorry. (I don't intend to become a regular contributor to go through the bother of setting it up.) ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2007-02-25 12:58 Message: Logged In: YES user_id=3066 Originator: NO Applied to the trunk in Subversion revision 53910. The bad use of {datadescni} was apparently based on precedents elsewhere in the etree documentation; this has been fixed throughout. ---------------------------------------------------------------------- Comment By: Fred L. Drake, Jr. (fdrake) Date: 2007-02-25 12:29 Message: Logged In: YES user_id=3066 Originator: NO The LaTeX formats without error, but the style appears a bit heavy. In particular, the use of the {datadescni} environment to present return value information is inappropriate: that should be presented a normal sentence in the method description. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 07:15 Message: Logged In: YES user_id=21627 Originator: NO Fredrik, can you please take a look? If not, assign it back to Fred. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1657613&group_id=5470 From noreply at sourceforge.net Sun Feb 25 20:50:53 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 11:50:53 -0800 Subject: [Patches] [ python-Patches-1624059 ] fast subclasses of builtin types Message-ID: Patches item #1624059, was opened at 2006-12-28 22:01 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1624059&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 2.6 >Status: Closed >Resolution: Fixed Priority: 5 Private: No Submitted By: Neal Norwitz (nnorwitz) Assigned to: Guido van Rossum (gvanrossum) Summary: fast subclasses of builtin types Initial Comment: This is similar to a patch posted on python-dev a few months ago (or more). I modified it to also handle subclassing exceptions which should speed up exception handling a bit. (This was proposed by Guido based on the original patch.) I also dropped an extra bit that was going to indicate if it was a builtin type or a subclass of a builtin type. ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-25 11:50 Message: Logged In: YES user_id=33168 Originator: YES Committed rev 53911. Hopefully the checkin comment explains most of what's going on. I simplified the patch as much as possible. I like to start with less code. If we can improve the speed, that can be optimized later. I didn't measure the little variaions. I had measured that it made a real diff in speed for using an int subclass a long time ago. This should help a fair amount for exceptions too. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-01-06 06:54 Message: Logged In: YES user_id=21627 Originator: NO File Added: a.c ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-01-06 06:24 Message: Logged In: YES user_id=21627 Originator: NO I made a couple of assembler experiments (see attached a.c), with gcc 4.1 on x86. A "bit mask enumeration" test (f) compiles into four instructions: movl 8(%eax), %eax andl $-268435456, %eax cmpl $1879048192, %eax je .L18 (fall-through being the else case) A single bit test of a flag (g) compiles to two instructions: testl $-1073741824, 8(%eax) je .L9 (fall-through being the if case) Adding an identity test (comparison with the address of a global), followed by a bit mask test (h), compiles into six instructions: cmpl $int_type, %eax je .L2 movl 8(%eax), %eax andl $-268435456, %eax cmpl $1879048192, %eax je .L2 (fall-through being the else case) In the common case, only two of these instructions are executed. So all-in-all, I would agree with Guido that adding bit flags is more efficient. However, existing bits cannot be recycled: in existing binary extension modules, these flags are set, so if the modules don't get recompiled, the type check would believe that the types are subtypes. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-01-03 19:59 Message: Logged In: YES user_id=6380 Originator: NO This looks fine, but I have some questions about alternative implementations: - Why does the typical PyFoo_Check() macro first call PyFoo_CheckExact() before calling the fast bit checking macro? Did you measure that this is in fact faster? True, it means always a pointer deref, so maybe it is -- but OTOH it is more instructions. - Why not have a separate bit for each type? Then you could make the fast macro test for (flags & mask) != 0 instead of testing for (flag & mask) == value. It would use up all the remaining bits, but I suspect there are some unused (or reusable) bits in lower positions: 1L<<2 is unused (was GC), and 1L<<11 also seems unused. And bits 18 through 23! And I'm guessing that INPLACEOPS (1L<<3) isn't all that interesting any more they were introduced in 2.0... So it really looks like you have plenty of bits. Of course I don't know if it matters; would be worth it perhaps to look at the machine code. - Oops, it looks like your comment is off. You claim to be using bits 24-27, leaving 28-31 free, but in fact you're using bits 28-31! BTW You're inroducing quite a few lines over 80 chars. Perhaps cut back a bit? ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-28 22:04 Message: Logged In: YES user_id=33168 Originator: YES I forgot to mention this patch works by using unused bits in tp_flags. This saves a function call when checking for a subclass of a builtin type. There's one funky thing about this patch, the change to Objects/exceptions.c. I didn't investigate why this was necessary, or more likely I did why when I added it and forgot. I know that without adding BASE_EXC_SUBCLASS to tp_flags, test_exceptions fails. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1624059&group_id=5470 From noreply at sourceforge.net Sun Feb 25 23:03:15 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 14:03:15 -0800 Subject: [Patches] [ python-Patches-1668482 ] don't use '-' and '_' in mkstemp Message-ID: Patches item #1668482, was opened at 2007-02-25 18:47 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668482&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Arvin Schnell (arvins) Assigned to: Nobody/Anonymous (nobody) Summary: don't use '-' and '_' in mkstemp Initial Comment: I suggest to not use '-' and '_' in the filename generated by mkstemp. Without a prefix you end up with filename that start with '-' witch are clumsy to delete in the shell. glibc's mkstemp also uses only alphanumerical characters. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2007-02-25 23:03 Message: Logged In: YES user_id=21627 Originator: NO I can see what's wrong with -, but what is the problem with _? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668482&group_id=5470 From noreply at sourceforge.net Sun Feb 25 23:44:49 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 14:44:49 -0800 Subject: [Patches] [ python-Patches-1668482 ] don't use '-' and '_' in mkstemp Message-ID: Patches item #1668482, was opened at 2007-02-25 18:47 Message generated for change (Comment added) made by arvins You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668482&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Arvin Schnell (arvins) Assigned to: Nobody/Anonymous (nobody) Summary: don't use '-' and '_' in mkstemp Initial Comment: I suggest to not use '-' and '_' in the filename generated by mkstemp. Without a prefix you end up with filename that start with '-' witch are clumsy to delete in the shell. glibc's mkstemp also uses only alphanumerical characters. ---------------------------------------------------------------------- >Comment By: Arvin Schnell (arvins) Date: 2007-02-25 23:44 Message: Logged In: YES user_id=698939 Originator: YES Nothing is wrong with '_'. I just wanted to make it consistent with glibc. Both 62**6 and 63**6 are huge numbers so it doesn't really matter. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-25 23:03 Message: Logged In: YES user_id=21627 Originator: NO I can see what's wrong with -, but what is the problem with _? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668482&group_id=5470 From noreply at sourceforge.net Mon Feb 26 02:33:15 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 17:33:15 -0800 Subject: [Patches] [ python-Patches-1667860 ] urllib2 raises an UnboundLocalError if "auth-int" is the qop Message-ID: Patches item #1667860, was opened at 2007-02-24 15:19 Message generated for change (Comment added) made by varmaa You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667860&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Atul Varma (varmaa) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2 raises an UnboundLocalError if "auth-int" is the qop Initial Comment: If a proxy server is connected to that specifies the "auth-int" quality of protection (qop) code--or any qop code aside from "auth", actually--urllib2 raises an UnboundLocalError exception. While this patch doesn't implement auth-int, it does modify the behavior of urllib2 such that it raises a URLError with the reason "qop '%s' is not supported', where %s is the name of the qop code. Two unit tests that test urllib2's functionality with an in-process proxy can be found at: http://varmaa.googlepages.com/Urllib2ProxyTests.py I will try to attach this file to this patch ticket if I can. I am also interested in integrating this unit testing suite with the Python standard lib unit tests, and am willing to work with you if this is desirable. One of the unit tests in this suite succeeds with the standard version of urllib2.py, but the other fails unless this patch is applied. This patch (and the unit test suite) applies to Python 2.4 and 2.5. ---------------------------------------------------------------------- >Comment By: Atul Varma (varmaa) Date: 2007-02-26 01:33 Message: Logged In: YES user_id=863202 Originator: YES File Added: urllib2_patch.diff ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2007-02-25 17:37 Message: Logged In: YES user_id=261020 Originator: NO This looks useful. It would be nice to test failure cases too (e.g. wrong password, .add_password() not called, ...). I think if this goes into a new file (which might be reasonable), it should be named something like test_urllib2_localnet.py , to allow for adding more loopback tests, which will not always involve proxy handling. I suppose it should NOT depend on the network resource flag (Lib/test/regrtest.py -u network) since it's only localhost. Style (see PEP 8): Don't put whitespace inside brackets -- e.g. bad: fn( arg ) good: fn(arg) Use this_kind_of_name or thiskind if short, not thisKind. Imports and classes etc. don't need comments to label them as such. Probably is useful to label the BaseHTTPServer code though (maybe '# Local proxy test infrastructure'). See e.g. end of Lib/test/test_urllib2net.py for appropriate stdlib test boilerplate. Your patch has inconsistent line endings, BTW. ---------------------------------------------------------------------- Comment By: Atul Varma (varmaa) Date: 2007-02-24 15:20 Message: Logged In: YES user_id=863202 Originator: YES File Added: Urllib2ProxyTests.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667860&group_id=5470 From noreply at sourceforge.net Mon Feb 26 02:38:02 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 17:38:02 -0800 Subject: [Patches] [ python-Patches-1667860 ] urllib2 raises an UnboundLocalError if "auth-int" is the qop Message-ID: Patches item #1667860, was opened at 2007-02-24 15:19 Message generated for change (Comment added) made by varmaa You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667860&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Atul Varma (varmaa) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2 raises an UnboundLocalError if "auth-int" is the qop Initial Comment: If a proxy server is connected to that specifies the "auth-int" quality of protection (qop) code--or any qop code aside from "auth", actually--urllib2 raises an UnboundLocalError exception. While this patch doesn't implement auth-int, it does modify the behavior of urllib2 such that it raises a URLError with the reason "qop '%s' is not supported', where %s is the name of the qop code. Two unit tests that test urllib2's functionality with an in-process proxy can be found at: http://varmaa.googlepages.com/Urllib2ProxyTests.py I will try to attach this file to this patch ticket if I can. I am also interested in integrating this unit testing suite with the Python standard lib unit tests, and am willing to work with you if this is desirable. One of the unit tests in this suite succeeds with the standard version of urllib2.py, but the other fails unless this patch is applied. This patch (and the unit test suite) applies to Python 2.4 and 2.5. ---------------------------------------------------------------------- >Comment By: Atul Varma (varmaa) Date: 2007-02-26 01:38 Message: Logged In: YES user_id=863202 Originator: YES File Added: test_urllib2_localnet.py ---------------------------------------------------------------------- Comment By: Atul Varma (varmaa) Date: 2007-02-26 01:33 Message: Logged In: YES user_id=863202 Originator: YES File Added: urllib2_patch.diff ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2007-02-25 17:37 Message: Logged In: YES user_id=261020 Originator: NO This looks useful. It would be nice to test failure cases too (e.g. wrong password, .add_password() not called, ...). I think if this goes into a new file (which might be reasonable), it should be named something like test_urllib2_localnet.py , to allow for adding more loopback tests, which will not always involve proxy handling. I suppose it should NOT depend on the network resource flag (Lib/test/regrtest.py -u network) since it's only localhost. Style (see PEP 8): Don't put whitespace inside brackets -- e.g. bad: fn( arg ) good: fn(arg) Use this_kind_of_name or thiskind if short, not thisKind. Imports and classes etc. don't need comments to label them as such. Probably is useful to label the BaseHTTPServer code though (maybe '# Local proxy test infrastructure'). See e.g. end of Lib/test/test_urllib2net.py for appropriate stdlib test boilerplate. Your patch has inconsistent line endings, BTW. ---------------------------------------------------------------------- Comment By: Atul Varma (varmaa) Date: 2007-02-24 15:20 Message: Logged In: YES user_id=863202 Originator: YES File Added: Urllib2ProxyTests.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667860&group_id=5470 From noreply at sourceforge.net Mon Feb 26 02:42:17 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 17:42:17 -0800 Subject: [Patches] [ python-Patches-1667860 ] urllib2 raises an UnboundLocalError if "auth-int" is the qop Message-ID: Patches item #1667860, was opened at 2007-02-24 15:19 Message generated for change (Comment added) made by varmaa You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667860&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Atul Varma (varmaa) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2 raises an UnboundLocalError if "auth-int" is the qop Initial Comment: If a proxy server is connected to that specifies the "auth-int" quality of protection (qop) code--or any qop code aside from "auth", actually--urllib2 raises an UnboundLocalError exception. While this patch doesn't implement auth-int, it does modify the behavior of urllib2 such that it raises a URLError with the reason "qop '%s' is not supported', where %s is the name of the qop code. Two unit tests that test urllib2's functionality with an in-process proxy can be found at: http://varmaa.googlepages.com/Urllib2ProxyTests.py I will try to attach this file to this patch ticket if I can. I am also interested in integrating this unit testing suite with the Python standard lib unit tests, and am willing to work with you if this is desirable. One of the unit tests in this suite succeeds with the standard version of urllib2.py, but the other fails unless this patch is applied. This patch (and the unit test suite) applies to Python 2.4 and 2.5. ---------------------------------------------------------------------- >Comment By: Atul Varma (varmaa) Date: 2007-02-26 01:42 Message: Logged In: YES user_id=863202 Originator: YES Thanks for the feedback, John--I've applied your suggestions to the testing suite (including renaming the file name) and the patch file. The only thing I haven't done yet is add tests for failure cases, but I wanted to submit the "refactored" version of the test suite before doing that so I know I'm on the right track. ---------------------------------------------------------------------- Comment By: Atul Varma (varmaa) Date: 2007-02-26 01:38 Message: Logged In: YES user_id=863202 Originator: YES File Added: test_urllib2_localnet.py ---------------------------------------------------------------------- Comment By: Atul Varma (varmaa) Date: 2007-02-26 01:33 Message: Logged In: YES user_id=863202 Originator: YES File Added: urllib2_patch.diff ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2007-02-25 17:37 Message: Logged In: YES user_id=261020 Originator: NO This looks useful. It would be nice to test failure cases too (e.g. wrong password, .add_password() not called, ...). I think if this goes into a new file (which might be reasonable), it should be named something like test_urllib2_localnet.py , to allow for adding more loopback tests, which will not always involve proxy handling. I suppose it should NOT depend on the network resource flag (Lib/test/regrtest.py -u network) since it's only localhost. Style (see PEP 8): Don't put whitespace inside brackets -- e.g. bad: fn( arg ) good: fn(arg) Use this_kind_of_name or thiskind if short, not thisKind. Imports and classes etc. don't need comments to label them as such. Probably is useful to label the BaseHTTPServer code though (maybe '# Local proxy test infrastructure'). See e.g. end of Lib/test/test_urllib2net.py for appropriate stdlib test boilerplate. Your patch has inconsistent line endings, BTW. ---------------------------------------------------------------------- Comment By: Atul Varma (varmaa) Date: 2007-02-24 15:20 Message: Logged In: YES user_id=863202 Originator: YES File Added: Urllib2ProxyTests.py ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1667860&group_id=5470 From noreply at sourceforge.net Mon Feb 26 05:12:14 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 20:12:14 -0800 Subject: [Patches] [ python-Patches-1668100 ] urllib2.urlopen() raises OSError instead of URLError Message-ID: Patches item #1668100, was opened at 2007-02-24 16:20 Message generated for change (Comment added) made by jseutter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668100&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Modules Group: Python 2.5 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Jerry Seutter (jseutter) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2.urlopen() raises OSError instead of URLError Initial Comment: See bug 1591774. The base problem is that when passing a file:// url into urllib2.urlopen() for a file that doesn't exist, the os.stat() call fails with an OSError exception which is not caught. The documentation for urllib2 states that urlopen() raises a URLError upon error. This bug can also be triggered when fetching a http:// url. A misconfigured webserver can cause urllib2.urlopen to use the FileHandler (see bug report). This patch catches OSError in the FileHandler class and raises it as a URLError. Unit test included. ---------------------------------------------------------------------- >Comment By: Jerry Seutter (jseutter) Date: 2007-02-25 21:12 Message: Logged In: YES user_id=1727609 Originator: YES Sorry, I looked at that commented out section and didn't immediately see what it was for, so ignored it. I'm not sure what the correct behavior would be in this case. Getting a OSError from a http:// url is unsettling. I guess that if I were using the library, I would only want to have to catch one type of exception. Having to catch both URLError and OSError is inconvenient. If I were looking for a file on disk I would use file(), so the only time someone probably uses the file:// url syntax is when the end user is entering the URL. The end user wouldn't distinguish between the two exception types in this case. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2007-02-25 10:01 Message: Logged In: YES user_id=261020 Originator: NO I recall that when I wrote the urllib2 tests there were a fair few cases similar to this -- indeed, this particular one is already marked in the tests with an "XXXX" in test_file(). What do other people think about these -- should they be fixed, or the docs changed? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1668100&group_id=5470 From noreply at sourceforge.net Mon Feb 26 08:08:53 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Sun, 25 Feb 2007 23:08:53 -0800 Subject: [Patches] [ python-Patches-1616125 ] Cached globals+builtins lookup optimization Message-ID: Patches item #1616125, was opened at 2006-12-15 01:44 Message generated for change (Comment added) made by ag6502 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1616125&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Performance Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Andrea Griffini (ag6502) Assigned to: Nobody/Anonymous (nobody) Summary: Cached globals+builtins lookup optimization Initial Comment: This patch implements a speed optimization by introducing a timestamp on dictionaries and by caching lookups of constants so that lookup doesn't need to be repeated if the dictionary didn't change. Currently the patch implements the cached lookup only for the LOAD_GLOBAL opcode and stores the cache as two extra member in the code object. I'm going to investigate if LOAD_ATTR in the special case of a module is worth caching too. A cache entry for LOAD_GLOBAL is defined by two timestamps (Py_ssize_t) and a *borrowed* pointer to a PyObject. The use of a borrowed pointer is safe because the value is used only if the dictionaries didn't change, so the result of lookup must still be alive because the reference in the dictionary that returned the cached value it is still in place. The assumptions are: - a LOAD_GLOBAL opcode always looks for the same symbol (i.e. co_names never changes) - the dictionary being searched (globals and builtins) are mostly constant On my system I got a speedup of 8%-10% on a real program (a tcp/ip server that handles an in-memory database of python objects) on the checkpoint load + rollforward steps (a few millions LOAD_GLOBAL operations in total) and the varying percentage depends on which compiler options are used to build python. On another little test program that calls a function def f(x): return math.sin(x) + math.cos(x) + abs(x) one million times the speedup with the default makefile is about 7%. To enable the patch the makefile should define the CACHED_LOOKUPS symbol. The change to dictobject also exports the timestamp, at the python level; other changes are instead invisible at the python level. I'm not sure at all exporting dict.timestamp() is a good idea however... Note that after applying the patch you may find an error in userdict if you simply use "make" (there's probably some dependency problem somewhere). After make clean + make the test suite runs fine on my system. I'm interested in hearing if also other platforms do actually see a real performance improvement. Andrea ---------------------------------------------------------------------- >Comment By: Andrea Griffini (ag6502) Date: 2007-02-26 08:08 Message: Logged In: YES user_id=1668946 Originator: YES There are indeed two symbols used for debugging; one is CACHED_LOOKUP_STATS (Python/ceval.c) that outputs the hit/miss ratios for the cache when python exits and the other is LOOKUP_DEBUG_TIMESTAMP_LIMIT (Objects/dictobject.c) that forces the global counter to wrap around every 10000 dict changes (instead of every 2**32 dict changes). Actually on this patch I'm getting confusing results on my machine (for example getting better results with statistics ON, that to me just make no sense at all). I got no results from anyone else so I can't say if this patch is actually speeding up or slowing down python. On my machine (trunk python) pystone gets 1.8 seconds without stats, 1.6 seconds with stats (!) and 1.7 seconds without the patch applied. Does anyone have the time to check on their setup/machines ? So far I'm quite confused... ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2007-02-25 18:46 Message: Logged In: YES user_id=4771 Originator: NO In the patch, the initial and the reset value of dict_timestamp_value are 0xFFFFFFFFU - 10000. I guess that it's a value you used while debugging the clearing code, and that it should now be 1 instead? ---------------------------------------------------------------------- Comment By: Andrea Griffini (ag6502) Date: 2006-12-30 13:47 Message: Logged In: YES user_id=1668946 Originator: YES I added the two missing parts and the patch should be now functionally complete; now there co_names reordering pass during compilation so that all names used in LOAD_GLOBAL/LOAD_ATTR are moved at the beginning of the tuple, this allows the lookup cache to be allocated only for the needed size. The reordering is done *before* actually assembling basic blocks to avoid to deal with the change of opcode size after reordering. Also now the timestamp is reset when rolling over to 0 and a sweep is done on all dictionaries and all code objects to clear cached lookups; I didn't implemented it in the gc because: 1) I'm not sure if all code objects and dictionaries are listed (how do I find code objects ?), 2) to do the traversing the code must be in gcmodule, or a richer interface must be exported, 3) what if garbage collecting is disabled ? The extra cost for dictionary creation is normally (i.e. when a dictionary is recovered from the free pool) just setting the timestamp to 1, the extra cost for dictionary update is higher but around the "noise" of timeit.Timer on my system. The memory cost is three more 32-bit words for every dict and 4 + n*3 more words for every code object where n is the number of elements of co_names that are used in LOAD_GLOBAL/LOAD_ATTR opcodes. The patch as it stands don't need to change marshalling of compiled code (the loaded code objects will simply be "unoptimized" so n will be 1+max(oparg) for all arguments of LOAD_GLOBAL/LOAD_ATTR - wasting some cache slots). The patch itself doesn't do anything unless the symbol CACHED_LOOKUPS is defined (this will cache LOAD_GLOBALs) or both CACHED_LOOKUPS and CACHED_MODULE_LOOKUPS are defined (this will cache both LOAD_GLOBALs and LOAD_ATTRs when the searched object is a module). This allows to have this optimization as an optional build option (does this make sense ?). File Added: cached_lookups_12.patch ---------------------------------------------------------------------- Comment By: Andrea Griffini (ag6502) Date: 2006-12-23 23:52 Message: Logged In: YES user_id=1668946 Originator: YES I was already changing the code in the direction you're pointing out. I attached the current version (still not complete!) of the patch; more specifically: - Cached lookups are now stored in an array of structures instead than in two separate arrays - Macros are used for handling of cache entries - The timestamp is now unsigned and there is optional support for 64-bit timestamps - Support for stalling timestamping at 32 bit wrap point - Removed the LOAD_FAST_HACK (was just an experiment with oprofile; was left there by mistake) - Fixed memory allocation failures handling in code object creation - Removed exporting of timestamp value at the python level - Used as cache size the largest LOAD_ATTR/LOAD_GLOBAL entry found Still missing are the sorting of co_names to pack cache slots and the cache reset at gc time if the dictionary timestamp is stalled. The cached lookup can be used at two levels: -DCACHED_LOOKUPS enables caching of LOAD_GLOBAL results -DCACHED_MODULE_LOOKUPS (in addition to -DCACHED_LOOKUPS) also caches the result of a lookup in a module. Speed results are sometimes very interesting and sometimes strange; I found measurable differences just moving around statements between logically equivalent places after checking what gcc was doing with register allocation (probably those places were indeed not equivalent if taking in account aliasing that isn't going to happen but that a c compiler must assume as possible). I have no idea if the speedups I've measured are better or worse on other processors/architectures. File Added: cached_lookups_10.patch ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-23 04:33 Message: Logged In: YES user_id=33168 Originator: NO I only reviewed the code in the patch. I have not thought conceptually about this, whether it's a good idea, how to break it, etc. I also did not look to verify all the necessary dict methods were updated to add TOUCHED. Have you profiled the slow down to dicts in normal cases? Things like: d = {} d[0] = 0 etc? All dict operations are going to be a tiny bit slower due to an additional assignment. It's probably not measurable, but you should still verify this. What is LOAD_FAST_HACK in ceval.c? It doesn't seem necessary for this patch. Please revert any whitespace/formatting changes that aren't part of the patch (there are a couple in dictobject.c and codeobject.c). It seems like a little structure would be better than parallel arrays. This would reduce some calculations and reduce the number of mallocs. Also there would only be a single assignment in the eval loop and the names would probably wind up being more clear. Would it be better to use PyObject_Malloc rather than PyMem_Malloc? It should be faster for sizes <= 256. (Despite the name, it doesn't have to handle PyObjects.) Why not use a size_t instead of Py_ssize_t for the timestamp? This is conceptually unsigned, no? Use the macro version PyTuple_GET_SIZE(names); in codeobject.c. It will be a little faster. If the cache allocs fail when allocating a code object, ceval will crash due to not checking for NULL. As for the timestamp, there should probably be a macro to access the field for a dict. This macro should be used in ceval.c/codeobject.c for getting/setting it. When returning the timestamp in dictobject.c you shouldn't cast it to a long. Use the appropriate method such as PyInt_FromSsize_t. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-16 23:57 Message: Logged In: YES user_id=21627 Originator: NO I think restricting yourself to 32-bit counters is fine (it's also an issue of memory overhead as you have one of these per access to a global, and per dictionary); you have to deal with the overflow, though. I suggest to use an unsigned value, e.g. size_t. Overflowing should be dealt with through visiting all objects, as you say. It could either happen right when it overflows, or it could get integrated into GC, and only reset when the GC runs. You'd then need to account for those periods when it already has overflowed, but GC hasn't run yet; in these periods, counting should be "on hold". ---------------------------------------------------------------------- Comment By: Andrea Griffini (ag6502) Date: 2006-12-16 18:13 Message: Logged In: YES user_id=1668946 Originator: YES I am experimenting with long long timestamps (based on "unsigned long long" if available or explicitly on two "unsigned long" otherwise). An alternative to slowing down lookups by using 64 bits on 32-bit computers is just using a single 32-bit counter and trigger a "cache flush" when the timestamp rolls over by visiting all live dictionaries and code objects. This would be IMO a *very* infrequent operation (for example on my P4/2.4GHz just doing nothing else but dict updates getting to 2**32 updates would require 14 minutes). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-16 12:01 Message: Logged In: YES user_id=21627 Originator: NO How does that deal with timestamp wraparound? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1616125&group_id=5470 From noreply at sourceforge.net Mon Feb 26 11:30:35 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 26 Feb 2007 02:30:35 -0800 Subject: [Patches] [ python-Patches-1660500 ] Hide iteration variable in list comprehensions Message-ID: Patches item #1660500, was opened at 2007-02-15 11:29 Message generated for change (Comment added) made by gbrandl You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660500&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Parser/Compiler Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Nick Coghlan (ncoghlan) Assigned to: Georg Brandl (gbrandl) Summary: Hide iteration variable in list comprehensions Initial Comment: This patch hides the iteration variable in list comprehensions. It adds new tests (modelled on the generator expression tests) and also removes some del statements from the standard library (where code previously cleaned up its own namespace). The changes to symtable.[ch] are more significant than strictly necessary - I found it necessary to spend some time cleaning up the code in order to understand what was needed for the list comprehension changes. Given that the 2.x and 3.0 compilers have already diverged fairly significantly, I don't believe this will make the process of keeping them in sync any more difficult than it is already. Assigning to Georg for initial review (as his set comprehensions patch provided a great deal of inspiration for this one). ---------------------------------------------------------------------- >Comment By: Georg Brandl (gbrandl) Date: 2007-02-26 10:30 Message: Logged In: YES user_id=849994 Originator: NO Okay, I looked at the patch, and apart from a refleak in one of the compiler methods I couldn't find anything wrong. I then manually applied the rest of my set comprehension patch. The result is attached. Grammar, AST and compilation for comprehensions are now really unified. It passes the tests for listcomps, genexps and setcomps. I couldn't check properly for refleaks since e.g. test_genexps leaks in the branch head, as well as some other tests. I'm currently searching for the offending revision(s)... File Added: new-set-comps.diff ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2007-02-15 12:13 Message: Logged In: YES user_id=1038590 Originator: YES For reference, the original set comprehensions patch & the py3k list discussion: http://www.python.org/sf/1548388 http://mail.python.org/pipermail/python-3000/2006-December/005188.html Note that the current patch ended up looking a *lot* like the original one (the main difference specific to list comprehensions is that the temporary list is built in the inner scope and then returned rather than being passed in as an argument. Additionally, the code has been unified only for the symtable stage - the AST and compilation stages still use separate code for listcomps and genexps). It turns out that there are some really curly scoping problems that using a nested function deals with automatically (see the new test_listcomps.py in the patch for examples). Having a more efficient mechanism specifically for 'transient' scopes like this is an interesting idea, but it's far from easy (mainly because the variables in the scope still need to be properly visible in scopes it *contains*). Adding set comprehensions on top of this patch should be pretty straightforward. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2007-02-15 11:53 Message: Logged In: YES user_id=1038590 Originator: YES Speed measurements show a significant speed up over trunk & Python 2.4 for module/class level code: (Python 2.4)$ python -m timeit -s "seq=range(1000)" "[[x for x in seq] for y in seq]" 10 loops, best of 3: 239 msec per loop (Python 2.x trunk)$ ./python -m timeit -s "seq=range(1000)" "[[x for x in seq] for y in seq]" 10 loops, best of 3: 193 msec per loop (Python 3000)$ ./python -m timeit -s "seq=range(1000)" "[[x for x in seq] for y in seq]" 10 loops, best of 3: 176 msec per loop This is almost certainly due to the variables and the list object becoming function locals. There is a slowdown inside a function (but we are still faster than Python 2.4): (Python 2.4)$ python -m timeit -s "seq=range(1000)" -s "def f(): return [[x for x in seq] for y in seq]" "f()" 10 loops, best of 3: 259 msec per loop (Python 2.x trunk)$ ./python -m timeit -s "seq=range(1000)" -s "def f(): return [[x for x in seq] for y in seq]" "f()" 10 loops, best of 3: 176 msec per loop (Python 3000)$ ./python -m timeit -s "seq=range(1000)" -s "def f(): return [[x for x in seq] for y in seq]" "f()" 10 loops, best of 3: 185 msec per loop ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1660500&group_id=5470 From noreply at sourceforge.net Mon Feb 26 16:19:09 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 26 Feb 2007 07:19:09 -0800 Subject: [Patches] [ python-Patches-1616125 ] Cached globals+builtins lookup optimization Message-ID: Patches item #1616125, was opened at 2006-12-14 16:44 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1616125&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Performance Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Andrea Griffini (ag6502) Assigned to: Nobody/Anonymous (nobody) Summary: Cached globals+builtins lookup optimization Initial Comment: This patch implements a speed optimization by introducing a timestamp on dictionaries and by caching lookups of constants so that lookup doesn't need to be repeated if the dictionary didn't change. Currently the patch implements the cached lookup only for the LOAD_GLOBAL opcode and stores the cache as two extra member in the code object. I'm going to investigate if LOAD_ATTR in the special case of a module is worth caching too. A cache entry for LOAD_GLOBAL is defined by two timestamps (Py_ssize_t) and a *borrowed* pointer to a PyObject. The use of a borrowed pointer is safe because the value is used only if the dictionaries didn't change, so the result of lookup must still be alive because the reference in the dictionary that returned the cached value it is still in place. The assumptions are: - a LOAD_GLOBAL opcode always looks for the same symbol (i.e. co_names never changes) - the dictionary being searched (globals and builtins) are mostly constant On my system I got a speedup of 8%-10% on a real program (a tcp/ip server that handles an in-memory database of python objects) on the checkpoint load + rollforward steps (a few millions LOAD_GLOBAL operations in total) and the varying percentage depends on which compiler options are used to build python. On another little test program that calls a function def f(x): return math.sin(x) + math.cos(x) + abs(x) one million times the speedup with the default makefile is about 7%. To enable the patch the makefile should define the CACHED_LOOKUPS symbol. The change to dictobject also exports the timestamp, at the python level; other changes are instead invisible at the python level. I'm not sure at all exporting dict.timestamp() is a good idea however... Note that after applying the patch you may find an error in userdict if you simply use "make" (there's probably some dependency problem somewhere). After make clean + make the test suite runs fine on my system. I'm interested in hearing if also other platforms do actually see a real performance improvement. Andrea ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-26 07:19 Message: Logged In: YES user_id=33168 Originator: NO What sort of results do you get with pybench (it's in Tools)? ---------------------------------------------------------------------- Comment By: Andrea Griffini (ag6502) Date: 2007-02-25 23:08 Message: Logged In: YES user_id=1668946 Originator: YES There are indeed two symbols used for debugging; one is CACHED_LOOKUP_STATS (Python/ceval.c) that outputs the hit/miss ratios for the cache when python exits and the other is LOOKUP_DEBUG_TIMESTAMP_LIMIT (Objects/dictobject.c) that forces the global counter to wrap around every 10000 dict changes (instead of every 2**32 dict changes). Actually on this patch I'm getting confusing results on my machine (for example getting better results with statistics ON, that to me just make no sense at all). I got no results from anyone else so I can't say if this patch is actually speeding up or slowing down python. On my machine (trunk python) pystone gets 1.8 seconds without stats, 1.6 seconds with stats (!) and 1.7 seconds without the patch applied. Does anyone have the time to check on their setup/machines ? So far I'm quite confused... ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2007-02-25 09:46 Message: Logged In: YES user_id=4771 Originator: NO In the patch, the initial and the reset value of dict_timestamp_value are 0xFFFFFFFFU - 10000. I guess that it's a value you used while debugging the clearing code, and that it should now be 1 instead? ---------------------------------------------------------------------- Comment By: Andrea Griffini (ag6502) Date: 2006-12-30 04:47 Message: Logged In: YES user_id=1668946 Originator: YES I added the two missing parts and the patch should be now functionally complete; now there co_names reordering pass during compilation so that all names used in LOAD_GLOBAL/LOAD_ATTR are moved at the beginning of the tuple, this allows the lookup cache to be allocated only for the needed size. The reordering is done *before* actually assembling basic blocks to avoid to deal with the change of opcode size after reordering. Also now the timestamp is reset when rolling over to 0 and a sweep is done on all dictionaries and all code objects to clear cached lookups; I didn't implemented it in the gc because: 1) I'm not sure if all code objects and dictionaries are listed (how do I find code objects ?), 2) to do the traversing the code must be in gcmodule, or a richer interface must be exported, 3) what if garbage collecting is disabled ? The extra cost for dictionary creation is normally (i.e. when a dictionary is recovered from the free pool) just setting the timestamp to 1, the extra cost for dictionary update is higher but around the "noise" of timeit.Timer on my system. The memory cost is three more 32-bit words for every dict and 4 + n*3 more words for every code object where n is the number of elements of co_names that are used in LOAD_GLOBAL/LOAD_ATTR opcodes. The patch as it stands don't need to change marshalling of compiled code (the loaded code objects will simply be "unoptimized" so n will be 1+max(oparg) for all arguments of LOAD_GLOBAL/LOAD_ATTR - wasting some cache slots). The patch itself doesn't do anything unless the symbol CACHED_LOOKUPS is defined (this will cache LOAD_GLOBALs) or both CACHED_LOOKUPS and CACHED_MODULE_LOOKUPS are defined (this will cache both LOAD_GLOBALs and LOAD_ATTRs when the searched object is a module). This allows to have this optimization as an optional build option (does this make sense ?). File Added: cached_lookups_12.patch ---------------------------------------------------------------------- Comment By: Andrea Griffini (ag6502) Date: 2006-12-23 14:52 Message: Logged In: YES user_id=1668946 Originator: YES I was already changing the code in the direction you're pointing out. I attached the current version (still not complete!) of the patch; more specifically: - Cached lookups are now stored in an array of structures instead than in two separate arrays - Macros are used for handling of cache entries - The timestamp is now unsigned and there is optional support for 64-bit timestamps - Support for stalling timestamping at 32 bit wrap point - Removed the LOAD_FAST_HACK (was just an experiment with oprofile; was left there by mistake) - Fixed memory allocation failures handling in code object creation - Removed exporting of timestamp value at the python level - Used as cache size the largest LOAD_ATTR/LOAD_GLOBAL entry found Still missing are the sorting of co_names to pack cache slots and the cache reset at gc time if the dictionary timestamp is stalled. The cached lookup can be used at two levels: -DCACHED_LOOKUPS enables caching of LOAD_GLOBAL results -DCACHED_MODULE_LOOKUPS (in addition to -DCACHED_LOOKUPS) also caches the result of a lookup in a module. Speed results are sometimes very interesting and sometimes strange; I found measurable differences just moving around statements between logically equivalent places after checking what gcc was doing with register allocation (probably those places were indeed not equivalent if taking in account aliasing that isn't going to happen but that a c compiler must assume as possible). I have no idea if the speedups I've measured are better or worse on other processors/architectures. File Added: cached_lookups_10.patch ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-22 19:33 Message: Logged In: YES user_id=33168 Originator: NO I only reviewed the code in the patch. I have not thought conceptually about this, whether it's a good idea, how to break it, etc. I also did not look to verify all the necessary dict methods were updated to add TOUCHED. Have you profiled the slow down to dicts in normal cases? Things like: d = {} d[0] = 0 etc? All dict operations are going to be a tiny bit slower due to an additional assignment. It's probably not measurable, but you should still verify this. What is LOAD_FAST_HACK in ceval.c? It doesn't seem necessary for this patch. Please revert any whitespace/formatting changes that aren't part of the patch (there are a couple in dictobject.c and codeobject.c). It seems like a little structure would be better than parallel arrays. This would reduce some calculations and reduce the number of mallocs. Also there would only be a single assignment in the eval loop and the names would probably wind up being more clear. Would it be better to use PyObject_Malloc rather than PyMem_Malloc? It should be faster for sizes <= 256. (Despite the name, it doesn't have to handle PyObjects.) Why not use a size_t instead of Py_ssize_t for the timestamp? This is conceptually unsigned, no? Use the macro version PyTuple_GET_SIZE(names); in codeobject.c. It will be a little faster. If the cache allocs fail when allocating a code object, ceval will crash due to not checking for NULL. As for the timestamp, there should probably be a macro to access the field for a dict. This macro should be used in ceval.c/codeobject.c for getting/setting it. When returning the timestamp in dictobject.c you shouldn't cast it to a long. Use the appropriate method such as PyInt_FromSsize_t. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-16 14:57 Message: Logged In: YES user_id=21627 Originator: NO I think restricting yourself to 32-bit counters is fine (it's also an issue of memory overhead as you have one of these per access to a global, and per dictionary); you have to deal with the overflow, though. I suggest to use an unsigned value, e.g. size_t. Overflowing should be dealt with through visiting all objects, as you say. It could either happen right when it overflows, or it could get integrated into GC, and only reset when the GC runs. You'd then need to account for those periods when it already has overflowed, but GC hasn't run yet; in these periods, counting should be "on hold". ---------------------------------------------------------------------- Comment By: Andrea Griffini (ag6502) Date: 2006-12-16 09:13 Message: Logged In: YES user_id=1668946 Originator: YES I am experimenting with long long timestamps (based on "unsigned long long" if available or explicitly on two "unsigned long" otherwise). An alternative to slowing down lookups by using 64 bits on 32-bit computers is just using a single 32-bit counter and trigger a "cache flush" when the timestamp rolls over by visiting all live dictionaries and code objects. This would be IMO a *very* infrequent operation (for example on my P4/2.4GHz just doing nothing else but dict updates getting to 2**32 updates would require 14 minutes). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2006-12-16 03:01 Message: Logged In: YES user_id=21627 Originator: NO How does that deal with timestamp wraparound? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1616125&group_id=5470 From noreply at sourceforge.net Mon Feb 26 20:18:47 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 26 Feb 2007 11:18:47 -0800 Subject: [Patches] [ python-Patches-1669379 ] bytes.fromhex() Message-ID: Patches item #1669379, was opened at 2007-02-26 19:18 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669379&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Georg Brandl (gbrandl) Assigned to: Neal Norwitz (nnorwitz) Summary: bytes.fromhex() Initial Comment: Implement bytes.fromhex(). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669379&group_id=5470 From noreply at sourceforge.net Mon Feb 26 21:58:35 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 26 Feb 2007 12:58:35 -0800 Subject: [Patches] [ python-Patches-1607548 ] Optional Argument Syntax Message-ID: Patches item #1607548, was opened at 2006-12-02 15:53 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607548&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 3000 Status: Open Resolution: Accepted Priority: 5 Private: No Submitted By: Tony Lownds (tonylownds) Assigned to: Guido van Rossum (gvanrossum) Summary: Optional Argument Syntax Initial Comment: This patch implements optional argument syntax for Python 3000. The patch still has issues; I am posting so that Collin Winters can add a link to the PEP. The syntax implemented is roughly: def f(arg:expr, (nested1:expr, nested2:expr)) -> expr: suite The function object has a new attribute, func_annotations that maps from argument names to the result of the expression. The return annotation is stored with a key of 'return'. Lambda's syntax doesn't support annotations. This patch alters the MAKE_FUNCTION opcode. I have an implementation that built the func_annotations dictionary in bytecode as well but it was bigger and slower. ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2007-02-26 15:58 Message: Logged In: YES user_id=6380 Originator: NO Tony, I am checking make_closure_fix.patch and peepholer_and_max_annotations.patch. Unfortunately, something happened to pydoc.py so that pydoc.patch no longer applies and I don't want to have to think about how to fix it. Can you have a look at this and submit a revised patch? ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-14 15:32 Message: Logged In: YES user_id=24100 Originator: YES File Added: make_closure_fix.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-14 15:31 Message: Logged In: YES user_id=24100 Originator: YES Combines the code paths for MAKE_FUNCTION and MAKE_CLOSURE. Fixes a crash where functions with closures and either annotations or keyword-only arguments result in MAKE_CLOSURE, but only MAKE_FUNCTION has the code to handle annotations or keyword-only arguments. Includes enough tests to trigger the bug. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-06 16:03 Message: Logged In: YES user_id=24100 Originator: YES I tried to implement getargspec() as described, and unfortunately there is another wrinkle to consider. Keyword-only arguments may or may not have defaults. So the invariant described in getargspec()'s docstring can't be maintained when simply appending keyword-only arguments. A tuple of four things is returned: (args, varargs, varkw, defaults). 'args' is a list of the argument names (it may contain nested lists). 'args' will include keyword-only argument names. 'varargs' and 'varkw' are the names of the * and ** arguments or None. 'defaults' is an n-tuple of the default values of the last n arguments. The attached patch adds an 'getfullargspec' API that returns complete information; 'getargspec' raises an error if information would be lost; the order of arguments in 'formatargspec' is backwards compatible, so that formatargspec(*getargspec(f)) == formatargspec(*getfullargspec(f)) when getargspec(f) does not raise an error. PEP 362 could and probably should replace the new getfullargspec() function, so I did not implement an API more complicated than a tuple. File Added: pydoc.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-06 15:05 Message: Logged In: YES user_id=24100 Originator: YES Change peepholer to not bail in the presence of EXTENDED_ARG + MAKE_FUNCTION. Enforce the natural 16-bit limit of annotations in compile.c. File Added: peepholer_and_max_annotations.patch ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-01-04 12:53 Message: Logged In: YES user_id=6380 Originator: NO I like the following approach: (1) the old API continues to work for all functions, but provides incomplete information (not losing the kw-only args completely, but losing the fact that they are kw-only); (2) add a new API that provides all the relevant information. Maybe the new API should not return a 7-tuple but rather a structure with named attributes; that makes it more future-proof. Sorry, I don't have any good suggestions for new names. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-04 02:12 Message: Logged In: YES user_id=24100 Originator: YES For getargs and getargvalues, including the names in positional args is an excellent strategy. There are uses (in cgitb) in the stdlib for getargvalues that then wouldn't need to be changed. The 2 uses of getargspec in the stdlib (one of which I missed, in DocXMLRPCServer) are both closely followed by formatargspec. I think those APIs should change or information will be lost. Alternatively, a new function (hopefully with a better name than getfullargspec :) could be made and getargspec could retain its API, but raise an error when keyword-only arguments are present. def getargspec(func): args, varargs, kwonlyargs, kwdefaults, varkw, defaults, ann = getfullargspec() if kwonlyargs: raise ValueError, "function has keyword-only arguments, use getfullargspec!" return args, varargs, varkw, defaults I'll update the patch to fix getargvalues and DocXMLRPCServer this weekend. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-01-04 00:22 Message: Logged In: YES user_id=6380 Originator: NO Well, it depends on the context whether that matters. The kw-only args could just be included in the positional args (which have names anyway) and that wouldn't be so bad for some apps. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-04 00:17 Message: Logged In: YES user_id=24100 Originator: YES I think everyone should update have to update their uses of getargspec and friends, because otherwise they will silently mis-handle keyword-only arguments. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-01-03 23:30 Message: Logged In: YES user_id=6380 Originator: NO I'm not sure it's right to just change the signature of the various functions in inspect.py; that would break all existing code using that module (and there definitely are other users besides pydoc). It would be better to add new methods that provide access to the additional functionality. Or do you think that everyone will have to change their code anyway? ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-28 01:53 Message: Logged In: YES user_id=33168 Originator: NO I'm skipping the pydoc patch. Didn't even look at it. I don't have the refleak, but I changed some calls and may have fixed it. Committed revision 53170. Leaving open to deal with the pydoc patch. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-27 22:04 Message: Logged In: YES user_id=24100 Originator: YES Nothing else on the C side of things. The pydoc patch works well for me; more tests ought to be added for function annotations and also for keyword-only arguments, but perhaps that can be added on as a later patch after checkin. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-12-27 20:38 Message: Logged In: YES user_id=6380 Originator: NO Thanks! Is there anything else that you think needs to be done before I check this in? The core code looks alright to me; I can't be bothered with reviewing the ast stuff or the compiler package since I don't know enough about these, but given that it compiles things correctly I'm not so worried about those. What's the status of the pydoc patch? Are you still working on that? ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-27 20:28 Message: Logged In: YES user_id=24100 Originator: YES Fixed in latest patch. Also added VISIT call for func_annotations. File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-12-27 19:40 Message: Logged In: YES user_id=6380 Originator: NO I believe I've found a leak in the code that adds annotations to a function object. See this session: >>> x = object() >>> import sys >>> sys.getrefcount(x) 2 >>> for i in range(100): ... def f(x: x): pass ... >>> del f >>> sys.getrefcount(x) 102 >>> At first I thought this could be due to the code added to the MAKE_FUNCTION opcode, but I don't see a leak there. More likely func_annotations is not being freed when a function object is deleted. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-23 14:05 Message: Logged In: YES user_id=24100 Originator: YES Initial patch to implement keyword-only arguments and annotations support for pydoc and inspect. Tests do not exercise these features, yet. Output for annotations that are types is special cased so that for: def intmin(*a: int) -> int: pass ...help(intmin) will display: intmin(*a: int) -> int File Added: pydoc.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-23 10:53 Message: Logged In: YES user_id=24100 Originator: YES Fixed the non-C89 style lines and the formatting (hopefully in compatible style :) File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-12-22 16:41 Message: Logged In: YES user_id=6380 Originator: NO Thanks for the progress! There are still a few lines ending in whitespace or lines that are longer than 80 chars (and weren't before). Mind cleaning those up? Also ceval.c:2305 and compile.c:1440 contain code that gcc 2.95 won't compile (the 'int' declarations ought to be moved to the start of the containing {...} block); I think this style is not C89 compatible. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-22 15:15 Message: Logged In: YES user_id=24100 Originator: YES Changes: 1. Fix crasher in Python/symtable.c -- annotations were visited inside the function scope 2. Fix Lib/compiler issues with Lib/test/test_complex_args. Output from Lib/compiler does not pass all tests, same failures as in HEAD of p3yk branch. File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-21 15:21 Message: Logged In: YES user_id=24100 Originator: YES Changes: 1. Address Neal's comments (I hope) 2. test_scope passes 3. Added some additional tests to test_compiler Open implementation issues: 1. Output from Lib/compiler does not pass test_complex_args, test_scope, possibly more. File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-20 17:13 Message: Logged In: YES user_id=24100 Originator: YES Changes: 1. Updated to apply cleanly 2. Fix to compile.c so that test_complex_args passes Open implementation issues: 1. Neal's comments 2. test_scope fails 3. Output from Lib/compiler does not pass test_complex_args File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-20 13:04 Message: Logged In: YES user_id=24100 Originator: YES I'll work on code formatting and the error checking and other cleanup. Open to other names than tname and vname, I created those non-terminals in order to use the same code for processing "def" and "lambda". Terminals are caps IIUC. I did add a test for the multi-paren situation. 2.5 had that bug too. Re: no changes to ceval, I tried generating the func_annotations dictionary using bytecodes. That doesn't change the ceval loop but was more code and was slower. So there is a way to avoid ceval changes. Re: deciding if lambda was going to require parens around the arguments, I don't think there was any decision, and yes annotations would be easily supportable. Happy to change if there is support, it's backwards incompatible. Re: return type syntax, I have only seen the -> syntax (vs a keyword 'as') on Guido's blog. Thanks for the comments! ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-20 04:25 Message: Logged In: YES user_id=33168 Originator: NO Nix this comment: I would definitely prefer the annotations baked into the code object so there are no changes to ceval. I see that Guido wants it the way it currently is which makes sense for nested functions. There should probably be a test with nested functions even though it really shouldn't be different. The test will verify that. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-20 03:38 Message: Logged In: YES user_id=33168 Originator: NO When regenerating the patch, can you also remove non-functional changes such as removing unneeded parens and whitespace changes. Also, please try to keep the same formatting in the file wrt tabs and spaces and don't move code around. I know this is a pain and inconsistent. I think I changed ast.c to be all 4 space indents with spaces only. In compiler_simple_arg(), don't you need to check if annotation is NULL when returned from ast_for_expr? Otherwise an undetected error would go through, wouldn't it? In compiler_complex_args(), don't you need to set the ast_error (or a SystemError) if the switch isn't a tname, vname, or LPAR? I don't like the names tname and vname. Also they seem inconsistent. Aren't all the other names all CAPS? In hunk, @@ -602,51 +625,75 @@ remove the commented out code. We shouldn't use any // style comments either. Can you improve the error msg for kwdefaults == NULL? (Thanks for adding it!) Check annotation for NULL if returned from ast_for_expr? BTW, the AST code in this area was tricky code which had some bugs. Did you test with adding extra parentheses and singleton tuples? I'm not sure if Guido preferred syntax -> vs a keyword 'as' for the return type. In symtable.c remove the printfs. They should probably be SystemErrors or something. I would definitely prefer the annotations baked into the code object so there are no changes to ceval. Did we decide if lambda was going to require parens around the arguments? If so, it could support annotations, right? (No comment on the usefulness of annotations for lambdas. :-) In compiler_visit_argannotation, you should return the result from PyList_Append and can remove the comment about checking for errors. Also, I believe the INCREF is not needed, it will be done by PyList_Append. Same deal with returning result of compiler_visit_argannotations() (the one with an s). Need to check for PyList_New() returning NULL in compiler_visit_annotations(). Lots more error checking needs to be added in this area. Dammit, I really want to use Mondrian for these comments! (Sorry Tony, not your fault, I'm just having some bad memories at this point cause I have to keep providing the references.) This patch looks very complete in that it updates things like the compiler package and the parsermodule.c. Good job! This is a great start. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-12-19 20:22 Message: Logged In: YES user_id=6380 Originator: NO Applying the patch fails, probably due to recent merge activities in the p3yk branch. Can I inconvenience you with a request to regenerate the patch from the branch head? ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-12-11 12:29 Message: Logged In: YES user_id=764593 Originator: NO Could you rename it to "argument annotations"? "optional argument" makes me think of the current keyword arguments, that can be but don't have to be passed. -jJ ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-03 20:24 Message: Logged In: YES user_id=24100 Originator: YES This patch implements optional argument syntax for Python 3000. The patch still has issues: 1. test_ast and test_scope fail. 2. Running the test suite after compiling the library with the compiler package causes failures 3. no docs 4. C-code reference counts and error checking needs a review The syntax implemented is roughly: def f(arg:expr, (nested1:expr, nested2:expr)) -> expr: suite The function object has a new attribute, func_annotations that maps from argument names to the result of the expression. The return annotation is stored with a key of 'return'. Lambda's syntax doesn't support annotations. The ast format has changed for the builtin compiler and the compiler package. A new token was added, '->' (called RARROW in token.h). token.py lost ERRORTOKEN after re-generating, I don't know why. I added it back manually. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607548&group_id=5470 From noreply at sourceforge.net Mon Feb 26 22:47:16 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 26 Feb 2007 13:47:16 -0800 Subject: [Patches] [ python-Patches-1669481 ] subprocess: Support close_fds on Win32 Message-ID: Patches item #1669481, was opened at 2007-02-26 21:47 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669481&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Jon Foster (jongfoster) Assigned to: Nobody/Anonymous (nobody) Summary: subprocess: Support close_fds on Win32 Initial Comment: There is an existing close_fds parameter to the subprocess.Popen() constructor that prevents FDs being copied to the child process on UNIX. This patch makes that parameter also work on Windows (with a caveat). Setting this parameter on Windows was explicitly prohibited by the code, because it prevents redirecting stdin/stdout/stderr. However, the check is too restrictive; it is perfectly reasonable to set this flag if you are not redirecting anything. That is what this patch does. The behaviour is unchanged if you are on Windows, set this flag, and simultaneously try to redirect stdin/stdout/stderr: the call will fail with an exception. This patch is against Python SVN trunk. Kind regards, Jon Foster ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669481&group_id=5470 From noreply at sourceforge.net Tue Feb 27 00:07:51 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 26 Feb 2007 15:07:51 -0800 Subject: [Patches] [ python-Patches-1669539 ] Change (fix!) os.path.isabs() semantics on Win32 Message-ID: Patches item #1669539, was opened at 2007-02-26 23:07 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669539&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Jon Foster (jongfoster) Assigned to: Nobody/Anonymous (nobody) Summary: Change (fix!) os.path.isabs() semantics on Win32 Initial Comment: Hi, I consider this to be a bug in os.path.isabs(): PythonWin 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32. >>> import os.path >>> s = '\\Python25' >>> os.path.isabs(s) True >>> os.path.abspath(s) 'C:\\Python25' >>> os.chdir('d:') >>> os.path.abspath(s) 'D:\\Python25' >>> If s is really an absolute path as isabs() claims, then why does abspath() return a different path (i.e. not s)? And worse, note that a call to os.chdir() changes the meaning of s! So s is clearly not an absolute path, and isabs() is wrong. It turns out that on Windows there are really 4 different kinds of paths: 1) Completely relative, e.g. foo\bar 2) Completely absolute, e.g. c:\foo\bar or \\server\share 3) Halfbreeds with no drive, e.g. \foo\bar 4) Halfbreeds relative to the current working directory on a specific drive, e.g. c:foo\bar Python 2.5's os.path.isabs() method considers both (2) and (3) to be absolute; I agree with the classification of (2) but strongly disagree about case (3). Oh, and os.path.join is also broken, e.g. os.path.join('foo', 'a:bar') gives 'foo\\a:bar', which is an invalid path. Another consequence of this is that os.path.isabs() is not enough to classify paths. Sometimes you really need a relative path, so we really need (at least) a new os.path.isrelative(), which can return "not isabs(s)" on POSIX platforms, and do the right thing on Win32. The attached patch: - Changes the behaviour of os.path.isabs() and os.path.join() on Win32, to classify pathnames as described above (including adding UNC support) - Adds os.path.isrelative() on all platforms - Changes a couple of Win32 os.path tests where I have deliberately broken backward compatibility - Adds lots of new tests for these 3 functions on Win32 This does change the behaviour of isabs(), and it is possible that existing applications might be depending on the current behaviour. Silently changing the behaviour might break those applications. I'm not sure what the best course of action is - applying this for 2.6, putting a new API in (os.path.isreallyabs()?), or waiting for Python 3000...? Kind regards, Jon Foster ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669539&group_id=5470 From noreply at sourceforge.net Tue Feb 27 00:15:21 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 26 Feb 2007 15:15:21 -0800 Subject: [Patches] [ python-Patches-1669539 ] Change (fix!) os.path.isabs() semantics on Win32 Message-ID: Patches item #1669539, was opened at 2007-02-26 23:07 Message generated for change (Comment added) made by jongfoster You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669539&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Jon Foster (jongfoster) Assigned to: Nobody/Anonymous (nobody) Summary: Change (fix!) os.path.isabs() semantics on Win32 Initial Comment: Hi, I consider this to be a bug in os.path.isabs(): PythonWin 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32. >>> import os.path >>> s = '\\Python25' >>> os.path.isabs(s) True >>> os.path.abspath(s) 'C:\\Python25' >>> os.chdir('d:') >>> os.path.abspath(s) 'D:\\Python25' >>> If s is really an absolute path as isabs() claims, then why does abspath() return a different path (i.e. not s)? And worse, note that a call to os.chdir() changes the meaning of s! So s is clearly not an absolute path, and isabs() is wrong. It turns out that on Windows there are really 4 different kinds of paths: 1) Completely relative, e.g. foo\bar 2) Completely absolute, e.g. c:\foo\bar or \\server\share 3) Halfbreeds with no drive, e.g. \foo\bar 4) Halfbreeds relative to the current working directory on a specific drive, e.g. c:foo\bar Python 2.5's os.path.isabs() method considers both (2) and (3) to be absolute; I agree with the classification of (2) but strongly disagree about case (3). Oh, and os.path.join is also broken, e.g. os.path.join('foo', 'a:bar') gives 'foo\\a:bar', which is an invalid path. Another consequence of this is that os.path.isabs() is not enough to classify paths. Sometimes you really need a relative path, so we really need (at least) a new os.path.isrelative(), which can return "not isabs(s)" on POSIX platforms, and do the right thing on Win32. The attached patch: - Changes the behaviour of os.path.isabs() and os.path.join() on Win32, to classify pathnames as described above (including adding UNC support) - Adds os.path.isrelative() on all platforms - Changes a couple of Win32 os.path tests where I have deliberately broken backward compatibility - Adds lots of new tests for these 3 functions on Win32 This does change the behaviour of isabs(), and it is possible that existing applications might be depending on the current behaviour. Silently changing the behaviour might break those applications. I'm not sure what the best course of action is - applying this for 2.6, putting a new API in (os.path.isreallyabs()?), or waiting for Python 3000...? Kind regards, Jon Foster ---------------------------------------------------------------------- >Comment By: Jon Foster (jongfoster) Date: 2007-02-26 23:15 Message: Logged In: YES user_id=199289 Originator: YES P.S. The patch is against Python SVN trunk ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669539&group_id=5470 From ctmc at blueheronmercantile.com Tue Feb 27 01:23:54 2007 From: ctmc at blueheronmercantile.com (Isolde Stark) Date: Mon, 26 Feb 2007 19:23:54 -0500 Subject: [Patches] You can do this in various ways such as through the Application Server Administration Console or by copying the CustomELResolver. Message-ID: <000d01c75a05$faf3da90$4c26a9c7@eoiepm> DataSource, and JAXB objects. If you don't, see "Chapter 24: Introduction to the Java Persistence API" in the Java EE 5 TutorialA sample packageaccompanies the tip. Welcome to the Java Technology Fundamentals Newsletter. NORTH, left, 25,SpringLayout. In the sample, this is done using an ant task with the target name compile-server. The JSP page then uses the stateless session bean to access the Order entity. Note that the client uses a JAXB binding configuration file, config-client. Make sure to add the SDK's bin directory to your path. Derby was previously available under its earlier name, Cloudscape, from its former owners: Cloudscape, Informix, and IBM. Because the JNDIELResolver simply adds JNDI capabilities to the EL, a null base argument means that it must look at the value of the property argument. An interceptor is a method that you can interpose in the invocation flow of an enterprise bean. You then create a header fragment and a navigation fragment and include these fragments in the home page. You can download the Java EE 5 SDK from the the Java EE 5 SDK Downloads page. The Provider interface supports methods of type SOAPMessage. It's used as a proxy to transmit requests to a server in a different domain. In Java EE 5, VariableResolver and PropertyResolver have been deprecated. The returned value of the method call then becomes the new setting for the same color property. In the top part of the GUI, a top-to-bottom box layout places a label above a scroll pane. Welcome to the Java Technology Fundamentals Newsletter. GROUP BY and HAVING ClausesThe Java Persistence Query Language also supports the GROUP BY and HAVING clauses. Although the previous example is sufficient, if you want to include a JColorChooser within your own window, more often than not you want the JColorChooser to appear in a separate popup window. The utility method getSourceFromSOAPMessage returns a Source object. " What the system does with the unitPrice property depends on the usage context. The significant amount of lab time illustrates the workshop nature of this course. 1 breaks down the expression into two parts: requestScope and user. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20070226/3a6584e7/attachment.htm -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 11506 bytes Desc: not available Url : http://mail.python.org/pipermail/patches/attachments/20070226/3a6584e7/attachment-0001.gif From noreply at sourceforge.net Tue Feb 27 02:45:31 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 26 Feb 2007 17:45:31 -0800 Subject: [Patches] [ python-Patches-1429818 ] trace.py needs to know about doctests Message-ID: Patches item #1429818, was opened at 2006-02-11 21:22 Message generated for change (Comment added) made by haypo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1429818&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.4 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Marius Gedminas (mgedmin) Assigned to: Nobody/Anonymous (nobody) Summary: trace.py needs to know about doctests Initial Comment: SchoolTool's test runner uses the 'trace' module to produce unit test coverage reports. Many unit tests in SchoolTool are actually doctests. When the test runner invokes trace.CoverageResults.write_results, it gets a lot of printed error messages like this: Not printing coverage data for '': [Errno 2] No such file or directory: '' and a lot of empty files with nimes like >> do_something(3) 4 """ if x == 0: return -1 return x + 1 if __name__ == '__main__': import doctest doctest.testmod() Now run python -m trace --cover --missing --coverdir=output foo.py trace.py will produce 'foo.cover' that looks like this 1: import doctest 1: def do_something(x): """ >>> do_something(3) 4 """ 1: if x == 0: >>>>>> return -1 1: return x + 1 1: if __name__ == '__main__': 1: import doctest 1: doctest.testmod() It will also produce an empty file called '': [Errno 2] No such file or directory: '' With my patch, trace.py will produce a 'foo.cover' file with the same content, and it will not create empty files nor produce any warnings. (I do not particularly care about tracing Python statements inside doctests themselves. I would like to keep this bug report focused, so if anyone wants doctest coverage in addition to regular Python code coverage, I suggest opening a new bug report.) Given this, I think the answers to your questions are (1) yes (2) what is 'the code that creates a test'? If you mean the '>>> do_something(4)' line, then I'm content if it remains untracked (3) whether doctests are in the same file or not is irrelevant, in my opinion (in SchoolTool most of the doctests are in separate files); I do not see where they might get mistaken for code There are no test-portions-that-didn't run, so there's no need for subdirectories. It would not be OK if some of the tests/doctests didn't actually run because of trace.py. Luckily, that is not the case. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-02-15 17:20 Message: Logged In: YES user_id=764593 Do I understand this correctly, then (1) doctests are still run, and contribute to coverage (2) The code that creates a test should not be measured for coverage. ??? (3) Because doctests are in the same file as the code, rather than the other tests, they get mistaken for code whose own coverage should be measured. If so, it seems like the right solution is to add a subdirectory for test-portions-that-didn't-run, and put doctests there instead of in the main directory. Or are you saying that is OK if some of the tests (and some of the doctests) don't actually run in full? -jJ ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1429818&group_id=5470 From noreply at sourceforge.net Tue Feb 27 04:04:19 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 26 Feb 2007 19:04:19 -0800 Subject: [Patches] [ python-Patches-1669633 ] methods for bytes Message-ID: Patches item #1669633, was opened at 2007-02-27 03:04 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Pete Shinners (shredwheat) Assigned to: Nobody/Anonymous (nobody) Summary: methods for bytes Initial Comment: This adds the methods for bytes objects as listed in PEP 358. It includes tests for the newly added methods. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 From noreply at sourceforge.net Tue Feb 27 06:18:10 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 26 Feb 2007 21:18:10 -0800 Subject: [Patches] [ python-Patches-1472639 ] make range be xrange Message-ID: Patches item #1472639, was opened at 2006-04-18 15:40 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1472639&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Thomas Wouters (twouters) Assigned to: Neal Norwitz (nnorwitz) Summary: make range be xrange Initial Comment: As discussed (in private email), make range an alias for xrange and see howmuch breaks ;) Aside from Python/bltinmodule.c, 31 files had to be changed: 3 modules (but one of them was a doctest in doctest), 2 test output files (for profile and cProfile), and 26 tests. The predominant breakage in tests was due to tests using range() to express expected-test-output, and comparing it with the output list. Another fair sized portion (particularly of doctests, and including the doctest in doctest itself that had to be updated) broke because of reliance on the repr of range(). Only a few tests broke because of xrange() being immutable (mostly tests that were actually testing list-behaviour, like item- and slice-assignment, on a list created by range()), but that were all two real breakages in actual modules. The only place that broke because xrange can't handle longs was the test for range() that tested whether it'd take longs. Overall, ~185 lines had to be changed. The patch still breaks a test: the test to see if range() does proper checks on its arguments (using the 'badint' class in test_builtin.) I didn't fix it to remind myself that xrange() should be made to operate on longs :) (It currently fails because xrange() will turn all its arguments into C long integers before it does any checks.) ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-26 21:18 Message: Logged In: YES user_id=33168 Originator: NO Updating a checkpoint which patches everything in the library. Also has a non-working version of xrange. I haven't ported this from 2.6 to the int/long unification. This is just in case my laptop crashes. Hopefully I'll finish this up tomorrow and check in. Most of the tests pass. There are only a few issues AFAIK, like test_xrange (big surprise) and test_repr. Basically the longs aren't done. I'm not sure about floats. Floats should be rejected in 3k. File Added: xrange.diff ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-12-27 17:39 Message: Logged In: YES user_id=6380 Originator: NO Ping? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-08-26 13:20 Message: Logged In: YES user_id=6380 For Neal to close when his range/xrange patch is uploaded. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1472639&group_id=5470 From noreply at sourceforge.net Tue Feb 27 06:35:27 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 26 Feb 2007 21:35:27 -0800 Subject: [Patches] [ python-Patches-1669633 ] methods for bytes Message-ID: Patches item #1669633, was opened at 2007-02-26 19:04 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Pete Shinners (shredwheat) Assigned to: Nobody/Anonymous (nobody) Summary: methods for bytes Initial Comment: This adds the methods for bytes objects as listed in PEP 358. It includes tests for the newly added methods. ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-26 21:35 Message: Logged In: YES user_id=33168 Originator: NO Thanks for the patch. I'd like to try to focus on reviewing only new code. How much of this code was copied straight from string, how much was modified and how much was new? There are a bunch of formatting changes. Could you revert those portions and add them to a separate patch (the file can be attached here if you want). Since you are adding a Fini method, I wonder if you ought to add a corresponding Init method. This would init nullbytes and save a comparison on each instantiation of a bytes object. _adjust_indices() looks generic, does this code exist elsewhere/can it be reused? All the methods that take a single argument should use METH_O (e.g., bytes_extend). Shouldn't value in bytes_remove(), bytes_insert(), and possibly elsewhere be limited to 0..255? If the value is > 255, it will traverse the entire bytes, then raise an exception rather than fail early. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 From noreply at sourceforge.net Tue Feb 27 07:41:11 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 26 Feb 2007 22:41:11 -0800 Subject: [Patches] [ python-Patches-1669379 ] bytes.fromhex() Message-ID: Patches item #1669379, was opened at 2007-02-26 11:18 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669379&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Georg Brandl (gbrandl) Assigned to: Neal Norwitz (nnorwitz) Summary: bytes.fromhex() Initial Comment: Implement bytes.fromhex(). ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-26 22:41 Message: Logged In: YES user_id=33168 Originator: NO I don't know if this is supposed to be in the API or not. I'm too tired to care, but here are my comments on the code. It would be better to use a variable name other than new, since new is a keyword in C++. isspace() checks for any whitespace, so not only can there be spaces, there can be tabs, carriage returns, new lines, etc. I don't know if the code is correct or the comment is. I see you have a test for other whitespace, so I would just update the comment to say _any_ whitespace. Does byteslen need the + 1? Is there a macro to get ob_bytes rather than accessing directly? buf = ((PyBytesObject *)new)->ob_bytes; It would be good to add the index position (i) in the error msg. It might be hard to see/non-obvious from the 2 chars (or potentially ambiguous). How about adding a test that passes a buffer('') and buffer('00')? These are all nits. Assuming Guido is fine with the API, I'd say clean it up and check it in. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669379&group_id=5470 From noreply at sourceforge.net Tue Feb 27 08:28:38 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Mon, 26 Feb 2007 23:28:38 -0800 Subject: [Patches] [ python-Patches-1669633 ] methods for bytes Message-ID: Patches item #1669633, was opened at 2007-02-27 03:04 Message generated for change (Comment added) made by shredwheat You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Pete Shinners (shredwheat) Assigned to: Nobody/Anonymous (nobody) Summary: methods for bytes Initial Comment: This adds the methods for bytes objects as listed in PEP 358. It includes tests for the newly added methods. ---------------------------------------------------------------------- >Comment By: Pete Shinners (shredwheat) Date: 2007-02-27 07:28 Message: Logged In: YES user_id=1076442 Originator: YES This is a split of the whitespacing. This matches the PEP7 style guide for 4-space indentation in C code. File Added: bytesspace.diff ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-27 05:35 Message: Logged In: YES user_id=33168 Originator: NO Thanks for the patch. I'd like to try to focus on reviewing only new code. How much of this code was copied straight from string, how much was modified and how much was new? There are a bunch of formatting changes. Could you revert those portions and add them to a separate patch (the file can be attached here if you want). Since you are adding a Fini method, I wonder if you ought to add a corresponding Init method. This would init nullbytes and save a comparison on each instantiation of a bytes object. _adjust_indices() looks generic, does this code exist elsewhere/can it be reused? All the methods that take a single argument should use METH_O (e.g., bytes_extend). Shouldn't value in bytes_remove(), bytes_insert(), and possibly elsewhere be limited to 0..255? If the value is > 255, it will traverse the entire bytes, then raise an exception rather than fail early. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 From noreply at sourceforge.net Tue Feb 27 09:42:33 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 27 Feb 2007 00:42:33 -0800 Subject: [Patches] [ python-Patches-1669379 ] bytes.fromhex() Message-ID: Patches item #1669379, was opened at 2007-02-26 19:18 Message generated for change (Comment added) made by gbrandl You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669379&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: None >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Georg Brandl (gbrandl) Assigned to: Neal Norwitz (nnorwitz) Summary: bytes.fromhex() Initial Comment: Implement bytes.fromhex(). ---------------------------------------------------------------------- >Comment By: Georg Brandl (gbrandl) Date: 2007-02-27 08:42 Message: Logged In: YES user_id=849994 Originator: YES It is in the PEP, so I assume it is in the API. I intended for all spaces to be allowed, but the PEP says otherwise, so I updated the implementation accordingly. I fixed the rest of your nits, fixed another bug and checked it in as rev. 53989. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-27 06:41 Message: Logged In: YES user_id=33168 Originator: NO I don't know if this is supposed to be in the API or not. I'm too tired to care, but here are my comments on the code. It would be better to use a variable name other than new, since new is a keyword in C++. isspace() checks for any whitespace, so not only can there be spaces, there can be tabs, carriage returns, new lines, etc. I don't know if the code is correct or the comment is. I see you have a test for other whitespace, so I would just update the comment to say _any_ whitespace. Does byteslen need the + 1? Is there a macro to get ob_bytes rather than accessing directly? buf = ((PyBytesObject *)new)->ob_bytes; It would be good to add the index position (i) in the error msg. It might be hard to see/non-obvious from the 2 chars (or potentially ambiguous). How about adding a test that passes a buffer('') and buffer('00')? These are all nits. Assuming Guido is fine with the API, I'd say clean it up and check it in. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669379&group_id=5470 From noreply at sourceforge.net Tue Feb 27 17:41:07 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 27 Feb 2007 08:41:07 -0800 Subject: [Patches] [ python-Patches-1669633 ] methods for bytes Message-ID: Patches item #1669633, was opened at 2007-02-27 03:04 Message generated for change (Comment added) made by shredwheat You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Pete Shinners (shredwheat) Assigned to: Nobody/Anonymous (nobody) Summary: methods for bytes Initial Comment: This adds the methods for bytes objects as listed in PEP 358. It includes tests for the newly added methods. ---------------------------------------------------------------------- >Comment By: Pete Shinners (shredwheat) Date: 2007-02-27 16:41 Message: Logged In: YES user_id=1076442 Originator: YES Attached is the updated patch. This should apply clean after the "fromhex" patch that was recently applied. This ignores the previous whitespace cleanup patch. Most of this code was copied from string. Minor modifications were made on these, but no real code changes. The following list of new methods are mostly modelled after list methods. extend, insert, append, reverse, pop, remove This adds an Init method to match the Fini. Also added a comment stating partition is the only dependency of all this init. If partiion is removed from bytes, so should this global stuff. _adjust_indices was copied directly from str. This is not used anywhere else in the codebase, so it should be unique again when str is removed. Added comment about this. Switched new single argument methods to METH_O. The 0-255 range is enforced on insert, append, and remove. Outside results in ValueError. File Added: bytesmethods2.diff ---------------------------------------------------------------------- Comment By: Pete Shinners (shredwheat) Date: 2007-02-27 07:28 Message: Logged In: YES user_id=1076442 Originator: YES This is a split of the whitespacing. This matches the PEP7 style guide for 4-space indentation in C code. File Added: bytesspace.diff ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-27 05:35 Message: Logged In: YES user_id=33168 Originator: NO Thanks for the patch. I'd like to try to focus on reviewing only new code. How much of this code was copied straight from string, how much was modified and how much was new? There are a bunch of formatting changes. Could you revert those portions and add them to a separate patch (the file can be attached here if you want). Since you are adding a Fini method, I wonder if you ought to add a corresponding Init method. This would init nullbytes and save a comparison on each instantiation of a bytes object. _adjust_indices() looks generic, does this code exist elsewhere/can it be reused? All the methods that take a single argument should use METH_O (e.g., bytes_extend). Shouldn't value in bytes_remove(), bytes_insert(), and possibly elsewhere be limited to 0..255? If the value is > 255, it will traverse the entire bytes, then raise an exception rather than fail early. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 From noreply at sourceforge.net Tue Feb 27 17:42:24 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 27 Feb 2007 08:42:24 -0800 Subject: [Patches] [ python-Patches-1669633 ] methods for bytes Message-ID: Patches item #1669633, was opened at 2007-02-27 03:04 Message generated for change (Settings changed) made by shredwheat You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Pete Shinners (shredwheat) >Assigned to: Neal Norwitz (nnorwitz) Summary: methods for bytes Initial Comment: This adds the methods for bytes objects as listed in PEP 358. It includes tests for the newly added methods. ---------------------------------------------------------------------- Comment By: Pete Shinners (shredwheat) Date: 2007-02-27 16:41 Message: Logged In: YES user_id=1076442 Originator: YES Attached is the updated patch. This should apply clean after the "fromhex" patch that was recently applied. This ignores the previous whitespace cleanup patch. Most of this code was copied from string. Minor modifications were made on these, but no real code changes. The following list of new methods are mostly modelled after list methods. extend, insert, append, reverse, pop, remove This adds an Init method to match the Fini. Also added a comment stating partition is the only dependency of all this init. If partiion is removed from bytes, so should this global stuff. _adjust_indices was copied directly from str. This is not used anywhere else in the codebase, so it should be unique again when str is removed. Added comment about this. Switched new single argument methods to METH_O. The 0-255 range is enforced on insert, append, and remove. Outside results in ValueError. File Added: bytesmethods2.diff ---------------------------------------------------------------------- Comment By: Pete Shinners (shredwheat) Date: 2007-02-27 07:28 Message: Logged In: YES user_id=1076442 Originator: YES This is a split of the whitespacing. This matches the PEP7 style guide for 4-space indentation in C code. File Added: bytesspace.diff ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-27 05:35 Message: Logged In: YES user_id=33168 Originator: NO Thanks for the patch. I'd like to try to focus on reviewing only new code. How much of this code was copied straight from string, how much was modified and how much was new? There are a bunch of formatting changes. Could you revert those portions and add them to a separate patch (the file can be attached here if you want). Since you are adding a Fini method, I wonder if you ought to add a corresponding Init method. This would init nullbytes and save a comparison on each instantiation of a bytes object. _adjust_indices() looks generic, does this code exist elsewhere/can it be reused? All the methods that take a single argument should use METH_O (e.g., bytes_extend). Shouldn't value in bytes_remove(), bytes_insert(), and possibly elsewhere be limited to 0..255? If the value is > 255, it will traverse the entire bytes, then raise an exception rather than fail early. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 From noreply at sourceforge.net Tue Feb 27 19:08:12 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 27 Feb 2007 10:08:12 -0800 Subject: [Patches] [ python-Patches-1670209 ] Remove Py_PROTO from socket in py3k Message-ID: Patches item #1670209, was opened at 2007-02-27 18:08 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1670209&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Pete Shinners (shredwheat) Assigned to: Nobody/Anonymous (nobody) Summary: Remove Py_PROTO from socket in py3k Initial Comment: A few files used to build the socket module have Py_PROTO macros used for forward function declarations. These are preventing the socket module from compiling under cygwin. Searching the python code does not show these being defined anywhere under Py3k. In the 2.6 trunk it is defined in pyport.h ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1670209&group_id=5470 From noreply at sourceforge.net Tue Feb 27 20:02:41 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 27 Feb 2007 11:02:41 -0800 Subject: [Patches] [ python-Patches-1669633 ] methods for bytes Message-ID: Patches item #1669633, was opened at 2007-02-26 19:04 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 3000 >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Pete Shinners (shredwheat) Assigned to: Neal Norwitz (nnorwitz) Summary: methods for bytes Initial Comment: This adds the methods for bytes objects as listed in PEP 358. It includes tests for the newly added methods. ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-27 11:02 Message: Logged In: YES user_id=33168 Originator: NO Thanks Pete! Committed revision 54000. ---------------------------------------------------------------------- Comment By: Pete Shinners (shredwheat) Date: 2007-02-27 08:41 Message: Logged In: YES user_id=1076442 Originator: YES Attached is the updated patch. This should apply clean after the "fromhex" patch that was recently applied. This ignores the previous whitespace cleanup patch. Most of this code was copied from string. Minor modifications were made on these, but no real code changes. The following list of new methods are mostly modelled after list methods. extend, insert, append, reverse, pop, remove This adds an Init method to match the Fini. Also added a comment stating partition is the only dependency of all this init. If partiion is removed from bytes, so should this global stuff. _adjust_indices was copied directly from str. This is not used anywhere else in the codebase, so it should be unique again when str is removed. Added comment about this. Switched new single argument methods to METH_O. The 0-255 range is enforced on insert, append, and remove. Outside results in ValueError. File Added: bytesmethods2.diff ---------------------------------------------------------------------- Comment By: Pete Shinners (shredwheat) Date: 2007-02-26 23:28 Message: Logged In: YES user_id=1076442 Originator: YES This is a split of the whitespacing. This matches the PEP7 style guide for 4-space indentation in C code. File Added: bytesspace.diff ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-26 21:35 Message: Logged In: YES user_id=33168 Originator: NO Thanks for the patch. I'd like to try to focus on reviewing only new code. How much of this code was copied straight from string, how much was modified and how much was new? There are a bunch of formatting changes. Could you revert those portions and add them to a separate patch (the file can be attached here if you want). Since you are adding a Fini method, I wonder if you ought to add a corresponding Init method. This would init nullbytes and save a comparison on each instantiation of a bytes object. _adjust_indices() looks generic, does this code exist elsewhere/can it be reused? All the methods that take a single argument should use METH_O (e.g., bytes_extend). Shouldn't value in bytes_remove(), bytes_insert(), and possibly elsewhere be limited to 0..255? If the value is > 255, it will traverse the entire bytes, then raise an exception rather than fail early. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 From noreply at sourceforge.net Tue Feb 27 20:06:36 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 27 Feb 2007 11:06:36 -0800 Subject: [Patches] [ python-Patches-1670209 ] Remove Py_PROTO from socket in py3k Message-ID: Patches item #1670209, was opened at 2007-02-27 10:08 Message generated for change (Comment added) made by nnorwitz You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1670209&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: Python 3000 >Status: Closed >Resolution: Accepted Priority: 5 Private: No Submitted By: Pete Shinners (shredwheat) Assigned to: Nobody/Anonymous (nobody) Summary: Remove Py_PROTO from socket in py3k Initial Comment: A few files used to build the socket module have Py_PROTO macros used for forward function declarations. These are preventing the socket module from compiling under cygwin. Searching the python code does not show these being defined anywhere under Py3k. In the 2.6 trunk it is defined in pyport.h ---------------------------------------------------------------------- >Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-27 11:06 Message: Logged In: YES user_id=33168 Originator: NO Committed revision 54001. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1670209&group_id=5470 From unxzmukyod at auna.net Tue Feb 27 21:04:01 2007 From: unxzmukyod at auna.net (Welsh & English Cain) Date: Tue, 27 Feb 2007 12:04:01 -0800 Subject: [Patches] ****SPAM**** Johnnie loves this program Message-ID: <1db901c75aa7$c6be0a50$2410a383@LQQBGD> Nice to get in touch with you Isabelle, Johnnie loves this program Get your start bonus now: - Visit our website http://fderolg.com/micro/16/ and download the playing module in your language (no installation needed) - Register as guest or player......and start winning Cheers Welsh & English Cain -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/patches/attachments/20070227/2f8aceac/attachment.htm From noreply at sourceforge.net Tue Feb 27 23:23:20 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 27 Feb 2007 14:23:20 -0800 Subject: [Patches] [ python-Patches-1669633 ] methods for bytes Message-ID: Patches item #1669633, was opened at 2007-02-27 03:04 Message generated for change (Comment added) made by shredwheat You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 3000 Status: Closed Resolution: Accepted Priority: 5 Private: No Submitted By: Pete Shinners (shredwheat) Assigned to: Neal Norwitz (nnorwitz) Summary: methods for bytes Initial Comment: This adds the methods for bytes objects as listed in PEP 358. It includes tests for the newly added methods. ---------------------------------------------------------------------- >Comment By: Pete Shinners (shredwheat) Date: 2007-02-27 22:23 Message: Logged In: YES user_id=1076442 Originator: YES This patch adds many more tests for bytes. It breaks the common string_tests into a lower level base class that can support bytes and strings. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-27 19:02 Message: Logged In: YES user_id=33168 Originator: NO Thanks Pete! Committed revision 54000. ---------------------------------------------------------------------- Comment By: Pete Shinners (shredwheat) Date: 2007-02-27 16:41 Message: Logged In: YES user_id=1076442 Originator: YES Attached is the updated patch. This should apply clean after the "fromhex" patch that was recently applied. This ignores the previous whitespace cleanup patch. Most of this code was copied from string. Minor modifications were made on these, but no real code changes. The following list of new methods are mostly modelled after list methods. extend, insert, append, reverse, pop, remove This adds an Init method to match the Fini. Also added a comment stating partition is the only dependency of all this init. If partiion is removed from bytes, so should this global stuff. _adjust_indices was copied directly from str. This is not used anywhere else in the codebase, so it should be unique again when str is removed. Added comment about this. Switched new single argument methods to METH_O. The 0-255 range is enforced on insert, append, and remove. Outside results in ValueError. File Added: bytesmethods2.diff ---------------------------------------------------------------------- Comment By: Pete Shinners (shredwheat) Date: 2007-02-27 07:28 Message: Logged In: YES user_id=1076442 Originator: YES This is a split of the whitespacing. This matches the PEP7 style guide for 4-space indentation in C code. File Added: bytesspace.diff ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-27 05:35 Message: Logged In: YES user_id=33168 Originator: NO Thanks for the patch. I'd like to try to focus on reviewing only new code. How much of this code was copied straight from string, how much was modified and how much was new? There are a bunch of formatting changes. Could you revert those portions and add them to a separate patch (the file can be attached here if you want). Since you are adding a Fini method, I wonder if you ought to add a corresponding Init method. This would init nullbytes and save a comparison on each instantiation of a bytes object. _adjust_indices() looks generic, does this code exist elsewhere/can it be reused? All the methods that take a single argument should use METH_O (e.g., bytes_extend). Shouldn't value in bytes_remove(), bytes_insert(), and possibly elsewhere be limited to 0..255? If the value is > 255, it will traverse the entire bytes, then raise an exception rather than fail early. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 From noreply at sourceforge.net Tue Feb 27 23:23:59 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 27 Feb 2007 14:23:59 -0800 Subject: [Patches] [ python-Patches-1669633 ] methods for bytes Message-ID: Patches item #1669633, was opened at 2007-02-27 03:04 Message generated for change (Comment added) made by shredwheat You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 3000 Status: Closed Resolution: Accepted Priority: 5 Private: No Submitted By: Pete Shinners (shredwheat) Assigned to: Neal Norwitz (nnorwitz) Summary: methods for bytes Initial Comment: This adds the methods for bytes objects as listed in PEP 358. It includes tests for the newly added methods. ---------------------------------------------------------------------- >Comment By: Pete Shinners (shredwheat) Date: 2007-02-27 22:23 Message: Logged In: YES user_id=1076442 Originator: YES File Added: bytestests.diff ---------------------------------------------------------------------- Comment By: Pete Shinners (shredwheat) Date: 2007-02-27 22:23 Message: Logged In: YES user_id=1076442 Originator: YES This patch adds many more tests for bytes. It breaks the common string_tests into a lower level base class that can support bytes and strings. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-27 19:02 Message: Logged In: YES user_id=33168 Originator: NO Thanks Pete! Committed revision 54000. ---------------------------------------------------------------------- Comment By: Pete Shinners (shredwheat) Date: 2007-02-27 16:41 Message: Logged In: YES user_id=1076442 Originator: YES Attached is the updated patch. This should apply clean after the "fromhex" patch that was recently applied. This ignores the previous whitespace cleanup patch. Most of this code was copied from string. Minor modifications were made on these, but no real code changes. The following list of new methods are mostly modelled after list methods. extend, insert, append, reverse, pop, remove This adds an Init method to match the Fini. Also added a comment stating partition is the only dependency of all this init. If partiion is removed from bytes, so should this global stuff. _adjust_indices was copied directly from str. This is not used anywhere else in the codebase, so it should be unique again when str is removed. Added comment about this. Switched new single argument methods to METH_O. The 0-255 range is enforced on insert, append, and remove. Outside results in ValueError. File Added: bytesmethods2.diff ---------------------------------------------------------------------- Comment By: Pete Shinners (shredwheat) Date: 2007-02-27 07:28 Message: Logged In: YES user_id=1076442 Originator: YES This is a split of the whitespacing. This matches the PEP7 style guide for 4-space indentation in C code. File Added: bytesspace.diff ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2007-02-27 05:35 Message: Logged In: YES user_id=33168 Originator: NO Thanks for the patch. I'd like to try to focus on reviewing only new code. How much of this code was copied straight from string, how much was modified and how much was new? There are a bunch of formatting changes. Could you revert those portions and add them to a separate patch (the file can be attached here if you want). Since you are adding a Fini method, I wonder if you ought to add a corresponding Init method. This would init nullbytes and save a comparison on each instantiation of a bytes object. _adjust_indices() looks generic, does this code exist elsewhere/can it be reused? All the methods that take a single argument should use METH_O (e.g., bytes_extend). Shouldn't value in bytes_remove(), bytes_insert(), and possibly elsewhere be limited to 0..255? If the value is > 255, it will traverse the entire bytes, then raise an exception rather than fail early. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1669633&group_id=5470 From noreply at sourceforge.net Wed Feb 28 01:29:17 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 27 Feb 2007 16:29:17 -0800 Subject: [Patches] [ python-Patches-1121142 ] ZipFile.open - read-only file-like obj for files in archive Message-ID: Patches item #1121142, was opened at 2005-02-11 19:08 Message generated for change (Comment added) made by alanmcintyre You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: Python 2.5 Status: Open Resolution: Out of Date Priority: 5 Private: No Submitted By: Alan McIntyre (alanmcintyre) Assigned to: Nobody/Anonymous (nobody) Summary: ZipFile.open - read-only file-like obj for files in archive Initial Comment: I originally started working on updating patch 992750, but decided after a short while to just start from scratch, so I'm posting it as a new patch. Sorry if this isn't appropriate. This patch provides a new open() method on ZipFile; this method returns a file-like object for the requested item in the archive. This file-like object only provides a read() method. ZipFile.read was modified to use the new open method (this was suggested by loewis in reference to patch 992750). The patched zipfile.py passed the existing tests in the test_zipfile.py from CVS. New tests were added to verify the operation of the object returned by open(). These tests were modeled after existing tests for ZipFile.read(); two read fixed-size chunks from the file-like object, and two others read random-sized chunks. I have only run the tests on Windows XP, using Python2.4 from the official Windows installer. I will test the patch out on Linux over the weekend. If the patch is accepted I'll also generate and submit patches for the appropriate documentation as well. ---------------------------------------------------------------------- >Comment By: Alan McIntyre (alanmcintyre) Date: 2007-02-27 19:29 Message: Logged In: YES user_id=1115903 Originator: YES This version of the patch includes the updates to the documentation. It passes regression tests as well as the test_zipfile64 tests on my Gentoo laptop. File Added: zipfile_patch7.tgz ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2007-02-18 13:41 Message: Logged In: YES user_id=1115903 Originator: YES Here is a version of the patch against the current trunk. It passes a "make test" on my Gentoo laptop at the moment, but I think I still need to add some more tests (like reading from encrypted compressed files), and I need to try out the test_zipfile64 stuff. I don't have time today to look through the module Glyph suggested, so it may be that after I do that I'll have some more tweaks to make. I'm pretty sure I'll be able to spend some more time on Wednesday this week. I also included the full contents of zipfile.py and test_zipfile.py just in case that's useful to somebody. File Added: zipfile_patch6.tgz ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2007-02-18 11:44 Message: Logged In: YES user_id=1115903 Originator: YES Thanks, glyph; I'll definitely have a look and see what I can steal. :-) ---------------------------------------------------------------------- Comment By: Glyph Lefkowitz (glyf) Date: 2007-02-16 22:48 Message: Logged In: YES user_id=226807 Originator: NO Twisted also contains an implementation of this functionality, available from http://twistedmatrix.com/trac/browser/trunk/twisted/python/zipstream.py As far as I can tell it doesn't have anything to recommend it over the attached patch, however (in fact, the test coverage of the attached patch looks better), but perhaps it has some behavior which might be desirable to steal. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2007-02-14 12:13 Message: Logged In: YES user_id=1115903 Originator: YES I will see if I can make some progress on this over the weekend. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2007-02-13 04:14 Message: Logged In: YES user_id=21627 Originator: NO Can you please update the patch to the current subversion trunk? I'd like to apply it, but the code of zipfile has changed so that the patch is currently out-of-date. When redoing it, notice that the read implementation has changed (I couldn't figure out how you moved code around). Please do use the trunk (not Python 2.5), and please submit the patch as a single 'svn diff' output (rather than a tar file containing multiple individual diff files). ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-05-30 21:56 Message: Logged In: YES user_id=1115903 Revision 5 of this patch has been in constant use with Python 2.4.1 in an application at my job for about a month; it seems to be stable and useful in that regard. If anybody has time to review and accept (or reject) it I would appreciate it. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-26 21:23 Message: Logged In: YES user_id=1115903 After testing on my large batch of large Zip files, I made one fix (version 4 of the patch didn't read all the content of large compressed archive items). The current set of changes is attached as zipfile_patch5.tgz. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-13 11:34 Message: Logged In: YES user_id=1115903 I found a problem with my universal newline handling code in readline(); if the first byte of an '\r\n' pair was read from file but the second byte didn't come in on that same read, it resulted in the next line having an inappropriate '\n' prepended to it. zipfile_patch4.tgz has a fix for this included (with everything else, of course). I'm going to test the open() capability in a real application with some reasonably large zip files (containing files up to just short of 2GB) over the next few days, so if any bugs or performance problems show up I may have some more changes. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-04-13 00:58 Message: Logged In: YES user_id=1115903 Uploaded the third revision of this patch; passes all regression tests against current CVS on WinXP. I think all the issues Martin brought up have been addressed except perhaps for the case of compression rate <1. I will have a look at that when I have time; just wanted to get an update here before the patch started to look abandoned. :) ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-03-14 09:37 Message: Logged In: YES user_id=1115903 Hmm...I could have sworn I did the diff in the correct order. I'll make sure next time. :) Here's my comments on your remarks (in order): - I'm adding support for universal newlines, and will reject all modes that aren't legal combinations of r, U, and b. - I'll see if I can make a Zip file store something with compression < 1, and if I can I'll add a test case for it. - I'll try work a .flush() in there on the compression object and come up with a test case if possible - .read(0) and .readline(0) will both return an empty string with no side-effects, since this seems to be what builtin files do. Right now ZipExtFile uses the ZipFile's file object, so you pretty much have to do whatever you're going to do with the ZipExtFile instance you get back from .open() before you use that ZipFile instance in a way that moves the file pointer around. I'm sure that somebody will eventually try to use the ZipFile in this way, so it appears my options are either to (1) give the ZipExtFile its own file object to use (when possible), or (2) make sure this limitation is documented. #1 feels (to me) to be the "right thing" to do, so that's what I'll try unless there's a good reason I shouldn't. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2005-03-01 02:59 Message: Logged In: YES user_id=21627 The patch is reversed: usually, diff is invoked as "-c old new". I think it is almost right, but I have a few remarks: - by tradition, open() should have a mode argument, defaulting to 'r'; it would be ok to raise exceptions if it is anything else. However, do consider implementing universal newlines; allowing 'b' as a no-op might also be reasonable. - I wonder what happens if the compression rate is < 1. It would appear that the code might use too few rawbytes. I suggest to recursively invoke read in this case. - I wonder whether it could ever happen that there is still data to uncompress in the zlib object, ie. whether it might be necessary to invoke .flush() after exhausting the rawbytes (and discard the zlib object afterwards) - it appears that atleast the builtin file object implements .read(0) as returning an empty string; the manual says that the entire file is meant only if size is omitted or negative. ---------------------------------------------------------------------- Comment By: Alan McIntyre (alanmcintyre) Date: 2005-02-27 01:28 Message: Logged In: YES user_id=1115903 zipfile_patch2.tgz: I updated the file-like object to support readline, readlines, __iter__ and next(). Added tests for those new methods. Also added a patch for the documentation. Passes regression tests on 2.5a0 built from CVS HEAD with MSVC .NET on Windows XP. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 From noreply at sourceforge.net Wed Feb 28 04:23:25 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Tue, 27 Feb 2007 19:23:25 -0800 Subject: [Patches] [ python-Patches-1607548 ] Optional Argument Syntax Message-ID: Patches item #1607548, was opened at 2006-12-02 20:53 Message generated for change (Comment added) made by tonylownds You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607548&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 3000 Status: Open Resolution: Accepted Priority: 5 Private: No Submitted By: Tony Lownds (tonylownds) Assigned to: Guido van Rossum (gvanrossum) Summary: Optional Argument Syntax Initial Comment: This patch implements optional argument syntax for Python 3000. The patch still has issues; I am posting so that Collin Winters can add a link to the PEP. The syntax implemented is roughly: def f(arg:expr, (nested1:expr, nested2:expr)) -> expr: suite The function object has a new attribute, func_annotations that maps from argument names to the result of the expression. The return annotation is stored with a key of 'return'. Lambda's syntax doesn't support annotations. This patch alters the MAKE_FUNCTION opcode. I have an implementation that built the func_annotations dictionary in bytecode as well but it was bigger and slower. ---------------------------------------------------------------------- >Comment By: Tony Lownds (tonylownds) Date: 2007-02-28 03:23 Message: Logged In: YES user_id=24100 Originator: YES File Added: pydoc.patch ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-02-26 20:58 Message: Logged In: YES user_id=6380 Originator: NO Tony, I am checking make_closure_fix.patch and peepholer_and_max_annotations.patch. Unfortunately, something happened to pydoc.py so that pydoc.patch no longer applies and I don't want to have to think about how to fix it. Can you have a look at this and submit a revised patch? ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-14 20:32 Message: Logged In: YES user_id=24100 Originator: YES File Added: make_closure_fix.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-14 20:31 Message: Logged In: YES user_id=24100 Originator: YES Combines the code paths for MAKE_FUNCTION and MAKE_CLOSURE. Fixes a crash where functions with closures and either annotations or keyword-only arguments result in MAKE_CLOSURE, but only MAKE_FUNCTION has the code to handle annotations or keyword-only arguments. Includes enough tests to trigger the bug. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-06 21:03 Message: Logged In: YES user_id=24100 Originator: YES I tried to implement getargspec() as described, and unfortunately there is another wrinkle to consider. Keyword-only arguments may or may not have defaults. So the invariant described in getargspec()'s docstring can't be maintained when simply appending keyword-only arguments. A tuple of four things is returned: (args, varargs, varkw, defaults). 'args' is a list of the argument names (it may contain nested lists). 'args' will include keyword-only argument names. 'varargs' and 'varkw' are the names of the * and ** arguments or None. 'defaults' is an n-tuple of the default values of the last n arguments. The attached patch adds an 'getfullargspec' API that returns complete information; 'getargspec' raises an error if information would be lost; the order of arguments in 'formatargspec' is backwards compatible, so that formatargspec(*getargspec(f)) == formatargspec(*getfullargspec(f)) when getargspec(f) does not raise an error. PEP 362 could and probably should replace the new getfullargspec() function, so I did not implement an API more complicated than a tuple. File Added: pydoc.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-06 20:05 Message: Logged In: YES user_id=24100 Originator: YES Change peepholer to not bail in the presence of EXTENDED_ARG + MAKE_FUNCTION. Enforce the natural 16-bit limit of annotations in compile.c. File Added: peepholer_and_max_annotations.patch ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-01-04 17:53 Message: Logged In: YES user_id=6380 Originator: NO I like the following approach: (1) the old API continues to work for all functions, but provides incomplete information (not losing the kw-only args completely, but losing the fact that they are kw-only); (2) add a new API that provides all the relevant information. Maybe the new API should not return a 7-tuple but rather a structure with named attributes; that makes it more future-proof. Sorry, I don't have any good suggestions for new names. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-04 07:12 Message: Logged In: YES user_id=24100 Originator: YES For getargs and getargvalues, including the names in positional args is an excellent strategy. There are uses (in cgitb) in the stdlib for getargvalues that then wouldn't need to be changed. The 2 uses of getargspec in the stdlib (one of which I missed, in DocXMLRPCServer) are both closely followed by formatargspec. I think those APIs should change or information will be lost. Alternatively, a new function (hopefully with a better name than getfullargspec :) could be made and getargspec could retain its API, but raise an error when keyword-only arguments are present. def getargspec(func): args, varargs, kwonlyargs, kwdefaults, varkw, defaults, ann = getfullargspec() if kwonlyargs: raise ValueError, "function has keyword-only arguments, use getfullargspec!" return args, varargs, varkw, defaults I'll update the patch to fix getargvalues and DocXMLRPCServer this weekend. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-01-04 05:22 Message: Logged In: YES user_id=6380 Originator: NO Well, it depends on the context whether that matters. The kw-only args could just be included in the positional args (which have names anyway) and that wouldn't be so bad for some apps. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-04 05:17 Message: Logged In: YES user_id=24100 Originator: YES I think everyone should update have to update their uses of getargspec and friends, because otherwise they will silently mis-handle keyword-only arguments. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-01-04 04:30 Message: Logged In: YES user_id=6380 Originator: NO I'm not sure it's right to just change the signature of the various functions in inspect.py; that would break all existing code using that module (and there definitely are other users besides pydoc). It would be better to add new methods that provide access to the additional functionality. Or do you think that everyone will have to change their code anyway? ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-28 06:53 Message: Logged In: YES user_id=33168 Originator: NO I'm skipping the pydoc patch. Didn't even look at it. I don't have the refleak, but I changed some calls and may have fixed it. Committed revision 53170. Leaving open to deal with the pydoc patch. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-28 03:04 Message: Logged In: YES user_id=24100 Originator: YES Nothing else on the C side of things. The pydoc patch works well for me; more tests ought to be added for function annotations and also for keyword-only arguments, but perhaps that can be added on as a later patch after checkin. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-12-28 01:38 Message: Logged In: YES user_id=6380 Originator: NO Thanks! Is there anything else that you think needs to be done before I check this in? The core code looks alright to me; I can't be bothered with reviewing the ast stuff or the compiler package since I don't know enough about these, but given that it compiles things correctly I'm not so worried about those. What's the status of the pydoc patch? Are you still working on that? ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-28 01:28 Message: Logged In: YES user_id=24100 Originator: YES Fixed in latest patch. Also added VISIT call for func_annotations. File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-12-28 00:40 Message: Logged In: YES user_id=6380 Originator: NO I believe I've found a leak in the code that adds annotations to a function object. See this session: >>> x = object() >>> import sys >>> sys.getrefcount(x) 2 >>> for i in range(100): ... def f(x: x): pass ... >>> del f >>> sys.getrefcount(x) 102 >>> At first I thought this could be due to the code added to the MAKE_FUNCTION opcode, but I don't see a leak there. More likely func_annotations is not being freed when a function object is deleted. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-23 19:05 Message: Logged In: YES user_id=24100 Originator: YES Initial patch to implement keyword-only arguments and annotations support for pydoc and inspect. Tests do not exercise these features, yet. Output for annotations that are types is special cased so that for: def intmin(*a: int) -> int: pass ...help(intmin) will display: intmin(*a: int) -> int File Added: pydoc.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-23 15:53 Message: Logged In: YES user_id=24100 Originator: YES Fixed the non-C89 style lines and the formatting (hopefully in compatible style :) File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-12-22 21:41 Message: Logged In: YES user_id=6380 Originator: NO Thanks for the progress! There are still a few lines ending in whitespace or lines that are longer than 80 chars (and weren't before). Mind cleaning those up? Also ceval.c:2305 and compile.c:1440 contain code that gcc 2.95 won't compile (the 'int' declarations ought to be moved to the start of the containing {...} block); I think this style is not C89 compatible. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-22 20:15 Message: Logged In: YES user_id=24100 Originator: YES Changes: 1. Fix crasher in Python/symtable.c -- annotations were visited inside the function scope 2. Fix Lib/compiler issues with Lib/test/test_complex_args. Output from Lib/compiler does not pass all tests, same failures as in HEAD of p3yk branch. File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-21 20:21 Message: Logged In: YES user_id=24100 Originator: YES Changes: 1. Address Neal's comments (I hope) 2. test_scope passes 3. Added some additional tests to test_compiler Open implementation issues: 1. Output from Lib/compiler does not pass test_complex_args, test_scope, possibly more. File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-20 22:13 Message: Logged In: YES user_id=24100 Originator: YES Changes: 1. Updated to apply cleanly 2. Fix to compile.c so that test_complex_args passes Open implementation issues: 1. Neal's comments 2. test_scope fails 3. Output from Lib/compiler does not pass test_complex_args File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-20 18:04 Message: Logged In: YES user_id=24100 Originator: YES I'll work on code formatting and the error checking and other cleanup. Open to other names than tname and vname, I created those non-terminals in order to use the same code for processing "def" and "lambda". Terminals are caps IIUC. I did add a test for the multi-paren situation. 2.5 had that bug too. Re: no changes to ceval, I tried generating the func_annotations dictionary using bytecodes. That doesn't change the ceval loop but was more code and was slower. So there is a way to avoid ceval changes. Re: deciding if lambda was going to require parens around the arguments, I don't think there was any decision, and yes annotations would be easily supportable. Happy to change if there is support, it's backwards incompatible. Re: return type syntax, I have only seen the -> syntax (vs a keyword 'as') on Guido's blog. Thanks for the comments! ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-20 09:25 Message: Logged In: YES user_id=33168 Originator: NO Nix this comment: I would definitely prefer the annotations baked into the code object so there are no changes to ceval. I see that Guido wants it the way it currently is which makes sense for nested functions. There should probably be a test with nested functions even though it really shouldn't be different. The test will verify that. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-20 08:38 Message: Logged In: YES user_id=33168 Originator: NO When regenerating the patch, can you also remove non-functional changes such as removing unneeded parens and whitespace changes. Also, please try to keep the same formatting in the file wrt tabs and spaces and don't move code around. I know this is a pain and inconsistent. I think I changed ast.c to be all 4 space indents with spaces only. In compiler_simple_arg(), don't you need to check if annotation is NULL when returned from ast_for_expr? Otherwise an undetected error would go through, wouldn't it? In compiler_complex_args(), don't you need to set the ast_error (or a SystemError) if the switch isn't a tname, vname, or LPAR? I don't like the names tname and vname. Also they seem inconsistent. Aren't all the other names all CAPS? In hunk, @@ -602,51 +625,75 @@ remove the commented out code. We shouldn't use any // style comments either. Can you improve the error msg for kwdefaults == NULL? (Thanks for adding it!) Check annotation for NULL if returned from ast_for_expr? BTW, the AST code in this area was tricky code which had some bugs. Did you test with adding extra parentheses and singleton tuples? I'm not sure if Guido preferred syntax -> vs a keyword 'as' for the return type. In symtable.c remove the printfs. They should probably be SystemErrors or something. I would definitely prefer the annotations baked into the code object so there are no changes to ceval. Did we decide if lambda was going to require parens around the arguments? If so, it could support annotations, right? (No comment on the usefulness of annotations for lambdas. :-) In compiler_visit_argannotation, you should return the result from PyList_Append and can remove the comment about checking for errors. Also, I believe the INCREF is not needed, it will be done by PyList_Append. Same deal with returning result of compiler_visit_argannotations() (the one with an s). Need to check for PyList_New() returning NULL in compiler_visit_annotations(). Lots more error checking needs to be added in this area. Dammit, I really want to use Mondrian for these comments! (Sorry Tony, not your fault, I'm just having some bad memories at this point cause I have to keep providing the references.) This patch looks very complete in that it updates things like the compiler package and the parsermodule.c. Good job! This is a great start. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-12-20 01:22 Message: Logged In: YES user_id=6380 Originator: NO Applying the patch fails, probably due to recent merge activities in the p3yk branch. Can I inconvenience you with a request to regenerate the patch from the branch head? ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-12-11 17:29 Message: Logged In: YES user_id=764593 Originator: NO Could you rename it to "argument annotations"? "optional argument" makes me think of the current keyword arguments, that can be but don't have to be passed. -jJ ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-04 01:24 Message: Logged In: YES user_id=24100 Originator: YES This patch implements optional argument syntax for Python 3000. The patch still has issues: 1. test_ast and test_scope fail. 2. Running the test suite after compiling the library with the compiler package causes failures 3. no docs 4. C-code reference counts and error checking needs a review The syntax implemented is roughly: def f(arg:expr, (nested1:expr, nested2:expr)) -> expr: suite The function object has a new attribute, func_annotations that maps from argument names to the result of the expression. The return annotation is stored with a key of 'return'. Lambda's syntax doesn't support annotations. The ast format has changed for the builtin compiler and the compiler package. A new token was added, '->' (called RARROW in token.h). token.py lost ERRORTOKEN after re-generating, I don't know why. I added it back manually. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607548&group_id=5470 From noreply at sourceforge.net Wed Feb 28 13:04:10 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 28 Feb 2007 04:04:10 -0800 Subject: [Patches] [ python-Patches-1670765 ] email.Generator: no header wrapping for multipart/signed Message-ID: Patches item #1670765, was opened at 2007-02-28 13:04 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1670765&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Library (Lib) Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Martin von Gagern (gagern) Assigned to: Nobody/Anonymous (nobody) Summary: email.Generator: no header wrapping for multipart/signed Initial Comment: The body of a multipart/signed message has to remain unmodified for the signature to stay intact. Rewrapping headers of nested MIME parts breaks signatures. So I disabled header rewrapping for multipart/signed, by adding a new handler to the email.Generator class. There still remains an issue about leading spaces in header values breaking signatures. The supplied patch is against the latest subversion sources, r54016. Related issues: Python bug #968430 http://tinyurl.com/2973or Mailman bug #815297 http://tinyurl.com/2a8dta ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1670765&group_id=5470 From noreply at sourceforge.net Wed Feb 28 16:41:54 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 28 Feb 2007 07:41:54 -0800 Subject: [Patches] [ python-Patches-1670993 ] Refactor test_threadedtempfile.py to use unittest. Message-ID: Patches item #1670993, was opened at 2007-02-28 08:41 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1670993&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Jerry Seutter (jseutter) Assigned to: Nobody/Anonymous (nobody) Summary: Refactor test_threadedtempfile.py to use unittest. Initial Comment: This patch turns test_threadedtempfile.py into a unittest test file. There should be no changes in what the file tests. Input on this patch is welcome before I go too far on more of the old test files. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1670993&group_id=5470 From noreply at sourceforge.net Wed Feb 28 19:40:03 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 28 Feb 2007 10:40:03 -0800 Subject: [Patches] [ python-Patches-1670993 ] Refactor test_threadedtempfile.py to use unittest. Message-ID: Patches item #1670993, was opened at 2007-02-28 08:41 Message generated for change (Comment added) made by jseutter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1670993&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Jerry Seutter (jseutter) Assigned to: Nobody/Anonymous (nobody) Summary: Refactor test_threadedtempfile.py to use unittest. Initial Comment: This patch turns test_threadedtempfile.py into a unittest test file. There should be no changes in what the file tests. Input on this patch is welcome before I go too far on more of the old test files. ---------------------------------------------------------------------- >Comment By: Jerry Seutter (jseutter) Date: 2007-02-28 11:40 Message: Logged In: YES user_id=1727609 Originator: YES Lib/test/output/test_threadedtempfile should also be removed when this patch is applied. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1670993&group_id=5470 From noreply at sourceforge.net Wed Feb 28 19:56:30 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 28 Feb 2007 10:56:30 -0800 Subject: [Patches] [ python-Patches-1670993 ] Refactor test_threadedtempfile.py to use unittest. Message-ID: Patches item #1670993, was opened at 2007-02-28 08:41 Message generated for change (Settings changed) made by jseutter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1670993&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. >Category: Tests >Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Jerry Seutter (jseutter) Assigned to: Nobody/Anonymous (nobody) Summary: Refactor test_threadedtempfile.py to use unittest. Initial Comment: This patch turns test_threadedtempfile.py into a unittest test file. There should be no changes in what the file tests. Input on this patch is welcome before I go too far on more of the old test files. ---------------------------------------------------------------------- Comment By: Jerry Seutter (jseutter) Date: 2007-02-28 11:40 Message: Logged In: YES user_id=1727609 Originator: YES Lib/test/output/test_threadedtempfile should also be removed when this patch is applied. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1670993&group_id=5470 From noreply at sourceforge.net Wed Feb 28 21:40:51 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 28 Feb 2007 12:40:51 -0800 Subject: [Patches] [ python-Patches-1671208 ] Class Decorators Message-ID: Patches item #1671208, was opened at 2007-02-28 15:40 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1671208&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None Group: None Status: Open Resolution: None Priority: 5 Private: No Submitted By: Jack Diederich (jackdied) Assigned to: Nobody/Anonymous (nobody) Summary: Class Decorators Initial Comment: Patch is against rev 54035 on the pyk3 branch PEP will be on the 3k list shortly. Grammar/Grammar is changed from funcdef: [decorators] 'def' NAME parameters ['->' test] ':' suite to decorated_thing: decorators (classdef | funcdef) funcdef: 'def' NAME parameters ['->' test] ':' suite ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1671208&group_id=5470 From noreply at sourceforge.net Wed Feb 28 21:51:34 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 28 Feb 2007 12:51:34 -0800 Subject: [Patches] [ python-Patches-1671208 ] Class Decorators Message-ID: Patches item #1671208, was opened at 2007-02-28 15:40 Message generated for change (Settings changed) made by jackdied You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1671208&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: None >Group: Python 3000 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Jack Diederich (jackdied) >Assigned to: Jack Diederich (jackdied) Summary: Class Decorators Initial Comment: Patch is against rev 54035 on the pyk3 branch PEP will be on the 3k list shortly. Grammar/Grammar is changed from funcdef: [decorators] 'def' NAME parameters ['->' test] ':' suite to decorated_thing: decorators (classdef | funcdef) funcdef: 'def' NAME parameters ['->' test] ':' suite ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1671208&group_id=5470 From noreply at sourceforge.net Wed Feb 28 23:05:11 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 28 Feb 2007 14:05:11 -0800 Subject: [Patches] [ python-Patches-1607548 ] Optional Argument Syntax Message-ID: Patches item #1607548, was opened at 2006-12-02 15:53 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607548&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 3000 >Status: Closed Resolution: Accepted Priority: 5 Private: No Submitted By: Tony Lownds (tonylownds) Assigned to: Guido van Rossum (gvanrossum) Summary: Optional Argument Syntax Initial Comment: This patch implements optional argument syntax for Python 3000. The patch still has issues; I am posting so that Collin Winters can add a link to the PEP. The syntax implemented is roughly: def f(arg:expr, (nested1:expr, nested2:expr)) -> expr: suite The function object has a new attribute, func_annotations that maps from argument names to the result of the expression. The return annotation is stored with a key of 'return'. Lambda's syntax doesn't support annotations. This patch alters the MAKE_FUNCTION opcode. I have an implementation that built the func_annotations dictionary in bytecode as well but it was bigger and slower. ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2007-02-28 17:05 Message: Logged In: YES user_id=6380 Originator: NO Thanks! Committed as r54043. If you ever feel the urge to add unittests for the new functions, just mail them to me, as I'm closing this patch. ;-) ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-02-27 22:23 Message: Logged In: YES user_id=24100 Originator: YES File Added: pydoc.patch ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-02-26 15:58 Message: Logged In: YES user_id=6380 Originator: NO Tony, I am checking make_closure_fix.patch and peepholer_and_max_annotations.patch. Unfortunately, something happened to pydoc.py so that pydoc.patch no longer applies and I don't want to have to think about how to fix it. Can you have a look at this and submit a revised patch? ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-14 15:32 Message: Logged In: YES user_id=24100 Originator: YES File Added: make_closure_fix.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-14 15:31 Message: Logged In: YES user_id=24100 Originator: YES Combines the code paths for MAKE_FUNCTION and MAKE_CLOSURE. Fixes a crash where functions with closures and either annotations or keyword-only arguments result in MAKE_CLOSURE, but only MAKE_FUNCTION has the code to handle annotations or keyword-only arguments. Includes enough tests to trigger the bug. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-06 16:03 Message: Logged In: YES user_id=24100 Originator: YES I tried to implement getargspec() as described, and unfortunately there is another wrinkle to consider. Keyword-only arguments may or may not have defaults. So the invariant described in getargspec()'s docstring can't be maintained when simply appending keyword-only arguments. A tuple of four things is returned: (args, varargs, varkw, defaults). 'args' is a list of the argument names (it may contain nested lists). 'args' will include keyword-only argument names. 'varargs' and 'varkw' are the names of the * and ** arguments or None. 'defaults' is an n-tuple of the default values of the last n arguments. The attached patch adds an 'getfullargspec' API that returns complete information; 'getargspec' raises an error if information would be lost; the order of arguments in 'formatargspec' is backwards compatible, so that formatargspec(*getargspec(f)) == formatargspec(*getfullargspec(f)) when getargspec(f) does not raise an error. PEP 362 could and probably should replace the new getfullargspec() function, so I did not implement an API more complicated than a tuple. File Added: pydoc.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-06 15:05 Message: Logged In: YES user_id=24100 Originator: YES Change peepholer to not bail in the presence of EXTENDED_ARG + MAKE_FUNCTION. Enforce the natural 16-bit limit of annotations in compile.c. File Added: peepholer_and_max_annotations.patch ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-01-04 12:53 Message: Logged In: YES user_id=6380 Originator: NO I like the following approach: (1) the old API continues to work for all functions, but provides incomplete information (not losing the kw-only args completely, but losing the fact that they are kw-only); (2) add a new API that provides all the relevant information. Maybe the new API should not return a 7-tuple but rather a structure with named attributes; that makes it more future-proof. Sorry, I don't have any good suggestions for new names. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-04 02:12 Message: Logged In: YES user_id=24100 Originator: YES For getargs and getargvalues, including the names in positional args is an excellent strategy. There are uses (in cgitb) in the stdlib for getargvalues that then wouldn't need to be changed. The 2 uses of getargspec in the stdlib (one of which I missed, in DocXMLRPCServer) are both closely followed by formatargspec. I think those APIs should change or information will be lost. Alternatively, a new function (hopefully with a better name than getfullargspec :) could be made and getargspec could retain its API, but raise an error when keyword-only arguments are present. def getargspec(func): args, varargs, kwonlyargs, kwdefaults, varkw, defaults, ann = getfullargspec() if kwonlyargs: raise ValueError, "function has keyword-only arguments, use getfullargspec!" return args, varargs, varkw, defaults I'll update the patch to fix getargvalues and DocXMLRPCServer this weekend. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-01-04 00:22 Message: Logged In: YES user_id=6380 Originator: NO Well, it depends on the context whether that matters. The kw-only args could just be included in the positional args (which have names anyway) and that wouldn't be so bad for some apps. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2007-01-04 00:17 Message: Logged In: YES user_id=24100 Originator: YES I think everyone should update have to update their uses of getargspec and friends, because otherwise they will silently mis-handle keyword-only arguments. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-01-03 23:30 Message: Logged In: YES user_id=6380 Originator: NO I'm not sure it's right to just change the signature of the various functions in inspect.py; that would break all existing code using that module (and there definitely are other users besides pydoc). It would be better to add new methods that provide access to the additional functionality. Or do you think that everyone will have to change their code anyway? ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-28 01:53 Message: Logged In: YES user_id=33168 Originator: NO I'm skipping the pydoc patch. Didn't even look at it. I don't have the refleak, but I changed some calls and may have fixed it. Committed revision 53170. Leaving open to deal with the pydoc patch. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-27 22:04 Message: Logged In: YES user_id=24100 Originator: YES Nothing else on the C side of things. The pydoc patch works well for me; more tests ought to be added for function annotations and also for keyword-only arguments, but perhaps that can be added on as a later patch after checkin. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-12-27 20:38 Message: Logged In: YES user_id=6380 Originator: NO Thanks! Is there anything else that you think needs to be done before I check this in? The core code looks alright to me; I can't be bothered with reviewing the ast stuff or the compiler package since I don't know enough about these, but given that it compiles things correctly I'm not so worried about those. What's the status of the pydoc patch? Are you still working on that? ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-27 20:28 Message: Logged In: YES user_id=24100 Originator: YES Fixed in latest patch. Also added VISIT call for func_annotations. File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-12-27 19:40 Message: Logged In: YES user_id=6380 Originator: NO I believe I've found a leak in the code that adds annotations to a function object. See this session: >>> x = object() >>> import sys >>> sys.getrefcount(x) 2 >>> for i in range(100): ... def f(x: x): pass ... >>> del f >>> sys.getrefcount(x) 102 >>> At first I thought this could be due to the code added to the MAKE_FUNCTION opcode, but I don't see a leak there. More likely func_annotations is not being freed when a function object is deleted. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-23 14:05 Message: Logged In: YES user_id=24100 Originator: YES Initial patch to implement keyword-only arguments and annotations support for pydoc and inspect. Tests do not exercise these features, yet. Output for annotations that are types is special cased so that for: def intmin(*a: int) -> int: pass ...help(intmin) will display: intmin(*a: int) -> int File Added: pydoc.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-23 10:53 Message: Logged In: YES user_id=24100 Originator: YES Fixed the non-C89 style lines and the formatting (hopefully in compatible style :) File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-12-22 16:41 Message: Logged In: YES user_id=6380 Originator: NO Thanks for the progress! There are still a few lines ending in whitespace or lines that are longer than 80 chars (and weren't before). Mind cleaning those up? Also ceval.c:2305 and compile.c:1440 contain code that gcc 2.95 won't compile (the 'int' declarations ought to be moved to the start of the containing {...} block); I think this style is not C89 compatible. ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-22 15:15 Message: Logged In: YES user_id=24100 Originator: YES Changes: 1. Fix crasher in Python/symtable.c -- annotations were visited inside the function scope 2. Fix Lib/compiler issues with Lib/test/test_complex_args. Output from Lib/compiler does not pass all tests, same failures as in HEAD of p3yk branch. File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-21 15:21 Message: Logged In: YES user_id=24100 Originator: YES Changes: 1. Address Neal's comments (I hope) 2. test_scope passes 3. Added some additional tests to test_compiler Open implementation issues: 1. Output from Lib/compiler does not pass test_complex_args, test_scope, possibly more. File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-20 17:13 Message: Logged In: YES user_id=24100 Originator: YES Changes: 1. Updated to apply cleanly 2. Fix to compile.c so that test_complex_args passes Open implementation issues: 1. Neal's comments 2. test_scope fails 3. Output from Lib/compiler does not pass test_complex_args File Added: opt_arg_ann.patch ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-20 13:04 Message: Logged In: YES user_id=24100 Originator: YES I'll work on code formatting and the error checking and other cleanup. Open to other names than tname and vname, I created those non-terminals in order to use the same code for processing "def" and "lambda". Terminals are caps IIUC. I did add a test for the multi-paren situation. 2.5 had that bug too. Re: no changes to ceval, I tried generating the func_annotations dictionary using bytecodes. That doesn't change the ceval loop but was more code and was slower. So there is a way to avoid ceval changes. Re: deciding if lambda was going to require parens around the arguments, I don't think there was any decision, and yes annotations would be easily supportable. Happy to change if there is support, it's backwards incompatible. Re: return type syntax, I have only seen the -> syntax (vs a keyword 'as') on Guido's blog. Thanks for the comments! ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-20 04:25 Message: Logged In: YES user_id=33168 Originator: NO Nix this comment: I would definitely prefer the annotations baked into the code object so there are no changes to ceval. I see that Guido wants it the way it currently is which makes sense for nested functions. There should probably be a test with nested functions even though it really shouldn't be different. The test will verify that. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2006-12-20 03:38 Message: Logged In: YES user_id=33168 Originator: NO When regenerating the patch, can you also remove non-functional changes such as removing unneeded parens and whitespace changes. Also, please try to keep the same formatting in the file wrt tabs and spaces and don't move code around. I know this is a pain and inconsistent. I think I changed ast.c to be all 4 space indents with spaces only. In compiler_simple_arg(), don't you need to check if annotation is NULL when returned from ast_for_expr? Otherwise an undetected error would go through, wouldn't it? In compiler_complex_args(), don't you need to set the ast_error (or a SystemError) if the switch isn't a tname, vname, or LPAR? I don't like the names tname and vname. Also they seem inconsistent. Aren't all the other names all CAPS? In hunk, @@ -602,51 +625,75 @@ remove the commented out code. We shouldn't use any // style comments either. Can you improve the error msg for kwdefaults == NULL? (Thanks for adding it!) Check annotation for NULL if returned from ast_for_expr? BTW, the AST code in this area was tricky code which had some bugs. Did you test with adding extra parentheses and singleton tuples? I'm not sure if Guido preferred syntax -> vs a keyword 'as' for the return type. In symtable.c remove the printfs. They should probably be SystemErrors or something. I would definitely prefer the annotations baked into the code object so there are no changes to ceval. Did we decide if lambda was going to require parens around the arguments? If so, it could support annotations, right? (No comment on the usefulness of annotations for lambdas. :-) In compiler_visit_argannotation, you should return the result from PyList_Append and can remove the comment about checking for errors. Also, I believe the INCREF is not needed, it will be done by PyList_Append. Same deal with returning result of compiler_visit_argannotations() (the one with an s). Need to check for PyList_New() returning NULL in compiler_visit_annotations(). Lots more error checking needs to be added in this area. Dammit, I really want to use Mondrian for these comments! (Sorry Tony, not your fault, I'm just having some bad memories at this point cause I have to keep providing the references.) This patch looks very complete in that it updates things like the compiler package and the parsermodule.c. Good job! This is a great start. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2006-12-19 20:22 Message: Logged In: YES user_id=6380 Originator: NO Applying the patch fails, probably due to recent merge activities in the p3yk branch. Can I inconvenience you with a request to regenerate the patch from the branch head? ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2006-12-11 12:29 Message: Logged In: YES user_id=764593 Originator: NO Could you rename it to "argument annotations"? "optional argument" makes me think of the current keyword arguments, that can be but don't have to be passed. -jJ ---------------------------------------------------------------------- Comment By: Tony Lownds (tonylownds) Date: 2006-12-03 20:24 Message: Logged In: YES user_id=24100 Originator: YES This patch implements optional argument syntax for Python 3000. The patch still has issues: 1. test_ast and test_scope fail. 2. Running the test suite after compiling the library with the compiler package causes failures 3. no docs 4. C-code reference counts and error checking needs a review The syntax implemented is roughly: def f(arg:expr, (nested1:expr, nested2:expr)) -> expr: suite The function object has a new attribute, func_annotations that maps from argument names to the result of the expression. The return annotation is stored with a key of 'return'. Lambda's syntax doesn't support annotations. The ast format has changed for the builtin compiler and the compiler package. A new token was added, '->' (called RARROW in token.h). token.py lost ERRORTOKEN after re-generating, I don't know why. I added it back manually. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1607548&group_id=5470 From noreply at sourceforge.net Wed Feb 28 23:06:24 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 28 Feb 2007 14:06:24 -0800 Subject: [Patches] [ python-Patches-1629305 ] The Unicode "lazy strings" patches Message-ID: Patches item #1629305, was opened at 2007-01-06 04:37 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1629305&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Core (C code) Group: Python 3000 >Status: Closed >Resolution: Rejected Priority: 5 Private: No Submitted By: Larry Hastings (lhastings) Assigned to: Nobody/Anonymous (nobody) Summary: The Unicode "lazy strings" patches Initial Comment: These are patches to add lazy processing to Unicode strings for Python 3000. I plan to post separate patches for both "lazy concatenation" and "lazy slices", as I suspect "lazy concatenation" has a much higher chance of being accepted. There is a long discussion about "lazy concatenation" here: http://mail.python.org/pipermail/python-dev/2006-October/069224.html And another long discussion about "lazy slices" here: http://mail.python.org/pipermail/python-dev/2006-October/069506.html Note that, unlike the 8-bit-character strings patches, I don't expect the "lazy slices" patch to be dependent on the "lazy concatenation" patch. Unicode objects are stored differently, and already use a pointer to a separately-allocated buffer. This was the big (and mildly controversial) change made by the 8-bit-character "lazy concatenation" patch, and "lazy slices" needed it too. Since Unicode objects already look like that, the Unicode lazy patches should be independent. ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2007-02-28 17:06 Message: Logged In: YES user_id=6380 Originator: NO Looks like I forgot to close and reject this. ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-20 04:16 Message: Logged In: YES user_id=364875 Originator: YES Whoops, sorry. I refreshed a summary page I had lying around, which I guess re-posted the comment! Didn't mean to spam you with extra updates. ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-20 04:14 Message: Logged In: YES user_id=364875 Originator: YES As discussed (briefly) over email, I'm moving this discussion back to the Python-3000 mailing list. But before I do I wanted to clear up something from your reply. "lazy concatenation" and "lazy slices" are really two patches, filed under the "lazy slices" penumbra. They are different optimizations, with different implementations and different behaviors. I implemented them cumulatively to save work because they intertwine when merged, but I had hoped they would be considered independently. I apologize if this point was unclear (and moreso if it was a bad idea). My reason for doing so: I suspected "lazy slices" were doomed from the start; doing the patch this way meant wasting less work. One downside of "lazy slices" is their ability to waste loads of memory in the worst-case. Now, "lazy concatenation" simply doesn't have that problem. Yet the fourth and fifth paragraphs of your most recent reply imply you think it can. A quick recap of lazy concatenation: a = u"a" b = u"b" concat = a + b "concat" is a PyUnicodeConcatenationObject holding references to a and b (or rather their values). Its "value" is NULL, indicating that it is unrendered. The moment someone asks for the value of "concat", the object allocates space for its value, constructs the value by walking its tree of children, and frees its children. The implementation is heavily optimized for the general case (concatenation) and avoids recursion where possible. The worst-case memory consumption behavior of lazy concatenation is adding lots and lots of tiny strings and never rendering; that will allocate lots of PyUnicodeConcatenationObjects. But it's nowhere near as bad as a short lazy slice of a long string. Does that make "lazy concatenation" more palatable? ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-15 13:54 Message: Logged In: YES user_id=364875 Originator: YES As discussed (briefly) over email, I'm moving this discussion back to the Python-3000 mailing list. But before I do I wanted to clear up something from your reply. "lazy concatenation" and "lazy slices" are really two patches, filed under the "lazy slices" penumbra. They are different optimizations, with different implementations and different behaviors. I implemented them cumulatively to save work because they intertwine when merged, but I had hoped they would be considered independently. I apologize if this point was unclear (and moreso if it was a bad idea). My reason for doing so: I suspected "lazy slices" were doomed from the start; doing the patch this way meant wasting less work. One downside of "lazy slices" is their ability to waste loads of memory in the worst-case. Now, "lazy concatenation" simply doesn't have that problem. Yet the fourth and fifth paragraphs of your most recent reply imply you think it can. A quick recap of lazy concatenation: a = u"a" b = u"b" concat = a + b "concat" is a PyUnicodeConcatenationObject holding references to a and b (or rather their values). Its "value" is NULL, indicating that it is unrendered. The moment someone asks for the value of "concat", the object allocates space for its value, constructs the value by walking its tree of children, and frees its children. The implementation is heavily optimized for the general case (concatenation) and avoids recursion where possible. The worst-case memory consumption behavior of lazy concatenation is adding lots and lots of tiny strings and never rendering; that will allocate lots of PyUnicodeConcatenationObjects. But it's nowhere near as bad as a short lazy slice of a long string. Does that make "lazy concatenation" more palatable? ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-15 13:53 Message: Logged In: YES user_id=364875 Originator: YES As discussed (briefly) over email, I'm moving this discussion back to the Python-3000 mailing list. But before I do I wanted to clear up something from your reply. "lazy concatenation" and "lazy slices" are really two patches, filed under the "lazy slices" penumbra. They are different optimizations, with different implementations and different behaviors. I implemented them cumulatively to save work because they intertwine when merged, but I had hoped they would be considered independently. I apologize if this point was unclear (and moreso if it was a bad idea). My reason for doing so: I suspected "lazy slices" were doomed from the start; doing the patch this way meant wasting less work. One downside of "lazy slices" is their ability to waste loads of memory in the worst-case. Now, "lazy concatenation" simply doesn't have that problem. Yet the fourth and fifth paragraphs of your most recent reply imply you think it can. A quick recap of lazy concatenation: a = u"a" b = u"b" concat = a + b "concat" is a PyUnicodeConcatenationObject holding references to a and b (or rather their values). Its "value" is NULL, indicating that it is unrendered. The moment someone asks for the value of "concat", the object allocates space for its value, constructs the value by walking its tree of children, and frees its children. The implementation is heavily optimized for the general case (concatenation) and avoids recursion where possible. The worst-case memory consumption behavior of lazy concatenation is adding lots and lots of tiny strings and never rendering; that will allocate lots of PyUnicodeConcatenationObjects. But it's nowhere near as bad as a short lazy slice of a long string. Does that make "lazy concatenation" more palatable? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-01-14 11:32 Message: Logged In: YES user_id=6380 Originator: NO Sorry, the test_array failure was due to not rebuilding after patching. Because extension modules are built using distutils, they don't get automatically rebuilt when a relevant header has changed. "grind to a halt": swapping, probably, due to memory filling up with 1M-character string objects, as you experienced yourself. Your proposal takes the edge off, although I can still come up with a worst-case scenario (just use 64K strings instead of 1M strings, and leave the rest the same). I am far from convinced that replacing one pathological case (O(N**2) concatenation, which is easily explained and avoided) with another (which is harder to explain due to the more complicated algorithms and heuristics involved) is a good trade-off. This is all the worse since your optimization doesn't have a clear time/space trade-off: it mostly attempts to preserve time *and* space, but in the worst case it can *waste* space. (And I'm not convinced there can't be a pathological case where it is slower, too.) And the gains are dependent on the ability to *avoid* ultimately rendering the string; if every string ends up being rendered, there is no net gain in space, and there might be no net gain in time either (at least not for slices). I believe I would rather not pursue this patch further at this time; a far more important programming task is the str/unicode unification (now that the int/long unification is mostly there). If you want to clean up the patch, I suggest that you add a large comment section somewhere (unicode.h?) describing the algorithms in a lot of detail, including edge cases and performance analysis, to make review of the code possible. But you're most welcome to withdraw it, too; it would save me a lot of headaches. ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-14 06:44 Message: Logged In: YES user_id=364875 Originator: YES Here's another possible fix for the worst-case scenario: #define MAX_SLICE_DELTA (64*1024) if ( ((size_of_slice + MAX_SLICE_DELTA) > size_of_original) || (size_of_slice > (size_of_original / 2)) ) use_lazy_slice(); else create_string_as_normal(); You'd still get the full benefit of lazy slices most of the time, but it takes the edge off the really pathological cases. How's that? ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-14 05:42 Message: Logged In: YES user_id=364875 Originator: YES Thanks for taking the time! > - Style: you set your tab stops to 4 spaces. That is an absolute > no-no! Sorry about that; I'll fix it if I resubmit. > - Segfault in test_array. It seems that it's receiving a unicode > slice object and treating it like a "classic" unicode object. I tested on Windows and Linux, and I haven't seen that behavior. Which test_array, by the way? In Lib/test, or Lib/ctypes/test? I'm having trouble with most of the DLL extensions on Windows; they complain that the module uses the incompatible python26.dll or python26_d.dll. So I haven't tested ctypes/test_array.py on Windows, but I have tested the other three permutations of Linux vs Windows and Lib/test/test_array vs Lib/ctypes/test/test_array. Can you give me a stack trace to the segfault? With that I bet I can fix it even without a reproducible test case. > - I got it to come to a grinding halt with the following worst-case > scenario: > > a = [] > while True: > x = u"x"*1000000 > x = x[30:60] # Short slice of long string > a.append(x) > > If you can't do better than that, I'll have to reject it. > > PS I used your combined patch, if it matters. It matters. The combined patch has "lazy slices", the other patch does not. When you say "grind to a halt" I'm not sure what you mean. Was it thrashing? How much CPU was it using? When I ran that test, my Windows computer got to 1035 iterations then threw a MemoryError. My Linux box behaved the same, except it got to 1605 iterations. Adding a call to .simplify() on the slice defeats this worst-case scenario: a = [] while True: x = u"x"*1000000 x = x[30:60].simplify() # Short slice of long string a.append(x) .simplify() forces lazy strings to render themselves. With that change, this test will run until the cows come home. Is that acceptable? Failing that, is there any sort of last-ditch garbage collection pass that gets called when a memory allocation fails but before it returns NULL? If so, I could hook in to that and try to render some slices. (I don't see such a pass, but maybe I missed it.) Failing that, I could add garbage-collect-and-retry-once logic to memory allocation myself, either just for unicodeobject.c or as a global change. But I'd be shocked if you were interested in that approach; if Python doesn't have such a thing by now, you probably don't want it. And failing that, "lazy slices" are probably toast. It always was a tradeoff of speed for worst-case memory use, and I always knew it might not fly. If that's the case, please take a look at the other patch, and in the meantime I'll see if anyone can come up with other ways to mitigate the worst-case scenario. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2007-01-13 18:59 Message: Logged In: YES user_id=6380 Originator: NO Problems so far: - Style: you set your tab stops to 4 spaces. That is an absolute no-no! You can indent using 4 spaces, but you should NEVER assume that a TAB character is anything except 8 spaces. - Segfault in test_array. It seems that it's receiving a unicode slice object and treating it like a "classic" unicode object. - I got it to come to a grinding halt with the following worst-case scenario: a = [] while True: x = u"x"*1000000 x = x[30:60] # Short slice of long string a.append(x) If you can't do better than that, I'll have to reject it. PS I used your combined patch, if it matters. ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-12 19:03 Message: Logged In: YES user_id=364875 Originator: YES File Added: pybench.first.results.zip ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-12 12:57 Message: Logged In: YES user_id=364875 Originator: YES josiahcarlson: I think you misunderstood options 2 and 3. The empty string (option 2) or nonempty but fixed size string (option 3) would *only* be returned in the event of an allocation failure, aka "the process is out of memory". Since it's out of memory yet trying to allocate more, it has *already* failed. My goal in proposing options 2 and 3 was that, when this happens (and it eventually will), Python would fail *gracefully* with an exception, rather than *miserably* with a bus error. As for writing a wrapper, I'm just not interested. I'm a strong believer in "There should be one--and preferably only one--obvious way to do it", and I feel a special-purpose wrapper class for good string performance adds mental clutter. The obvious way to do string concatenation is with "+"; the obvious way to to string slices is with "[:]". My goal is to make those fast so that you can use them *everywhere*--even in performance-critical code. I don't want a wrapper class, and have no interest in contributing to one. For what it's worth, I came up with a fifth approach this morning while posting to the Python-3000 mailing list: pre-allocate the str buffer, updating it to the correct size whenever the lazy object changes size. That would certainly fix the problem; the error would occur in a much more reportable place. But it would also slow down the code quite a lot, negating many of the speed gains of this approach. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-01-12 01:55 Message: Logged In: YES user_id=341410 Originator: NO I don't think that changing the possible return of PyUnicode_AS_UNICODE is reasonable. (option 1) Option 2 breaks the buffer interface. Option 3 severely limits the size of potential unicode strings. If you are only manipulating tiny unicode strings (8k?), then the effect of fast concatenation, slicing, etc., isn't terribly significant. Option 4 is possible, but I know I would feel bad if all of this work went to waste. Note what M. A. Lemburg mentioned. The functionality is useful, it's the polymorphic representation that is the issue. Rather than attempting to change the unicode representation, what about a wrapper type? Keep the base unicode representation simple (both Guido and M. A. have talked about this). Guido has also stated that he wouldn't be against views (slicing and/or concatenation) if they could be shown to have real use-cases. The use-cases you have offered here are still applicable, and because it wouldn't necessitate a (not insignificant) change in semantics and 3rd party code, would make it acceptable. ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-11 23:32 Message: Logged In: YES user_id=364875 Originator: YES Just fixed the build under Linux--sorry, should have done that before posting the original patch. Patches now built and tested under Win32 and Linux, and produce the same output as an unpatched py3k trunk. lemburg: A minor correction: the full "lazy strings" patch (with "lazy slices") also touches "stringlib/partition.h", "stringlib/readme.txt", and "Objects/stringobject.c", in addition to the two unicodeobject.* files. The changes to these three files are minuscule, and don't affect their maintainability, so the gist of my statements still hold. (Besides, all three of those files will probably go away before Py3k ships.) File Added: lch.py3k.unicode.lazy.slice.and.concat.patch.53392.txt ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-11 23:25 Message: Logged In: YES user_id=364875 Originator: YES File Added: lch.py3k.unicode.lazy.concat.patch.53392.txt ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-11 22:12 Message: Logged In: YES user_id=364875 Originator: YES Attached below you will find the full "lazy strings" patch, which has both "lazy concatenation" and "lazy slices". The diff is against the current revision of the Py3k branch, #53392. On my machine (Win32) rt.bat produces identical output before and after the patch, for both debug and release builds. As I mentioned in a previous comment, you can read the description (and ensuing conversation) about "lazy slices" here: http://mail.python.org/pipermail/python-dev/2006-October/069506.html One new feature of this version: I added a method on a Unicode string, s.simplify(), which forces the string to "render" if it's one of my exotic string subtypes (a lazy concatenation or lazy slice). My goal is to assuage fears about pathological memory-use cases where you have long-lived tiny slices of gigantic strings. If you realize you're having that problem, simply add calls to .simplify() on the slices and the problem should go away. As for the semantics of .simplify(), it returns a reference to the string s. Honestly I wasn't sure whether it should return a new string or just monkey with the existing string. Really, rendering doesn't change the string; it's the same string, with the exact same external behavior, just with different bits floating around underneath. For now it monkeys with the existing string, as that seemed best. (But I'd be happy to switch it to returning a new string if it'd help.) I had planned to make the "lazy slices" patch independent of the "lazy concatenation" patch. However, it wound up being a bigger pain that I thought, and anyway I figure the likelyhood that "lazy slices" would be accepted and "lazy concatenation" would not is effectively zero. So I didn't bother. If there's genuine interest in "lazy slices" without "lazy concatenation", I can produce such a thing. File Added: lch.py3k.unicode.lazy.slice.and.concat.patch.53392.txt ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-11 21:50 Message: Logged In: YES user_id=364875 Originator: YES File Added: lch.py3k.unicode.lazy.concat.patch.53392.txt ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-11 21:42 Message: Logged In: YES user_id=364875 Originator: YES lemburg: You're right, the possibility of PyUnicode_AS_UNICODE() returning NULL is new behavior, and this could conceivably result in crashes. To be clear: NULL return values will only happen when allocation of the final "str" buffer fails during lazy rendering. This will only happen in out-of-memory conditions; for right now, while the patch is under early review, I suspect that's okay. So far I've come up with four possible ways to resolve this problem, which I will list here from least-likely to most-likely: 1. Redefine the API such that PyUnicode_AS_UNICODE() is allowed to return NULL, and fix every place in the Python source tree that calls it to check for a NULL return. Document this with strong language for external C module authors. 2. Change the length to 0 and return a constant empty string. Suggest that users of the Unicode API ask for the pointer *first* and the length *second*. 3. Change the length to 0 and return a previously-allocated buffer of some hopefully-big-enough-size (4096 bytes? 8192 bytes?), such that even if the caller iterates over the buffer, odds are good they'll stop before they hit the end. Again, suggest that users of the Unicode API ask for the pointer *first* and the length *second*. 4. The patch is not accepted. Of course, I'm open to suggestions of other approaches. (Not to mention patches!) Regarding your memory usage and "slice integers" comments, perhaps you'll be interested in the full lazy patch, which I hope to post later today. "Lazy concatenation" is only one of the features of the full patch; the other is "lazy slices". For a full description of my "lazy slices" implementation, see this posting (and the subsequent conversation) to Python-Dev: http://mail.python.org/pipermail/python-dev/2006-October/069506.html And yes, lazy slices suffer from the same possible-NULL-return-from-PyUnicode_AS_UNICODE() problem that lazy concatenation does. As for your final statement, I never claimed that this was a particularly clean design. I merely claim it makes things faster and is (so far) self-contained. For the Unicode versions of my lazy strings patches, the only files I touched were "Include/unicodeobject.h" and "Objects/unicodeobject.c". I freely admit my patch makes those files *even fussier* to work on than they already are. But if you don't touch those files, you won't notice the difference*, and the patch makes some Python string operations faster without making anything else slower. At the very least I suggest the patches are worthy of examination. * Barring API changes to rectify the possible NULL return from PyUnicode_AS_UNICODE() problem, that is. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2007-01-10 15:59 Message: Logged In: YES user_id=38388 Originator: NO Larry, I probably wasn't clear enough: PyUnicode_AS_UNICODE() returns a pointer to the underlying Py_UNICODE buffer. No API using this macro checks for a NULL return value of the macro since a Unicode object is guaranteed to have a non-NULL Py_UNICODE buffer. As a result, a memory caused during the concatenation process cannot be passed back up the call stack. The NULL return value would result in a plain segfault in the calling API. Regarding the tradeoff and trying such an approach: I've done such tests myself (not with Unicode but with 8-bit strings) and it didn't pay off. The memory consumption outweighs the performance you gain by using the 'x += y' approach. The ''.join(list) approach also doesn't really help if you're after performance (for much the same reasons). In mxTextTools I used slice integers pointing into the original parsed string to work around these problems, which works great and avoids creating short strings altogether (so you gain speed and memory). A patch I would find a lot more useful is one to create a Unicode alternative to cStringIO - for strings, this is by far the most performant way of creating a larger string from lots of small pieces. To complement this, a smart slice type might also be an attractive target; one that breaks up a larger string into slices and provides operations on these, including joining them to form a new string. I'm not convinced that murking with the underlying object type and doing "subtyping" on-the-fly is a clean design. ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-10 15:30 Message: Logged In: YES user_id=364875 Originator: YES Much of what I do in Python is text processing. My largest Python project to date was an IDL which spewed out loads of text; I've also written an HTML formatter or two. I seem to do an awful lot of string concatenation in Python, and I'd like it to be fast. I'm not alone in this, as there have been several patches to Python in recent years to speed up string concatenation. Perhaps you aren't familiar with my original justification for the patch. I've always hated the "".join() idiom for string concatenation, as it violates the "There should be one--and preferably only one--obvious way to do it" principle (and arguably others). With lazy concatenation, the obvious way (using +) becomes competitive with "".join(), thus dispensing with the need for this inobvious and distracting idiom. For a more thorough dissection of the (original) patch, including its implementation and lots of discussion from other people, please see the original thread on c.l.p: http://groups.google.com/group/comp.lang.python/browse_frm/thread/b8a8f20bc3c81bcf Please ignore the benchmarks there, as they were quite flawed. And, no, I haven't seen a lot of code manipulating Unicode strings yet, but then I'm not a Python shaker-and-mover. Obviously I expect to see a whole lot more when Py3k is adopted. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-01-10 13:24 Message: Logged In: YES user_id=341410 Originator: NO >From what I understand, the point of the lazy strings patch is to make certain operations faster. What operations? Generally speaking, looped concatenation (x += y), and other looping operations that have traditionally been slow; O(n^2). While this error is still common among new users of Python, generally users only get bit once. They ask about it on python-list and are told: z = []; z.append(y); x = ''.join(z) . Then again, the only place where I've seen the iterative building up of *text* is really in document reformatting (like textwrap). Basically all other use-cases (that I have seen) generally involve the manipulation of binary data. Larry, out of curiosity, have you found code out there that currently loops and concatenates unicode? ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-08 20:26 Message: Logged In: YES user_id=364875 Originator: YES Continuing the comedy of errors, concat patch #2 was actually the same as #1, it didn't have the fix for detecting a NULL return of PyMem_NEW(). Fixed in concat patch #3. (Deleting concat patch #2.) File Added: lch.py3k.unicode.lazy.concat.patch.3.txt ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-08 20:10 Message: Logged In: YES user_id=364875 Originator: YES Revised the lazy concatenation patch to add (doh!) a check for when PyMem_NEW() fails in PyUnicode_AsUnicode(). File Added: lch.py3k.unicode.lazy.concat.patch.2.txt ---------------------------------------------------------------------- Comment By: Larry Hastings (lhastings) Date: 2007-01-08 13:50 Message: Logged In: YES user_id=364875 Originator: YES jcarlson: The first time someone calls PyUnicode_AsUnicode() on a concatenation object, it renders the string, and that's an O(something) operation. In general this rendering is O(i), aka linear time, though linear related to *what* depends. (It iterates over the m concatenated strings, and each of the n characters in those strings, and whether n or m is more important depends on their values.) After rendering, the object behaves like any other Unicode string, including O(1) for array element lookup. If you're referring to GvR's statement "I mention performance because s[i] should remain an O(1) operation.", here: http://mail.python.org/pipermail/python-3000/2006-December/005281.html I suspect this refers to the UCS-2 vs. UTF-16 debate. lemberg: Your criticisms are fair; lazy evaluation is a tradeoff. In general my response to theories about how it will affect performance is "I invite you to try it and see". As for causing memory errors, the only problem I see is not checking for a NULL return from PyMem_NEW() in PyUnicode_AsUnicode(). But that's a bug, not a flaw in my approach, and I'll fix that bug today. I don't see how "[my] approach can cause memory errors" in any sort of larger sense. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2007-01-08 05:59 Message: Logged In: YES user_id=38388 Originator: NO While I don't think the added complexity in the implementation is worth it, given that there are other ways of achieving the same kind of performance (e.g. list of Unicode strings), some comments: * you add a long field to every Unicode object - so every single object in the system pays 4-8 bytes for the small performance advantage * Unicode objects are often references using PyUnicode_AS_UNICODE(); this operation doesn't allow passing back errors, yet your lazy evaluation approach can cause memory errors - how are you going to deal with them ? (currently you don't even test for them) * the lazy approach keeps all partial Unicode objects alive until they finally get concatenated; if you have lots of those (e.g. if you use x += y in a loop), then you pay the complete Python object overhead for every single partial Unicode object in the list of strings - given that most such operations use short strings, you are likely creating a memory overhead far greater than the the total length of all the strings ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2007-01-07 00:08 Message: Logged In: YES user_id=341410 Originator: NO What are the performance characteristics of each operation? I presume that a + b for unicode strings a and b is O(1) time (if I understand your implementation correctly). But according to my reading, (a + b + c + ...)[i] is O(number of concatenations performed). Is this correct? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1629305&group_id=5470 From noreply at sourceforge.net Wed Feb 28 23:38:44 2007 From: noreply at sourceforge.net (SourceForge.net) Date: Wed, 28 Feb 2007 14:38:44 -0800 Subject: [Patches] [ python-Patches-1671298 ] Refactor test_class to use unittest lib Message-ID: Patches item #1671298, was opened at 2007-02-28 15:38 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1671298&group_id=5470 Please note that this message will contain a full copy of the comment thread, including the initial issue submission, for this request, not just the latest update. Category: Tests Group: Python 2.6 Status: Open Resolution: None Priority: 5 Private: No Submitted By: Mike Verdone (jyzude) Assigned to: Nobody/Anonymous (nobody) Summary: Refactor test_class to use unittest lib Initial Comment: Refactored Lib/test/test_class.py to use unittest library instead of icky output comparison tests. Also have to delete output/test_class after adding this patch. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1671298&group_id=5470