From noreply at sourceforge.net Sat May 1 06:52:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 1 06:52:24 2004 Subject: [ python-Bugs-906405 ] Build fails on XP Message-ID: Bugs item #906405, was opened at 2004-02-28 11:08 Message generated for change (Comment added) made by shrodingersdog You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=906405&group_id=5470 Category: Build Group: Python 2.2.3 Status: Open Resolution: None Priority: 5 Submitted By: Con (shrodingersdog) Assigned to: Nobody/Anonymous (nobody) Summary: Build fails on XP Initial Comment: When trying to build the source: Modules/posixmodule.c fails to build. I'm running on XP SP1 using mingw32-make ver 3.80 Is there any known issues on XP? ---------------------------------------------------------------------- >Comment By: Con (shrodingersdog) Date: 2004-05-01 10:52 Message: Logged In: YES user_id=986333 Sorry, crash is not the right word. I've attached the output from make. Let me know if that helps or if you need other info. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-04-24 08:08 Message: Logged In: YES user_id=21627 What exactly does "it crashes" mean? Do you get a blue screen? Is your hard disk making a crashing sound? ---------------------------------------------------------------------- Comment By: Con (shrodingersdog) Date: 2004-04-23 09:27 Message: Logged In: YES user_id=986333 When I run make it crashes when it gets to Modules/posixmodule.c. This is compiling with the Ming / (Cygwin?) compiler though. I have recently tried with MSVC 6 which builds without any problems. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-03-29 22:35 Message: Logged In: YES user_id=21627 In what way precisely does it fail? It builds fine for me with VC.NET 2003, which is the official build environment for Python on XP. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=906405&group_id=5470 From noreply at sourceforge.net Sat May 1 07:39:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 1 07:39:12 2004 Subject: [ python-Bugs-945861 ] dir(object) does not list __name__ Message-ID: Bugs item #945861, was opened at 2004-05-01 07:39 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=945861&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Thomas Philips (tkpmep) Assigned to: Nobody/Anonymous (nobody) Summary: dir(object) does not list __name__ Initial Comment: dir(object) returns ['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce _ex__', '__repr__', '__setattr__', '__str__'] The __name__ method is not reported ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=945861&group_id=5470 From noreply at sourceforge.net Sat May 1 18:30:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 1 18:30:33 2004 Subject: [ python-Bugs-946130 ] xmlrpclib omits charset in Content-Type HTTP header Message-ID: Bugs item #946130, was opened at 2004-05-01 22:30 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Christian Schmidt (c960657) Assigned to: Nobody/Anonymous (nobody) Summary: xmlrpclib omits charset in Content-Type HTTP header Initial Comment: When xmlrpclib makes an HTTP request, it always sends the HTTP header line "Content-Type: text/xml". The encoding of the XML document is specified in the tag, e.g. . However, when XML is transferred over HTTP, the charset specified in the HTTP Content-Type header takes precedence over that in the document itself, i.e. the encoding specified in th tag should be ignored (RFC 3023 section 3.1). If the charset is not specified in the Content-Type header, it defaults to us-ascii. xmlrpclib currently specifies the charset in the encoding attribute of the tag and not in the HTTP header. The XML-RPC server thus treats the XML document as us-ascii instead of the specified encoding. xmlrpclib should specify the encoding in the Content-Type header. Disclaimer: I am no expert in XML and MIME-types, so I might be wrong about this. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 From noreply at sourceforge.net Sat May 1 21:02:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 1 21:02:15 2004 Subject: [ python-Bugs-945861 ] dir(object) does not list __name__ Message-ID: Bugs item #945861, was opened at 2004-05-01 06:39 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=945861&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Thomas Philips (tkpmep) Assigned to: Nobody/Anonymous (nobody) Summary: dir(object) does not list __name__ Initial Comment: dir(object) returns ['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce _ex__', '__repr__', '__setattr__', '__str__'] The __name__ method is not reported ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2004-05-01 20:02 Message: Logged In: YES user_id=44345 Are you sure that the object has an actual __name__ attribute (and not something computed by a __getattr__ method)? >>> import sys >>> dir(sys) ['__displayhook__', '__doc__', '__excepthook__', '__name__', '__stderr__', '__stdin__', '__stdout__', '_getframe', 'api_version', 'argv', 'builtin_module_names', 'byteorder', 'call_tracing', 'callstats', 'copyright', 'displayhook', 'exc_clear', 'exc_info', 'exc_type', 'excepthook', 'exec_prefix', 'executable', 'exit', 'exitfunc', 'getcheckinterval', 'getdefaultencoding', 'getdlopenflags', 'getfilesystemencoding', 'getrecursionlimit', 'getrefcount', 'hexversion', 'maxint', 'maxunicode', 'meta_path', 'modules', 'path', 'path_hooks', 'path_importer_cache', 'platform', 'prefix', 'ps1', 'ps2', 'setcheckinterval', 'setdlopenflags', 'setprofile', 'setrecursionlimit', 'settrace', 'stderr', 'stdin', 'stdout', 'version', 'version_info', 'warnoptions'] >>> sys.__name__ 'sys' ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=945861&group_id=5470 From noreply at sourceforge.net Sat May 1 22:50:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 1 22:50:27 2004 Subject: [ python-Bugs-942952 ] Weakness in tuple hash Message-ID: Bugs item #942952, was opened at 2004-04-27 14:41 Message generated for change (Comment added) made by ygale You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 6 Submitted By: Steve Tregidgo (smst) Assigned to: Tim Peters (tim_one) Summary: Weakness in tuple hash Initial Comment: I've encountered some performance problems when constructing dictionaries with keys of a particular form, and have pinned the problem down to the hashing function. I've reproduced the effect in Python 1.5.2, Python 2.2 and Python 2.3.3. I came across this when loading a marshalled dictionary with about 40000 entries. Loading other dictionaries of this size has always been fast, but in this case I killed the interpreter after a few minutes of CPU thrashing. The performance problem was caused because every key in the dictionary had the same hash value. The problem is as follows: for hashable values X and Y, hash( (X, (X, Y)) ) == hash(Y). This is always true (except in the corner case where hash((X, Y)) is internally calculated to be -1 (the error value) and so -2 is forced as the return value). With data in this form where X varies more than Y (Y is constant, or chosen from relatively few values compared to X) chances of collision become high as X is effectively ignored. The hash algorithm for tuples starts with a seed value, then generates a new value for each item in the tuple by multiplying the iteration's starting value by a constant (keeping the lowest 32 bits) and XORing with the hash of the item. The final value is then XORed with the tuple's length. In Python (ignoring the careful business with -1): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) The tuple (X, Y) therefore has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash(Y) ^ 2 ...and the tuple (X, (X, Y)) has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash((X, Y)) ^ 2 The outer multiplication is repeated, and is XORed with itself (cancelling both of them). The XORed 2s cancel also, leaving just hash(Y). Note that this cancellation property also means that the same hash value is shared by (X, (X, (X, (X, Y)))), and (X, (X, (X, (X, (X, (X, Y)))))), and so on, and (X, Z, (X, Z, Y)) and so on. I realise that choosing a hash function is a difficult task, so it may be that the behaviour seen here is a tradeoff against other desireable properties -- I don't have the expertise to tell. My naive suggestion would be that an extra multiplication is necessary, which presumably has a performance impact (multiplication being more expensive than XOR) but would reduce the possibility of cancellation. On the other hand, perhaps this particular case is rare enough that it's not worth the effort. For my own application I'm fortunate in that I can probably rearrange the data structure to avoid this case. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-02 05:50 Message: Logged In: YES user_id=1033539 I suggest leaving the function as is, but replacing the constant with a value that varies as we step through the tuple. Take the values from a fixed table. When we reach the end of the table, start again from the beginning and mung the values slightly (e.g., increment them). True, there will always be the possibilities of collisions, but this will make it very unlikely except in very weird cases. Using a table of constants is a standard technique for hashes. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 20:45 Message: Logged In: YES user_id=764593 Wouldn't that just shift the pathological case from (x, (x, y)) to (x, (-x, y))? My point was that any hash function will act badly on *some* pattern of input, and if a pattern must be penalized, this might be a good pattern to choose. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 19:36 Message: Logged In: YES user_id=80475 The OP was not referring to "some collisions"; his app collapsed all entries to a single hash value. Changing XOR to + would partially eliminate the self cancelling property of this hash function. Also, I am concerned about the tuple hash using the same multiplier as the hash for other objects. In sets.py, a naive combination of the component hash values caused many distinct sets to collapse to a handful of possibilities -- while tuples do not have an identical issue, it does highlight the risks involved. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 17:50 Message: Logged In: YES user_id=764593 Any hash will have some collisions. If there is going to be predictably bad data, this is probably a good place to have it. The obvious alternatives are a more complicated hash (slows everything down), a different hash for embedded tuples (bad, since hash can't be cached then) or ignoring some elements when determining the hash (bad in the normal case of different data). I would also expect your workaround of data rearrangement to be sensible almost any time (X, (X, Y)) is really a common case. (The intuitive meaning for me is "X - then map X to Y", which could be done as (X, Y) or at least (X, (None, Y)), or perhaps d[X]=(X,Y).) ---------------------------------------------------------------------- Comment By: Steve Tregidgo (smst) Date: 2004-04-27 14:45 Message: Logged In: YES user_id=42335 I'll repeat the tuple hashing algorithm in a fixed-width font (with the first line following "for..." being the loop body): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 From noreply at sourceforge.net Sun May 2 01:42:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 2 01:42:32 2004 Subject: [ python-Bugs-946247 ] CGI QUERY_STRING Message-ID: Bugs item #946247, was opened at 2004-05-01 22:42 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946247&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Larry (bugbee) Assigned to: Nobody/Anonymous (nobody) Summary: CGI QUERY_STRING Initial Comment: CGIHTTPServer.py QUERY_STRING kept carrying forward the old value especially with subsequent calls that didn't have a query string. QUERY_STRING was not being cleared. In CGIHTTPServer.py, lines 150 and 151 currently read: if query: env['QUERY_STRING'] = query and should read: if query: env['QUERY_STRING'] = query else: env['QUERY_STRING'] = '' # '' or None ??? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946247&group_id=5470 From noreply at sourceforge.net Sun May 2 06:59:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 2 06:59:26 2004 Subject: [ python-Bugs-946130 ] xmlrpclib omits charset in Content-Type HTTP header Message-ID: Bugs item #946130, was opened at 2004-05-02 00:30 Message generated for change (Comment added) made by lemburg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Christian Schmidt (c960657) Assigned to: Nobody/Anonymous (nobody) Summary: xmlrpclib omits charset in Content-Type HTTP header Initial Comment: When xmlrpclib makes an HTTP request, it always sends the HTTP header line "Content-Type: text/xml". The encoding of the XML document is specified in the tag, e.g. . However, when XML is transferred over HTTP, the charset specified in the HTTP Content-Type header takes precedence over that in the document itself, i.e. the encoding specified in th tag should be ignored (RFC 3023 section 3.1). If the charset is not specified in the Content-Type header, it defaults to us-ascii. xmlrpclib currently specifies the charset in the encoding attribute of the tag and not in the HTTP header. The XML-RPC server thus treats the XML document as us-ascii instead of the specified encoding. xmlrpclib should specify the encoding in the Content-Type header. Disclaimer: I am no expert in XML and MIME-types, so I might be wrong about this. ---------------------------------------------------------------------- >Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-02 12:59 Message: Logged In: YES user_id=38388 I don't see anything wrong with the way xmlrpclib deals with the encoding. You right on one point: HTTP defaults to Latin-1 as charset, but since the content may well be non-Latin-1, xmlrpclib should probably also place the encoding information into the HTTP header (for requests it sends out). However, this is rarely a problem, since clients usually don't follow the HTTP way of interpreting the charset when seeing text/xml as content type... xmlrpclib itself certainly doesn't :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 From noreply at sourceforge.net Sun May 2 08:25:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 2 08:25:24 2004 Subject: [ python-Bugs-946130 ] xmlrpclib omits charset in Content-Type HTTP header Message-ID: Bugs item #946130, was opened at 2004-05-02 00:30 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 Category: None >Group: Not a Bug >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Christian Schmidt (c960657) Assigned to: Nobody/Anonymous (nobody) Summary: xmlrpclib omits charset in Content-Type HTTP header Initial Comment: When xmlrpclib makes an HTTP request, it always sends the HTTP header line "Content-Type: text/xml". The encoding of the XML document is specified in the tag, e.g. . However, when XML is transferred over HTTP, the charset specified in the HTTP Content-Type header takes precedence over that in the document itself, i.e. the encoding specified in th tag should be ignored (RFC 3023 section 3.1). If the charset is not specified in the Content-Type header, it defaults to us-ascii. xmlrpclib currently specifies the charset in the encoding attribute of the tag and not in the HTTP header. The XML-RPC server thus treats the XML document as us-ascii instead of the specified encoding. xmlrpclib should specify the encoding in the Content-Type header. Disclaimer: I am no expert in XML and MIME-types, so I might be wrong about this. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-02 14:25 Message: Logged In: YES user_id=21627 The XML-RPC spec is very clear that the value of the Content-Type header is "text/xml". Following the traditional interpretation of the XML-RPC spec (where examples are considered normative), it would be a protocol violation to add a charset= parameter to Content-Type. Until the XML-RPC spec is changed, or the status of using charset= in XML-RPC is officially clarified, we can't change our implementation. Closing this as not-a-bug. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-02 12:59 Message: Logged In: YES user_id=38388 I don't see anything wrong with the way xmlrpclib deals with the encoding. You right on one point: HTTP defaults to Latin-1 as charset, but since the content may well be non-Latin-1, xmlrpclib should probably also place the encoding information into the HTTP header (for requests it sends out). However, this is rarely a problem, since clients usually don't follow the HTTP way of interpreting the charset when seeing text/xml as content type... xmlrpclib itself certainly doesn't :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 From noreply at sourceforge.net Sun May 2 10:39:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 2 10:39:11 2004 Subject: [ python-Bugs-946130 ] xmlrpclib omits charset in Content-Type HTTP header Message-ID: Bugs item #946130, was opened at 2004-05-01 22:30 Message generated for change (Comment added) made by c960657 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 Category: None Group: Not a Bug Status: Closed Resolution: Invalid Priority: 5 Submitted By: Christian Schmidt (c960657) Assigned to: Nobody/Anonymous (nobody) Summary: xmlrpclib omits charset in Content-Type HTTP header Initial Comment: When xmlrpclib makes an HTTP request, it always sends the HTTP header line "Content-Type: text/xml". The encoding of the XML document is specified in the tag, e.g. . However, when XML is transferred over HTTP, the charset specified in the HTTP Content-Type header takes precedence over that in the document itself, i.e. the encoding specified in th tag should be ignored (RFC 3023 section 3.1). If the charset is not specified in the Content-Type header, it defaults to us-ascii. xmlrpclib currently specifies the charset in the encoding attribute of the tag and not in the HTTP header. The XML-RPC server thus treats the XML document as us-ascii instead of the specified encoding. xmlrpclib should specify the encoding in the Content-Type header. Disclaimer: I am no expert in XML and MIME-types, so I might be wrong about this. ---------------------------------------------------------------------- >Comment By: Christian Schmidt (c960657) Date: 2004-05-02 14:39 Message: Logged In: YES user_id=32013 Hmm, interesting. I agree that according to the letter of the spec, the encoding cannot be specified in the Content-Type header. But -- XML-RPC uses HTTP so I would argue that RFC 3023 still applies. If it does, a server should ignore any encoding specified in the , and the default encoding for text/xml is us-ascii. So in order to represent non-us-ascii characters in an XML-RPC message, they should be encoded using the &#xx; notation. xmlrpclib doesn't do this, so I suggest reopening this bug. FYI: The actual problem I am having is making xmlrpclib work with the XML_RPC_Server that is part of PEAR (PEAR is the "official" PHP Extension and Application Repository). This server does not inspect the tag for an encoding but always assumes that the input is UTF-8. According to RFC 3023 it should assume it to be us-ascii, but since us-ascii is a subset of UTF-8, the current behaviour of the server should be safe, as long as the client either sends us-ascii or UTF-8 (I have submitted a patch to the XML_RPC_Server maintainer that extends the encoding detection to look in both the Content-Type header and the tag). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-02 12:25 Message: Logged In: YES user_id=21627 The XML-RPC spec is very clear that the value of the Content-Type header is "text/xml". Following the traditional interpretation of the XML-RPC spec (where examples are considered normative), it would be a protocol violation to add a charset= parameter to Content-Type. Until the XML-RPC spec is changed, or the status of using charset= in XML-RPC is officially clarified, we can't change our implementation. Closing this as not-a-bug. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-02 10:59 Message: Logged In: YES user_id=38388 I don't see anything wrong with the way xmlrpclib deals with the encoding. You right on one point: HTTP defaults to Latin-1 as charset, but since the content may well be non-Latin-1, xmlrpclib should probably also place the encoding information into the HTTP header (for requests it sends out). However, this is rarely a problem, since clients usually don't follow the HTTP way of interpreting the charset when seeing text/xml as content type... xmlrpclib itself certainly doesn't :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 From noreply at sourceforge.net Sun May 2 12:28:57 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 2 12:29:03 2004 Subject: [ python-Bugs-946130 ] xmlrpclib omits charset in Content-Type HTTP header Message-ID: Bugs item #946130, was opened at 2004-05-02 00:30 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 Category: None Group: Not a Bug Status: Closed Resolution: Invalid Priority: 5 Submitted By: Christian Schmidt (c960657) Assigned to: Nobody/Anonymous (nobody) Summary: xmlrpclib omits charset in Content-Type HTTP header Initial Comment: When xmlrpclib makes an HTTP request, it always sends the HTTP header line "Content-Type: text/xml". The encoding of the XML document is specified in the tag, e.g. . However, when XML is transferred over HTTP, the charset specified in the HTTP Content-Type header takes precedence over that in the document itself, i.e. the encoding specified in th tag should be ignored (RFC 3023 section 3.1). If the charset is not specified in the Content-Type header, it defaults to us-ascii. xmlrpclib currently specifies the charset in the encoding attribute of the tag and not in the HTTP header. The XML-RPC server thus treats the XML document as us-ascii instead of the specified encoding. xmlrpclib should specify the encoding in the Content-Type header. Disclaimer: I am no expert in XML and MIME-types, so I might be wrong about this. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-02 18:28 Message: Logged In: YES user_id=21627 When it comes to XML-RPC, the only person to argue with is Dave Winer (arguing with us is futile). If you can make him say, in public, that adding charset= is ok for XML-RPC implementations, we can change Python. As for your current problem: It would be best to use US-ASCII for encoding your XML document, representing non-ASCII characters as character references. Of course, that is currently not supported in xmlrpclib; patches welcome. ---------------------------------------------------------------------- Comment By: Christian Schmidt (c960657) Date: 2004-05-02 16:39 Message: Logged In: YES user_id=32013 Hmm, interesting. I agree that according to the letter of the spec, the encoding cannot be specified in the Content-Type header. But -- XML-RPC uses HTTP so I would argue that RFC 3023 still applies. If it does, a server should ignore any encoding specified in the , and the default encoding for text/xml is us-ascii. So in order to represent non-us-ascii characters in an XML-RPC message, they should be encoded using the &#xx; notation. xmlrpclib doesn't do this, so I suggest reopening this bug. FYI: The actual problem I am having is making xmlrpclib work with the XML_RPC_Server that is part of PEAR (PEAR is the "official" PHP Extension and Application Repository). This server does not inspect the tag for an encoding but always assumes that the input is UTF-8. According to RFC 3023 it should assume it to be us-ascii, but since us-ascii is a subset of UTF-8, the current behaviour of the server should be safe, as long as the client either sends us-ascii or UTF-8 (I have submitted a patch to the XML_RPC_Server maintainer that extends the encoding detection to look in both the Content-Type header and the tag). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-02 14:25 Message: Logged In: YES user_id=21627 The XML-RPC spec is very clear that the value of the Content-Type header is "text/xml". Following the traditional interpretation of the XML-RPC spec (where examples are considered normative), it would be a protocol violation to add a charset= parameter to Content-Type. Until the XML-RPC spec is changed, or the status of using charset= in XML-RPC is officially clarified, we can't change our implementation. Closing this as not-a-bug. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-02 12:59 Message: Logged In: YES user_id=38388 I don't see anything wrong with the way xmlrpclib deals with the encoding. You right on one point: HTTP defaults to Latin-1 as charset, but since the content may well be non-Latin-1, xmlrpclib should probably also place the encoding information into the HTTP header (for requests it sends out). However, this is rarely a problem, since clients usually don't follow the HTTP way of interpreting the charset when seeing text/xml as content type... xmlrpclib itself certainly doesn't :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 From noreply at sourceforge.net Sun May 2 15:47:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 2 15:48:00 2004 Subject: [ python-Bugs-945861 ] dir(object) does not list __name__ Message-ID: Bugs item #945861, was opened at 2004-05-01 06:39 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=945861&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Thomas Philips (tkpmep) Assigned to: Nobody/Anonymous (nobody) Summary: dir(object) does not list __name__ Initial Comment: dir(object) returns ['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce _ex__', '__repr__', '__setattr__', '__str__'] The __name__ method is not reported ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2004-05-02 14:47 Message: Logged In: YES user_id=44345 After a message from the submitter, it's apparent he was referring to class objects not showing '__name__' attributes in dir() output. This is a case of an attribute not being visible to dir() because it's not directly present in the object's __dict__ and is trapped at evaluation time by __getattr__(). Short of hacking dir() or adding a special attribute ("__attributes__"?) to objects which have __getattr__() methods I don't see a way around this problem. Wasn't there discussion of such an attribute which would expose such dynamic attributes to dir()? I don't see anything in the implementation of PyObject_Dir(). ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-05-01 20:02 Message: Logged In: YES user_id=44345 Are you sure that the object has an actual __name__ attribute (and not something computed by a __getattr__ method)? >>> import sys >>> dir(sys) ['__displayhook__', '__doc__', '__excepthook__', '__name__', '__stderr__', '__stdin__', '__stdout__', '_getframe', 'api_version', 'argv', 'builtin_module_names', 'byteorder', 'call_tracing', 'callstats', 'copyright', 'displayhook', 'exc_clear', 'exc_info', 'exc_type', 'excepthook', 'exec_prefix', 'executable', 'exit', 'exitfunc', 'getcheckinterval', 'getdefaultencoding', 'getdlopenflags', 'getfilesystemencoding', 'getrecursionlimit', 'getrefcount', 'hexversion', 'maxint', 'maxunicode', 'meta_path', 'modules', 'path', 'path_hooks', 'path_importer_cache', 'platform', 'prefix', 'ps1', 'ps2', 'setcheckinterval', 'setdlopenflags', 'setprofile', 'setrecursionlimit', 'settrace', 'stderr', 'stdin', 'stdout', 'version', 'version_info', 'warnoptions'] >>> sys.__name__ 'sys' ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=945861&group_id=5470 From noreply at sourceforge.net Sun May 2 17:04:44 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 2 17:04:51 2004 Subject: [ python-Bugs-946130 ] xmlrpclib omits charset in Content-Type HTTP header Message-ID: Bugs item #946130, was opened at 2004-05-01 22:30 Message generated for change (Comment added) made by c960657 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 Category: None Group: Not a Bug Status: Closed Resolution: Invalid Priority: 5 Submitted By: Christian Schmidt (c960657) Assigned to: Nobody/Anonymous (nobody) Summary: xmlrpclib omits charset in Content-Type HTTP header Initial Comment: When xmlrpclib makes an HTTP request, it always sends the HTTP header line "Content-Type: text/xml". The encoding of the XML document is specified in the tag, e.g. . However, when XML is transferred over HTTP, the charset specified in the HTTP Content-Type header takes precedence over that in the document itself, i.e. the encoding specified in th tag should be ignored (RFC 3023 section 3.1). If the charset is not specified in the Content-Type header, it defaults to us-ascii. xmlrpclib currently specifies the charset in the encoding attribute of the tag and not in the HTTP header. The XML-RPC server thus treats the XML document as us-ascii instead of the specified encoding. xmlrpclib should specify the encoding in the Content-Type header. Disclaimer: I am no expert in XML and MIME-types, so I might be wrong about this. ---------------------------------------------------------------------- >Comment By: Christian Schmidt (c960657) Date: 2004-05-02 21:04 Message: Logged In: YES user_id=32013 It was not my intention to argue with you whether XML-RPC should allow adding a charset= or not. But I am arguing that RFC 3023 applies for XML-RPC, and that implies that as long as charset= is not specified, the server should treat the encoding as us-ascii, even if another encoding is specified in the tag (the XML-RPC spec doesn't mention specifying an encoding in the tag, so as such there is no contradiction between the spec and RFC 3023). And this implies that XML-RPC must encode non-us-ascii characters with &#xx;. So my point is that in order to be compliant with both the XML-RPC spec and RFC 3023, xmlrpclib should encode non-us-ascii characters as &#xx;. This may not be that important, but I believe it _is_ a bug, i.e. I disagree that this is marked "Not a bug". I only started coding Pyhton two weeks ago, so I may not be the best to provide a patch :-) But I am willing to give it a try. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-02 16:28 Message: Logged In: YES user_id=21627 When it comes to XML-RPC, the only person to argue with is Dave Winer (arguing with us is futile). If you can make him say, in public, that adding charset= is ok for XML-RPC implementations, we can change Python. As for your current problem: It would be best to use US-ASCII for encoding your XML document, representing non-ASCII characters as character references. Of course, that is currently not supported in xmlrpclib; patches welcome. ---------------------------------------------------------------------- Comment By: Christian Schmidt (c960657) Date: 2004-05-02 14:39 Message: Logged In: YES user_id=32013 Hmm, interesting. I agree that according to the letter of the spec, the encoding cannot be specified in the Content-Type header. But -- XML-RPC uses HTTP so I would argue that RFC 3023 still applies. If it does, a server should ignore any encoding specified in the , and the default encoding for text/xml is us-ascii. So in order to represent non-us-ascii characters in an XML-RPC message, they should be encoded using the &#xx; notation. xmlrpclib doesn't do this, so I suggest reopening this bug. FYI: The actual problem I am having is making xmlrpclib work with the XML_RPC_Server that is part of PEAR (PEAR is the "official" PHP Extension and Application Repository). This server does not inspect the tag for an encoding but always assumes that the input is UTF-8. According to RFC 3023 it should assume it to be us-ascii, but since us-ascii is a subset of UTF-8, the current behaviour of the server should be safe, as long as the client either sends us-ascii or UTF-8 (I have submitted a patch to the XML_RPC_Server maintainer that extends the encoding detection to look in both the Content-Type header and the tag). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-02 12:25 Message: Logged In: YES user_id=21627 The XML-RPC spec is very clear that the value of the Content-Type header is "text/xml". Following the traditional interpretation of the XML-RPC spec (where examples are considered normative), it would be a protocol violation to add a charset= parameter to Content-Type. Until the XML-RPC spec is changed, or the status of using charset= in XML-RPC is officially clarified, we can't change our implementation. Closing this as not-a-bug. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-02 10:59 Message: Logged In: YES user_id=38388 I don't see anything wrong with the way xmlrpclib deals with the encoding. You right on one point: HTTP defaults to Latin-1 as charset, but since the content may well be non-Latin-1, xmlrpclib should probably also place the encoding information into the HTTP header (for requests it sends out). However, this is rarely a problem, since clients usually don't follow the HTTP way of interpreting the charset when seeing text/xml as content type... xmlrpclib itself certainly doesn't :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 From noreply at sourceforge.net Sun May 2 17:25:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 2 17:25:49 2004 Subject: [ python-Bugs-946130 ] xmlrpclib omits charset in Content-Type HTTP header Message-ID: Bugs item #946130, was opened at 2004-05-02 00:30 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 Category: None Group: Not a Bug Status: Closed Resolution: Invalid Priority: 5 Submitted By: Christian Schmidt (c960657) Assigned to: Nobody/Anonymous (nobody) Summary: xmlrpclib omits charset in Content-Type HTTP header Initial Comment: When xmlrpclib makes an HTTP request, it always sends the HTTP header line "Content-Type: text/xml". The encoding of the XML document is specified in the tag, e.g. . However, when XML is transferred over HTTP, the charset specified in the HTTP Content-Type header takes precedence over that in the document itself, i.e. the encoding specified in th tag should be ignored (RFC 3023 section 3.1). If the charset is not specified in the Content-Type header, it defaults to us-ascii. xmlrpclib currently specifies the charset in the encoding attribute of the tag and not in the HTTP header. The XML-RPC server thus treats the XML document as us-ascii instead of the specified encoding. xmlrpclib should specify the encoding in the Content-Type header. Disclaimer: I am no expert in XML and MIME-types, so I might be wrong about this. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-02 23:25 Message: Logged In: YES user_id=21627 The bug, as reported, is not a bug. Adding charset= to the Content-type would be a violation of the XML-RPC protocol. Not doing so might be a violation of the HTTP protocol, but xmlrpclib does not claim to implement HTTP; it claims to implement XML-RPC. Using character references if unencodable characters are found might be a good idea, but this is not the subject of the bug report. Asking for a change to xmlrpclib to use character references where necessary would not be a bug report, it would be a feature request (because xmlrpclib is currently not misbehaving in this respect). If you want to look into implementing a change in this direction, you should try to use the 'xmlcharrefreplace' error handler for the Unicode .encode method. The tricky part is to get the quoting of ASCII characters (lt, gt, ...) right. ---------------------------------------------------------------------- Comment By: Christian Schmidt (c960657) Date: 2004-05-02 23:04 Message: Logged In: YES user_id=32013 It was not my intention to argue with you whether XML-RPC should allow adding a charset= or not. But I am arguing that RFC 3023 applies for XML-RPC, and that implies that as long as charset= is not specified, the server should treat the encoding as us-ascii, even if another encoding is specified in the tag (the XML-RPC spec doesn't mention specifying an encoding in the tag, so as such there is no contradiction between the spec and RFC 3023). And this implies that XML-RPC must encode non-us-ascii characters with &#xx;. So my point is that in order to be compliant with both the XML-RPC spec and RFC 3023, xmlrpclib should encode non-us-ascii characters as &#xx;. This may not be that important, but I believe it _is_ a bug, i.e. I disagree that this is marked "Not a bug". I only started coding Pyhton two weeks ago, so I may not be the best to provide a patch :-) But I am willing to give it a try. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-02 18:28 Message: Logged In: YES user_id=21627 When it comes to XML-RPC, the only person to argue with is Dave Winer (arguing with us is futile). If you can make him say, in public, that adding charset= is ok for XML-RPC implementations, we can change Python. As for your current problem: It would be best to use US-ASCII for encoding your XML document, representing non-ASCII characters as character references. Of course, that is currently not supported in xmlrpclib; patches welcome. ---------------------------------------------------------------------- Comment By: Christian Schmidt (c960657) Date: 2004-05-02 16:39 Message: Logged In: YES user_id=32013 Hmm, interesting. I agree that according to the letter of the spec, the encoding cannot be specified in the Content-Type header. But -- XML-RPC uses HTTP so I would argue that RFC 3023 still applies. If it does, a server should ignore any encoding specified in the , and the default encoding for text/xml is us-ascii. So in order to represent non-us-ascii characters in an XML-RPC message, they should be encoded using the &#xx; notation. xmlrpclib doesn't do this, so I suggest reopening this bug. FYI: The actual problem I am having is making xmlrpclib work with the XML_RPC_Server that is part of PEAR (PEAR is the "official" PHP Extension and Application Repository). This server does not inspect the tag for an encoding but always assumes that the input is UTF-8. According to RFC 3023 it should assume it to be us-ascii, but since us-ascii is a subset of UTF-8, the current behaviour of the server should be safe, as long as the client either sends us-ascii or UTF-8 (I have submitted a patch to the XML_RPC_Server maintainer that extends the encoding detection to look in both the Content-Type header and the tag). ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-02 14:25 Message: Logged In: YES user_id=21627 The XML-RPC spec is very clear that the value of the Content-Type header is "text/xml". Following the traditional interpretation of the XML-RPC spec (where examples are considered normative), it would be a protocol violation to add a charset= parameter to Content-Type. Until the XML-RPC spec is changed, or the status of using charset= in XML-RPC is officially clarified, we can't change our implementation. Closing this as not-a-bug. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-02 12:59 Message: Logged In: YES user_id=38388 I don't see anything wrong with the way xmlrpclib deals with the encoding. You right on one point: HTTP defaults to Latin-1 as charset, but since the content may well be non-Latin-1, xmlrpclib should probably also place the encoding information into the HTTP header (for requests it sends out). However, this is rarely a problem, since clients usually don't follow the HTTP way of interpreting the charset when seeing text/xml as content type... xmlrpclib itself certainly doesn't :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=946130&group_id=5470 From noreply at sourceforge.net Mon May 3 09:40:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 3 09:40:25 2004 Subject: [ python-Bugs-942952 ] Weakness in tuple hash Message-ID: Bugs item #942952, was opened at 2004-04-27 07:41 Message generated for change (Comment added) made by jimjjewett You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 6 Submitted By: Steve Tregidgo (smst) Assigned to: Tim Peters (tim_one) Summary: Weakness in tuple hash Initial Comment: I've encountered some performance problems when constructing dictionaries with keys of a particular form, and have pinned the problem down to the hashing function. I've reproduced the effect in Python 1.5.2, Python 2.2 and Python 2.3.3. I came across this when loading a marshalled dictionary with about 40000 entries. Loading other dictionaries of this size has always been fast, but in this case I killed the interpreter after a few minutes of CPU thrashing. The performance problem was caused because every key in the dictionary had the same hash value. The problem is as follows: for hashable values X and Y, hash( (X, (X, Y)) ) == hash(Y). This is always true (except in the corner case where hash((X, Y)) is internally calculated to be -1 (the error value) and so -2 is forced as the return value). With data in this form where X varies more than Y (Y is constant, or chosen from relatively few values compared to X) chances of collision become high as X is effectively ignored. The hash algorithm for tuples starts with a seed value, then generates a new value for each item in the tuple by multiplying the iteration's starting value by a constant (keeping the lowest 32 bits) and XORing with the hash of the item. The final value is then XORed with the tuple's length. In Python (ignoring the careful business with -1): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) The tuple (X, Y) therefore has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash(Y) ^ 2 ...and the tuple (X, (X, Y)) has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash((X, Y)) ^ 2 The outer multiplication is repeated, and is XORed with itself (cancelling both of them). The XORed 2s cancel also, leaving just hash(Y). Note that this cancellation property also means that the same hash value is shared by (X, (X, (X, (X, Y)))), and (X, (X, (X, (X, (X, (X, Y)))))), and so on, and (X, Z, (X, Z, Y)) and so on. I realise that choosing a hash function is a difficult task, so it may be that the behaviour seen here is a tradeoff against other desireable properties -- I don't have the expertise to tell. My naive suggestion would be that an extra multiplication is necessary, which presumably has a performance impact (multiplication being more expensive than XOR) but would reduce the possibility of cancellation. On the other hand, perhaps this particular case is rare enough that it's not worth the effort. For my own application I'm fortunate in that I can probably rearrange the data structure to avoid this case. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-03 09:40 Message: Logged In: YES user_id=764593 Note that in this case, the problem was because of nested tuples. X was the first element of both tuples, and both tuples had the same length -- so the X's would still have been multiplied by the same constant. (Not the same constant as Y, and hash(X, Y), but the same constant as each other.) A non-linear function might work, but would do bad things to other data. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-01 22:50 Message: Logged In: YES user_id=1033539 I suggest leaving the function as is, but replacing the constant with a value that varies as we step through the tuple. Take the values from a fixed table. When we reach the end of the table, start again from the beginning and mung the values slightly (e.g., increment them). True, there will always be the possibilities of collisions, but this will make it very unlikely except in very weird cases. Using a table of constants is a standard technique for hashes. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 13:45 Message: Logged In: YES user_id=764593 Wouldn't that just shift the pathological case from (x, (x, y)) to (x, (-x, y))? My point was that any hash function will act badly on *some* pattern of input, and if a pattern must be penalized, this might be a good pattern to choose. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 12:36 Message: Logged In: YES user_id=80475 The OP was not referring to "some collisions"; his app collapsed all entries to a single hash value. Changing XOR to + would partially eliminate the self cancelling property of this hash function. Also, I am concerned about the tuple hash using the same multiplier as the hash for other objects. In sets.py, a naive combination of the component hash values caused many distinct sets to collapse to a handful of possibilities -- while tuples do not have an identical issue, it does highlight the risks involved. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 10:50 Message: Logged In: YES user_id=764593 Any hash will have some collisions. If there is going to be predictably bad data, this is probably a good place to have it. The obvious alternatives are a more complicated hash (slows everything down), a different hash for embedded tuples (bad, since hash can't be cached then) or ignoring some elements when determining the hash (bad in the normal case of different data). I would also expect your workaround of data rearrangement to be sensible almost any time (X, (X, Y)) is really a common case. (The intuitive meaning for me is "X - then map X to Y", which could be done as (X, Y) or at least (X, (None, Y)), or perhaps d[X]=(X,Y).) ---------------------------------------------------------------------- Comment By: Steve Tregidgo (smst) Date: 2004-04-27 07:45 Message: Logged In: YES user_id=42335 I'll repeat the tuple hashing algorithm in a fixed-width font (with the first line following "for..." being the loop body): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 From noreply at sourceforge.net Mon May 3 20:46:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 3 20:46:49 2004 Subject: [ python-Bugs-947380 ] sys.path is wrong in some cases Message-ID: Bugs item #947380, was opened at 2004-05-04 03:46 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947380&group_id=5470 Category: Python Interpreter Core Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Ori Berger (orib) Assigned to: Nobody/Anonymous (nobody) Summary: sys.path is wrong in some cases Initial Comment: Python version tested: 2.3.3 (But if I read the cvs annotate correctly, this goes all the way back to 2.0) OS Version tested: Win2K (but any win32 version should behave the same). On Windows, sys,path sometimes contains the 'current working directory', in which the Python process was started, while the interpreter is still initializing; it shouldn't be there until after the interpreter had completed initializing and is ready for batch or interactive execution. How to reproduce: Use plain-vanilla 2.3.3 Python, without _any_ additional module installed. The HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.3\PythonPath registry key must NOT have any subkey (this is true for a fresh install, but might not be true if packages were installed earlier). Go to a subdirectory of your choice, e.g., C:\HoverCraft\Eels, and run (assuming Python is in c:\python23) > c:\python23\python -c "import sys; print sys.path" The first entry should be the current directory or ''. That's ok - it was added at the end of initialization. The second entry will be a .zip file where the .dll is; That's also ok. The third entry should be the current directory (or '') again. THIS IS NOT OK. How this was discovered: To give credit where credit is due, my friend Oren Gampel created a file called 'stat.py' in a directory, and from that moment on, Python started complaining about failing to import site. The reason is that 'site' imports 'ntpath', which imports 'stat', which unfortunately imported the new 'stat' rather than the library 'stat'. In some convoluted settings, this might have security implications - e.g., if an administrator starts a Python script in a directory to which a user has write permissions, this could result in a privelege escalation (even if the script has no import statements at all - the implicit "import site" is sufficient). I'm submitting a 2-line patch to the patch tracker (and one of them is a comment!) that seems to solve this problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947380&group_id=5470 From noreply at sourceforge.net Mon May 3 20:59:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 3 20:59:52 2004 Subject: [ python-Bugs-947380 ] sys.path is wrong in some cases Message-ID: Bugs item #947380, was opened at 2004-05-04 03:46 Message generated for change (Comment added) made by orib You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947380&group_id=5470 Category: Python Interpreter Core Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Ori Berger (orib) Assigned to: Nobody/Anonymous (nobody) Summary: sys.path is wrong in some cases Initial Comment: Python version tested: 2.3.3 (But if I read the cvs annotate correctly, this goes all the way back to 2.0) OS Version tested: Win2K (but any win32 version should behave the same). On Windows, sys,path sometimes contains the 'current working directory', in which the Python process was started, while the interpreter is still initializing; it shouldn't be there until after the interpreter had completed initializing and is ready for batch or interactive execution. How to reproduce: Use plain-vanilla 2.3.3 Python, without _any_ additional module installed. The HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.3\PythonPath registry key must NOT have any subkey (this is true for a fresh install, but might not be true if packages were installed earlier). Go to a subdirectory of your choice, e.g., C:\HoverCraft\Eels, and run (assuming Python is in c:\python23) > c:\python23\python -c "import sys; print sys.path" The first entry should be the current directory or ''. That's ok - it was added at the end of initialization. The second entry will be a .zip file where the .dll is; That's also ok. The third entry should be the current directory (or '') again. THIS IS NOT OK. How this was discovered: To give credit where credit is due, my friend Oren Gampel created a file called 'stat.py' in a directory, and from that moment on, Python started complaining about failing to import site. The reason is that 'site' imports 'ntpath', which imports 'stat', which unfortunately imported the new 'stat' rather than the library 'stat'. In some convoluted settings, this might have security implications - e.g., if an administrator starts a Python script in a directory to which a user has write permissions, this could result in a privelege escalation (even if the script has no import statements at all - the implicit "import site" is sufficient). I'm submitting a 2-line patch to the patch tracker (and one of them is a comment!) that seems to solve this problem. ---------------------------------------------------------------------- >Comment By: Ori Berger (orib) Date: 2004-05-04 03:59 Message: Logged In: YES user_id=67862 Patch uploaded to http://python.org/sf/947386 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947380&group_id=5470 From noreply at sourceforge.net Mon May 3 21:32:02 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 3 21:35:14 2004 Subject: [ python-Bugs-947405 ] os.utime() raises bad exception for unicode filenames Message-ID: Bugs item #947405, was opened at 2004-05-04 11:32 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947405&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Mark Hammond (mhammond) Assigned to: Thomas Heller (theller) Summary: os.utime() raises bad exception for unicode filenames Initial Comment: os.utime handles unicode filenames - but when it raises an exception, it always uses the 'char *' filename. When unicode is passed to this function, the 'char *' is garbage. For example: python -c "import os;os.utime(u'\xe', None)" Traceback (most recent call last): File "", line 1, in ? OSError: [Errno 2] No such file or directory: '\x05' Note the bad filename. Once this patch is applied, the output is: Traceback (most recent call last): File "", line 1, in ? OSError: [Errno 2] No such file or directory: u'\xe' The patch is against the 2.3 tree, but still applies against the head. I believe this should be checked into both places (and am happy to do it :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947405&group_id=5470 From noreply at sourceforge.net Tue May 4 01:48:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 01:48:30 2004 Subject: [ python-Bugs-905389 ] str.join() intercepts TypeError raised by iterator Message-ID: Bugs item #905389, was opened at 2004-02-26 16:19 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=905389&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Lenard Lindstrom (kermode) >Assigned to: Nobody/Anonymous (nobody) Summary: str.join() intercepts TypeError raised by iterator Initial Comment: For str.join(), if it is passed an iterator and that iterator raises a TypeError, that exception is caught by the join method and replaced by its own TypeError exception. SyntaxError and IndexError exceptions are uneffected. Example: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 ... IDLE 1.0.2 >>> def gen(n): if not isinstance(n, int): raise TypeError, "gen() TypeError" if n<0: raise IndexError, "gen() IndexError" for i in range(n): yield str(i) >>> ''.join(gen(5)) '01234' >>> ''.join(gen(-1)) Traceback (most recent call last): File "", line 1, in -toplevel- ''.join(gen(-1)) File "", line 5, in gen raise IndexError, "gen() IndexError" IndexError: gen() IndexError >>> ''.join(gen(None)) Traceback (most recent call last): File "", line 1, in -toplevel- ''.join(gen(None)) TypeError: sequence expected, generator found >>> ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=905389&group_id=5470 From noreply at sourceforge.net Tue May 4 01:52:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 01:52:17 2004 Subject: [ python-Bugs-764504 ] doctest fails with TypeError Message-ID: Bugs item #764504, was opened at 2003-07-02 05:30 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=764504&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: Postponed Priority: 3 Submitted By: Mark J (average) >Assigned to: Nobody/Anonymous (nobody) Summary: doctest fails with TypeError Initial Comment: #mymod.py: class base(dict): myupdate = dict.update #FINE def myfunc(self): pass class derive(base): myname = base.myfunc #FAILS import doctest, mymod doctest.testmod(mymod) ****** $ python2.3b2 mymod.py Traceback (most recent call last): File "mymod.py", line 8, in ? import doctest, mymod File "/home/me/mymod.py", line 9, in ? doctest.testmod(mymod) File "/usr/local/lib/python2.3/doctest.py", line 1137, in testmod f, t = tester.rundict(m.__dict__, name, m) File "/usr/local/lib/python2.3/doctest.py", line 900, in rundict f2, t2 = self.__runone(value, name + "." + thisname) File "/usr/local/lib/python2.3/doctest.py", line 1061, in __runone return self.rundoc(target, name) File "/usr/local/lib/python2.3/doctest.py", line 820, in rundoc f2, t2 = self.run__test__(d, name) File "/usr/local/lib/python2.3/doctest.py", line 929, in run__test__ raise TypeError("Tester.run__test__: values in " TypeError: Tester.run__test__: values in dict must be strings, functions or classes; ****** Does not appear to be specific to Python v2.3. Thanks! Mark ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-07-17 01:27 Message: Logged In: YES user_id=80475 Agreed. There is no compelling use case for changing this in Py2.3. Marking as postponed. For Py2.4, there will be plenty of time for merging/improving the underlying search behaviors. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2003-07-16 16:08 Message: Logged In: YES user_id=31435 If it has to be fixed , the only principle I can conjure up is that redundant testing is a waste of time and surprising. On that ground, in the specific example I'd rather skip it, since if the thing had a doctest, it would be most natural to run it from base.myfunc.__doc__. If you're feeling more ambitious, I recently slammed in some useful doctest extensions from Jim Fulton. As I know you've already discovered, there are some warts. The deepest wart isn't obvious: Jim's extensions use their own ways of finding doctests to run. It would be good if everthing used the same ways of finding doctests to run. This bug report shows that doctest's old ways don't always work. It's unknown in how many situations the new ways don't work, and/or deliver different results. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-07-09 00:48 Message: Logged In: YES user_id=80475 Tim, would you like me to fix this one by searching the unbound method or by skipping the unbound method? Since classes are searched recursively, it may be reasonable to search the unbound method. However, imports are not searched, so there is a precedent for skipping it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=764504&group_id=5470 From noreply at sourceforge.net Tue May 4 03:35:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 03:36:03 2004 Subject: [ python-Bugs-947405 ] os.utime() raises bad exception for unicode filenames Message-ID: Bugs item #947405, was opened at 2004-05-04 03:32 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947405&group_id=5470 Category: Windows Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Mark Hammond (mhammond) >Assigned to: Mark Hammond (mhammond) Summary: os.utime() raises bad exception for unicode filenames Initial Comment: os.utime handles unicode filenames - but when it raises an exception, it always uses the 'char *' filename. When unicode is passed to this function, the 'char *' is garbage. For example: python -c "import os;os.utime(u'\xe', None)" Traceback (most recent call last): File "", line 1, in ? OSError: [Errno 2] No such file or directory: '\x05' Note the bad filename. Once this patch is applied, the output is: Traceback (most recent call last): File "", line 1, in ? OSError: [Errno 2] No such file or directory: u'\xe' The patch is against the 2.3 tree, but still applies against the head. I believe this should be checked into both places (and am happy to do it :) ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-05-04 09:35 Message: Logged In: YES user_id=11105 Patch looks good, if you ask me. Please check in quickly, so that this goes into 2.3.4. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947405&group_id=5470 From noreply at sourceforge.net Tue May 4 04:11:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 04:11:10 2004 Subject: [ python-Bugs-947405 ] os.utime() raises bad exception for unicode filenames Message-ID: Bugs item #947405, was opened at 2004-05-04 11:32 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947405&group_id=5470 Category: Windows Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Mark Hammond (mhammond) Assigned to: Mark Hammond (mhammond) Summary: os.utime() raises bad exception for unicode filenames Initial Comment: os.utime handles unicode filenames - but when it raises an exception, it always uses the 'char *' filename. When unicode is passed to this function, the 'char *' is garbage. For example: python -c "import os;os.utime(u'\xe', None)" Traceback (most recent call last): File "", line 1, in ? OSError: [Errno 2] No such file or directory: '\x05' Note the bad filename. Once this patch is applied, the output is: Traceback (most recent call last): File "", line 1, in ? OSError: [Errno 2] No such file or directory: u'\xe' The patch is against the 2.3 tree, but still applies against the head. I believe this should be checked into both places (and am happy to do it :) ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-05-04 18:11 Message: Logged In: YES user_id=14198 Fixed! 2.3: Checking in posixmodule.c; new revision: 2.300.8.9; previous revision: 2.300.8.8 head: new revision: 2.317; previous revision: 2.316 ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-04 17:35 Message: Logged In: YES user_id=11105 Patch looks good, if you ask me. Please check in quickly, so that this goes into 2.3.4. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947405&group_id=5470 From noreply at sourceforge.net Tue May 4 05:57:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 05:58:03 2004 Subject: [ python-Bugs-947571 ] urlib.urlopen() fails to raise exception Message-ID: Bugs item #947571, was opened at 2004-05-04 11:57 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947571&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: M.-A. Lemburg (lemburg) Assigned to: Nobody/Anonymous (nobody) Summary: urlib.urlopen() fails to raise exception Initial Comment: I've come across a strange problem: even though the docs say that urllib.urlopen() should raise an IOError for server errors (e.g. 404s), all versions of Python that I've tested (1.5.2 - 2.3) fail to do so. Example: >>> import urllib >>> f = urllib.urlopen('http://www.example.net/this-url-does-not-exist/') >>> print f.read() 404 Not Found

Not Found

The requested URL /this-url-does-not-exist/ was not found on this server.

Apache/1.3.27 Server at www.example.com Port 80
Either the docs are wrong or the implementation has a really long standing bug or I am missing something. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947571&group_id=5470 From noreply at sourceforge.net Tue May 4 05:58:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 05:58:50 2004 Subject: [ python-Bugs-947571 ] urllib.urlopen() fails to raise exception Message-ID: Bugs item #947571, was opened at 2004-05-04 11:57 Message generated for change (Settings changed) made by lemburg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947571&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: M.-A. Lemburg (lemburg) Assigned to: Nobody/Anonymous (nobody) >Summary: urllib.urlopen() fails to raise exception Initial Comment: I've come across a strange problem: even though the docs say that urllib.urlopen() should raise an IOError for server errors (e.g. 404s), all versions of Python that I've tested (1.5.2 - 2.3) fail to do so. Example: >>> import urllib >>> f = urllib.urlopen('http://www.example.net/this-url-does-not-exist/') >>> print f.read() 404 Not Found

Not Found

The requested URL /this-url-does-not-exist/ was not found on this server.

Apache/1.3.27 Server at www.example.com Port 80
Either the docs are wrong or the implementation has a really long standing bug or I am missing something. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947571&group_id=5470 From noreply at sourceforge.net Tue May 4 10:00:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 10:01:02 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Tue May 4 10:44:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 10:44:52 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Tue May 4 14:20:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 14:20:05 2004 Subject: [ python-Bugs-947894 ] calendar.weekheader() undocumented Message-ID: Bugs item #947894, was opened at 2004-05-04 15:20 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947894&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Leonardo Rochael Almeida (rochael) Assigned to: Nobody/Anonymous (nobody) Summary: calendar.weekheader() undocumented Initial Comment: http://www.python.org/doc/current/lib/module-calendar.html makes no mention of calendar.weekheader() ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947894&group_id=5470 From noreply at sourceforge.net Tue May 4 14:38:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 14:38:35 2004 Subject: [ python-Bugs-947906 ] calendar.weekheader(n): n should mean chars not bytes Message-ID: Bugs item #947906, was opened at 2004-05-04 15:38 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947906&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Leonardo Rochael Almeida (rochael) Assigned to: Nobody/Anonymous (nobody) Summary: calendar.weekheader(n): n should mean chars not bytes Initial Comment: calendar.weekheader(n) is locale aware, which is good in principle. The parameter n, however, is interpreted as meaning bytes, not chars, which can generate broken strings for, e.g. localized weekday names: >>> calendar.weekheader(2) 'Mo Tu We Th Fr Sa Su' >>> locale.setlocale(locale.LC_ALL, "pt_BR.UTF-8") 'pt_BR.UTF-8' >>> calendar.weekheader(2) 'Se Te Qu Qu Se S\xc3 Do' Notice how "S?bado" (Saturday) above is missing the second utf-8 byte for the encoding of "?": >>> u"S?".encode("utf-8") 'S\xc3\xa1' The implementation of weekheader (and of all of calendar.py, it seems) is based on localized 8 bit strings. I suppose the correct fix for this bug will involve a roundtrip thru unicode. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947906&group_id=5470 From noreply at sourceforge.net Tue May 4 21:37:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 21:37:38 2004 Subject: [ python-Bugs-814606 ] 5454 - documentation wrong for ossaudiodev mixer device Message-ID: Bugs item #814606, was opened at 2003-09-29 14:16 Message generated for change (Settings changed) made by gward You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=814606&group_id=5470 Category: Documentation Group: None >Status: Closed Resolution: None Priority: 5 Submitted By: Alfred Morgan (zectbumo) Assigned to: Greg Ward (gward) Summary: 5454 - documentation wrong for ossaudiodev mixer device Initial Comment: http://www.python.org/doc/current/lib/mixer-device-objects.html methods controls() in the mixer device object have sample code that does not work. It seems that every reference in the document to the channels() method should be controls(). Also stereocontrols() references channels() and that should be controls() as well. -alfred python: 2.3 ---------------------------------------------------------------------- >Comment By: Greg Ward (gward) Date: 2004-05-04 21:37 Message: Logged In: YES user_id=14422 Fixed in Doc/lib/libossaudiodev.tex: rev 1.10.10.2 (2.3 maintenance branch) rev 1.12 (trunk) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=814606&group_id=5470 From noreply at sourceforge.net Tue May 4 21:40:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 21:40:16 2004 Subject: [ python-Feature Requests-819178 ] optparse "append" action should always make the em Message-ID: Feature Requests item #819178, was opened at 2003-10-07 06:47 Message generated for change (Settings changed) made by gward You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=819178&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jeremy Fincher (jemfinch) >Assigned to: Nobody/Anonymous (nobody) >Summary: optparse "append" action should always make the em Initial Comment: I was torn between "Bug" and "RFE" on this one. Anyway, with options whose action is "append", optparse.OptionParser will set the option to None if it's never called. Instead, I think it should set the option to the empty list, so code can iterate over the list regardless of whether the option was ever given. It keeps users from having to guard their iteration with "if options.foo:" Such a change would be slightly backwards-incompatible; users who use "if option.foo is None:" form instead of just "if not options.foo:" would run into trouble, as well as those users who used an "else" statement in their loops. The latter seemed like a big problem, until I realized that those users' code *already* guards against the empty case, and thus shouldn't run into difficulties. ---------------------------------------------------------------------- Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-04-01 03:16 Message: Logged In: YES user_id=3066 I'd agree that supplying [] as the default is the right thing to do, but the compatibility issue pretty much kills it. This should probably become a documentation issue; a hint to use default=[] would go a long way, and works with the existing code. ---------------------------------------------------------------------- Comment By: Jeremy Fincher (jemfinch) Date: 2003-10-07 06:48 Message: Logged In: YES user_id=99508 So, I *thought* I could change this to an RFE after submitting it, but apparently not. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=819178&group_id=5470 From noreply at sourceforge.net Tue May 4 21:42:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 21:42:17 2004 Subject: [ python-Feature Requests-819178 ] optparse "append" action should always make the em Message-ID: Feature Requests item #819178, was opened at 2003-10-07 06:47 Message generated for change (Settings changed) made by gward You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=819178&group_id=5470 Category: None Group: None >Status: Closed >Resolution: Rejected Priority: 5 Submitted By: Jeremy Fincher (jemfinch) Assigned to: Nobody/Anonymous (nobody) Summary: optparse "append" action should always make the em Initial Comment: I was torn between "Bug" and "RFE" on this one. Anyway, with options whose action is "append", optparse.OptionParser will set the option to None if it's never called. Instead, I think it should set the option to the empty list, so code can iterate over the list regardless of whether the option was ever given. It keeps users from having to guard their iteration with "if options.foo:" Such a change would be slightly backwards-incompatible; users who use "if option.foo is None:" form instead of just "if not options.foo:" would run into trouble, as well as those users who used an "else" statement in their loops. The latter seemed like a big problem, until I realized that those users' code *already* guards against the empty case, and thus shouldn't run into difficulties. ---------------------------------------------------------------------- >Comment By: Greg Ward (gward) Date: 2004-05-04 21:42 Message: Logged In: YES user_id=14422 First of all, this should have been filed with the 'optik' project, not Python. Oh well, no matter, because I'm going to refuse it anyways. Adding "default=[]" to the add_option() call is not such a burden, and Optik/optparse is mature enough that I don't want to break b/w compatibility. ---------------------------------------------------------------------- Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-04-01 03:16 Message: Logged In: YES user_id=3066 I'd agree that supplying [] as the default is the right thing to do, but the compatibility issue pretty much kills it. This should probably become a documentation issue; a hint to use default=[] would go a long way, and works with the existing code. ---------------------------------------------------------------------- Comment By: Jeremy Fincher (jemfinch) Date: 2003-10-07 06:48 Message: Logged In: YES user_id=99508 So, I *thought* I could change this to an RFE after submitting it, but apparently not. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=819178&group_id=5470 From noreply at sourceforge.net Tue May 4 21:45:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 21:46:18 2004 Subject: [ python-Feature Requests-927543 ] "with self:" statement Message-ID: Feature Requests item #927543, was opened at 2004-04-01 09:36 Message generated for change (Comment added) made by gward You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=927543&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Thomas Dunne (tomdunne) Assigned to: Nobody/Anonymous (nobody) Summary: "with self:" statement Initial Comment: please forgive if this is documented somewhere; I looked and searched and could not find any up to date PEPs or comments about the status of this idea. I managed to find a thread from 2002 that talked about this in great detail: http://mail.python.org/pipermail/python-list/2002-July/112888.html but there was no conclusion... I know it's not a new idea and it shows up often... but here goes: How about a "with self:" statement, but more in the style of (forgive) VB then Pascal, ie referenced items in the self instance are preceded with a dot. present syntax: with self: ??returnvalue = self.x + self.y * self.width proposed syntax: with self: ??returnvalue = .x + .y * .width If no preceding dot is used, you run into an ambiguity when assigning values. How do you know if the variable is to be assigned in self or to an object in a higher scope... the preceding dot avoids that problem since it explicitly tags self items. (using nested "with" statements will lead to the same problem when assigning values. Assignment will have to be to the innermost "with object". Reading values should work ok though. ) I understand that I could just simply use "s." instead of "self." or assign all needed values in one preceding line: ??x,y,width = self.x, self.y, self.width This is a question that shows up every so often, the points made in the thread are good: http://mail.python.org/pipermail/python-list/2002-July/113831.html has this been recently discussed? is there a patch or PEP? (I might be able to provide a patch...) ---------------------------------------------------------------------- >Comment By: Greg Ward (gward) Date: 2004-05-04 21:45 Message: Logged In: YES user_id=14422 A change of this magnitude demands a PEP. I don't recall having seen one, but I haven't been following python-dev for a while. If you want to do the right thing, write up a PEP and submit it to python-dev for the obligatory shooting-down-in-flames. Document the reasons why the feature was rejected in the PEP, and then the next time someone asks for it, your PEP can be trotted out. And don't take it personally when your idea gets shot down in flames. (Heck, who knows, maybe the python-dev crew will latch on and someone will implement it. But I doubt it...) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=927543&group_id=5470 From noreply at sourceforge.net Tue May 4 21:48:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 21:48:49 2004 Subject: [ python-Feature Requests-904770 ] http://doc.python.org Message-ID: Feature Requests item #904770, was opened at 2004-02-25 21:55 Message generated for change (Comment added) made by gward You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=904770&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Bob Ippolito (etrepum) Assigned to: Thomas Wouters (twouters) Summary: http://doc.python.org Initial Comment: If http://python.org/doc/ were mirrored at http://doc.python.org/ it would be much easier to search, especially via google. f.ex. "datetime site:doc.python.org" ---------------------------------------------------------------------- >Comment By: Greg Ward (gward) Date: 2004-05-04 21:48 Message: Logged In: YES user_id=14422 I just noticed that doc.python.org redirects to docs.python.org, which works just fine. Can this feature request be closed? ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-03-16 01:03 Message: Logged In: YES user_id=12800 +1 I believe Thomas Wouters has effective control over DNS now, so if he adds the host I'll set up Apache. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 16:42 Message: Logged In: YES user_id=80475 This use case seems pluasable enough to warrant consideration. Aahz, what do you think? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=904770&group_id=5470 From noreply at sourceforge.net Tue May 4 21:56:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 21:57:03 2004 Subject: [ python-Bugs-832236 ] Build fails in ossaudiodev.c with missing macros Message-ID: Bugs item #832236, was opened at 2003-10-29 04:05 Message generated for change (Comment added) made by gward You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=832236&group_id=5470 Category: Build Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Daniel Parks (danielparks) Assigned to: Greg Ward (gward) Summary: Build fails in ossaudiodev.c with missing macros Initial Comment: I'm building Python 2.3.2 with no configure flags on MkLinux Pre-R1. uname -a: Linux gondor.middle.earth 2.0.37-osfmach3 GENERIC_08alpha-20 Fri Jul 30 11:07:38 PDT 1999 ppc unknown The missing macros are: SOUND_MIXER_DIGITAL1, SOUND_MIXER_DIGITAL2, SOUND_MIXER_DIGITAL3, SOUND_MIXER_PHONEIN, SOUND_MIXER_PHONEOUT, SOUND_MIXER_VIDEO, SOUND_MIXER_RADIO, SOUND_MIXER_MONITOR, SNDCTL_DSP_GETODELAY I commented out two lines in setup.py and it seems to have built correctly. I would test it, but I need to go to bed, and I will forget to add this bug tomorrow. I will update this if Python actually works. ---------------------------------------------------------------------- >Comment By: Greg Ward (gward) Date: 2004-05-04 21:56 Message: Logged In: YES user_id=14422 Attached patch should fix the problem; I've emailed it to the original reporter to verify that it does. Will checkin/merge/close slip when I hear from him, or in a few days if I don't hear from him. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2003-12-23 14:21 Message: Logged In: YES user_id=11375 The fix is probably straightforward; add #ifdef ...#endif around each use of the problematic macros. If you do this, please submit a patch. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=832236&group_id=5470 From noreply at sourceforge.net Tue May 4 21:59:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 21:59:18 2004 Subject: [ python-Feature Requests-904770 ] http://doc.python.org Message-ID: Feature Requests item #904770, was opened at 2004-02-25 21:55 Message generated for change (Comment added) made by etrepum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=904770&group_id=5470 Category: Documentation Group: None >Status: Closed Resolution: None Priority: 5 Submitted By: Bob Ippolito (etrepum) Assigned to: Thomas Wouters (twouters) Summary: http://doc.python.org Initial Comment: If http://python.org/doc/ were mirrored at http://doc.python.org/ it would be much easier to search, especially via google. f.ex. "datetime site:doc.python.org" ---------------------------------------------------------------------- >Comment By: Bob Ippolito (etrepum) Date: 2004-05-04 21:59 Message: Logged In: YES user_id=139309 Yeah, seems like it happened -- I didn't even know about it! :) ---------------------------------------------------------------------- Comment By: Greg Ward (gward) Date: 2004-05-04 21:48 Message: Logged In: YES user_id=14422 I just noticed that doc.python.org redirects to docs.python.org, which works just fine. Can this feature request be closed? ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-03-16 01:03 Message: Logged In: YES user_id=12800 +1 I believe Thomas Wouters has effective control over DNS now, so if he adds the host I'll set up Apache. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-13 16:42 Message: Logged In: YES user_id=80475 This use case seems pluasable enough to warrant consideration. Aahz, what do you think? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=904770&group_id=5470 From noreply at sourceforge.net Tue May 4 22:01:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 4 22:01:41 2004 Subject: [ python-Feature Requests-819178 ] optparse "append" action should always make the em Message-ID: Feature Requests item #819178, was opened at 2003-10-07 05:47 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=819178&group_id=5470 Category: None Group: None Status: Closed Resolution: Rejected Priority: 5 Submitted By: Jeremy Fincher (jemfinch) >Assigned to: Greg Ward (gward) Summary: optparse "append" action should always make the em Initial Comment: I was torn between "Bug" and "RFE" on this one. Anyway, with options whose action is "append", optparse.OptionParser will set the option to None if it's never called. Instead, I think it should set the option to the empty list, so code can iterate over the list regardless of whether the option was ever given. It keeps users from having to guard their iteration with "if options.foo:" Such a change would be slightly backwards-incompatible; users who use "if option.foo is None:" form instead of just "if not options.foo:" would run into trouble, as well as those users who used an "else" statement in their loops. The latter seemed like a big problem, until I realized that those users' code *already* guards against the empty case, and thus shouldn't run into difficulties. ---------------------------------------------------------------------- Comment By: Greg Ward (gward) Date: 2004-05-04 20:42 Message: Logged In: YES user_id=14422 First of all, this should have been filed with the 'optik' project, not Python. Oh well, no matter, because I'm going to refuse it anyways. Adding "default=[]" to the add_option() call is not such a burden, and Optik/optparse is mature enough that I don't want to break b/w compatibility. ---------------------------------------------------------------------- Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-04-01 03:16 Message: Logged In: YES user_id=3066 I'd agree that supplying [] as the default is the right thing to do, but the compatibility issue pretty much kills it. This should probably become a documentation issue; a hint to use default=[] would go a long way, and works with the existing code. ---------------------------------------------------------------------- Comment By: Jeremy Fincher (jemfinch) Date: 2003-10-07 05:48 Message: Logged In: YES user_id=99508 So, I *thought* I could change this to an RFE after submitting it, but apparently not. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=819178&group_id=5470 From noreply at sourceforge.net Wed May 5 00:02:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 00:02:35 2004 Subject: [ python-Bugs-882332 ] boolobject.h documentation missing from Python/C API Message-ID: Bugs item #882332, was opened at 2004-01-22 14:13 Message generated for change (Settings changed) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=882332&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Greg Chapman (glchapman) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: boolobject.h documentation missing from Python/C API Initial Comment: The items in boolobject.h do not appear to be documented in the current documentation. I just checked the index of the online docs and PyBool_FromLong, Py_True, and Py_False are all missing. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=882332&group_id=5470 From noreply at sourceforge.net Wed May 5 00:03:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 00:03:34 2004 Subject: [ python-Bugs-889154 ] PyBool_* not documented in C API Message-ID: Bugs item #889154, was opened at 2004-02-02 11:04 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=889154&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Duplicate Priority: 5 Submitted By: Fred L. Drake, Jr. (fdrake) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: PyBool_* not documented in C API Initial Comment: The PyBool_* values and functions in the C API are not documented; these should get a section in the "Concrete Objects Layer" chapter. Reported via email by Paul DuBois. ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-05 00:03 Message: Logged In: YES user_id=3066 duplicate of #882332 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=889154&group_id=5470 From noreply at sourceforge.net Wed May 5 00:18:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 00:18:44 2004 Subject: [ python-Bugs-837929 ] socket.gethostbyname raises gaierror, not herror Message-ID: Bugs item #837929, was opened at 2003-11-07 10:11 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=837929&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Per Cederqvist (ceder) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: socket.gethostbyname raises gaierror, not herror Initial Comment: http://www.python.org/doc/current/lib/module-socket.html states that socket.gethostbyaddr() raises herror if the host isn't found. That isn't true in Python 2.3.2: Python 2.3.2 (#1, Oct 5 2003, 12:10:05) [GCC 3.3.1] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import socket >>> socket.gethostbyname('finnsinte.ingate.se') Traceback (most recent call last): File "", line 1, in ? socket.gaierror: (-2, 'Name or service not known') >>> Also, the description of gaierror should mention that the error value is set to one of the socket.EAI_* constants. ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-05 00:18 Message: Logged In: YES user_id=3066 The offending misinformation about which exception is raised has already been removed from CVS. I've added a note about gaierror and the EAI_* constants for Python 2.3.4 and 2.4. Doc/lib/libsocket.tex revisions 1.82, 1.76.6.2 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=837929&group_id=5470 From noreply at sourceforge.net Wed May 5 00:57:34 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 00:57:39 2004 Subject: [ python-Bugs-873246 ] pickle: __getinitargs__ not used for new-style objects Message-ID: Bugs item #873246, was opened at 2004-01-08 14:17 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=873246&group_id=5470 Category: Documentation Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Fred L. Drake, Jr. (fdrake) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: pickle: __getinitargs__ not used for new-style objects Initial Comment: The documentation (libref section 3.14.5.1) doesn't state that __getinitargs__() is not supported on new-style objects. The pickle docs are generally in a sad state. ;-( ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-05 00:57 Message: Logged In: YES user_id=3066 Noted that __getinitargs__() is only used for old-style classes. Also described __getnewargs__() for new-style types as used with protocol 2. Doc/lib/libpickle.tex revisions 1.48, 1.49 (index entry), 1.42.10.3 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=873246&group_id=5470 From noreply at sourceforge.net Wed May 5 04:28:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 04:28:56 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 23:28 Message generated for change (Comment added) made by kjetilja You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 08:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 14:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 14:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 23:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Wed May 5 08:24:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 08:25:30 2004 Subject: [ python-Bugs-738973 ] urllib2 CacheFTPHandler doesn't work on multiple dirs Message-ID: Bugs item #738973, was opened at 2003-05-17 08:18 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=738973&group_id=5470 Category: Python Library Group: Python 2.2.2 Status: Open Resolution: None >Priority: 6 Submitted By: Shahms E. King (shahms) >Assigned to: Mark Hammond (mhammond) Summary: urllib2 CacheFTPHandler doesn't work on multiple dirs Initial Comment: using the CacheFTPHandler for ftp requests in urllib2 works as expected for files in the same directory as the original file, however, as ftpwrapper() changes the directory only after the initial connection, any urllib2.urlopen('ftp://...') that is in a different directory that the initial urlopen() call will fail with a "450: File Not Found" ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-05-05 22:24 Message: Logged In: YES user_id=14198 I reviewed and tested the patch, and it looks good to me. I'll take this on (for trunk and 2.3.4) unless someone objects. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2003-12-01 09:12 Message: Logged In: YES user_id=261020 I've submitted a fix for this as patch 851736. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=738973&group_id=5470 From noreply at sourceforge.net Wed May 5 08:26:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 08:26:55 2004 Subject: [ python-Bugs-846133 ] os.chmod/os.utime/shutil do not work with unicode filenames Message-ID: Bugs item #846133, was opened at 2003-11-21 08:27 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=846133&group_id=5470 Category: Unicode Group: None >Status: Closed Resolution: None Priority: 5 Submitted By: Eric Meyer (meyeet) Assigned to: Mark Hammond (mhammond) Summary: os.chmod/os.utime/shutil do not work with unicode filenames Initial Comment: I have a filename that contains Kanji characters and I'm trying change the permissions on the file. I am running Python 2.3.1 on Windows 2000. Also I have the japanese language pack installed so that I can view the kanji characters in Windows explorer. >>> part u'\u5171\u6709\u3055\u308c\u308b.txt' >>> os.chmod(part, 0777) Traceback (most recent call last): File "", line 1, in ? OSError: [Errno 22] Invalid argument: '?????.txt' >>> I attached the above named file for you to test against. Thanks. ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-05-05 22:26 Message: Logged In: YES user_id=14198 I'm fairly sure this has been nailed (including the test failure) for some time? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-12-04 18:18 Message: Logged In: YES user_id=21627 2.3 maint should be fine: the problems are more likely in the new test cases than in the code itself. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2003-12-04 06:21 Message: Logged In: YES user_id=31435 meyeet, 2.3.3 should be released this month (December). Mark, I reopened this, because test_unicode_filename fails on Win98SE now (see Python-Dev report; that was on the trunk; I don't know about 2.3 maint). ---------------------------------------------------------------------- Comment By: Eric Meyer (meyeet) Date: 2003-12-04 06:16 Message: Logged In: YES user_id=913976 Is there an approximate date (or month) when 2.3.3 is likely to be released? ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2003-12-03 12:33 Message: Logged In: YES user_id=14198 release23-maint: Checking in posixmodule.c; new revision: 2.300.8.5; previous revision: 2.300.8.4 trunk: Checking in posixmodule.c; new revision: 2.309; previous revision: 2.308 Checking in test_support.py; new revision: 1.59; previous revision: 1.58 Checking in test_unicode_file.py; new revision: 1.11; previous revision: 1.10 Removing output/test_unicode_file; new revision: delete; previous revision: 1.1 ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-12-02 08:39 Message: Logged In: YES user_id=21627 The patches to posixmodule.c are fine for both 2.3 and 2.4. Can you apply them before 2.3.3 is frozen? The patches to the test suite are fine for 2.4 only, and they probably need to be relaxed. For example, on OSX, there simply is no file name that fails to work for the normal file system API: the file system encoding is UTF-8, so it supports all file names. You should consider changing test_pep277.py instead. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2003-11-29 12:32 Message: Logged In: YES user_id=14198 I created www.python.org/sf/850997 about the MBCS encoding issue. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2003-11-28 20:58 Message: Logged In: YES user_id=14198 I opened http://www.python.org/sf/846133 regarding os.utime, which I found via the "shutil" module, via SpamBayes, also on a Japanese system (see that bug for details), but then I saw this and decided to tackle them both. I rolled my fix for that in with a fix for chmod. I also hacked the test suite radically: * Creation of a test_support.TESTFN_UNICODE_UNENCODEABLE variable, which is a Unicode string that can *not* be encoded using the file system encoding. This will cause functions with 'encoding' support but without Unicode support (such as utime/chmod) to fail. * Made functions of all the test cases, so more combinations of unicode/encoded can be tested. Many are redundant, but that is OK. * Added shutil tests of the filenames * While I was there, converted to a unittest test. The new test case blows up with a couple of errors before the posixmodule patch is applied, and passes after. Note that shutil.move/copy etc can not handle being passed one string and one unicode arg, and therefore this combination is skipped. I'd like any opinions on whether this is a bug in shutil or not. Also note that the new comment in test_support.py regarding a potential bug in the 'mbcs' encoding - it appears as if it always works as though errors=ignore. Comments/reviews? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-11-25 09:21 Message: Logged In: YES user_id=21627 If you look at the source of os.chmod, it is not at all surprising that it does not work for characters outside the file system encoding: it is simply not implemented. Patches are welcome. ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2003-11-22 11:51 Message: Logged In: YES user_id=671362 Hi, Eric. My previous post was maybe wrong. This is the problem of os.chmod. I've confirmed two kinds of exceptions are raised when using os.chmod for unicode filenames. The first one is [Errno 22] Invalid argument. You can read/write a file but cannot use os.chmod. The second one is [Errno 2] No such file or directory. Although there exists a file, Python complains "No such file or directory" test.test_codecs has a bunch of international unicode characters, so I borrowed them for testing. >>> import os >>> from test.test_codecs import punycode_testcases >>> def unicode_test(name): try: f = file(name, 'w') f.close() except IOError, e: print e return try: os.chmod(name, 0777) except OSError, e: print e >>> for i, (uni, puny) in enumerate (punycode_testcases): print i unicode_test(uni) I ran this script on Windows 2000(Japanese edition) using Python 2.3 and got "[Errno 22]" for 0,1,2,3,4,5,7,10 and "[Errno 2]" for 9. ---------------------------------------------------------------------- Comment By: Eric Meyer (meyeet) Date: 2003-11-22 03:18 Message: Logged In: YES user_id=913976 George, I tried the following but I had to specify one of the japanese codecs during the unicode() call. What is your default encoding set to? Below are my results. >>> import os >>> os.listdir('.') [] >>> u1 = unicode('\x82\xa0', 'cp932') >>> u2 = u'\x82\xa0' >>> u1, u2 (u'\u3042', u'\x82\xa0') >>> print >> file(u1, 'w'), "hello world" >>> os.listdir('.') ['?'] >>> os.chmod(u1, 0777) Traceback (most recent call last): File "", line 1, in ? OSError: [Errno 22] Invalid argument: '?' ---------------------------------------------------------------------- Comment By: George Yoshida (quiver) Date: 2003-11-21 11:07 Message: Logged In: YES user_id=671362 I'm running Python in almost the same environment. I guess this results from the different bihavior of u'' and unicode(''). If you convert a multi-byte character to a unicode character, u'' and unicode('') don't return the same string. unicode'' works as intended but u'' doesn't. This is probably caused by the bug of Japanese codecs package. Eric, please try the session below and tell me what happens. NOTE: Japanese codecs needs to be installed to test the code below. Otherwise, UnicodeDecodeError will be raised. --- >>> import os >>> os.listdir('.') [] >>> lst = ['\x82', '\xa0'] # japanese character >>> u1 = unicode('\x82\xa0') >>> u2 = u'\x82\xa0' >>> u1 == u2 False >>> u1, u2 (u'\u3042', u'\x82\xa0') # u2 is odd >>> print >> file(u1, 'w'), "hello world" >>> os.listdir('.') ['あ'] >>> os.chmod(u1, 0777) >>> os.chmod(u2, 0777) Traceback (most recent call last): File "", line 1, in -toplevel- os.chmod(u2, 0777) OSError: [Errno 22] Invalid argument: '??' ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=846133&group_id=5470 From noreply at sourceforge.net Wed May 5 08:30:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 08:31:05 2004 Subject: [ python-Bugs-604387 ] long list in Pythonwin -> weird text Message-ID: Bugs item #604387, was opened at 2002-09-04 16:36 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=604387&group_id=5470 Category: Windows Group: 3rd Party >Status: Closed >Resolution: Invalid Priority: 2 Submitted By: James Briggs (jbriggs66) Assigned to: Mark Hammond (mhammond) Summary: long list in Pythonwin -> weird text Initial Comment: Using Pythonwin (build 148) with ActiveState 2.2.1 >>> import sys >>> sys.modules.keys() ['code', 'pywin.framework.cmdline', .... after about 15 lines the output goes weird. Text is mostly stuck on 1st column, some text appears and disappears as the scroll bar is moved. Tried this with Python 2.1 and Pythonwin (build 147 also) ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-05-05 22:30 Message: Logged In: YES user_id=14198 This should be in the pywin32 bug database, which will say it should be in the scintilla bug list, which will mark it "wont fix" :) Pythonwin gets around it these days by setting scintilla's word-wrap feature on, but an editor like scintilla is always going to struggle handling huge lines both 100% correctly and quickly. ---------------------------------------------------------------------- Comment By: James Briggs (jbriggs66) Date: 2002-09-05 10:06 Message: Logged In: YES user_id=605472 Make that - screen width effects what you see, but not when the bug appears. ---------------------------------------------------------------------- Comment By: James Briggs (jbriggs66) Date: 2002-09-05 10:04 Message: Logged In: YES user_id=605472 Another folloup. All tests were done with the default window size, and full size window on 1024x768. Screen width effects the where you see the Changing to non-fixed size font gets the range(1,000) to work, but the display still goes wierd on the range(1,2000). Fonts sizes seem to make no difference. Fonts used were Fixed width - Courier New 10 pt, Proportional - Arial 10 pt. ---------------------------------------------------------------------- Comment By: James Briggs (jbriggs66) Date: 2002-09-05 09:06 Message: Logged In: YES user_id=605472 Yes, I should have mentioned it worked in IDLE and on cthe command line. Currently using Windows 2000 (Sp2). Any very long sequence with enough text seems to cause the problem. Try also: >>> range(1,1000) # will go wierd after 771 >>> s=str(range(1,780)) It seems to be when the total string length gets around 3750 characters. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2002-09-05 01:35 Message: Logged In: YES user_id=31435 Note that PythonWin isn't part of the core Python project, so marked 3rdParty and assigned to Mark Hammond. I don't see any problems under a DOS box, or under IDLE, so if there is a problem here, it seems unlikely to be in the core. Help Mark by giving more info: which flavor of Windows are you using? Is it a non-US version (French, Japanese, etc)? Is the problem specific to sys.modules.keys(), or is it common across every list that requires more than "about 15 lines" to display?: How wide is your display window? Which font and point size are you using? Etc. As is, I doubt there's enough info here for anyone to help. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=604387&group_id=5470 From noreply at sourceforge.net Wed May 5 08:57:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 08:57:54 2004 Subject: [ python-Bugs-602245 ] os.popen() negative error code IOError Message-ID: Bugs item #602245, was opened at 2002-08-30 16:25 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=602245&group_id=5470 Category: Windows Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Tim Delaney (tcdelaney) Assigned to: Mark Hammond (mhammond) Summary: os.popen() negative error code IOError Initial Comment: Windows 2000, Python 2.0.1, 2.1.1, 2.2. When a negative return code is received by the os.popen () family, an IOError is raised when the last pipe from the process is closed. The following code demonstrates the problem: import sys import os import traceback if __name__ == '__main__': if len(sys.argv) == 1: try: r = os.popen('%s %s %s' % (sys.executable, sys.argv[0], -1,)) r.close() except IOError: traceback.print_exc() try: w, r = os.popen2('%s %s %s' % (sys.executable, sys.argv[0], -1,)) w.close() r.close() except IOError: traceback.print_exc() try: w, r, e = os.popen3('%s %s %s' % (sys.executable, sys.argv[0], -1,)) w.close() r.close() e.close() except IOError: traceback.print_exc() else: sys.exit(int(sys.argv[1])) ---------- Run ---------- Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 11, in ? r.close() IOError: (0, 'Error') Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 18, in ? r.close() IOError: (0, 'Error') Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 26, in ? e.close() IOError: (0, 'Error') ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-05-05 22:57 Message: Logged In: YES user_id=14198 I'm not sure what to do about this. Python internally checks for -1 from the close function. The safest thing to do would be to translate the return code from -1 to something else. Changing the fclose semantics to check for PyErr_Occurred is another alternative, but doesn't seem that reasonable just for this. Any thoughts? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2002-08-31 00:05 Message: Logged In: YES user_id=6380 I suppose this is one for Mark Hammond... ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2002-08-30 22:38 Message: Logged In: YES user_id=44345 Attached Tim's code so it's more readily accessible to testers. BTW, works fine for me on Linux. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=602245&group_id=5470 From noreply at sourceforge.net Wed May 5 09:18:57 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 09:19:04 2004 Subject: [ python-Bugs-602245 ] os.popen() negative error code IOError Message-ID: Bugs item #602245, was opened at 2002-08-30 02:25 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=602245&group_id=5470 Category: Windows Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Tim Delaney (tcdelaney) Assigned to: Mark Hammond (mhammond) Summary: os.popen() negative error code IOError Initial Comment: Windows 2000, Python 2.0.1, 2.1.1, 2.2. When a negative return code is received by the os.popen () family, an IOError is raised when the last pipe from the process is closed. The following code demonstrates the problem: import sys import os import traceback if __name__ == '__main__': if len(sys.argv) == 1: try: r = os.popen('%s %s %s' % (sys.executable, sys.argv[0], -1,)) r.close() except IOError: traceback.print_exc() try: w, r = os.popen2('%s %s %s' % (sys.executable, sys.argv[0], -1,)) w.close() r.close() except IOError: traceback.print_exc() try: w, r, e = os.popen3('%s %s %s' % (sys.executable, sys.argv[0], -1,)) w.close() r.close() e.close() except IOError: traceback.print_exc() else: sys.exit(int(sys.argv[1])) ---------- Run ---------- Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 11, in ? r.close() IOError: (0, 'Error') Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 18, in ? r.close() IOError: (0, 'Error') Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 26, in ? e.close() IOError: (0, 'Error') ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-05 09:18 Message: Logged In: YES user_id=6380 How about changing the line "result = exit_code;" in _PyPclose() to truncate the value to the lower 16 bits, ensuring it is always >= 0? I think that should do it. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-05 08:57 Message: Logged In: YES user_id=14198 I'm not sure what to do about this. Python internally checks for -1 from the close function. The safest thing to do would be to translate the return code from -1 to something else. Changing the fclose semantics to check for PyErr_Occurred is another alternative, but doesn't seem that reasonable just for this. Any thoughts? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2002-08-30 10:05 Message: Logged In: YES user_id=6380 I suppose this is one for Mark Hammond... ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2002-08-30 08:38 Message: Logged In: YES user_id=44345 Attached Tim's code so it's more readily accessible to testers. BTW, works fine for me on Linux. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=602245&group_id=5470 From noreply at sourceforge.net Wed May 5 10:36:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 10:36:56 2004 Subject: [ python-Bugs-948517 ] LaTeX not required Message-ID: Bugs item #948517, was opened at 2004-05-05 07:36 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=948517&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Aahz (aahz) Assigned to: Nobody/Anonymous (nobody) Summary: LaTeX not required Initial Comment: /doc.html and about.html ("Comments and Questions") should make clear that LaTeX is not required for submitting additions and changes to the docs. Suggested wording: Although we prefer additions and changes to be in the format prescribed by /doc.html, we welcome plain-text, too -- we're happy to handle the formatting for you. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=948517&group_id=5470 From noreply at sourceforge.net Wed May 5 12:59:13 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 12:59:18 2004 Subject: [ python-Bugs-944119 ] Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty Message-ID: Bugs item #944119, was opened at 2004-04-28 18:05 Message generated for change (Comment added) made by levis501 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944119&group_id=5470 Category: None Group: Platform-specific >Status: Deleted Resolution: None Priority: 5 Submitted By: Steve Levis (levis501) Assigned to: Nobody/Anonymous (nobody) Summary: Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty Initial Comment: The attached python file reproduces a bug whereby Python does not receive alarm signals from FreeBSD 4.9. This also appears to be the case in FreeBSD 4.8, and may be related to the addition of HyperThreading in FreeBSD 4.8. If Python (either 2.2.3 or 2.3.3) is configured with -?without-threads before compiling, the bug does not exist. ---------------------------------------------------------------------- >Comment By: Steve Levis (levis501) Date: 2004-05-05 09:59 Message: Logged In: YES user_id=309698 I was apparently not running on a standard FreeBSD kernel. It's working now. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944119&group_id=5470 From noreply at sourceforge.net Wed May 5 13:08:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 13:08:55 2004 Subject: [ python-Bugs-944119 ] Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty Message-ID: Bugs item #944119, was opened at 2004-04-28 18:05 Message generated for change (Comment added) made by levis501 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944119&group_id=5470 Category: None Group: Platform-specific >Status: Open Resolution: None Priority: 5 Submitted By: Steve Levis (levis501) Assigned to: Nobody/Anonymous (nobody) Summary: Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty Initial Comment: The attached python file reproduces a bug whereby Python does not receive alarm signals from FreeBSD 4.9. This also appears to be the case in FreeBSD 4.8, and may be related to the addition of HyperThreading in FreeBSD 4.8. If Python (either 2.2.3 or 2.3.3) is configured with -?without-threads before compiling, the bug does not exist. ---------------------------------------------------------------------- >Comment By: Steve Levis (levis501) Date: 2004-05-05 10:08 Message: Logged In: YES user_id=309698 Ooops, again! I ran on the standard kernel but had python compiled --without-threads, so of course it worked. So, to make a long story short, the bug exists -- ignore my last post. ---------------------------------------------------------------------- Comment By: Steve Levis (levis501) Date: 2004-05-05 09:59 Message: Logged In: YES user_id=309698 I was apparently not running on a standard FreeBSD kernel. It's working now. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944119&group_id=5470 From noreply at sourceforge.net Wed May 5 13:10:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 13:10:16 2004 Subject: [ python-Bugs-810879 ] shutil.copyfile fails when dst exists read-only Message-ID: Bugs item #810879, was opened at 2003-09-22 19:21 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=810879&group_id=5470 Category: Windows Group: Python 2.2.3 Status: Open Resolution: None Priority: 5 Submitted By: Ray Pasco (pascor) Assigned to: A.M. Kuchling (akuchling) Summary: shutil.copyfile fails when dst exists read-only Initial Comment: (WIN32): shutil.copyfile fails when the dst file already exists, but in read-only mode. As the old file is about to be clobbered, a fix I've been using is to delete the orginal file first. The attached my be inelegant, but works fine ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-05-05 13:10 Message: Logged In: YES user_id=11375 On Unix it fails with the same os.error. I'll tweak the wording of the docs a bit, and then close this bug. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-02-16 18:52 Message: Logged In: YES user_id=31435 Andrew, what does this do on Unix? I don't *want* copyfile() to clobber read-only files, so I view this report as a small lack in the docs (which unconditionally say dst will be replaced). If the user lacks sufficient write permissions on Unix (a decent approximation to Windows "read only" state), does copyfile() succeed anyway? I sure hope not. On Windows, fdst = open(dst, 'wb') dies with a (thoroughly appropriate, according to me) IOError: [Errno 13] Permission denied: when dst is read-only. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2003-09-25 14:37 Message: Logged In: YES user_id=11375 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=810879&group_id=5470 From noreply at sourceforge.net Wed May 5 13:22:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 13:22:31 2004 Subject: [ python-Bugs-810879 ] shutil.copyfile fails when dst exists read-only Message-ID: Bugs item #810879, was opened at 2003-09-22 19:21 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=810879&group_id=5470 Category: Windows Group: Python 2.2.3 >Status: Closed >Resolution: Wont Fix Priority: 5 Submitted By: Ray Pasco (pascor) Assigned to: A.M. Kuchling (akuchling) Summary: shutil.copyfile fails when dst exists read-only Initial Comment: (WIN32): shutil.copyfile fails when the dst file already exists, but in read-only mode. As the old file is about to be clobbered, a fix I've been using is to delete the orginal file first. The attached my be inelegant, but works fine ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-05-05 13:22 Message: Logged In: YES user_id=11375 Closing. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2004-05-05 13:10 Message: Logged In: YES user_id=11375 On Unix it fails with the same os.error. I'll tweak the wording of the docs a bit, and then close this bug. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-02-16 18:52 Message: Logged In: YES user_id=31435 Andrew, what does this do on Unix? I don't *want* copyfile() to clobber read-only files, so I view this report as a small lack in the docs (which unconditionally say dst will be replaced). If the user lacks sufficient write permissions on Unix (a decent approximation to Windows "read only" state), does copyfile() succeed anyway? I sure hope not. On Windows, fdst = open(dst, 'wb') dies with a (thoroughly appropriate, according to me) IOError: [Errno 13] Permission denied: when dst is read-only. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2003-09-25 14:37 Message: Logged In: YES user_id=11375 There's no uploaded file! You have to check the checkbox labeled "Check to Upload & Attach File" when you upload a file. Please try again. (This is a SourceForge annoyance that we can do nothing about. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=810879&group_id=5470 From noreply at sourceforge.net Wed May 5 13:24:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 13:24:58 2004 Subject: [ python-Bugs-765819 ] unable to specify another compiler Message-ID: Bugs item #765819, was opened at 2003-07-04 05:04 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=765819&group_id=5470 Category: Distutils Group: Python 2.2.3 >Status: Closed Resolution: Works For Me Priority: 5 Submitted By: Matthias Klose (doko) Assigned to: A.M. Kuchling (akuchling) Summary: unable to specify another compiler Initial Comment: [forwarded from http://bugs.debian.org/197537] Assume, python was built using `gcc' (version 3.3), but an extension module needs to be built using `gcc-2.95', because the library it depends on is still built by g++-2.95, which has another ABI than g++-3.3. I'm unable (or fail to see how) I can overwrite the compiler used by distutils. ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-05-05 13:24 Message: Logged In: YES user_id=11375 At this point it doesn't seem likely there will be another 2.2.x release, so backporting the fix isn't worthwhile. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-07-06 06:14 Message: Logged In: YES user_id=21627 Please try this with python2.3 instead. See Lib/distutils/sysconfig.py:customize_compiler. 2.3 has a block if os.environ.has_key('CC'): cc = os.environ['CC'] if os.environ.has_key('CXX'): cxx = os.environ['CXX'] which was added through patch #588809. Regrouping this as 2.2.3 issue. I'm assigning this to Andrew for consideration whether the patch should be backported. As this is, strictly speaking, a new feature, it might not be eligible for back-porting. Meanwhile, modifying setup.py to read from distutils.sysconfig import parse_makefile, get_config_vars config_vars = get_config_vars() config_vars['CC'] = 'gcc-2.95' employs the other approach of configuration. ---------------------------------------------------------------------- Comment By: Matthias Klose (doko) Date: 2003-07-05 15:25 Message: Logged In: YES user_id=60903 Doesn't work for me. I can reproduce this at least for the python-apt package on Debian (apt-get source python-apt) CC=gcc-2.95 /usr/bin/python setup.py build gcc still gets used. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-07-05 12:25 Message: Logged In: YES user_id=21627 This is not a bug. The C compiler can be replaced by either setting the CC/CXX environment variable, or by modifying distutils.sysconfig._config_vars. Notice that replacing the C compiler may result in an inoperable module, if mixing different compilers is not supported in the system. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=765819&group_id=5470 From noreply at sourceforge.net Wed May 5 16:34:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 16:34:09 2004 Subject: [ python-Bugs-944396 ] urllib2 doesn't handle username/password in url Message-ID: Bugs item #944396, was opened at 2004-04-29 07:04 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944396&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Chris Withers (fresh) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2 doesn't handle username/password in url Initial Comment: >>> urllib2.urlopen('http://username:password@server') Traceback (most recent call last): File "", line 1, in ? File "C:\PYTHON23\lib\urllib2.py", line 129, in urlopen return _opener.open(url, data) File "C:\PYTHON23\lib\urllib2.py", line 326, in open '_open', req) File "C:\PYTHON23\lib\urllib2.py", line 306, in _call_chain result = func(*args) File "C:\PYTHON23\lib\urllib2.py", line 901, in http_open return self.do_open(httplib.HTTP, req) File "C:\PYTHON23\lib\urllib2.py", line 860, in do_open h = http_class(host) # will parse host:port File "C:\Python23\lib\httplib.py", line 1009, in __init__ self._setup(self._connection_class(host, port, strict)) File "C:\Python23\lib\httplib.py", line 507, in __init__ self._set_hostport(host, port) File "C:\Python23\lib\httplib.py", line 518, in _set_hostport raise InvalidURL("nonnumeric port: '%s'" % host[i+1:]) httplib.InvalidURL: nonnumeric port: 'password@server' cheers, Chris ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-05 16:34 Message: Logged In: YES user_id=119306 Although allowing a username and password in the URL is a common client extension, it is not part of the standard ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944396&group_id=5470 From noreply at sourceforge.net Wed May 5 16:48:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 16:48:44 2004 Subject: [ python-Feature Requests-927543 ] "with self:" statement Message-ID: Feature Requests item #927543, was opened at 2004-04-01 16:36 Message generated for change (Comment added) made by tomdunne You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=927543&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Thomas Dunne (tomdunne) Assigned to: Nobody/Anonymous (nobody) Summary: "with self:" statement Initial Comment: please forgive if this is documented somewhere; I looked and searched and could not find any up to date PEPs or comments about the status of this idea. I managed to find a thread from 2002 that talked about this in great detail: http://mail.python.org/pipermail/python-list/2002-July/112888.html but there was no conclusion... I know it's not a new idea and it shows up often... but here goes: How about a "with self:" statement, but more in the style of (forgive) VB then Pascal, ie referenced items in the self instance are preceded with a dot. present syntax: with self: ??returnvalue = self.x + self.y * self.width proposed syntax: with self: ??returnvalue = .x + .y * .width If no preceding dot is used, you run into an ambiguity when assigning values. How do you know if the variable is to be assigned in self or to an object in a higher scope... the preceding dot avoids that problem since it explicitly tags self items. (using nested "with" statements will lead to the same problem when assigning values. Assignment will have to be to the innermost "with object". Reading values should work ok though. ) I understand that I could just simply use "s." instead of "self." or assign all needed values in one preceding line: ??x,y,width = self.x, self.y, self.width This is a question that shows up every so often, the points made in the thread are good: http://mail.python.org/pipermail/python-list/2002-July/113831.html has this been recently discussed? is there a patch or PEP? (I might be able to provide a patch...) ---------------------------------------------------------------------- >Comment By: Thomas Dunne (tomdunne) Date: 2004-05-05 22:48 Message: Logged In: YES user_id=1005553 Hey Greg, thanks for the suggestion! I posted a PEP to peps@python.org, flame on! ;) (want a copy?) ---------------------------------------------------------------------- Comment By: Greg Ward (gward) Date: 2004-05-05 03:45 Message: Logged In: YES user_id=14422 A change of this magnitude demands a PEP. I don't recall having seen one, but I haven't been following python-dev for a while. If you want to do the right thing, write up a PEP and submit it to python-dev for the obligatory shooting-down-in-flames. Document the reasons why the feature was rejected in the PEP, and then the next time someone asks for it, your PEP can be trotted out. And don't take it personally when your idea gets shot down in flames. (Heck, who knows, maybe the python-dev crew will latch on and someone will implement it. But I doubt it...) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=927543&group_id=5470 From noreply at sourceforge.net Wed May 5 17:01:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 17:02:06 2004 Subject: [ python-Bugs-944396 ] urllib2 doesn't handle username/password in url Message-ID: Bugs item #944396, was opened at 2004-04-29 11:04 Message generated for change (Comment added) made by fresh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944396&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Chris Withers (fresh) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2 doesn't handle username/password in url Initial Comment: >>> urllib2.urlopen('http://username:password@server') Traceback (most recent call last): File "", line 1, in ? File "C:\PYTHON23\lib\urllib2.py", line 129, in urlopen return _opener.open(url, data) File "C:\PYTHON23\lib\urllib2.py", line 326, in open '_open', req) File "C:\PYTHON23\lib\urllib2.py", line 306, in _call_chain result = func(*args) File "C:\PYTHON23\lib\urllib2.py", line 901, in http_open return self.do_open(httplib.HTTP, req) File "C:\PYTHON23\lib\urllib2.py", line 860, in do_open h = http_class(host) # will parse host:port File "C:\Python23\lib\httplib.py", line 1009, in __init__ self._setup(self._connection_class(host, port, strict)) File "C:\Python23\lib\httplib.py", line 507, in __init__ self._set_hostport(host, port) File "C:\Python23\lib\httplib.py", line 518, in _set_hostport raise InvalidURL("nonnumeric port: '%s'" % host[i+1:]) httplib.InvalidURL: nonnumeric port: 'password@server' cheers, Chris ---------------------------------------------------------------------- >Comment By: Chris Withers (fresh) Date: 2004-05-05 21:01 Message: Logged In: YES user_id=24723 However, given that the original urllib supported this, it is suprising that urllib2 doesn't. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-05 20:34 Message: Logged In: YES user_id=119306 Although allowing a username and password in the URL is a common client extension, it is not part of the standard ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944396&group_id=5470 From noreply at sourceforge.net Wed May 5 17:01:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 17:02:12 2004 Subject: [ python-Feature Requests-927543 ] "with self:" statement Message-ID: Feature Requests item #927543, was opened at 2004-04-01 16:36 Message generated for change (Comment added) made by tomdunne You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=927543&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Thomas Dunne (tomdunne) Assigned to: Nobody/Anonymous (nobody) Summary: "with self:" statement Initial Comment: please forgive if this is documented somewhere; I looked and searched and could not find any up to date PEPs or comments about the status of this idea. I managed to find a thread from 2002 that talked about this in great detail: http://mail.python.org/pipermail/python-list/2002-July/112888.html but there was no conclusion... I know it's not a new idea and it shows up often... but here goes: How about a "with self:" statement, but more in the style of (forgive) VB then Pascal, ie referenced items in the self instance are preceded with a dot. present syntax: with self: ??returnvalue = self.x + self.y * self.width proposed syntax: with self: ??returnvalue = .x + .y * .width If no preceding dot is used, you run into an ambiguity when assigning values. How do you know if the variable is to be assigned in self or to an object in a higher scope... the preceding dot avoids that problem since it explicitly tags self items. (using nested "with" statements will lead to the same problem when assigning values. Assignment will have to be to the innermost "with object". Reading values should work ok though. ) I understand that I could just simply use "s." instead of "self." or assign all needed values in one preceding line: ??x,y,width = self.x, self.y, self.width This is a question that shows up every so often, the points made in the thread are good: http://mail.python.org/pipermail/python-list/2002-July/113831.html has this been recently discussed? is there a patch or PEP? (I might be able to provide a patch...) ---------------------------------------------------------------------- >Comment By: Thomas Dunne (tomdunne) Date: 2004-05-05 23:01 Message: Logged In: YES user_id=1005553 Hey Greg, completely OT, but thought you should know: your sourceforge address redirect isn't correct! I sent an email to gward at users.sourceforge.net and the host (don't want to post for everyone to see here) responded that the redirected address didn't exist... ---------------------------------------------------------------------- Comment By: Thomas Dunne (tomdunne) Date: 2004-05-05 22:48 Message: Logged In: YES user_id=1005553 Hey Greg, thanks for the suggestion! I posted a PEP to peps@python.org, flame on! ;) (want a copy?) ---------------------------------------------------------------------- Comment By: Greg Ward (gward) Date: 2004-05-05 03:45 Message: Logged In: YES user_id=14422 A change of this magnitude demands a PEP. I don't recall having seen one, but I haven't been following python-dev for a while. If you want to do the right thing, write up a PEP and submit it to python-dev for the obligatory shooting-down-in-flames. Document the reasons why the feature was rejected in the PEP, and then the next time someone asks for it, your PEP can be trotted out. And don't take it personally when your idea gets shot down in flames. (Heck, who knows, maybe the python-dev crew will latch on and someone will implement it. But I doubt it...) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=927543&group_id=5470 From noreply at sourceforge.net Wed May 5 17:22:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 17:28:35 2004 Subject: [ python-Bugs-944396 ] urllib2 doesn't handle username/password in url Message-ID: Bugs item #944396, was opened at 2004-04-29 13:04 Message generated for change (Comment added) made by sjoerd You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944396&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Chris Withers (fresh) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2 doesn't handle username/password in url Initial Comment: >>> urllib2.urlopen('http://username:password@server') Traceback (most recent call last): File "", line 1, in ? File "C:\PYTHON23\lib\urllib2.py", line 129, in urlopen return _opener.open(url, data) File "C:\PYTHON23\lib\urllib2.py", line 326, in open '_open', req) File "C:\PYTHON23\lib\urllib2.py", line 306, in _call_chain result = func(*args) File "C:\PYTHON23\lib\urllib2.py", line 901, in http_open return self.do_open(httplib.HTTP, req) File "C:\PYTHON23\lib\urllib2.py", line 860, in do_open h = http_class(host) # will parse host:port File "C:\Python23\lib\httplib.py", line 1009, in __init__ self._setup(self._connection_class(host, port, strict)) File "C:\Python23\lib\httplib.py", line 507, in __init__ self._set_hostport(host, port) File "C:\Python23\lib\httplib.py", line 518, in _set_hostport raise InvalidURL("nonnumeric port: '%s'" % host[i+1:]) httplib.InvalidURL: nonnumeric port: 'password@server' cheers, Chris ---------------------------------------------------------------------- >Comment By: Sjoerd Mullender (sjoerd) Date: 2004-05-05 23:22 Message: Logged In: YES user_id=43607 I don't know (nor care) about RFC 1738, but it's successor RFC 2396 *does* mention @: as a possible "server". See section 3.2.2. I admit, it also says that it is not recommended, but it does specifically allow username + password in the URI. ---------------------------------------------------------------------- Comment By: Chris Withers (fresh) Date: 2004-05-05 23:01 Message: Logged In: YES user_id=24723 However, given that the original urllib supported this, it is suprising that urllib2 doesn't. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-05 22:34 Message: Logged In: YES user_id=119306 Although allowing a username and password in the URL is a common client extension, it is not part of the standard ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944396&group_id=5470 From noreply at sourceforge.net Wed May 5 19:42:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 19:43:04 2004 Subject: [ python-Bugs-948881 ] texi2html.py not working Message-ID: Bugs item #948881, was opened at 2004-05-06 08:42 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=948881&group_id=5470 Category: Demos and Tools Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Nobody/Anonymous (nobody) Summary: texi2html.py not working Initial Comment: Tools/scripts/texi2html.py have several syntactic problems(inproper indent level, unclosed brace, etc) and raises a SyntaxError. For example, at line 809 self.write(self.FN_SOURCE_PATTERN % {'id': repr(id)) should read self.write(self.FN_SOURCE_PATTERN % {'id': repr(id)}) Looks like Revision 1.14 to 1.16 introduced these bugs. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=948881&group_id=5470 From noreply at sourceforge.net Wed May 5 21:41:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 21:42:04 2004 Subject: [ python-Bugs-944082 ] urllib2 authentication mishandles empty password Message-ID: Bugs item #944082, was opened at 2004-04-29 02:02 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944082&group_id=5470 Category: Python Library Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: sc0rp (yangir) Assigned to: Nobody/Anonymous (nobody) Summary: urllib2 authentication mishandles empty password Initial Comment: If example.org requires authentication, then following code: host = 'example.org' user = 'testuser' password = '' url = 'http://%s/' % host authInfo = urllib2.HTTPPasswordMgrWithDefaultRealm() authInfo.add_password( None, host, user, password ) authHandler = urllib2.HTTPBasicAuthHandler( authInfo ) opener = urllib2.build_opener( authHandler ) urlFile = opener.open( url ) print urlFile.read() will die by throwing HTTPError 401: File "/usr/lib/python2.3/urllib2.py", line 419, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 401: Authorization Required even if authenticating with 'testuser' and empty password is valid. Empty password is mishandled (i.e. authentication with empty password string is ignored) in AbstractBasicAuthHandler.retry_http_basic_auth def retry_http_basic_auth(self, host, req, realm): user,pw = self.passwd.find_user_password(realm, host) if pw: [...] It can be fixed by changing: if pw: to if pw is not None: Python 2.3.2 (#1, Oct 9 2003, 12:03:29) [GCC 3.3.1 (cygming special)] on cygwin Type "help", "copyright", "credits" or "license" for more information. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-06 03:41 Message: Logged In: YES user_id=21627 This is fixed with patch #944110. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944082&group_id=5470 From noreply at sourceforge.net Wed May 5 21:56:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 21:56:11 2004 Subject: [ python-Bugs-780300 ] pyexpat LexicalHandler swaps system_id and public_id Message-ID: Bugs item #780300, was opened at 2003-07-30 17:35 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=780300&group_id=5470 Category: XML Group: Python 2.2.3 >Status: Closed >Resolution: Fixed Priority: 7 Submitted By: Holger Floerke (floerke) Assigned to: Martin v. L?wis (loewis) Summary: pyexpat LexicalHandler swaps system_id and public_id Initial Comment: XML: test DTD: you can imagine ... ch = MyHandler() parser = make_parser() parser.setProperty(handler.property_lexical_handler, ch) ... class MyHandler(XMLGenerator): ... def startDTD(self, name, public_id, system_id): self._out.write(public_id) ... -> "a.dtd" (but a.dtd should be the system_id) and the system_id in method startDTD is None a glimpse into pyexpat.c from python 2.2.3 ... VOID_HANDLER(StartDoctypeDecl, ... STRING_CONV_FUNC,doctypeName, STRING_CONV_FUNC,sysid, STRING_CONV_FUNC,pubid, has_internal_subset)) ... maybe sysid and pubid is swapped. I'm not a python specialist and maybe this bug is fixed or is not a bug. Please let me know... HolgeR ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-06 03:56 Message: Logged In: YES user_id=21627 This has been fixed in expatreader.py 1.33. Backporting the fix to 2.3 is not possible, since applications may rely on the bug. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-08-01 09:10 Message: Logged In: YES user_id=21627 This is indeed a bug in Python 2.3, which has been fixed in PyXML. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=780300&group_id=5470 From noreply at sourceforge.net Wed May 5 21:58:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 21:58:36 2004 Subject: [ python-Bugs-924294 ] IPV6 not correctly ifdef'd in socketmodule.c Message-ID: Bugs item #924294, was opened at 2004-03-27 01:07 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924294&group_id=5470 Category: Build Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: David Meleedy (dmeleedy) Assigned to: Martin v. L?wis (loewis) Summary: IPV6 not correctly ifdef'd in socketmodule.c Initial Comment: buckaroo-75: diff -c3 socketmodule.c-dist socketmodule.c *** socketmodule.c-dist Fri Mar 26 18:51:52 2004 --- socketmodule.c Fri Mar 26 18:52:47 2004 *************** *** 2971,2977 **** return NULL; } ! #ifndef ENABLE_IPV6 if(af == AF_INET6) { PyErr_SetString(socket_error, "can't use AF_INET6, IPv6 is disabled"); --- 2971,2977 ---- return NULL; } ! #ifdef ENABLE_IPV6 if(af == AF_INET6) { PyErr_SetString(socket_error, "can't use AF_INET6, IPv6 is disabled"); ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-06 03:58 Message: Logged In: YES user_id=21627 Can you please elaborate? Why should Python raise an exception that IPv6 is disabled if ENABLE_IPV6 is defined? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924294&group_id=5470 From noreply at sourceforge.net Wed May 5 22:05:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 22:05:19 2004 Subject: [ python-Bugs-936637 ] XMLFilterBase has bogus startElementNS Message-ID: Bugs item #936637, was opened at 2004-04-17 00:05 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=936637&group_id=5470 Category: XML Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 7 Submitted By: Magnus Lie Hetland (mlh) Assigned to: Martin v. L?wis (loewis) Summary: XMLFilterBase has bogus startElementNS Initial Comment: The version in XMLFilterBase is: def startElementNS(self, name, qname, attrs): self._cont_handler.startElement(name, attrs) It should be: def startElementNS(self, name, qname, attrs): self._cont_handler.startElementNS(name, qname, attrs) I've seen examples using this method, so in some version somewhere it must work -- but not in my 2.4 standard libraries, at least. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-06 04:05 Message: Logged In: YES user_id=21627 Thanks for pointing that out. Fixed in saxutils.py 1.21.10.1, 1.22 NEWS 1.831.4.104 Fix in PyXML is pending. ---------------------------------------------------------------------- Comment By: Magnus Lie Hetland (mlh) Date: 2004-04-17 00:06 Message: Logged In: YES user_id=20535 Err.. Make that: "not in my 2.3 standard libraries" ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=936637&group_id=5470 From noreply at sourceforge.net Wed May 5 22:40:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 22:40:55 2004 Subject: [ python-Bugs-938076 ] XMLGenerator ignores encoding in output Message-ID: Bugs item #938076, was opened at 2004-04-19 19:18 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=938076&group_id=5470 Category: XML Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 7 Submitted By: Magnus Lie Hetland (mlh) Assigned to: Martin v. L?wis (loewis) Summary: XMLGenerator ignores encoding in output Initial Comment: When XMLGenerator is supplied with an encoding such as 'utf-8' and subsequently with some non-ASCII Unicode characters, it crashes, because of its characters() method. The current version is: def characters(self, content): self._out.write(escape(content)) This completely ignores the encoding, and will (when writing to something such as a StringIO or the like) simply try to convert this into an ASCII string. The encoding is only used in the XML header, not as the real encoding! It may be that I've gotten things wrong, but I would suggest the following fix: def characters(self, content): self._out.write(escape(content).encode(self._encoding)) This seems to work well for me, at least. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-06 04:40 Message: Logged In: YES user_id=21627 Thanks for pointing this out. Fixed in saxutils.py 1.21.10.2 1.23 NEWS 1.831.4.105 Fix in PyXML is pending. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-04-20 21:46 Message: Logged In: YES user_id=21627 In general, it would be even better to generate character references for characters not representable in the output encoding. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=938076&group_id=5470 From noreply at sourceforge.net Wed May 5 23:26:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 23:27:04 2004 Subject: [ python-Bugs-948517 ] LaTeX not required Message-ID: Bugs item #948517, was opened at 2004-05-05 10:36 Message generated for change (Settings changed) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=948517&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Aahz (aahz) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: LaTeX not required Initial Comment: /doc.html and about.html ("Comments and Questions") should make clear that LaTeX is not required for submitting additions and changes to the docs. Suggested wording: Although we prefer additions and changes to be in the format prescribed by /doc.html, we welcome plain-text, too -- we're happy to handle the formatting for you. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=948517&group_id=5470 From noreply at sourceforge.net Wed May 5 23:49:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 5 23:49:32 2004 Subject: [ python-Bugs-881707 ] xml.sax.handler.ContentHandler missing argument Message-ID: Bugs item #881707, was opened at 2004-01-21 15:59 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=881707&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: James Kew (jkew) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: xml.sax.handler.ContentHandler missing argument Initial Comment: On xml.sax.handler.ContentHandler (section 13.10.1 in the Library Reference), the ignorableWhitespace method is documented as taking no args. The code, both in the stock xml package and in PyXML, has it taking one arg, whitespace. FWIW also the docstring for the method says "the application must attempt to read from the array outside of the specified range" -- not meaningful in the current version of the interface, which only passes content rather than content, offset, length. ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-05 23:49 Message: Logged In: YES user_id=3066 Fixed in CVS for Python 2.3.4 and 2.4. Doc/lib/xmlsaxhandler.tex 1.11, 1.10.12.1 Lib/xml/sax/handler.py 1.11, 1.10.10.1 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=881707&group_id=5470 From noreply at sourceforge.net Thu May 6 00:00:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 00:03:38 2004 Subject: [ python-Bugs-922964 ] file built-in object is after file() function Message-ID: Bugs item #922964, was opened at 2004-03-25 03:18 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=922964&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Out of Date Priority: 5 Submitted By: Francesco Ricciardi (nerby) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: file built-in object is after file() function Initial Comment: In section 2.1 of the HTML and PDF-A4 library references (so supposedly in all lib references), in the description of the file() function, it says that "Return a new file object (described earlier under Built-in Types)", but the Built-in Types section is later not earlier. ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-06 00:00 Message: Logged In: YES user_id=3066 Closing as out-of-date; as Tim notes, the CVS version of the documentation has already been made direction-agnostic. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-04-05 19:55 Message: Logged In: YES user_id=31435 Assigned to Fred. The LaTeX currently has """ Return a new file object (described in section~\ref{bltin-file-objects}, ``\ulink{File Objects}{bltin-file-objects.html}''). """ so I expect this should just be closed, as already fixed. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-03-29 12:57 Message: Logged In: YES user_id=6656 That's not what he's complaining about, though. Seems to me that just removing the 'earlier' would suffice. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-03-25 03:28 Message: Logged In: YES user_id=80475 Forward references are only problematic in the tutorial. The rest of the docs tend to be used in a random access fashion rather than sequentially. That's a good thing because forward references are everywhere. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=922964&group_id=5470 From noreply at sourceforge.net Thu May 6 02:30:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 02:30:44 2004 Subject: [ python-Bugs-948970 ] PyExc_* not in index Message-ID: Bugs item #948970, was opened at 2004-05-06 08:30 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=948970&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Thomas Heller (theller) Assigned to: Nobody/Anonymous (nobody) Summary: PyExc_* not in index Initial Comment: The PyExc_* variables are not in the Python/C API index. This prevents finding them easily. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=948970&group_id=5470 From noreply at sourceforge.net Thu May 6 06:22:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 06:22:47 2004 Subject: [ python-Bugs-949052 ] winzip cannot extrat tarfile created tars Message-ID: Bugs item #949052, was opened at 2004-05-06 12:22 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949052&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Simon Eisenmann (longsleep) Assigned to: Nobody/Anonymous (nobody) Summary: winzip cannot extrat tarfile created tars Initial Comment: The tarfile module uses the posix way to handle long file names per default. This is not supported by lots of extractors (like winzip). This should be changed not to be the default. Long file names should be handled by the gnu extension anytime. right now this is reached by setting $tarobject.posix = False manually. In addition the tarfile module sets a file size on directory entries. This isnt supported as well on well known extractors like winzip. The size for directory entries should be zero anytime. I attached a patch to fix the latter problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949052&group_id=5470 From noreply at sourceforge.net Thu May 6 07:42:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 07:42:51 2004 Subject: [ python-Bugs-949100 ] refresh() missing curses reference Message-ID: Bugs item #949100, was opened at 2004-05-06 13:42 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949100&group_id=5470 Category: Documentation Group: Not a Bug Status: Open Resolution: None Priority: 5 Submitted By: Alexander Neumann (alexandern) Assigned to: Nobody/Anonymous (nobody) Summary: refresh() missing curses reference Initial Comment: The refresh() function of window objects is completely missing from the library reference documentation about curses. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949100&group_id=5470 From noreply at sourceforge.net Thu May 6 08:28:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 08:28:47 2004 Subject: [ python-Bugs-800828 ] Doc bug in PEP-0278 Message-ID: Bugs item #800828, was opened at 2003-09-04 22:18 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=800828&group_id=5470 Category: Documentation Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: logistix (logistix) Assigned to: Jack Jansen (jackjansen) Summary: Doc bug in PEP-0278 Initial Comment: PEP 278 suggests that the proper way to test for universal newline support is 'hasattr (sys.stdout, "newline")' However, this doesn't work too well when sys.stdout is overridden. One big example of this is PythonWin's sys.stdout, which doesn't have an newlines attribute even when it is supported by python23.dll. Testing 'hasattr(file, "newlines")' seems to be a more reliable way to test. ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-06 08:28 Message: Logged In: YES user_id=3066 Fixed. I changed the test to use "hasattr(open, 'newlines')" to be compatible with older versions of Python as well as avoiding the replacement problem you noted. nondist/peps/pep-0278.txt revision 1.9 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=800828&group_id=5470 From noreply at sourceforge.net Thu May 6 08:46:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 08:46:23 2004 Subject: [ python-Bugs-817742 ] Need "new style" note regarding descriptors Message-ID: Bugs item #817742, was opened at 2003-10-04 11:26 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=817742&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: David Abrahams (david_abrahams) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: Need "new style" note regarding descriptors Initial Comment: When you split my patch into http://www.python.org/doc/current/ref/new-style- attribute-access.html and http://www.python.org/doc/current/ref/descriptors.html, the descriptors info doesn't say "new style classes only anymore" ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-06 08:46 Message: Logged In: YES user_id=3066 Fixed in CVS for Python 2.3.4 and 2.4. Doc/ref/ref3.tex revisions 1.119, 1.114.4.5 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=817742&group_id=5470 From noreply at sourceforge.net Thu May 6 08:50:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 08:50:17 2004 Subject: [ python-Bugs-737734 ] need doc for new trace module Message-ID: Bugs item #737734, was opened at 2003-05-14 12:22 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=737734&group_id=5470 Category: Documentation Group: None >Status: Closed >Resolution: Duplicate Priority: 5 Submitted By: Jeremy Hylton (jhylton) Assigned to: Jeremy Hylton (jhylton) Summary: need doc for new trace module Initial Comment: see summary ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-06 08:50 Message: Logged In: YES user_id=3066 duplicate of SF bug #779976 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=737734&group_id=5470 From noreply at sourceforge.net Thu May 6 09:05:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 09:05:33 2004 Subject: [ python-Bugs-949100 ] refresh() missing curses reference Message-ID: Bugs item #949100, was opened at 2004-05-06 07:42 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949100&group_id=5470 Category: Documentation Group: Not a Bug >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Alexander Neumann (alexandern) >Assigned to: A.M. Kuchling (akuchling) Summary: refresh() missing curses reference Initial Comment: The refresh() function of window objects is completely missing from the library reference documentation about curses. ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-05-06 09:05 Message: Logged In: YES user_id=11375 It doesn't seem to be missing; see http://docs.python.org/lib/curses-window-objects.html . The text starts "Update the display immediately...". ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949100&group_id=5470 From noreply at sourceforge.net Thu May 6 09:20:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 09:20:34 2004 Subject: [ python-Bugs-945063 ] cgitb has backwards logic on log file creation Message-ID: Bugs item #945063, was opened at 2004-04-30 01:23 Message generated for change (Comment added) made by akuchling You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=945063&group_id=5470 >Category: Python Library Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Danny Yoo (dyoo) >Assigned to: A.M. Kuchling (akuchling) Summary: cgitb has backwards logic on log file creation Initial Comment: In Python 2.3.3, the cgitb module has some strange logic on line 276: suffix = ['.html', '.txt'][self.format=="html"] This looks incorrect; the corrected code should be: suffix = ['.txt', '.html'][self.format=='html'] Slightly discussed on Python-tutor: http://mail.python.org/pipermail/tutor/2004-April/029447.html http://mail.python.org/pipermail/tutor/2004-April/029455.html ---------------------------------------------------------------------- >Comment By: A.M. Kuchling (akuchling) Date: 2004-05-06 09:20 Message: Logged In: YES user_id=11375 Verified and fixed in CVS, both the 2.4 HEAD and the 2.3 maintenance branch. Thanks! ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=945063&group_id=5470 From noreply at sourceforge.net Thu May 6 12:02:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 12:02:32 2004 Subject: [ python-Bugs-949301 ] Memory "read" instruction failed Message-ID: Bugs item #949301, was opened at 2004-05-06 18:02 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 Category: Windows Group: None Status: Open Resolution: None Priority: 5 Submitted By: AnToine (alexion) Assigned to: Thomas Heller (theller) Summary: Memory "read" instruction failed Initial Comment: When starting a compiled (.exe) program (Python en wxWindows) it sometimes hapens that it won't start and display's a error message, saying something like: Intruction on 0x00de3ce9 directs to 0x00000028. The write or readinstruction ("read") on the memmory failed. Klick OK to end the programm (see attached file with the Dutch error message) Our programm is free for download at: http://www.alexion.nl/pynx/registrationalexion.html (only e-mail and (fake)name required) It's a mistery, only on some systems (winXP) it happens. Sometimes after a re-install of WinXP everything is OK again... ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 From noreply at sourceforge.net Thu May 6 12:52:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 12:52:12 2004 Subject: [ python-Bugs-949329 ] stringprep.py: sets.Set() --> set() Message-ID: Bugs item #949329, was opened at 2004-05-06 11:51 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949329&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Martin v. L?wis (loewis) Summary: stringprep.py: sets.Set() --> set() Initial Comment: Suggest replacing sets.Set() with set() in stringprep.py. The __contains__ test for the builtin runs several times faster. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949329&group_id=5470 From noreply at sourceforge.net Thu May 6 14:05:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 14:06:09 2004 Subject: [ python-Bugs-949301 ] Memory "read" instruction failed Message-ID: Bugs item #949301, was opened at 2004-05-06 18:02 Message generated for change (Settings changed) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 Category: Windows Group: None >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: AnToine (alexion) >Assigned to: Nobody/Anonymous (nobody) Summary: Memory "read" instruction failed Initial Comment: When starting a compiled (.exe) program (Python en wxWindows) it sometimes hapens that it won't start and display's a error message, saying something like: Intruction on 0x00de3ce9 directs to 0x00000028. The write or readinstruction ("read") on the memmory failed. Klick OK to end the programm (see attached file with the Dutch error message) Our programm is free for download at: http://www.alexion.nl/pynx/registrationalexion.html (only e-mail and (fake)name required) It's a mistery, only on some systems (winXP) it happens. Sometimes after a re-install of WinXP everything is OK again... ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-05-06 20:05 Message: Logged In: YES user_id=11105 It seems you are using Installer to create the exe. The problem is most certainly a bug either in wxPython, other third party extensions, or Installer itself. Unassigning, and closed as invalid. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 From noreply at sourceforge.net Thu May 6 16:05:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 16:05:22 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None >Priority: 7 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Thu May 6 21:31:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 21:31:10 2004 Subject: [ python-Bugs-949605 ] UnicodeEncodeError message has syntax error Message-ID: Bugs item #949605, was opened at 2004-05-07 01:31 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949605&group_id=5470 Category: Unicode Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Tom Lynn (tlynn) Assigned to: M.-A. Lemburg (lemburg) Summary: UnicodeEncodeError message has syntax error Initial Comment: It's a bit nit-picking, but shouldn't the '\ua3' in the UnicodeEncodeError message below be '\xa3' or u'\xa3'? http://www.python.org/doc/current/ref/strings.html only allows for \uxxxx and \Uxxxxxxxx format specifiers. >>> u"\N{POUND SIGN}" u'\xa3' >>> _.encode("ascii") Traceback (most recent call last): File "", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character '\ua3' in position 0: ordinal not in range(128) >>> '\ua3' '\ua3' >>> u'\ua3' UnicodeDecodeError: 'unicodeescape' codec can't decode bytes in position 0-3: end of string in escape sequence ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949605&group_id=5470 From noreply at sourceforge.net Thu May 6 21:33:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 6 21:33:58 2004 Subject: [ python-Bugs-949605 ] UnicodeEncodeError message has syntax error Message-ID: Bugs item #949605, was opened at 2004-05-07 01:31 Message generated for change (Settings changed) made by tlynn You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949605&group_id=5470 Category: Unicode Group: Python 2.3 Status: Open Resolution: None >Priority: 2 Submitted By: Tom Lynn (tlynn) Assigned to: M.-A. Lemburg (lemburg) Summary: UnicodeEncodeError message has syntax error Initial Comment: It's a bit nit-picking, but shouldn't the '\ua3' in the UnicodeEncodeError message below be '\xa3' or u'\xa3'? http://www.python.org/doc/current/ref/strings.html only allows for \uxxxx and \Uxxxxxxxx format specifiers. >>> u"\N{POUND SIGN}" u'\xa3' >>> _.encode("ascii") Traceback (most recent call last): File "", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character '\ua3' in position 0: ordinal not in range(128) >>> '\ua3' '\ua3' >>> u'\ua3' UnicodeDecodeError: 'unicodeescape' codec can't decode bytes in position 0-3: end of string in escape sequence ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949605&group_id=5470 From noreply at sourceforge.net Fri May 7 01:26:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 01:27:01 2004 Subject: [ python-Bugs-949667 ] file write() method and non-blocking mode. Message-ID: Bugs item #949667, was opened at 2004-05-07 15:26 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949667&group_id=5470 Category: Python Library Group: Feature Request Status: Open Resolution: None Priority: 5 Submitted By: Donovan Baarda (abo) Assigned to: Nobody/Anonymous (nobody) Summary: file write() method and non-blocking mode. Initial Comment: Currently file object's write() method returns nothing. It would be very nice if it returned the number of bytes written. It would also be nice if file objects had a setblocking() method, similar to socket objects. Under various situations, such as communicating with popen'ed processes, write() can block. To avoid this, you must set the file to non-blocking mode, or do something like fork a thread to service the process so that it can "unblock" the write. Currently, setting a file to unblocking mode requires using fnctl on the file objects fileno(). It would be nice if you could directly modify this file behaviour using a setblocking() method. When you do set a file to non-blocking mode, the file object's write() method will raise an exception if not all the data could be written. This is useless, as there is no indication how much data was successfuly written before the blocking happened, so you have no idea how much data remains to be written when the file becomes unblocked. Even using things like select() loops are not reliable, as write() on an unblocked file can block part way through even a small data buffer. The only sure way to avoid a file object's write() from blocking or raising an unhelpful exception is to write a single byte at a time. This means you must use os.write() instead. The os.write() method returns the number of bytes written, which tells how many bytes of data remain to be written next time when the file becomes unblocked. The existing read() method behaves the same as the os.read() method and can be used in non-blocking mode. If the file write() method behaved the same as the os.write() method, returning the number of bytes written instead of raising an exception, then it could also be used in non-blocking mode. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949667&group_id=5470 From noreply at sourceforge.net Fri May 7 03:19:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 03:19:47 2004 Subject: [ python-Bugs-931848 ] re seems broken on 64-bit machines/linux Message-ID: Bugs item #931848, was opened at 2004-04-08 19:02 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=931848&group_id=5470 Category: Regular Expressions Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 6 Submitted By: Mihai Ibanescu (misa) Assigned to: Martin v. L?wis (loewis) Summary: re seems broken on 64-bit machines/linux Initial Comment: Hello, Tested the following piece of code both on ia64 and amd64. python -c 'import re; print re.compile(u"[\u002E\u3002\uFF0E\uFF61]").split("a.b.c")' Expected result: ['a', 'b', 'c'] Actual result: varies depending on the version of glibc (probably). On glibc-2.3.2 (Red Hat Enterprise Linux 3 WS) I get back: ['a.b.c'] On glibc-2.3.3 (Fedora Core 2) I get back: ['a.b.', ''] This doesn't happen on i386 architectures. The above string that I try to compile comes from encodings/idna.py (and it is used to split a domain name into components). Let me know if you need more information on how to reproduce this. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-07 09:19 Message: Logged In: YES user_id=21627 This is fixed now in sre_compile.py 1.55 test_re.py 1.49 sre.h 2.25 sre_compile.py 1.49.6.1 NEWS 1.831.4.106 sre.h 2.22.16.1 Are you sure it happens in UCS-2 mode as well? The fix I made can only apply to UCS-4 mode, and is then independent of the C library. ---------------------------------------------------------------------- Comment By: Mihai Ibanescu (misa) Date: 2004-04-08 19:11 Message: Logged In: YES user_id=205865 In both cases, python was built with UCS4, but it happens with UCS2 just as well. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=931848&group_id=5470 From noreply at sourceforge.net Fri May 7 04:02:08 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 04:02:16 2004 Subject: [ python-Bugs-949301 ] Memory "read" instruction failed Message-ID: Bugs item #949301, was opened at 2004-05-06 18:02 Message generated for change (Comment added) made by alexion You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 Category: Windows Group: None Status: Closed Resolution: Invalid Priority: 5 Submitted By: AnToine (alexion) Assigned to: Nobody/Anonymous (nobody) Summary: Memory "read" instruction failed Initial Comment: When starting a compiled (.exe) program (Python en wxWindows) it sometimes hapens that it won't start and display's a error message, saying something like: Intruction on 0x00de3ce9 directs to 0x00000028. The write or readinstruction ("read") on the memmory failed. Klick OK to end the programm (see attached file with the Dutch error message) Our programm is free for download at: http://www.alexion.nl/pynx/registrationalexion.html (only e-mail and (fake)name required) It's a mistery, only on some systems (winXP) it happens. Sometimes after a re-install of WinXP everything is OK again... ---------------------------------------------------------------------- >Comment By: AnToine (alexion) Date: 2004-05-07 10:02 Message: Logged In: YES user_id=1036567 A message in Win2000 is saying a problem with Python23.dll: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Is this something what makes sense? ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-06 20:05 Message: Logged In: YES user_id=11105 It seems you are using Installer to create the exe. The problem is most certainly a bug either in wxPython, other third party extensions, or Installer itself. Unassigning, and closed as invalid. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 From noreply at sourceforge.net Fri May 7 04:11:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 04:11:58 2004 Subject: [ python-Bugs-949706 ] Invalid page in module PYTHON23.DLL Message-ID: Bugs item #949706, was opened at 2004-05-07 10:11 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949706&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: AnToine (alexion) Assigned to: Nobody/Anonymous (nobody) Summary: Invalid page in module PYTHON23.DLL Initial Comment: In Windows2000 we got the following error message: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Can you do something with this? ongeldige pagina = unvalid page (or something I guess). ARM one can request a license for free at http://www.alexion.nl/pynx/registrationalexion.html Or direct download at http://www.alexion.nl/download/ARM- v1.0_RC1_build_1002.exe With the following registration code: Geregistreerd door: Python sourceforge Email adres: support@alexion.nl Aantal medewerkers: 1 Registratie sleutel: 5C12731086396B5C38CAFEB82D0E4E9A Use these to fill in when the registration asks for it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949706&group_id=5470 From noreply at sourceforge.net Fri May 7 04:23:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 04:23:25 2004 Subject: [ python-Bugs-949301 ] Memory "read" instruction failed Message-ID: Bugs item #949301, was opened at 2004-05-06 18:02 Message generated for change (Comment added) made by alexion You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 Category: Windows Group: None Status: Closed Resolution: Invalid Priority: 5 Submitted By: AnToine (alexion) Assigned to: Nobody/Anonymous (nobody) Summary: Memory "read" instruction failed Initial Comment: When starting a compiled (.exe) program (Python en wxWindows) it sometimes hapens that it won't start and display's a error message, saying something like: Intruction on 0x00de3ce9 directs to 0x00000028. The write or readinstruction ("read") on the memmory failed. Klick OK to end the programm (see attached file with the Dutch error message) Our programm is free for download at: http://www.alexion.nl/pynx/registrationalexion.html (only e-mail and (fake)name required) It's a mistery, only on some systems (winXP) it happens. Sometimes after a re-install of WinXP everything is OK again... ---------------------------------------------------------------------- >Comment By: AnToine (alexion) Date: 2004-05-07 10:23 Message: Logged In: YES user_id=1036567 A message in Win2000 is saying a problem with Python23.dll: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Is this something what makes sense? ---------------------------------------------------------------------- Comment By: AnToine (alexion) Date: 2004-05-07 10:02 Message: Logged In: YES user_id=1036567 A message in Win2000 is saying a problem with Python23.dll: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Is this something what makes sense? ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-06 20:05 Message: Logged In: YES user_id=11105 It seems you are using Installer to create the exe. The problem is most certainly a bug either in wxPython, other third party extensions, or Installer itself. Unassigning, and closed as invalid. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 From noreply at sourceforge.net Fri May 7 04:38:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 04:38:37 2004 Subject: [ python-Bugs-949301 ] Memory "read" instruction failed Message-ID: Bugs item #949301, was opened at 2004-05-06 18:02 Message generated for change (Comment added) made by alexion You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 Category: Windows Group: None >Status: Open Resolution: Invalid Priority: 5 Submitted By: AnToine (alexion) Assigned to: Nobody/Anonymous (nobody) Summary: Memory "read" instruction failed Initial Comment: When starting a compiled (.exe) program (Python en wxWindows) it sometimes hapens that it won't start and display's a error message, saying something like: Intruction on 0x00de3ce9 directs to 0x00000028. The write or readinstruction ("read") on the memmory failed. Klick OK to end the programm (see attached file with the Dutch error message) Our programm is free for download at: http://www.alexion.nl/pynx/registrationalexion.html (only e-mail and (fake)name required) It's a mistery, only on some systems (winXP) it happens. Sometimes after a re-install of WinXP everything is OK again... ---------------------------------------------------------------------- >Comment By: AnToine (alexion) Date: 2004-05-07 10:38 Message: Logged In: YES user_id=1036567 A message in Win2000 is saying a problem with Python23.dll: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Is this something what makes sense? ---------------------------------------------------------------------- Comment By: AnToine (alexion) Date: 2004-05-07 10:23 Message: Logged In: YES user_id=1036567 A message in Win2000 is saying a problem with Python23.dll: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Is this something what makes sense? ---------------------------------------------------------------------- Comment By: AnToine (alexion) Date: 2004-05-07 10:02 Message: Logged In: YES user_id=1036567 A message in Win2000 is saying a problem with Python23.dll: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Is this something what makes sense? ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-06 20:05 Message: Logged In: YES user_id=11105 It seems you are using Installer to create the exe. The problem is most certainly a bug either in wxPython, other third party extensions, or Installer itself. Unassigning, and closed as invalid. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 From noreply at sourceforge.net Fri May 7 04:39:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 04:39:30 2004 Subject: [ python-Bugs-949706 ] Invalid page in module PYTHON23.DLL Message-ID: Bugs item #949706, was opened at 2004-05-07 10:11 Message generated for change (Settings changed) made by alexion You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949706&group_id=5470 >Category: Windows >Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: AnToine (alexion) Assigned to: Nobody/Anonymous (nobody) Summary: Invalid page in module PYTHON23.DLL Initial Comment: In Windows2000 we got the following error message: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Can you do something with this? ongeldige pagina = unvalid page (or something I guess). ARM one can request a license for free at http://www.alexion.nl/pynx/registrationalexion.html Or direct download at http://www.alexion.nl/download/ARM- v1.0_RC1_build_1002.exe With the following registration code: Geregistreerd door: Python sourceforge Email adres: support@alexion.nl Aantal medewerkers: 1 Registratie sleutel: 5C12731086396B5C38CAFEB82D0E4E9A Use these to fill in when the registration asks for it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949706&group_id=5470 From noreply at sourceforge.net Fri May 7 04:40:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 04:40:24 2004 Subject: [ python-Bugs-949301 ] Memory "read" instruction failed Message-ID: Bugs item #949301, was opened at 2004-05-06 18:02 Message generated for change (Settings changed) made by alexion You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 Category: Windows Group: None Status: Open Resolution: Invalid Priority: 5 Submitted By: AnToine (alexion) >Assigned to: Thomas Heller (theller) Summary: Memory "read" instruction failed Initial Comment: When starting a compiled (.exe) program (Python en wxWindows) it sometimes hapens that it won't start and display's a error message, saying something like: Intruction on 0x00de3ce9 directs to 0x00000028. The write or readinstruction ("read") on the memmory failed. Klick OK to end the programm (see attached file with the Dutch error message) Our programm is free for download at: http://www.alexion.nl/pynx/registrationalexion.html (only e-mail and (fake)name required) It's a mistery, only on some systems (winXP) it happens. Sometimes after a re-install of WinXP everything is OK again... ---------------------------------------------------------------------- Comment By: AnToine (alexion) Date: 2004-05-07 10:38 Message: Logged In: YES user_id=1036567 A message in Win2000 is saying a problem with Python23.dll: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Is this something what makes sense? ---------------------------------------------------------------------- Comment By: AnToine (alexion) Date: 2004-05-07 10:23 Message: Logged In: YES user_id=1036567 A message in Win2000 is saying a problem with Python23.dll: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Is this something what makes sense? ---------------------------------------------------------------------- Comment By: AnToine (alexion) Date: 2004-05-07 10:02 Message: Logged In: YES user_id=1036567 A message in Win2000 is saying a problem with Python23.dll: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Is this something what makes sense? ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-06 20:05 Message: Logged In: YES user_id=11105 It seems you are using Installer to create the exe. The problem is most certainly a bug either in wxPython, other third party extensions, or Installer itself. Unassigning, and closed as invalid. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 From noreply at sourceforge.net Fri May 7 07:09:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 07:10:03 2004 Subject: [ python-Bugs-931848 ] re seems broken on 64-bit machines/linux Message-ID: Bugs item #931848, was opened at 2004-04-08 13:02 Message generated for change (Comment added) made by misa You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=931848&group_id=5470 Category: Regular Expressions Group: Python 2.3 Status: Closed Resolution: Fixed Priority: 6 Submitted By: Mihai Ibanescu (misa) Assigned to: Martin v. L?wis (loewis) Summary: re seems broken on 64-bit machines/linux Initial Comment: Hello, Tested the following piece of code both on ia64 and amd64. python -c 'import re; print re.compile(u"[\u002E\u3002\uFF0E\uFF61]").split("a.b.c")' Expected result: ['a', 'b', 'c'] Actual result: varies depending on the version of glibc (probably). On glibc-2.3.2 (Red Hat Enterprise Linux 3 WS) I get back: ['a.b.c'] On glibc-2.3.3 (Fedora Core 2) I get back: ['a.b.', ''] This doesn't happen on i386 architectures. The above string that I try to compile comes from encodings/idna.py (and it is used to split a domain name into components). Let me know if you need more information on how to reproduce this. ---------------------------------------------------------------------- >Comment By: Mihai Ibanescu (misa) Date: 2004-05-07 07:09 Message: Logged In: YES user_id=205865 I will check to see how UCS2, I remember testing it though and issuing the same results. Thanks for the fixes, testing them... ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-07 03:19 Message: Logged In: YES user_id=21627 This is fixed now in sre_compile.py 1.55 test_re.py 1.49 sre.h 2.25 sre_compile.py 1.49.6.1 NEWS 1.831.4.106 sre.h 2.22.16.1 Are you sure it happens in UCS-2 mode as well? The fix I made can only apply to UCS-4 mode, and is then independent of the C library. ---------------------------------------------------------------------- Comment By: Mihai Ibanescu (misa) Date: 2004-04-08 13:11 Message: Logged In: YES user_id=205865 In both cases, python was built with UCS4, but it happens with UCS2 just as well. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=931848&group_id=5470 From noreply at sourceforge.net Fri May 7 07:23:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 07:23:39 2004 Subject: [ python-Bugs-949605 ] UnicodeEncodeError message has syntax error Message-ID: Bugs item #949605, was opened at 2004-05-07 03:31 Message generated for change (Comment added) made by doerwalter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949605&group_id=5470 Category: Unicode Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 2 Submitted By: Tom Lynn (tlynn) Assigned to: M.-A. Lemburg (lemburg) Summary: UnicodeEncodeError message has syntax error Initial Comment: It's a bit nit-picking, but shouldn't the '\ua3' in the UnicodeEncodeError message below be '\xa3' or u'\xa3'? http://www.python.org/doc/current/ref/strings.html only allows for \uxxxx and \Uxxxxxxxx format specifiers. >>> u"\N{POUND SIGN}" u'\xa3' >>> _.encode("ascii") Traceback (most recent call last): File "", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character '\ua3' in position 0: ordinal not in range(128) >>> '\ua3' '\ua3' >>> u'\ua3' UnicodeDecodeError: 'unicodeescape' codec can't decode bytes in position 0-3: end of string in escape sequence ---------------------------------------------------------------------- >Comment By: Walter D?rwald (doerwalter) Date: 2004-05-07 13:23 Message: Logged In: YES user_id=89016 This has already been fixed in Python/exception.c 1.46 and 1.44.10.1 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949605&group_id=5470 From noreply at sourceforge.net Fri May 7 07:57:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 07:57:57 2004 Subject: [ python-Bugs-949706 ] Invalid page in module PYTHON23.DLL Message-ID: Bugs item #949706, was opened at 2004-05-07 10:11 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949706&group_id=5470 Category: Windows Group: Python 2.3 >Status: Deleted >Resolution: Duplicate Priority: 5 Submitted By: AnToine (alexion) Assigned to: Nobody/Anonymous (nobody) Summary: Invalid page in module PYTHON23.DLL Initial Comment: In Windows2000 we got the following error message: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Can you do something with this? ongeldige pagina = unvalid page (or something I guess). ARM one can request a license for free at http://www.alexion.nl/pynx/registrationalexion.html Or direct download at http://www.alexion.nl/download/ARM- v1.0_RC1_build_1002.exe With the following registration code: Geregistreerd door: Python sourceforge Email adres: support@alexion.nl Aantal medewerkers: 1 Registratie sleutel: 5C12731086396B5C38CAFEB82D0E4E9A Use these to fill in when the registration asks for it. ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-05-07 13:57 Message: Logged In: YES user_id=11105 Duplicate of 949301. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949706&group_id=5470 From noreply at sourceforge.net Fri May 7 08:04:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 08:04:28 2004 Subject: [ python-Bugs-949301 ] Memory "read" instruction failed Message-ID: Bugs item #949301, was opened at 2004-05-06 18:02 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 Category: Windows Group: None >Status: Closed Resolution: Invalid Priority: 5 Submitted By: AnToine (alexion) >Assigned to: Nobody/Anonymous (nobody) Summary: Memory "read" instruction failed Initial Comment: When starting a compiled (.exe) program (Python en wxWindows) it sometimes hapens that it won't start and display's a error message, saying something like: Intruction on 0x00de3ce9 directs to 0x00000028. The write or readinstruction ("read") on the memmory failed. Klick OK to end the programm (see attached file with the Dutch error message) Our programm is free for download at: http://www.alexion.nl/pynx/registrationalexion.html (only e-mail and (fake)name required) It's a mistery, only on some systems (winXP) it happens. Sometimes after a re-install of WinXP everything is OK again... ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-05-07 14:04 Message: Logged In: YES user_id=11105 The stack dump doesn't help at all. Python23.dll is a library providing a lot of functions, and Windows complaining about Python23.dll doesn't say whether the problem is in the Python dll or in the code calling it. If you can reproduce this problem in the original Python script (without freezing it into a standalone exe) it might be easier to spot the problem (and to find out whether it is in Python itself, in a third party extension like wxPython or win32all). If you *can not* reproduce the problem in this way, the bug probably lies inside the Installer code. Over and out. And please stop assigning bugs to developers yourself, normally these pick up the bugs they want to work on themselves. We're all volunteers here... ---------------------------------------------------------------------- Comment By: AnToine (alexion) Date: 2004-05-07 10:38 Message: Logged In: YES user_id=1036567 A message in Win2000 is saying a problem with Python23.dll: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Is this something what makes sense? ---------------------------------------------------------------------- Comment By: AnToine (alexion) Date: 2004-05-07 10:23 Message: Logged In: YES user_id=1036567 A message in Win2000 is saying a problem with Python23.dll: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Is this something what makes sense? ---------------------------------------------------------------------- Comment By: AnToine (alexion) Date: 2004-05-07 10:02 Message: Logged In: YES user_id=1036567 A message in Win2000 is saying a problem with Python23.dll: ARM heeft een fout met betrekking tot een ongeldige pagina veroorzaakt in module PYTHON23.DLL op018f:1e01561d. Registers: EAX=00000000 CS=018f EIP=1e01561d EFLGS=00010202 EBX=00776930 SS=0197 ESP=0064ead4 EBP=00000001 ECX=00000000 DS=0197 ESI=00776a80 FS=943f EDX=1e0bb488 ES=0197 EDI=00c08088 GS=0000 Bytes in CS:EIP: 8b 08 41 89 08 8b 0e 85 c9 89 06 74 11 8b 01 48 Stackdump: 00000000 0137eb04 01cea4b0 00000000 00000000 17edbb04 00000000 00776a84 00770870 1e040998 01b62860 00000002 00776930 0137eb04 00000002 00000000 Is this something what makes sense? ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-06 20:05 Message: Logged In: YES user_id=11105 It seems you are using Installer to create the exe. The problem is most certainly a bug either in wxPython, other third party extensions, or Installer itself. Unassigning, and closed as invalid. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949301&group_id=5470 From noreply at sourceforge.net Fri May 7 08:39:34 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 08:39:45 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-19 09:28 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None Priority: 7 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 22:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 06:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 18:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-05 00:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-05 00:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-19 09:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Fri May 7 08:44:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 08:44:38 2004 Subject: [ python-Bugs-949832 ] Problem w/6.27.2.2 GNUTranslations ungettext() example code Message-ID: Bugs item #949832, was opened at 2004-05-07 05:44 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949832&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Martin Miller (mrmiller) Assigned to: Nobody/Anonymous (nobody) Summary: Problem w/6.27.2.2 GNUTranslations ungettext() example code Initial Comment: > Here is an example: > > n = len(os.listdir('.')) > cat = GNUTranslations(somefile) > message = cat.ungettext( > 'There is %(num)d file in this directory', > 'There are %(num)d files in this directory', > n) % {'n': n} The last line of code in the example should be: > n) % {'num': n} Also, I don't think the example illustrates a realistic usage of the ungettext() method, as it is unlikely that the message catalog is going to have different message id's for all the possible variations of the string each with a different number of files in them -- exactly the problem that ungettext() is suppose to address. A better example would just use ungettext() to pick either the word "file" or "files" based on the value of n. It would be more realistic and the example code would probably be simpler. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949832&group_id=5470 From noreply at sourceforge.net Fri May 7 08:56:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 08:57:08 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-19 00:28 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None Priority: 7 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-07 13:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 13:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 21:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 09:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 15:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 15:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-19 00:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Fri May 7 09:06:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 09:07:02 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-19 09:28 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None Priority: 7 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 23:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 22:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 22:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 06:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 18:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-05 00:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-05 00:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-19 09:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Fri May 7 09:48:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 09:49:03 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None Priority: 7 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Fri May 7 09:59:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 09:59:34 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None Priority: 7 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Fri May 7 12:50:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 12:51:13 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None Priority: 7 Submitted By: Greg Jones (morngnstar) >Assigned to: Guido van Rossum (gvanrossum) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Fri May 7 14:09:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 14:09:58 2004 Subject: [ python-Bugs-949329 ] stringprep.py: sets.Set() --> set() Message-ID: Bugs item #949329, was opened at 2004-05-06 11:51 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949329&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Martin v. L?wis (loewis) Summary: stringprep.py: sets.Set() --> set() Initial Comment: Suggest replacing sets.Set() with set() in stringprep.py. The __contains__ test for the builtin runs several times faster. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-07 13:09 Message: Logged In: YES user_id=80475 Patch is attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949329&group_id=5470 From noreply at sourceforge.net Fri May 7 14:18:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 14:18:45 2004 Subject: [ python-Bugs-947906 ] calendar.weekheader(n): n should mean chars not bytes Message-ID: Bugs item #947906, was opened at 2004-05-04 13:38 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947906&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None >Priority: 7 Submitted By: Leonardo Rochael Almeida (rochael) Assigned to: Nobody/Anonymous (nobody) Summary: calendar.weekheader(n): n should mean chars not bytes Initial Comment: calendar.weekheader(n) is locale aware, which is good in principle. The parameter n, however, is interpreted as meaning bytes, not chars, which can generate broken strings for, e.g. localized weekday names: >>> calendar.weekheader(2) 'Mo Tu We Th Fr Sa Su' >>> locale.setlocale(locale.LC_ALL, "pt_BR.UTF-8") 'pt_BR.UTF-8' >>> calendar.weekheader(2) 'Se Te Qu Qu Se S\xc3 Do' Notice how "S?bado" (Saturday) above is missing the second utf-8 byte for the encoding of "?": >>> u"S?".encode("utf-8") 'S\xc3\xa1' The implementation of weekheader (and of all of calendar.py, it seems) is based on localized 8 bit strings. I suppose the correct fix for this bug will involve a roundtrip thru unicode. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947906&group_id=5470 From noreply at sourceforge.net Fri May 7 14:26:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 14:26:03 2004 Subject: [ python-Bugs-950055 ] Is cmdqueue attribute used? Message-ID: Bugs item #950055, was opened at 2004-05-07 13:25 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950055&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Guido van Rossum (gvanrossum) Summary: Is cmdqueue attribute used? Initial Comment: In cmd.py, the cmdqueue attribute is set to a list and checked for pending items but never is filled by any method nor is it in the documented API. Is the interface incomplete or is the attribute bogus? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950055&group_id=5470 From noreply at sourceforge.net Fri May 7 14:32:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 14:32:44 2004 Subject: [ python-Bugs-950057 ] itertools.chain doesn't "process" exceptions as they occur Message-ID: Bugs item #950057, was opened at 2004-05-07 20:32 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950057&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Tristan Seligmann (mithrandi) Assigned to: Nobody/Anonymous (nobody) Summary: itertools.chain doesn't "process" exceptions as they occur Initial Comment: Python 2.3.3 (#2, Feb 24 2004, 09:29:20) [GCC 3.3.3 (Debian)] on linux2 The following code demonstrates the problem: import itertools def gen1(): print 'Before yield 1' yield 1 print 'After yield 1' assert False def gen2(x = False): print 'Before yield 2' yield 2 print 'After yield 2' if x: raise StopIteration for elem in itertools.chain(gen1(), gen2()): print elem The exception only propagates up after itertools.chain completes iteration; if you pass True to gen2(), then the AssertionError will be completely masked, concealing the fact that any exception has been raised. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950057&group_id=5470 From noreply at sourceforge.net Fri May 7 14:34:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 14:35:11 2004 Subject: [ python-Bugs-950057 ] itertools.chain doesn't "process" exceptions as they occur Message-ID: Bugs item #950057, was opened at 2004-05-07 20:32 Message generated for change (Comment added) made by mithrandi You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950057&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Tristan Seligmann (mithrandi) >Assigned to: Raymond Hettinger (rhettinger) Summary: itertools.chain doesn't "process" exceptions as they occur Initial Comment: Python 2.3.3 (#2, Feb 24 2004, 09:29:20) [GCC 3.3.3 (Debian)] on linux2 The following code demonstrates the problem: import itertools def gen1(): print 'Before yield 1' yield 1 print 'After yield 1' assert False def gen2(x = False): print 'Before yield 2' yield 2 print 'After yield 2' if x: raise StopIteration for elem in itertools.chain(gen1(), gen2()): print elem The exception only propagates up after itertools.chain completes iteration; if you pass True to gen2(), then the AssertionError will be completely masked, concealing the fact that any exception has been raised. ---------------------------------------------------------------------- >Comment By: Tristan Seligmann (mithrandi) Date: 2004-05-07 20:34 Message: Logged In: YES user_id=557968 Oops, providing example code as an attachment this time instead. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950057&group_id=5470 From noreply at sourceforge.net Fri May 7 15:15:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 15:15:55 2004 Subject: [ python-Bugs-929689 ] Python 2.3.3: socketmodule doesn't build on UnixWare 7.1.3 Message-ID: Bugs item #929689, was opened at 2004-04-05 13:28 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=929689&group_id=5470 Category: Build Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Larry Rosenman (ler) Assigned to: Nobody/Anonymous (nobody) Summary: Python 2.3.3: socketmodule doesn't build on UnixWare 7.1.3 Initial Comment: We seem to be running afoul of headers and redefinitions/etc for the socketmodule. The virgin source tarball manages to not get included before . If I force that issue, we get inconsistent definitions of h_errno for the threaded case. I can supply an account for anyone that wants to look. I've looked, and can't seem to fix the errors with h_errno easily. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-07 21:15 Message: Logged In: YES user_id=21627 The problem is not that sys/types.h is not included. The problem is that sys/types.h is included, but fails to define u_long. This, in itself, is not a bug, since when requesting XOPEN API, it may be reasonable not to define u_long. However, the real bug is that netinet/tcp.h, which should not use u_long if it is not defined. Somebody should report this bug to SCO. I've added a work-around in configure 1.416.4.14 configure.in 1.427.4.13 NEWS 1.831.4.107 configure 1.444 configure.in 1.455 ---------------------------------------------------------------------- Comment By: Larry Rosenman (ler) Date: 2004-04-05 13:31 Message: Logged In: YES user_id=36452 FWIW, the 2.2.2 version DOES compile on the same platform. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=929689&group_id=5470 From noreply at sourceforge.net Fri May 7 15:20:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 15:20:54 2004 Subject: [ python-Bugs-929689 ] Python 2.3.3: socketmodule doesn't build on UnixWare 7.1.3 Message-ID: Bugs item #929689, was opened at 2004-04-05 06:28 Message generated for change (Comment added) made by ler You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=929689&group_id=5470 Category: Build Group: Python 2.3 Status: Closed Resolution: Fixed Priority: 5 Submitted By: Larry Rosenman (ler) Assigned to: Nobody/Anonymous (nobody) Summary: Python 2.3.3: socketmodule doesn't build on UnixWare 7.1.3 Initial Comment: We seem to be running afoul of headers and redefinitions/etc for the socketmodule. The virgin source tarball manages to not get included before . If I force that issue, we get inconsistent definitions of h_errno for the threaded case. I can supply an account for anyone that wants to look. I've looked, and can't seem to fix the errors with h_errno easily. ---------------------------------------------------------------------- >Comment By: Larry Rosenman (ler) Date: 2004-05-07 14:20 Message: Logged In: YES user_id=36452 I've reported this to my SCO contacts. Thanks. Can I get a revised tarball from somewhere? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-07 14:15 Message: Logged In: YES user_id=21627 The problem is not that sys/types.h is not included. The problem is that sys/types.h is included, but fails to define u_long. This, in itself, is not a bug, since when requesting XOPEN API, it may be reasonable not to define u_long. However, the real bug is that netinet/tcp.h, which should not use u_long if it is not defined. Somebody should report this bug to SCO. I've added a work-around in configure 1.416.4.14 configure.in 1.427.4.13 NEWS 1.831.4.107 configure 1.444 configure.in 1.455 ---------------------------------------------------------------------- Comment By: Larry Rosenman (ler) Date: 2004-04-05 06:31 Message: Logged In: YES user_id=36452 FWIW, the 2.2.2 version DOES compile on the same platform. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=929689&group_id=5470 From noreply at sourceforge.net Fri May 7 15:29:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 15:29:22 2004 Subject: [ python-Bugs-929689 ] Python 2.3.3: socketmodule doesn't build on UnixWare 7.1.3 Message-ID: Bugs item #929689, was opened at 2004-04-05 13:28 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=929689&group_id=5470 Category: Build Group: Python 2.3 Status: Closed Resolution: Fixed Priority: 5 Submitted By: Larry Rosenman (ler) Assigned to: Nobody/Anonymous (nobody) Summary: Python 2.3.3: socketmodule doesn't build on UnixWare 7.1.3 Initial Comment: We seem to be running afoul of headers and redefinitions/etc for the socketmodule. The virgin source tarball manages to not get included before . If I force that issue, we get inconsistent definitions of h_errno for the threaded case. I can supply an account for anyone that wants to look. I've looked, and can't seem to fix the errors with h_errno easily. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-07 21:29 Message: Logged In: YES user_id=21627 Python 2.3.4 will be released shortly. I have attached the configure patch to this report. ---------------------------------------------------------------------- Comment By: Larry Rosenman (ler) Date: 2004-05-07 21:20 Message: Logged In: YES user_id=36452 I've reported this to my SCO contacts. Thanks. Can I get a revised tarball from somewhere? ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-07 21:15 Message: Logged In: YES user_id=21627 The problem is not that sys/types.h is not included. The problem is that sys/types.h is included, but fails to define u_long. This, in itself, is not a bug, since when requesting XOPEN API, it may be reasonable not to define u_long. However, the real bug is that netinet/tcp.h, which should not use u_long if it is not defined. Somebody should report this bug to SCO. I've added a work-around in configure 1.416.4.14 configure.in 1.427.4.13 NEWS 1.831.4.107 configure 1.444 configure.in 1.455 ---------------------------------------------------------------------- Comment By: Larry Rosenman (ler) Date: 2004-04-05 13:31 Message: Logged In: YES user_id=36452 FWIW, the 2.2.2 version DOES compile on the same platform. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=929689&group_id=5470 From noreply at sourceforge.net Fri May 7 16:50:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 16:50:51 2004 Subject: [ python-Bugs-950055 ] Is cmdqueue attribute used? Message-ID: Bugs item #950055, was opened at 2004-05-07 14:25 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950055&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Closed Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Guido van Rossum (gvanrossum) Summary: Is cmdqueue attribute used? Initial Comment: In cmd.py, the cmdqueue attribute is set to a list and checked for pending items but never is filled by any method nor is it in the documented API. Is the interface incomplete or is the attribute bogus? ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 16:50 Message: Logged In: YES user_id=6380 It may not be documented, but it is intended for use by subclasses, e.g. pdb.py. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950055&group_id=5470 From noreply at sourceforge.net Fri May 7 17:07:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 17:08:07 2004 Subject: [ python-Bugs-950055 ] Is cmdqueue attribute used? Message-ID: Bugs item #950055, was opened at 2004-05-07 13:25 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950055&group_id=5470 Category: Python Library Group: Python 2.3 Status: Closed Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Guido van Rossum (gvanrossum) Summary: Is cmdqueue attribute used? Initial Comment: In cmd.py, the cmdqueue attribute is set to a list and checked for pending items but never is filled by any method nor is it in the documented API. Is the interface incomplete or is the attribute bogus? ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-07 16:07 Message: Logged In: YES user_id=80475 It looks like the interface is limited to an append method. Would it be reasonable to make cmdqueue a deque() object and replace the pop(0) with a popleft()? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 15:50 Message: Logged In: YES user_id=6380 It may not be documented, but it is intended for use by subclasses, e.g. pdb.py. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950055&group_id=5470 From noreply at sourceforge.net Fri May 7 17:26:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 17:27:16 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None Priority: 7 Submitted By: Greg Jones (morngnstar) >Assigned to: Tim Peters (tim_one) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Fri May 7 17:28:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 17:29:30 2004 Subject: [ python-Bugs-950055 ] Is cmdqueue attribute used? Message-ID: Bugs item #950055, was opened at 2004-05-07 14:25 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950055&group_id=5470 Category: Python Library Group: Python 2.3 Status: Closed Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Guido van Rossum (gvanrossum) Summary: Is cmdqueue attribute used? Initial Comment: In cmd.py, the cmdqueue attribute is set to a list and checked for pending items but never is filled by any method nor is it in the documented API. Is the interface incomplete or is the attribute bogus? ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:28 Message: Logged In: YES user_id=6380 And what problem would making it a deque solve? And why shouldn't someone be allowed to remove a protocol from the list? Why not leave well enough alone? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-07 17:07 Message: Logged In: YES user_id=80475 It looks like the interface is limited to an append method. Would it be reasonable to make cmdqueue a deque() object and replace the pop(0) with a popleft()? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 16:50 Message: Logged In: YES user_id=6380 It may not be documented, but it is intended for use by subclasses, e.g. pdb.py. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950055&group_id=5470 From noreply at sourceforge.net Fri May 7 17:40:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 17:40:49 2004 Subject: [ python-Bugs-950055 ] Is cmdqueue attribute used? Message-ID: Bugs item #950055, was opened at 2004-05-07 13:25 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950055&group_id=5470 Category: Python Library Group: Python 2.3 Status: Closed Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Guido van Rossum (gvanrossum) Summary: Is cmdqueue attribute used? Initial Comment: In cmd.py, the cmdqueue attribute is set to a list and checked for pending items but never is filled by any method nor is it in the documented API. Is the interface incomplete or is the attribute bogus? ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-07 16:40 Message: Logged In: YES user_id=80475 Okay. Just checking. The idea was to establish habit and example for O(1) algorithms over O(n) approaches. Queing applications are what deque's where designed for -- they scale well and popleft() reads more clearly than .pop(0) Of course, this attribute only gets accessed once every million years, so the only real benefit is setting an example. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 16:28 Message: Logged In: YES user_id=6380 And what problem would making it a deque solve? And why shouldn't someone be allowed to remove a protocol from the list? Why not leave well enough alone? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-07 16:07 Message: Logged In: YES user_id=80475 It looks like the interface is limited to an append method. Would it be reasonable to make cmdqueue a deque() object and replace the pop(0) with a popleft()? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 15:50 Message: Logged In: YES user_id=6380 It may not be documented, but it is intended for use by subclasses, e.g. pdb.py. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950055&group_id=5470 From noreply at sourceforge.net Fri May 7 17:50:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 17:50:53 2004 Subject: [ python-Bugs-950055 ] Is cmdqueue attribute used? Message-ID: Bugs item #950055, was opened at 2004-05-07 14:25 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950055&group_id=5470 Category: Python Library Group: Python 2.3 Status: Closed Resolution: None Priority: 5 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Guido van Rossum (gvanrossum) Summary: Is cmdqueue attribute used? Initial Comment: In cmd.py, the cmdqueue attribute is set to a list and checked for pending items but never is filled by any method nor is it in the documented API. Is the interface incomplete or is the attribute bogus? ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:50 Message: Logged In: YES user_id=6380 Well, there is a reason why it took Python 14+ years to grow a specialized queue/deque datatype. One of Python's premises is that having a small number of fast general data structures is better than having many specialized ones, even if the specialized ones are sometimes faster: msot of the time you don't care so much about speed, and you just end up wasting time worrying (or arguing :-) about the right choice of data structure. Many Python programmers don't know O(n) from O(1), and shouldn't have to care, because their problems are never large enough for the difference to matter. I would suggest that the deque should be used only in situations where a concern for performance with a large number of items is realistic, otherwise I fear we risk make everyone speed freaks, worrying so much about optimizations that they forget to write correct code first. If you want examples, put them in the documentation; I'm not for making small changes to the stdlib all the time to use new features. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-07 17:40 Message: Logged In: YES user_id=80475 Okay. Just checking. The idea was to establish habit and example for O(1) algorithms over O(n) approaches. Queing applications are what deque's where designed for -- they scale well and popleft() reads more clearly than .pop(0) Of course, this attribute only gets accessed once every million years, so the only real benefit is setting an example. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:28 Message: Logged In: YES user_id=6380 And what problem would making it a deque solve? And why shouldn't someone be allowed to remove a protocol from the list? Why not leave well enough alone? ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-07 17:07 Message: Logged In: YES user_id=80475 It looks like the interface is limited to an append method. Would it be reasonable to make cmdqueue a deque() object and replace the pop(0) with a popleft()? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 16:50 Message: Logged In: YES user_id=6380 It may not be documented, but it is intended for use by subclasses, e.g. pdb.py. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950055&group_id=5470 From noreply at sourceforge.net Fri May 7 19:57:57 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 7 19:58:05 2004 Subject: [ python-Bugs-947906 ] calendar.weekheader(n): n should mean chars not bytes Message-ID: Bugs item #947906, was opened at 2004-05-05 03:38 Message generated for change (Comment added) made by perky You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947906&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Leonardo Rochael Almeida (rochael) Assigned to: Nobody/Anonymous (nobody) Summary: calendar.weekheader(n): n should mean chars not bytes Initial Comment: calendar.weekheader(n) is locale aware, which is good in principle. The parameter n, however, is interpreted as meaning bytes, not chars, which can generate broken strings for, e.g. localized weekday names: >>> calendar.weekheader(2) 'Mo Tu We Th Fr Sa Su' >>> locale.setlocale(locale.LC_ALL, "pt_BR.UTF-8") 'pt_BR.UTF-8' >>> calendar.weekheader(2) 'Se Te Qu Qu Se S\xc3 Do' Notice how "S?bado" (Saturday) above is missing the second utf-8 byte for the encoding of "?": >>> u"S?".encode("utf-8") 'S\xc3\xa1' The implementation of weekheader (and of all of calendar.py, it seems) is based on localized 8 bit strings. I suppose the correct fix for this bug will involve a roundtrip thru unicode. ---------------------------------------------------------------------- >Comment By: Hye-Shik Chang (perky) Date: 2004-05-08 08:57 Message: Logged In: YES user_id=55188 I think calendar.weekheader should mean not chars nor bytes but width. Because the function is currectly used for fixed width representations of calendars. Yes. They are same for western alphabets. But, for many of CJK characters are in full width. So, they need only 1 character for calendar.weekheader(2); and it's conventional in real life, too. But, we don't have unicode.width() support to implement the feature yet. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947906&group_id=5470 From noreply at sourceforge.net Sat May 8 00:02:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 8 00:02:34 2004 Subject: [ python-Bugs-950274 ] buffer() can point to freed memory Message-ID: Bugs item #950274, was opened at 2004-05-07 21:02 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950274&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: L. Peter Deutsch (lpd) Assigned to: Nobody/Anonymous (nobody) Summary: buffer() can point to freed memory Initial Comment: This is related to the issue in bug 216405, but different enough that I think it needs consideration. Sample code: import array t = array.array('b', 100 * 'x') u = buffer(t, 0, 100) del t[:] print u[:] You'll probably get something that is *not* 100 x's, because the buffer will point to (partly) deallocated storage. This problem can arise with any buffer if the object it references can change length. Buffers currently refuse to reference lists, but they don't refuse arrays. Frankly, I don't know what to recommend here. I was surprised when I found out that arrays can change length: my first suggestion is to remove this capability. If this isn't acceptable, I don't know what to suggest next. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950274&group_id=5470 From noreply at sourceforge.net Sat May 8 12:14:26 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 8 12:14:31 2004 Subject: [ python-Bugs-950482 ] -fconstant-cfstrings required for Xcode 1.2 Message-ID: Bugs item #950482, was opened at 2004-05-08 12:14 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950482&group_id=5470 Category: Macintosh Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Bob Ippolito (etrepum) Assigned to: Jack Jansen (jackjansen) Summary: -fconstant-cfstrings required for Xcode 1.2 Initial Comment: Xcode 1.2's compiler requires -fconstant-cfstrings in order to use CFSTR(...), so it should be added to Darwin's BASECFLAGS. It might need to be conditional, I'm not sure if previous versions of GCC supported the flag. cc (GCC) 3.3 20030304 (Apple Computer, Inc. build 1640) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950482&group_id=5470 From noreply at sourceforge.net Sat May 8 12:58:02 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 8 12:58:08 2004 Subject: [ python-Bugs-950274 ] buffer() can point to freed memory Message-ID: Bugs item #950274, was opened at 2004-05-08 04:02 Message generated for change (Comment added) made by nascheme You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950274&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: L. Peter Deutsch (lpd) >Assigned to: Neil Schemenauer (nascheme) Summary: buffer() can point to freed memory Initial Comment: This is related to the issue in bug 216405, but different enough that I think it needs consideration. Sample code: import array t = array.array('b', 100 * 'x') u = buffer(t, 0, 100) del t[:] print u[:] You'll probably get something that is *not* 100 x's, because the buffer will point to (partly) deallocated storage. This problem can arise with any buffer if the object it references can change length. Buffers currently refuse to reference lists, but they don't refuse arrays. Frankly, I don't know what to recommend here. I was surprised when I found out that arrays can change length: my first suggestion is to remove this capability. If this isn't acceptable, I don't know what to suggest next. ---------------------------------------------------------------------- >Comment By: Neil Schemenauer (nascheme) Date: 2004-05-08 16:58 Message: Logged In: YES user_id=35752 This problem has been fixed in the CVS version of Python for a while now. buffer objects no longer store pointers to memory but use the buffer API to get fresh ones whenever they are needed. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950274&group_id=5470 From noreply at sourceforge.net Sat May 8 15:28:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 8 15:28:47 2004 Subject: [ python-Bugs-932563 ] logging: need a way to discard Logger objects Message-ID: Bugs item #932563, was opened at 2004-04-09 21:51 Message generated for change (Comment added) made by vsajip You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=932563&group_id=5470 Category: Python Library Group: Feature Request Status: Open Resolution: None Priority: 5 Submitted By: Fred L. Drake, Jr. (fdrake) Assigned to: Vinay Sajip (vsajip) Summary: logging: need a way to discard Logger objects Initial Comment: There needs to be a way to tell the logging package that an application is done with a particular logger object. This is important for long-running processes that want to use a logger to represent a related set of activities over a relatively short period of time (compared to the life of the process). This is useful to allow creating per-connection loggers for internet servers, for example. Using a connection-specific logger allows the application to provide an identifier for the session that can be automatically included in the logs without having the application encode it into each message (a far more error prone approach). ---------------------------------------------------------------------- >Comment By: Vinay Sajip (vsajip) Date: 2004-05-08 19:28 Message: Logged In: YES user_id=308438 The problem with disposing of Logger objects programmatically is that you don't know who is referencing them. How about the following approach? I'm making no assumptions about the actual connection classes used; if you need to make it even less error prone, you can create delegating methods in the server class which do the appropriate wrapping. class ConnectionWrapper: def __init__(self, conn): self.conn = conn def message(self, msg): return "[connection: %s]: %s" % (self.conn, msg) and then use this like so... class Server: def get_connection(self, request): # return the connection for this request def handle_request(self, request): conn = self.get_connection(request) # we use a cache of connection wrappers if conn in self.conn_cache: cw = self.conn_cache[conn] else: cw = ConnectionWrapper(conn) self.conn_cache[conn] = cw #process request, and if events need to be logged, you can e.g. logger.debug(cw.message("Network packet truncated at %d bytes"), n) #The logged message will contain the connection ID ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=932563&group_id=5470 From noreply at sourceforge.net Sat May 8 16:37:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 8 16:37:53 2004 Subject: [ python-Bugs-950057 ] itertools.chain doesn't "process" exceptions as they occur Message-ID: Bugs item #950057, was opened at 2004-05-07 13:32 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950057&group_id=5470 Category: Python Library Group: None >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Tristan Seligmann (mithrandi) Assigned to: Raymond Hettinger (rhettinger) Summary: itertools.chain doesn't "process" exceptions as they occur Initial Comment: Python 2.3.3 (#2, Feb 24 2004, 09:29:20) [GCC 3.3.3 (Debian)] on linux2 The following code demonstrates the problem: import itertools def gen1(): print 'Before yield 1' yield 1 print 'After yield 1' assert False def gen2(x = False): print 'Before yield 2' yield 2 print 'After yield 2' if x: raise StopIteration for elem in itertools.chain(gen1(), gen2()): print elem The exception only propagates up after itertools.chain completes iteration; if you pass True to gen2(), then the AssertionError will be completely masked, concealing the fact that any exception has been raised. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-08 15:37 Message: Logged In: YES user_id=80475 Fix applied as Modules/itertoolsmodule.c 1.33 and 1.18.6.4. Thanks for the report. ---------------------------------------------------------------------- Comment By: Tristan Seligmann (mithrandi) Date: 2004-05-07 13:34 Message: Logged In: YES user_id=557968 Oops, providing example code as an attachment this time instead. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950057&group_id=5470 From noreply at sourceforge.net Sat May 8 19:57:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 8 19:57:15 2004 Subject: [ python-Bugs-832236 ] Build fails in ossaudiodev.c with missing macros Message-ID: Bugs item #832236, was opened at 2003-10-29 01:05 Message generated for change (Comment added) made by danielparks You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=832236&group_id=5470 Category: Build Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Daniel Parks (danielparks) Assigned to: Greg Ward (gward) Summary: Build fails in ossaudiodev.c with missing macros Initial Comment: I'm building Python 2.3.2 with no configure flags on MkLinux Pre-R1. uname -a: Linux gondor.middle.earth 2.0.37-osfmach3 GENERIC_08alpha-20 Fri Jul 30 11:07:38 PDT 1999 ppc unknown The missing macros are: SOUND_MIXER_DIGITAL1, SOUND_MIXER_DIGITAL2, SOUND_MIXER_DIGITAL3, SOUND_MIXER_PHONEIN, SOUND_MIXER_PHONEOUT, SOUND_MIXER_VIDEO, SOUND_MIXER_RADIO, SOUND_MIXER_MONITOR, SNDCTL_DSP_GETODELAY I commented out two lines in setup.py and it seems to have built correctly. I would test it, but I need to go to bed, and I will forget to add this bug tomorrow. I will update this if Python actually works. ---------------------------------------------------------------------- >Comment By: Daniel Parks (danielparks) Date: 2004-05-08 16:57 Message: Logged In: YES user_id=317142 Compiled Python 2.3.3 with the attached patch (sf-832236.patch). The build was successful. This is all I really wanted. [daniel@gondor Python-2.3.3]$ sudo ./python Lib/test/ test_ossaudiodev.py Password: playing test sound file... elapsed time: 5.4 sec Traceback (most recent call last): File "Lib/test/test_ossaudiodev.py", line 124, in ? test() File "Lib/test/test_ossaudiodev.py", line 122, in test test_setparameters() File "Lib/test/test_ossaudiodev.py", line 87, in test_setparameters raise RuntimeError("unable to set audio sampling parameters: " RuntimeError: unable to set audio sampling parameters: you must have really weird audio hardware I had to run the test as root because /dev/dsp was not accessible by other users. The audio clip played, but completely was unintelligible -- it sounded like a mistuned radio. This is the same machine as above; it is a PowerMac 7200/75, with whatever sound hardware is on the motherboard. I know that esound works properly on a PowerMac 6100/66 (which probably has similar audio hardware), but this is not nearly important enough for me to pursue. However, I am happy to test any patches. ---------------------------------------------------------------------- Comment By: Greg Ward (gward) Date: 2004-05-04 18:56 Message: Logged In: YES user_id=14422 Attached patch should fix the problem; I've emailed it to the original reporter to verify that it does. Will checkin/merge/close slip when I hear from him, or in a few days if I don't hear from him. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2003-12-23 11:21 Message: Logged In: YES user_id=11375 The fix is probably straightforward; add #ifdef ...#endif around each use of the problematic macros. If you do this, please submit a patch. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=832236&group_id=5470 From noreply at sourceforge.net Sat May 8 21:52:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 8 21:53:02 2004 Subject: [ python-Feature Requests-950644 ] Allow any lvalue for function definitions Message-ID: Feature Requests item #950644, was opened at 2004-05-09 01:52 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=950644&group_id=5470 Category: Parser/Compiler Group: None Status: Open Resolution: None Priority: 5 Submitted By: David Albert Torpey (dtorp) Assigned to: Nobody/Anonymous (nobody) Summary: Allow any lvalue for function definitions Initial Comment: A definition like: def M(x): return 2*x is the same as: M = lambda x: 2*x With the latter form, I can use any lvalue: A[0] = lambda x: 2*x B.f = lambda x: 2*x But with the first form, you're locked into just using a plain variable name. If this were fixed, it wouldn't break anything else but would be useful for making method definitons outside of a class definition: This came up when I was experimenting with David MacQuigg's ideas for prototype OO. I want to write something like: Account = Object.clone() Account.balance = 0 def Account.deposit(self, v): self.balance += v Unfortunately, the latter has to be written: def Account.deposit(self, v): self.balance += v Account.deposit = deposit ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=950644&group_id=5470 From noreply at sourceforge.net Sat May 8 21:54:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 8 21:54:28 2004 Subject: [ python-Feature Requests-935915 ] os.nullfilename Message-ID: Feature Requests item #935915, was opened at 2004-04-15 20:44 Message generated for change (Comment added) made by dtorp You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=935915&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: John Belmonte (jbelmonte) Assigned to: Nobody/Anonymous (nobody) Summary: os.nullfilename Initial Comment: Just as the os library provides os.sep, etc., for the current OS, it should provide the name of the null file (e.g., "/dev/null" or "nul:"), so that there is a portable way to open a null file. Use of an object such as class nullFile: def write(self, data): pass is not sufficient because it does not provide a full file object interface (no access to file descriptor, etc.). See discussion at . ---------------------------------------------------------------------- Comment By: David Albert Torpey (dtorp) Date: 2004-05-09 01:54 Message: Logged In: YES user_id=681258 I like this idea. It is more portable. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-04-15 20:52 Message: Logged In: YES user_id=21627 Move to feature requests tracker ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=935915&group_id=5470 From noreply at sourceforge.net Sun May 9 06:19:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 9 06:19:15 2004 Subject: [ python-Bugs-950747 ] email: mishandles Content-Transfer-Enc. for text/* messages Message-ID: Bugs item #950747, was opened at 2004-05-09 10:19 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950747&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Matthias Klose (doko) Assigned to: Nobody/Anonymous (nobody) Summary: email: mishandles Content-Transfer-Enc. for text/* messages Initial Comment: [forwarded from http://bugs.debian.org/247792] Normally when creating MIME attachements with the email module, the user must explicitly encode (base64 or quoted-printable) the message using encode_base64() or encode_quopri(), for example. That sets the Content-Transfer-Encoding header, and replaces the payload with an encoded version which is then emitted verbatim by the Generator. However, for text/ objects, the Content-Transfer-Encoding header is set when the payload is attached, according to the character set. The payload is not encoded at that point, but is instead encoded according to the charset's default body encoding by the Generator at flatten() time. This means in particular that using encode_*() on a text/* message with a non-None charset will result in a malformed message: it will have multiple Content-Transfer-Encoding headers, and multiply encoded body. For added confusion the multiple body encodings won't even be applied in the same order as the duplicate headers appear (as they would if multiple encode_*() functions were applied in sequence). It also means it is impossible to override a charset's default encoding. For example utf-8 text will always be base64 encoded, even though utf-8 text in western languages is likely to be almost entirely 7bit, so quoted-printable or even 8bit encoding would be more appropriate. The payload data of the Message object should consistently hold either encoded or unencoded data. If the latter, the Generator should take its cue from the Content-Transfer-Encoding header, not from the charset. If the former, the encode_* functions should recognize an already encoded message and recode it, or at least throw an exception rather than generating a malformed MIME message. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950747&group_id=5470 From noreply at sourceforge.net Sun May 9 12:08:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 9 12:08:23 2004 Subject: [ python-Feature Requests-935915 ] os.nullfilename Message-ID: Feature Requests item #935915, was opened at 2004-04-15 22:44 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=935915&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: John Belmonte (jbelmonte) Assigned to: Nobody/Anonymous (nobody) Summary: os.nullfilename Initial Comment: Just as the os library provides os.sep, etc., for the current OS, it should provide the name of the null file (e.g., "/dev/null" or "nul:"), so that there is a portable way to open a null file. Use of an object such as class nullFile: def write(self, data): pass is not sufficient because it does not provide a full file object interface (no access to file descriptor, etc.). See discussion at . ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-09 18:08 Message: Logged In: YES user_id=21627 Would you like to work on a patch? ---------------------------------------------------------------------- Comment By: David Albert Torpey (dtorp) Date: 2004-05-09 03:54 Message: Logged In: YES user_id=681258 I like this idea. It is more portable. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-04-15 22:52 Message: Logged In: YES user_id=21627 Move to feature requests tracker ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=935915&group_id=5470 From noreply at sourceforge.net Sun May 9 13:20:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 9 13:20:23 2004 Subject: [ python-Bugs-895510 ] email library Parser can't parse some messages Message-ID: Bugs item #895510, was opened at 2004-02-12 02:42 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=895510&group_id=5470 Category: Python Library Group: Python 2.2.3 >Status: Closed >Resolution: Wont Fix Priority: 5 Submitted By: Jan Ondrej (ondrejj) Assigned to: Barry A. Warsaw (bwarsaw) Summary: email library Parser can't parse some messages Initial Comment: The email library can't decode some emails with one bad part. This part is ignored (not decoded) by other software. I am developing an antivir software and I need to decode these messages. There is the code, which raises an error: if line[0] in ' \t': if not lastheader: print "Errors.HeaderParseError" raise Errors.HeaderParseError( 'Continuation line seen before first header') There is an example attached. It contains a space before "Content-Type:" in first part. This part is buggy, but next part can contain important data. If it is not a bug but an RFC implementation, please tell me, how I can decode these emails. I think, this problem is in all versions of this library (also in CVS), because this code is not changed in it. ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-09 13:20 Message: Logged In: YES user_id=12800 Sorry, but that's a limitation of the design of the email2 Parser. You should probably have a look at the email3 FeedParser which will parse the sample message, adding 'defects' for the bogus headers. email3 will be part of Python 2.4. Please join the email-sig if you want to contribute to the design. http://www.python.org/sigs/email-sig ---------------------------------------------------------------------- Comment By: Jan Ondrej (ondrejj) Date: 2004-02-12 02:45 Message: Logged In: YES user_id=799485 I am sorry, file was not attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=895510&group_id=5470 From noreply at sourceforge.net Sun May 9 14:01:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 9 14:01:21 2004 Subject: [ python-Bugs-884030 ] MIMEMultipart _subparts constructor unusable Message-ID: Bugs item #884030, was opened at 2004-01-25 02:19 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=884030&group_id=5470 Category: Python Library Group: None >Status: Closed >Resolution: Accepted Priority: 5 Submitted By: Amit Aronovitch (amitar) Assigned to: Barry A. Warsaw (bwarsaw) Summary: MIMEMultipart _subparts constructor unusable Initial Comment: It is not clear how to pass initial list of messages to MIMEMultipart. >From doc it seems that _subparts should be a single list-like object, from source it seems that it expects seperate parameters - neither work... My suggested patch implemets the single parameter way (after applying it, the second try in the example below does work). ((to apply the patch, use : cp MIMEMultipart.py MIMEMultipart_orig.py patch -i MIMEMultipart_2.3.3.diff -o MIMEMultipart.py MIMEMultipart_orig.py )) example: >>> msgs = [MIMEText('a'),MIMEText('b')] >>> MIMEMultipart('mixed',None,msgs[0],msgs[1]) Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.3/email/MIMEMultipart.py", line 35, in __init__ self.attach(*list(_subparts)) TypeError: attach() takes exactly 2 arguments (3 given) >>> MIMEMultipart('mixed',None,msgs).as_string() Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.3/email/Message.py", line 130, in as_string g.flatten(self, unixfrom=unixfrom) File "/usr/lib/python2.3/email/Generator.py", line 102, in flatten self._write(msg) File "/usr/lib/python2.3/email/Generator.py", line 130, in _write self._dispatch(msg) File "/usr/lib/python2.3/email/Generator.py", line 156, in _dispatch meth(msg) File "/usr/lib/python2.3/email/Generator.py", line 230, in _handle_multipart g.flatten(part, unixfrom=False) File "/usr/lib/python2.3/email/Generator.py", line 102, in flatten self._write(msg) File "/usr/lib/python2.3/email/Generator.py", line 130, in _write self._dispatch(msg) File "/usr/lib/python2.3/email/Generator.py", line 147, in _dispatch main = msg.get_content_maintype() AttributeError: 'list' object has no attribute 'get_content_maintype' >>> ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-09 14:01 Message: Logged In: YES user_id=12800 Your patch seems right since it implements what's documented. I'll apply this for Python 2.3.4 and Python 2.4 (with a bit of reformatting and a test case). ---------------------------------------------------------------------- Comment By: Amit Aronovitch (amitar) Date: 2004-01-25 02:28 Message: Logged In: YES user_id=564711 This refers to the email package... To my example I should have added at the beginning: >>> from email.MIMEText import MIMEText >>> from email.MIMEMultipart import MIMEMultipart ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=884030&group_id=5470 From noreply at sourceforge.net Sun May 9 14:23:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 9 14:23:39 2004 Subject: [ python-Bugs-467924 ] Improve the ZipFile Interface Message-ID: Bugs item #467924, was opened at 2001-10-04 11:54 Message generated for change (Comment added) made by myers_carpenter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=467924&group_id=5470 Category: Python Library Group: Feature Request Status: Open Resolution: None Priority: 5 Submitted By: Nobody/Anonymous (nobody) Assigned to: Greg Ward (gward) Summary: Improve the ZipFile Interface Initial Comment: There exist two methods to write to a ZipFile write(self, filename, arcname=None, compress_type=None) writestr(self, zinfo, bytes) but only one to read from it read(self, name) Additionally, the two 'write's behave differently with respect to compression. --- (a) 'read' does not fit to 'write', since 'write' takes a file and adds it to a ZipFile, but 'read' is not the reverse operation. 'read' should be called 'readstr' since it much better matches to 'writestr'. (b) It is confusing what 'write' and 'read' actually mean. Does 'write' write a file, or into the ZipFile? It would be more obvious if ZipFile has 4 methods which pair-wise fit together: writestr (self, zinfo, bytes) # same as now readstr (self, name) # returns bytes (as string), currently called 'read' # 'read' could still live but should be deprecated add (self, filename, arcname=None, compress_type=None) # currently 'write' # 'write' could still live but should be deprecated extract (self, name, filename, arcname=None) # new, desired functionality (c) BOTH, 'writestr' and 'add' should by default use the 'compress_type' that was passed to the constructor of 'ZipFile'. Currently, 'write' does it, 'writestr' via zinfo does it not. 'ZipInfo' sets the compression strict to 'ZIP_STORED' :-( It should not do that! It rather should: - allow more parameters in the signature of the constructor to also pass the compression type (and some other attributes, too) - default to 'None', so that 'writestr' can see this, and then take the default from the 'ZipFile' instance. ---------------------------------------------------------------------- Comment By: Myers Carpenter (myers_carpenter) Date: 2004-05-09 14:23 Message: Logged In: YES user_id=335935 The zipfile interface should match the tarfile interface. At the mininum is should work for this example: import zipfile zip = zipfile.open("sample.zip", "r") for zipinfo in zip: print tarinfo.name, "is", tarinfo.size, "bytes in size and is", zip.extract(zipinfo) zip.close() This closely matchs the 'tarfile' module. ---------------------------------------------------------------------- Comment By: Matt Zimmerman (mzimmerman) Date: 2003-07-31 10:22 Message: Logged In: YES user_id=196786 It would also be very useful to be able to have ZipFile read/write the uncompressed file data from/to a file-like object, instead of just strings and files (respectively). I would like to use this module to work with zip files containing large files, but this is unworkable because the current implementation would use excessive amounts of memory. Currently, read() reads all of the compressed data into memory, then uncompresses it into memory. For files which may be hundreds of megabytes compressed, this is undesirable. Likewise for write(), I would like to be able to stream data into a zip file, passing in a ZipInfo to specify the metadata as is done with writestr(). The implementation of this functionality is quite straightforward, but I am not sure whether (or how) the interface should change. Some other parts of the library allow for a file object to be passed to the same interface which accepts a filename. The object is examined to see if it has the necessary read/write methods and if not, it is assumed to be a filename. Would this be the correct way to do it? I, too, am a bit irked by the lack of symmetry exhibited by read vs. write/writestr, and think that the interface suggested above would be a significant improvement. ---------------------------------------------------------------------- Comment By: Just van Rossum (jvr) Date: 2003-01-05 15:54 Message: Logged In: YES user_id=92689 In Python 2.3, writestr() has an enhanced signature: the first arg may now also be an archive name, in which case the correct default settings are used (ie. the compression value is taken from the file). See patch #651621. extract() could be moderately useful (although I don't understand the 'arcname' arg, how's that different from 'name'?) but would have to deal with file modes (bin/text). The file mode isn't in the archive so would have to (optionally) be supplied by the caller. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=467924&group_id=5470 From noreply at sourceforge.net Sun May 9 19:45:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 9 19:45:56 2004 Subject: [ python-Bugs-602245 ] os.popen() negative error code IOError Message-ID: Bugs item #602245, was opened at 2002-08-30 16:25 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=602245&group_id=5470 Category: Windows Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Tim Delaney (tcdelaney) Assigned to: Mark Hammond (mhammond) Summary: os.popen() negative error code IOError Initial Comment: Windows 2000, Python 2.0.1, 2.1.1, 2.2. When a negative return code is received by the os.popen () family, an IOError is raised when the last pipe from the process is closed. The following code demonstrates the problem: import sys import os import traceback if __name__ == '__main__': if len(sys.argv) == 1: try: r = os.popen('%s %s %s' % (sys.executable, sys.argv[0], -1,)) r.close() except IOError: traceback.print_exc() try: w, r = os.popen2('%s %s %s' % (sys.executable, sys.argv[0], -1,)) w.close() r.close() except IOError: traceback.print_exc() try: w, r, e = os.popen3('%s %s %s' % (sys.executable, sys.argv[0], -1,)) w.close() r.close() e.close() except IOError: traceback.print_exc() else: sys.exit(int(sys.argv[1])) ---------- Run ---------- Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 11, in ? r.close() IOError: (0, 'Error') Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 18, in ? r.close() IOError: (0, 'Error') Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 26, in ? e.close() IOError: (0, 'Error') ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-05-10 09:45 Message: Logged In: YES user_id=14198 I tried truncating as you suggest, and this works fine - although it gives different results than Linux. Digging deeper, I found for linux: """ The value from the wait() system call is split into three parts mask 0xff00 is the exit status mask 0x007f is the signal causing the exit mask 0x0080 is the cord dumped flag in combination with a signal """ To get the same results as Linux for negative exit codes, it seems: result = (int)((exit_code & 0x000000FF) << 8); Will do the job. Counter-intuitively, this will make sys.exit(-1) yield an exit code of 65280, but this exactly what happens on Linux. Should we try and strive for compatibility within Python, or compatibility with the host OS? I lean towards compatibility with Linux - if they want win32 specific stuff, use win32 specific code. Thoughts? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-05 23:18 Message: Logged In: YES user_id=6380 How about changing the line "result = exit_code;" in _PyPclose() to truncate the value to the lower 16 bits, ensuring it is always >= 0? I think that should do it. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-05 22:57 Message: Logged In: YES user_id=14198 I'm not sure what to do about this. Python internally checks for -1 from the close function. The safest thing to do would be to translate the return code from -1 to something else. Changing the fclose semantics to check for PyErr_Occurred is another alternative, but doesn't seem that reasonable just for this. Any thoughts? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2002-08-31 00:05 Message: Logged In: YES user_id=6380 I suppose this is one for Mark Hammond... ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2002-08-30 22:38 Message: Logged In: YES user_id=44345 Attached Tim's code so it's more readily accessible to testers. BTW, works fine for me on Linux. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=602245&group_id=5470 From noreply at sourceforge.net Sun May 9 21:13:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 9 21:14:15 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None Priority: 7 Submitted By: Greg Jones (morngnstar) >Assigned to: Guido van Rossum (gvanrossum) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Mon May 10 03:17:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 03:17:31 2004 Subject: [ python-Bugs-951088 ] str(email) breaks message by adding newline Message-ID: Bugs item #951088, was opened at 2004-05-10 09:17 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951088&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Gerrit Holl (gerrit) Assigned to: Nobody/Anonymous (nobody) Summary: str(email) breaks message by adding newline Initial Comment: Since last nightly build, the following code breaks: import email message="""From gerrit@topjaklont.student.utwente.nl Mon May 10 09:06:55 2004 Return-Path: X-Original-To: gerrit Delivered-To: gerrit@topjaklont.student.utwente.nl Received: by topjaklont.student.utwente.nl (Postfix, from userid 500) id 866FE1F835; Mon, 10 May 2004 09:06:55 +0200 (CEST) To: gerrit@topjaklont.student.utwente.nl Message-Id: <20040510070655.866FE1F835@topjaklont.student.utwente.nl> Date: Mon, 10 May 2004 09:06:55 +0200 (CEST) From: gerrit@topjaklont.student.utwente.nl (Gerrit Holl) content """ msg = email.message_from_string(message) print str(msg) The output is: >From gerrit@topjaklont.student.utwente.nl Mon May 10 09:06:55 2004 Return-Path: X-Original-To: gerrit Delivered-To: gerrit@topjaklont.student.utwente.nl Received: by topjaklont.student.utwente.nl (Postfix, from userid 500) id 866FE1F835; Mon, 10 May 2004 09:06:55 +0200 (CEST) To: gerrit@topjaklont.student.utwente.nl Message-Id: <20040510070655.866FE1F835@topjaklont.student.utwente.nl> Date: Mon, 10 May 2004 09:06:55 +0200 (CEST) From: gerrit@topjaklont.student.utwente.nl (Gerrit Holl) content My mail client doesn't like this. The test script has been attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951088&group_id=5470 From noreply at sourceforge.net Mon May 10 03:39:08 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 03:40:37 2004 Subject: [ python-Bugs-738973 ] urllib2 CacheFTPHandler doesn't work on multiple dirs Message-ID: Bugs item #738973, was opened at 2003-05-17 08:18 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=738973&group_id=5470 Category: Python Library Group: Python 2.2.2 >Status: Closed >Resolution: Fixed Priority: 6 Submitted By: Shahms E. King (shahms) Assigned to: Mark Hammond (mhammond) Summary: urllib2 CacheFTPHandler doesn't work on multiple dirs Initial Comment: using the CacheFTPHandler for ftp requests in urllib2 works as expected for files in the same directory as the original file, however, as ftpwrapper() changes the directory only after the initial connection, any urllib2.urlopen('ftp://...') that is in a different directory that the initial urlopen() call will fail with a "450: File Not Found" ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-05-10 17:39 Message: Logged In: YES user_id=14198 Fixed on 2.3 branch: Checking in urllib2.py; new revision: 1.53.6.6; previous revision: 1.53.6.5 Fixed on trunk: new revision: 1.66; previous revision: 1.65 ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-05 22:24 Message: Logged In: YES user_id=14198 I reviewed and tested the patch, and it looks good to me. I'll take this on (for trunk and 2.3.4) unless someone objects. ---------------------------------------------------------------------- Comment By: John J Lee (jjlee) Date: 2003-12-01 09:12 Message: Logged In: YES user_id=261020 I've submitted a fix for this as patch 851736. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=738973&group_id=5470 From noreply at sourceforge.net Mon May 10 05:52:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 05:52:26 2004 Subject: [ python-Bugs-933068 ] Different behaviour using super() and explicit base class. Message-ID: Bugs item #933068, was opened at 2004-04-11 13:59 Message generated for change (Comment added) made by ncoghlan You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=933068&group_id=5470 Category: Threads Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Lawrence Allan Jones (lajones) Assigned to: Nobody/Anonymous (nobody) Summary: Different behaviour using super() and explicit base class. Initial Comment: I created a child class of threading.Thread. To invoke the parent __init__ method, I called super(C, self).__init__(). When called at run-time, Python threw an assertion exception informing me that the group parameter must be None. (The detailed error message was assert group is None, "group argument must be None for now".) I had not changed this parameter. I created a similar class and explicitly called threading.Thread.__init__(). This call did not exhibit the fault at run-time. ---------------------------------------------------------------------- Comment By: Nick Coghlan (ncoghlan) Date: 2004-05-10 19:52 Message: Logged In: YES user_id=1038590 This looks like a usage problem to me - the call to super() returns an already bound method object, hence 'self' does not need to be supplied as an argument to the __init__ call. That is, the correct call to fix the 'Broken' class is: super(Broken, self).__init__() ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=933068&group_id=5470 From noreply at sourceforge.net Mon May 10 09:18:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 09:18:57 2004 Subject: [ python-Bugs-951235 ] strptime doesn't account for %p in output Message-ID: Bugs item #951235, was opened at 2004-05-10 08:18 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951235&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Skip Montanaro (montanaro) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't account for %p in output Initial Comment: Correct use of the "%p" format is not obvious: >>> time.strptime("3:15pm", '%H:%M%p') (1900, 1, 1, 3, 15, 0, 0, 1, -1) >>> time.strptime("3:15", '%H:%M') (1900, 1, 1, 3, 15, 0, 0, 1, -1) >>> time.strptime("3:15am", '%H:%M%p') (1900, 1, 1, 3, 15, 0, 0, 1, -1) Other than successfully parsing 'am' or 'pm', there's no indication in the returned time tuple of a 12-hour offset in the 'pm' case. I think the hour field returned from the first call should be 15, not 3. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951235&group_id=5470 From noreply at sourceforge.net Mon May 10 09:21:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 09:23:27 2004 Subject: [ python-Bugs-951235 ] strptime doesn't account for %p in output Message-ID: Bugs item #951235, was opened at 2004-05-10 08:18 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951235&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Skip Montanaro (montanaro) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't account for %p in output Initial Comment: Correct use of the "%p" format is not obvious: >>> time.strptime("3:15pm", '%H:%M%p') (1900, 1, 1, 3, 15, 0, 0, 1, -1) >>> time.strptime("3:15", '%H:%M') (1900, 1, 1, 3, 15, 0, 0, 1, -1) >>> time.strptime("3:15am", '%H:%M%p') (1900, 1, 1, 3, 15, 0, 0, 1, -1) Other than successfully parsing 'am' or 'pm', there's no indication in the returned time tuple of a 12-hour offset in the 'pm' case. I think the hour field returned from the first call should be 15, not 3. ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2004-05-10 08:21 Message: Logged In: YES user_id=44345 Another corner case involving am/pm: >>> time.strptime("15:15pm", '%H:%M%p') (1900, 1, 1, 15, 15, 0, 0, 1, -1) >>> time.strptime("15:15am", '%H:%M%p') (1900, 1, 1, 15, 15, 0, 0, 1, -1) Shouldn't both raise ValueError since the input hour was not < 13? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951235&group_id=5470 From noreply at sourceforge.net Mon May 10 09:49:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 09:49:22 2004 Subject: [ python-Bugs-951088 ] str(email) breaks message by adding newline Message-ID: Bugs item #951088, was opened at 2004-05-10 03:17 Message generated for change (Settings changed) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951088&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Gerrit Holl (gerrit) >Assigned to: Barry A. Warsaw (bwarsaw) Summary: str(email) breaks message by adding newline Initial Comment: Since last nightly build, the following code breaks: import email message="""From gerrit@topjaklont.student.utwente.nl Mon May 10 09:06:55 2004 Return-Path: X-Original-To: gerrit Delivered-To: gerrit@topjaklont.student.utwente.nl Received: by topjaklont.student.utwente.nl (Postfix, from userid 500) id 866FE1F835; Mon, 10 May 2004 09:06:55 +0200 (CEST) To: gerrit@topjaklont.student.utwente.nl Message-Id: <20040510070655.866FE1F835@topjaklont.student.utwente.nl> Date: Mon, 10 May 2004 09:06:55 +0200 (CEST) From: gerrit@topjaklont.student.utwente.nl (Gerrit Holl) content """ msg = email.message_from_string(message) print str(msg) The output is: >From gerrit@topjaklont.student.utwente.nl Mon May 10 09:06:55 2004 Return-Path: X-Original-To: gerrit Delivered-To: gerrit@topjaklont.student.utwente.nl Received: by topjaklont.student.utwente.nl (Postfix, from userid 500) id 866FE1F835; Mon, 10 May 2004 09:06:55 +0200 (CEST) To: gerrit@topjaklont.student.utwente.nl Message-Id: <20040510070655.866FE1F835@topjaklont.student.utwente.nl> Date: Mon, 10 May 2004 09:06:55 +0200 (CEST) From: gerrit@topjaklont.student.utwente.nl (Gerrit Holl) content My mail client doesn't like this. The test script has been attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951088&group_id=5470 From noreply at sourceforge.net Mon May 10 10:09:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 10:09:46 2004 Subject: [ python-Bugs-950747 ] email: mishandles Content-Transfer-Enc. for text/* messages Message-ID: Bugs item #950747, was opened at 2004-05-09 06:19 Message generated for change (Settings changed) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950747&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Matthias Klose (doko) >Assigned to: Barry A. Warsaw (bwarsaw) Summary: email: mishandles Content-Transfer-Enc. for text/* messages Initial Comment: [forwarded from http://bugs.debian.org/247792] Normally when creating MIME attachements with the email module, the user must explicitly encode (base64 or quoted-printable) the message using encode_base64() or encode_quopri(), for example. That sets the Content-Transfer-Encoding header, and replaces the payload with an encoded version which is then emitted verbatim by the Generator. However, for text/ objects, the Content-Transfer-Encoding header is set when the payload is attached, according to the character set. The payload is not encoded at that point, but is instead encoded according to the charset's default body encoding by the Generator at flatten() time. This means in particular that using encode_*() on a text/* message with a non-None charset will result in a malformed message: it will have multiple Content-Transfer-Encoding headers, and multiply encoded body. For added confusion the multiple body encodings won't even be applied in the same order as the duplicate headers appear (as they would if multiple encode_*() functions were applied in sequence). It also means it is impossible to override a charset's default encoding. For example utf-8 text will always be base64 encoded, even though utf-8 text in western languages is likely to be almost entirely 7bit, so quoted-printable or even 8bit encoding would be more appropriate. The payload data of the Message object should consistently hold either encoded or unencoded data. If the latter, the Generator should take its cue from the Content-Transfer-Encoding header, not from the charset. If the former, the encode_* functions should recognize an already encoded message and recode it, or at least throw an exception rather than generating a malformed MIME message. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=950747&group_id=5470 From noreply at sourceforge.net Mon May 10 10:18:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 10:18:46 2004 Subject: [ python-Bugs-951270 ] Generator expression used in doc example Message-ID: Bugs item #951270, was opened at 2004-05-11 00:18 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951270&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Nick Coghlan (ncoghlan) Assigned to: Nobody/Anonymous (nobody) Summary: Generator expression used in doc example Initial Comment: One of the examples in test_deque uses a generator expression to populate the deque. This means the doctest run fails for test_deque when run without the generator expressions patch. ncoghlan@localhost src]$ ./python ./Lib/test/test_deque.py test_basics (__main__.TestBasic) ... ok test_big_queue_popleft (__main__.TestBasic) ... ok test_big_queue_popright (__main__.TestBasic) ... ok test_big_stack_left (__main__.TestBasic) ... ok test_big_stack_right (__main__.TestBasic) ... ok test_clear (__main__.TestBasic) ... ok test_comparisons (__main__.TestBasic) ... ok test_copy (__main__.TestBasic) ... ok test_deepcopy (__main__.TestBasic) ... ok test_extend (__main__.TestBasic) ... ok test_extendleft (__main__.TestBasic) ... ok test_getitem (__main__.TestBasic) ... ok test_hash (__main__.TestBasic) ... ok test_len (__main__.TestBasic) ... ok test_long_steadystate_queue_popleft (__main__.TestBasic) ... ok test_long_steadystate_queue_popright (__main__.TestBasic) ... ok test_pickle (__main__.TestBasic) ... ok test_print (__main__.TestBasic) ... ok test_repr (__main__.TestBasic) ... ok test_reversed (__main__.TestBasic) ... ok test_rotate (__main__.TestBasic) ... ok test_roundtrip_iter_init (__main__.TestBasic) ... ok test_setitem (__main__.TestBasic) ... ok test_underflow (__main__.TestBasic) ... ok test_constructor (__main__.TestVariousIteratorArgs) ... ok test_iter_with_altered_data (__main__.TestVariousIteratorArgs) ... ok test_basics (__main__.TestSubclass) ... ok test_copy_pickle (__main__.TestSubclass) ... ok ---------------------------------------------------------------------- Ran 28 tests in 1.127s OK ***************************************************************** Failure in example: def roundrobin(*iterables): pending = deque(iter(i) for i in iterables) while pending: task = pending.popleft() try: yield task.next() except StopIteration: continue pending.append(task) from line #66 of test.test_deque.__test__.libreftest Exception raised: Traceback (most recent call last): File "/home/ncoghlan/python/src/python/dist/src/Lib/doctest.py", line 442, in _run_examples_inner compileflags, 1) in globs File "", line 2 pending = deque(iter(i) for i in iterables) ^ SyntaxError: invalid syntax ***************************************************************** Failure in example: for value in roundrobin('abc', 'd', 'efgh'): print value from line #77 of test.test_deque.__test__.libreftest Exception raised: Traceback (most recent call last): File "/home/ncoghlan/python/src/python/dist/src/Lib/doctest.py", line 442, in _run_examples_inner compileflags, 1) in globs File "", line 1, in ? NameError: name 'roundrobin' is not defined ***************************************************************** 1 items had failures: 2 of 32 in test.test_deque.__test__.libreftest ***Test Failed*** 2 failures. Traceback (most recent call last): File "./Lib/test/test_deque.py", line 561, in ? test_main(verbose=True) File "./Lib/test/test_deque.py", line 558, in test_main test_support.run_doctest(test_deque, verbose) File "/home/ncoghlan/python/src/python/dist/src/Lib/test/test_support.py", line 318, in run_doctest raise TestFailed("%d of %d doctests failed" % (f, t)) test.test_support.TestFailed: 2 of 32 doctests failed [ncoghlan@localhost src]$ ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951270&group_id=5470 From noreply at sourceforge.net Mon May 10 10:29:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 10:29:19 2004 Subject: [ python-Bugs-951275 ] ihooks chokes on BioPython package Message-ID: Bugs item #951275, was opened at 2004-05-10 09:29 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951275&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: mpalmer (waterloo) Assigned to: Nobody/Anonymous (nobody) Summary: ihooks chokes on BioPython package Initial Comment: I'm trying to use the BioPython package (www.biopython.org) with Webware. Webware uses ihooks. Ihooks chokes on importing Bio: >>>import ihooks >>>ihooks.install() >>>import Bio gives an error. Here is the traceback: Traceback (most recent call last): File "weg.py", line 5, in ? from Bio import GenBank File "C:\PYTHON23\lib\ihooks.py", line 398, in import_module q, tail = self.find_head_package(parent, str(name)) File "C:\PYTHON23\lib\ihooks.py", line 434, in find_head_package q = self.import_it(head, qname, parent) File "C:\PYTHON23\lib\ihooks.py", line 489, in import_it m = self.loader.load_module(fqname, stuff) File "C:\PYTHON23\lib\ihooks.py", line 274, in load_module m = self.hooks.load_package(name, filename, file) File "C:\PYTHON23\lib\ihooks.py", line 174, in load_package return imp.load_module(name, file, filename, ("", "", PKG_DIRECTORY)) File "C:\PYTHON23\Lib\site-packages\Bio\__init__.py", line 106, in ? _load_registries() File "C:\PYTHON23\Lib\site-packages\Bio\__init__.py", line 83, in _load_registries config_imports = __import__("Bio.config", {}, {}, ["Bio"]) File "C:\PYTHON23\lib\ihooks.py", line 403, in import_module self.ensure_fromlist(m, fromlist) File "C:\PYTHON23\lib\ihooks.py", line 470, in ensure_fromlist raise ImportError, "No module named " + subname ImportError: No module named Bio.config.Bio After >>>ihooks.uninstall() the import works. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951275&group_id=5470 From noreply at sourceforge.net Mon May 10 10:32:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 10:32:29 2004 Subject: [ python-Bugs-951270 ] Generator expression used in doc example Message-ID: Bugs item #951270, was opened at 2004-05-10 09:18 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951270&group_id=5470 Category: Python Library Group: Python 2.4 >Status: Pending >Resolution: Remind Priority: 5 Submitted By: Nick Coghlan (ncoghlan) >Assigned to: Raymond Hettinger (rhettinger) Summary: Generator expression used in doc example Initial Comment: One of the examples in test_deque uses a generator expression to populate the deque. This means the doctest run fails for test_deque when run without the generator expressions patch. ncoghlan@localhost src]$ ./python ./Lib/test/test_deque.py test_basics (__main__.TestBasic) ... ok test_big_queue_popleft (__main__.TestBasic) ... ok test_big_queue_popright (__main__.TestBasic) ... ok test_big_stack_left (__main__.TestBasic) ... ok test_big_stack_right (__main__.TestBasic) ... ok test_clear (__main__.TestBasic) ... ok test_comparisons (__main__.TestBasic) ... ok test_copy (__main__.TestBasic) ... ok test_deepcopy (__main__.TestBasic) ... ok test_extend (__main__.TestBasic) ... ok test_extendleft (__main__.TestBasic) ... ok test_getitem (__main__.TestBasic) ... ok test_hash (__main__.TestBasic) ... ok test_len (__main__.TestBasic) ... ok test_long_steadystate_queue_popleft (__main__.TestBasic) ... ok test_long_steadystate_queue_popright (__main__.TestBasic) ... ok test_pickle (__main__.TestBasic) ... ok test_print (__main__.TestBasic) ... ok test_repr (__main__.TestBasic) ... ok test_reversed (__main__.TestBasic) ... ok test_rotate (__main__.TestBasic) ... ok test_roundtrip_iter_init (__main__.TestBasic) ... ok test_setitem (__main__.TestBasic) ... ok test_underflow (__main__.TestBasic) ... ok test_constructor (__main__.TestVariousIteratorArgs) ... ok test_iter_with_altered_data (__main__.TestVariousIteratorArgs) ... ok test_basics (__main__.TestSubclass) ... ok test_copy_pickle (__main__.TestSubclass) ... ok ---------------------------------------------------------------------- Ran 28 tests in 1.127s OK ***************************************************************** Failure in example: def roundrobin(*iterables): pending = deque(iter(i) for i in iterables) while pending: task = pending.popleft() try: yield task.next() except StopIteration: continue pending.append(task) from line #66 of test.test_deque.__test__.libreftest Exception raised: Traceback (most recent call last): File "/home/ncoghlan/python/src/python/dist/src/Lib/doctest.py", line 442, in _run_examples_inner compileflags, 1) in globs File "", line 2 pending = deque(iter(i) for i in iterables) ^ SyntaxError: invalid syntax ***************************************************************** Failure in example: for value in roundrobin('abc', 'd', 'efgh'): print value from line #77 of test.test_deque.__test__.libreftest Exception raised: Traceback (most recent call last): File "/home/ncoghlan/python/src/python/dist/src/Lib/doctest.py", line 442, in _run_examples_inner compileflags, 1) in globs File "", line 1, in ? NameError: name 'roundrobin' is not defined ***************************************************************** 1 items had failures: 2 of 32 in test.test_deque.__test__.libreftest ***Test Failed*** 2 failures. Traceback (most recent call last): File "./Lib/test/test_deque.py", line 561, in ? test_main(verbose=True) File "./Lib/test/test_deque.py", line 558, in test_main test_support.run_doctest(test_deque, verbose) File "/home/ncoghlan/python/src/python/dist/src/Lib/test/test_support.py", line 318, in run_doctest raise TestFailed("%d of %d doctests failed" % (f, t)) test.test_support.TestFailed: 2 of 32 doctests failed [ncoghlan@localhost src]$ ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-10 09:32 Message: Logged In: YES user_id=80475 Temporarily disabled the test. Pending this report to remind me to put it back in later. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951270&group_id=5470 From noreply at sourceforge.net Mon May 10 10:48:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 10:48:39 2004 Subject: [ python-Bugs-951088 ] str(email) breaks message by adding newline Message-ID: Bugs item #951088, was opened at 2004-05-10 03:17 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951088&group_id=5470 Category: Python Library Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Gerrit Holl (gerrit) Assigned to: Barry A. Warsaw (bwarsaw) Summary: str(email) breaks message by adding newline Initial Comment: Since last nightly build, the following code breaks: import email message="""From gerrit@topjaklont.student.utwente.nl Mon May 10 09:06:55 2004 Return-Path: X-Original-To: gerrit Delivered-To: gerrit@topjaklont.student.utwente.nl Received: by topjaklont.student.utwente.nl (Postfix, from userid 500) id 866FE1F835; Mon, 10 May 2004 09:06:55 +0200 (CEST) To: gerrit@topjaklont.student.utwente.nl Message-Id: <20040510070655.866FE1F835@topjaklont.student.utwente.nl> Date: Mon, 10 May 2004 09:06:55 +0200 (CEST) From: gerrit@topjaklont.student.utwente.nl (Gerrit Holl) content """ msg = email.message_from_string(message) print str(msg) The output is: >From gerrit@topjaklont.student.utwente.nl Mon May 10 09:06:55 2004 Return-Path: X-Original-To: gerrit Delivered-To: gerrit@topjaklont.student.utwente.nl Received: by topjaklont.student.utwente.nl (Postfix, from userid 500) id 866FE1F835; Mon, 10 May 2004 09:06:55 +0200 (CEST) To: gerrit@topjaklont.student.utwente.nl Message-Id: <20040510070655.866FE1F835@topjaklont.student.utwente.nl> Date: Mon, 10 May 2004 09:06:55 +0200 (CEST) From: gerrit@topjaklont.student.utwente.nl (Gerrit Holl) content My mail client doesn't like this. The test script has been attached. ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-10 10:48 Message: Logged In: YES user_id=12800 Fixed. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951088&group_id=5470 From noreply at sourceforge.net Mon May 10 10:50:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 10:50:21 2004 Subject: [ python-Bugs-854102 ] Message.get_filename() problem Message-ID: Bugs item #854102, was opened at 2003-12-04 10:26 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854102&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Ervin N?meth (nemethe) Assigned to: Barry A. Warsaw (bwarsaw) Summary: Message.get_filename() problem Initial Comment: There are obviously e-mails with non-standard encodings (who wants to join me hating pine?) See attached backtrace. A local patch of - return unicode(newvalue[2], newvalue[0] or 'us-ascii') + try: + return unicode(newvalue[2], newvalue[0] or 'us-ascii') + except LookupError: + return unicode(newvalue[2], 'us-ascii', 'replace') solved the problem here. ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-10 10:50 Message: Logged In: YES user_id=12800 Please attach a sample test case to illustrate the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854102&group_id=5470 From noreply at sourceforge.net Mon May 10 11:04:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 11:04:10 2004 Subject: [ python-Bugs-852347 ] add support for cjkcodecs to Python email Message-ID: Bugs item #852347, was opened at 2003-12-01 17:05 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=852347&group_id=5470 Category: Python Library Group: Feature Request >Status: Closed Resolution: Accepted Priority: 5 Submitted By: Jason R. Mastaler (jasonrm) Assigned to: Barry A. Warsaw (bwarsaw) Summary: add support for cjkcodecs to Python email Initial Comment: As discussed last week on the email-sig list, the attached patch adds support for the CJKCodecs package as an alternative to the {Chinese,Japanese,Korean}Codecs packages. CJKCodecs 1.0.2 and above should work with this patch. This is advantageous because the Chinese and KoreanCodecs packages are no longer supported,maintained or available for download. This patch does not break compatibility with {Chinese,Japanese,Korean}Codecs, so they can still be used if desired. Lastly, this patch fixes a small typo that broke GB2312. ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-10 11:04 Message: Logged In: YES user_id=12800 Closing this without applying it to 2.3. The general consensus was not to mess with this in a bug fix release, so it will have to wait until Python 2.4/email3. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-12-30 11:52 Message: Logged In: YES user_id=12800 Applied to Python trunk (2.4). This will be applied to Python 2.3 and closed when that branch's freeze is lifted. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-12-30 10:16 Message: Logged In: YES user_id=12800 Latest version of the patch, with updated comments as per Jason's followup, and including Tokio Kikuchi's fix for the test suite regression. ---------------------------------------------------------------------- Comment By: Jason R. Mastaler (jasonrm) Date: 2003-12-30 00:59 Message: Logged In: YES user_id=85984 comments regarding Barry's Charset.py.diff: You shouldn't mention KoreanCodecs and ChineseCodecs in the comments as alternatives to CJKCodecs. Both are no longer maintained, or even available for download. Both have been completely replaced by CJKCodecs. Only JapaneseCodecs remains as a substitute package. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-12-29 23:33 Message: Logged In: YES user_id=12800 I'm not even sure this patch is correct, since it breaks the test suite. The problem is that self.output_codec ends up being different with the patch than without it (in Charset.__init__()). For example: Python 2.3.3 (#1, Dec 19 2003, 11:33:00) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from email.Charset import Charset >>> c = Charset('euc-jp') >>> c.output_codec 'japanese.iso-2022-jp' >>> But now with Charset.py.diff applied: ... >>> c.output_codec 'euc-jp' We need to figure out what the right thing to do here is. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-12-29 10:19 Message: Logged In: YES user_id=12800 One more rev of Charset.py.diff ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-12-29 09:52 Message: Logged In: YES user_id=12800 Oops, with the typo fix for gb2312. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-12-29 09:49 Message: Logged In: YES user_id=12800 Correct me if I'm wrong, but won't the attached patch work better? It simply removes the entries from CODEC_MAP that are already provided by cjkcodecs.aliases (and japanese.aliases and korean.aliases). See Charset.py.diff ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=852347&group_id=5470 From noreply at sourceforge.net Mon May 10 11:13:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 11:14:13 2004 Subject: [ python-Bugs-736407 ] Problem With email.MIMEText Package Message-ID: Bugs item #736407, was opened at 2003-05-12 05:41 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=736407&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Pending Resolution: None Priority: 5 Submitted By: John Abel (judasiscariot) Assigned to: Barry A. Warsaw (bwarsaw) Summary: Problem With email.MIMEText Package Initial Comment: Problem with email.MIMEText. I created a MIMEText message (following the example in the documentation). The problem occurs when as_string() is called. This is the traceback: File "/usr/local/lib/python2.3/email/Message.py", line 113, in as_string g.flatten(self, unixfrom=unixfrom) File "/usr/local/lib/python2.3/email/Generator.py", line 102, in flatten self._write(msg) File "/usr/local/lib/python2.3/email/Generator.py", line 137, in _write self._write_headers(msg) File "/usr/local/lib/python2.3/email/Generator.py", line 183, in _write_headers header_name=h, continuation_ws='\t').encode() File "/usr/local/lib/python2.3/email/Header.py", line 415, in encode return self._encode_chunks(newchunks, maxlinelen) File "/usr/local/lib/python2.3/email/Header.py", line 375, in _encode_chunks _max_append(chunks, s, maxlinelen, extra) File "/usr/local/lib/python2.3/email/quopriMIME.py", line 84, in _max_append L.append(s.lstrip()) AttributeError: 'list' object has no attribute 'lstrip' I'm using 2.3b1, on RH9. The same piece of code works with 2.2.2. ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-10 11:13 Message: Logged In: YES user_id=12800 I need to know what the value of oldFiles is when you run BareMail.py. I've had no problems with this example, but I suppose it's possible you might if you were running on a system with unicode file names and there were some funky characters in your file names. FWIW, this code works fine for me in Python 2.3.3 and Python 2.4cvs. I'm re-pending this bug report for more information. ---------------------------------------------------------------------- Comment By: John Abel (judasiscariot) Date: 2004-03-22 06:35 Message: Logged In: YES user_id=221478 OK, stripped everything down to a minimum. I now have a class, based upon the example from the email docs. It still gives the same error, when mailMessage.as_string() is called. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-11-21 15:29 Message: Logged In: YES user_id=12800 There have no follow ups to this message in months, so I'm moving it to Pending status. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-09-03 08:15 Message: Logged In: YES user_id=12800 Sorry, but you're going to have to work harder at trying to boil down the buggy code. When I run ServerMonitor.py, I get an immediate traceback Traceback (most recent call last): File "ServerMonitor.py", line 18, in ? import ThreadUtils, NetMon ImportError: No module named ThreadUtils Here's one thing you can try to do: catch the exception in your code, then pickle the Message object up and post the pickle. I can at least reconstitute the pickle and look at the structure. Also, I'm guessing the problem is somewhere in ProcessMail() in PostITControl.py. Be sure that the mailBody argument is a string and nothing else. ---------------------------------------------------------------------- Comment By: John Abel (judasiscariot) Date: 2003-05-21 04:16 Message: Logged In: YES user_id=221478 The Config.ini will need to modifying to suit your machine settings. Run the ServerMonitor.py, and then run PostITControl.py. PostITControl.py crashes when trying to send a mail. ---------------------------------------------------------------------- Comment By: John Abel (judasiscariot) Date: 2003-05-21 04:13 Message: Logged In: YES user_id=221478 Sorry, been trying to narrow the code down. If I have a bare minimum script, it works, yet the original still fails. I'll upload the required files (3). ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-05-19 15:59 Message: Logged In: YES user_id=12800 I'm moving this to Pending since wthout more information I can't help. This starts the 14 day clock ticking. After 2 weeks this bug report will be closed, unless someone follows up with more information. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-05-15 10:07 Message: Logged In: YES user_id=12800 Sorry, the attached code sample is incomplete. It is not sufficient for me to reproduce the problem. Please boil down your example to a self-contained -- but as small as possible -- example. ---------------------------------------------------------------------- Comment By: John Abel (judasiscariot) Date: 2003-05-15 05:32 Message: Logged In: YES user_id=221478 Sorry, forgot the comment. When running the script using python2.2 (email 2.4.3), I don't receive any errors, mail gets sent. It only seems to happen with 2.3b. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-05-13 11:45 Message: Logged In: YES user_id=12800 Can you upload the exact code that you wrote that caused this crash? Please don't paste it. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=736407&group_id=5470 From noreply at sourceforge.net Mon May 10 12:08:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 12:09:45 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-19 09:28 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None Priority: 7 Submitted By: Greg Jones (morngnstar) Assigned to: Guido van Rossum (gvanrossum) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-11 02:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 11:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-08 07:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-08 02:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 23:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 23:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 23:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 22:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 22:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 06:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 18:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-05 00:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-05 00:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-19 09:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Mon May 10 14:19:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 14:19:55 2004 Subject: [ python-Bugs-951235 ] strptime doesn't account for %p in output Message-ID: Bugs item #951235, was opened at 2004-05-10 06:18 Message generated for change (Settings changed) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951235&group_id=5470 Category: Python Library Group: None >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Skip Montanaro (montanaro) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't account for %p in output Initial Comment: Correct use of the "%p" format is not obvious: >>> time.strptime("3:15pm", '%H:%M%p') (1900, 1, 1, 3, 15, 0, 0, 1, -1) >>> time.strptime("3:15", '%H:%M') (1900, 1, 1, 3, 15, 0, 0, 1, -1) >>> time.strptime("3:15am", '%H:%M%p') (1900, 1, 1, 3, 15, 0, 0, 1, -1) Other than successfully parsing 'am' or 'pm', there's no indication in the returned time tuple of a 12-hour offset in the 'pm' case. I think the hour field returned from the first call should be 15, not 3. ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-05-10 11:19 Message: Logged In: YES user_id=357491 It's a misuse of %H. If you look at the 'time' module docs (http:// docs.python.org/lib/module-time.html so you don't have to click to it), it states %H is "Hour (24-hour clock) as a decimal number" while %I is "Hour (12-hour clock) as a decimal number". Use of %p is only triggered when %I is used since otherwise it is not supposed to be needed. This also deals with your corner case, Skip. If you feel this should be handled another way, let me know. But since putting in extra info that is not used is not against the rules, the code just doesn't bother using %p information unless %I is used. I guess I could raise an exception, but that seems harsh. Could do a warning, but not sure if that is the right way to go either. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-05-10 06:21 Message: Logged In: YES user_id=44345 Another corner case involving am/pm: >>> time.strptime("15:15pm", '%H:%M%p') (1900, 1, 1, 15, 15, 0, 0, 1, -1) >>> time.strptime("15:15am", '%H:%M%p') (1900, 1, 1, 15, 15, 0, 0, 1, -1) Shouldn't both raise ValueError since the input hour was not < 13? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951235&group_id=5470 From noreply at sourceforge.net Mon May 10 14:45:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 14:45:34 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.2.2 Status: Open Resolution: None Priority: 7 Submitted By: Greg Jones (morngnstar) Assigned to: Guido van Rossum (gvanrossum) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Mon May 10 14:54:34 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 14:55:46 2004 Subject: [ python-Bugs-951235 ] strptime doesn't account for %p in output Message-ID: Bugs item #951235, was opened at 2004-05-10 08:18 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951235&group_id=5470 Category: Python Library Group: None Status: Closed Resolution: Invalid Priority: 5 Submitted By: Skip Montanaro (montanaro) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't account for %p in output Initial Comment: Correct use of the "%p" format is not obvious: >>> time.strptime("3:15pm", '%H:%M%p') (1900, 1, 1, 3, 15, 0, 0, 1, -1) >>> time.strptime("3:15", '%H:%M') (1900, 1, 1, 3, 15, 0, 0, 1, -1) >>> time.strptime("3:15am", '%H:%M%p') (1900, 1, 1, 3, 15, 0, 0, 1, -1) Other than successfully parsing 'am' or 'pm', there's no indication in the returned time tuple of a 12-hour offset in the 'pm' case. I think the hour field returned from the first call should be 15, not 3. ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2004-05-10 13:54 Message: Logged In: YES user_id=44345 I checked in a change to Doc/lib/libtime.tex (v. 1.64). See if that seems correct to you. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-05-10 13:19 Message: Logged In: YES user_id=357491 It's a misuse of %H. If you look at the 'time' module docs (http:// docs.python.org/lib/module-time.html so you don't have to click to it), it states %H is "Hour (24-hour clock) as a decimal number" while %I is "Hour (12-hour clock) as a decimal number". Use of %p is only triggered when %I is used since otherwise it is not supposed to be needed. This also deals with your corner case, Skip. If you feel this should be handled another way, let me know. But since putting in extra info that is not used is not against the rules, the code just doesn't bother using %p information unless %I is used. I guess I could raise an exception, but that seems harsh. Could do a warning, but not sure if that is the right way to go either. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-05-10 08:21 Message: Logged In: YES user_id=44345 Another corner case involving am/pm: >>> time.strptime("15:15pm", '%H:%M%p') (1900, 1, 1, 15, 15, 0, 0, 1, -1) >>> time.strptime("15:15am", '%H:%M%p') (1900, 1, 1, 15, 15, 0, 0, 1, -1) Shouldn't both raise ValueError since the input hour was not < 13? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951235&group_id=5470 From noreply at sourceforge.net Mon May 10 15:47:22 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 15:48:34 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads >Group: Python 2.4 Status: Open Resolution: None >Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Guido van Rossum (gvanrossum) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 15:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Mon May 10 16:01:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 16:01:59 2004 Subject: [ python-Bugs-951482 ] New module: doc versus action Message-ID: Bugs item #951482, was opened at 2004-05-10 16:01 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951482&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Terry J. Reedy (tjreedy) Assigned to: Nobody/Anonymous (nobody) Summary: New module: doc versus action Initial Comment: Because the functions in the new module give direct access to API functions, syntactically legal code that uses them can be more fragile than normal -- leading to possible core dumps (which are normally considered to be interpreter bugs). Should Lib Ref 3.27 warn about this possibility? Can the interfacing be strengthened to cover at least some bad inputs? Can ceval be strengthed without slowing it down? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951482&group_id=5470 From noreply at sourceforge.net Mon May 10 16:03:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 16:03:17 2004 Subject: [ python-Bugs-951482 ] New module: doc versus action Message-ID: Bugs item #951482, was opened at 2004-05-10 16:01 Message generated for change (Settings changed) made by tjreedy You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951482&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Terry J. Reedy (tjreedy) >Assigned to: Raymond Hettinger (rhettinger) Summary: New module: doc versus action Initial Comment: Because the functions in the new module give direct access to API functions, syntactically legal code that uses them can be more fragile than normal -- leading to possible core dumps (which are normally considered to be interpreter bugs). Should Lib Ref 3.27 warn about this possibility? Can the interfacing be strengthened to cover at least some bad inputs? Can ceval be strengthed without slowing it down? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951482&group_id=5470 From noreply at sourceforge.net Mon May 10 16:06:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 16:06:33 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) >Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-10 16:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 15:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Mon May 10 18:20:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 18:20:10 2004 Subject: [ python-Bugs-875646 ] email.Utils.formataddr doesn't handle whitespace too well Message-ID: Bugs item #875646, was opened at 2004-01-12 15:30 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=875646&group_id=5470 Category: Python Library Group: Python 2.2.3 >Status: Closed >Resolution: Wont Fix Priority: 5 Submitted By: Matthias Klose (doko) Assigned to: Barry A. Warsaw (bwarsaw) Summary: email.Utils.formataddr doesn't handle whitespace too well Initial Comment: [forwarded from http://bugs.debian.org/178777] originally reported for mailman, then reassigned to python It seems like this problem is because email.Utils.formataddr doesn't handle newlines embedded in headers too well. : tfheen@yiwaz ..man/mailman-2.1.3/Mailman > python Python 2.3.3a0 (#2, Nov 21 2003, 00:36:53) [GCC 3.3.2 (Debian)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> from email.Utils import formataddr >>> formataddr(("Books is a list for discussing all aspects of the audio\nand text boox", 'books@spanky.braille.uwo.ca')) 'Books is a list for discussing all aspects of the audio\nand text boox ' >>> IMHO, it should either strip newlines or at least insert a space in front of them so they aren't interpreted as a new header. ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-10 18:20 Message: Logged In: YES user_id=12800 It's questionable whether formataddr should do this or not. Its docstring could be interpreted either way ('suitable for an RFC 2822 [...] header'). My inclination is to keep the magic out of formataddr and leave it up to the application to sanitize the real name. I'm closing this as won't fix, but if you still disagree, feel free to bring it up on the email-sig. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=875646&group_id=5470 From noreply at sourceforge.net Mon May 10 18:46:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 18:46:51 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 18:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 16:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 15:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Mon May 10 19:12:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 19:12:38 2004 Subject: [ python-Bugs-586899 ] email package does not work with mailbox Message-ID: Bugs item #586899, was opened at 2002-07-26 05:19 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=586899&group_id=5470 Category: Python Library Group: Python 2.2 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Paul Moore (pmoore) Assigned to: Barry A. Warsaw (bwarsaw) Summary: email package does not work with mailbox Initial Comment: The email.message_from_file function does not seem to work correctly when used as the "factory" argument to the mailbox.UnixMailbox function. The "From_" separator line gets included in the preceding mail message. For a demonstration of this, see the attached zip file. The Python code loads the first message in the mailbox file and then displays the final line. This shows the From_ line from the subsequent message (correctly quoted with a ">"!!!). This is with Python 2.2 on Windows 2000. The Python banner line is Python 2.2 (#28, Dec 21 2001, 12:21:22) [MSC 32 bit (Intel)] on win32 ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-10 19:12 Message: Logged In: YES user_id=12800 It doesn't look like anybody has the motivation to fix mailbox.py so I will (finally!) add this note to libmailbox.tex (for both Python 2.3 and 2.4): \note{For reasons of this module's internal implementation, you will probably want to open the \var{fp} object in binary mode. This is especially important on Windows.} ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-02-16 19:06 Message: Logged In: YES user_id=31435 Sorry for the delay. Paul is certainly correct that the only thing you can reliably do with a tell() result from a file opened in text mode is seek() to it later. Arithmetic on tell() results makes no sense for text-mode files. That's ANSI C rules, and they matter on Windows. Since we're going on two years without a patch here, sounds like changing the docs is better than nothing. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-03-10 12:05 Message: Logged In: YES user_id=12800 Since this makes no difference for *nix and I don't have a Windows box to play with, I'm assigning to Tim. He can thank me later. Paul's explanation seems reasonable, but without a patch, I doubt any of us have motivation to fix it. I'd be happy with a documentation change. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2002-07-26 11:59 Message: Logged In: YES user_id=12800 I don't have time to comment except to say that email.message_from_file() definitely works as the factory function for mailbox.UnixMailbox. As proof, I have the following code in Mailman: class Mailbox(mailbox.PortableUnixMailbox): def __init__(self, fp): mailbox.PortableUnixMailbox.__init__(self, fp, email.message_from_file) so I know it works. :) I'll look at the comments Paul's made when I get a chance. I'm assigning this bug report to me. ---------------------------------------------------------------------- Comment By: Paul Moore (pmoore) Date: 2002-07-26 10:06 Message: Logged In: YES user_id=113328 This is a text vs binary file issue with the mailbox module, not with email. The "fp" parameter to the [Portable]UnixMailbox class must be opened in binary mode, or incorrect results are obtained. This is caused by the implementation of _Subfile.read() in mailbox.py, which assumes that number of characters read is the same as the difference between tell() values, an assumption which fails for text mode files on Windows. I would consider this behaviour a bug, and would prefer it to be fixed. However, if it is not to be fixed, the documentation should be changed to note that the "fp" parameter for the mailbox constructor must be opened in binary mode. For MMDF and Babyl format mailboxes, it's arguably correct, as the file format is binary. For Unix mailboxes, the file is text format, so opening the file in text mode is not unreasonable. Question: is the universal newline support in Python 2.3 going to mess this up further? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=586899&group_id=5470 From noreply at sourceforge.net Mon May 10 21:15:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 21:18:38 2004 Subject: [ python-Bugs-951235 ] strptime doesn't account for %p in output Message-ID: Bugs item #951235, was opened at 2004-05-10 06:18 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951235&group_id=5470 Category: Python Library Group: None Status: Closed Resolution: Invalid Priority: 5 Submitted By: Skip Montanaro (montanaro) Assigned to: Brett Cannon (bcannon) Summary: strptime doesn't account for %p in output Initial Comment: Correct use of the "%p" format is not obvious: >>> time.strptime("3:15pm", '%H:%M%p') (1900, 1, 1, 3, 15, 0, 0, 1, -1) >>> time.strptime("3:15", '%H:%M') (1900, 1, 1, 3, 15, 0, 0, 1, -1) >>> time.strptime("3:15am", '%H:%M%p') (1900, 1, 1, 3, 15, 0, 0, 1, -1) Other than successfully parsing 'am' or 'pm', there's no indication in the returned time tuple of a 12-hour offset in the 'pm' case. I think the hour field returned from the first call should be 15, not 3. ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-05-10 18:15 Message: Logged In: YES user_id=357491 Yep, works for me. Thanks, Skip. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-05-10 11:54 Message: Logged In: YES user_id=44345 I checked in a change to Doc/lib/libtime.tex (v. 1.64). See if that seems correct to you. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-05-10 11:19 Message: Logged In: YES user_id=357491 It's a misuse of %H. If you look at the 'time' module docs (http:// docs.python.org/lib/module-time.html so you don't have to click to it), it states %H is "Hour (24-hour clock) as a decimal number" while %I is "Hour (12-hour clock) as a decimal number". Use of %p is only triggered when %I is used since otherwise it is not supposed to be needed. This also deals with your corner case, Skip. If you feel this should be handled another way, let me know. But since putting in extra info that is not used is not against the rules, the code just doesn't bother using %p information unless %I is used. I guess I could raise an exception, but that seems harsh. Could do a warning, but not sure if that is the right way to go either. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-05-10 06:21 Message: Logged In: YES user_id=44345 Another corner case involving am/pm: >>> time.strptime("15:15pm", '%H:%M%p') (1900, 1, 1, 15, 15, 0, 0, 1, -1) >>> time.strptime("15:15am", '%H:%M%p') (1900, 1, 1, 15, 15, 0, 0, 1, -1) Shouldn't both raise ValueError since the input hour was not < 13? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951235&group_id=5470 From noreply at sourceforge.net Mon May 10 21:35:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 21:35:44 2004 Subject: [ python-Bugs-832236 ] Build fails in ossaudiodev.c with missing macros Message-ID: Bugs item #832236, was opened at 2003-10-29 04:05 Message generated for change (Comment added) made by gward You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=832236&group_id=5470 Category: Build Group: Platform-specific >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Daniel Parks (danielparks) Assigned to: Greg Ward (gward) Summary: Build fails in ossaudiodev.c with missing macros Initial Comment: I'm building Python 2.3.2 with no configure flags on MkLinux Pre-R1. uname -a: Linux gondor.middle.earth 2.0.37-osfmach3 GENERIC_08alpha-20 Fri Jul 30 11:07:38 PDT 1999 ppc unknown The missing macros are: SOUND_MIXER_DIGITAL1, SOUND_MIXER_DIGITAL2, SOUND_MIXER_DIGITAL3, SOUND_MIXER_PHONEIN, SOUND_MIXER_PHONEOUT, SOUND_MIXER_VIDEO, SOUND_MIXER_RADIO, SOUND_MIXER_MONITOR, SNDCTL_DSP_GETODELAY I commented out two lines in setup.py and it seems to have built correctly. I would test it, but I need to go to bed, and I will forget to add this bug tomorrow. I will update this if Python actually works. ---------------------------------------------------------------------- >Comment By: Greg Ward (gward) Date: 2004-05-10 21:35 Message: Logged In: YES user_id=14422 OK, that's good enough -- I was just concerned about breaking the build, not about getting audio working under an obsolete version of Linux on ancient hardware. Fixed in Modules/ossaudiodev.c rev 1.34.8.1 (release23-maint branch) and rev 1.35 (trunk). ---------------------------------------------------------------------- Comment By: Daniel Parks (danielparks) Date: 2004-05-08 19:57 Message: Logged In: YES user_id=317142 Compiled Python 2.3.3 with the attached patch (sf-832236.patch). The build was successful. This is all I really wanted. [daniel@gondor Python-2.3.3]$ sudo ./python Lib/test/ test_ossaudiodev.py Password: playing test sound file... elapsed time: 5.4 sec Traceback (most recent call last): File "Lib/test/test_ossaudiodev.py", line 124, in ? test() File "Lib/test/test_ossaudiodev.py", line 122, in test test_setparameters() File "Lib/test/test_ossaudiodev.py", line 87, in test_setparameters raise RuntimeError("unable to set audio sampling parameters: " RuntimeError: unable to set audio sampling parameters: you must have really weird audio hardware I had to run the test as root because /dev/dsp was not accessible by other users. The audio clip played, but completely was unintelligible -- it sounded like a mistuned radio. This is the same machine as above; it is a PowerMac 7200/75, with whatever sound hardware is on the motherboard. I know that esound works properly on a PowerMac 6100/66 (which probably has similar audio hardware), but this is not nearly important enough for me to pursue. However, I am happy to test any patches. ---------------------------------------------------------------------- Comment By: Greg Ward (gward) Date: 2004-05-04 21:56 Message: Logged In: YES user_id=14422 Attached patch should fix the problem; I've emailed it to the original reporter to verify that it does. Will checkin/merge/close slip when I hear from him, or in a few days if I don't hear from him. ---------------------------------------------------------------------- Comment By: A.M. Kuchling (akuchling) Date: 2003-12-23 14:21 Message: Logged In: YES user_id=11375 The fix is probably straightforward; add #ifdef ...#endif around each use of the problematic macros. If you do this, please submit a patch. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=832236&group_id=5470 From noreply at sourceforge.net Mon May 10 22:06:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 10 22:06:18 2004 Subject: [ python-Bugs-824977 ] Memory error on AIX in email.Utils._qdecode Message-ID: Bugs item #824977, was opened at 2003-10-16 13:40 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=824977&group_id=5470 Category: Extension Modules Group: Python 2.2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Stuart D. Gathman (customdesigned) >Assigned to: Nobody/Anonymous (nobody) Summary: Memory error on AIX in email.Utils._qdecode Initial Comment: The following scriptlet works correctly on RedHat python2-2.2.3, but gets a MemoryError on AIX python-2.2.3. This is the only anomoly for AIX python I have seen. import email fp = open('analfail') msg = email.message_from_file(fp) for part in msg.walk(): if part.get_main_type() == 'text': txt = part.get_payload(decode=True) #del msg["content-transfer-encoding"] msg.set_payload(txt) fp.close() print msg.as_string() It doesn't matter whether the 'del' is there. ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-10 22:05 Message: Logged In: YES user_id=12800 Fixed, finally, by replacing the calloc/free calls with calls to PyMem_Malloc and PyMem_Free. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-11-21 16:29 Message: Logged In: YES user_id=12800 Whoops, you're right. I don't think PyMem_MALLOC guarantees that, so the patch isn't quite right (we'd have to zero it out ourselves). ---------------------------------------------------------------------- Comment By: Stuart D. Gathman (customdesigned) Date: 2003-11-21 16:08 Message: Logged In: YES user_id=142072 Are you sure that the following code doesn't depend on calloc clearing the memory to 0? Does PyMem_MALLOC clear to 0? ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2003-11-21 14:56 Message: Logged In: YES user_id=12800 Or at least PyMem_MALLOC, right? See attached patch candidate. If this fixes the problem for you (and if it passes Michael's muster), I'll commit this to the 2.3cvs. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2003-10-17 07:46 Message: Logged In: YES user_id=6656 Why does binascii call calloc directly? Shouldn't it be calling PyMem_New? ---------------------------------------------------------------------- Comment By: Stuart D. Gathman (customdesigned) Date: 2003-10-16 16:14 Message: Logged In: YES user_id=142072 Might as well fix b2a_qp while we are at it: --- ./Modules/binascii.c.aix Mon Mar 17 06:34:43 2003 +++ ./Modules/binascii.c Thu Oct 16 16:08:25 2003 @@ -1036,7 +1036,7 @@ /* We allocate the output same size as input, this is overkill */ odata = (unsigned char *) calloc(1, datalen); - if (odata == NULL) { + if (odata == NULL && datalen > 0) { PyErr_NoMemory(); return NULL; } @@ -1206,7 +1206,7 @@ odata = (unsigned char *) calloc(1, odatalen); - if (odata == NULL) { + if (odata == NULL && odatalen > 0) { PyErr_NoMemory(); return NULL; } ---------------------------------------------------------------------- Comment By: Stuart D. Gathman (customdesigned) Date: 2003-10-16 16:00 Message: Logged In: YES user_id=142072 Fixed binascii as follows: --- ./Modules/binascii.c.bms Mon Mar 17 06:34:43 2003 +++ ./Modules/binascii.c Thu Oct 16 15:55:34 2003 @@ -1036,7 +1036,7 @@ /* We allocate the output same size as input, this is overkill */ odata = (unsigned char *) calloc(1, datalen); - if (odata == NULL) { + if (odata == NULL && datalen > 0) { PyErr_NoMemory(); return NULL; } This bug will manifest not just on AIX, but any system which returns NULL from malloc or calloc when allocated size is 0. ---------------------------------------------------------------------- Comment By: Stuart D. Gathman (customdesigned) Date: 2003-10-16 13:47 Message: Logged In: YES user_id=142072 It looks like the problem is with a2b_qp in the binascii extension module. So I am changing the Category. ---------------------------------------------------------------------- Comment By: Stuart D. Gathman (customdesigned) Date: 2003-10-16 13:43 Message: Logged In: YES user_id=142072 I forgot to include the traceback. $ python t.py Traceback (most recent call last): File "t.py", line 8, in ? txt = part.get_payload(decode=True) File "/usr/local/lib/python2.2/email/Message.py", line 197, in get_payload return Utils._qdecode(payload) File "/usr/local/lib/python2.2/quopri.py", line 161, in decodestring return a2b_qp(s, header = header) MemoryError ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=824977&group_id=5470 From noreply at sourceforge.net Tue May 11 04:52:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 04:53:17 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-19 00:28 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-11 09:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 23:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 21:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 20:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 19:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 17:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 02:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 22:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 17:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 14:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 14:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 14:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 13:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 13:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 21:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 09:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 15:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 15:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-19 00:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Tue May 11 07:02:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 07:02:50 2004 Subject: [ python-Bugs-951851 ] Crash when reading "import table" of certain windows DLLs Message-ID: Bugs item #951851, was opened at 2004-05-11 21:02 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 Category: Windows Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Mark Hammond (mhammond) Assigned to: Thomas Heller (theller) Summary: Crash when reading "import table" of certain windows DLLs Initial Comment: As diagnosed by Thomas Heller, via the python-win32 list. On Windows 2000, if your sys.path includes the Windows system32 directory, 'import wmi' will crash Python. To reproduce, change to the system32 directory, start Python, and execute 'import wmi'. Note that Windows XP does not crash. The problem is in GetPythonImport(), in code that tries to walk the PE 'import table'. AFAIK, this is code that checks the correct Python version is used, but I've never seen this code before. I'm not sure why the code is actually crashing (ie, what assumptions made about the import table are wrong), but I have a patch that checks a the pointers are valid before they are de-referenced. After the patch is applied, the result is the correct: "ImportError: dynamic module does not define init function (initwmi)" exception. Assigning to thomas for his review, then I'm happy to check in. I propose this as a 2.3 bugfix. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 From noreply at sourceforge.net Tue May 11 07:05:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 07:05:41 2004 Subject: [ python-Bugs-951851 ] Crash when reading "import table" of certain windows DLLs Message-ID: Bugs item #951851, was opened at 2004-05-11 21:02 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 Category: Windows Group: Python 2.3 Status: Open Resolution: None Priority: 7 Submitted By: Mark Hammond (mhammond) Assigned to: Thomas Heller (theller) Summary: Crash when reading "import table" of certain windows DLLs Initial Comment: As diagnosed by Thomas Heller, via the python-win32 list. On Windows 2000, if your sys.path includes the Windows system32 directory, 'import wmi' will crash Python. To reproduce, change to the system32 directory, start Python, and execute 'import wmi'. Note that Windows XP does not crash. The problem is in GetPythonImport(), in code that tries to walk the PE 'import table'. AFAIK, this is code that checks the correct Python version is used, but I've never seen this code before. I'm not sure why the code is actually crashing (ie, what assumptions made about the import table are wrong), but I have a patch that checks a the pointers are valid before they are de-referenced. After the patch is applied, the result is the correct: "ImportError: dynamic module does not define init function (initwmi)" exception. Assigning to thomas for his review, then I'm happy to check in. I propose this as a 2.3 bugfix. ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-05-11 21:05 Message: Logged In: YES user_id=14198 Actually, I guess a comment regarding the pointer checks and referencing this bug would be nice :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 From noreply at sourceforge.net Tue May 11 09:42:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 09:42:24 2004 Subject: [ python-Feature Requests-831643 ] Set option for subprocess in setup.py Message-ID: Feature Requests item #831643, was opened at 2003-10-28 10:43 Message generated for change (Comment added) made by pimbuur You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=831643&group_id=5470 Category: Distutils Group: None Status: Open Resolution: None Priority: 5 Submitted By: Pim Buurman (pimbuur) Assigned to: Nobody/Anonymous (nobody) Summary: Set option for subprocess in setup.py Initial Comment: It is now only possible to prescribe options for subprocesses (e.g. build_ext) in the file setup.cfg But some of these options are obligatory to get the correct result for the action. These options can be set in the file setup.cfg, but it is not clear that these options should not be changed, nor is it clear in the setup.py file that the setup.cfg is relevant. I tend now to not to use the setup.cfg and call 'python setup.py' (without arguments). The setup.py will then fill in the necessary options in sys.argv . It would be much nicer if these options could be explicitly given as defaults for the subprocess. ---------------------------------------------------------------------- >Comment By: Pim Buurman (pimbuur) Date: 2004-05-11 15:42 Message: Logged In: YES user_id=157121 The described behaviour is solved by using the 'options' option. This option is not described, I found it by accident (somebody else used it). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=831643&group_id=5470 From noreply at sourceforge.net Tue May 11 09:59:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 10:00:32 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 09:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 04:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 18:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 16:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 15:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Tue May 11 10:19:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 10:19:42 2004 Subject: [ python-Bugs-948517 ] LaTeX not required Message-ID: Bugs item #948517, was opened at 2004-05-05 10:36 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=948517&group_id=5470 Category: Documentation Group: None >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Aahz (aahz) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: LaTeX not required Initial Comment: /doc.html and about.html ("Comments and Questions") should make clear that LaTeX is not required for submitting additions and changes to the docs. Suggested wording: Although we prefer additions and changes to be in the format prescribed by /doc.html, we welcome plain-text, too -- we're happy to handle the formatting for you. ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-11 10:19 Message: Logged In: YES user_id=3066 I don't see anything in the locations you cite (if I understood correctly; they weren't completely clear) that even suggests LaTeX, much less implies a requirement for LaTeX. Closing as Invalid; if I'm not seeing something, please re-open with detailed information on what current content you find misleading. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=948517&group_id=5470 From noreply at sourceforge.net Tue May 11 10:49:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 10:49:38 2004 Subject: [ python-Bugs-951851 ] Crash when reading "import table" of certain windows DLLs Message-ID: Bugs item #951851, was opened at 2004-05-11 13:02 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 Category: Windows Group: Python 2.3 Status: Open >Resolution: Accepted Priority: 7 Submitted By: Mark Hammond (mhammond) >Assigned to: Mark Hammond (mhammond) Summary: Crash when reading "import table" of certain windows DLLs Initial Comment: As diagnosed by Thomas Heller, via the python-win32 list. On Windows 2000, if your sys.path includes the Windows system32 directory, 'import wmi' will crash Python. To reproduce, change to the system32 directory, start Python, and execute 'import wmi'. Note that Windows XP does not crash. The problem is in GetPythonImport(), in code that tries to walk the PE 'import table'. AFAIK, this is code that checks the correct Python version is used, but I've never seen this code before. I'm not sure why the code is actually crashing (ie, what assumptions made about the import table are wrong), but I have a patch that checks a the pointers are valid before they are de-referenced. After the patch is applied, the result is the correct: "ImportError: dynamic module does not define init function (initwmi)" exception. Assigning to thomas for his review, then I'm happy to check in. I propose this as a 2.3 bugfix. ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-05-11 16:49 Message: Logged In: YES user_id=11105 IMO, IsBadReadPointer(import_data, 12 + sizeof(DWORD)) should be enough. Yes, please add a comment in the code. This is a little bit hackish, but it fixes the problem. And the real problem can always be fixed later, if needed. And, BTW, python 2.3.3 crashes on Windows XP as well. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-11 13:05 Message: Logged In: YES user_id=14198 Actually, I guess a comment regarding the pointer checks and referencing this bug would be nice :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 From noreply at sourceforge.net Tue May 11 11:12:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 11:12:44 2004 Subject: [ python-Bugs-876278 ] Unbounded recursion in modulefinder Message-ID: Bugs item #876278, was opened at 2004-01-13 19:46 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=876278&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Thomas Heller (theller) >Assigned to: Thomas Heller (theller) Summary: Unbounded recursion in modulefinder Initial Comment: modulefinder goes into unbounded recursion when a module named 'path.py', containing 'import os', is on sys.path. Last part of the traceback: File "c:\python23\lib\modulefinder.py", line 255, in import_module m = self.load_module(fqname, fp, pathname, stuff) File "c:\python23\lib\modulefinder.py", line 285, in load_module self.scan_code(co, m) File "c:\python23\lib\modulefinder.py", line 342, in scan_code self._safe_import_hook(name, m, fromlist) File "c:\python23\lib\modulefinder.py", line 300, in _safe_import_hook self.import_hook(name, caller) File "c:\python23\lib\modulefinder.py", line 124, in import_hook q, tail = self.find_head_package(parent, name) File "c:\python23\lib\modulefinder.py", line 166, in find_head_package q = self.import_module(head, qname, parent) File "c:\python23\lib\modulefinder.py", line 237, in import_module self.msgin(3, "import_module", partname, fqname, parent) RuntimeError: maximum recursion depth exceeded Running modulefinder with the '-x os.path' command line option prevents the recursion. This is not really a contrived situation, there's a possibly useful path.py module from Jason Orendorff at ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-05-11 17:12 Message: Logged In: YES user_id=11105 Thanks, Mark. Checked in as Lib/modulefinder.py CVS revision 1.7.6.2 and 1.11. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-04-28 09:11 Message: Logged In: YES user_id=14198 the patch looks reasonable to me, and I can demonstrate that it does fix the original problem as described. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-03-30 21:10 Message: Logged In: YES user_id=11105 I think I have found something, but I still have to check if this cures the original problem. Imagine an (possibly empty) module foo.py. Without this patch, modulefinder would even find 'import foo.foo.foo' modules. The assert in the second hunk is commented out, although I think it should be correct. Just, any comments? ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-03-25 22:09 Message: Logged In: YES user_id=11105 It has something to do with os.path. Adding 'os.path' to the excludes list, the problem does not appear. Adding print statements it seems that mf is looking for os.os.os.os.os.......something, it seems that it recurses without noting that something failed. If I had to fix this myself, I probably had to first step through these imports with the C debugger, and then through mf. ---------------------------------------------------------------------- Comment By: Just van Rossum (jvr) Date: 2004-03-25 21:53 Message: Logged In: YES user_id=92689 Hm, not really. Do you have any idea what's causing it? ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-03-25 21:06 Message: Logged In: YES user_id=11105 Just, do you have time to look into this? ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-01-28 21:36 Message: Logged In: YES user_id=11105 Just, do you have time to look into this? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=876278&group_id=5470 From noreply at sourceforge.net Tue May 11 11:16:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 11:16:10 2004 Subject: [ python-Bugs-935091 ] bdist_winist post-install script fails on non-admin Python Message-ID: Bugs item #935091, was opened at 2004-04-14 19:54 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=935091&group_id=5470 Category: Distutils Group: None Status: Open Resolution: None Priority: 5 Submitted By: David R. Handy (davidhandy) Assigned to: Thomas Heller (theller) Summary: bdist_winist post-install script fails on non-admin Python Initial Comment: Python version 2.3.3 Windows 98 If you install Python on Windows, and you selected a "Non Admin" install, then later on when you run any distutils-created installer that has a post-install script, the post-install script will not run. This in turn causes shortcuts to not be created, skips additional install steps, etc. The reason for this is pretty simple: a non-admin install of Python puts python23.dll in the C:\Python23 directory instead of in C:\WINDOWS\SYSTEM (or the equivalent on your version of windows). The extractor stub used by bdist_wininst does a simple LoadLibrary(), with no path information, to get python23.dll. This only looks in the current directory or in the PATH. What the extractor stub program ought to do in this case is a LoadLibrary() giving the full path (i.e. C:\Python23\python23.dll) since the Python installation directory is known. ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-05-11 17:16 Message: Logged In: YES user_id=11105 This is already fixed in the release23-maint branch, so it should be fixed in Python 2.3.4. I will port this to the trunk later. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=935091&group_id=5470 From noreply at sourceforge.net Tue May 11 11:20:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 11:20:45 2004 Subject: [ python-Bugs-951851 ] Crash when reading "import table" of certain windows DLLs Message-ID: Bugs item #951851, was opened at 2004-05-11 07:02 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 Category: Windows Group: Python 2.3 Status: Open Resolution: Accepted Priority: 7 Submitted By: Mark Hammond (mhammond) Assigned to: Mark Hammond (mhammond) Summary: Crash when reading "import table" of certain windows DLLs Initial Comment: As diagnosed by Thomas Heller, via the python-win32 list. On Windows 2000, if your sys.path includes the Windows system32 directory, 'import wmi' will crash Python. To reproduce, change to the system32 directory, start Python, and execute 'import wmi'. Note that Windows XP does not crash. The problem is in GetPythonImport(), in code that tries to walk the PE 'import table'. AFAIK, this is code that checks the correct Python version is used, but I've never seen this code before. I'm not sure why the code is actually crashing (ie, what assumptions made about the import table are wrong), but I have a patch that checks a the pointers are valid before they are de-referenced. After the patch is applied, the result is the correct: "ImportError: dynamic module does not define init function (initwmi)" exception. Assigning to thomas for his review, then I'm happy to check in. I propose this as a 2.3 bugfix. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-11 11:20 Message: Logged In: YES user_id=31435 Mark, while you may not have seen this code before, you checked it in . IIRC, though, the person who *created* the patch was never heard from again. I don't understand what the code thinks it's doing either, exactly. The obvious concern: if the import table format has changed, won't we also fail to import legit C extensions? I haven't seen that happen yet, but I haven't yet built any extensions using VC 7.1 either. In any case, I'd agree it's better to get a mysterious import failure than a crash. Maybe the detail in the ImportError could be changed to indicate whan an import failure is due to a bad pointer. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-11 10:49 Message: Logged In: YES user_id=11105 IMO, IsBadReadPointer(import_data, 12 + sizeof(DWORD)) should be enough. Yes, please add a comment in the code. This is a little bit hackish, but it fixes the problem. And the real problem can always be fixed later, if needed. And, BTW, python 2.3.3 crashes on Windows XP as well. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-11 07:05 Message: Logged In: YES user_id=14198 Actually, I guess a comment regarding the pointer checks and referencing this bug would be nice :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 From noreply at sourceforge.net Tue May 11 11:42:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 11:42:58 2004 Subject: [ python-Bugs-951851 ] Crash when reading "import table" of certain windows DLLs Message-ID: Bugs item #951851, was opened at 2004-05-11 13:02 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 Category: Windows Group: Python 2.3 Status: Open Resolution: Accepted Priority: 7 Submitted By: Mark Hammond (mhammond) Assigned to: Mark Hammond (mhammond) Summary: Crash when reading "import table" of certain windows DLLs Initial Comment: As diagnosed by Thomas Heller, via the python-win32 list. On Windows 2000, if your sys.path includes the Windows system32 directory, 'import wmi' will crash Python. To reproduce, change to the system32 directory, start Python, and execute 'import wmi'. Note that Windows XP does not crash. The problem is in GetPythonImport(), in code that tries to walk the PE 'import table'. AFAIK, this is code that checks the correct Python version is used, but I've never seen this code before. I'm not sure why the code is actually crashing (ie, what assumptions made about the import table are wrong), but I have a patch that checks a the pointers are valid before they are de-referenced. After the patch is applied, the result is the correct: "ImportError: dynamic module does not define init function (initwmi)" exception. Assigning to thomas for his review, then I'm happy to check in. I propose this as a 2.3 bugfix. ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-05-11 17:42 Message: Logged In: YES user_id=11105 Tim, I don't think the import table format has changed, instead wmi.dll doesn't have an import table (for whatever reason). Maybe the code isn't able to handle that correctly. Since Python 2.3 as well at it's extensions are still built with MSVC 6, I *think* we should be safe with this code. I'll attach the output of running MSVC.NET 2003's 'dumpbin.exe \windows\system32\wmi.dll' on my WinXP Pro SP1 for the curious. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-11 17:20 Message: Logged In: YES user_id=31435 Mark, while you may not have seen this code before, you checked it in . IIRC, though, the person who *created* the patch was never heard from again. I don't understand what the code thinks it's doing either, exactly. The obvious concern: if the import table format has changed, won't we also fail to import legit C extensions? I haven't seen that happen yet, but I haven't yet built any extensions using VC 7.1 either. In any case, I'd agree it's better to get a mysterious import failure than a crash. Maybe the detail in the ImportError could be changed to indicate whan an import failure is due to a bad pointer. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-11 16:49 Message: Logged In: YES user_id=11105 IMO, IsBadReadPointer(import_data, 12 + sizeof(DWORD)) should be enough. Yes, please add a comment in the code. This is a little bit hackish, but it fixes the problem. And the real problem can always be fixed later, if needed. And, BTW, python 2.3.3 crashes on Windows XP as well. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-11 13:05 Message: Logged In: YES user_id=14198 Actually, I guess a comment regarding the pointer checks and referencing this bug would be nice :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 From noreply at sourceforge.net Tue May 11 11:45:09 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 11:45:13 2004 Subject: [ python-Bugs-951851 ] Crash when reading "import table" of certain windows DLLs Message-ID: Bugs item #951851, was opened at 2004-05-11 13:02 Message generated for change (Comment added) made by theller You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 Category: Windows Group: Python 2.3 Status: Open Resolution: Accepted Priority: 7 Submitted By: Mark Hammond (mhammond) Assigned to: Mark Hammond (mhammond) Summary: Crash when reading "import table" of certain windows DLLs Initial Comment: As diagnosed by Thomas Heller, via the python-win32 list. On Windows 2000, if your sys.path includes the Windows system32 directory, 'import wmi' will crash Python. To reproduce, change to the system32 directory, start Python, and execute 'import wmi'. Note that Windows XP does not crash. The problem is in GetPythonImport(), in code that tries to walk the PE 'import table'. AFAIK, this is code that checks the correct Python version is used, but I've never seen this code before. I'm not sure why the code is actually crashing (ie, what assumptions made about the import table are wrong), but I have a patch that checks a the pointers are valid before they are de-referenced. After the patch is applied, the result is the correct: "ImportError: dynamic module does not define init function (initwmi)" exception. Assigning to thomas for his review, then I'm happy to check in. I propose this as a 2.3 bugfix. ---------------------------------------------------------------------- >Comment By: Thomas Heller (theller) Date: 2004-05-11 17:45 Message: Logged In: YES user_id=11105 Oh, we have to give the /all option to dumpbin ;-) ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-11 17:42 Message: Logged In: YES user_id=11105 Tim, I don't think the import table format has changed, instead wmi.dll doesn't have an import table (for whatever reason). Maybe the code isn't able to handle that correctly. Since Python 2.3 as well at it's extensions are still built with MSVC 6, I *think* we should be safe with this code. I'll attach the output of running MSVC.NET 2003's 'dumpbin.exe \windows\system32\wmi.dll' on my WinXP Pro SP1 for the curious. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-11 17:20 Message: Logged In: YES user_id=31435 Mark, while you may not have seen this code before, you checked it in . IIRC, though, the person who *created* the patch was never heard from again. I don't understand what the code thinks it's doing either, exactly. The obvious concern: if the import table format has changed, won't we also fail to import legit C extensions? I haven't seen that happen yet, but I haven't yet built any extensions using VC 7.1 either. In any case, I'd agree it's better to get a mysterious import failure than a crash. Maybe the detail in the ImportError could be changed to indicate whan an import failure is due to a bad pointer. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-11 16:49 Message: Logged In: YES user_id=11105 IMO, IsBadReadPointer(import_data, 12 + sizeof(DWORD)) should be enough. Yes, please add a comment in the code. This is a little bit hackish, but it fixes the problem. And the real problem can always be fixed later, if needed. And, BTW, python 2.3.3 crashes on Windows XP as well. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-11 13:05 Message: Logged In: YES user_id=14198 Actually, I guess a comment regarding the pointer checks and referencing this bug would be nice :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 From noreply at sourceforge.net Tue May 11 15:52:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 15:53:27 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 16:28 Message generated for change (Comment added) made by nobody You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 12:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 06:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 01:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 15:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 13:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 12:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 11:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 09:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 18:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 14:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 09:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 06:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 06:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 06:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 05:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 05:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 13:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 01:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 07:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 07:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 16:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Tue May 11 15:57:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 15:59:20 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 18:28 Message generated for change (Comment added) made by jasonlowe You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 14:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 14:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 08:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 03:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 17:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 15:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 14:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 13:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 11:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 20:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 16:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 11:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 08:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 08:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 07:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 07:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 15:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 03:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 09:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 09:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 18:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Tue May 11 18:26:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 18:26:57 2004 Subject: [ python-Bugs-952212 ] __name__ should be in 2.3.10 Special Attributes Message-ID: Bugs item #952212, was opened at 2004-05-11 22:26 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952212&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Robert Brewer (aminusfu) Assigned to: Nobody/Anonymous (nobody) Summary: __name__ should be in 2.3.10 Special Attributes Initial Comment: Although dir(object) doesn't yet list __name__, it is a common "special attribute" of many objects. The documentation should include a short description of __name__ in section 2.3.10 "Special Attributes". ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952212&group_id=5470 From noreply at sourceforge.net Tue May 11 18:27:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 18:27:55 2004 Subject: [ python-Bugs-952212 ] __name__ should be in 2.3.10 Special Attributes Message-ID: Bugs item #952212, was opened at 2004-05-11 22:26 Message generated for change (Settings changed) made by aminusfu You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952212&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Robert Brewer (aminusfu) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: __name__ should be in 2.3.10 Special Attributes Initial Comment: Although dir(object) doesn't yet list __name__, it is a common "special attribute" of many objects. The documentation should include a short description of __name__ in section 2.3.10 "Special Attributes". ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952212&group_id=5470 From noreply at sourceforge.net Tue May 11 18:46:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 18:48:15 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 18:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 15:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 15:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 09:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 04:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 18:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 16:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 15:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Tue May 11 18:54:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 18:55:20 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 18:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 18:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 15:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 15:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 09:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 04:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 18:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 16:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 15:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Tue May 11 19:00:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 19:00:35 2004 Subject: [ python-Bugs-951851 ] Crash when reading "import table" of certain windows DLLs Message-ID: Bugs item #951851, was opened at 2004-05-11 21:02 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 Category: Windows Group: Python 2.3 Status: Open Resolution: Accepted Priority: 7 Submitted By: Mark Hammond (mhammond) Assigned to: Mark Hammond (mhammond) Summary: Crash when reading "import table" of certain windows DLLs Initial Comment: As diagnosed by Thomas Heller, via the python-win32 list. On Windows 2000, if your sys.path includes the Windows system32 directory, 'import wmi' will crash Python. To reproduce, change to the system32 directory, start Python, and execute 'import wmi'. Note that Windows XP does not crash. The problem is in GetPythonImport(), in code that tries to walk the PE 'import table'. AFAIK, this is code that checks the correct Python version is used, but I've never seen this code before. I'm not sure why the code is actually crashing (ie, what assumptions made about the import table are wrong), but I have a patch that checks a the pointers are valid before they are de-referenced. After the patch is applied, the result is the correct: "ImportError: dynamic module does not define init function (initwmi)" exception. Assigning to thomas for his review, then I'm happy to check in. I propose this as a 2.3 bugfix. ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-05-12 09:00 Message: Logged In: YES user_id=14198 OK - will change to 12+so(WORD) And yes, I had seen this code - I meant "familiar with" :) Tim: Note that the import failure is not due to a bad import table (but the crash was). This code is trying to determine if a different version of Python is used. We are effectively skipping that check, and landing directly in the "does it have an init function?", then faling normally - ie, the code is now correctly *not* finding other Python versions linked against it. Thus, a different error message doesn't make much sense to me. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-12 01:45 Message: Logged In: YES user_id=11105 Oh, we have to give the /all option to dumpbin ;-) ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-12 01:42 Message: Logged In: YES user_id=11105 Tim, I don't think the import table format has changed, instead wmi.dll doesn't have an import table (for whatever reason). Maybe the code isn't able to handle that correctly. Since Python 2.3 as well at it's extensions are still built with MSVC 6, I *think* we should be safe with this code. I'll attach the output of running MSVC.NET 2003's 'dumpbin.exe \windows\system32\wmi.dll' on my WinXP Pro SP1 for the curious. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-12 01:20 Message: Logged In: YES user_id=31435 Mark, while you may not have seen this code before, you checked it in . IIRC, though, the person who *created* the patch was never heard from again. I don't understand what the code thinks it's doing either, exactly. The obvious concern: if the import table format has changed, won't we also fail to import legit C extensions? I haven't seen that happen yet, but I haven't yet built any extensions using VC 7.1 either. In any case, I'd agree it's better to get a mysterious import failure than a crash. Maybe the detail in the ImportError could be changed to indicate whan an import failure is due to a bad pointer. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-12 00:49 Message: Logged In: YES user_id=11105 IMO, IsBadReadPointer(import_data, 12 + sizeof(DWORD)) should be enough. Yes, please add a comment in the code. This is a little bit hackish, but it fixes the problem. And the real problem can always be fixed later, if needed. And, BTW, python 2.3.3 crashes on Windows XP as well. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-11 21:05 Message: Logged In: YES user_id=14198 Actually, I guess a comment regarding the pointer checks and referencing this bug would be nice :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 From noreply at sourceforge.net Tue May 11 19:04:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 19:04:31 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 19:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 18:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 18:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 15:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 15:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 09:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 04:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 18:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 16:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 15:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Tue May 11 19:41:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 19:42:25 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 19:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 19:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 18:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 18:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 15:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 15:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 09:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 04:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 18:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 16:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 15:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Tue May 11 21:56:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 21:56:59 2004 Subject: [ python-Bugs-951851 ] Crash when reading "import table" of certain windows DLLs Message-ID: Bugs item #951851, was opened at 2004-05-11 21:02 Message generated for change (Comment added) made by mhammond You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 Category: Windows Group: Python 2.3 Status: Open Resolution: Accepted Priority: 7 Submitted By: Mark Hammond (mhammond) Assigned to: Mark Hammond (mhammond) Summary: Crash when reading "import table" of certain windows DLLs Initial Comment: As diagnosed by Thomas Heller, via the python-win32 list. On Windows 2000, if your sys.path includes the Windows system32 directory, 'import wmi' will crash Python. To reproduce, change to the system32 directory, start Python, and execute 'import wmi'. Note that Windows XP does not crash. The problem is in GetPythonImport(), in code that tries to walk the PE 'import table'. AFAIK, this is code that checks the correct Python version is used, but I've never seen this code before. I'm not sure why the code is actually crashing (ie, what assumptions made about the import table are wrong), but I have a patch that checks a the pointers are valid before they are de-referenced. After the patch is applied, the result is the correct: "ImportError: dynamic module does not define init function (initwmi)" exception. Assigning to thomas for his review, then I'm happy to check in. I propose this as a 2.3 bugfix. ---------------------------------------------------------------------- >Comment By: Mark Hammond (mhammond) Date: 2004-05-12 11:56 Message: Logged In: YES user_id=14198 Seeing as it was the de-referencing of 'import_name' that crashed, I think a better patch is to terminate the outer while look as soon as we hit a bad string. Otherwise, this outer loop continues, 20 bytes at a time, until the outer pointer ('import_data') also goes bad or happens to reference \0. Attaching a slightly different patch, with comments and sizeof() change. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-12 09:00 Message: Logged In: YES user_id=14198 OK - will change to 12+so(WORD) And yes, I had seen this code - I meant "familiar with" :) Tim: Note that the import failure is not due to a bad import table (but the crash was). This code is trying to determine if a different version of Python is used. We are effectively skipping that check, and landing directly in the "does it have an init function?", then faling normally - ie, the code is now correctly *not* finding other Python versions linked against it. Thus, a different error message doesn't make much sense to me. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-12 01:45 Message: Logged In: YES user_id=11105 Oh, we have to give the /all option to dumpbin ;-) ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-12 01:42 Message: Logged In: YES user_id=11105 Tim, I don't think the import table format has changed, instead wmi.dll doesn't have an import table (for whatever reason). Maybe the code isn't able to handle that correctly. Since Python 2.3 as well at it's extensions are still built with MSVC 6, I *think* we should be safe with this code. I'll attach the output of running MSVC.NET 2003's 'dumpbin.exe \windows\system32\wmi.dll' on my WinXP Pro SP1 for the curious. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-12 01:20 Message: Logged In: YES user_id=31435 Mark, while you may not have seen this code before, you checked it in . IIRC, though, the person who *created* the patch was never heard from again. I don't understand what the code thinks it's doing either, exactly. The obvious concern: if the import table format has changed, won't we also fail to import legit C extensions? I haven't seen that happen yet, but I haven't yet built any extensions using VC 7.1 either. In any case, I'd agree it's better to get a mysterious import failure than a crash. Maybe the detail in the ImportError could be changed to indicate whan an import failure is due to a bad pointer. ---------------------------------------------------------------------- Comment By: Thomas Heller (theller) Date: 2004-05-12 00:49 Message: Logged In: YES user_id=11105 IMO, IsBadReadPointer(import_data, 12 + sizeof(DWORD)) should be enough. Yes, please add a comment in the code. This is a little bit hackish, but it fixes the problem. And the real problem can always be fixed later, if needed. And, BTW, python 2.3.3 crashes on Windows XP as well. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-11 21:05 Message: Logged In: YES user_id=14198 Actually, I guess a comment regarding the pointer checks and referencing this bug would be nice :) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951851&group_id=5470 From noreply at sourceforge.net Tue May 11 22:49:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 22:49:36 2004 Subject: [ python-Bugs-952212 ] __name__ should be in 2.3.10 Special Attributes Message-ID: Bugs item #952212, was opened at 2004-05-11 18:26 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952212&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Robert Brewer (aminusfu) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: __name__ should be in 2.3.10 Special Attributes Initial Comment: Although dir(object) doesn't yet list __name__, it is a common "special attribute" of many objects. The documentation should include a short description of __name__ in section 2.3.10 "Special Attributes". ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-11 22:49 Message: Logged In: YES user_id=3066 Fixed in Doc/lib/libstdtypes.tex revisions 1.153, 1.129.8.9, just in time for Python 2.3.4. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952212&group_id=5470 From noreply at sourceforge.net Tue May 11 22:55:22 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 22:55:30 2004 Subject: [ python-Bugs-945861 ] dir(object) does not list __name__ Message-ID: Bugs item #945861, was opened at 2004-05-01 07:39 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=945861&group_id=5470 >Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Thomas Philips (tkpmep) Assigned to: Nobody/Anonymous (nobody) Summary: dir(object) does not list __name__ Initial Comment: dir(object) returns ['__class__', '__delattr__', '__doc__', '__getattribute__', '__hash__', '__init__', '__new__', '__reduce__', '__reduce _ex__', '__repr__', '__setattr__', '__str__'] The __name__ method is not reported ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-11 22:55 Message: Logged In: YES user_id=3066 I'm not convinced that attributes dynamically provided by __getattr__() aren't actual attributes; it would be best if dir() reported them if they're available via getattr(ob, name). Whether or not this is practical is another matter. I've just closed documentation bug #952212, so at least the presence of the __name__ attribute on types and classes is mentioned somewhere. I'm re-classifying this bug report, since the dynamic behavior of dir() is not a documentation issue. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-05-02 15:47 Message: Logged In: YES user_id=44345 After a message from the submitter, it's apparent he was referring to class objects not showing '__name__' attributes in dir() output. This is a case of an attribute not being visible to dir() because it's not directly present in the object's __dict__ and is trapped at evaluation time by __getattr__(). Short of hacking dir() or adding a special attribute ("__attributes__"?) to objects which have __getattr__() methods I don't see a way around this problem. Wasn't there discussion of such an attribute which would expose such dynamic attributes to dir()? I don't see anything in the implementation of PyObject_Dir(). ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-05-01 21:02 Message: Logged In: YES user_id=44345 Are you sure that the object has an actual __name__ attribute (and not something computed by a __getattr__ method)? >>> import sys >>> dir(sys) ['__displayhook__', '__doc__', '__excepthook__', '__name__', '__stderr__', '__stdin__', '__stdout__', '_getframe', 'api_version', 'argv', 'builtin_module_names', 'byteorder', 'call_tracing', 'callstats', 'copyright', 'displayhook', 'exc_clear', 'exc_info', 'exc_type', 'excepthook', 'exec_prefix', 'executable', 'exit', 'exitfunc', 'getcheckinterval', 'getdefaultencoding', 'getdlopenflags', 'getfilesystemencoding', 'getrecursionlimit', 'getrefcount', 'hexversion', 'maxint', 'maxunicode', 'meta_path', 'modules', 'path', 'path_hooks', 'path_importer_cache', 'platform', 'prefix', 'ps1', 'ps2', 'setcheckinterval', 'setdlopenflags', 'setprofile', 'setrecursionlimit', 'settrace', 'stderr', 'stdin', 'stdout', 'version', 'version_info', 'warnoptions'] >>> sys.__name__ 'sys' ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=945861&group_id=5470 From noreply at sourceforge.net Tue May 11 23:54:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 11 23:55:18 2004 Subject: [ python-Bugs-602245 ] os.popen() negative error code IOError Message-ID: Bugs item #602245, was opened at 2002-08-30 02:25 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=602245&group_id=5470 Category: Windows Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Tim Delaney (tcdelaney) Assigned to: Mark Hammond (mhammond) Summary: os.popen() negative error code IOError Initial Comment: Windows 2000, Python 2.0.1, 2.1.1, 2.2. When a negative return code is received by the os.popen () family, an IOError is raised when the last pipe from the process is closed. The following code demonstrates the problem: import sys import os import traceback if __name__ == '__main__': if len(sys.argv) == 1: try: r = os.popen('%s %s %s' % (sys.executable, sys.argv[0], -1,)) r.close() except IOError: traceback.print_exc() try: w, r = os.popen2('%s %s %s' % (sys.executable, sys.argv[0], -1,)) w.close() r.close() except IOError: traceback.print_exc() try: w, r, e = os.popen3('%s %s %s' % (sys.executable, sys.argv[0], -1,)) w.close() r.close() e.close() except IOError: traceback.print_exc() else: sys.exit(int(sys.argv[1])) ---------- Run ---------- Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 11, in ? r.close() IOError: (0, 'Error') Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 18, in ? r.close() IOError: (0, 'Error') Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 26, in ? e.close() IOError: (0, 'Error') ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 23:54 Message: Logged In: YES user_id=6380 Yeah, that behavior is not just Linux, but any sane Unix (although the POSIX standard abstracts it away in macros). I think it's right to do that. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-09 19:45 Message: Logged In: YES user_id=14198 I tried truncating as you suggest, and this works fine - although it gives different results than Linux. Digging deeper, I found for linux: """ The value from the wait() system call is split into three parts mask 0xff00 is the exit status mask 0x007f is the signal causing the exit mask 0x0080 is the cord dumped flag in combination with a signal """ To get the same results as Linux for negative exit codes, it seems: result = (int)((exit_code & 0x000000FF) << 8); Will do the job. Counter-intuitively, this will make sys.exit(-1) yield an exit code of 65280, but this exactly what happens on Linux. Should we try and strive for compatibility within Python, or compatibility with the host OS? I lean towards compatibility with Linux - if they want win32 specific stuff, use win32 specific code. Thoughts? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-05 09:18 Message: Logged In: YES user_id=6380 How about changing the line "result = exit_code;" in _PyPclose() to truncate the value to the lower 16 bits, ensuring it is always >= 0? I think that should do it. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-05 08:57 Message: Logged In: YES user_id=14198 I'm not sure what to do about this. Python internally checks for -1 from the close function. The safest thing to do would be to translate the return code from -1 to something else. Changing the fclose semantics to check for PyErr_Occurred is another alternative, but doesn't seem that reasonable just for this. Any thoughts? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2002-08-30 10:05 Message: Logged In: YES user_id=6380 I suppose this is one for Mark Hammond... ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2002-08-30 08:38 Message: Logged In: YES user_id=44345 Attached Tim's code so it's more readily accessible to testers. BTW, works fine for me on Linux. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=602245&group_id=5470 From noreply at sourceforge.net Wed May 12 04:32:21 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 12 04:32:33 2004 Subject: [ python-Bugs-602245 ] os.popen() negative error code IOError Message-ID: Bugs item #602245, was opened at 2002-08-30 07:25 Message generated for change (Comment added) made by pmoore You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=602245&group_id=5470 Category: Windows Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Tim Delaney (tcdelaney) Assigned to: Mark Hammond (mhammond) Summary: os.popen() negative error code IOError Initial Comment: Windows 2000, Python 2.0.1, 2.1.1, 2.2. When a negative return code is received by the os.popen () family, an IOError is raised when the last pipe from the process is closed. The following code demonstrates the problem: import sys import os import traceback if __name__ == '__main__': if len(sys.argv) == 1: try: r = os.popen('%s %s %s' % (sys.executable, sys.argv[0], -1,)) r.close() except IOError: traceback.print_exc() try: w, r = os.popen2('%s %s %s' % (sys.executable, sys.argv[0], -1,)) w.close() r.close() except IOError: traceback.print_exc() try: w, r, e = os.popen3('%s %s %s' % (sys.executable, sys.argv[0], -1,)) w.close() r.close() e.close() except IOError: traceback.print_exc() else: sys.exit(int(sys.argv[1])) ---------- Run ---------- Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 11, in ? r.close() IOError: (0, 'Error') Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 18, in ? r.close() IOError: (0, 'Error') Traceback (most recent call last): File "Q:\Viper\src\webvis\tests\test.py", line 26, in ? e.close() IOError: (0, 'Error') ---------------------------------------------------------------------- Comment By: Paul Moore (pmoore) Date: 2004-05-12 09:32 Message: Logged In: YES user_id=113328 Do you propose to implement all of the WIFEXITED, WEXITSTATUS series of functions for Windows as well? Expecting people to unpack the return value manually - especially Windows users who won't be aware of the Unix heritage of the format - is probably unreasonable. If you implement the W... functions, I'm OK with this change. Otherwise I'm -1. After all, many uses of popen are probably to run non-portable programs anyway, so portability is only one aspect of the issue... ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 04:54 Message: Logged In: YES user_id=6380 Yeah, that behavior is not just Linux, but any sane Unix (although the POSIX standard abstracts it away in macros). I think it's right to do that. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-10 00:45 Message: Logged In: YES user_id=14198 I tried truncating as you suggest, and this works fine - although it gives different results than Linux. Digging deeper, I found for linux: """ The value from the wait() system call is split into three parts mask 0xff00 is the exit status mask 0x007f is the signal causing the exit mask 0x0080 is the cord dumped flag in combination with a signal """ To get the same results as Linux for negative exit codes, it seems: result = (int)((exit_code & 0x000000FF) << 8); Will do the job. Counter-intuitively, this will make sys.exit(-1) yield an exit code of 65280, but this exactly what happens on Linux. Should we try and strive for compatibility within Python, or compatibility with the host OS? I lean towards compatibility with Linux - if they want win32 specific stuff, use win32 specific code. Thoughts? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-05 14:18 Message: Logged In: YES user_id=6380 How about changing the line "result = exit_code;" in _PyPclose() to truncate the value to the lower 16 bits, ensuring it is always >= 0? I think that should do it. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2004-05-05 13:57 Message: Logged In: YES user_id=14198 I'm not sure what to do about this. Python internally checks for -1 from the close function. The safest thing to do would be to translate the return code from -1 to something else. Changing the fclose semantics to check for PyErr_Occurred is another alternative, but doesn't seem that reasonable just for this. Any thoughts? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2002-08-30 15:05 Message: Logged In: YES user_id=6380 I suppose this is one for Mark Hammond... ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2002-08-30 13:38 Message: Logged In: YES user_id=44345 Attached Tim's code so it's more readily accessible to testers. BTW, works fine for me on Linux. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=602245&group_id=5470 From noreply at sourceforge.net Wed May 12 09:54:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 12 09:56:17 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 18:28 Message generated for change (Comment added) made by jasonlowe You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 08:54 Message: Logged In: YES user_id=56897 SIGINT is 'special' because that's the signal behind the problems reported in bug 465673. Given the readline module's setjmp/longjmp mechanism to process SIGINT, we simply cannot allow one thread to do the setjmp() and another thread to do the longjmp() when it receives SIGINT. Without the setjmp/longjmp stuff, SIGINT is no more special than any other asynchronous signal like SIGTERM, SIGUSR1, etc. It'd be great if we could get the desired behavior for SIGINT out of the readline module without setjmp/longjmp, but without help from the readline library I don't see an easy way to do this. The readline library insists on continuing the readline() call after a SIGINT is handled, and there doesn't appear to be any way to get it to abort the current readline() call short of modifying the readline library. If we're stuck with the setjmp/longjmp mechanism, I think we can solve the issues regarding readline() being called from another thread and exec'd processes from threads by using the pthread_kill() technique mentioned earlier. The steps would look something like this: - Do not block any signals (including SIGINT) in any threads. - When we initialize the readline module's jmp_buf via setjmp(), save off the current thread ID. Probably want to check for existing ownership of jmp_buf and flag an error if detected. - When the readline module SIGINT handler is invoked, check if the current thread owns jmp_buf. If we are the owning thread then execute the longjmp (or siglongjmp). If we're not the owning thread, then have the current thread execute pthread_kill(jmp_buf_owner_thread, SIGINT) and little else. This will defer the SIGINT to the only thread that can really process it correctly. - Since SIGINT isn't blocked in any thread, processes exec'd from threads should get the default behavior for SIGINT rather than having it blocked. The above algorithm has a race condition on thread implementations where all threads receive SIGINT. The race can cause SIGINT to be processed more than once. The jmp_buf owning thread might finish the processing of SIGINT before another thread starts its processing and re-sends SIGINT to the jmp_buf owning thread. If there's a way to know via configure that we're on a thread implementation that broadcasts SIGINT, we could #ifdef the code to use something like the getpid() hack in signalmodule.c to do the right thing. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 18:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 18:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 17:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 17:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 14:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 14:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 08:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 03:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 17:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 15:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 14:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 13:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 11:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 20:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 16:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 11:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 08:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 08:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 07:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 07:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 15:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 03:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 09:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 09:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 18:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Wed May 12 10:11:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 12 10:13:23 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 10:11 Message: Logged In: YES user_id=6380 Sounds good. This solves the problem in the readline module, where it originates. BTW, if we can simplify things by only allowing readline() to be called from the main thread, that's fine with me. Doing console I/O from threads is insane anyway. We can start by assuming the signal broadcast problem is restricted to IRIX, and configure appropriately: define a test symbol for this and in configure, set this when IRIX is detected. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 09:54 Message: Logged In: YES user_id=56897 SIGINT is 'special' because that's the signal behind the problems reported in bug 465673. Given the readline module's setjmp/longjmp mechanism to process SIGINT, we simply cannot allow one thread to do the setjmp() and another thread to do the longjmp() when it receives SIGINT. Without the setjmp/longjmp stuff, SIGINT is no more special than any other asynchronous signal like SIGTERM, SIGUSR1, etc. It'd be great if we could get the desired behavior for SIGINT out of the readline module without setjmp/longjmp, but without help from the readline library I don't see an easy way to do this. The readline library insists on continuing the readline() call after a SIGINT is handled, and there doesn't appear to be any way to get it to abort the current readline() call short of modifying the readline library. If we're stuck with the setjmp/longjmp mechanism, I think we can solve the issues regarding readline() being called from another thread and exec'd processes from threads by using the pthread_kill() technique mentioned earlier. The steps would look something like this: - Do not block any signals (including SIGINT) in any threads. - When we initialize the readline module's jmp_buf via setjmp(), save off the current thread ID. Probably want to check for existing ownership of jmp_buf and flag an error if detected. - When the readline module SIGINT handler is invoked, check if the current thread owns jmp_buf. If we are the owning thread then execute the longjmp (or siglongjmp). If we're not the owning thread, then have the current thread execute pthread_kill(jmp_buf_owner_thread, SIGINT) and little else. This will defer the SIGINT to the only thread that can really process it correctly. - Since SIGINT isn't blocked in any thread, processes exec'd from threads should get the default behavior for SIGINT rather than having it blocked. The above algorithm has a race condition on thread implementations where all threads receive SIGINT. The race can cause SIGINT to be processed more than once. The jmp_buf owning thread might finish the processing of SIGINT before another thread starts its processing and re-sends SIGINT to the jmp_buf owning thread. If there's a way to know via configure that we're on a thread implementation that broadcasts SIGINT, we could #ifdef the code to use something like the getpid() hack in signalmodule.c to do the right thing. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 19:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 19:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 18:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 18:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 15:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 15:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 09:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 04:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 18:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 16:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 15:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Wed May 12 10:28:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 12 10:29:05 2004 Subject: [ python-Bugs-854102 ] Message.get_filename() problem Message-ID: Bugs item #854102, was opened at 2003-12-04 16:26 Message generated for change (Comment added) made by nemethe You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854102&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Ervin N?meth (nemethe) Assigned to: Barry A. Warsaw (bwarsaw) Summary: Message.get_filename() problem Initial Comment: There are obviously e-mails with non-standard encodings (who wants to join me hating pine?) See attached backtrace. A local patch of - return unicode(newvalue[2], newvalue[0] or 'us-ascii') + try: + return unicode(newvalue[2], newvalue[0] or 'us-ascii') + except LookupError: + return unicode(newvalue[2], 'us-ascii', 'replace') solved the problem here. ---------------------------------------------------------------------- >Comment By: Ervin N?meth (nemethe) Date: 2004-05-12 16:28 Message: Logged In: YES user_id=768522 Barry, sorry but it is 6 month since the report. The mail causing the problem is long lost. You could try contacting Gregor Maier who commented on mailman bug 803729 recently. He is at http://sourceforge.net/users/maiergre/ ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-10 16:50 Message: Logged In: YES user_id=12800 Please attach a sample test case to illustrate the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=854102&group_id=5470 From noreply at sourceforge.net Wed May 12 10:39:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 12 10:40:37 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-19 09:28 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-13 00:39 Message: Logged In: YES user_id=29957 This seems like a pragmatic and sensible approach to take, to me. It should probably be tested on the HP/UX boxes (google for 'HP/UX testdrive') I particularly like the idea of just putting a test in to block readline in the non-main thread. It seems the pythonic approach - since we can't guarantee behaviour that's anything but sane, it seems like a plan. Or at least make it issue a warning saying "don't do this" when readline is invoked from a non-main thread. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-13 00:11 Message: Logged In: YES user_id=6380 Sounds good. This solves the problem in the readline module, where it originates. BTW, if we can simplify things by only allowing readline() to be called from the main thread, that's fine with me. Doing console I/O from threads is insane anyway. We can start by assuming the signal broadcast problem is restricted to IRIX, and configure appropriately: define a test symbol for this and in configure, set this when IRIX is detected. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 23:54 Message: Logged In: YES user_id=56897 SIGINT is 'special' because that's the signal behind the problems reported in bug 465673. Given the readline module's setjmp/longjmp mechanism to process SIGINT, we simply cannot allow one thread to do the setjmp() and another thread to do the longjmp() when it receives SIGINT. Without the setjmp/longjmp stuff, SIGINT is no more special than any other asynchronous signal like SIGTERM, SIGUSR1, etc. It'd be great if we could get the desired behavior for SIGINT out of the readline module without setjmp/longjmp, but without help from the readline library I don't see an easy way to do this. The readline library insists on continuing the readline() call after a SIGINT is handled, and there doesn't appear to be any way to get it to abort the current readline() call short of modifying the readline library. If we're stuck with the setjmp/longjmp mechanism, I think we can solve the issues regarding readline() being called from another thread and exec'd processes from threads by using the pthread_kill() technique mentioned earlier. The steps would look something like this: - Do not block any signals (including SIGINT) in any threads. - When we initialize the readline module's jmp_buf via setjmp(), save off the current thread ID. Probably want to check for existing ownership of jmp_buf and flag an error if detected. - When the readline module SIGINT handler is invoked, check if the current thread owns jmp_buf. If we are the owning thread then execute the longjmp (or siglongjmp). If we're not the owning thread, then have the current thread execute pthread_kill(jmp_buf_owner_thread, SIGINT) and little else. This will defer the SIGINT to the only thread that can really process it correctly. - Since SIGINT isn't blocked in any thread, processes exec'd from threads should get the default behavior for SIGINT rather than having it blocked. The above algorithm has a race condition on thread implementations where all threads receive SIGINT. The race can cause SIGINT to be processed more than once. The jmp_buf owning thread might finish the processing of SIGINT before another thread starts its processing and re-sends SIGINT to the jmp_buf owning thread. If there's a way to know via configure that we're on a thread implementation that broadcasts SIGINT, we could #ifdef the code to use something like the getpid() hack in signalmodule.c to do the right thing. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-12 09:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 09:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 08:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-12 08:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 05:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-12 05:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 23:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 18:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 08:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-11 06:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 05:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 04:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-11 02:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 11:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-08 07:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-08 02:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 23:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 23:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 23:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 22:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 22:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 06:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 18:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-05 00:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-05 00:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-19 09:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Wed May 12 14:01:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 12 14:01:52 2004 Subject: [ python-Bugs-952737 ] Funny keywords in library index Message-ID: Bugs item #952737, was opened at 2004-05-12 20:01 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952737&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Thomas Heller (theller) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: Funny keywords in library index Initial Comment: The library index contains the keywords \UNIX and \POSIX. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952737&group_id=5470 From noreply at sourceforge.net Wed May 12 15:30:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 12 16:02:31 2004 Subject: [ python-Bugs-952807 ] segfault in subclassing datetime.date & pickling Message-ID: Bugs item #952807, was opened at 2004-05-12 21:30 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952807&group_id=5470 Category: Extension Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Thomas Wouters (twouters) Assigned to: Nobody/Anonymous (nobody) Summary: segfault in subclassing datetime.date & pickling Initial Comment: datetime.date does not take subclassing into account properly. datetime.date's tp_new has special code for unpickling (the single-string argument) which calls PyObject_New() directly, which doesn't account for the fact that subclasses may participate in cycle-gc (even if datetime.date objects do not.) The result is a segfault in code that unpickles instances of subclasses of datetime.date: import pickle, datetime class mydate(datetime.date): pass s = pickle.dumps(mydate.today()) broken = pickle.loads(s) del broken The 'del broken' is what causes the segfault: the 'mydate' class/type is supposed to participate in GC, but because of datetime.date's shortcut, that part of the object is never initialized (nor allocated, I presume.) The 'broken' instance reaches 0 refcounts, the GC gets triggered and it reads garbage memory. To 'prove' that the problem isn't caused by pickle itself: class mydate(datetime.date): pass broken = mydate('\x07\xd4\x05\x0c') del broken causes the same crash, in the GC code. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952807&group_id=5470 From noreply at sourceforge.net Wed May 12 16:49:26 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 12 16:49:30 2004 Subject: [ python-Bugs-952852 ] readline broke between 2.2.2 and 2.2.3 Message-ID: Bugs item #952852, was opened at 2004-05-12 15:49 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952852&group_id=5470 Category: Extension Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jon Nelson (jnelson) Assigned to: Nobody/Anonymous (nobody) Summary: readline broke between 2.2.2 and 2.2.3 Initial Comment: Patches readline.c to readline.c (2.48->2.49) and 2.41.6.2 -> 2.41.6.3 broke readline for many applications, with no recourse. Readline's behavior since 1.5.2 has been to append a space to the end of completions. Many completers depend on this functionality to provide enhanced completion. Those CVS commits broke all of them with no recourse. Instead of adding a function to set the append character, and then making rlcompleter use that, the readline module itself was changed and now nobody can do enhanced completion. Please *please* fix this. http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Modules/readline.c?r1=2.48&r2=2.49 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952852&group_id=5470 From noreply at sourceforge.net Wed May 12 17:17:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 12 17:17:25 2004 Subject: [ python-Bugs-952866 ] "can't multiply sequence *by* non-int" Message-ID: Bugs item #952866, was opened at 2004-05-12 21:17 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952866&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Tom Lynn (tlynn) Assigned to: Nobody/Anonymous (nobody) Summary: "can't multiply sequence *by* non-int" Initial Comment: >>> ""*"" Traceback (most recent call last): File "", line 1, in ? TypeError: can't multiply sequence to non-int Shouldn't that say "can't multiply sequence by non-int"? (or "with", or "and", but not "to") ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952866&group_id=5470 From noreply at sourceforge.net Wed May 12 17:17:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 12 17:17:46 2004 Subject: [ python-Bugs-952866 ] "can't multiply sequence *by* non-int" Message-ID: Bugs item #952866, was opened at 2004-05-12 21:17 Message generated for change (Settings changed) made by tlynn You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952866&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None >Priority: 3 Submitted By: Tom Lynn (tlynn) Assigned to: Nobody/Anonymous (nobody) Summary: "can't multiply sequence *by* non-int" Initial Comment: >>> ""*"" Traceback (most recent call last): File "", line 1, in ? TypeError: can't multiply sequence to non-int Shouldn't that say "can't multiply sequence by non-int"? (or "with", or "and", but not "to") ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952866&group_id=5470 From noreply at sourceforge.net Wed May 12 17:36:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 12 17:36:21 2004 Subject: [ python-Bugs-952866 ] "can't multiply sequence *by* non-int" Message-ID: Bugs item #952866, was opened at 2004-05-12 16:17 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952866&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 3 Submitted By: Tom Lynn (tlynn) Assigned to: Nobody/Anonymous (nobody) Summary: "can't multiply sequence *by* non-int" Initial Comment: >>> ""*"" Traceback (most recent call last): File "", line 1, in ? TypeError: can't multiply sequence to non-int Shouldn't that say "can't multiply sequence by non-int"? (or "with", or "and", but not "to") ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-12 16:36 Message: Logged In: YES user_id=80475 Fixed for Py2.4 Thanks for the bug report. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952866&group_id=5470 From noreply at sourceforge.net Wed May 12 20:54:44 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 12 20:54:48 2004 Subject: [ python-Bugs-952953 ] execve rejects empty argument list Message-ID: Bugs item #952953, was opened at 2004-05-13 00:54 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952953&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: newsham (newsham) Assigned to: Nobody/Anonymous (nobody) Summary: execve rejects empty argument list Initial Comment: The exec* family of calls in the os package does not allow an empty list of arguments to be passed in. Passing is an empty list (ie. allow argv[0] to be NULL and argc to be zero) is rarely desired, but perfectly legal. Verify by: import os os.execve("/bin/ls", [], {}) Tim N. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952953&group_id=5470 From noreply at sourceforge.net Thu May 13 05:59:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 05:59:16 2004 Subject: [ python-Bugs-952852 ] readline broke between 2.2.2 and 2.2.3 Message-ID: Bugs item #952852, was opened at 2004-05-12 21:49 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952852&group_id=5470 Category: Extension Modules Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jon Nelson (jnelson) Assigned to: Nobody/Anonymous (nobody) Summary: readline broke between 2.2.2 and 2.2.3 Initial Comment: Patches readline.c to readline.c (2.48->2.49) and 2.41.6.2 -> 2.41.6.3 broke readline for many applications, with no recourse. Readline's behavior since 1.5.2 has been to append a space to the end of completions. Many completers depend on this functionality to provide enhanced completion. Those CVS commits broke all of them with no recourse. Instead of adding a function to set the append character, and then making rlcompleter use that, the readline module itself was changed and now nobody can do enhanced completion. Please *please* fix this. http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Modules/readline.c?r1=2.48&r2=2.49 ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-13 10:59 Message: Logged In: YES user_id=6656 Why can't you add the space in your completion function? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952852&group_id=5470 From noreply at sourceforge.net Thu May 13 07:15:31 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 07:15:34 2004 Subject: [ python-Bugs-953177 ] cgi module documentation could mention getlist Message-ID: Bugs item #953177, was opened at 2004-05-13 21:15 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=953177&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Richard Jones (richard) Assigned to: Nobody/Anonymous (nobody) Summary: cgi module documentation could mention getlist Initial Comment: Section "11.2.2 Using the cgi module" at http://www.python.org/dev/doc/devel/lib/node411.html has a discussion about how the module handles multiple values with the same name. It even presents a section of code describing how to handle the situation. It could alternatively just make mention of its own getlist() method. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=953177&group_id=5470 From noreply at sourceforge.net Thu May 13 12:20:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 12:21:29 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 18:28 Message generated for change (Comment added) made by jasonlowe You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 11:20 Message: Logged In: YES user_id=56897 Argh! I thought we had a relatively clean solution, but it appears there's a stumbling block with the pthread_kill() approach. pthread_kill() is not guaranteed to be async-signal-safe, and nuggets found on the net indicate there's no portable way to redirect a process signal from one thread to another: http://groups.google.com/groups?q=pthread_kill+async+safe&start=30&hl=en&lr=&selm=3662B6A8.861984D5%40zko.dec.com&rnum=32 http://www.redhat.com/archives/phil-list/2003-December/msg00049.html Given that we can't safely call pthread_kill() from the SIGINT handler directly, there might be another way to solve our problems with pthread_atfork(). Here's my thinking: - Block SIGINT in all threads except the main (readline) thread. - Register via child process handler via pthread_atfork() that sets the SIGINT action for the child process back to the default. Unfortunately this fix isn't localized to the readline module as desired, but it may solve the problems. SIGINT routing will be forced to the readline thread, and child processes won't have SIGINT blocked, solving bug 756940. The IRIX thread signal delivery model (i.e.: broadcast) may cause problems since SIGINT may be pending when we attempt to set the action to default. Having SIGINT pending when the handler is changed to default would kill the child process. Maybe having the child process set the disposition to ignore and then to default would safely clear any pending SIGINT signal? I'll try to run some experiments with the pthread_atfork() approach soon, but work and home life for me is particularly busy lately. Apologies in advance if it takes me a while to respond or submit patches. If we're interested in a timely fix, would it be useful to break up the fix in two stages? I think we can all agree that the current approach of blocking ALL signals in created threads is a Bad Thing. What if we implement a quick, partial fix by simply change the existing code to only block SIGINT? This should be a two-line change to thread_pthread.h where "sigemptyset(&newmask); sigaddset(&newmask, SIGINT);" is used instead of "sigfillset(&newmask);". I see this partial fix having a number of benefits: - Easy change to make. No extra stuff to check for in configure or calls to things that may not exist or work properly. - Much less risky than trying to fix all the problems at once. The change only opens up signals to threads that Python-2.1 is already allowing through. - Should solve the SIGSEGV zombie problem and Guido's SIGTERM annoyance, although it would still have the problem reported in bug 756940. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-12 09:39 Message: Logged In: YES user_id=29957 This seems like a pragmatic and sensible approach to take, to me. It should probably be tested on the HP/UX boxes (google for 'HP/UX testdrive') I particularly like the idea of just putting a test in to block readline in the non-main thread. It seems the pythonic approach - since we can't guarantee behaviour that's anything but sane, it seems like a plan. Or at least make it issue a warning saying "don't do this" when readline is invoked from a non-main thread. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 09:11 Message: Logged In: YES user_id=6380 Sounds good. This solves the problem in the readline module, where it originates. BTW, if we can simplify things by only allowing readline() to be called from the main thread, that's fine with me. Doing console I/O from threads is insane anyway. We can start by assuming the signal broadcast problem is restricted to IRIX, and configure appropriately: define a test symbol for this and in configure, set this when IRIX is detected. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 08:54 Message: Logged In: YES user_id=56897 SIGINT is 'special' because that's the signal behind the problems reported in bug 465673. Given the readline module's setjmp/longjmp mechanism to process SIGINT, we simply cannot allow one thread to do the setjmp() and another thread to do the longjmp() when it receives SIGINT. Without the setjmp/longjmp stuff, SIGINT is no more special than any other asynchronous signal like SIGTERM, SIGUSR1, etc. It'd be great if we could get the desired behavior for SIGINT out of the readline module without setjmp/longjmp, but without help from the readline library I don't see an easy way to do this. The readline library insists on continuing the readline() call after a SIGINT is handled, and there doesn't appear to be any way to get it to abort the current readline() call short of modifying the readline library. If we're stuck with the setjmp/longjmp mechanism, I think we can solve the issues regarding readline() being called from another thread and exec'd processes from threads by using the pthread_kill() technique mentioned earlier. The steps would look something like this: - Do not block any signals (including SIGINT) in any threads. - When we initialize the readline module's jmp_buf via setjmp(), save off the current thread ID. Probably want to check for existing ownership of jmp_buf and flag an error if detected. - When the readline module SIGINT handler is invoked, check if the current thread owns jmp_buf. If we are the owning thread then execute the longjmp (or siglongjmp). If we're not the owning thread, then have the current thread execute pthread_kill(jmp_buf_owner_thread, SIGINT) and little else. This will defer the SIGINT to the only thread that can really process it correctly. - Since SIGINT isn't blocked in any thread, processes exec'd from threads should get the default behavior for SIGINT rather than having it blocked. The above algorithm has a race condition on thread implementations where all threads receive SIGINT. The race can cause SIGINT to be processed more than once. The jmp_buf owning thread might finish the processing of SIGINT before another thread starts its processing and re-sends SIGINT to the jmp_buf owning thread. If there's a way to know via configure that we're on a thread implementation that broadcasts SIGINT, we could #ifdef the code to use something like the getpid() hack in signalmodule.c to do the right thing. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 18:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 18:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 17:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 17:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 14:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 14:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 08:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 03:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 17:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 15:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 14:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 13:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 11:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 20:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 16:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 11:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 08:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 08:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 07:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 07:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 15:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 03:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 09:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 09:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 18:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Thu May 13 12:25:22 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 12:26:06 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-19 00:28 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-13 17:25 Message: Logged In: YES user_id=6656 Just to make life more entertaining, pthread_atfork isn't what you want, either. http://mail.python.org/pipermail/python-dev/2003-December/041309.html ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 17:20 Message: Logged In: YES user_id=56897 Argh! I thought we had a relatively clean solution, but it appears there's a stumbling block with the pthread_kill() approach. pthread_kill() is not guaranteed to be async-signal-safe, and nuggets found on the net indicate there's no portable way to redirect a process signal from one thread to another: http://groups.google.com/groups?q=pthread_kill+async+safe&start=30&hl=en&lr=&selm=3662B6A8.861984D5%40zko.dec.com&rnum=32 http://www.redhat.com/archives/phil-list/2003-December/msg00049.html Given that we can't safely call pthread_kill() from the SIGINT handler directly, there might be another way to solve our problems with pthread_atfork(). Here's my thinking: - Block SIGINT in all threads except the main (readline) thread. - Register via child process handler via pthread_atfork() that sets the SIGINT action for the child process back to the default. Unfortunately this fix isn't localized to the readline module as desired, but it may solve the problems. SIGINT routing will be forced to the readline thread, and child processes won't have SIGINT blocked, solving bug 756940. The IRIX thread signal delivery model (i.e.: broadcast) may cause problems since SIGINT may be pending when we attempt to set the action to default. Having SIGINT pending when the handler is changed to default would kill the child process. Maybe having the child process set the disposition to ignore and then to default would safely clear any pending SIGINT signal? I'll try to run some experiments with the pthread_atfork() approach soon, but work and home life for me is particularly busy lately. Apologies in advance if it takes me a while to respond or submit patches. If we're interested in a timely fix, would it be useful to break up the fix in two stages? I think we can all agree that the current approach of blocking ALL signals in created threads is a Bad Thing. What if we implement a quick, partial fix by simply change the existing code to only block SIGINT? This should be a two-line change to thread_pthread.h where "sigemptyset(&newmask); sigaddset(&newmask, SIGINT);" is used instead of "sigfillset(&newmask);". I see this partial fix having a number of benefits: - Easy change to make. No extra stuff to check for in configure or calls to things that may not exist or work properly. - Much less risky than trying to fix all the problems at once. The change only opens up signals to threads that Python-2.1 is already allowing through. - Should solve the SIGSEGV zombie problem and Guido's SIGTERM annoyance, although it would still have the problem reported in bug 756940. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-12 15:39 Message: Logged In: YES user_id=29957 This seems like a pragmatic and sensible approach to take, to me. It should probably be tested on the HP/UX boxes (google for 'HP/UX testdrive') I particularly like the idea of just putting a test in to block readline in the non-main thread. It seems the pythonic approach - since we can't guarantee behaviour that's anything but sane, it seems like a plan. Or at least make it issue a warning saying "don't do this" when readline is invoked from a non-main thread. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 15:11 Message: Logged In: YES user_id=6380 Sounds good. This solves the problem in the readline module, where it originates. BTW, if we can simplify things by only allowing readline() to be called from the main thread, that's fine with me. Doing console I/O from threads is insane anyway. We can start by assuming the signal broadcast problem is restricted to IRIX, and configure appropriately: define a test symbol for this and in configure, set this when IRIX is detected. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 14:54 Message: Logged In: YES user_id=56897 SIGINT is 'special' because that's the signal behind the problems reported in bug 465673. Given the readline module's setjmp/longjmp mechanism to process SIGINT, we simply cannot allow one thread to do the setjmp() and another thread to do the longjmp() when it receives SIGINT. Without the setjmp/longjmp stuff, SIGINT is no more special than any other asynchronous signal like SIGTERM, SIGUSR1, etc. It'd be great if we could get the desired behavior for SIGINT out of the readline module without setjmp/longjmp, but without help from the readline library I don't see an easy way to do this. The readline library insists on continuing the readline() call after a SIGINT is handled, and there doesn't appear to be any way to get it to abort the current readline() call short of modifying the readline library. If we're stuck with the setjmp/longjmp mechanism, I think we can solve the issues regarding readline() being called from another thread and exec'd processes from threads by using the pthread_kill() technique mentioned earlier. The steps would look something like this: - Do not block any signals (including SIGINT) in any threads. - When we initialize the readline module's jmp_buf via setjmp(), save off the current thread ID. Probably want to check for existing ownership of jmp_buf and flag an error if detected. - When the readline module SIGINT handler is invoked, check if the current thread owns jmp_buf. If we are the owning thread then execute the longjmp (or siglongjmp). If we're not the owning thread, then have the current thread execute pthread_kill(jmp_buf_owner_thread, SIGINT) and little else. This will defer the SIGINT to the only thread that can really process it correctly. - Since SIGINT isn't blocked in any thread, processes exec'd from threads should get the default behavior for SIGINT rather than having it blocked. The above algorithm has a race condition on thread implementations where all threads receive SIGINT. The race can cause SIGINT to be processed more than once. The jmp_buf owning thread might finish the processing of SIGINT before another thread starts its processing and re-sends SIGINT to the jmp_buf owning thread. If there's a way to know via configure that we're on a thread implementation that broadcasts SIGINT, we could #ifdef the code to use something like the getpid() hack in signalmodule.c to do the right thing. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-12 00:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 00:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 23:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 23:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 20:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 20:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 14:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 09:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 23:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 21:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 20:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 19:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 17:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 02:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 22:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 17:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 14:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 14:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 14:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 13:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 13:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 21:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 09:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 15:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 15:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-19 00:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Thu May 13 12:48:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 12:51:27 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-13 12:48 Message: Logged In: YES user_id=119306 pthread_kill(). That is annoying, I have something nearly done that used it. I didn't double check the safety of pthread_kill. I saw that posix says that kill is safe to call from interrupt handlers and went from there. Can we note that we need a pthread_kill in a call to Py_AddPendingCall, and then handle it later? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-13 12:25 Message: Logged In: YES user_id=6656 Just to make life more entertaining, pthread_atfork isn't what you want, either. http://mail.python.org/pipermail/python-dev/2003-December/041309.html ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 12:20 Message: Logged In: YES user_id=56897 Argh! I thought we had a relatively clean solution, but it appears there's a stumbling block with the pthread_kill() approach. pthread_kill() is not guaranteed to be async-signal-safe, and nuggets found on the net indicate there's no portable way to redirect a process signal from one thread to another: http://groups.google.com/groups?q=pthread_kill+async+safe&start=30&hl=en&lr=&selm=3662B6A8.861984D5%40zko.dec.com&rnum=32 http://www.redhat.com/archives/phil-list/2003-December/msg00049.html Given that we can't safely call pthread_kill() from the SIGINT handler directly, there might be another way to solve our problems with pthread_atfork(). Here's my thinking: - Block SIGINT in all threads except the main (readline) thread. - Register via child process handler via pthread_atfork() that sets the SIGINT action for the child process back to the default. Unfortunately this fix isn't localized to the readline module as desired, but it may solve the problems. SIGINT routing will be forced to the readline thread, and child processes won't have SIGINT blocked, solving bug 756940. The IRIX thread signal delivery model (i.e.: broadcast) may cause problems since SIGINT may be pending when we attempt to set the action to default. Having SIGINT pending when the handler is changed to default would kill the child process. Maybe having the child process set the disposition to ignore and then to default would safely clear any pending SIGINT signal? I'll try to run some experiments with the pthread_atfork() approach soon, but work and home life for me is particularly busy lately. Apologies in advance if it takes me a while to respond or submit patches. If we're interested in a timely fix, would it be useful to break up the fix in two stages? I think we can all agree that the current approach of blocking ALL signals in created threads is a Bad Thing. What if we implement a quick, partial fix by simply change the existing code to only block SIGINT? This should be a two-line change to thread_pthread.h where "sigemptyset(&newmask); sigaddset(&newmask, SIGINT);" is used instead of "sigfillset(&newmask);". I see this partial fix having a number of benefits: - Easy change to make. No extra stuff to check for in configure or calls to things that may not exist or work properly. - Much less risky than trying to fix all the problems at once. The change only opens up signals to threads that Python-2.1 is already allowing through. - Should solve the SIGSEGV zombie problem and Guido's SIGTERM annoyance, although it would still have the problem reported in bug 756940. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-12 10:39 Message: Logged In: YES user_id=29957 This seems like a pragmatic and sensible approach to take, to me. It should probably be tested on the HP/UX boxes (google for 'HP/UX testdrive') I particularly like the idea of just putting a test in to block readline in the non-main thread. It seems the pythonic approach - since we can't guarantee behaviour that's anything but sane, it seems like a plan. Or at least make it issue a warning saying "don't do this" when readline is invoked from a non-main thread. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 10:11 Message: Logged In: YES user_id=6380 Sounds good. This solves the problem in the readline module, where it originates. BTW, if we can simplify things by only allowing readline() to be called from the main thread, that's fine with me. Doing console I/O from threads is insane anyway. We can start by assuming the signal broadcast problem is restricted to IRIX, and configure appropriately: define a test symbol for this and in configure, set this when IRIX is detected. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 09:54 Message: Logged In: YES user_id=56897 SIGINT is 'special' because that's the signal behind the problems reported in bug 465673. Given the readline module's setjmp/longjmp mechanism to process SIGINT, we simply cannot allow one thread to do the setjmp() and another thread to do the longjmp() when it receives SIGINT. Without the setjmp/longjmp stuff, SIGINT is no more special than any other asynchronous signal like SIGTERM, SIGUSR1, etc. It'd be great if we could get the desired behavior for SIGINT out of the readline module without setjmp/longjmp, but without help from the readline library I don't see an easy way to do this. The readline library insists on continuing the readline() call after a SIGINT is handled, and there doesn't appear to be any way to get it to abort the current readline() call short of modifying the readline library. If we're stuck with the setjmp/longjmp mechanism, I think we can solve the issues regarding readline() being called from another thread and exec'd processes from threads by using the pthread_kill() technique mentioned earlier. The steps would look something like this: - Do not block any signals (including SIGINT) in any threads. - When we initialize the readline module's jmp_buf via setjmp(), save off the current thread ID. Probably want to check for existing ownership of jmp_buf and flag an error if detected. - When the readline module SIGINT handler is invoked, check if the current thread owns jmp_buf. If we are the owning thread then execute the longjmp (or siglongjmp). If we're not the owning thread, then have the current thread execute pthread_kill(jmp_buf_owner_thread, SIGINT) and little else. This will defer the SIGINT to the only thread that can really process it correctly. - Since SIGINT isn't blocked in any thread, processes exec'd from threads should get the default behavior for SIGINT rather than having it blocked. The above algorithm has a race condition on thread implementations where all threads receive SIGINT. The race can cause SIGINT to be processed more than once. The jmp_buf owning thread might finish the processing of SIGINT before another thread starts its processing and re-sends SIGINT to the jmp_buf owning thread. If there's a way to know via configure that we're on a thread implementation that broadcasts SIGINT, we could #ifdef the code to use something like the getpid() hack in signalmodule.c to do the right thing. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 19:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 19:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 18:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 18:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 15:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 15:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 09:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 04:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 18:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 16:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 15:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Thu May 13 12:59:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 12:59:49 2004 Subject: [ python-Bugs-952852 ] readline broke between 2.2.2 and 2.2.3 Message-ID: Bugs item #952852, was opened at 2004-05-12 15:49 Message generated for change (Comment added) made by jnelson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952852&group_id=5470 Category: Extension Modules Group: None >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Jon Nelson (jnelson) Assigned to: Nobody/Anonymous (nobody) Summary: readline broke between 2.2.2 and 2.2.3 Initial Comment: Patches readline.c to readline.c (2.48->2.49) and 2.41.6.2 -> 2.41.6.3 broke readline for many applications, with no recourse. Readline's behavior since 1.5.2 has been to append a space to the end of completions. Many completers depend on this functionality to provide enhanced completion. Those CVS commits broke all of them with no recourse. Instead of adding a function to set the append character, and then making rlcompleter use that, the readline module itself was changed and now nobody can do enhanced completion. Please *please* fix this. http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Modules/readline.c?r1=2.48&r2=2.49 ---------------------------------------------------------------------- >Comment By: Jon Nelson (jnelson) Date: 2004-05-13 11:59 Message: Logged In: YES user_id=8446 I ended up having trouble with the cmd.py file. At /some/ point it added the completekey parameter, which blasted my completer function. Everything works now, this bug can be closed, sorry for the trouble. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-13 04:59 Message: Logged In: YES user_id=6656 Why can't you add the space in your completion function? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952852&group_id=5470 From noreply at sourceforge.net Thu May 13 13:00:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 13:00:18 2004 Subject: [ python-Bugs-952852 ] readline broke between 2.2.2 and 2.2.3 Message-ID: Bugs item #952852, was opened at 2004-05-12 15:49 Message generated for change (Comment added) made by jnelson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952852&group_id=5470 Category: Extension Modules Group: None Status: Closed Resolution: Invalid Priority: 5 Submitted By: Jon Nelson (jnelson) Assigned to: Nobody/Anonymous (nobody) Summary: readline broke between 2.2.2 and 2.2.3 Initial Comment: Patches readline.c to readline.c (2.48->2.49) and 2.41.6.2 -> 2.41.6.3 broke readline for many applications, with no recourse. Readline's behavior since 1.5.2 has been to append a space to the end of completions. Many completers depend on this functionality to provide enhanced completion. Those CVS commits broke all of them with no recourse. Instead of adding a function to set the append character, and then making rlcompleter use that, the readline module itself was changed and now nobody can do enhanced completion. Please *please* fix this. http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Modules/readline.c?r1=2.48&r2=2.49 ---------------------------------------------------------------------- >Comment By: Jon Nelson (jnelson) Date: 2004-05-13 12:00 Message: Logged In: YES user_id=8446 I ended up having trouble with the cmd.py file. At /some/ point it added the completekey parameter, which blasted my completer function. Everything works now, this bug can be closed, sorry for the trouble. ---------------------------------------------------------------------- Comment By: Jon Nelson (jnelson) Date: 2004-05-13 11:59 Message: Logged In: YES user_id=8446 I ended up having trouble with the cmd.py file. At /some/ point it added the completekey parameter, which blasted my completer function. Everything works now, this bug can be closed, sorry for the trouble. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-13 04:59 Message: Logged In: YES user_id=6656 Why can't you add the space in your completion function? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=952852&group_id=5470 From noreply at sourceforge.net Thu May 13 13:15:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 13:16:09 2004 Subject: [ python-Bugs-874842 ] httplib fails on Akamai URLs Message-ID: Bugs item #874842, was opened at 2004-01-11 03:16 Message generated for change (Comment added) made by zwoop You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=874842&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: Accepted Priority: 5 Submitted By: Leif Hedstrom (zwoop) Assigned to: Jeremy Hylton (jhylton) Summary: httplib fails on Akamai URLs Initial Comment: Using Python 2.3.2 and httplib, reading from Akamai URLs will always hang at the end of the transacation. As common as this must be, I couldn't find anything related to it on any search engines, nor on the bug list here. The problem is that Akamai returns an HTTP/1.0 response, with a header like: Connection: keep-alive httplib does not recognize this response properly (the Connection: header parsing is only done for HTTP/1.1 responses). I'm not sure exactly what the right solution is, but I'm supplying one alternative solution that does solve the problem. I'm attaching a diff against httplib.py. ---------------------------------------------------------------------- >Comment By: Leif Hedstrom (zwoop) Date: 2004-05-13 10:15 Message: Logged In: YES user_id=480913 Hate to beat on a dead horse here, but what was the final outcome of this discussion? Anything I can do to help produce a better patch, documentation or anything? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-04-20 09:53 Message: Logged In: YES user_id=6380 It's great that the ASF can wield such power over the likes of AOL. But I don't want to presume the same for Python (we're not the #1 web language, not even #2). I'd be more concerned if adding this hack would *break* anything, but that doesn't seem to be the case. So I still think Jeremy can check it in. ---------------------------------------------------------------------- Comment By: Leif Hedstrom (zwoop) Date: 2004-04-20 08:36 Message: Logged In: YES user_id=480913 Yeah, the second solution is what I ended up doing, although it's definitely not obvious for anyone using httplib.py that this is required to support Akamai (see my original blog post at http://www.ogre.com/tiki-view_blog_post.php?blogId=3&postId=30 for both alternative solutions). At a minimum, I think we should provide the AkamaiHTTPResponse class in one way or another, and clearly document that this is required for correct support of Akamai URLs. My vote would probably be to "hack" the original HTTPResponse class, since anyone using HTTPlib for anything that might hit Akamai (perhaps as a referral/redirect) will have to use the fixed version anyways. Unfortunately I don't have any contacts left at Akamai, so I'm not sure how to inform them of their problems. I completely agree that we need to inform them about this problem, my point was that since Akamai works with pretty much everything else (browsers, other modules etc.), I think it'll be quite slow to get them to change. And until then, we're stuck with a module that is effectively semi-broken. Thanks, -- Leif ---------------------------------------------------------------------- Comment By: Greg Stein (gstein) Date: 2004-04-19 23:44 Message: Logged In: YES user_id=6501 Falling into line with "oh, but they won't change it" is why we end up with a whole bunch of bad implementations. If everybody said that, then we wouldn't get anywhere. A long while back, AOL came out with a busted proxy implementation which didn't work with Apache servers. The ASF said, "sorry AOL: you're wrong. fix your proxies." And they did. If we put a hack in for every busted thing that came out over the next ten years, then imagine the craphole we'd be in... :-) That said: yes, you can workaround the issue with a subclass of HTTPResponse which overrides the _check_close() method. You can then create an HTTPConnection subclass which overrides the class variable 'response_class', or you can set that field in an HTTPConnection instance as an instance variable. For example: conn = HTTPConnection(...) conn.response_class = AkamaiBugHandler When the response arrives, the HTTPConnection class uses self.response_class, so there are a few options to get your custom response class into the chain of events. ---------------------------------------------------------------------- Comment By: Leif Hedstrom (zwoop) Date: 2004-04-19 17:57 Message: Logged In: YES user_id=480913 As I said, no matter what we do, it's a hack on something that's broken on the web (now there's a shocker :-). I don't feel terribly strongly on this issue, I merely filed the bug report because I had this problem, and it took me several hours to figure out why my daemon would stall on Akamai URLs. I'm guessing other users of httplib.py might run into the same problem. As for the patch, the comments would of course have to change, I didn't want to impose more changes in the diff than necessary. Besides the suggested patch, an alternative solution is to provide a specialized implementation of the HTTPResponse class, which works with Akamai. The users of the httplib.py module would then have to explicitly request that httplib.HTTPConnection should instantiate that class instead of the default one. Preferably this would be passed as a new argument to the constructor for HTTPConnection. And I agree that it's a hack to have to code around poor server implementations. But not sure what our odds are to get Akamai to fix their servers any time soon, since pretty much any web browser in existance works with their broken implementation. Cheers, -- leif ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-04-19 15:32 Message: Logged In: YES user_id=6380 I won't reject the patch on that basis. Like HTML, it's more useful to be able to handle what we see in the real world than to stick to the standard. Clearly the OP needs to be able to access Akamai servers. He doesn't have the power to fix the Akamai servers,so saying "the server is wrong" doesn't do him any good. (The comment should stateclearly that Akamai *is* wrong though!) Or do you have a different suggestion for how the poster can work around the problem? ---------------------------------------------------------------------- Comment By: Greg Stein (gstein) Date: 2004-04-19 15:26 Message: Logged In: YES user_id=6501 I have a philosophical problem with compensating for servers that obviously break protocols. The server should be fixed, not *every* client on the planet. From that standpoint, this problem/fix should be rejected, though I defer to Guido on that choice. That said, the comment right above the patch should be fixed. The whole point of that comment is, "the header shouldn't be there, so we shouldn't bother to examine the thing." Obviously, the new code does, so the two comments should be merged. The comment about Akamai should also be strengthened to note that it is violating the HTTP protocol (see section 8.1.2.1 of RFC 2616). Summary: I'd reject it, but will leave that to Guido to choose (i.e. "we'll help users even tho it violates protocols"). If he wants it, then +1 if the comments are fixed up. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2004-04-15 14:59 Message: Logged In: YES user_id=31392 Looks good to me. I want to see if I can come up with a simple test module for httplib with the network resource enabled. I'll see if I can do that tonight. ---------------------------------------------------------------------- Comment By: Leif Hedstrom (zwoop) Date: 2004-04-12 13:54 Message: Logged In: YES user_id=480913 Heh, yeah, I'm pretty sure that's the problem, Akamai being confused about protocols. They claim to be a v1.0 HTTP proxy, yet they use v1.1 HTTP headers :-/. This is why I mentioned I wasn't sure exactly what the right solution is. And no matter what we do, it'll be a hack. Maybe the original author of the module has some insight ? Unfortunately, there's a lot of Akamai content out there that are affected by this. Cheers, -- Leif ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-04-12 13:32 Message: Logged In: YES user_id=6380 Hmm... Indeed. read() checks will_close and apparently setting that to False will do the right thing. I don't know HTTP and this code well enough to approve this fix though. Also, the comment right above your patch should probably be fixed; it claims that connection headers on HTTP/1.0 are due to confused proxies. (Maybe that's what Akamai servers are? :-) ---------------------------------------------------------------------- Comment By: Leif Hedstrom (zwoop) Date: 2004-04-12 13:13 Message: Logged In: YES user_id=480913 Yeah, that works for me to. But the problem is in the HTTPResponse class from the httplib.py module. For example, this code (butchered from my application) will hang on Akamai URLs: #!/usr/bin/python import httplib def testHTTPlib(host, url): http = httplib.HTTPConnection(host) try: http.request('GET', url) response = http.getresponse() except IOError: self._log.warning("Can't connect to %s", url) return False except socket.error: self._log.error("Socket error retrieving %s", url) return False except socket.timeout: self._log.warning("Timeout connecting to %s", url) return False else: try: data = response.read() return True except socket.timeout: self._log.warning("Timeout reading from %s", url) return False return False print testHTTPlib("www.ogre.com", "/") print testHTTPlib("www.akamai.com", "/") Granted, I think Akamai aren't strictly following the protocols, but it's inconvenient that this piece of code stalls here (and only for akamai.com domains, I've tried a lot of them). Thanks! -- Leif ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-04-12 12:36 Message: Logged In: YES user_id=6380 Can you give a complete program that reproduces this? I've tried this: >>> import urllib >>> urllib.urlopen("http://www.akamai.com").read() and it doesn't hang for me. I tried a number of Python versions from 2.2 through 2.4a0. ---------------------------------------------------------------------- Comment By: Leif Hedstrom (zwoop) Date: 2004-01-11 11:37 Message: Logged In: YES user_id=480913 Oh, I forgot, this is easiest reproduced by simple requesting the URL http://www.akamai.com/ Fortunately they Akamai their home page as well. :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=874842&group_id=5470 From noreply at sourceforge.net Thu May 13 13:25:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 13:27:56 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 18:28 Message generated for change (Comment added) made by jasonlowe You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 12:25 Message: Logged In: YES user_id=56897 There didn't seem to be an outcome from the python-dev discussion regarding system() and pthread_atfork(). The thread indicates that system() indeed is supposed to call atfork handlers, so therefore RedHat 9 is violating the pthread standard in that sense. (Whether or not they'll fix it is another issue.) There's also mention that os.system() may be changed to not call system() because of the atfork() problem. If the changes to avoid system() are implemented, would the pthread_atfork() approach still be problematic? As Martin Loewis points out, we could always implement the signal fixup in the child directly after the fork() if Python routines are being used to do the fork() in the first place. However if we're concerned about native modules that directly call fork() then it seems our choices are a pthread_atfork() approach or an approach where SIGINT isn't blocked. Without an async-signal-safe way to route a signal from one thread to another, I don't see how we can leave SIGINT unblocked in all threads. Re: Py_AddPendingCall. That approach might work in many cases, but I assume it doesn't work well when all threads are currently busy in native modules that are not well-behaved. For example, I have two threads: one in readline() and the other blocked in a native call that, like readline(), doesn't return control on EINTR. If the SIGINT is sent to the readline thread, the signal handler could check the thread ID and do the longjmp() since we're the proper thread to do so. If the SIGINT is sent to the other thread, the callback added by Py_AddPendingCall() won't necessarily be processed any time soon because no threads are going to return control (in a timely manner) to Python. To make matters worse, apparently even something as simple as pthread_self(), which we'd use to get the thread ID, isn't async-signal-safe on all platforms. From what I've read, none of the pthread functions are guaranteed to be async-signal-safe. :-( ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-13 11:48 Message: Logged In: YES user_id=119306 pthread_kill(). That is annoying, I have something nearly done that used it. I didn't double check the safety of pthread_kill. I saw that posix says that kill is safe to call from interrupt handlers and went from there. Can we note that we need a pthread_kill in a call to Py_AddPendingCall, and then handle it later? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-13 11:25 Message: Logged In: YES user_id=6656 Just to make life more entertaining, pthread_atfork isn't what you want, either. http://mail.python.org/pipermail/python-dev/2003-December/041309.html ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 11:20 Message: Logged In: YES user_id=56897 Argh! I thought we had a relatively clean solution, but it appears there's a stumbling block with the pthread_kill() approach. pthread_kill() is not guaranteed to be async-signal-safe, and nuggets found on the net indicate there's no portable way to redirect a process signal from one thread to another: http://groups.google.com/groups?q=pthread_kill+async+safe&start=30&hl=en&lr=&selm=3662B6A8.861984D5%40zko.dec.com&rnum=32 http://www.redhat.com/archives/phil-list/2003-December/msg00049.html Given that we can't safely call pthread_kill() from the SIGINT handler directly, there might be another way to solve our problems with pthread_atfork(). Here's my thinking: - Block SIGINT in all threads except the main (readline) thread. - Register via child process handler via pthread_atfork() that sets the SIGINT action for the child process back to the default. Unfortunately this fix isn't localized to the readline module as desired, but it may solve the problems. SIGINT routing will be forced to the readline thread, and child processes won't have SIGINT blocked, solving bug 756940. The IRIX thread signal delivery model (i.e.: broadcast) may cause problems since SIGINT may be pending when we attempt to set the action to default. Having SIGINT pending when the handler is changed to default would kill the child process. Maybe having the child process set the disposition to ignore and then to default would safely clear any pending SIGINT signal? I'll try to run some experiments with the pthread_atfork() approach soon, but work and home life for me is particularly busy lately. Apologies in advance if it takes me a while to respond or submit patches. If we're interested in a timely fix, would it be useful to break up the fix in two stages? I think we can all agree that the current approach of blocking ALL signals in created threads is a Bad Thing. What if we implement a quick, partial fix by simply change the existing code to only block SIGINT? This should be a two-line change to thread_pthread.h where "sigemptyset(&newmask); sigaddset(&newmask, SIGINT);" is used instead of "sigfillset(&newmask);". I see this partial fix having a number of benefits: - Easy change to make. No extra stuff to check for in configure or calls to things that may not exist or work properly. - Much less risky than trying to fix all the problems at once. The change only opens up signals to threads that Python-2.1 is already allowing through. - Should solve the SIGSEGV zombie problem and Guido's SIGTERM annoyance, although it would still have the problem reported in bug 756940. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-12 09:39 Message: Logged In: YES user_id=29957 This seems like a pragmatic and sensible approach to take, to me. It should probably be tested on the HP/UX boxes (google for 'HP/UX testdrive') I particularly like the idea of just putting a test in to block readline in the non-main thread. It seems the pythonic approach - since we can't guarantee behaviour that's anything but sane, it seems like a plan. Or at least make it issue a warning saying "don't do this" when readline is invoked from a non-main thread. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 09:11 Message: Logged In: YES user_id=6380 Sounds good. This solves the problem in the readline module, where it originates. BTW, if we can simplify things by only allowing readline() to be called from the main thread, that's fine with me. Doing console I/O from threads is insane anyway. We can start by assuming the signal broadcast problem is restricted to IRIX, and configure appropriately: define a test symbol for this and in configure, set this when IRIX is detected. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 08:54 Message: Logged In: YES user_id=56897 SIGINT is 'special' because that's the signal behind the problems reported in bug 465673. Given the readline module's setjmp/longjmp mechanism to process SIGINT, we simply cannot allow one thread to do the setjmp() and another thread to do the longjmp() when it receives SIGINT. Without the setjmp/longjmp stuff, SIGINT is no more special than any other asynchronous signal like SIGTERM, SIGUSR1, etc. It'd be great if we could get the desired behavior for SIGINT out of the readline module without setjmp/longjmp, but without help from the readline library I don't see an easy way to do this. The readline library insists on continuing the readline() call after a SIGINT is handled, and there doesn't appear to be any way to get it to abort the current readline() call short of modifying the readline library. If we're stuck with the setjmp/longjmp mechanism, I think we can solve the issues regarding readline() being called from another thread and exec'd processes from threads by using the pthread_kill() technique mentioned earlier. The steps would look something like this: - Do not block any signals (including SIGINT) in any threads. - When we initialize the readline module's jmp_buf via setjmp(), save off the current thread ID. Probably want to check for existing ownership of jmp_buf and flag an error if detected. - When the readline module SIGINT handler is invoked, check if the current thread owns jmp_buf. If we are the owning thread then execute the longjmp (or siglongjmp). If we're not the owning thread, then have the current thread execute pthread_kill(jmp_buf_owner_thread, SIGINT) and little else. This will defer the SIGINT to the only thread that can really process it correctly. - Since SIGINT isn't blocked in any thread, processes exec'd from threads should get the default behavior for SIGINT rather than having it blocked. The above algorithm has a race condition on thread implementations where all threads receive SIGINT. The race can cause SIGINT to be processed more than once. The jmp_buf owning thread might finish the processing of SIGINT before another thread starts its processing and re-sends SIGINT to the jmp_buf owning thread. If there's a way to know via configure that we're on a thread implementation that broadcasts SIGINT, we could #ifdef the code to use something like the getpid() hack in signalmodule.c to do the right thing. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 18:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 18:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 17:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 17:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 14:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 14:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 08:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 03:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 17:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 15:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 14:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 13:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 11:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 20:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 16:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 11:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 08:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 08:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 07:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 07:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 15:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 03:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 09:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 09:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 18:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Thu May 13 14:53:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 14:53:24 2004 Subject: [ python-Bugs-815924 ] tkMessageBox functions reject type and ico Message-ID: Bugs item #815924, was opened at 2003-10-01 17:19 Message generated for change (Settings changed) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=815924&group_id=5470 Category: Tkinter Group: None Status: Open Resolution: None Priority: 5 Submitted By: Tim Diggins (tdiggins) Assigned to: Martin v. L?wis (loewis) >Summary: tkMessageBox functions reject type and ico Initial Comment: Python 2.2.x -> Python 2.3 I've noticed a small separation between documentation and reality in tkMessageBox. According to Fredrik Lundh's documentation of tkMessageBox (in the various online guides & ?printed) you can send type and icon keywords to tkMessageBox's convenience functions (like askquestion) However this doesn't work. Don't know if it used to. So I'm not sure if the docs should be amended or amend the code (originally submitted by Fredrik). An attempt to clear up this bug but was rejected (I think, because they also added a feature). see BUG 761144 https://sourceforge.net/tracker/? func=detail&atid=105470&aid=761144&group_id=5470) A simple way to clean up this would be to amend the first three lines of _show as attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=815924&group_id=5470 From noreply at sourceforge.net Thu May 13 15:26:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 15:27:17 2004 Subject: [ python-Bugs-949329 ] stringprep.py: sets.Set() --> set() Message-ID: Bugs item #949329, was opened at 2004-05-06 18:51 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949329&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Raymond Hettinger (rhettinger) >Assigned to: Raymond Hettinger (rhettinger) Summary: stringprep.py: sets.Set() --> set() Initial Comment: Suggest replacing sets.Set() with set() in stringprep.py. The __contains__ test for the builtin runs several times faster. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-13 21:26 Message: Logged In: YES user_id=21627 The patch is fine, please apply, and regenerate stringprep.py. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-07 20:09 Message: Logged In: YES user_id=80475 Patch is attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949329&group_id=5470 From noreply at sourceforge.net Thu May 13 17:12:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 17:13:11 2004 Subject: [ python-Bugs-949329 ] stringprep.py: sets.Set() --> set() Message-ID: Bugs item #949329, was opened at 2004-05-06 11:51 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949329&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Raymond Hettinger (rhettinger) >Assigned to: Martin v. L?wis (loewis) Summary: stringprep.py: sets.Set() --> set() Initial Comment: Suggest replacing sets.Set() with set() in stringprep.py. The __contains__ test for the builtin runs several times faster. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-13 16:12 Message: Logged In: YES user_id=80475 On Windows with MSVC++ 6.0, I haven't been able to make a UCS-4 build. Can you apply this one for me. Thanks, Raymond ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-13 14:26 Message: Logged In: YES user_id=21627 The patch is fine, please apply, and regenerate stringprep.py. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-07 13:09 Message: Logged In: YES user_id=80475 Patch is attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949329&group_id=5470 From noreply at sourceforge.net Thu May 13 17:47:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 17:47:23 2004 Subject: [ python-Bugs-953599 ] asyncore misses socket closes when poll is used Message-ID: Bugs item #953599, was opened at 2004-05-13 23:47 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=953599&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Shane Kerr (shane_kerr) Assigned to: Nobody/Anonymous (nobody) Summary: asyncore misses socket closes when poll is used Initial Comment: Problem: If the loop() function of asyncore is invoked with poll rather than select, the function readwrite() is used when I/O is available on a socket. However, this function does not check for hangup - provided by POLLHUP. If a socket is attempting to write, then POLLOUT never gets set, so the socket hangs. Because poll() is returning immediately, but the return value is never used, asyncore busy-loops, consuming all available CPU. Possible solutions: The easy solution is to check for POLLHUP in the readwrite() function: if flags & (select.POLLOUT | select.POLLHUP): obj.handle_write_event() This makes the poll work exactly like the select - the application raises a socket.error set to EPIPE. An alternate solution - possibly more graceful - is to invoke the handle_close() method of the object: if flags & select.POLLHUP: obj.handle_close() else: if flags & select.POLLIN: obj.handle_read_event() if flags & select.pollout: obj.handle_write_event() This is incompatible with the select model, but it means that the read and write logic is now the same for socket hangups - handle_close() is invoked. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=953599&group_id=5470 From noreply at sourceforge.net Thu May 13 18:30:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 18:30:13 2004 Subject: [ python-Bugs-846938 ] email.Parser.Parser doesn't check for valid Content-Type Message-ID: Bugs item #846938, was opened at 2003-11-21 17:25 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=846938&group_id=5470 Category: Python Library >Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Jason R. Mastaler (jasonrm) Assigned to: Barry A. Warsaw (bwarsaw) Summary: email.Parser.Parser doesn't check for valid Content-Type Initial Comment: See the attached traceback illustrating how to reproduce the problem. A sample triggering message is also attached. The problem is that the Content-Type is 'multipour alternative', but the Content-Type field has a valid 'boundary=' parameter. The parser doesn't check that Content-Type is valid; it just creates sub-Message objects for each part it finds that is separated by the boundary. Then, when Generator goes to check Content-Type, it discovers that the content-type *is* invalid and it returns 'text/plain' instead! Generator tries to _handle_text and instead finds the list of sub-Message objects. BOOM! ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-13 18:30 Message: Logged In: YES user_id=12800 This is going to be difficult to fix in email 2.5. Can you try the new parser in email3/Python 2.4cvs? It doesn't fail on this message. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=846938&group_id=5470 From noreply at sourceforge.net Thu May 13 19:09:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 19:09:38 2004 Subject: [ python-Bugs-846938 ] email.Parser.Parser doesn't check for valid Content-Type Message-ID: Bugs item #846938, was opened at 2003-11-21 17:25 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=846938&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Jason R. Mastaler (jasonrm) Assigned to: Barry A. Warsaw (bwarsaw) Summary: email.Parser.Parser doesn't check for valid Content-Type Initial Comment: See the attached traceback illustrating how to reproduce the problem. A sample triggering message is also attached. The problem is that the Content-Type is 'multipour alternative', but the Content-Type field has a valid 'boundary=' parameter. The parser doesn't check that Content-Type is valid; it just creates sub-Message objects for each part it finds that is separated by the boundary. Then, when Generator goes to check Content-Type, it discovers that the content-type *is* invalid and it returns 'text/plain' instead! Generator tries to _handle_text and instead finds the list of sub-Message objects. BOOM! ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-13 19:09 Message: Logged In: YES user_id=12800 Never mind that -- the fix is trivial. I'll add this to email 2.5.5 / Python 2.3.4. ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-13 18:30 Message: Logged In: YES user_id=12800 This is going to be difficult to fix in email 2.5. Can you try the new parser in email3/Python 2.4cvs? It doesn't fail on this message. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=846938&group_id=5470 From noreply at sourceforge.net Thu May 13 20:29:57 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 13 20:30:11 2004 Subject: [ python-Bugs-874842 ] httplib fails on Akamai URLs Message-ID: Bugs item #874842, was opened at 2004-01-11 06:16 Message generated for change (Comment added) made by gvanrossum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=874842&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: Accepted Priority: 5 Submitted By: Leif Hedstrom (zwoop) Assigned to: Jeremy Hylton (jhylton) Summary: httplib fails on Akamai URLs Initial Comment: Using Python 2.3.2 and httplib, reading from Akamai URLs will always hang at the end of the transacation. As common as this must be, I couldn't find anything related to it on any search engines, nor on the bug list here. The problem is that Akamai returns an HTTP/1.0 response, with a header like: Connection: keep-alive httplib does not recognize this response properly (the Connection: header parsing is only done for HTTP/1.1 responses). I'm not sure exactly what the right solution is, but I'm supplying one alternative solution that does solve the problem. I'm attaching a diff against httplib.py. ---------------------------------------------------------------------- >Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-13 20:29 Message: Logged In: YES user_id=6380 Looks like nothing happened in CVS... :-( It's too late for 2.3.4 now, Anthony issued the release candidate already. There will be a 2.3.5 though. ---------------------------------------------------------------------- Comment By: Leif Hedstrom (zwoop) Date: 2004-05-13 13:15 Message: Logged In: YES user_id=480913 Hate to beat on a dead horse here, but what was the final outcome of this discussion? Anything I can do to help produce a better patch, documentation or anything? ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-04-20 12:53 Message: Logged In: YES user_id=6380 It's great that the ASF can wield such power over the likes of AOL. But I don't want to presume the same for Python (we're not the #1 web language, not even #2). I'd be more concerned if adding this hack would *break* anything, but that doesn't seem to be the case. So I still think Jeremy can check it in. ---------------------------------------------------------------------- Comment By: Leif Hedstrom (zwoop) Date: 2004-04-20 11:36 Message: Logged In: YES user_id=480913 Yeah, the second solution is what I ended up doing, although it's definitely not obvious for anyone using httplib.py that this is required to support Akamai (see my original blog post at http://www.ogre.com/tiki-view_blog_post.php?blogId=3&postId=30 for both alternative solutions). At a minimum, I think we should provide the AkamaiHTTPResponse class in one way or another, and clearly document that this is required for correct support of Akamai URLs. My vote would probably be to "hack" the original HTTPResponse class, since anyone using HTTPlib for anything that might hit Akamai (perhaps as a referral/redirect) will have to use the fixed version anyways. Unfortunately I don't have any contacts left at Akamai, so I'm not sure how to inform them of their problems. I completely agree that we need to inform them about this problem, my point was that since Akamai works with pretty much everything else (browsers, other modules etc.), I think it'll be quite slow to get them to change. And until then, we're stuck with a module that is effectively semi-broken. Thanks, -- Leif ---------------------------------------------------------------------- Comment By: Greg Stein (gstein) Date: 2004-04-20 02:44 Message: Logged In: YES user_id=6501 Falling into line with "oh, but they won't change it" is why we end up with a whole bunch of bad implementations. If everybody said that, then we wouldn't get anywhere. A long while back, AOL came out with a busted proxy implementation which didn't work with Apache servers. The ASF said, "sorry AOL: you're wrong. fix your proxies." And they did. If we put a hack in for every busted thing that came out over the next ten years, then imagine the craphole we'd be in... :-) That said: yes, you can workaround the issue with a subclass of HTTPResponse which overrides the _check_close() method. You can then create an HTTPConnection subclass which overrides the class variable 'response_class', or you can set that field in an HTTPConnection instance as an instance variable. For example: conn = HTTPConnection(...) conn.response_class = AkamaiBugHandler When the response arrives, the HTTPConnection class uses self.response_class, so there are a few options to get your custom response class into the chain of events. ---------------------------------------------------------------------- Comment By: Leif Hedstrom (zwoop) Date: 2004-04-19 20:57 Message: Logged In: YES user_id=480913 As I said, no matter what we do, it's a hack on something that's broken on the web (now there's a shocker :-). I don't feel terribly strongly on this issue, I merely filed the bug report because I had this problem, and it took me several hours to figure out why my daemon would stall on Akamai URLs. I'm guessing other users of httplib.py might run into the same problem. As for the patch, the comments would of course have to change, I didn't want to impose more changes in the diff than necessary. Besides the suggested patch, an alternative solution is to provide a specialized implementation of the HTTPResponse class, which works with Akamai. The users of the httplib.py module would then have to explicitly request that httplib.HTTPConnection should instantiate that class instead of the default one. Preferably this would be passed as a new argument to the constructor for HTTPConnection. And I agree that it's a hack to have to code around poor server implementations. But not sure what our odds are to get Akamai to fix their servers any time soon, since pretty much any web browser in existance works with their broken implementation. Cheers, -- leif ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-04-19 18:32 Message: Logged In: YES user_id=6380 I won't reject the patch on that basis. Like HTML, it's more useful to be able to handle what we see in the real world than to stick to the standard. Clearly the OP needs to be able to access Akamai servers. He doesn't have the power to fix the Akamai servers,so saying "the server is wrong" doesn't do him any good. (The comment should stateclearly that Akamai *is* wrong though!) Or do you have a different suggestion for how the poster can work around the problem? ---------------------------------------------------------------------- Comment By: Greg Stein (gstein) Date: 2004-04-19 18:26 Message: Logged In: YES user_id=6501 I have a philosophical problem with compensating for servers that obviously break protocols. The server should be fixed, not *every* client on the planet. From that standpoint, this problem/fix should be rejected, though I defer to Guido on that choice. That said, the comment right above the patch should be fixed. The whole point of that comment is, "the header shouldn't be there, so we shouldn't bother to examine the thing." Obviously, the new code does, so the two comments should be merged. The comment about Akamai should also be strengthened to note that it is violating the HTTP protocol (see section 8.1.2.1 of RFC 2616). Summary: I'd reject it, but will leave that to Guido to choose (i.e. "we'll help users even tho it violates protocols"). If he wants it, then +1 if the comments are fixed up. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2004-04-15 17:59 Message: Logged In: YES user_id=31392 Looks good to me. I want to see if I can come up with a simple test module for httplib with the network resource enabled. I'll see if I can do that tonight. ---------------------------------------------------------------------- Comment By: Leif Hedstrom (zwoop) Date: 2004-04-12 16:54 Message: Logged In: YES user_id=480913 Heh, yeah, I'm pretty sure that's the problem, Akamai being confused about protocols. They claim to be a v1.0 HTTP proxy, yet they use v1.1 HTTP headers :-/. This is why I mentioned I wasn't sure exactly what the right solution is. And no matter what we do, it'll be a hack. Maybe the original author of the module has some insight ? Unfortunately, there's a lot of Akamai content out there that are affected by this. Cheers, -- Leif ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-04-12 16:32 Message: Logged In: YES user_id=6380 Hmm... Indeed. read() checks will_close and apparently setting that to False will do the right thing. I don't know HTTP and this code well enough to approve this fix though. Also, the comment right above your patch should probably be fixed; it claims that connection headers on HTTP/1.0 are due to confused proxies. (Maybe that's what Akamai servers are? :-) ---------------------------------------------------------------------- Comment By: Leif Hedstrom (zwoop) Date: 2004-04-12 16:13 Message: Logged In: YES user_id=480913 Yeah, that works for me to. But the problem is in the HTTPResponse class from the httplib.py module. For example, this code (butchered from my application) will hang on Akamai URLs: #!/usr/bin/python import httplib def testHTTPlib(host, url): http = httplib.HTTPConnection(host) try: http.request('GET', url) response = http.getresponse() except IOError: self._log.warning("Can't connect to %s", url) return False except socket.error: self._log.error("Socket error retrieving %s", url) return False except socket.timeout: self._log.warning("Timeout connecting to %s", url) return False else: try: data = response.read() return True except socket.timeout: self._log.warning("Timeout reading from %s", url) return False return False print testHTTPlib("www.ogre.com", "/") print testHTTPlib("www.akamai.com", "/") Granted, I think Akamai aren't strictly following the protocols, but it's inconvenient that this piece of code stalls here (and only for akamai.com domains, I've tried a lot of them). Thanks! -- Leif ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-04-12 15:36 Message: Logged In: YES user_id=6380 Can you give a complete program that reproduces this? I've tried this: >>> import urllib >>> urllib.urlopen("http://www.akamai.com").read() and it doesn't hang for me. I tried a number of Python versions from 2.2 through 2.4a0. ---------------------------------------------------------------------- Comment By: Leif Hedstrom (zwoop) Date: 2004-01-11 14:37 Message: Logged In: YES user_id=480913 Oh, I forgot, this is easiest reproduced by simple requesting the URL http://www.akamai.com/ Fortunately they Akamai their home page as well. :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=874842&group_id=5470 From noreply at sourceforge.net Fri May 14 11:18:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 14 11:19:02 2004 Subject: [ python-Bugs-954030 ] simple typo in pprint module Message-ID: Bugs item #954030, was opened at 2004-05-14 17:18 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954030&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Michal Pasternak (mpasternak) Assigned to: Nobody/Anonymous (nobody) Summary: simple typo in pprint module Initial Comment: import pprint help(pprint) ------------------------------------------------ pprint(object, stream=None) Pretty-print a Python object to a stream [default is sys.sydout]. ------------------------------------------------ sydout -> stdout ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954030&group_id=5470 From noreply at sourceforge.net Fri May 14 12:32:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 14 12:32:35 2004 Subject: [ python-Bugs-954030 ] simple typo in pprint module Message-ID: Bugs item #954030, was opened at 2004-05-14 10:18 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954030&group_id=5470 Category: Documentation Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Michal Pasternak (mpasternak) Assigned to: Nobody/Anonymous (nobody) Summary: simple typo in pprint module Initial Comment: import pprint help(pprint) ------------------------------------------------ pprint(object, stream=None) Pretty-print a Python object to a stream [default is sys.sydout]. ------------------------------------------------ sydout -> stdout ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2004-05-14 11:32 Message: Logged In: YES user_id=44345 thanks - fixed in 1.30 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954030&group_id=5470 From noreply at sourceforge.net Fri May 14 14:02:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 14 14:02:13 2004 Subject: [ python-Bugs-513572 ] isdir behavior getting odder on UNC path Message-ID: Bugs item #513572, was opened at 2002-02-05 17:07 Message generated for change (Comment added) made by glchapman You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=513572&group_id=5470 Category: Python Library Group: Python 2.2 Status: Open Resolution: None Priority: 5 Submitted By: Gary Herron (herron) Assigned to: Mark Hammond (mhammond) Summary: isdir behavior getting odder on UNC path Initial Comment: It's been documented in earlier version of Python on windows that os.path.isdir returns true on a UNC directory only if there was an extra backslash at the end of the argument. In Python2.2 (at least on windows 2000) it appears that *TWO* extra backslashes are needed. Python 2.2 (#28, Dec 21 2001, 12:21:22) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> >>> import os >>> os.path.isdir('\\trainer\island') 0 >>> os.path.isdir('\\trainer\island\') 0 >>> os.path.isdir('\\trainer\island\\') 1 >>> In a perfect world, the first call should return 1, but never has. In older versions of python, the second returned 1, but no longer. In limited tests, appending 2 or more backslashes to the end of any pathname returns the correct answer in both isfile and isdir. ---------------------------------------------------------------------- Comment By: Greg Chapman (glchapman) Date: 2004-05-14 10:02 Message: Logged In: YES user_id=86307 I took a stab at fixing this, see: www.python.org/sf/954115 ---------------------------------------------------------------------- Comment By: Greg Chapman (glchapman) Date: 2004-04-20 10:21 Message: Logged In: YES user_id=86307 I just ran into this bug. I checked the CVS and it appears that no patch has yet been committed for it. Does a patch exist? Am I correct that the suggested change is essentially: if (IsRootUNCName(path)) EnsureTrailingSlash(path); else if (!IsRootDir(path)) NukeTrailingSlashIfPresent(path); stat(path, st); ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2002-04-17 17:10 Message: Logged In: YES user_id=31435 Sounds good to me! I agree it shouldn't be all that hard to special-case UNC roots too -- what I wonder about is how many other forms of "root" syntax MS will make up out of thin air next year . ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2002-04-17 06:30 Message: Logged In: YES user_id=14198 I have done a little analysis of how we use stat and how it performs by instrumenting posixmodule.c. It seems that Tim's concern about Python starup/import is largely unfounded. While Python does call stat() repeatedly at startup, it does so from C rather than os.stat(). Thus, starting and stopping Python yields the following (with my instrumentation): Success: 9 in 1.47592ms, avg 0.164 Failure: 2 in 0.334504ms, avg 0.1673 (ie, os.stat() is called with a valid file 9 times, and invalid file twice. Average time for stat() is 0.16ms per call.) python -c "import os, string, httplib, urllib" shows the same results (ie, no extra stats for imports) However, this is not the typical case. The Python test suite (which takes ~110 seconds wall time on my PC) yields the following: Success: 383 in 84.3571ms, avg 0.2203 Failure: 1253 in 3805.52ms, avg 3.037 egads - 4 seconds spent in failed stat calls, averaging 3ms each!! Further instrumentation shows that stat() can be very slow on directories with many files. In this case, os.stat() in the %TEMP% directory for tempfiles() occasionally took extremely long. OK - so assuming this tempfile behaviour is also not "typical", I tried the COM test suite: Success: 972 in 303.856ms, avg 0.3126 Failure: 16 in 2.60549ms, avg 0.1628 (also with some extremely long times on files that did exist in a directory with many files) So - all this implies to me that: * stat() can be quite slow in some cases, success or failure * We probably shouldn't make this twice as long in every case that fails! So, I am moving back to trying to outguess the stat() implementation. Looking at it shows that indeed UNC roots are treated specially along with the root directory case already handled by Python (courtesy of Tim). Adding an extra check for a UNC root shouldn't be too hard, and can't possibly be as expensive as an extra stat() :) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2002-04-04 16:51 Message: Logged In: YES user_id=31435 Nice to see you, Mark! If you want to pursue this, the caution I had about my idea, but forgot to write down, is that Python does lots of stats during imports, and especially stats on things that usually don't exist (is it there with a .pyd suffix? a .dll suffix? a .py suffix? a .pyw suffix? a .pyc suffix?). If the idea has a bad effect on startup time, that may kill it; startup time is already a sore point for some. OTOH, on Windows we should really, say, be using FindFirstFile() with a wildcard extension for that purpose anyway. ---------------------------------------------------------------------- Comment By: Mark Hammond (mhammond) Date: 2002-04-04 15:46 Message: Logged In: YES user_id=14198 Sorry - I missed this bug. It is not that I wasn't paying attention, but rather that SF's Tracker didn't get my attention :( Have I mentioned how much I have SF and love Bugzilla yet? :) I quite like Tim's algorithm. One extra stat in that case is OK IMO. I can't imagine too many speed sensitive bits of code that continuously check for a non-existent directory. Everyone still OK with that? ---------------------------------------------------------------------- Comment By: Trent Mick (tmick) Date: 2002-04-04 09:08 Message: Logged In: YES user_id=34892 I have struggled with this too. Currently I tend to use this _isdir(). Hopefully this is helpful. def _isdir(dirname): """os.path.isdir() doesn't work for UNC mount points. Fake it. # For an existing mount point # (want: _isdir() == 1) os.path.ismount(r"\crimper\apps") -> 1 os.path.exists(r"\crimper\apps") -> 0 os.path.isdir(r"\crimper\apps") -> 0 os.listdir(r"\crimper\apps") -> [...contents...] # For a non-existant mount point # (want: _isdir() == 0) os.path.ismount(r"\crimper\foo") -> 1 os.path.exists(r"\crimper\foo") -> 0 os.path.isdir(r"\crimper\foo") -> 0 os.listdir(r"\crimper\foo") -> WindowsError # For an existing dir under a mount point # (want: _isdir() == 1) os.path.mount(r"\crimper\apps\Komodo") -> 0 os.path.exists(r"\crimper\apps\Komodo") -> 1 os.path.isdir(r"\crimper\apps\Komodo") -> 1 os.listdir(r"\crimper\apps\Komodo") -> [...contents...] # For a non-existant dir/file under a mount point # (want: _isdir() == 0) os.path.ismount(r"\crimper\apps\foo") -> 0 os.path.exists(r"\crimper\apps\foo") -> 0 os.path.isdir(r"\crimper\apps\foo") -> 0 os.listdir(r"\crimper\apps\foo") -> [] # as if empty contents # For an existing file under a mount point # (want: _isdir() == 0) os.path.ismount(r"\crimper\apps\Komodo\exists.txt") -> 0 os.path.exists(r"\crimper\apps\Komodo\exists.txt") -> 1 os.path.isdir(r"\crimper\apps\Komodo\exists.txt") -> 0 os.listdir(r"\crimper\apps\Komodo\exists.txt") -> WindowsError """ if sys.platform[:3] == 'win' and dirname[:2] == r'\': if os.path.exists(dirname): return os.path.isdir(dirname) try: os.listdir(dirname) except WindowsError: return 0 else: return os.path.ismount(dirname) else: return os.path.isdir(dirname) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2002-03-10 00:03 Message: Logged In: YES user_id=31435 Gordon, none of those are UNC roots -- they follow the rules exactly as stated for non-UNC paths: MS stat() recognizes \ME\E\java if and only if there's no trailing backslash. That's why your first example succeeds. The complication is that Python removes one trailing backslash "by magic" unless the path "looks like a root", and none of these do. That's why your third example works. Your second and fourth examples fail because you specified two trailing backslashes in those, and Python only removes one of them by magic. An example of "a UNC root" would be \ME\E. The MS stat() recognizes a root directory if and only if it *does* have a trailing backslash, and Python's magical backslash removal doesn't know UNC roots from a Euro symbol. So the only way to get Python's isdir() (etc) to recognize \ME\E is to follow it with two backslashes, one because Python strips one away (due to not realizing "it looks like a root"), and another else MS stat() refuses to recognize it. Anyway, I'm unassigning this now, cuz MarkH isn't paying any attentino. If someone wants to write a pile of tedious code to "recognize a UNC root when it sees one", I'd accept the patch. I doubt I'll get it to it myself in this lifetime. ---------------------------------------------------------------------- Comment By: Gordon B. McMillan (gmcm) Date: 2002-03-07 06:31 Message: Logged In: YES user_id=4923 Data point: run on a win2k box, where \ME is an NT box Python 2.2 (#28, Dec 21 2001, 12:21:22) [MSC 32 bit (Intel)] on win32 >>> os.path.isdir(r"\ME\E\java") 1 >>> os.path.isdir(r"\ME\E\java\") 0 >>> os.path.isdir("\\ME\E\java\") 1 >>> os.path.isdir("\\ME\E\java\\") 0 ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2002-02-10 23:28 Message: Logged In: YES user_id=31435 Mark, what do you think about a different approach here? 1. Leave the string alone and *try* stat. If it succeeds, great, we're done. 2. Else if the string doesn't have a trailing (back)slash, append one and try again. Win or lose, that's the end. 3. Else the string does have a trailing (back)slash. If the string has more than one character, strip a trailing (back)slash and try again. Win or lose, that's the end. 4. Else the string is a single (back)slash, yet stat() failed. This shouldn't be possible. It doubles the number of stats in cases where the file path doesn't correspond to anything that exists. OTOH, MS's (back)slash rules are undocumented and incomprehensible (read their implementation of stat() for the whole truth -- we're not out-thinking lots of it now, and the gimmick added after 1.5.2 to out-think part of it is at least breaking Gary's thoroughly sensible use). ---------------------------------------------------------------------- Comment By: Gary Herron (herron) Date: 2002-02-10 23:03 Message: Logged In: YES user_id=395736 Sorry, but I don't have much of an idea which versions I was refering to. I picked up the idea of an extra backslashes in a faq from a web site, the search for which I can't seem to reproduce. It claimed one backslash was enough, but did not specify a python version. It *might* have been old enough to be pre 1.5.2. The two versions I can test are 1.5.1 (where one backslash is enough) and 2.2 (where two are required). This seems to me to support (or at least not contradict) Tim's hypothesis. Gary ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2002-02-10 09:57 Message: Logged In: YES user_id=31435 Gary, exactly what do you mean by "older versions of Python"? That is, specifically which versions? The Microsoft stat() function is extremely picky about trailing (back)slashes. For example, if you have a directory c:/python, and pass "c:/python/" to the MS stat (), it claims no such thing exists. This isn't documented by MS, but that's how it works: a trailing (back)slash is required if and only if the path passed in "is a root". So MS stat() doesn't understand "/python/", and doesn't understand "d:" either. The former doesn't tolerate a (back)slash, while the latter requires one. This is impossible for people to keep straight, so after 1.5.2 Python started removing (back)slashes on its own to make MS stat() happy. The code currently leaves a trailing (back)slash alone if and only if one exists, and in addition of these obtains: 1) The (back)slash is the only character in the path. or 2) The path has 3 characters, and the middle one is a colon. UNC roots don't fit either of those, so do get one (back) slash chopped off. However, just as for any other roots, the MS stat() refuses to recognize them as valid unless they do have a trailing (back)slash. Indeed, the last time I applied a contributed patch to this code, I added a /* XXX UNC root drives should also be exempted? */ comment there. However, this explanation doesn't make sense unless by "older versions of Python" you mean nothing more recent than 1.5.2. If I'm understanding the source of the problem, it should exist in all Pythons after 1.5.2. So if you don't see the same problem in 1.6, 2.0 or 2.1, I'm on the wrong track. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2002-02-08 14:33 Message: Logged In: YES user_id=31435 BTW, it occurs to me that this *may* be a consequence of whatever was done in 2.2 to encode/decode filename strings for system calls on Windows. I didn't follow that, and Mark may be the only one who fully understands the details. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2002-02-08 14:17 Message: Logged In: YES user_id=31435 Here's the implementation of Windows isdir(): def isdir(path): . """Test whether a path is a directory""" . try: . st = os.stat(path) . except os.error: . return 0 . return stat.S_ISDIR(st[stat.ST_MODE]) That is, we return whatever Microsoft's stat() tells us, and our code is the same in 2.2 as in 2.1. I don't have Win2K here, and my Win98 box isn't on a Windows network so I can't even try real UNC paths here. Reassigning to MarkH in case he can do better on either count. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2002-02-08 13:05 Message: Logged In: YES user_id=6380 Tim, I hate to do this to you, but you're the only person I trust with researching this. (My laptop is currently off the net again. :-( ) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=513572&group_id=5470 From noreply at sourceforge.net Sat May 15 00:19:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 15 00:19:47 2004 Subject: [ python-Bugs-954320 ] New email pkg FeedParser croaks on some spam Message-ID: Bugs item #954320, was opened at 2004-05-14 23:19 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954320&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Skip Montanaro (montanaro) Assigned to: Barry A. Warsaw (bwarsaw) Summary: New email pkg FeedParser croaks on some spam Initial Comment: After cvs up today, the Spambayes sb_filter.py app croaks on the attached spam (spam10). I suspect it's got something to do with the new FeedParser in the email package. Running 'sb_filter.py spam10' I get this traceback: Traceback (most recent call last): File "/Users/skip/local/bin/sb_filter.py", line 257, in ? main() File "/Users/skip/local/bin/sb_filter.py", line 245, in main mbox = mboxutils.getmbox(fname) File "/Users/skip/local/lib/python2.4/site-packages/ spambayes/mboxutils.py", line 66, in getmbox return [get_message(sys.stdin)] File "/Users/skip/local/lib/python2.4/site-packages/ spambayes/mboxutils.py", line 129, in get_message msg = email.message_from_string(obj) File "/Users/skip/local/lib/python2.4/email/__init__.py", line 45, in message_from_string return Parser(_class, strict=strict).parsestr(s) File "/Users/skip/local/lib/python2.4/email/Parser.py", line 67, in parsestr return self.parse(StringIO(text), headersonly=headersonly) File "/Users/skip/local/lib/python2.4/email/Parser.py", line 56, in parse feedparser.feed(data) File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 145, in feed self._call_parse() File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 149, in _call_parse self._parse() File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 317, in _parsegen mo = boundaryre.match(line) TypeError: expected string or buffer ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954320&group_id=5470 From noreply at sourceforge.net Sat May 15 04:41:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 15 04:41:39 2004 Subject: [ python-Bugs-954364 ] 2.3 inspect.getframeinfo wrong tb line numbers Message-ID: Bugs item #954364, was opened at 2004-05-15 08:41 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954364&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Robin Becker (rgbecker) Assigned to: Nobody/Anonymous (nobody) Summary: 2.3 inspect.getframeinfo wrong tb line numbers Initial Comment: inspect.getframeinfo always uses f_lineno even when passed a tb. In practice it is not always true that tb.tb_frame.f_lineno==tb.tb_lineno so user functions like inspect.getinnerframes (and hence cgitb) will get wrong information back sometimes. I fixed this using the attached patch. This script illustrates via cgitb. ############ def raise_an_error(): a = 3 b = 4 c = 0 try: a = a/c except: import sys, cgitb, traceback, inspect tbt,tbv,tb = sys.exc_info() print 'traceback\n',''.join(traceback.format_exception(tbt,tbv,tb)) print '\n\ncgitb\n',cgitb.text((tbt,tbv,tb),1) raise_an_error() ############ ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954364&group_id=5470 From noreply at sourceforge.net Sat May 15 11:59:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 15 12:01:00 2004 Subject: [ python-Bugs-954320 ] New email pkg FeedParser croaks on some spam Message-ID: Bugs item #954320, was opened at 2004-05-14 23:19 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954320&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Skip Montanaro (montanaro) Assigned to: Barry A. Warsaw (bwarsaw) Summary: New email pkg FeedParser croaks on some spam Initial Comment: After cvs up today, the Spambayes sb_filter.py app croaks on the attached spam (spam10). I suspect it's got something to do with the new FeedParser in the email package. Running 'sb_filter.py spam10' I get this traceback: Traceback (most recent call last): File "/Users/skip/local/bin/sb_filter.py", line 257, in ? main() File "/Users/skip/local/bin/sb_filter.py", line 245, in main mbox = mboxutils.getmbox(fname) File "/Users/skip/local/lib/python2.4/site-packages/ spambayes/mboxutils.py", line 66, in getmbox return [get_message(sys.stdin)] File "/Users/skip/local/lib/python2.4/site-packages/ spambayes/mboxutils.py", line 129, in get_message msg = email.message_from_string(obj) File "/Users/skip/local/lib/python2.4/email/__init__.py", line 45, in message_from_string return Parser(_class, strict=strict).parsestr(s) File "/Users/skip/local/lib/python2.4/email/Parser.py", line 67, in parsestr return self.parse(StringIO(text), headersonly=headersonly) File "/Users/skip/local/lib/python2.4/email/Parser.py", line 56, in parse feedparser.feed(data) File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 145, in feed self._call_parse() File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 149, in _call_parse self._parse() File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 317, in _parsegen mo = boundaryre.match(line) TypeError: expected string or buffer ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2004-05-15 10:59 Message: Logged In: YES user_id=44345 After looking at the source, this error seems to be pretty shallow. Attached is FeedParser.diff, which seems to fix the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954320&group_id=5470 From noreply at sourceforge.net Sat May 15 12:18:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 15 12:19:47 2004 Subject: [ python-Bugs-954320 ] New email pkg FeedParser croaks on some spam Message-ID: Bugs item #954320, was opened at 2004-05-15 00:19 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954320&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Skip Montanaro (montanaro) Assigned to: Barry A. Warsaw (bwarsaw) Summary: New email pkg FeedParser croaks on some spam Initial Comment: After cvs up today, the Spambayes sb_filter.py app croaks on the attached spam (spam10). I suspect it's got something to do with the new FeedParser in the email package. Running 'sb_filter.py spam10' I get this traceback: Traceback (most recent call last): File "/Users/skip/local/bin/sb_filter.py", line 257, in ? main() File "/Users/skip/local/bin/sb_filter.py", line 245, in main mbox = mboxutils.getmbox(fname) File "/Users/skip/local/lib/python2.4/site-packages/ spambayes/mboxutils.py", line 66, in getmbox return [get_message(sys.stdin)] File "/Users/skip/local/lib/python2.4/site-packages/ spambayes/mboxutils.py", line 129, in get_message msg = email.message_from_string(obj) File "/Users/skip/local/lib/python2.4/email/__init__.py", line 45, in message_from_string return Parser(_class, strict=strict).parsestr(s) File "/Users/skip/local/lib/python2.4/email/Parser.py", line 67, in parsestr return self.parse(StringIO(text), headersonly=headersonly) File "/Users/skip/local/lib/python2.4/email/Parser.py", line 56, in parse feedparser.feed(data) File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 145, in feed self._call_parse() File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 149, in _call_parse self._parse() File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 317, in _parsegen mo = boundaryre.match(line) TypeError: expected string or buffer ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-15 12:18 Message: Logged In: YES user_id=12800 That's exactly the right fix -- I have the same one waiting to be checked in, but I've been trying to boil down the example so I can add it to the test suite. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-05-15 11:59 Message: Logged In: YES user_id=44345 After looking at the source, this error seems to be pretty shallow. Attached is FeedParser.diff, which seems to fix the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954320&group_id=5470 From noreply at sourceforge.net Sat May 15 12:28:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 15 12:29:11 2004 Subject: [ python-Bugs-954320 ] New email pkg FeedParser croaks on some spam Message-ID: Bugs item #954320, was opened at 2004-05-15 00:19 Message generated for change (Comment added) made by bwarsaw You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954320&group_id=5470 Category: Python Library Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Skip Montanaro (montanaro) Assigned to: Barry A. Warsaw (bwarsaw) Summary: New email pkg FeedParser croaks on some spam Initial Comment: After cvs up today, the Spambayes sb_filter.py app croaks on the attached spam (spam10). I suspect it's got something to do with the new FeedParser in the email package. Running 'sb_filter.py spam10' I get this traceback: Traceback (most recent call last): File "/Users/skip/local/bin/sb_filter.py", line 257, in ? main() File "/Users/skip/local/bin/sb_filter.py", line 245, in main mbox = mboxutils.getmbox(fname) File "/Users/skip/local/lib/python2.4/site-packages/ spambayes/mboxutils.py", line 66, in getmbox return [get_message(sys.stdin)] File "/Users/skip/local/lib/python2.4/site-packages/ spambayes/mboxutils.py", line 129, in get_message msg = email.message_from_string(obj) File "/Users/skip/local/lib/python2.4/email/__init__.py", line 45, in message_from_string return Parser(_class, strict=strict).parsestr(s) File "/Users/skip/local/lib/python2.4/email/Parser.py", line 67, in parsestr return self.parse(StringIO(text), headersonly=headersonly) File "/Users/skip/local/lib/python2.4/email/Parser.py", line 56, in parse feedparser.feed(data) File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 145, in feed self._call_parse() File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 149, in _call_parse self._parse() File "/Users/skip/local/lib/python2.4/email/FeedParser.py", line 317, in _parsegen mo = boundaryre.match(line) TypeError: expected string or buffer ---------------------------------------------------------------------- >Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-15 12:28 Message: Logged In: YES user_id=12800 Actually, I won't add a test case for this one -- it'll happens because of the 8092 byte blocking factor of Parser.parse(). ---------------------------------------------------------------------- Comment By: Barry A. Warsaw (bwarsaw) Date: 2004-05-15 12:18 Message: Logged In: YES user_id=12800 That's exactly the right fix -- I have the same one waiting to be checked in, but I've been trying to boil down the example so I can add it to the test suite. ---------------------------------------------------------------------- Comment By: Skip Montanaro (montanaro) Date: 2004-05-15 11:59 Message: Logged In: YES user_id=44345 After looking at the source, this error seems to be pretty shallow. Attached is FeedParser.diff, which seems to fix the problem. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954320&group_id=5470 From noreply at sourceforge.net Sat May 15 13:59:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 15 13:59:59 2004 Subject: [ python-Bugs-954530 ] bsddb3 test errors Message-ID: Bugs item #954530, was opened at 2004-05-15 13:59 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954530&group_id=5470 Category: Extension Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: bsddb3 test errors Initial Comment: test_bsddb3 has 30 errors in 2.3.4c1, WinXP Pro SP1. They all appear to come from BTree tests, and are a mix of DBNotFoundError and TypeError: unsubscriptable object bsddb3.txt is a screen scrap of the whole test_bsddb3 run. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954530&group_id=5470 From noreply at sourceforge.net Sat May 15 14:01:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 15 14:01:49 2004 Subject: [ python-Bugs-954530 ] bsddb3 test errors Message-ID: Bugs item #954530, was opened at 2004-05-15 13:59 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954530&group_id=5470 Category: Extension Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: bsddb3 test errors Initial Comment: test_bsddb3 has 30 errors in 2.3.4c1, WinXP Pro SP1. They all appear to come from BTree tests, and are a mix of DBNotFoundError and TypeError: unsubscriptable object bsddb3.txt is a screen scrap of the whole test_bsddb3 run. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-15 14:01 Message: Logged In: YES user_id=31435 Attaching bsddb3.txt (again?). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954530&group_id=5470 From noreply at sourceforge.net Sat May 15 16:21:57 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 15 16:22:02 2004 Subject: [ python-Bugs-954530 ] bsddb3 test errors Message-ID: Bugs item #954530, was opened at 2004-05-15 10:59 Message generated for change (Comment added) made by greg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954530&group_id=5470 Category: Extension Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Tim Peters (tim_one) Assigned to: Nobody/Anonymous (nobody) Summary: bsddb3 test errors Initial Comment: test_bsddb3 has 30 errors in 2.3.4c1, WinXP Pro SP1. They all appear to come from BTree tests, and are a mix of DBNotFoundError and TypeError: unsubscriptable object bsddb3.txt is a screen scrap of the whole test_bsddb3 run. ---------------------------------------------------------------------- >Comment By: Gregory P. Smith (greg) Date: 2004-05-15 13:21 Message: Logged In: YES user_id=413 some of the errors are occurring in code added by a patch i applied in february; i'm looking at reproducing this (on linux) now. hopefully i just missed something when porting it over. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-15 11:01 Message: Logged In: YES user_id=31435 Attaching bsddb3.txt (again?). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954530&group_id=5470 From noreply at sourceforge.net Sat May 15 17:39:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 15 17:39:59 2004 Subject: [ python-Bugs-954530 ] bsddb3 test errors Message-ID: Bugs item #954530, was opened at 2004-05-15 10:59 Message generated for change (Comment added) made by greg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954530&group_id=5470 Category: Extension Modules Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Tim Peters (tim_one) >Assigned to: Gregory P. Smith (greg) Summary: bsddb3 test errors Initial Comment: test_bsddb3 has 30 errors in 2.3.4c1, WinXP Pro SP1. They all appear to come from BTree tests, and are a mix of DBNotFoundError and TypeError: unsubscriptable object bsddb3.txt is a screen scrap of the whole test_bsddb3 run. ---------------------------------------------------------------------- >Comment By: Gregory P. Smith (greg) Date: 2004-05-15 14:39 Message: Logged In: YES user_id=413 Revision 1.7.6.2 of Lib/bsddb/test/test_basics.py has the offending added test case removed. it doesn't port directly from the 2.4 HEAD due to a different default behaviour. ---------------------------------------------------------------------- Comment By: Gregory P. Smith (greg) Date: 2004-05-15 13:21 Message: Logged In: YES user_id=413 some of the errors are occurring in code added by a patch i applied in february; i'm looking at reproducing this (on linux) now. hopefully i just missed something when porting it over. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-15 11:01 Message: Logged In: YES user_id=31435 Attaching bsddb3.txt (again?). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954530&group_id=5470 From noreply at sourceforge.net Sun May 16 08:25:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 16 08:25:47 2004 Subject: [ python-Bugs-942952 ] Weakness in tuple hash Message-ID: Bugs item #942952, was opened at 2004-04-27 14:41 Message generated for change (Comment added) made by ygale You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 6 Submitted By: Steve Tregidgo (smst) Assigned to: Tim Peters (tim_one) Summary: Weakness in tuple hash Initial Comment: I've encountered some performance problems when constructing dictionaries with keys of a particular form, and have pinned the problem down to the hashing function. I've reproduced the effect in Python 1.5.2, Python 2.2 and Python 2.3.3. I came across this when loading a marshalled dictionary with about 40000 entries. Loading other dictionaries of this size has always been fast, but in this case I killed the interpreter after a few minutes of CPU thrashing. The performance problem was caused because every key in the dictionary had the same hash value. The problem is as follows: for hashable values X and Y, hash( (X, (X, Y)) ) == hash(Y). This is always true (except in the corner case where hash((X, Y)) is internally calculated to be -1 (the error value) and so -2 is forced as the return value). With data in this form where X varies more than Y (Y is constant, or chosen from relatively few values compared to X) chances of collision become high as X is effectively ignored. The hash algorithm for tuples starts with a seed value, then generates a new value for each item in the tuple by multiplying the iteration's starting value by a constant (keeping the lowest 32 bits) and XORing with the hash of the item. The final value is then XORed with the tuple's length. In Python (ignoring the careful business with -1): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) The tuple (X, Y) therefore has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash(Y) ^ 2 ...and the tuple (X, (X, Y)) has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash((X, Y)) ^ 2 The outer multiplication is repeated, and is XORed with itself (cancelling both of them). The XORed 2s cancel also, leaving just hash(Y). Note that this cancellation property also means that the same hash value is shared by (X, (X, (X, (X, Y)))), and (X, (X, (X, (X, (X, (X, Y)))))), and so on, and (X, Z, (X, Z, Y)) and so on. I realise that choosing a hash function is a difficult task, so it may be that the behaviour seen here is a tradeoff against other desireable properties -- I don't have the expertise to tell. My naive suggestion would be that an extra multiplication is necessary, which presumably has a performance impact (multiplication being more expensive than XOR) but would reduce the possibility of cancellation. On the other hand, perhaps this particular case is rare enough that it's not worth the effort. For my own application I'm fortunate in that I can probably rearrange the data structure to avoid this case. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-16 15:25 Message: Logged In: YES user_id=1033539 Hmm, you are correct. This is appears to be an off-by-one problem: the original seed always gets multiplied by the constant (which is silly), and the last item in the tuple does not get multiplied (which causes the bug). The correct solution is to change: value = my_mul(const, value) ^ hash(item) in Steve's pseudo-code to: value = my_mul(const, value ^ hash(item)) Of course, you still get a lot more robustness for almost no cost if you vary "const" across the tuple via a table. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-03 16:40 Message: Logged In: YES user_id=764593 Note that in this case, the problem was because of nested tuples. X was the first element of both tuples, and both tuples had the same length -- so the X's would still have been multiplied by the same constant. (Not the same constant as Y, and hash(X, Y), but the same constant as each other.) A non-linear function might work, but would do bad things to other data. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-02 05:50 Message: Logged In: YES user_id=1033539 I suggest leaving the function as is, but replacing the constant with a value that varies as we step through the tuple. Take the values from a fixed table. When we reach the end of the table, start again from the beginning and mung the values slightly (e.g., increment them). True, there will always be the possibilities of collisions, but this will make it very unlikely except in very weird cases. Using a table of constants is a standard technique for hashes. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 20:45 Message: Logged In: YES user_id=764593 Wouldn't that just shift the pathological case from (x, (x, y)) to (x, (-x, y))? My point was that any hash function will act badly on *some* pattern of input, and if a pattern must be penalized, this might be a good pattern to choose. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 19:36 Message: Logged In: YES user_id=80475 The OP was not referring to "some collisions"; his app collapsed all entries to a single hash value. Changing XOR to + would partially eliminate the self cancelling property of this hash function. Also, I am concerned about the tuple hash using the same multiplier as the hash for other objects. In sets.py, a naive combination of the component hash values caused many distinct sets to collapse to a handful of possibilities -- while tuples do not have an identical issue, it does highlight the risks involved. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 17:50 Message: Logged In: YES user_id=764593 Any hash will have some collisions. If there is going to be predictably bad data, this is probably a good place to have it. The obvious alternatives are a more complicated hash (slows everything down), a different hash for embedded tuples (bad, since hash can't be cached then) or ignoring some elements when determining the hash (bad in the normal case of different data). I would also expect your workaround of data rearrangement to be sensible almost any time (X, (X, Y)) is really a common case. (The intuitive meaning for me is "X - then map X to Y", which could be done as (X, Y) or at least (X, (None, Y)), or perhaps d[X]=(X,Y).) ---------------------------------------------------------------------- Comment By: Steve Tregidgo (smst) Date: 2004-04-27 14:45 Message: Logged In: YES user_id=42335 I'll repeat the tuple hashing algorithm in a fixed-width font (with the first line following "for..." being the loop body): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 From noreply at sourceforge.net Sun May 16 11:30:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 16 11:30:41 2004 Subject: [ python-Bugs-870120 ] segmentation fault in test_re Message-ID: Bugs item #870120, was opened at 2004-01-03 23:44 Message generated for change (Comment added) made by hoel You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=870120&group_id=5470 Category: None Group: None >Status: Closed Resolution: None Priority: 5 Submitted By: Berthold H?llmann (hoel) Assigned to: Nobody/Anonymous (nobody) Summary: segmentation fault in test_re Initial Comment: I'm building python 2.3.3 on a SuSE Linux 9.0 with kernel 2.4.21-144-athlon using "gcc (GCC) 3.3.1 (SuSE Linux)". When using ./configure --with-thread --with-fpectl --with-signal-module --with-pymalloc --enable-shared --with-cxx=g++ make test I get .... test_quopri test_random test_re make: *** [test] Segmentation fault Leaving out the "--enable-shared" switch testing succeeds. After some conversation on comp.lang.python i fiddeled with USE_RECURSION_LIMIT in Modules/_sre.c. Setting it to a value of 6835 testing succeeds with "--enable-shared". Setting it 6840 I get the segmentation fault. ---------------------------------------------------------------------- >Comment By: Berthold H?llmann (hoel) Date: 2004-05-16 17:30 Message: Logged In: YES user_id=81913 Tis is fixed in CVS, but bug persists in 2.3.4c1 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=870120&group_id=5470 From noreply at sourceforge.net Mon May 17 00:36:22 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 17 00:36:33 2004 Subject: [ python-Bugs-754449 ] Exceptions when a thread exits Message-ID: Bugs item #754449, was opened at 2003-06-14 03:32 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=754449&group_id=5470 Category: Threads Group: Python 2.3 Status: Open Resolution: None Priority: 3 Submitted By: Matthias Klose (doko) Assigned to: Brett Cannon (bcannon) Summary: Exceptions when a thread exits Initial Comment: [forwarded from http://bugs.debian.org/195812] The application mentioned is offlineimap, available from ftp://ftp.debian.org/dists/unstable/main/source/. This behavior is new to Python 2.3. When my application exits, I get a lot of these messages: Traceback (most recent call last): File "/usr/lib/python2.3/threading.py", line 426, in __bootstrap self.__stop() File "/usr/lib/python2.3/threading.py", line 435, in __stop self.__block.notifyAll() File "/usr/lib/python2.3/threading.py", line 239, in notifyAll self.notify(len(self.__waiters)) File "/usr/lib/python2.3/threading.py", line 221, in notify currentThread() # for side-effect TypeError: 'NoneType' object is not callable jgoerzen@christoph:~/tree/offlineimap-3.99.18$ ./offlineimap.py -l log -d maildir -a Personal,Excel Unhandled exception in thread started by > Traceback (most recent call last): File "/usr/lib/python2.3/threading.py", line 426, in __bootstrap self.__stop() File "/usr/lib/python2.3/threading.py", line 435, in __stop self.__block.notifyAll() File "/usr/lib/python2.3/threading.py", line 239, in notifyAll self.notify(len(self.__waiters)) File "/usr/lib/python2.3/threading.py", line 221, in notify currentThread() # for side-effect TypeError: 'NoneType' object is not callable Unhandled exception in thread started by > Traceback (most recent call last): File "/usr/lib/python2.3/threading.py", line 426, in __bootstrap self.__stop() File "/usr/lib/python2.3/threading.py", line 435, in __stop self.__block.notifyAll() File "/usr/lib/python2.3/threading.py", line 239, in notifyAll self.notify(len(self.__waiters)) File "/usr/lib/python2.3/threading.py", line 221, in notify currentThread() # for side-effect TypeError: 'NoneType' object is not callable ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-05-16 21:36 Message: Logged In: YES user_id=357491 How to reproduce the bug: * Follow the instructions at http://mnetproject.org/repos/ on how to get a copy of the in-dev version of mnet (I used the instructions for using wget and it worked fine). * Run ``python setup.py``. Do realize that this app will download other Python code (crypto stuff mostly) to finish its install. Takes up about 54 MB on my machine after it is compiled. * Run ``python setup.py test -a``. This executes the testing suite. The bug manifests itself after the testing suite finishes execution. This will consistently cause the bug every time. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-05-16 11:45 Message: Logged In: YES user_id=357491 Discovered this is not fixed after all (previous fix didn't go far enough; still needed, though). Patch 954922 is the second attempt to fix this. This time, though, I had code that could trigger the problem reliably and thus this should be the proper, and final, fix. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-03-08 14:36 Message: Logged In: YES user_id=357491 OK, this has been fixed in Python 2.4 (rev. 1.41 of Lib/threading.py; just removed the two calls to currentThread in the _Condition class that were commented to be for their side-effect). Just because I don't feel rock-solid on my fix to the point of it not changing some obscure semantics that I would be willing to put a gun to my head over it I am not going to backport this until I hear from someone who had this bug and reports that they have not had any issues for a long time. Otherwise I will just leave the fix in 2.4 only. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-03-08 14:32 Message: Logged In: YES user_id=357491 OK, this has been fixed in Python 2.4 (rev. 1.41 of Lib/threading.py; just removed the two calls to currentThread in the _Condition class that were commented to be for their side-effect). Just because I don't feel rock-solid on my fix to the point of it not changing some obscure semantics that I would be willing to put a gun to my head over it I am not going to backport this until I hear from someone who had this bug and reports that they have not had any issues for a long time. Otherwise I will just leave the fix in 2.4 only. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-03-02 20:29 Message: Logged In: YES user_id=357491 To force this error I inserted code to set all attributes of the threading module to None and then throw and exception. After commenting out the two calls to currentThread in _Condition the thrown exception to trigger the problem propogated to the top correctly. I did have the clean-up function give out since I set *everything* to None, but it doesn't look like a normal issue as shown in the exception traceback in the OP. I don't think it is an issue. So it seems commenting the two calls to currentThread in _Condition solves the problem. But since this is threading code and the comment explicitly states it is for the side-effect I am going to try to get a second opinion. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-02-16 20:59 Message: Logged In: YES user_id=357491 Does anyone know how to cause this error in isolation? I have tried a bunch of different ways but cannot cause an exception to be raised at the correct point in Threading.__bootstrap() to lead to self.__stop() to be called while the interpreter is tearing itself down. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-02-09 17:18 Message: Logged In: YES user_id=357491 After staring at the code, I am not even sure if the calls for its side- effect are needed. If you call currentThread(), it either returns a value from _active which is a dict of running Thread instances indexed by thread idents, or a new _DummyThread instance that inserts itself into _active. Now the two calls in the _Condition class are purely for this possible side-effect. And yet at no point is anything from _active, through currentThread or direct access, even touched by a _Condition method. The only chance this might be an issue is if a _Condition instance uses an RLock instance for its locking, which is the default behavior. But this still makes the side-effect need useless. RLocks will call currentThread on their own and since this is all in the same thread the result won't change. And in this specific case of this bug, the _Condition instance is created with a Lock instance since that is what the Thread instance uses for constructing the _Condition instance it use, which is just thread.allocate_lock() which is obviously not an RLock. In other words I can't find the point to the side-effect in _Condition. I will try to come up with some testing code that reproduces the error and then see if just removing the calls in _Condition remove the error and still pass the regression tests. ---------------------------------------------------------------------- Comment By: John Goerzen (jgoerzen) Date: 2003-06-16 06:26 Message: Logged In: YES user_id=491567 I can confirm that this behavior is not present in Python 2.2 in the same version that I am using to test against Python 2.3. I will be on vacation for most of this and next week. I'll try to get to the logging script before I leave, but I might not get to it until I return. FYI, you can also obtain OfflineIMAP at http://quux.org/devel/offlineimap. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-15 23:58 Message: Logged In: YES user_id=357491 OK, following Tim's advice I checked and it seems that Thread calls a method while shutting itself down that calls Condition.notifyAll which calls currentThread which is a global. It would appear that offlineimap is leaving its threads running, the program gets shut down, the threads raise an error while shutting down (probably because things are being torn down), this triggers the stopping method in Thread, and this raises its own exception because of the teardown which is what we are seeing as the TypeError. So the question is whether Condition should store a local reference to currentThread or not. It is not the most pure solution since this shutdown issue is not Condition's, but the only other solution I can think of is to have Thread keep a reference to currentThread, inject it into the current frame's global namespace, call Condition.notifyAll, and then remove the reference from the frame again. I vote for the cleaner, less pure solution. =) Am I insane on this or does it at least sound like this is the problem and proper solution? ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-15 23:19 Message: Logged In: YES user_id=357491 Nuts. For some reason I thought the OP had said when threads were exiting. I will ask on python-dev for a good explanation of what happens when Python is shutting down. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2003-06-15 18:05 Message: Logged In: YES user_id=31435 Note that the OP said "when my application exits". When Python is tearing down the universe, it systematically sets module-global bindings to None. currentThread() is a module global. I can't make more time for this now, so for more info talk about it on Python-Dev. ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-15 16:44 Message: Logged In: YES user_id=357491 Well, I'm stumped. I checked the diff from when 2.2 was initially released until now and the only change that seems to be related to any of this is that what is returned by currentThread is not saved in a variable. But since the error is the calling of currentThread itself and not saving the return value I don't see how that would affect anything. I also went through offlineimap but I didn't see anything there that seemed to be setting currentThread to None. Although since several files do ``import *`` so there still is a possibility of overwriting currentThread locally. So, for my owning learning and to help solve this, I have written a tracing function that writes to stderr using the logging package when it detects that either currentThread or threading.currentThread has been set to None, locally or globally (I assume the code is not injecting into builtins so I didn't bother checking there). The file is named tracer.py and I have attached it to this bug report. If you can execute ``sys.settrace(tracer.trace_currentThread)`` before offlinemap starts executing and immediately within each thread (it has to be called in *every* thread since tracing functions are no inherited from the main thread) it should print out a message when currentThread becomes None. If you *really* want to make this robust you can also have it check sys.modules['threading'] every time as well, but I figure there is not going to be much renaming and masking of currentThread. ---------------------------------------------------------------------- Comment By: Matthias Klose (doko) Date: 2003-06-15 01:44 Message: Logged In: YES user_id=60903 Please see http://packages.debian.org/unstable/mail/offlineimap.html or for the tarball: http://ftp.debian.org/debian/pool/main/o/offlineimap/offlineimap_3.99.18.tar.gz ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2003-06-15 01:19 Message: Logged In: YES user_id=357491 I went to the FTP site and all I found was some huge, compressed files (after changing the path to ftp://ftp.debian.org/ debian/dists/sid/main/source); no specific program called offlinemap. If it is in one of those files can you just add the file to the bug report? As for the reported bug, it looks like currentThread is being redefined, although how that is happening is beyond me. I checked the 'threading' module and no where is currentThread redefined which could lead to None. Has the app being run been changed at all since Python 2.2 was released? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=754449&group_id=5470 From noreply at sourceforge.net Mon May 17 06:36:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 17 06:36:10 2004 Subject: [ python-Bugs-955197 ] CFLAGS, CPPFLAGS, LDFLAGS are NOT used when building modules Message-ID: Bugs item #955197, was opened at 2004-05-17 12:36 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955197&group_id=5470 Category: Build Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Niki W. Waibel (nikiwaibel) Assigned to: Nobody/Anonymous (nobody) Summary: CFLAGS, CPPFLAGS, LDFLAGS are NOT used when building modules Initial Comment: i am trying to compile python in solaris 2.8. ncurses incl files and libs are not in theis usual places (/usr/include, /usr/lib)! so i usually set CPPFLAGS, CFLAGS, LDFLAGS and LD_RUN_PATH when compiling programs. that works nicely if all of the automake, autoconf and libtool machinery is used. but it seems that the python team has decided to do sthg unusual when building the modules... === building '_curses' extension gcc -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -fPIC -fno-strict-aliasing -I. -I/tmp/Python-2.3.3/./Include -I/tmp/Python-2.3. 3/Include -I/tmp/Python-2.3.3 -c /tmp/Python-2.3. 3/Modules/_cursesmodule.c -o build/temp.solaris-2.8-sun4u-2. 3/_cursesmodule.o In file included from /tmp/Python-2.3.3/Modules/_cursesmodule.c:113: /tmp/Python-2.3.3/./Include/py_curses.h:43:21: ncurses.h: No such file or directory === adding CPPFLAGS or CFLAGS to the compile command would sort things out. the same thing happens to other modules as well (bz2 ie). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955197&group_id=5470 From noreply at sourceforge.net Sun May 16 17:33:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 17 08:06:42 2004 Subject: [ python-Bugs-954981 ] Error in urllib2 Examples Message-ID: Bugs item #954981, was opened at 2004-05-16 23:33 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954981&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Phillip (phillip_) Assigned to: Nobody/Anonymous (nobody) Summary: Error in urllib2 Examples Initial Comment: I'll quote from the urllib2 examples: >>> Here we are sending a data-stream to the stdin of a CGI and reading the data it returns to us: >>> import urllib2 >>> req = urllib2.Request(url='https://localhost/cgi-bin/test.cgi', ... data='This data is passed to stdin of the CGI') >>> f = urllib2.urlopen(req) ... <<< urllib2 returns: urllib2.URLError: (This is the Errormsg in 2.3.3, 2.2.3 is a different syntax but says the same. This is seriously misleading, as it implies that somehow urllib2 can handle https, which it can't in this manner. At least not in the way exampled here. Please correct or delete this example. It would be nice if you could provide a working example for handling https. Examples are a good thing in the whole httplib / urllib / urllib2 area, since it appears somewhat hard to overlook. I'm sorry I can't provide a working version of this example, as I am trying to figure out how to handle https myself. Anyway, it's a documentation bug and needs fixing. Thanks. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=954981&group_id=5470 From noreply at sourceforge.net Mon May 17 09:09:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 17 09:09:37 2004 Subject: [ python-Bugs-942952 ] Weakness in tuple hash Message-ID: Bugs item #942952, was opened at 2004-04-27 14:41 Message generated for change (Comment added) made by ygale You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 6 Submitted By: Steve Tregidgo (smst) Assigned to: Nobody/Anonymous (nobody) Summary: Weakness in tuple hash Initial Comment: I've encountered some performance problems when constructing dictionaries with keys of a particular form, and have pinned the problem down to the hashing function. I've reproduced the effect in Python 1.5.2, Python 2.2 and Python 2.3.3. I came across this when loading a marshalled dictionary with about 40000 entries. Loading other dictionaries of this size has always been fast, but in this case I killed the interpreter after a few minutes of CPU thrashing. The performance problem was caused because every key in the dictionary had the same hash value. The problem is as follows: for hashable values X and Y, hash( (X, (X, Y)) ) == hash(Y). This is always true (except in the corner case where hash((X, Y)) is internally calculated to be -1 (the error value) and so -2 is forced as the return value). With data in this form where X varies more than Y (Y is constant, or chosen from relatively few values compared to X) chances of collision become high as X is effectively ignored. The hash algorithm for tuples starts with a seed value, then generates a new value for each item in the tuple by multiplying the iteration's starting value by a constant (keeping the lowest 32 bits) and XORing with the hash of the item. The final value is then XORed with the tuple's length. In Python (ignoring the careful business with -1): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) The tuple (X, Y) therefore has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash(Y) ^ 2 ...and the tuple (X, (X, Y)) has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash((X, Y)) ^ 2 The outer multiplication is repeated, and is XORed with itself (cancelling both of them). The XORed 2s cancel also, leaving just hash(Y). Note that this cancellation property also means that the same hash value is shared by (X, (X, (X, (X, Y)))), and (X, (X, (X, (X, (X, (X, Y)))))), and so on, and (X, Z, (X, Z, Y)) and so on. I realise that choosing a hash function is a difficult task, so it may be that the behaviour seen here is a tradeoff against other desireable properties -- I don't have the expertise to tell. My naive suggestion would be that an extra multiplication is necessary, which presumably has a performance impact (multiplication being more expensive than XOR) but would reduce the possibility of cancellation. On the other hand, perhaps this particular case is rare enough that it's not worth the effort. For my own application I'm fortunate in that I can probably rearrange the data structure to avoid this case. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-17 16:09 Message: Logged In: YES user_id=1033539 Tim, your test fixture is very nice. I verified your result that there are no collisions over the fixture set for the algorithm you specified. Based on that result, I would recommend that we NOT use your algorithm. The probability of no collisions is near zero for 250000 random samples taken from a set of 2^32 elements. So the result shows that your algorithm is far from a good hash function, though it is constructed to be great for that specific fixture set. I ran my algorithm on your fixture set using the table of 64 constants taken from MD5. I got 13 collisions, a reasonable result. It is not true that I repeat multipliers (very often) - note that I increment the elements of the table each time around, so the sequence repeats itself only after 2^38 tuple elements. Also, my algorithm runs at esentially the same speed as the current one: no additional multiplies, only a few adds and increments and such. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-17 00:54 Message: Logged In: YES user_id=31435 I can't make more time for this, so unassigned myself. This seems to make a good set of test inputs: N = whatever base = range(N) xp = [(i, j) for i in base for j in base] inps = base + [(i, j) for i in base for j in xp] + [(i, j) for i in xp for j in base] When N == 50, inps contains 250,050 distinct elements, and 66,966 of them generate a duplicated hash code. Most simple changes to the current algorithm don't do significantly better, and some do much worse. For example, just replacing the xor with an add zooms it to 119,666 duplicated hash codes across that set. Here's a hash function that yields no collisions across that set: def hx(x): if isinstance(x, tuple): result = 0x345678L mult = 1000003 for elt in x: y = hx(elt) result = (((result + y) & 0xffffffffL) * mult) & 0xffffffffL mult = (mult * 69069) & 0xffffffffL result ^= len(x) if result == -1: result = -2 else: result = hash(x) return result In C, none of the "& 0xffffffffL" clauses are needed; in Python code, they're needed to cut results back to 32 bits. 69069 is a well-known multiplier for a better-than-most pure multiplicative PRNG. This does add a multiply per loop trip over what we have today, but it can proceed in parallel with the existing multiply. Unlike a canned table, it won't repeat multipliers, (unless the tuple has more than a billion elements). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-17 00:09 Message: Logged In: YES user_id=31435 Noting that (X, (Y, Z)) and (Y, (X, Z)) hash to the same thing today too (module -1 endcases). For example, >>> hash((3, (4, 8))) == hash((4, (3, 8))) True >>> ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-16 15:25 Message: Logged In: YES user_id=1033539 Hmm, you are correct. This is appears to be an off-by-one problem: the original seed always gets multiplied by the constant (which is silly), and the last item in the tuple does not get multiplied (which causes the bug). The correct solution is to change: value = my_mul(const, value) ^ hash(item) in Steve's pseudo-code to: value = my_mul(const, value ^ hash(item)) Of course, you still get a lot more robustness for almost no cost if you vary "const" across the tuple via a table. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-03 16:40 Message: Logged In: YES user_id=764593 Note that in this case, the problem was because of nested tuples. X was the first element of both tuples, and both tuples had the same length -- so the X's would still have been multiplied by the same constant. (Not the same constant as Y, and hash(X, Y), but the same constant as each other.) A non-linear function might work, but would do bad things to other data. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-02 05:50 Message: Logged In: YES user_id=1033539 I suggest leaving the function as is, but replacing the constant with a value that varies as we step through the tuple. Take the values from a fixed table. When we reach the end of the table, start again from the beginning and mung the values slightly (e.g., increment them). True, there will always be the possibilities of collisions, but this will make it very unlikely except in very weird cases. Using a table of constants is a standard technique for hashes. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 20:45 Message: Logged In: YES user_id=764593 Wouldn't that just shift the pathological case from (x, (x, y)) to (x, (-x, y))? My point was that any hash function will act badly on *some* pattern of input, and if a pattern must be penalized, this might be a good pattern to choose. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 19:36 Message: Logged In: YES user_id=80475 The OP was not referring to "some collisions"; his app collapsed all entries to a single hash value. Changing XOR to + would partially eliminate the self cancelling property of this hash function. Also, I am concerned about the tuple hash using the same multiplier as the hash for other objects. In sets.py, a naive combination of the component hash values caused many distinct sets to collapse to a handful of possibilities -- while tuples do not have an identical issue, it does highlight the risks involved. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 17:50 Message: Logged In: YES user_id=764593 Any hash will have some collisions. If there is going to be predictably bad data, this is probably a good place to have it. The obvious alternatives are a more complicated hash (slows everything down), a different hash for embedded tuples (bad, since hash can't be cached then) or ignoring some elements when determining the hash (bad in the normal case of different data). I would also expect your workaround of data rearrangement to be sensible almost any time (X, (X, Y)) is really a common case. (The intuitive meaning for me is "X - then map X to Y", which could be done as (X, Y) or at least (X, (None, Y)), or perhaps d[X]=(X,Y).) ---------------------------------------------------------------------- Comment By: Steve Tregidgo (smst) Date: 2004-04-27 14:45 Message: Logged In: YES user_id=42335 I'll repeat the tuple hashing algorithm in a fixed-width font (with the first line following "for..." being the loop body): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 From noreply at sourceforge.net Mon May 17 09:39:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 17 09:39:32 2004 Subject: [ python-Bugs-942952 ] Weakness in tuple hash Message-ID: Bugs item #942952, was opened at 2004-04-27 14:41 Message generated for change (Comment added) made by ygale You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 6 Submitted By: Steve Tregidgo (smst) Assigned to: Nobody/Anonymous (nobody) Summary: Weakness in tuple hash Initial Comment: I've encountered some performance problems when constructing dictionaries with keys of a particular form, and have pinned the problem down to the hashing function. I've reproduced the effect in Python 1.5.2, Python 2.2 and Python 2.3.3. I came across this when loading a marshalled dictionary with about 40000 entries. Loading other dictionaries of this size has always been fast, but in this case I killed the interpreter after a few minutes of CPU thrashing. The performance problem was caused because every key in the dictionary had the same hash value. The problem is as follows: for hashable values X and Y, hash( (X, (X, Y)) ) == hash(Y). This is always true (except in the corner case where hash((X, Y)) is internally calculated to be -1 (the error value) and so -2 is forced as the return value). With data in this form where X varies more than Y (Y is constant, or chosen from relatively few values compared to X) chances of collision become high as X is effectively ignored. The hash algorithm for tuples starts with a seed value, then generates a new value for each item in the tuple by multiplying the iteration's starting value by a constant (keeping the lowest 32 bits) and XORing with the hash of the item. The final value is then XORed with the tuple's length. In Python (ignoring the careful business with -1): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) The tuple (X, Y) therefore has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash(Y) ^ 2 ...and the tuple (X, (X, Y)) has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash((X, Y)) ^ 2 The outer multiplication is repeated, and is XORed with itself (cancelling both of them). The XORed 2s cancel also, leaving just hash(Y). Note that this cancellation property also means that the same hash value is shared by (X, (X, (X, (X, Y)))), and (X, (X, (X, (X, (X, (X, Y)))))), and so on, and (X, Z, (X, Z, Y)) and so on. I realise that choosing a hash function is a difficult task, so it may be that the behaviour seen here is a tradeoff against other desireable properties -- I don't have the expertise to tell. My naive suggestion would be that an extra multiplication is necessary, which presumably has a performance impact (multiplication being more expensive than XOR) but would reduce the possibility of cancellation. On the other hand, perhaps this particular case is rare enough that it's not worth the effort. For my own application I'm fortunate in that I can probably rearrange the data structure to avoid this case. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-17 16:39 Message: Logged In: YES user_id=1033539 Here is the Python code for the hash function that I am describing: def hx(x): if isinstance(x, tuple): result = 0x345678L for n, elt in enumerate(x): y = (tbl[n & 0x3f] + (n >> 6)) & 0xffffffffL result = (y * (result ^ hx(elt))) & 0xffffffffL result ^= len(x) else: result = hash(x) return result # 64 constants taken from MD5 # (see Modules/md5c.c in the Python sources) tbl = ( -0x28955b88, -0x173848aa, 0x242070db, -0x3e423112, -0xa83f051, 0x4787c62a, -0x57cfb9ed, -0x2b96aff, 0x698098d8, -0x74bb0851, -0xa44f, -0x76a32842, 0x6b901122, -0x2678e6d, -0x5986bc72, 0x49b40821, -0x9e1da9e, -0x3fbf4cc0, 0x265e5a51, -0x16493856, -0x29d0efa3, 0x2441453, -0x275e197f, -0x182c0438, 0x21e1cde6, -0x3cc8f82a, -0xb2af279, 0x455a14ed, -0x561c16fb, -0x3105c08, 0x676f02d9, -0x72d5b376, -0x5c6be, -0x788e097f, 0x6d9d6122, -0x21ac7f4, -0x5b4115bc, 0x4bdecfa9, -0x944b4a0, -0x41404390, 0x289b7ec6, -0x155ed806, -0x2b10cf7b, 0x4881d05, -0x262b2fc7, -0x1924661b, 0x1fa27cf8, -0x3b53a99b, -0xbd6ddbc, 0x432aff97, -0x546bdc59, -0x36c5fc7, 0x655b59c3, -0x70f3336e, -0x100b83, -0x7a7ba22f, 0x6fa87e4f, -0x1d31920, -0x5cfebcec, 0x4e0811a1, -0x8ac817e, -0x42c50dcb, 0x2ad7d2bb, -0x14792c6f ) ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-17 16:09 Message: Logged In: YES user_id=1033539 Tim, your test fixture is very nice. I verified your result that there are no collisions over the fixture set for the algorithm you specified. Based on that result, I would recommend that we NOT use your algorithm. The probability of no collisions is near zero for 250000 random samples taken from a set of 2^32 elements. So the result shows that your algorithm is far from a good hash function, though it is constructed to be great for that specific fixture set. I ran my algorithm on your fixture set using the table of 64 constants taken from MD5. I got 13 collisions, a reasonable result. It is not true that I repeat multipliers (very often) - note that I increment the elements of the table each time around, so the sequence repeats itself only after 2^38 tuple elements. Also, my algorithm runs at esentially the same speed as the current one: no additional multiplies, only a few adds and increments and such. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-17 00:54 Message: Logged In: YES user_id=31435 I can't make more time for this, so unassigned myself. This seems to make a good set of test inputs: N = whatever base = range(N) xp = [(i, j) for i in base for j in base] inps = base + [(i, j) for i in base for j in xp] + [(i, j) for i in xp for j in base] When N == 50, inps contains 250,050 distinct elements, and 66,966 of them generate a duplicated hash code. Most simple changes to the current algorithm don't do significantly better, and some do much worse. For example, just replacing the xor with an add zooms it to 119,666 duplicated hash codes across that set. Here's a hash function that yields no collisions across that set: def hx(x): if isinstance(x, tuple): result = 0x345678L mult = 1000003 for elt in x: y = hx(elt) result = (((result + y) & 0xffffffffL) * mult) & 0xffffffffL mult = (mult * 69069) & 0xffffffffL result ^= len(x) if result == -1: result = -2 else: result = hash(x) return result In C, none of the "& 0xffffffffL" clauses are needed; in Python code, they're needed to cut results back to 32 bits. 69069 is a well-known multiplier for a better-than-most pure multiplicative PRNG. This does add a multiply per loop trip over what we have today, but it can proceed in parallel with the existing multiply. Unlike a canned table, it won't repeat multipliers, (unless the tuple has more than a billion elements). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-17 00:09 Message: Logged In: YES user_id=31435 Noting that (X, (Y, Z)) and (Y, (X, Z)) hash to the same thing today too (module -1 endcases). For example, >>> hash((3, (4, 8))) == hash((4, (3, 8))) True >>> ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-16 15:25 Message: Logged In: YES user_id=1033539 Hmm, you are correct. This is appears to be an off-by-one problem: the original seed always gets multiplied by the constant (which is silly), and the last item in the tuple does not get multiplied (which causes the bug). The correct solution is to change: value = my_mul(const, value) ^ hash(item) in Steve's pseudo-code to: value = my_mul(const, value ^ hash(item)) Of course, you still get a lot more robustness for almost no cost if you vary "const" across the tuple via a table. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-03 16:40 Message: Logged In: YES user_id=764593 Note that in this case, the problem was because of nested tuples. X was the first element of both tuples, and both tuples had the same length -- so the X's would still have been multiplied by the same constant. (Not the same constant as Y, and hash(X, Y), but the same constant as each other.) A non-linear function might work, but would do bad things to other data. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-02 05:50 Message: Logged In: YES user_id=1033539 I suggest leaving the function as is, but replacing the constant with a value that varies as we step through the tuple. Take the values from a fixed table. When we reach the end of the table, start again from the beginning and mung the values slightly (e.g., increment them). True, there will always be the possibilities of collisions, but this will make it very unlikely except in very weird cases. Using a table of constants is a standard technique for hashes. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 20:45 Message: Logged In: YES user_id=764593 Wouldn't that just shift the pathological case from (x, (x, y)) to (x, (-x, y))? My point was that any hash function will act badly on *some* pattern of input, and if a pattern must be penalized, this might be a good pattern to choose. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 19:36 Message: Logged In: YES user_id=80475 The OP was not referring to "some collisions"; his app collapsed all entries to a single hash value. Changing XOR to + would partially eliminate the self cancelling property of this hash function. Also, I am concerned about the tuple hash using the same multiplier as the hash for other objects. In sets.py, a naive combination of the component hash values caused many distinct sets to collapse to a handful of possibilities -- while tuples do not have an identical issue, it does highlight the risks involved. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 17:50 Message: Logged In: YES user_id=764593 Any hash will have some collisions. If there is going to be predictably bad data, this is probably a good place to have it. The obvious alternatives are a more complicated hash (slows everything down), a different hash for embedded tuples (bad, since hash can't be cached then) or ignoring some elements when determining the hash (bad in the normal case of different data). I would also expect your workaround of data rearrangement to be sensible almost any time (X, (X, Y)) is really a common case. (The intuitive meaning for me is "X - then map X to Y", which could be done as (X, Y) or at least (X, (None, Y)), or perhaps d[X]=(X,Y).) ---------------------------------------------------------------------- Comment By: Steve Tregidgo (smst) Date: 2004-04-27 14:45 Message: Logged In: YES user_id=42335 I'll repeat the tuple hashing algorithm in a fixed-width font (with the first line following "for..." being the loop body): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 From noreply at sourceforge.net Mon May 17 20:47:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 17 20:47:47 2004 Subject: [ python-Bugs-857881 ] ConfigParser bug Message-ID: Bugs item #857881, was opened at 2003-12-10 17:00 Message generated for change (Settings changed) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=857881&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Michel Martin (mrsmile) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: ConfigParser bug Initial Comment: Python Version: 2.3.2c1 OS: Windows 2000 ----------------------- Hello, I was trying to use the ConfigParser provided with Python when I realized that there is a problem with the case in the string passed in the Interpolation feature. Explicitely, test1 works but test2 doesn't. You can see by yourself with the attachment. =========================== [Test1] logFullPath: c:/logs/%(logfilename)s logFilename=test.log [Test2] logFullPath: c:/logs/%(logFilename)s logFilename=test.log =========================== Using a capital letter in %('string')s doesn't work, even if the key is declared with a capital letter. How could we fix this? Thanks for your help, Best regards, Michel Martin ---------------------------------------------------------------------- Comment By: Jordan R McCoy (jrm) Date: 2003-12-24 12:28 Message: Logged In: YES user_id=813983 See patch request #865455 for an explanation of the problem and a patch solution. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=857881&group_id=5470 From noreply at sourceforge.net Mon May 17 22:14:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 17 22:20:29 2004 Subject: [ python-Bugs-942952 ] Weakness in tuple hash Message-ID: Bugs item #942952, was opened at 2004-04-27 07:41 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 6 Submitted By: Steve Tregidgo (smst) Assigned to: Nobody/Anonymous (nobody) Summary: Weakness in tuple hash Initial Comment: I've encountered some performance problems when constructing dictionaries with keys of a particular form, and have pinned the problem down to the hashing function. I've reproduced the effect in Python 1.5.2, Python 2.2 and Python 2.3.3. I came across this when loading a marshalled dictionary with about 40000 entries. Loading other dictionaries of this size has always been fast, but in this case I killed the interpreter after a few minutes of CPU thrashing. The performance problem was caused because every key in the dictionary had the same hash value. The problem is as follows: for hashable values X and Y, hash( (X, (X, Y)) ) == hash(Y). This is always true (except in the corner case where hash((X, Y)) is internally calculated to be -1 (the error value) and so -2 is forced as the return value). With data in this form where X varies more than Y (Y is constant, or chosen from relatively few values compared to X) chances of collision become high as X is effectively ignored. The hash algorithm for tuples starts with a seed value, then generates a new value for each item in the tuple by multiplying the iteration's starting value by a constant (keeping the lowest 32 bits) and XORing with the hash of the item. The final value is then XORed with the tuple's length. In Python (ignoring the careful business with -1): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) The tuple (X, Y) therefore has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash(Y) ^ 2 ...and the tuple (X, (X, Y)) has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash((X, Y)) ^ 2 The outer multiplication is repeated, and is XORed with itself (cancelling both of them). The XORed 2s cancel also, leaving just hash(Y). Note that this cancellation property also means that the same hash value is shared by (X, (X, (X, (X, Y)))), and (X, (X, (X, (X, (X, (X, Y)))))), and so on, and (X, Z, (X, Z, Y)) and so on. I realise that choosing a hash function is a difficult task, so it may be that the behaviour seen here is a tradeoff against other desireable properties -- I don't have the expertise to tell. My naive suggestion would be that an extra multiplication is necessary, which presumably has a performance impact (multiplication being more expensive than XOR) but would reduce the possibility of cancellation. On the other hand, perhaps this particular case is rare enough that it's not worth the effort. For my own application I'm fortunate in that I can probably rearrange the data structure to avoid this case. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-17 22:14 Message: Logged In: YES user_id=31435 Nothing wrong with being "better than random" -- it's not the purpose of Python's hash() to randomize, but to minimize hash collisions. That's why, e.g., hash(int) returns int unchanged. Then when indexing a dict by any contiguous range of ints (excepting -1), there are no collisions at all. That's a good thing. There's nothing in the structure of my example function geared toward the specific test instances used. It *should* do a good job of "spreading out" inputs that are "close together", and that's an important case, and the test inputs have a lot of that. It's a good thing that it avoids all collisions across them. About speed, the only answer is to time both, and across several major platforms. An extra mult may or may not cost more than blowing extra cache lines to suck up a table of ints. About the table of ints, it's always a dubious idea to multiply by an even number. Because the high bits of the product are thrown away, multiplying by an even number throws away information needlessly (it throws away as many useful bits as there are trailing zero bits in the multiplier). odd_number * 69069**i is always odd, so the multiplier in the table-free way is never even (or, worse, divisible by 4, 8, 16, etc). ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-17 09:39 Message: Logged In: YES user_id=1033539 Here is the Python code for the hash function that I am describing: def hx(x): if isinstance(x, tuple): result = 0x345678L for n, elt in enumerate(x): y = (tbl[n & 0x3f] + (n >> 6)) & 0xffffffffL result = (y * (result ^ hx(elt))) & 0xffffffffL result ^= len(x) else: result = hash(x) return result # 64 constants taken from MD5 # (see Modules/md5c.c in the Python sources) tbl = ( -0x28955b88, -0x173848aa, 0x242070db, -0x3e423112, -0xa83f051, 0x4787c62a, -0x57cfb9ed, -0x2b96aff, 0x698098d8, -0x74bb0851, -0xa44f, -0x76a32842, 0x6b901122, -0x2678e6d, -0x5986bc72, 0x49b40821, -0x9e1da9e, -0x3fbf4cc0, 0x265e5a51, -0x16493856, -0x29d0efa3, 0x2441453, -0x275e197f, -0x182c0438, 0x21e1cde6, -0x3cc8f82a, -0xb2af279, 0x455a14ed, -0x561c16fb, -0x3105c08, 0x676f02d9, -0x72d5b376, -0x5c6be, -0x788e097f, 0x6d9d6122, -0x21ac7f4, -0x5b4115bc, 0x4bdecfa9, -0x944b4a0, -0x41404390, 0x289b7ec6, -0x155ed806, -0x2b10cf7b, 0x4881d05, -0x262b2fc7, -0x1924661b, 0x1fa27cf8, -0x3b53a99b, -0xbd6ddbc, 0x432aff97, -0x546bdc59, -0x36c5fc7, 0x655b59c3, -0x70f3336e, -0x100b83, -0x7a7ba22f, 0x6fa87e4f, -0x1d31920, -0x5cfebcec, 0x4e0811a1, -0x8ac817e, -0x42c50dcb, 0x2ad7d2bb, -0x14792c6f ) ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-17 09:09 Message: Logged In: YES user_id=1033539 Tim, your test fixture is very nice. I verified your result that there are no collisions over the fixture set for the algorithm you specified. Based on that result, I would recommend that we NOT use your algorithm. The probability of no collisions is near zero for 250000 random samples taken from a set of 2^32 elements. So the result shows that your algorithm is far from a good hash function, though it is constructed to be great for that specific fixture set. I ran my algorithm on your fixture set using the table of 64 constants taken from MD5. I got 13 collisions, a reasonable result. It is not true that I repeat multipliers (very often) - note that I increment the elements of the table each time around, so the sequence repeats itself only after 2^38 tuple elements. Also, my algorithm runs at esentially the same speed as the current one: no additional multiplies, only a few adds and increments and such. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-16 17:54 Message: Logged In: YES user_id=31435 I can't make more time for this, so unassigned myself. This seems to make a good set of test inputs: N = whatever base = range(N) xp = [(i, j) for i in base for j in base] inps = base + [(i, j) for i in base for j in xp] + [(i, j) for i in xp for j in base] When N == 50, inps contains 250,050 distinct elements, and 66,966 of them generate a duplicated hash code. Most simple changes to the current algorithm don't do significantly better, and some do much worse. For example, just replacing the xor with an add zooms it to 119,666 duplicated hash codes across that set. Here's a hash function that yields no collisions across that set: def hx(x): if isinstance(x, tuple): result = 0x345678L mult = 1000003 for elt in x: y = hx(elt) result = (((result + y) & 0xffffffffL) * mult) & 0xffffffffL mult = (mult * 69069) & 0xffffffffL result ^= len(x) if result == -1: result = -2 else: result = hash(x) return result In C, none of the "& 0xffffffffL" clauses are needed; in Python code, they're needed to cut results back to 32 bits. 69069 is a well-known multiplier for a better-than-most pure multiplicative PRNG. This does add a multiply per loop trip over what we have today, but it can proceed in parallel with the existing multiply. Unlike a canned table, it won't repeat multipliers, (unless the tuple has more than a billion elements). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-16 17:09 Message: Logged In: YES user_id=31435 Noting that (X, (Y, Z)) and (Y, (X, Z)) hash to the same thing today too (module -1 endcases). For example, >>> hash((3, (4, 8))) == hash((4, (3, 8))) True >>> ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-16 08:25 Message: Logged In: YES user_id=1033539 Hmm, you are correct. This is appears to be an off-by-one problem: the original seed always gets multiplied by the constant (which is silly), and the last item in the tuple does not get multiplied (which causes the bug). The correct solution is to change: value = my_mul(const, value) ^ hash(item) in Steve's pseudo-code to: value = my_mul(const, value ^ hash(item)) Of course, you still get a lot more robustness for almost no cost if you vary "const" across the tuple via a table. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-03 09:40 Message: Logged In: YES user_id=764593 Note that in this case, the problem was because of nested tuples. X was the first element of both tuples, and both tuples had the same length -- so the X's would still have been multiplied by the same constant. (Not the same constant as Y, and hash(X, Y), but the same constant as each other.) A non-linear function might work, but would do bad things to other data. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-01 22:50 Message: Logged In: YES user_id=1033539 I suggest leaving the function as is, but replacing the constant with a value that varies as we step through the tuple. Take the values from a fixed table. When we reach the end of the table, start again from the beginning and mung the values slightly (e.g., increment them). True, there will always be the possibilities of collisions, but this will make it very unlikely except in very weird cases. Using a table of constants is a standard technique for hashes. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 13:45 Message: Logged In: YES user_id=764593 Wouldn't that just shift the pathological case from (x, (x, y)) to (x, (-x, y))? My point was that any hash function will act badly on *some* pattern of input, and if a pattern must be penalized, this might be a good pattern to choose. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 12:36 Message: Logged In: YES user_id=80475 The OP was not referring to "some collisions"; his app collapsed all entries to a single hash value. Changing XOR to + would partially eliminate the self cancelling property of this hash function. Also, I am concerned about the tuple hash using the same multiplier as the hash for other objects. In sets.py, a naive combination of the component hash values caused many distinct sets to collapse to a handful of possibilities -- while tuples do not have an identical issue, it does highlight the risks involved. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 10:50 Message: Logged In: YES user_id=764593 Any hash will have some collisions. If there is going to be predictably bad data, this is probably a good place to have it. The obvious alternatives are a more complicated hash (slows everything down), a different hash for embedded tuples (bad, since hash can't be cached then) or ignoring some elements when determining the hash (bad in the normal case of different data). I would also expect your workaround of data rearrangement to be sensible almost any time (X, (X, Y)) is really a common case. (The intuitive meaning for me is "X - then map X to Y", which could be done as (X, Y) or at least (X, (None, Y)), or perhaps d[X]=(X,Y).) ---------------------------------------------------------------------- Comment By: Steve Tregidgo (smst) Date: 2004-04-27 07:45 Message: Logged In: YES user_id=42335 I'll repeat the tuple hashing algorithm in a fixed-width font (with the first line following "for..." being the loop body): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 From noreply at sourceforge.net Mon May 17 22:28:46 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 17 22:28:51 2004 Subject: [ python-Bugs-857881 ] ConfigParser bug Message-ID: Bugs item #857881, was opened at 2003-12-10 17:00 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=857881&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Michel Martin (mrsmile) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: ConfigParser bug Initial Comment: Python Version: 2.3.2c1 OS: Windows 2000 ----------------------- Hello, I was trying to use the ConfigParser provided with Python when I realized that there is a problem with the case in the string passed in the Interpolation feature. Explicitely, test1 works but test2 doesn't. You can see by yourself with the attachment. =========================== [Test1] logFullPath: c:/logs/%(logfilename)s logFilename=test.log [Test2] logFullPath: c:/logs/%(logFilename)s logFilename=test.log =========================== Using a capital letter in %('string')s doesn't work, even if the key is declared with a capital letter. How could we fix this? Thanks for your help, Best regards, Michel Martin ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-17 22:28 Message: Logged In: YES user_id=3066 I've committed a patch that fixes this on the Python development trunk (what will become Python 2.4). The fix is similar to SF patch #865455. Doc/lib/libcfgparser.tex 1.35 Lib/ConfigParser.py 1.64 Lib/test/test_cfgparser.py 1.21 ---------------------------------------------------------------------- Comment By: Jordan R McCoy (jrm) Date: 2003-12-24 12:28 Message: Logged In: YES user_id=813983 See patch request #865455 for an explanation of the problem and a patch solution. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=857881&group_id=5470 From noreply at sourceforge.net Mon May 17 23:32:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 17 23:32:49 2004 Subject: [ python-Bugs-810843 ] Support for non-string data in ConfigParser unclear/buggy Message-ID: Bugs item #810843, was opened at 2003-09-22 17:58 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=810843&group_id=5470 Category: None Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: curby (kirbysdl) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: Support for non-string data in ConfigParser unclear/buggy Initial Comment: In my current project, I'm using ConfigParser to read configuration files as well as manage internal configuration. The following test script generates an error because values of options must be strings in order for ConfigParser to run smoothly. (It's short enough that I'm including it here for reference; I'm also attaching it). #----- BEGIN TEST ----- #!/usr/bin/env python import ConfigParser import sys conf = ConfigParser.ConfigParser() conf.add_section("sect") conf.set("sect","option",2) conf.write(sys.stderr) conf.get("sect","option") #----- END TEST ----- >From the ConfigParser code, I see that the value isn't checked to make sure it is a string before string methods are applied to it. I'm currently using the raw parameter to get() (see http://www.python.org/doc/lib/ConfigParser-objects.html#l2h-1261) in order to circumvent the problem. I'd suggest one or more of the following: (1) Casting the option to a string explicitly instead of assuming it is a string (this could also be done in the set() method). (2) Checking that the option is a string instead of assuming it is a string. (3) Stating in the documentation that the set() method should only take values of type string. (4) Stating in the documentation that the raw parameter should be used when an option's value might not be of type string (this might be the intended usage, but it isn't made clear). I hope none of these are too difficult to implement, but I understand that this is a noncritical problem. Nonetheless, I was shocked to see what I thought was a straightforward use of ConfigParser throw off errors. ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-17 23:32 Message: Logged In: YES user_id=3066 Fixed using methods (2) and (3): ConfigParser.set() now checks that the value is some flavor of string, and raises TypeError if it isn't. This is documented behavior; regression tests have been added. Doc/lib/libcfgparser.tex 1.36 Lib/ConfigParser.py 1.65 Lib/test/test_cfgparser.py 1.22 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=810843&group_id=5470 From noreply at sourceforge.net Mon May 17 23:42:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 17 23:42:59 2004 Subject: [ python-Bugs-864063 ] ConfigParser.getint failure on 2.3.3 Message-ID: Bugs item #864063, was opened at 2003-12-21 13:37 Message generated for change (Comment added) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=864063&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Pending Resolution: None Priority: 5 Submitted By: Jarek Zgoda (zgoda) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: ConfigParser.getint failure on 2.3.3 Initial Comment: It fails with this traceback: jarek:~/projekty/jpawrk/trunk$ ./jpa.py Traceback (most recent call last): File "/home/jarek/projekty/jpawrk/trunk/lib/jt_mainwindow.py", line 328, in OnMessageEdit self.cfg) File "/home/jarek/projekty/jpawrk/trunk/lib/jt_msgeditwindow.py", line 35, in editMessage dlg = MsgEditDlg(tmp, config) File "/home/jarek/projekty/jpawrk/trunk/lib/jt_msgeditwindow.py", line 69, in __init__ self.__setProperties(config) File "/home/jarek/projekty/jpawrk/trunk/lib/jt_msgeditwindow.py", line 76, in __setProperties winSize = cfg.getWindowSize('msgedit') File "/home/jarek/projekty/jpawrk/trunk/lib/jt_cfg.py", line 66, in getWindowSize return (self.cp.getint(windowName, 'width'), File "/usr/lib/python2.3/ConfigParser.py", line 315, in getint return self._get(section, int, option) File "/usr/lib/python2.3/ConfigParser.py", line 312, in _get return conv(self.get(section, option)) File "/usr/lib/python2.3/ConfigParser.py", line 518, in get return self._interpolate(section, option, value, d) File "/usr/lib/python2.3/ConfigParser.py", line 557, in _interpolate if value.find("%(") != -1: AttributeError: 'int' object has no attribute 'find' The offending line in ConfigParser.py is line 553 which reads: value = rawval In case of non-string values this will fail later, since find() is a string method. Changing this line to: value = str(rawval) helps in case of numeric values (didn't check with booleans, though). ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-17 23:42 Message: Logged In: YES user_id=3066 Are you passing non-string values to the .set() method? The _interpolate() method is called before ConfigParser even performs data conversion, so it should only see strings. The script being run in the traceback you show was not attached, so I can't be sure how you're using ConfigParser. The documentation for the .set() method has been found lacking; this has been fixed in response to another bug report. Starting with Python 2.4, passing a non-string as the value to the .set() method will cause a TypeError to be raised. If you were *not* passing a non-integer as the value argument of the .set() method, please respond to this bug report; a short script that demonstrates the problem would help us diagnose the problem more carefully. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=864063&group_id=5470 From noreply at sourceforge.net Mon May 17 23:59:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 17 23:59:56 2004 Subject: [ python-Bugs-830449 ] Config parser don't raise DuplicateSectionError when reading Message-ID: Bugs item #830449, was opened at 2003-10-26 07:20 Message generated for change (Settings changed) made by fdrake You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=830449&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Miki Tebeka (tebeka) >Assigned to: Fred L. Drake, Jr. (fdrake) Summary: Config parser don't raise DuplicateSectionError when reading Initial Comment: According to the documentation of ConfigParser "DuplicateSectionError" it is raised if multiple sections with the same name are found. In the code this is not the case (it continues to update the section with the same name (see ConfigParser.py line 447). Either fix the code (which I prefer :-) of the documentation. Miki ---------------------------------------------------------------------- >Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-17 23:59 Message: Logged In: YES user_id=3066 I've fixed the documentation; changing the semantics is too invasive to existing applications. ConfigParser doesn't have enough internal state to know whether a section was defined twice in a single input file or if it was defined in two different files. The later case should be allowed, while the former case is questionable at best. Doc/lib/libcfgparser.tex 1.37 ---------------------------------------------------------------------- Comment By: Johan M. Hahn (johahn) Date: 2003-10-26 16:18 Message: Logged In: YES user_id=887415 I hereby confirm the bug and I've included some sample code of it as well. Though I don't know which solution is the best. Fixing the code might break some production code that uses duplicate sections and it is therefor easier to just change the documentation. I also don't see reason to why they should be disallowed in the first place. >>> import ConfigParser >>> cp = ConfigParser.ConfigParser() >>> from StringIO import StringIO >>> fp = StringIO(""" ; [section1] var1=option1 [section2] var2=option2 [section1] var3=option3 """) >>> cp.readfp(fp) >>> [x for x in cp.items('sec1')] [('var1', 'option1'), ('var3', 'option3')] >>> ...johahn ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=830449&group_id=5470 From noreply at sourceforge.net Tue May 18 05:03:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 05:03:34 2004 Subject: [ python-Bugs-944119 ] Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty Message-ID: Bugs item #944119, was opened at 2004-04-29 11:05 Message generated for change (Comment added) made by aimacintyre You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944119&group_id=5470 Category: None Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Steve Levis (levis501) Assigned to: Nobody/Anonymous (nobody) Summary: Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty Initial Comment: The attached python file reproduces a bug whereby Python does not receive alarm signals from FreeBSD 4.9. This also appears to be the case in FreeBSD 4.8, and may be related to the addition of HyperThreading in FreeBSD 4.8. If Python (either 2.2.3 or 2.3.3) is configured with -?without-threads before compiling, the bug does not exist. ---------------------------------------------------------------------- >Comment By: Andrew I MacIntyre (aimacintyre) Date: 2004-05-18 19:03 Message: Logged In: YES user_id=250749 Hmm... are you in a position to write a C test case for this? The reason I ask is that I have seen other signal related problems involving a Python interpreter built --with-threads on FreeBSD 4.x which were traced back to issues with libc_r, and I suspect that this problem is similar. ---------------------------------------------------------------------- Comment By: Steve Levis (levis501) Date: 2004-05-06 03:08 Message: Logged In: YES user_id=309698 Ooops, again! I ran on the standard kernel but had python compiled --without-threads, so of course it worked. So, to make a long story short, the bug exists -- ignore my last post. ---------------------------------------------------------------------- Comment By: Steve Levis (levis501) Date: 2004-05-06 02:59 Message: Logged In: YES user_id=309698 I was apparently not running on a standard FreeBSD kernel. It's working now. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944119&group_id=5470 From noreply at sourceforge.net Tue May 18 05:05:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 05:05:53 2004 Subject: [ python-Bugs-944119 ] Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty Message-ID: Bugs item #944119, was opened at 2004-04-29 11:05 Message generated for change (Comment added) made by aimacintyre You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944119&group_id=5470 Category: None Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Steve Levis (levis501) Assigned to: Nobody/Anonymous (nobody) Summary: Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty Initial Comment: The attached python file reproduces a bug whereby Python does not receive alarm signals from FreeBSD 4.9. This also appears to be the case in FreeBSD 4.8, and may be related to the addition of HyperThreading in FreeBSD 4.8. If Python (either 2.2.3 or 2.3.3) is configured with -?without-threads before compiling, the bug does not exist. ---------------------------------------------------------------------- >Comment By: Andrew I MacIntyre (aimacintyre) Date: 2004-05-18 19:05 Message: Logged In: YES user_id=250749 Hmm... are you in a position to write a C test case for this? The reason I ask is that I have seen other signal related problems involving a Python interpreter built --with-threads on FreeBSD 4.x which were traced back to issues with libc_r, and I suspect that this problem is similar. ---------------------------------------------------------------------- Comment By: Andrew I MacIntyre (aimacintyre) Date: 2004-05-18 19:03 Message: Logged In: YES user_id=250749 Hmm... are you in a position to write a C test case for this? The reason I ask is that I have seen other signal related problems involving a Python interpreter built --with-threads on FreeBSD 4.x which were traced back to issues with libc_r, and I suspect that this problem is similar. ---------------------------------------------------------------------- Comment By: Steve Levis (levis501) Date: 2004-05-06 03:08 Message: Logged In: YES user_id=309698 Ooops, again! I ran on the standard kernel but had python compiled --without-threads, so of course it worked. So, to make a long story short, the bug exists -- ignore my last post. ---------------------------------------------------------------------- Comment By: Steve Levis (levis501) Date: 2004-05-06 02:59 Message: Logged In: YES user_id=309698 I was apparently not running on a standard FreeBSD kernel. It's working now. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944119&group_id=5470 From noreply at sourceforge.net Tue May 18 06:04:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 06:04:48 2004 Subject: [ python-Bugs-955772 ] Nested generator terminates prematurely Message-ID: Bugs item #955772, was opened at 2004-05-18 13:02 Message generated for change (Comment added) made by ygale You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 5 Submitted By: Yitz Gale (ygale) Assigned to: Nobody/Anonymous (nobody) Summary: Nested generator terminates prematurely Initial Comment: def g(x, y): for i in x: for j in y: yield i, j r2 = (0, 1) [e for e in g(r2, g(r2, r2))] Expected result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)), (1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))] Actual result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1))] ---------------------------------------------------------------------- >Comment By: Yitz Gale (ygale) Date: 2004-05-18 13:04 Message: Logged In: YES user_id=1033539 Trying again to get the indentation correct: def g(x, y): for i in x: for j in y: yield i, j ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 From noreply at sourceforge.net Tue May 18 06:02:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 06:06:58 2004 Subject: [ python-Bugs-955772 ] Nested generator terminates prematurely Message-ID: Bugs item #955772, was opened at 2004-05-18 13:02 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 5 Submitted By: Yitz Gale (ygale) Assigned to: Nobody/Anonymous (nobody) Summary: Nested generator terminates prematurely Initial Comment: def g(x, y): for i in x: for j in y: yield i, j r2 = (0, 1) [e for e in g(r2, g(r2, r2))] Expected result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)), (1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))] Actual result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1))] ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 From noreply at sourceforge.net Tue May 18 07:24:34 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 07:24:39 2004 Subject: [ python-Bugs-955772 ] Nested generator terminates prematurely Message-ID: Bugs item #955772, was opened at 2004-05-18 11:02 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 5 Submitted By: Yitz Gale (ygale) Assigned to: Nobody/Anonymous (nobody) Summary: Nested generator terminates prematurely Initial Comment: def g(x, y): for i in x: for j in y: yield i, j r2 = (0, 1) [e for e in g(r2, g(r2, r2))] Expected result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)), (1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))] Actual result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1))] ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-18 12:24 Message: Logged In: YES user_id=6656 Um. I think the answer to this is "generators are not reiterable". ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-18 11:04 Message: Logged In: YES user_id=1033539 Trying again to get the indentation correct: def g(x, y): for i in x: for j in y: yield i, j ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 From noreply at sourceforge.net Tue May 18 07:47:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 07:59:23 2004 Subject: [ python-Bugs-864063 ] ConfigParser.getint failure on 2.3.3 Message-ID: Bugs item #864063, was opened at 2003-12-21 19:37 Message generated for change (Comment added) made by zgoda You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=864063&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Jarek Zgoda (zgoda) Assigned to: Fred L. Drake, Jr. (fdrake) Summary: ConfigParser.getint failure on 2.3.3 Initial Comment: It fails with this traceback: jarek:~/projekty/jpawrk/trunk$ ./jpa.py Traceback (most recent call last): File "/home/jarek/projekty/jpawrk/trunk/lib/jt_mainwindow.py", line 328, in OnMessageEdit self.cfg) File "/home/jarek/projekty/jpawrk/trunk/lib/jt_msgeditwindow.py", line 35, in editMessage dlg = MsgEditDlg(tmp, config) File "/home/jarek/projekty/jpawrk/trunk/lib/jt_msgeditwindow.py", line 69, in __init__ self.__setProperties(config) File "/home/jarek/projekty/jpawrk/trunk/lib/jt_msgeditwindow.py", line 76, in __setProperties winSize = cfg.getWindowSize('msgedit') File "/home/jarek/projekty/jpawrk/trunk/lib/jt_cfg.py", line 66, in getWindowSize return (self.cp.getint(windowName, 'width'), File "/usr/lib/python2.3/ConfigParser.py", line 315, in getint return self._get(section, int, option) File "/usr/lib/python2.3/ConfigParser.py", line 312, in _get return conv(self.get(section, option)) File "/usr/lib/python2.3/ConfigParser.py", line 518, in get return self._interpolate(section, option, value, d) File "/usr/lib/python2.3/ConfigParser.py", line 557, in _interpolate if value.find("%(") != -1: AttributeError: 'int' object has no attribute 'find' The offending line in ConfigParser.py is line 553 which reads: value = rawval In case of non-string values this will fail later, since find() is a string method. Changing this line to: value = str(rawval) helps in case of numeric values (didn't check with booleans, though). ---------------------------------------------------------------------- >Comment By: Jarek Zgoda (zgoda) Date: 2004-05-18 13:47 Message: Logged In: YES user_id=92222 Thanks for clarification. And yes, the value passed to .set() is of integer type, as may be found in attached script. I am closing this report with resolution "Fixed". ---------------------------------------------------------------------- Comment By: Fred L. Drake, Jr. (fdrake) Date: 2004-05-18 05:42 Message: Logged In: YES user_id=3066 Are you passing non-string values to the .set() method? The _interpolate() method is called before ConfigParser even performs data conversion, so it should only see strings. The script being run in the traceback you show was not attached, so I can't be sure how you're using ConfigParser. The documentation for the .set() method has been found lacking; this has been fixed in response to another bug report. Starting with Python 2.4, passing a non-string as the value to the .set() method will cause a TypeError to be raised. If you were *not* passing a non-integer as the value argument of the .set() method, please respond to this bug report; a short script that demonstrates the problem would help us diagnose the problem more carefully. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=864063&group_id=5470 From noreply at sourceforge.net Tue May 18 10:51:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 10:51:08 2004 Subject: [ python-Bugs-955985 ] PyDateTime_Check macro unusable Message-ID: Bugs item #955985, was opened at 2004-05-18 14:51 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955985&group_id=5470 Category: Extension Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Jon Willeke (willeke) Assigned to: Nobody/Anonymous (nobody) Summary: PyDateTime_Check macro unusable Initial Comment: I'm using Python 2.3.3 on SuSE Linux 8.2. I'm trying to use the PyDateTime_Check() macro defined in datetime.h, but it refers to the undeclared PyDateTime_DateTimeType. I added the following declaration: PyAPI_DATA(PyTypeObject) PyDateTime_DateTimeType; Looking at the source, I can why this doesn't help: it is defined static. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955985&group_id=5470 From noreply at sourceforge.net Tue May 18 11:06:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 11:06:45 2004 Subject: [ python-Bugs-955985 ] PyDateTime_Check macro unusable Message-ID: Bugs item #955985, was opened at 2004-05-18 10:51 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955985&group_id=5470 Category: Extension Modules Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Jon Willeke (willeke) Assigned to: Nobody/Anonymous (nobody) Summary: PyDateTime_Check macro unusable Initial Comment: I'm using Python 2.3.3 on SuSE Linux 8.2. I'm trying to use the PyDateTime_Check() macro defined in datetime.h, but it refers to the undeclared PyDateTime_DateTimeType. I added the following declaration: PyAPI_DATA(PyTypeObject) PyDateTime_DateTimeType; Looking at the source, I can why this doesn't help: it is defined static. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-18 11:06 Message: Logged In: YES user_id=31435 That's so. We ran out of time before giving datetime a usable C API. There's a patch currently open, seeking to "fix that" for Python 2.4. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955985&group_id=5470 From noreply at sourceforge.net Tue May 18 14:07:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 14:07:10 2004 Subject: [ python-Bugs-955985 ] PyDateTime_Check macro unusable Message-ID: Bugs item #955985, was opened at 2004-05-18 14:51 Message generated for change (Comment added) made by willeke You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955985&group_id=5470 Category: Extension Modules Group: Python 2.3 >Status: Closed >Resolution: Duplicate Priority: 5 Submitted By: Jon Willeke (willeke) Assigned to: Nobody/Anonymous (nobody) Summary: PyDateTime_Check macro unusable Initial Comment: I'm using Python 2.3.3 on SuSE Linux 8.2. I'm trying to use the PyDateTime_Check() macro defined in datetime.h, but it refers to the undeclared PyDateTime_DateTimeType. I added the following declaration: PyAPI_DATA(PyTypeObject) PyDateTime_DateTimeType; Looking at the source, I can why this doesn't help: it is defined static. ---------------------------------------------------------------------- >Comment By: Jon Willeke (willeke) Date: 2004-05-18 18:07 Message: Logged In: YES user_id=185468 Thanks. I searched the bugs, but not the patches. Closing as a duplicate. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-18 15:06 Message: Logged In: YES user_id=31435 That's so. We ran out of time before giving datetime a usable C API. There's a patch currently open, seeking to "fix that" for Python 2.4. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955985&group_id=5470 From noreply at sourceforge.net Tue May 18 15:55:44 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 15:55:52 2004 Subject: [ python-Bugs-944119 ] Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty Message-ID: Bugs item #944119, was opened at 2004-04-28 18:05 Message generated for change (Comment added) made by levis501 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944119&group_id=5470 Category: None Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Steve Levis (levis501) Assigned to: Nobody/Anonymous (nobody) Summary: Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty Initial Comment: The attached python file reproduces a bug whereby Python does not receive alarm signals from FreeBSD 4.9. This also appears to be the case in FreeBSD 4.8, and may be related to the addition of HyperThreading in FreeBSD 4.8. If Python (either 2.2.3 or 2.3.3) is configured with -?without-threads before compiling, the bug does not exist. ---------------------------------------------------------------------- >Comment By: Steve Levis (levis501) Date: 2004-05-18 12:55 Message: Logged In: YES user_id=309698 I've uploaded failure_demo.c, which can be compiled with gcc -o failure_demo failure_demo.c -lutil. Written in C, the signal is properly delivered, and the test does not fail on either FreeBSD 4.7 or 4.9. ---------------------------------------------------------------------- Comment By: Andrew I MacIntyre (aimacintyre) Date: 2004-05-18 02:05 Message: Logged In: YES user_id=250749 Hmm... are you in a position to write a C test case for this? The reason I ask is that I have seen other signal related problems involving a Python interpreter built --with-threads on FreeBSD 4.x which were traced back to issues with libc_r, and I suspect that this problem is similar. ---------------------------------------------------------------------- Comment By: Andrew I MacIntyre (aimacintyre) Date: 2004-05-18 02:03 Message: Logged In: YES user_id=250749 Hmm... are you in a position to write a C test case for this? The reason I ask is that I have seen other signal related problems involving a Python interpreter built --with-threads on FreeBSD 4.x which were traced back to issues with libc_r, and I suspect that this problem is similar. ---------------------------------------------------------------------- Comment By: Steve Levis (levis501) Date: 2004-05-05 10:08 Message: Logged In: YES user_id=309698 Ooops, again! I ran on the standard kernel but had python compiled --without-threads, so of course it worked. So, to make a long story short, the bug exists -- ignore my last post. ---------------------------------------------------------------------- Comment By: Steve Levis (levis501) Date: 2004-05-05 09:59 Message: Logged In: YES user_id=309698 I was apparently not running on a standard FreeBSD kernel. It's working now. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944119&group_id=5470 From noreply at sourceforge.net Tue May 18 17:12:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 17:12:04 2004 Subject: [ python-Bugs-956246 ] csv.writer refuses a variable parameter Message-ID: Bugs item #956246, was opened at 2004-05-18 23:12 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956246&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Xavier BASSERY (balthus) Assigned to: Nobody/Anonymous (nobody) Summary: csv.writer refuses a variable parameter Initial Comment: Hi, I've reproduced that bug on both the win client and the cygwin build (windows 2000). When i do a call on writer like this : cvsWriter = csv.writer(pseudoFichier, 'excel', delimiter=delimiteur) with delimiteur being a variable == ';' , pseudoFichier : a cStringIO I get that error message : cvsWriter = csv.writer(pseudoFichier, 'excel', delimiter=delimiteur) TypeError: bad argument type for built-in operation A way to fix it is to replace the variable delimiteur with it's value, so cvsWriter = csv.writer(pseudoFichier, 'excel', delimiter=';') works like a bliss ! I've tried to look at the source of csv.py just to see that the method that i wanted to see is imported from a c library (_csv.lib). I'm not strong enough to look further. I hope someone here will be able to find the reason of what seems to be a bug. Greetings, Xavier ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956246&group_id=5470 From noreply at sourceforge.net Tue May 18 17:57:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 17:57:53 2004 Subject: [ python-Bugs-956246 ] csv.writer refuses a variable parameter Message-ID: Bugs item #956246, was opened at 2004-05-18 23:12 Message generated for change (Settings changed) made by balthus You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956246&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Deleted Resolution: None Priority: 5 Submitted By: Xavier BASSERY (balthus) Assigned to: Nobody/Anonymous (nobody) Summary: csv.writer refuses a variable parameter Initial Comment: Hi, I've reproduced that bug on both the win client and the cygwin build (windows 2000). When i do a call on writer like this : cvsWriter = csv.writer(pseudoFichier, 'excel', delimiter=delimiteur) with delimiteur being a variable == ';' , pseudoFichier : a cStringIO I get that error message : cvsWriter = csv.writer(pseudoFichier, 'excel', delimiter=delimiteur) TypeError: bad argument type for built-in operation A way to fix it is to replace the variable delimiteur with it's value, so cvsWriter = csv.writer(pseudoFichier, 'excel', delimiter=';') works like a bliss ! I've tried to look at the source of csv.py just to see that the method that i wanted to see is imported from a c library (_csv.lib). I'm not strong enough to look further. I hope someone here will be able to find the reason of what seems to be a bug. Greetings, Xavier ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956246&group_id=5470 From noreply at sourceforge.net Tue May 18 18:45:54 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 18:45:57 2004 Subject: [ python-Bugs-956303 ] Potential data corruption bug in save_pers in pickle module. Message-ID: Bugs item #956303, was opened at 2004-05-18 22:45 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956303&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Allan Crooks (amc1) Assigned to: Nobody/Anonymous (nobody) Summary: Potential data corruption bug in save_pers in pickle module. Initial Comment: There is a bug in save_pers in both the pickle and cPickle modules in Python. It occurs when someone uses a Pickler instance which is using an ASCII protocol and also has persistent_id defined which can return a persistent reference that can contain newline characters in. The current implementation of save_pers in the pickle module is as follows: ---- def save_pers(self, pid): # Save a persistent id reference if self.bin: self.save(pid) self.write(BINPERSID) else: self.write(PERSID + str(pid) + '\n') ---- The else clause assumes that the 'pid' will not be a string which one or more newline characters. If the pickler pickles a persistent ID which has a newline in it, then an unpickler with a corresponding persistent_load method will incorrectly unpickle the data - usually interpreting the character after the newline as a marker indicating what type of data should be expected (usually resulting in an exception being raised when the remaining data is not in the format expected). I have attached an example file which illustrates in what circumstances the error occurs. Workarounds for this bug are: 1) Use binary mode for picklers. 2) Modify subclass implementations of save_pers to ensure that newlines are not returned for persistent ID's. Although you may assume in general that this bug would only occur on rare occasions (due to the unlikely situation where someone would implement persistent_id so that it would return a string with a newline character embedded), it may occur more frequently if the subclass implementation of persistent_id uses a string which has been constructed using the marshal module. This bug was discovered when our code implemented the persistent_id method, which was returning the marshalled format of a tuple which contained strings. It occurred when one or more of the strings had a length of ten characters - the marshalled format of that string contains the string's length, where the byte used to represent the number 10 is the same as the one which represents the newline character: >>> marshal.dumps('a' * 10) 's\n\x00\x00\x00aaaaaaaaaa' >>> chr(10) '\n' I have replicated this bug on Python 1.5.2 and Python 2.3b1, and I believe it is present on all 2.x versions of Python. Many thanks to SourceForge user (and fellow colleague) SMST who diagnosed the bug and provided the test cases attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956303&group_id=5470 From noreply at sourceforge.net Tue May 18 18:51:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 18:51:35 2004 Subject: [ python-Bugs-956303 ] Potential data corruption bug in save_pers in pickle module. Message-ID: Bugs item #956303, was opened at 2004-05-18 22:45 Message generated for change (Settings changed) made by amc1 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956303&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None >Priority: 7 Submitted By: Allan Crooks (amc1) Assigned to: Nobody/Anonymous (nobody) Summary: Potential data corruption bug in save_pers in pickle module. Initial Comment: There is a bug in save_pers in both the pickle and cPickle modules in Python. It occurs when someone uses a Pickler instance which is using an ASCII protocol and also has persistent_id defined which can return a persistent reference that can contain newline characters in. The current implementation of save_pers in the pickle module is as follows: ---- def save_pers(self, pid): # Save a persistent id reference if self.bin: self.save(pid) self.write(BINPERSID) else: self.write(PERSID + str(pid) + '\n') ---- The else clause assumes that the 'pid' will not be a string which one or more newline characters. If the pickler pickles a persistent ID which has a newline in it, then an unpickler with a corresponding persistent_load method will incorrectly unpickle the data - usually interpreting the character after the newline as a marker indicating what type of data should be expected (usually resulting in an exception being raised when the remaining data is not in the format expected). I have attached an example file which illustrates in what circumstances the error occurs. Workarounds for this bug are: 1) Use binary mode for picklers. 2) Modify subclass implementations of save_pers to ensure that newlines are not returned for persistent ID's. Although you may assume in general that this bug would only occur on rare occasions (due to the unlikely situation where someone would implement persistent_id so that it would return a string with a newline character embedded), it may occur more frequently if the subclass implementation of persistent_id uses a string which has been constructed using the marshal module. This bug was discovered when our code implemented the persistent_id method, which was returning the marshalled format of a tuple which contained strings. It occurred when one or more of the strings had a length of ten characters - the marshalled format of that string contains the string's length, where the byte used to represent the number 10 is the same as the one which represents the newline character: >>> marshal.dumps('a' * 10) 's\n\x00\x00\x00aaaaaaaaaa' >>> chr(10) '\n' I have replicated this bug on Python 1.5.2 and Python 2.3b1, and I believe it is present on all 2.x versions of Python. Many thanks to SourceForge user (and fellow colleague) SMST who diagnosed the bug and provided the test cases attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956303&group_id=5470 From noreply at sourceforge.net Tue May 18 19:22:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 19:22:23 2004 Subject: [ python-Feature Requests-920680 ] readline not implemented for UTF-16 Message-ID: Feature Requests item #920680, was opened at 2004-03-21 17:37 Message generated for change (Comment added) made by etrepum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 3 Submitted By: Bob Ippolito (etrepum) Assigned to: Nobody/Anonymous (nobody) Summary: readline not implemented for UTF-16 Initial Comment: The StreamReader for UTF-16 (all three of them) doesn't implement readline. ---------------------------------------------------------------------- >Comment By: Bob Ippolito (etrepum) Date: 2004-05-18 19:22 Message: Logged In: YES user_id=139309 I've attached a monkeypatch to get readline support for utf-16 codecs.. import utf16reader utf16reader.install() It can be trivially inserted into the utf16 encodings implementation.. it would be really cool if someone would audit the implementation and sneak it in before Python 2.4 :) ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-03-21 17:54 Message: Logged In: YES user_id=139309 I don't need it enough to write a patch, but this is what I used instead.. and it seems like it might work: try: for line in inFile: tline = translator(line) outFile.write(tline) except NotImplementedError: BUFFER = 16384 bytes = inFile.read(BUFFER) while bytes: lines = bytes.split(u'\n') bytes = lines.pop() for line in lines: tline = translator(line) outFile.write(tline) newbytes = inFile.read(BUFFER) bytes += newbytes if not newbytes and bytes: bytes += u'\n' ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-21 17:44 Message: Logged In: YES user_id=38388 Patches are welcome :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 From noreply at sourceforge.net Tue May 18 21:11:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 21:14:33 2004 Subject: [ python-Bugs-956246 ] csv.writer refuses a variable parameter Message-ID: Bugs item #956246, was opened at 2004-05-18 16:12 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956246&group_id=5470 Category: Python Library Group: Python 2.3 Status: Deleted Resolution: None Priority: 5 Submitted By: Xavier BASSERY (balthus) >Assigned to: Skip Montanaro (montanaro) Summary: csv.writer refuses a variable parameter Initial Comment: Hi, I've reproduced that bug on both the win client and the cygwin build (windows 2000). When i do a call on writer like this : cvsWriter = csv.writer(pseudoFichier, 'excel', delimiter=delimiteur) with delimiteur being a variable == ';' , pseudoFichier : a cStringIO I get that error message : cvsWriter = csv.writer(pseudoFichier, 'excel', delimiter=delimiteur) TypeError: bad argument type for built-in operation A way to fix it is to replace the variable delimiteur with it's value, so cvsWriter = csv.writer(pseudoFichier, 'excel', delimiter=';') works like a bliss ! I've tried to look at the source of csv.py just to see that the method that i wanted to see is imported from a c library (_csv.lib). I'm not strong enough to look further. I hope someone here will be able to find the reason of what seems to be a bug. Greetings, Xavier ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956246&group_id=5470 From noreply at sourceforge.net Tue May 18 21:52:13 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 21:52:22 2004 Subject: [ python-Bugs-924301 ] A leak case with cmd.py & readline & exception Message-ID: Bugs item #924301, was opened at 2004-03-26 19:28 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924301&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Sverker Nilsson (svenil) >Assigned to: Michael Hudson (mwh) Summary: A leak case with cmd.py & readline & exception Initial Comment: A leak to do with readline & cmd, in Python 2.3. I found out what hold on to my interactive objects too long ('for ever') in certain circumstances. The circumstance had to do with an exception being raised in Cmd.cmdloop and handled (or not handled) outside of Cmd.cmdloop. In cmd.py, class Cmd, in cmdloop(), if an exception is raised and propagated out from the interior of cmdloop, the function postloop() is not called. The default function of this, (in 2.3) when the readline library is present, is to restore the completer, via: readline.set_completer(self.old_completer) If this is not done, the newly (by preloop) inserted completer will remain. Even if the loop is called again and run without exception, the new completer will remain, because then in postloop the old completer will be set to our new completer. When we exit, the completer will remain the one we set. This will hold on to our object, aka 'leak'. - In cmd.py in 2.2 no attempt was made to restore the completer, so this was also a kind of leak, but it was replaced the next time a Cmd instance was initialized. Now, however, the next time we will not replace the old completer, but both of them will remain in memory. The old one will be stored as self.old_completer. If we terminate with an exception, bad luck... the stored completer retains both of the instances. If we terminate normally, the old one will be retained. In no case do we restore the space of the first instance. The only way that would happen, would be if the second instance first exited the loop with an exception, and then entered the loop again an exited normally. But then, the second instance is retained instead! If each instance happens to terminate with an exception, it seems well possible that an ever increasing chain of leaking instances will be accumulated. My fix is to always call the postloop, given the preloop succeeded. This is done via a try:finally clause. def cmdloop(self, intro=None): ... self.preloop() try: ... finally: # Make sure postloop called self.postloop() I am attaching my patched version of cmd.py. It was originally from the tarball of Python 2.3.3 downloaded from Python.org some month or so ago in which cmd.py had this size & date: 14504 Feb 19 2003 cmd.py Best regards, Sverker Nilsson ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-18 20:52 Message: Logged In: YES user_id=80475 Michael, this touches some of your code. Do you want to handle this one? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924301&group_id=5470 From noreply at sourceforge.net Tue May 18 21:54:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 21:54:42 2004 Subject: [ python-Bugs-956408 ] Simplifiy coding in cmd.py Message-ID: Bugs item #956408, was opened at 2004-05-18 20:54 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956408&group_id=5470 Category: None Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Anthony Baxter (anthonybaxter) Summary: Simplifiy coding in cmd.py Initial Comment: In the cmd.py 1.35 checkin on 2/6/2003, there are many lines like: self.stdout.write("%s\n"%str(header)) I believe the str() call in unnecessary. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956408&group_id=5470 From noreply at sourceforge.net Tue May 18 21:55:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 21:55:22 2004 Subject: [ python-Bugs-956246 ] csv.writer refuses a variable parameter Message-ID: Bugs item #956246, was opened at 2004-05-18 16:12 Message generated for change (Comment added) made by montanaro You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956246&group_id=5470 Category: Python Library Group: Python 2.3 Status: Deleted Resolution: None Priority: 5 Submitted By: Xavier BASSERY (balthus) Assigned to: Skip Montanaro (montanaro) Summary: csv.writer refuses a variable parameter Initial Comment: Hi, I've reproduced that bug on both the win client and the cygwin build (windows 2000). When i do a call on writer like this : cvsWriter = csv.writer(pseudoFichier, 'excel', delimiter=delimiteur) with delimiteur being a variable == ';' , pseudoFichier : a cStringIO I get that error message : cvsWriter = csv.writer(pseudoFichier, 'excel', delimiter=delimiteur) TypeError: bad argument type for built-in operation A way to fix it is to replace the variable delimiteur with it's value, so cvsWriter = csv.writer(pseudoFichier, 'excel', delimiter=';') works like a bliss ! I've tried to look at the source of csv.py just to see that the method that i wanted to see is imported from a c library (_csv.lib). I'm not strong enough to look further. I hope someone here will be able to find the reason of what seems to be a bug. Greetings, Xavier ---------------------------------------------------------------------- >Comment By: Skip Montanaro (montanaro) Date: 2004-05-18 20:55 Message: Logged In: YES user_id=44345 I can't reproduce the problem: >>> import csv >>> import cStringIO >>> pseudoFichier = cStringIO.StringIO() >>> delimiteur = ';' >>> writer = csv.writer(pseudoFichier, 'excel', delimiter=delimiteur) >>> writer.writerow([1,2,3,4]) >>> pseudoFichier.getvalue() '1;2;3;4\r\n' ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956246&group_id=5470 From noreply at sourceforge.net Tue May 18 23:02:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 18 23:02:53 2004 Subject: [ python-Bugs-956303 ] Potential data corruption bug in save_pers in pickle module. Message-ID: Bugs item #956303, was opened at 2004-05-18 18:45 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956303&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 7 Submitted By: Allan Crooks (amc1) Assigned to: Nobody/Anonymous (nobody) Summary: Potential data corruption bug in save_pers in pickle module. Initial Comment: There is a bug in save_pers in both the pickle and cPickle modules in Python. It occurs when someone uses a Pickler instance which is using an ASCII protocol and also has persistent_id defined which can return a persistent reference that can contain newline characters in. The current implementation of save_pers in the pickle module is as follows: ---- def save_pers(self, pid): # Save a persistent id reference if self.bin: self.save(pid) self.write(BINPERSID) else: self.write(PERSID + str(pid) + '\n') ---- The else clause assumes that the 'pid' will not be a string which one or more newline characters. If the pickler pickles a persistent ID which has a newline in it, then an unpickler with a corresponding persistent_load method will incorrectly unpickle the data - usually interpreting the character after the newline as a marker indicating what type of data should be expected (usually resulting in an exception being raised when the remaining data is not in the format expected). I have attached an example file which illustrates in what circumstances the error occurs. Workarounds for this bug are: 1) Use binary mode for picklers. 2) Modify subclass implementations of save_pers to ensure that newlines are not returned for persistent ID's. Although you may assume in general that this bug would only occur on rare occasions (due to the unlikely situation where someone would implement persistent_id so that it would return a string with a newline character embedded), it may occur more frequently if the subclass implementation of persistent_id uses a string which has been constructed using the marshal module. This bug was discovered when our code implemented the persistent_id method, which was returning the marshalled format of a tuple which contained strings. It occurred when one or more of the strings had a length of ten characters - the marshalled format of that string contains the string's length, where the byte used to represent the number 10 is the same as the one which represents the newline character: >>> marshal.dumps('a' * 10) 's\n\x00\x00\x00aaaaaaaaaa' >>> chr(10) '\n' I have replicated this bug on Python 1.5.2 and Python 2.3b1, and I believe it is present on all 2.x versions of Python. Many thanks to SourceForge user (and fellow colleague) SMST who diagnosed the bug and provided the test cases attached. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-18 23:02 Message: Logged In: YES user_id=31435 The only documentation is the "Pickling and unpickling external objects" section of the Library Reference Manual, which says: """ Such objects are referenced by a ``persistent id'', which is just an arbitrary string of printable ASCII characters. """ A newline is universally considered to be a control character, not a printable character (e.g., try isprint('\n') under your local C compiler). So this is functioning as designed and as documented. If you don't find the docs clear, we should call this a documentation bug. If you think the semantics should change to allow more than printable characters, then this should become a feature request, and more is needed to define exactly which characters should be allowed. The current implementation is correct for persistent ids that meet the documented requirement. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956303&group_id=5470 From noreply at sourceforge.net Wed May 19 04:19:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 04:19:17 2004 Subject: [ python-Feature Requests-920680 ] readline not implemented for UTF-16 Message-ID: Feature Requests item #920680, was opened at 2004-03-21 23:37 Message generated for change (Comment added) made by lemburg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 3 Submitted By: Bob Ippolito (etrepum) >Assigned to: M.-A. Lemburg (lemburg) Summary: readline not implemented for UTF-16 Initial Comment: The StreamReader for UTF-16 (all three of them) doesn't implement readline. ---------------------------------------------------------------------- >Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-19 10:19 Message: Logged In: YES user_id=38388 Thanks for the patch. Some comments: * Unicode has a lot more line-end markers than just LF; you should use .splitlines() to break lines at all of them * please collapse both methods (sized + unsized) into one method and default to 256 bytes for the buffer size ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-05-19 01:22 Message: Logged In: YES user_id=139309 I've attached a monkeypatch to get readline support for utf-16 codecs.. import utf16reader utf16reader.install() It can be trivially inserted into the utf16 encodings implementation.. it would be really cool if someone would audit the implementation and sneak it in before Python 2.4 :) ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-03-21 23:54 Message: Logged In: YES user_id=139309 I don't need it enough to write a patch, but this is what I used instead.. and it seems like it might work: try: for line in inFile: tline = translator(line) outFile.write(tline) except NotImplementedError: BUFFER = 16384 bytes = inFile.read(BUFFER) while bytes: lines = bytes.split(u'\n') bytes = lines.pop() for line in lines: tline = translator(line) outFile.write(tline) newbytes = inFile.read(BUFFER) bytes += newbytes if not newbytes and bytes: bytes += u'\n' ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-21 23:44 Message: Logged In: YES user_id=38388 Patches are welcome :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 From noreply at sourceforge.net Wed May 19 04:28:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 04:28:40 2004 Subject: [ python-Bugs-951270 ] Generator expression used in doc example Message-ID: Bugs item #951270, was opened at 2004-05-10 09:18 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951270&group_id=5470 Category: Python Library Group: Python 2.4 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Nick Coghlan (ncoghlan) Assigned to: Raymond Hettinger (rhettinger) Summary: Generator expression used in doc example Initial Comment: One of the examples in test_deque uses a generator expression to populate the deque. This means the doctest run fails for test_deque when run without the generator expressions patch. ncoghlan@localhost src]$ ./python ./Lib/test/test_deque.py test_basics (__main__.TestBasic) ... ok test_big_queue_popleft (__main__.TestBasic) ... ok test_big_queue_popright (__main__.TestBasic) ... ok test_big_stack_left (__main__.TestBasic) ... ok test_big_stack_right (__main__.TestBasic) ... ok test_clear (__main__.TestBasic) ... ok test_comparisons (__main__.TestBasic) ... ok test_copy (__main__.TestBasic) ... ok test_deepcopy (__main__.TestBasic) ... ok test_extend (__main__.TestBasic) ... ok test_extendleft (__main__.TestBasic) ... ok test_getitem (__main__.TestBasic) ... ok test_hash (__main__.TestBasic) ... ok test_len (__main__.TestBasic) ... ok test_long_steadystate_queue_popleft (__main__.TestBasic) ... ok test_long_steadystate_queue_popright (__main__.TestBasic) ... ok test_pickle (__main__.TestBasic) ... ok test_print (__main__.TestBasic) ... ok test_repr (__main__.TestBasic) ... ok test_reversed (__main__.TestBasic) ... ok test_rotate (__main__.TestBasic) ... ok test_roundtrip_iter_init (__main__.TestBasic) ... ok test_setitem (__main__.TestBasic) ... ok test_underflow (__main__.TestBasic) ... ok test_constructor (__main__.TestVariousIteratorArgs) ... ok test_iter_with_altered_data (__main__.TestVariousIteratorArgs) ... ok test_basics (__main__.TestSubclass) ... ok test_copy_pickle (__main__.TestSubclass) ... ok ---------------------------------------------------------------------- Ran 28 tests in 1.127s OK ***************************************************************** Failure in example: def roundrobin(*iterables): pending = deque(iter(i) for i in iterables) while pending: task = pending.popleft() try: yield task.next() except StopIteration: continue pending.append(task) from line #66 of test.test_deque.__test__.libreftest Exception raised: Traceback (most recent call last): File "/home/ncoghlan/python/src/python/dist/src/Lib/doctest.py", line 442, in _run_examples_inner compileflags, 1) in globs File "", line 2 pending = deque(iter(i) for i in iterables) ^ SyntaxError: invalid syntax ***************************************************************** Failure in example: for value in roundrobin('abc', 'd', 'efgh'): print value from line #77 of test.test_deque.__test__.libreftest Exception raised: Traceback (most recent call last): File "/home/ncoghlan/python/src/python/dist/src/Lib/doctest.py", line 442, in _run_examples_inner compileflags, 1) in globs File "", line 1, in ? NameError: name 'roundrobin' is not defined ***************************************************************** 1 items had failures: 2 of 32 in test.test_deque.__test__.libreftest ***Test Failed*** 2 failures. Traceback (most recent call last): File "./Lib/test/test_deque.py", line 561, in ? test_main(verbose=True) File "./Lib/test/test_deque.py", line 558, in test_main test_support.run_doctest(test_deque, verbose) File "/home/ncoghlan/python/src/python/dist/src/Lib/test/test_support.py", line 318, in run_doctest raise TestFailed("%d of %d doctests failed" % (f, t)) test.test_support.TestFailed: 2 of 32 doctests failed [ncoghlan@localhost src]$ ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-10 09:32 Message: Logged In: YES user_id=80475 Temporarily disabled the test. Pending this report to remind me to put it back in later. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=951270&group_id=5470 From noreply at sourceforge.net Wed May 19 05:38:32 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 05:38:44 2004 Subject: [ python-Bugs-924301 ] A leak case with cmd.py & readline & exception Message-ID: Bugs item #924301, was opened at 2004-03-27 00:28 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924301&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Sverker Nilsson (svenil) Assigned to: Michael Hudson (mwh) Summary: A leak case with cmd.py & readline & exception Initial Comment: A leak to do with readline & cmd, in Python 2.3. I found out what hold on to my interactive objects too long ('for ever') in certain circumstances. The circumstance had to do with an exception being raised in Cmd.cmdloop and handled (or not handled) outside of Cmd.cmdloop. In cmd.py, class Cmd, in cmdloop(), if an exception is raised and propagated out from the interior of cmdloop, the function postloop() is not called. The default function of this, (in 2.3) when the readline library is present, is to restore the completer, via: readline.set_completer(self.old_completer) If this is not done, the newly (by preloop) inserted completer will remain. Even if the loop is called again and run without exception, the new completer will remain, because then in postloop the old completer will be set to our new completer. When we exit, the completer will remain the one we set. This will hold on to our object, aka 'leak'. - In cmd.py in 2.2 no attempt was made to restore the completer, so this was also a kind of leak, but it was replaced the next time a Cmd instance was initialized. Now, however, the next time we will not replace the old completer, but both of them will remain in memory. The old one will be stored as self.old_completer. If we terminate with an exception, bad luck... the stored completer retains both of the instances. If we terminate normally, the old one will be retained. In no case do we restore the space of the first instance. The only way that would happen, would be if the second instance first exited the loop with an exception, and then entered the loop again an exited normally. But then, the second instance is retained instead! If each instance happens to terminate with an exception, it seems well possible that an ever increasing chain of leaking instances will be accumulated. My fix is to always call the postloop, given the preloop succeeded. This is done via a try:finally clause. def cmdloop(self, intro=None): ... self.preloop() try: ... finally: # Make sure postloop called self.postloop() I am attaching my patched version of cmd.py. It was originally from the tarball of Python 2.3.3 downloaded from Python.org some month or so ago in which cmd.py had this size & date: 14504 Feb 19 2003 cmd.py Best regards, Sverker Nilsson ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-19 10:38 Message: Logged In: YES user_id=6656 This is where I go "I wish I'd reviewed that patch more carefully". In particular, the documentation of {pre,post}loop is now out of date. I wonder setting/getting the completer in these functions was a good idea. Hmm. This bug report confuses me :-) but I can certainly see the intent of the patch... ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-19 02:52 Message: Logged In: YES user_id=80475 Michael, this touches some of your code. Do you want to handle this one? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924301&group_id=5470 From noreply at sourceforge.net Wed May 19 06:31:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 06:31:15 2004 Subject: [ python-Bugs-956303 ] Potential data corruption bug in save_pers in pickle module. Message-ID: Bugs item #956303, was opened at 2004-05-18 22:45 Message generated for change (Comment added) made by smst You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956303&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 7 Submitted By: Allan Crooks (amc1) Assigned to: Nobody/Anonymous (nobody) Summary: Potential data corruption bug in save_pers in pickle module. Initial Comment: There is a bug in save_pers in both the pickle and cPickle modules in Python. It occurs when someone uses a Pickler instance which is using an ASCII protocol and also has persistent_id defined which can return a persistent reference that can contain newline characters in. The current implementation of save_pers in the pickle module is as follows: ---- def save_pers(self, pid): # Save a persistent id reference if self.bin: self.save(pid) self.write(BINPERSID) else: self.write(PERSID + str(pid) + '\n') ---- The else clause assumes that the 'pid' will not be a string which one or more newline characters. If the pickler pickles a persistent ID which has a newline in it, then an unpickler with a corresponding persistent_load method will incorrectly unpickle the data - usually interpreting the character after the newline as a marker indicating what type of data should be expected (usually resulting in an exception being raised when the remaining data is not in the format expected). I have attached an example file which illustrates in what circumstances the error occurs. Workarounds for this bug are: 1) Use binary mode for picklers. 2) Modify subclass implementations of save_pers to ensure that newlines are not returned for persistent ID's. Although you may assume in general that this bug would only occur on rare occasions (due to the unlikely situation where someone would implement persistent_id so that it would return a string with a newline character embedded), it may occur more frequently if the subclass implementation of persistent_id uses a string which has been constructed using the marshal module. This bug was discovered when our code implemented the persistent_id method, which was returning the marshalled format of a tuple which contained strings. It occurred when one or more of the strings had a length of ten characters - the marshalled format of that string contains the string's length, where the byte used to represent the number 10 is the same as the one which represents the newline character: >>> marshal.dumps('a' * 10) 's\n\x00\x00\x00aaaaaaaaaa' >>> chr(10) '\n' I have replicated this bug on Python 1.5.2 and Python 2.3b1, and I believe it is present on all 2.x versions of Python. Many thanks to SourceForge user (and fellow colleague) SMST who diagnosed the bug and provided the test cases attached. ---------------------------------------------------------------------- Comment By: Steve Tregidgo (smst) Date: 2004-05-19 10:31 Message: Logged In: YES user_id=42335 I'd overlooked that note in the documentation before, and in fact developed the opposite view on what was allowed by seeing that the binary pickle format happens to allow persistent IDs containing non-printable ASCII characters. Given that the non-binary format can represent strings (containing any character, printable or not) by escaping them, it seems unfortunate that the same escaping was not applied to the saving of persistent IDs. It might be helpful if the documentation indicated that the acceptance by the binary pickle format of strings without restriction is not to be relied upon, underlining the fact that printable ASCII is all that's allowed by the format. Personally I'd like to see the restriction on persistent IDs lifted in a future version of the pickle module, but I don't have a compelling reason for it (other than it seeming to be unnecessary). On the other hand, it seems to be a limitation which hasn't caused much grief (if any) over the years... perhaps such a change (albeit a minor one) in the specifications should be left until another protocol is introduced. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-19 03:02 Message: Logged In: YES user_id=31435 The only documentation is the "Pickling and unpickling external objects" section of the Library Reference Manual, which says: """ Such objects are referenced by a ``persistent id'', which is just an arbitrary string of printable ASCII characters. """ A newline is universally considered to be a control character, not a printable character (e.g., try isprint('\n') under your local C compiler). So this is functioning as designed and as documented. If you don't find the docs clear, we should call this a documentation bug. If you think the semantics should change to allow more than printable characters, then this should become a feature request, and more is needed to define exactly which characters should be allowed. The current implementation is correct for persistent ids that meet the documented requirement. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956303&group_id=5470 From noreply at sourceforge.net Wed May 19 06:57:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 06:57:46 2004 Subject: [ python-Bugs-955772 ] Nested generator terminates prematurely Message-ID: Bugs item #955772, was opened at 2004-05-18 13:02 Message generated for change (Comment added) made by ygale You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 5 Submitted By: Yitz Gale (ygale) Assigned to: Nobody/Anonymous (nobody) Summary: Nested generator terminates prematurely Initial Comment: def g(x, y): for i in x: for j in y: yield i, j r2 = (0, 1) [e for e in g(r2, g(r2, r2))] Expected result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)), (1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))] Actual result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1))] ---------------------------------------------------------------------- >Comment By: Yitz Gale (ygale) Date: 2004-05-19 13:57 Message: Logged In: YES user_id=1033539 Too bad. What exactly is the restriction? I didn't find anything in the docs. And things like this often do work and are useful. For example: def primes(): yield 2 for n in count(3): for p in primes(): if p > sqrt(n): yield n break if n % p == 0: break ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-18 14:24 Message: Logged In: YES user_id=6656 Um. I think the answer to this is "generators are not reiterable". ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-18 13:04 Message: Logged In: YES user_id=1033539 Trying again to get the indentation correct: def g(x, y): for i in x: for j in y: yield i, j ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 From noreply at sourceforge.net Wed May 19 06:59:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 06:59:57 2004 Subject: [ python-Bugs-955772 ] Nested generator terminates prematurely Message-ID: Bugs item #955772, was opened at 2004-05-18 11:02 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 5 Submitted By: Yitz Gale (ygale) Assigned to: Nobody/Anonymous (nobody) Summary: Nested generator terminates prematurely Initial Comment: def g(x, y): for i in x: for j in y: yield i, j r2 = (0, 1) [e for e in g(r2, g(r2, r2))] Expected result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)), (1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))] Actual result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1))] ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-19 11:59 Message: Logged In: YES user_id=6656 Well, it's impossible in general. You'd have to store any arguments the generator took somewhere too, wouldn't you? What about things like: def foo(aList): while aList: yield aList.pop() ? ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-19 11:57 Message: Logged In: YES user_id=1033539 Too bad. What exactly is the restriction? I didn't find anything in the docs. And things like this often do work and are useful. For example: def primes(): yield 2 for n in count(3): for p in primes(): if p > sqrt(n): yield n break if n % p == 0: break ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-18 12:24 Message: Logged In: YES user_id=6656 Um. I think the answer to this is "generators are not reiterable". ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-18 11:04 Message: Logged In: YES user_id=1033539 Trying again to get the indentation correct: def g(x, y): for i in x: for j in y: yield i, j ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 From noreply at sourceforge.net Wed May 19 07:59:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 07:59:23 2004 Subject: [ python-Bugs-955772 ] Nested generator terminates prematurely Message-ID: Bugs item #955772, was opened at 2004-05-18 13:02 Message generated for change (Comment added) made by ygale You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 5 Submitted By: Yitz Gale (ygale) Assigned to: Nobody/Anonymous (nobody) Summary: Nested generator terminates prematurely Initial Comment: def g(x, y): for i in x: for j in y: yield i, j r2 = (0, 1) [e for e in g(r2, g(r2, r2))] Expected result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)), (1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))] Actual result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1))] ---------------------------------------------------------------------- >Comment By: Yitz Gale (ygale) Date: 2004-05-19 14:59 Message: Logged In: YES user_id=1033539 Python functions can be called recursively in general. They know how to save their local namespace in a separate frame for each call. That includes arguments, since the arguments live in the local namespace of the function. Generator functions also seem to be supported, as my example shows. There is a restriction on a generator object that you may not call its next() method again while a previous call to next() is still running. But this is definitely not a case of that restriction - we have two separate generator instances, and each ought to have its own frame. If there is some other restriction, I think it ought to be documented. And if possible, it should raise an exception, like the other restriction. This smells like a bug to me, though. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-19 13:59 Message: Logged In: YES user_id=6656 Well, it's impossible in general. You'd have to store any arguments the generator took somewhere too, wouldn't you? What about things like: def foo(aList): while aList: yield aList.pop() ? ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-19 13:57 Message: Logged In: YES user_id=1033539 Too bad. What exactly is the restriction? I didn't find anything in the docs. And things like this often do work and are useful. For example: def primes(): yield 2 for n in count(3): for p in primes(): if p > sqrt(n): yield n break if n % p == 0: break ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-18 14:24 Message: Logged In: YES user_id=6656 Um. I think the answer to this is "generators are not reiterable". ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-18 13:04 Message: Logged In: YES user_id=1033539 Trying again to get the indentation correct: def g(x, y): for i in x: for j in y: yield i, j ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 From noreply at sourceforge.net Wed May 19 08:55:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 09:06:41 2004 Subject: [ python-Bugs-955772 ] Nested generator terminates prematurely Message-ID: Bugs item #955772, was opened at 2004-05-18 10:02 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 5 Submitted By: Yitz Gale (ygale) Assigned to: Nobody/Anonymous (nobody) Summary: Nested generator terminates prematurely Initial Comment: def g(x, y): for i in x: for j in y: yield i, j r2 = (0, 1) [e for e in g(r2, g(r2, r2))] Expected result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)), (1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))] Actual result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1))] ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-19 12:55 Message: Logged In: YES user_id=4771 Your issue is that you only create a total of two generator instances. The 'inner' one is immediately exhausted. Afterwards, this same instance is used again on other 'for' loops but this has no effect, as it has already been exhausted. The difference is the same as between r2 and iter(r2). If you do that: it = iter(r2) for x in it: print x for x in it: print x the second loop will be empty beause the first loop has exhausted the iterator. Generators are iterators (like it) and not sequences (like r2). Using the same iterator on several 'for' loops is useful, though (e.g. if the first loop can be interrupted with 'break'), so there is no way your code could raise an exception, short of saying that it is not allowed to call next() on already-exhausted iterators -- this would be too big a change. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-19 11:59 Message: Logged In: YES user_id=1033539 Python functions can be called recursively in general. They know how to save their local namespace in a separate frame for each call. That includes arguments, since the arguments live in the local namespace of the function. Generator functions also seem to be supported, as my example shows. There is a restriction on a generator object that you may not call its next() method again while a previous call to next() is still running. But this is definitely not a case of that restriction - we have two separate generator instances, and each ought to have its own frame. If there is some other restriction, I think it ought to be documented. And if possible, it should raise an exception, like the other restriction. This smells like a bug to me, though. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-19 10:59 Message: Logged In: YES user_id=6656 Well, it's impossible in general. You'd have to store any arguments the generator took somewhere too, wouldn't you? What about things like: def foo(aList): while aList: yield aList.pop() ? ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-19 10:57 Message: Logged In: YES user_id=1033539 Too bad. What exactly is the restriction? I didn't find anything in the docs. And things like this often do work and are useful. For example: def primes(): yield 2 for n in count(3): for p in primes(): if p > sqrt(n): yield n break if n % p == 0: break ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-18 11:24 Message: Logged In: YES user_id=6656 Um. I think the answer to this is "generators are not reiterable". ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-18 10:04 Message: Logged In: YES user_id=1033539 Trying again to get the indentation correct: def g(x, y): for i in x: for j in y: yield i, j ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 From noreply at sourceforge.net Wed May 19 11:10:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 11:13:57 2004 Subject: [ python-Bugs-942952 ] Weakness in tuple hash Message-ID: Bugs item #942952, was opened at 2004-04-27 11:41 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 6 Submitted By: Steve Tregidgo (smst) Assigned to: Nobody/Anonymous (nobody) Summary: Weakness in tuple hash Initial Comment: I've encountered some performance problems when constructing dictionaries with keys of a particular form, and have pinned the problem down to the hashing function. I've reproduced the effect in Python 1.5.2, Python 2.2 and Python 2.3.3. I came across this when loading a marshalled dictionary with about 40000 entries. Loading other dictionaries of this size has always been fast, but in this case I killed the interpreter after a few minutes of CPU thrashing. The performance problem was caused because every key in the dictionary had the same hash value. The problem is as follows: for hashable values X and Y, hash( (X, (X, Y)) ) == hash(Y). This is always true (except in the corner case where hash((X, Y)) is internally calculated to be -1 (the error value) and so -2 is forced as the return value). With data in this form where X varies more than Y (Y is constant, or chosen from relatively few values compared to X) chances of collision become high as X is effectively ignored. The hash algorithm for tuples starts with a seed value, then generates a new value for each item in the tuple by multiplying the iteration's starting value by a constant (keeping the lowest 32 bits) and XORing with the hash of the item. The final value is then XORed with the tuple's length. In Python (ignoring the careful business with -1): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) The tuple (X, Y) therefore has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash(Y) ^ 2 ...and the tuple (X, (X, Y)) has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash((X, Y)) ^ 2 The outer multiplication is repeated, and is XORed with itself (cancelling both of them). The XORed 2s cancel also, leaving just hash(Y). Note that this cancellation property also means that the same hash value is shared by (X, (X, (X, (X, Y)))), and (X, (X, (X, (X, (X, (X, Y)))))), and so on, and (X, Z, (X, Z, Y)) and so on. I realise that choosing a hash function is a difficult task, so it may be that the behaviour seen here is a tradeoff against other desireable properties -- I don't have the expertise to tell. My naive suggestion would be that an extra multiplication is necessary, which presumably has a performance impact (multiplication being more expensive than XOR) but would reduce the possibility of cancellation. On the other hand, perhaps this particular case is rare enough that it's not worth the effort. For my own application I'm fortunate in that I can probably rearrange the data structure to avoid this case. ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-19 15:10 Message: Logged In: YES user_id=4771 Tim: > Most simple changes to the current algorithm don't > do significantly better, and some do much worse. The simple change suggested by ygale (put the xor before the multiply) does a quite reasonable job in your test suite. The number of collisions drop from 66966 to 1466. It also solves the original problem with (x, (x,y)), which should not be ignored because it could naturally occur if someone is using d.items() to compute a hash out of a dict built by d[x] = x,y. With this change it seems more difficult to find examples of regular families of tuples that all turn out to have the same hash value. Such examples do exist: (x, (1684537936,x)) yields 2000004 for any x. But the value 1684537936 has been carefully constructed for this purpose, it's the only number to misbehave like this :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-18 02:14 Message: Logged In: YES user_id=31435 Nothing wrong with being "better than random" -- it's not the purpose of Python's hash() to randomize, but to minimize hash collisions. That's why, e.g., hash(int) returns int unchanged. Then when indexing a dict by any contiguous range of ints (excepting -1), there are no collisions at all. That's a good thing. There's nothing in the structure of my example function geared toward the specific test instances used. It *should* do a good job of "spreading out" inputs that are "close together", and that's an important case, and the test inputs have a lot of that. It's a good thing that it avoids all collisions across them. About speed, the only answer is to time both, and across several major platforms. An extra mult may or may not cost more than blowing extra cache lines to suck up a table of ints. About the table of ints, it's always a dubious idea to multiply by an even number. Because the high bits of the product are thrown away, multiplying by an even number throws away information needlessly (it throws away as many useful bits as there are trailing zero bits in the multiplier). odd_number * 69069**i is always odd, so the multiplier in the table-free way is never even (or, worse, divisible by 4, 8, 16, etc). ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-17 13:39 Message: Logged In: YES user_id=1033539 Here is the Python code for the hash function that I am describing: def hx(x): if isinstance(x, tuple): result = 0x345678L for n, elt in enumerate(x): y = (tbl[n & 0x3f] + (n >> 6)) & 0xffffffffL result = (y * (result ^ hx(elt))) & 0xffffffffL result ^= len(x) else: result = hash(x) return result # 64 constants taken from MD5 # (see Modules/md5c.c in the Python sources) tbl = ( -0x28955b88, -0x173848aa, 0x242070db, -0x3e423112, -0xa83f051, 0x4787c62a, -0x57cfb9ed, -0x2b96aff, 0x698098d8, -0x74bb0851, -0xa44f, -0x76a32842, 0x6b901122, -0x2678e6d, -0x5986bc72, 0x49b40821, -0x9e1da9e, -0x3fbf4cc0, 0x265e5a51, -0x16493856, -0x29d0efa3, 0x2441453, -0x275e197f, -0x182c0438, 0x21e1cde6, -0x3cc8f82a, -0xb2af279, 0x455a14ed, -0x561c16fb, -0x3105c08, 0x676f02d9, -0x72d5b376, -0x5c6be, -0x788e097f, 0x6d9d6122, -0x21ac7f4, -0x5b4115bc, 0x4bdecfa9, -0x944b4a0, -0x41404390, 0x289b7ec6, -0x155ed806, -0x2b10cf7b, 0x4881d05, -0x262b2fc7, -0x1924661b, 0x1fa27cf8, -0x3b53a99b, -0xbd6ddbc, 0x432aff97, -0x546bdc59, -0x36c5fc7, 0x655b59c3, -0x70f3336e, -0x100b83, -0x7a7ba22f, 0x6fa87e4f, -0x1d31920, -0x5cfebcec, 0x4e0811a1, -0x8ac817e, -0x42c50dcb, 0x2ad7d2bb, -0x14792c6f ) ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-17 13:09 Message: Logged In: YES user_id=1033539 Tim, your test fixture is very nice. I verified your result that there are no collisions over the fixture set for the algorithm you specified. Based on that result, I would recommend that we NOT use your algorithm. The probability of no collisions is near zero for 250000 random samples taken from a set of 2^32 elements. So the result shows that your algorithm is far from a good hash function, though it is constructed to be great for that specific fixture set. I ran my algorithm on your fixture set using the table of 64 constants taken from MD5. I got 13 collisions, a reasonable result. It is not true that I repeat multipliers (very often) - note that I increment the elements of the table each time around, so the sequence repeats itself only after 2^38 tuple elements. Also, my algorithm runs at esentially the same speed as the current one: no additional multiplies, only a few adds and increments and such. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-16 21:54 Message: Logged In: YES user_id=31435 I can't make more time for this, so unassigned myself. This seems to make a good set of test inputs: N = whatever base = range(N) xp = [(i, j) for i in base for j in base] inps = base + [(i, j) for i in base for j in xp] + [(i, j) for i in xp for j in base] When N == 50, inps contains 250,050 distinct elements, and 66,966 of them generate a duplicated hash code. Most simple changes to the current algorithm don't do significantly better, and some do much worse. For example, just replacing the xor with an add zooms it to 119,666 duplicated hash codes across that set. Here's a hash function that yields no collisions across that set: def hx(x): if isinstance(x, tuple): result = 0x345678L mult = 1000003 for elt in x: y = hx(elt) result = (((result + y) & 0xffffffffL) * mult) & 0xffffffffL mult = (mult * 69069) & 0xffffffffL result ^= len(x) if result == -1: result = -2 else: result = hash(x) return result In C, none of the "& 0xffffffffL" clauses are needed; in Python code, they're needed to cut results back to 32 bits. 69069 is a well-known multiplier for a better-than-most pure multiplicative PRNG. This does add a multiply per loop trip over what we have today, but it can proceed in parallel with the existing multiply. Unlike a canned table, it won't repeat multipliers, (unless the tuple has more than a billion elements). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-16 21:09 Message: Logged In: YES user_id=31435 Noting that (X, (Y, Z)) and (Y, (X, Z)) hash to the same thing today too (module -1 endcases). For example, >>> hash((3, (4, 8))) == hash((4, (3, 8))) True >>> ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-16 12:25 Message: Logged In: YES user_id=1033539 Hmm, you are correct. This is appears to be an off-by-one problem: the original seed always gets multiplied by the constant (which is silly), and the last item in the tuple does not get multiplied (which causes the bug). The correct solution is to change: value = my_mul(const, value) ^ hash(item) in Steve's pseudo-code to: value = my_mul(const, value ^ hash(item)) Of course, you still get a lot more robustness for almost no cost if you vary "const" across the tuple via a table. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-03 13:40 Message: Logged In: YES user_id=764593 Note that in this case, the problem was because of nested tuples. X was the first element of both tuples, and both tuples had the same length -- so the X's would still have been multiplied by the same constant. (Not the same constant as Y, and hash(X, Y), but the same constant as each other.) A non-linear function might work, but would do bad things to other data. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-02 02:50 Message: Logged In: YES user_id=1033539 I suggest leaving the function as is, but replacing the constant with a value that varies as we step through the tuple. Take the values from a fixed table. When we reach the end of the table, start again from the beginning and mung the values slightly (e.g., increment them). True, there will always be the possibilities of collisions, but this will make it very unlikely except in very weird cases. Using a table of constants is a standard technique for hashes. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 17:45 Message: Logged In: YES user_id=764593 Wouldn't that just shift the pathological case from (x, (x, y)) to (x, (-x, y))? My point was that any hash function will act badly on *some* pattern of input, and if a pattern must be penalized, this might be a good pattern to choose. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 16:36 Message: Logged In: YES user_id=80475 The OP was not referring to "some collisions"; his app collapsed all entries to a single hash value. Changing XOR to + would partially eliminate the self cancelling property of this hash function. Also, I am concerned about the tuple hash using the same multiplier as the hash for other objects. In sets.py, a naive combination of the component hash values caused many distinct sets to collapse to a handful of possibilities -- while tuples do not have an identical issue, it does highlight the risks involved. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 14:50 Message: Logged In: YES user_id=764593 Any hash will have some collisions. If there is going to be predictably bad data, this is probably a good place to have it. The obvious alternatives are a more complicated hash (slows everything down), a different hash for embedded tuples (bad, since hash can't be cached then) or ignoring some elements when determining the hash (bad in the normal case of different data). I would also expect your workaround of data rearrangement to be sensible almost any time (X, (X, Y)) is really a common case. (The intuitive meaning for me is "X - then map X to Y", which could be done as (X, Y) or at least (X, (None, Y)), or perhaps d[X]=(X,Y).) ---------------------------------------------------------------------- Comment By: Steve Tregidgo (smst) Date: 2004-04-27 11:45 Message: Logged In: YES user_id=42335 I'll repeat the tuple hashing algorithm in a fixed-width font (with the first line following "for..." being the loop body): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 From noreply at sourceforge.net Wed May 19 11:30:02 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 11:30:09 2004 Subject: [ python-Bugs-956303 ] Potential data corruption bug in save_pers in pickle module. Message-ID: Bugs item #956303, was opened at 2004-05-18 22:45 Message generated for change (Comment added) made by amc1 You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956303&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 7 Submitted By: Allan Crooks (amc1) Assigned to: Nobody/Anonymous (nobody) Summary: Potential data corruption bug in save_pers in pickle module. Initial Comment: There is a bug in save_pers in both the pickle and cPickle modules in Python. It occurs when someone uses a Pickler instance which is using an ASCII protocol and also has persistent_id defined which can return a persistent reference that can contain newline characters in. The current implementation of save_pers in the pickle module is as follows: ---- def save_pers(self, pid): # Save a persistent id reference if self.bin: self.save(pid) self.write(BINPERSID) else: self.write(PERSID + str(pid) + '\n') ---- The else clause assumes that the 'pid' will not be a string which one or more newline characters. If the pickler pickles a persistent ID which has a newline in it, then an unpickler with a corresponding persistent_load method will incorrectly unpickle the data - usually interpreting the character after the newline as a marker indicating what type of data should be expected (usually resulting in an exception being raised when the remaining data is not in the format expected). I have attached an example file which illustrates in what circumstances the error occurs. Workarounds for this bug are: 1) Use binary mode for picklers. 2) Modify subclass implementations of save_pers to ensure that newlines are not returned for persistent ID's. Although you may assume in general that this bug would only occur on rare occasions (due to the unlikely situation where someone would implement persistent_id so that it would return a string with a newline character embedded), it may occur more frequently if the subclass implementation of persistent_id uses a string which has been constructed using the marshal module. This bug was discovered when our code implemented the persistent_id method, which was returning the marshalled format of a tuple which contained strings. It occurred when one or more of the strings had a length of ten characters - the marshalled format of that string contains the string's length, where the byte used to represent the number 10 is the same as the one which represents the newline character: >>> marshal.dumps('a' * 10) 's\n\x00\x00\x00aaaaaaaaaa' >>> chr(10) '\n' I have replicated this bug on Python 1.5.2 and Python 2.3b1, and I believe it is present on all 2.x versions of Python. Many thanks to SourceForge user (and fellow colleague) SMST who diagnosed the bug and provided the test cases attached. ---------------------------------------------------------------------- >Comment By: Allan Crooks (amc1) Date: 2004-05-19 15:30 Message: Logged In: YES user_id=39733 I would at least like the documentation modified to make it clearer that certain characters are not permitted for persistent ID's. I think the text which indicates the requirement of printable ASCII characters is too subtle - there should be something which makes the requirement more obvious, the use of a "must" or "should" would help get the point across (as would some text after the statement indicating that characters such as '\b', '\n', '\r' are not permitted). Perhaps it would be an idea for save_pers to do some argument checking before storing the persistent ID, perhaps using an assertion statement to verify that the ID doesn't contain non-permitted characters (or at least, checking for the presence of a '\n' character embedded in the string). I think it is preferable to have safeguards implemented in Pickler to prevent potentially dodgy data being stored - I would rather have an error raised when I'm trying to pickle something than have the data stored and corrupted, only to notice it when it is unpickled (when it is too late). Confusingly, the code in save_pers in the pickle module seems to indicate that it would happily accept non-String based persistent ID's: ---- else: self.write(PERSID + str(pid) + '\n') ---- I don't understand why we are using the str function if we are expecting pid to be a string in the first place. I would rather that this method would raise an error if it tried to perform string concatenation on something which isn't a string. I agree with SMST, I would like the restriction removed over what persistent ID's we can use, it seems somewhat arbitary - there's no reason, for example, why we can't use any simple data type which can be marshalled as an ID. Apart from the reason that it wouldn't be backwardly compatible, which is probably a good enough reason. :) ---------------------------------------------------------------------- Comment By: Steve Tregidgo (smst) Date: 2004-05-19 10:31 Message: Logged In: YES user_id=42335 I'd overlooked that note in the documentation before, and in fact developed the opposite view on what was allowed by seeing that the binary pickle format happens to allow persistent IDs containing non-printable ASCII characters. Given that the non-binary format can represent strings (containing any character, printable or not) by escaping them, it seems unfortunate that the same escaping was not applied to the saving of persistent IDs. It might be helpful if the documentation indicated that the acceptance by the binary pickle format of strings without restriction is not to be relied upon, underlining the fact that printable ASCII is all that's allowed by the format. Personally I'd like to see the restriction on persistent IDs lifted in a future version of the pickle module, but I don't have a compelling reason for it (other than it seeming to be unnecessary). On the other hand, it seems to be a limitation which hasn't caused much grief (if any) over the years... perhaps such a change (albeit a minor one) in the specifications should be left until another protocol is introduced. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-19 03:02 Message: Logged In: YES user_id=31435 The only documentation is the "Pickling and unpickling external objects" section of the Library Reference Manual, which says: """ Such objects are referenced by a ``persistent id'', which is just an arbitrary string of printable ASCII characters. """ A newline is universally considered to be a control character, not a printable character (e.g., try isprint('\n') under your local C compiler). So this is functioning as designed and as documented. If you don't find the docs clear, we should call this a documentation bug. If you think the semantics should change to allow more than printable characters, then this should become a feature request, and more is needed to define exactly which characters should be allowed. The current implementation is correct for persistent ids that meet the documented requirement. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=956303&group_id=5470 From noreply at sourceforge.net Wed May 19 12:15:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 12:15:52 2004 Subject: [ python-Bugs-942952 ] Weakness in tuple hash Message-ID: Bugs item #942952, was opened at 2004-04-27 07:41 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 6 Submitted By: Steve Tregidgo (smst) Assigned to: Nobody/Anonymous (nobody) Summary: Weakness in tuple hash Initial Comment: I've encountered some performance problems when constructing dictionaries with keys of a particular form, and have pinned the problem down to the hashing function. I've reproduced the effect in Python 1.5.2, Python 2.2 and Python 2.3.3. I came across this when loading a marshalled dictionary with about 40000 entries. Loading other dictionaries of this size has always been fast, but in this case I killed the interpreter after a few minutes of CPU thrashing. The performance problem was caused because every key in the dictionary had the same hash value. The problem is as follows: for hashable values X and Y, hash( (X, (X, Y)) ) == hash(Y). This is always true (except in the corner case where hash((X, Y)) is internally calculated to be -1 (the error value) and so -2 is forced as the return value). With data in this form where X varies more than Y (Y is constant, or chosen from relatively few values compared to X) chances of collision become high as X is effectively ignored. The hash algorithm for tuples starts with a seed value, then generates a new value for each item in the tuple by multiplying the iteration's starting value by a constant (keeping the lowest 32 bits) and XORing with the hash of the item. The final value is then XORed with the tuple's length. In Python (ignoring the careful business with -1): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) The tuple (X, Y) therefore has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash(Y) ^ 2 ...and the tuple (X, (X, Y)) has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash((X, Y)) ^ 2 The outer multiplication is repeated, and is XORed with itself (cancelling both of them). The XORed 2s cancel also, leaving just hash(Y). Note that this cancellation property also means that the same hash value is shared by (X, (X, (X, (X, Y)))), and (X, (X, (X, (X, (X, (X, Y)))))), and so on, and (X, Z, (X, Z, Y)) and so on. I realise that choosing a hash function is a difficult task, so it may be that the behaviour seen here is a tradeoff against other desireable properties -- I don't have the expertise to tell. My naive suggestion would be that an extra multiplication is necessary, which presumably has a performance impact (multiplication being more expensive than XOR) but would reduce the possibility of cancellation. On the other hand, perhaps this particular case is rare enough that it's not worth the effort. For my own application I'm fortunate in that I can probably rearrange the data structure to avoid this case. ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-19 12:15 Message: Logged In: YES user_id=31435 If 250,050 balls are tossed into 2**32 bins at random, the mean number of occupied bins is ~250042.7 (== expected number of collisions is ~7.3), with std deviation ~2.7. While we're happy to get "better than random" distributions, "astronomically worse than random" distributions usually predict more of the same kind of problem reported later. That said, I certainly agree that would be a major, easy, and cheap improvement over the status quo. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-19 11:10 Message: Logged In: YES user_id=4771 Tim: > Most simple changes to the current algorithm don't > do significantly better, and some do much worse. The simple change suggested by ygale (put the xor before the multiply) does a quite reasonable job in your test suite. The number of collisions drop from 66966 to 1466. It also solves the original problem with (x, (x,y)), which should not be ignored because it could naturally occur if someone is using d.items() to compute a hash out of a dict built by d[x] = x,y. With this change it seems more difficult to find examples of regular families of tuples that all turn out to have the same hash value. Such examples do exist: (x, (1684537936,x)) yields 2000004 for any x. But the value 1684537936 has been carefully constructed for this purpose, it's the only number to misbehave like this :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-17 22:14 Message: Logged In: YES user_id=31435 Nothing wrong with being "better than random" -- it's not the purpose of Python's hash() to randomize, but to minimize hash collisions. That's why, e.g., hash(int) returns int unchanged. Then when indexing a dict by any contiguous range of ints (excepting -1), there are no collisions at all. That's a good thing. There's nothing in the structure of my example function geared toward the specific test instances used. It *should* do a good job of "spreading out" inputs that are "close together", and that's an important case, and the test inputs have a lot of that. It's a good thing that it avoids all collisions across them. About speed, the only answer is to time both, and across several major platforms. An extra mult may or may not cost more than blowing extra cache lines to suck up a table of ints. About the table of ints, it's always a dubious idea to multiply by an even number. Because the high bits of the product are thrown away, multiplying by an even number throws away information needlessly (it throws away as many useful bits as there are trailing zero bits in the multiplier). odd_number * 69069**i is always odd, so the multiplier in the table-free way is never even (or, worse, divisible by 4, 8, 16, etc). ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-17 09:39 Message: Logged In: YES user_id=1033539 Here is the Python code for the hash function that I am describing: def hx(x): if isinstance(x, tuple): result = 0x345678L for n, elt in enumerate(x): y = (tbl[n & 0x3f] + (n >> 6)) & 0xffffffffL result = (y * (result ^ hx(elt))) & 0xffffffffL result ^= len(x) else: result = hash(x) return result # 64 constants taken from MD5 # (see Modules/md5c.c in the Python sources) tbl = ( -0x28955b88, -0x173848aa, 0x242070db, -0x3e423112, -0xa83f051, 0x4787c62a, -0x57cfb9ed, -0x2b96aff, 0x698098d8, -0x74bb0851, -0xa44f, -0x76a32842, 0x6b901122, -0x2678e6d, -0x5986bc72, 0x49b40821, -0x9e1da9e, -0x3fbf4cc0, 0x265e5a51, -0x16493856, -0x29d0efa3, 0x2441453, -0x275e197f, -0x182c0438, 0x21e1cde6, -0x3cc8f82a, -0xb2af279, 0x455a14ed, -0x561c16fb, -0x3105c08, 0x676f02d9, -0x72d5b376, -0x5c6be, -0x788e097f, 0x6d9d6122, -0x21ac7f4, -0x5b4115bc, 0x4bdecfa9, -0x944b4a0, -0x41404390, 0x289b7ec6, -0x155ed806, -0x2b10cf7b, 0x4881d05, -0x262b2fc7, -0x1924661b, 0x1fa27cf8, -0x3b53a99b, -0xbd6ddbc, 0x432aff97, -0x546bdc59, -0x36c5fc7, 0x655b59c3, -0x70f3336e, -0x100b83, -0x7a7ba22f, 0x6fa87e4f, -0x1d31920, -0x5cfebcec, 0x4e0811a1, -0x8ac817e, -0x42c50dcb, 0x2ad7d2bb, -0x14792c6f ) ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-17 09:09 Message: Logged In: YES user_id=1033539 Tim, your test fixture is very nice. I verified your result that there are no collisions over the fixture set for the algorithm you specified. Based on that result, I would recommend that we NOT use your algorithm. The probability of no collisions is near zero for 250000 random samples taken from a set of 2^32 elements. So the result shows that your algorithm is far from a good hash function, though it is constructed to be great for that specific fixture set. I ran my algorithm on your fixture set using the table of 64 constants taken from MD5. I got 13 collisions, a reasonable result. It is not true that I repeat multipliers (very often) - note that I increment the elements of the table each time around, so the sequence repeats itself only after 2^38 tuple elements. Also, my algorithm runs at esentially the same speed as the current one: no additional multiplies, only a few adds and increments and such. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-16 17:54 Message: Logged In: YES user_id=31435 I can't make more time for this, so unassigned myself. This seems to make a good set of test inputs: N = whatever base = range(N) xp = [(i, j) for i in base for j in base] inps = base + [(i, j) for i in base for j in xp] + [(i, j) for i in xp for j in base] When N == 50, inps contains 250,050 distinct elements, and 66,966 of them generate a duplicated hash code. Most simple changes to the current algorithm don't do significantly better, and some do much worse. For example, just replacing the xor with an add zooms it to 119,666 duplicated hash codes across that set. Here's a hash function that yields no collisions across that set: def hx(x): if isinstance(x, tuple): result = 0x345678L mult = 1000003 for elt in x: y = hx(elt) result = (((result + y) & 0xffffffffL) * mult) & 0xffffffffL mult = (mult * 69069) & 0xffffffffL result ^= len(x) if result == -1: result = -2 else: result = hash(x) return result In C, none of the "& 0xffffffffL" clauses are needed; in Python code, they're needed to cut results back to 32 bits. 69069 is a well-known multiplier for a better-than-most pure multiplicative PRNG. This does add a multiply per loop trip over what we have today, but it can proceed in parallel with the existing multiply. Unlike a canned table, it won't repeat multipliers, (unless the tuple has more than a billion elements). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-16 17:09 Message: Logged In: YES user_id=31435 Noting that (X, (Y, Z)) and (Y, (X, Z)) hash to the same thing today too (module -1 endcases). For example, >>> hash((3, (4, 8))) == hash((4, (3, 8))) True >>> ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-16 08:25 Message: Logged In: YES user_id=1033539 Hmm, you are correct. This is appears to be an off-by-one problem: the original seed always gets multiplied by the constant (which is silly), and the last item in the tuple does not get multiplied (which causes the bug). The correct solution is to change: value = my_mul(const, value) ^ hash(item) in Steve's pseudo-code to: value = my_mul(const, value ^ hash(item)) Of course, you still get a lot more robustness for almost no cost if you vary "const" across the tuple via a table. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-03 09:40 Message: Logged In: YES user_id=764593 Note that in this case, the problem was because of nested tuples. X was the first element of both tuples, and both tuples had the same length -- so the X's would still have been multiplied by the same constant. (Not the same constant as Y, and hash(X, Y), but the same constant as each other.) A non-linear function might work, but would do bad things to other data. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-01 22:50 Message: Logged In: YES user_id=1033539 I suggest leaving the function as is, but replacing the constant with a value that varies as we step through the tuple. Take the values from a fixed table. When we reach the end of the table, start again from the beginning and mung the values slightly (e.g., increment them). True, there will always be the possibilities of collisions, but this will make it very unlikely except in very weird cases. Using a table of constants is a standard technique for hashes. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 13:45 Message: Logged In: YES user_id=764593 Wouldn't that just shift the pathological case from (x, (x, y)) to (x, (-x, y))? My point was that any hash function will act badly on *some* pattern of input, and if a pattern must be penalized, this might be a good pattern to choose. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 12:36 Message: Logged In: YES user_id=80475 The OP was not referring to "some collisions"; his app collapsed all entries to a single hash value. Changing XOR to + would partially eliminate the self cancelling property of this hash function. Also, I am concerned about the tuple hash using the same multiplier as the hash for other objects. In sets.py, a naive combination of the component hash values caused many distinct sets to collapse to a handful of possibilities -- while tuples do not have an identical issue, it does highlight the risks involved. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 10:50 Message: Logged In: YES user_id=764593 Any hash will have some collisions. If there is going to be predictably bad data, this is probably a good place to have it. The obvious alternatives are a more complicated hash (slows everything down), a different hash for embedded tuples (bad, since hash can't be cached then) or ignoring some elements when determining the hash (bad in the normal case of different data). I would also expect your workaround of data rearrangement to be sensible almost any time (X, (X, Y)) is really a common case. (The intuitive meaning for me is "X - then map X to Y", which could be done as (X, Y) or at least (X, (None, Y)), or perhaps d[X]=(X,Y).) ---------------------------------------------------------------------- Comment By: Steve Tregidgo (smst) Date: 2004-04-27 07:45 Message: Logged In: YES user_id=42335 I'll repeat the tuple hashing algorithm in a fixed-width font (with the first line following "for..." being the loop body): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 From noreply at sourceforge.net Wed May 19 12:36:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 12:36:17 2004 Subject: [ python-Bugs-955772 ] Nested generator terminates prematurely Message-ID: Bugs item #955772, was opened at 2004-05-18 05:02 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 Category: Python Interpreter Core Group: None >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: Yitz Gale (ygale) Assigned to: Nobody/Anonymous (nobody) Summary: Nested generator terminates prematurely Initial Comment: def g(x, y): for i in x: for j in y: yield i, j r2 = (0, 1) [e for e in g(r2, g(r2, r2))] Expected result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)), (1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))] Actual result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1))] ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-19 11:36 Message: Logged In: YES user_id=80475 Marking this as invalid and closing. Sorry, non-re-iterability is documented fact of life in the world of generators and iterators. The work arounds include making the inner generator into a list or re-instantiating a new generator on every loop: def g(x, y): for i in x for j in g(x, y) yield i, j ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-19 07:55 Message: Logged In: YES user_id=4771 Your issue is that you only create a total of two generator instances. The 'inner' one is immediately exhausted. Afterwards, this same instance is used again on other 'for' loops but this has no effect, as it has already been exhausted. The difference is the same as between r2 and iter(r2). If you do that: it = iter(r2) for x in it: print x for x in it: print x the second loop will be empty beause the first loop has exhausted the iterator. Generators are iterators (like it) and not sequences (like r2). Using the same iterator on several 'for' loops is useful, though (e.g. if the first loop can be interrupted with 'break'), so there is no way your code could raise an exception, short of saying that it is not allowed to call next() on already-exhausted iterators -- this would be too big a change. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-19 06:59 Message: Logged In: YES user_id=1033539 Python functions can be called recursively in general. They know how to save their local namespace in a separate frame for each call. That includes arguments, since the arguments live in the local namespace of the function. Generator functions also seem to be supported, as my example shows. There is a restriction on a generator object that you may not call its next() method again while a previous call to next() is still running. But this is definitely not a case of that restriction - we have two separate generator instances, and each ought to have its own frame. If there is some other restriction, I think it ought to be documented. And if possible, it should raise an exception, like the other restriction. This smells like a bug to me, though. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-19 05:59 Message: Logged In: YES user_id=6656 Well, it's impossible in general. You'd have to store any arguments the generator took somewhere too, wouldn't you? What about things like: def foo(aList): while aList: yield aList.pop() ? ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-19 05:57 Message: Logged In: YES user_id=1033539 Too bad. What exactly is the restriction? I didn't find anything in the docs. And things like this often do work and are useful. For example: def primes(): yield 2 for n in count(3): for p in primes(): if p > sqrt(n): yield n break if n % p == 0: break ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-18 06:24 Message: Logged In: YES user_id=6656 Um. I think the answer to this is "generators are not reiterable". ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-18 05:04 Message: Logged In: YES user_id=1033539 Trying again to get the indentation correct: def g(x, y): for i in x: for j in y: yield i, j ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 From noreply at sourceforge.net Wed May 19 14:21:35 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 14:22:00 2004 Subject: [ python-Bugs-215126 ] Over restricted type checking on eval() function Message-ID: Bugs item #215126, was opened at 2000-09-22 19:36 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=215126&group_id=5470 Category: Python Interpreter Core Group: Feature Request Status: Open Resolution: Later Priority: 5 Submitted By: Nobody/Anonymous (nobody) Assigned to: Raymond Hettinger (rhettinger) Summary: Over restricted type checking on eval() function Initial Comment: The built-in function eval() takes a string argument and a dictionary. The second argument should allow any instance which defines __getitem__ as opposed to just dictionaries. The following example creates a type error: eval, argument 2: expected dictionary, instance found class SpreadSheet: _cells = {} def __setitem__( self, key, formula ): self._cells[key] = formula def __getitem__( self, key ): return eval( self._cells[key], self ) ss = SpreadSheet() ss['a1'] = '5' ss['a2'] = 'a1*5' ss['a2'] ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-19 18:21 Message: Logged In: YES user_id=4771 With minimal work and performance impact, we could allow frame->f_locals to be something else than a real dictionary. It looks like it would be easily possible as long as we don't touch the time-critical frame->f_globals. Code compiled by eval() always uses LOAD_NAME and STORE_NAME, which is anyway allowed to be a bit slower than LOAD_FAST or LOAD_GLOBAL. Note that PyDict_GetItem() & co., as called by LOAD/STORE_NAME, do a PyDict_Check() anyway. We could just replace it with PyDict_CheckExact() and if it fails fall back to calling the appropriate ob_type->tp_as_mapping method. This would turn some of the PyDict_Xxx() API into a general mapping API, half-way between its current dict-only usage and the fully abstract PyObject_Xxx() API. This is maybe a bit strange from the user's point of view, because eval("expr", mydict) still wouldn't work: only eval("expr", {}, mydict) would. which does which does Now, there is no way I can think of that code compiled by eval() could contain LOAD_GLOBAL or STORE_GLOBAL. The only way to tell the difference between eval("expr", mydict) and eval("expr", {}, mydict) appears to be by calling globals() or somehow inspecting the frame directly. Therefore it might be acceptable to redefine the two-argument eval(expr, dict) to be equivalent not to eval(expr, dict, dict) but eval(expr, {}, dict). This hack might be useful enough even if it won't work with the exec statement. ---------------------------------------------------------------------- Comment By: Philippe Fremy (pfremy) Date: 2004-03-03 09:40 Message: Logged In: YES user_id=233844 I have exactly the same need as the original poster. The only difference in my case is that I inhreted a dictionnary. I want to use the eval() function as a simple expression evaluator. I have the follwing dictionnary: d['a']='1' d['b']='2' d['c']='a+b' I want the following results: d[a] -> 1 d[b] -> 2 d[c] -> 3 To do that, I was planning to use the eval() function and overloading the __getitem__ of the global or local dictionnary: class MyDict( dict ) : def __getitem__( self, key ): print "__getitem__", key val = dict.__getitem__( self, key ) print "val = '%s'" % val return eval( val , self ) But it does not work: d[a]: __getitem__ a val = '1' -> 1 d[b]: __getitem__ b val = '2' -> 2 d[c]: __getitem__ c val = 'e+1' ERROR Traceback (most recent call last): File "test_parse_jaycos_config.py", line 83, in testMyDict self.assertEquals( d['c'], 2 ) File "parse_config_file.py", line 10, in __getitem__ return eval( val , self ) File "", line 0, in ? TypeError: cannot concatenate 'str' and 'int' objects d['c'] did fetch the 'a+1' value, which was passed to eval. However, eval() tried to evaluate the expression using the content of the dictionnary instead of using __getitem__. So it fetched the string '1' for 'a' instead of the value 1, so the result is not suitable for the addition. ---------------------------------------------------------------------- Comment By: Michael Chermside (mcherm) Date: 2004-01-12 22:01 Message: Logged In: YES user_id=99874 Hmm... I like this! Of course, I am wary of adding *yet another* special double- underscore function to satisfy a single special purpose, but this would satisfy all of *my* needs, and without any performance impact for lookups that are FOUND. Lookups that are NOT found would have a slight performance degrade (I know better than to speculate about the size of the effect without measuring it). I'm not really sure what percentage of dict lookups succeed. At any rate, what are these "other contexts" you mention in which a __keyerror__ would "also be useful"? Because if we can find other places where it is useful, that significantly improves the usefulness. OTOH, if the tests can be done ONLY for eval (I don't really think that's possible), then I'm certain no one cares about the performance of eval. ---------------------------------------------------------------------- Comment By: Gregory Smith (gregsmith) Date: 2004-01-12 19:38 Message: Logged In: YES user_id=292741 This may be a compromise solution, which could also be useful in other contexts: What if the object passed is derived from dict - presumably that doesn't help because the point is to do low-level calls to the actual dict's lookup functions. Now, suppose we modify the basic dict type, so that before throwing a KeyError, it checks to see if it is really a derived object with a method __keyerror__, and if so, calls that and returns its result (or lets it throw)? Now you can make objects that look like dicts, and act like them at the low level, but can automatically populate themselves when non-existent keys are requested. Of course, __keyerror__('x') does not have to make an 'x' entry in the dict; it could make no change, or add several entries, depending on the desired semantics regarding future lookups. It could be set up so that every lookup fails and is forwarded by __keyerror__ to the __getitem__ of another object, for instance. The cost of this to the 'normal' dict lookup is that the need to do PyDict_CheckExact() each time a lookup fails. ---------------------------------------------------------------------- Comment By: Michael Chermside (mcherm) Date: 2003-12-16 15:37 Message: Logged In: YES user_id=99874 I'll just add my voice as somebody who would find this to be "darn handy" if it could ever done without noticably impacting the speed of python code that DOESN'T use eval(). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2000-09-23 04:18 Message: Added to PEP 42. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2000-09-22 19:42 Message: Changed Group to Feature Request. Should be added to PEP 42 (I'll do that if nobody beats me to it). CPython requires a genuine dict now for speed. I believe JPython allows any mapping object. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=215126&group_id=5470 From noreply at sourceforge.net Wed May 19 14:38:39 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 14:38:52 2004 Subject: [ python-Feature Requests-920680 ] readline not implemented for UTF-16 Message-ID: Feature Requests item #920680, was opened at 2004-03-21 17:37 Message generated for change (Comment added) made by etrepum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 3 Submitted By: Bob Ippolito (etrepum) Assigned to: M.-A. Lemburg (lemburg) Summary: readline not implemented for UTF-16 Initial Comment: The StreamReader for UTF-16 (all three of them) doesn't implement readline. ---------------------------------------------------------------------- >Comment By: Bob Ippolito (etrepum) Date: 2004-05-19 14:38 Message: Logged In: YES user_id=139309 Attaching a revised monkeypatch: * splitlines is used (I wasn't aware of the other unicode EOL markers) * 256 bytes is the new default buffer size Why do you want sized and unsized to be in the same function? They're both dispatched from readline as appropriate, and they are very different code paths. It would be much uglier as one function, so I'm not going to do it in my own code. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-19 04:19 Message: Logged In: YES user_id=38388 Thanks for the patch. Some comments: * Unicode has a lot more line-end markers than just LF; you should use .splitlines() to break lines at all of them * please collapse both methods (sized + unsized) into one method and default to 256 bytes for the buffer size ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-05-18 19:22 Message: Logged In: YES user_id=139309 I've attached a monkeypatch to get readline support for utf-16 codecs.. import utf16reader utf16reader.install() It can be trivially inserted into the utf16 encodings implementation.. it would be really cool if someone would audit the implementation and sneak it in before Python 2.4 :) ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-03-21 17:54 Message: Logged In: YES user_id=139309 I don't need it enough to write a patch, but this is what I used instead.. and it seems like it might work: try: for line in inFile: tline = translator(line) outFile.write(tline) except NotImplementedError: BUFFER = 16384 bytes = inFile.read(BUFFER) while bytes: lines = bytes.split(u'\n') bytes = lines.pop() for line in lines: tline = translator(line) outFile.write(tline) newbytes = inFile.read(BUFFER) bytes += newbytes if not newbytes and bytes: bytes += u'\n' ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-21 17:44 Message: Logged In: YES user_id=38388 Patches are welcome :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 From noreply at sourceforge.net Wed May 19 15:11:30 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 15:11:48 2004 Subject: [ python-Bugs-949329 ] stringprep.py: sets.Set() --> set() Message-ID: Bugs item #949329, was opened at 2004-05-06 16:51 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949329&group_id=5470 Category: Python Library Group: Python 2.4 >Status: Closed >Resolution: Accepted Priority: 3 Submitted By: Raymond Hettinger (rhettinger) Assigned to: Martin v. L?wis (loewis) Summary: stringprep.py: sets.Set() --> set() Initial Comment: Suggest replacing sets.Set() with set() in stringprep.py. The __contains__ test for the builtin runs several times faster. ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-19 19:11 Message: Logged In: YES user_id=4771 Checked in: Lib/stringprep.py: rev 1.3 Tools/unicode/mkstringprep.py: rev 1.2 ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-13 21:12 Message: Logged In: YES user_id=80475 On Windows with MSVC++ 6.0, I haven't been able to make a UCS-4 build. Can you apply this one for me. Thanks, Raymond ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-13 19:26 Message: Logged In: YES user_id=21627 The patch is fine, please apply, and regenerate stringprep.py. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-07 18:09 Message: Logged In: YES user_id=80475 Patch is attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=949329&group_id=5470 From noreply at sourceforge.net Wed May 19 15:56:08 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 15:56:13 2004 Subject: [ python-Bugs-957003 ] RFE: Extend smtplib.py with support for LMTP Message-ID: Bugs item #957003, was opened at 2004-05-19 12:56 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957003&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Leif Hedstrom (zwoop) Assigned to: Nobody/Anonymous (nobody) Summary: RFE: Extend smtplib.py with support for LMTP Initial Comment: Hi, attached is a proposal to extend the existing smtplib.py module with support for LMTP (RFC2033). I find it very useful together with IMAP servers like Cyrus. Thanks, -- leif ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957003&group_id=5470 From noreply at sourceforge.net Wed May 19 17:33:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 17:33:44 2004 Subject: [ python-Bugs-955772 ] Nested generator terminates prematurely Message-ID: Bugs item #955772, was opened at 2004-05-18 13:02 Message generated for change (Comment added) made by ygale You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 Category: Python Interpreter Core Group: None Status: Closed Resolution: Invalid Priority: 5 Submitted By: Yitz Gale (ygale) Assigned to: Nobody/Anonymous (nobody) Summary: Nested generator terminates prematurely Initial Comment: def g(x, y): for i in x: for j in y: yield i, j r2 = (0, 1) [e for e in g(r2, g(r2, r2))] Expected result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1)), (1, (0, 0)), (1, (0, 1)), (1, (1, 0)), (1, (1, 1))] Actual result: [(0, (0, 0)), (0, (0, 1)), (0, (1, 0)), (0, (1, 1))] ---------------------------------------------------------------------- >Comment By: Yitz Gale (ygale) Date: 2004-05-20 00:33 Message: Logged In: YES user_id=1033539 OK. I can get the semantics I want using the following: def g(x, y): for i in x: for j in y: yield i, j g = restartable(g) where I have defined: class restartable: def __init__(self, genfn): self.genfn = genfn def __call__(self, *args): return restartable_generator(self.genfn, *args) class restartable_generator: def __init__(self, genfn, *args): self.genfn = genfn self.args = args def __iter__(self): return self.genfn(*self.args) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-19 19:36 Message: Logged In: YES user_id=80475 Marking this as invalid and closing. Sorry, non-re-iterability is documented fact of life in the world of generators and iterators. The work arounds include making the inner generator into a list or re-instantiating a new generator on every loop: def g(x, y): for i in x for j in g(x, y) yield i, j ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-19 15:55 Message: Logged In: YES user_id=4771 Your issue is that you only create a total of two generator instances. The 'inner' one is immediately exhausted. Afterwards, this same instance is used again on other 'for' loops but this has no effect, as it has already been exhausted. The difference is the same as between r2 and iter(r2). If you do that: it = iter(r2) for x in it: print x for x in it: print x the second loop will be empty beause the first loop has exhausted the iterator. Generators are iterators (like it) and not sequences (like r2). Using the same iterator on several 'for' loops is useful, though (e.g. if the first loop can be interrupted with 'break'), so there is no way your code could raise an exception, short of saying that it is not allowed to call next() on already-exhausted iterators -- this would be too big a change. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-19 14:59 Message: Logged In: YES user_id=1033539 Python functions can be called recursively in general. They know how to save their local namespace in a separate frame for each call. That includes arguments, since the arguments live in the local namespace of the function. Generator functions also seem to be supported, as my example shows. There is a restriction on a generator object that you may not call its next() method again while a previous call to next() is still running. But this is definitely not a case of that restriction - we have two separate generator instances, and each ought to have its own frame. If there is some other restriction, I think it ought to be documented. And if possible, it should raise an exception, like the other restriction. This smells like a bug to me, though. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-19 13:59 Message: Logged In: YES user_id=6656 Well, it's impossible in general. You'd have to store any arguments the generator took somewhere too, wouldn't you? What about things like: def foo(aList): while aList: yield aList.pop() ? ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-19 13:57 Message: Logged In: YES user_id=1033539 Too bad. What exactly is the restriction? I didn't find anything in the docs. And things like this often do work and are useful. For example: def primes(): yield 2 for n in count(3): for p in primes(): if p > sqrt(n): yield n break if n % p == 0: break ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-18 14:24 Message: Logged In: YES user_id=6656 Um. I think the answer to this is "generators are not reiterable". ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-18 13:04 Message: Logged In: YES user_id=1033539 Trying again to get the indentation correct: def g(x, y): for i in x: for j in y: yield i, j ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=955772&group_id=5470 From noreply at sourceforge.net Wed May 19 19:10:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 19:10:48 2004 Subject: [ python-Feature Requests-920680 ] readline not implemented for UTF-16 Message-ID: Feature Requests item #920680, was opened at 2004-03-21 17:37 Message generated for change (Comment added) made by jimjjewett You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 3 Submitted By: Bob Ippolito (etrepum) Assigned to: M.-A. Lemburg (lemburg) Summary: readline not implemented for UTF-16 Initial Comment: The StreamReader for UTF-16 (all three of them) doesn't implement readline. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-19 19:10 Message: Logged In: YES user_id=764593 It might be just an upload/download quirk, but when I tried, this concatenated short lines. u"\n".join(...) worked better, but I'm not sure how that plays with other line breaks. It might work better to stick a class around the realine functions, so that self.buff can always be a (state-preserved) list; just return the first row, until the list length gets to one, then concatenate to that and resplit. ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-05-19 14:38 Message: Logged In: YES user_id=139309 Attaching a revised monkeypatch: * splitlines is used (I wasn't aware of the other unicode EOL markers) * 256 bytes is the new default buffer size Why do you want sized and unsized to be in the same function? They're both dispatched from readline as appropriate, and they are very different code paths. It would be much uglier as one function, so I'm not going to do it in my own code. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-19 04:19 Message: Logged In: YES user_id=38388 Thanks for the patch. Some comments: * Unicode has a lot more line-end markers than just LF; you should use .splitlines() to break lines at all of them * please collapse both methods (sized + unsized) into one method and default to 256 bytes for the buffer size ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-05-18 19:22 Message: Logged In: YES user_id=139309 I've attached a monkeypatch to get readline support for utf-16 codecs.. import utf16reader utf16reader.install() It can be trivially inserted into the utf16 encodings implementation.. it would be really cool if someone would audit the implementation and sneak it in before Python 2.4 :) ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-03-21 17:54 Message: Logged In: YES user_id=139309 I don't need it enough to write a patch, but this is what I used instead.. and it seems like it might work: try: for line in inFile: tline = translator(line) outFile.write(tline) except NotImplementedError: BUFFER = 16384 bytes = inFile.read(BUFFER) while bytes: lines = bytes.split(u'\n') bytes = lines.pop() for line in lines: tline = translator(line) outFile.write(tline) newbytes = inFile.read(BUFFER) bytes += newbytes if not newbytes and bytes: bytes += u'\n' ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-21 17:44 Message: Logged In: YES user_id=38388 Patches are welcome :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 From noreply at sourceforge.net Wed May 19 19:24:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 19:24:20 2004 Subject: [ python-Bugs-942952 ] Weakness in tuple hash Message-ID: Bugs item #942952, was opened at 2004-04-27 14:41 Message generated for change (Comment added) made by ygale You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 6 Submitted By: Steve Tregidgo (smst) Assigned to: Nobody/Anonymous (nobody) Summary: Weakness in tuple hash Initial Comment: I've encountered some performance problems when constructing dictionaries with keys of a particular form, and have pinned the problem down to the hashing function. I've reproduced the effect in Python 1.5.2, Python 2.2 and Python 2.3.3. I came across this when loading a marshalled dictionary with about 40000 entries. Loading other dictionaries of this size has always been fast, but in this case I killed the interpreter after a few minutes of CPU thrashing. The performance problem was caused because every key in the dictionary had the same hash value. The problem is as follows: for hashable values X and Y, hash( (X, (X, Y)) ) == hash(Y). This is always true (except in the corner case where hash((X, Y)) is internally calculated to be -1 (the error value) and so -2 is forced as the return value). With data in this form where X varies more than Y (Y is constant, or chosen from relatively few values compared to X) chances of collision become high as X is effectively ignored. The hash algorithm for tuples starts with a seed value, then generates a new value for each item in the tuple by multiplying the iteration's starting value by a constant (keeping the lowest 32 bits) and XORing with the hash of the item. The final value is then XORed with the tuple's length. In Python (ignoring the careful business with -1): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) The tuple (X, Y) therefore has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash(Y) ^ 2 ...and the tuple (X, (X, Y)) has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash((X, Y)) ^ 2 The outer multiplication is repeated, and is XORed with itself (cancelling both of them). The XORed 2s cancel also, leaving just hash(Y). Note that this cancellation property also means that the same hash value is shared by (X, (X, (X, (X, Y)))), and (X, (X, (X, (X, (X, (X, Y)))))), and so on, and (X, Z, (X, Z, Y)) and so on. I realise that choosing a hash function is a difficult task, so it may be that the behaviour seen here is a tradeoff against other desireable properties -- I don't have the expertise to tell. My naive suggestion would be that an extra multiplication is necessary, which presumably has a performance impact (multiplication being more expensive than XOR) but would reduce the possibility of cancellation. On the other hand, perhaps this particular case is rare enough that it's not worth the effort. For my own application I'm fortunate in that I can probably rearrange the data structure to avoid this case. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-20 02:24 Message: Logged In: YES user_id=1033539 I uploaded my current diff and some unit tests to patch #957122. Please look over the unit tests carefully. I am sure there is much more we can do. Note that besides the addition of the table containing the constants, only three lines of code are changed from the original function. By inspection, this code should run at essentially the same speed as the old code on all platforms (tests anyone?) Based on the results of the unit tests, I had to do some tweaking. First of all, Tim is correct: constants that are divisible by any small prime (e.g., 2) are a problem. To be on the safe side, I replaced each md5 constant with a nearby prime. I ran into a problem with empty tuples nested in tuples. The seed kept getting xored with itself. It turns out that no matter how nice the md5 constants are, they all produce the same value when multiplied by zero. Find "++x" in the code to see the fix for this. Finally, I moved the xor with the tuple length from the end of the calculation to the beginning. That amplifies the effect very nicely. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-19 19:15 Message: Logged In: YES user_id=31435 If 250,050 balls are tossed into 2**32 bins at random, the mean number of occupied bins is ~250042.7 (== expected number of collisions is ~7.3), with std deviation ~2.7. While we're happy to get "better than random" distributions, "astronomically worse than random" distributions usually predict more of the same kind of problem reported later. That said, I certainly agree that would be a major, easy, and cheap improvement over the status quo. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-19 18:10 Message: Logged In: YES user_id=4771 Tim: > Most simple changes to the current algorithm don't > do significantly better, and some do much worse. The simple change suggested by ygale (put the xor before the multiply) does a quite reasonable job in your test suite. The number of collisions drop from 66966 to 1466. It also solves the original problem with (x, (x,y)), which should not be ignored because it could naturally occur if someone is using d.items() to compute a hash out of a dict built by d[x] = x,y. With this change it seems more difficult to find examples of regular families of tuples that all turn out to have the same hash value. Such examples do exist: (x, (1684537936,x)) yields 2000004 for any x. But the value 1684537936 has been carefully constructed for this purpose, it's the only number to misbehave like this :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-18 05:14 Message: Logged In: YES user_id=31435 Nothing wrong with being "better than random" -- it's not the purpose of Python's hash() to randomize, but to minimize hash collisions. That's why, e.g., hash(int) returns int unchanged. Then when indexing a dict by any contiguous range of ints (excepting -1), there are no collisions at all. That's a good thing. There's nothing in the structure of my example function geared toward the specific test instances used. It *should* do a good job of "spreading out" inputs that are "close together", and that's an important case, and the test inputs have a lot of that. It's a good thing that it avoids all collisions across them. About speed, the only answer is to time both, and across several major platforms. An extra mult may or may not cost more than blowing extra cache lines to suck up a table of ints. About the table of ints, it's always a dubious idea to multiply by an even number. Because the high bits of the product are thrown away, multiplying by an even number throws away information needlessly (it throws away as many useful bits as there are trailing zero bits in the multiplier). odd_number * 69069**i is always odd, so the multiplier in the table-free way is never even (or, worse, divisible by 4, 8, 16, etc). ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-17 16:39 Message: Logged In: YES user_id=1033539 Here is the Python code for the hash function that I am describing: def hx(x): if isinstance(x, tuple): result = 0x345678L for n, elt in enumerate(x): y = (tbl[n & 0x3f] + (n >> 6)) & 0xffffffffL result = (y * (result ^ hx(elt))) & 0xffffffffL result ^= len(x) else: result = hash(x) return result # 64 constants taken from MD5 # (see Modules/md5c.c in the Python sources) tbl = ( -0x28955b88, -0x173848aa, 0x242070db, -0x3e423112, -0xa83f051, 0x4787c62a, -0x57cfb9ed, -0x2b96aff, 0x698098d8, -0x74bb0851, -0xa44f, -0x76a32842, 0x6b901122, -0x2678e6d, -0x5986bc72, 0x49b40821, -0x9e1da9e, -0x3fbf4cc0, 0x265e5a51, -0x16493856, -0x29d0efa3, 0x2441453, -0x275e197f, -0x182c0438, 0x21e1cde6, -0x3cc8f82a, -0xb2af279, 0x455a14ed, -0x561c16fb, -0x3105c08, 0x676f02d9, -0x72d5b376, -0x5c6be, -0x788e097f, 0x6d9d6122, -0x21ac7f4, -0x5b4115bc, 0x4bdecfa9, -0x944b4a0, -0x41404390, 0x289b7ec6, -0x155ed806, -0x2b10cf7b, 0x4881d05, -0x262b2fc7, -0x1924661b, 0x1fa27cf8, -0x3b53a99b, -0xbd6ddbc, 0x432aff97, -0x546bdc59, -0x36c5fc7, 0x655b59c3, -0x70f3336e, -0x100b83, -0x7a7ba22f, 0x6fa87e4f, -0x1d31920, -0x5cfebcec, 0x4e0811a1, -0x8ac817e, -0x42c50dcb, 0x2ad7d2bb, -0x14792c6f ) ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-17 16:09 Message: Logged In: YES user_id=1033539 Tim, your test fixture is very nice. I verified your result that there are no collisions over the fixture set for the algorithm you specified. Based on that result, I would recommend that we NOT use your algorithm. The probability of no collisions is near zero for 250000 random samples taken from a set of 2^32 elements. So the result shows that your algorithm is far from a good hash function, though it is constructed to be great for that specific fixture set. I ran my algorithm on your fixture set using the table of 64 constants taken from MD5. I got 13 collisions, a reasonable result. It is not true that I repeat multipliers (very often) - note that I increment the elements of the table each time around, so the sequence repeats itself only after 2^38 tuple elements. Also, my algorithm runs at esentially the same speed as the current one: no additional multiplies, only a few adds and increments and such. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-17 00:54 Message: Logged In: YES user_id=31435 I can't make more time for this, so unassigned myself. This seems to make a good set of test inputs: N = whatever base = range(N) xp = [(i, j) for i in base for j in base] inps = base + [(i, j) for i in base for j in xp] + [(i, j) for i in xp for j in base] When N == 50, inps contains 250,050 distinct elements, and 66,966 of them generate a duplicated hash code. Most simple changes to the current algorithm don't do significantly better, and some do much worse. For example, just replacing the xor with an add zooms it to 119,666 duplicated hash codes across that set. Here's a hash function that yields no collisions across that set: def hx(x): if isinstance(x, tuple): result = 0x345678L mult = 1000003 for elt in x: y = hx(elt) result = (((result + y) & 0xffffffffL) * mult) & 0xffffffffL mult = (mult * 69069) & 0xffffffffL result ^= len(x) if result == -1: result = -2 else: result = hash(x) return result In C, none of the "& 0xffffffffL" clauses are needed; in Python code, they're needed to cut results back to 32 bits. 69069 is a well-known multiplier for a better-than-most pure multiplicative PRNG. This does add a multiply per loop trip over what we have today, but it can proceed in parallel with the existing multiply. Unlike a canned table, it won't repeat multipliers, (unless the tuple has more than a billion elements). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-17 00:09 Message: Logged In: YES user_id=31435 Noting that (X, (Y, Z)) and (Y, (X, Z)) hash to the same thing today too (module -1 endcases). For example, >>> hash((3, (4, 8))) == hash((4, (3, 8))) True >>> ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-16 15:25 Message: Logged In: YES user_id=1033539 Hmm, you are correct. This is appears to be an off-by-one problem: the original seed always gets multiplied by the constant (which is silly), and the last item in the tuple does not get multiplied (which causes the bug). The correct solution is to change: value = my_mul(const, value) ^ hash(item) in Steve's pseudo-code to: value = my_mul(const, value ^ hash(item)) Of course, you still get a lot more robustness for almost no cost if you vary "const" across the tuple via a table. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-03 16:40 Message: Logged In: YES user_id=764593 Note that in this case, the problem was because of nested tuples. X was the first element of both tuples, and both tuples had the same length -- so the X's would still have been multiplied by the same constant. (Not the same constant as Y, and hash(X, Y), but the same constant as each other.) A non-linear function might work, but would do bad things to other data. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-02 05:50 Message: Logged In: YES user_id=1033539 I suggest leaving the function as is, but replacing the constant with a value that varies as we step through the tuple. Take the values from a fixed table. When we reach the end of the table, start again from the beginning and mung the values slightly (e.g., increment them). True, there will always be the possibilities of collisions, but this will make it very unlikely except in very weird cases. Using a table of constants is a standard technique for hashes. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 20:45 Message: Logged In: YES user_id=764593 Wouldn't that just shift the pathological case from (x, (x, y)) to (x, (-x, y))? My point was that any hash function will act badly on *some* pattern of input, and if a pattern must be penalized, this might be a good pattern to choose. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 19:36 Message: Logged In: YES user_id=80475 The OP was not referring to "some collisions"; his app collapsed all entries to a single hash value. Changing XOR to + would partially eliminate the self cancelling property of this hash function. Also, I am concerned about the tuple hash using the same multiplier as the hash for other objects. In sets.py, a naive combination of the component hash values caused many distinct sets to collapse to a handful of possibilities -- while tuples do not have an identical issue, it does highlight the risks involved. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 17:50 Message: Logged In: YES user_id=764593 Any hash will have some collisions. If there is going to be predictably bad data, this is probably a good place to have it. The obvious alternatives are a more complicated hash (slows everything down), a different hash for embedded tuples (bad, since hash can't be cached then) or ignoring some elements when determining the hash (bad in the normal case of different data). I would also expect your workaround of data rearrangement to be sensible almost any time (X, (X, Y)) is really a common case. (The intuitive meaning for me is "X - then map X to Y", which could be done as (X, Y) or at least (X, (None, Y)), or perhaps d[X]=(X,Y).) ---------------------------------------------------------------------- Comment By: Steve Tregidgo (smst) Date: 2004-04-27 14:45 Message: Logged In: YES user_id=42335 I'll repeat the tuple hashing algorithm in a fixed-width font (with the first line following "for..." being the loop body): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 From noreply at sourceforge.net Wed May 19 20:56:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 20:56:32 2004 Subject: [ python-Feature Requests-950644 ] Allow any lvalue for function definitions Message-ID: Feature Requests item #950644, was opened at 2004-05-08 21:52 Message generated for change (Comment added) made by mcherm You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=950644&group_id=5470 Category: Parser/Compiler Group: None Status: Open Resolution: None Priority: 5 Submitted By: David Albert Torpey (dtorp) Assigned to: Nobody/Anonymous (nobody) Summary: Allow any lvalue for function definitions Initial Comment: A definition like: def M(x): return 2*x is the same as: M = lambda x: 2*x With the latter form, I can use any lvalue: A[0] = lambda x: 2*x B.f = lambda x: 2*x But with the first form, you're locked into just using a plain variable name. If this were fixed, it wouldn't break anything else but would be useful for making method definitons outside of a class definition: This came up when I was experimenting with David MacQuigg's ideas for prototype OO. I want to write something like: Account = Object.clone() Account.balance = 0 def Account.deposit(self, v): self.balance += v Unfortunately, the latter has to be written: def Account.deposit(self, v): self.balance += v Account.deposit = deposit ---------------------------------------------------------------------- >Comment By: Michael Chermside (mcherm) Date: 2004-05-19 20:56 Message: Logged In: YES user_id=99874 I'm highly dubious of this. I see little advantage to doing the definition and storing the value in a single line, mostly because I rarely want to do such a thing. Your example may be convincing in Prothon or some relative, but in Python the sensible way to do it is a normal method. I'd suggest that if you want this idea to go anywhere that you try posting this to c.l.py and seeing if you can drum up interest and support there. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=950644&group_id=5470 From noreply at sourceforge.net Wed May 19 23:36:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 19 23:36:20 2004 Subject: [ python-Bugs-957198 ] C/C++ extensions w/ Python + Mingw (Windows) Message-ID: Bugs item #957198, was opened at 2004-05-20 03:36 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957198&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Connelly (connelly) Assigned to: Nobody/Anonymous (nobody) Summary: C/C++ extensions w/ Python + Mingw (Windows) Initial Comment: I am asking you to distribute libpython23.a with Python. This is a library built by the Mingw compiler (www.mingw.org). When this file is present, it greatly simplifies the process of installing a C/C++ extension. The below document explains how make this library, and install a C/C++ extension with the Mingw compiler. Currently, a library is provided for the proprietary MSVC++ compiler (python23.lib), but not the open source Mingw compiler. Normally, one uses the following procedure to build and install a C/C++ extension: python setup.py build --compiler=your_compiler python setup.py install For Python 2.3.3 on Windows, with the Mingw (Minimalist GNU) compiler, the following steps must be taken: 1. Find your Mingw bin directory. Copy gcc.exe to cc.exe. 2. Get PExports from either of: http://sebsauvage.net/python/pexports-0.42h.zip http://starship.python.net/crew/kernr/mingw32/pexports-0.42h.zip Extract pexports.exe to your Mingw bin directory. 3. Find pythonxx.dll. It should be in your main Python directory. Do the following: pexports python23.dll > python23.def dlltool --dllname python23.dll --def python23.def --output-lib libpython23.a 4. Copy libpythonxx.a to \python\libs. 5. Patch distutils. Locate \python\lib\distutils\msvccompiler.py, open it, and find the following lines (around line 211): if len (self.__paths) == 0: raise DistutilsPlatformError, ("Python was built with version %s of Visual Studio, " "and extensions need to be built with the same " "version of the compiler, but it isn't installed." % self.__version) Delete these. 6. Move back to the directory of your extension. Do the following: python setup.py build --compiler=mingw32 python setup.py install Ideally, only step 6 should be required to install an extension. I submitted the patch for step 5 to python.sourceforge.net. Steps 2-4 can be avoided if the libpythonxx.a file is distributed with Python. Step 1 can probably be avoided with another patch. This document is based on http://sebsauvage.net/python/mingw.html, which was written for Mingw + Python 2.2. Thanks, Connelly ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957198&group_id=5470 From noreply at sourceforge.net Thu May 20 02:28:44 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 02:28:51 2004 Subject: [ python-Feature Requests-491033 ] asyncore - api doesn't provide doneevent Message-ID: Feature Requests item #491033, was opened at 2001-12-09 22:00 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=491033&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Joseph Wayne Norton (natsukashi) Assigned to: Nobody/Anonymous (nobody) Summary: asyncore - api doesn't provide doneevent Initial Comment: The asyncore api doesn't provide a way to invoke the "loop" method but to only perform one iteration. Since the loop method does some selection of which poll method to call before entering the while loop, I have to duplicate the same behavior in my own "dooneiteration" method. I would like some functionality similar to the Tk interp model where one can enter the mainloop or simply do one event. regards, - joe n. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-19 23:28 Message: Logged In: YES user_id=341410 I've implemented and submitted a patch to implement the desired functionality in asyncore.loop via a 4th keyword argument called 'count'. The patch to current CVS can be found here: python.org/sf/957240 ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2001-12-10 09:41 Message: Logged In: YES user_id=31435 The Feature Requests tracker didn't have any Categories defined. I defined the same ones as the Bug tracker has, and purt this report in category "Python Library". ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2001-12-10 09:15 Message: Logged In: YES user_id=31435 Moved to the Feature Request tracker. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2001-12-10 06:47 Message: Logged In: YES user_id=6380 If you can supply a decent patch, we'd happily apply it to Python 2.3. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=491033&group_id=5470 From noreply at sourceforge.net Thu May 20 02:45:11 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 02:45:20 2004 Subject: [ python-Feature Requests-491033 ] asyncore - api doesn't provide doneevent Message-ID: Feature Requests item #491033, was opened at 2001-12-09 22:00 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=491033&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Joseph Wayne Norton (natsukashi) Assigned to: Nobody/Anonymous (nobody) Summary: asyncore - api doesn't provide doneevent Initial Comment: The asyncore api doesn't provide a way to invoke the "loop" method but to only perform one iteration. Since the loop method does some selection of which poll method to call before entering the while loop, I have to duplicate the same behavior in my own "dooneiteration" method. I would like some functionality similar to the Tk interp model where one can enter the mainloop or simply do one event. regards, - joe n. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-19 23:45 Message: Logged In: YES user_id=341410 I have posted the patch to implement the desired functionality. SF patch #957240 ; python.org/sf/957240 ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-19 23:28 Message: Logged In: YES user_id=341410 I've implemented and submitted a patch to implement the desired functionality in asyncore.loop via a 4th keyword argument called 'count'. The patch to current CVS can be found here: python.org/sf/957240 ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2001-12-10 09:41 Message: Logged In: YES user_id=31435 The Feature Requests tracker didn't have any Categories defined. I defined the same ones as the Bug tracker has, and purt this report in category "Python Library". ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2001-12-10 09:15 Message: Logged In: YES user_id=31435 Moved to the Feature Request tracker. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2001-12-10 06:47 Message: Logged In: YES user_id=6380 If you can supply a decent patch, we'd happily apply it to Python 2.3. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=491033&group_id=5470 From noreply at sourceforge.net Thu May 20 03:16:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 03:16:52 2004 Subject: [ python-Bugs-942952 ] Weakness in tuple hash Message-ID: Bugs item #942952, was opened at 2004-04-27 14:41 Message generated for change (Comment added) made by ygale You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 6 Submitted By: Steve Tregidgo (smst) Assigned to: Nobody/Anonymous (nobody) Summary: Weakness in tuple hash Initial Comment: I've encountered some performance problems when constructing dictionaries with keys of a particular form, and have pinned the problem down to the hashing function. I've reproduced the effect in Python 1.5.2, Python 2.2 and Python 2.3.3. I came across this when loading a marshalled dictionary with about 40000 entries. Loading other dictionaries of this size has always been fast, but in this case I killed the interpreter after a few minutes of CPU thrashing. The performance problem was caused because every key in the dictionary had the same hash value. The problem is as follows: for hashable values X and Y, hash( (X, (X, Y)) ) == hash(Y). This is always true (except in the corner case where hash((X, Y)) is internally calculated to be -1 (the error value) and so -2 is forced as the return value). With data in this form where X varies more than Y (Y is constant, or chosen from relatively few values compared to X) chances of collision become high as X is effectively ignored. The hash algorithm for tuples starts with a seed value, then generates a new value for each item in the tuple by multiplying the iteration's starting value by a constant (keeping the lowest 32 bits) and XORing with the hash of the item. The final value is then XORed with the tuple's length. In Python (ignoring the careful business with -1): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) The tuple (X, Y) therefore has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash(Y) ^ 2 ...and the tuple (X, (X, Y)) has hash value: my_mul(const, my_mul(const, seed) ^ hash(X)) ^ hash((X, Y)) ^ 2 The outer multiplication is repeated, and is XORed with itself (cancelling both of them). The XORed 2s cancel also, leaving just hash(Y). Note that this cancellation property also means that the same hash value is shared by (X, (X, (X, (X, Y)))), and (X, (X, (X, (X, (X, (X, Y)))))), and so on, and (X, Z, (X, Z, Y)) and so on. I realise that choosing a hash function is a difficult task, so it may be that the behaviour seen here is a tradeoff against other desireable properties -- I don't have the expertise to tell. My naive suggestion would be that an extra multiplication is necessary, which presumably has a performance impact (multiplication being more expensive than XOR) but would reduce the possibility of cancellation. On the other hand, perhaps this particular case is rare enough that it's not worth the effort. For my own application I'm fortunate in that I can probably rearrange the data structure to avoid this case. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-20 10:16 Message: Logged In: YES user_id=1033539 Re Tim's concern about cache hits: The trade-off is a slight risk of slight performance degradation when hashing very long tuples, versus the risk of complete app failure in certain rare cases like Steve's. I think it is worthwhile to use more constants. Anecdotally, I did not experience any qualitative slowdown while exercising these things with the unit tests. OTH, I didn't time anything, and anyway I am using a creaky old Celeron. Another alternative: use two constants: x = (len & 0x1 ? 0xd76aa471 : 0xe8c7b75f) * (++x ^ y); That passes all of the current unit tests, including Tim's set. However, with that I am less confident that this bug won't come up again in the future. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-20 02:24 Message: Logged In: YES user_id=1033539 I uploaded my current diff and some unit tests to patch #957122. Please look over the unit tests carefully. I am sure there is much more we can do. Note that besides the addition of the table containing the constants, only three lines of code are changed from the original function. By inspection, this code should run at essentially the same speed as the old code on all platforms (tests anyone?) Based on the results of the unit tests, I had to do some tweaking. First of all, Tim is correct: constants that are divisible by any small prime (e.g., 2) are a problem. To be on the safe side, I replaced each md5 constant with a nearby prime. I ran into a problem with empty tuples nested in tuples. The seed kept getting xored with itself. It turns out that no matter how nice the md5 constants are, they all produce the same value when multiplied by zero. Find "++x" in the code to see the fix for this. Finally, I moved the xor with the tuple length from the end of the calculation to the beginning. That amplifies the effect very nicely. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-19 19:15 Message: Logged In: YES user_id=31435 If 250,050 balls are tossed into 2**32 bins at random, the mean number of occupied bins is ~250042.7 (== expected number of collisions is ~7.3), with std deviation ~2.7. While we're happy to get "better than random" distributions, "astronomically worse than random" distributions usually predict more of the same kind of problem reported later. That said, I certainly agree that would be a major, easy, and cheap improvement over the status quo. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-19 18:10 Message: Logged In: YES user_id=4771 Tim: > Most simple changes to the current algorithm don't > do significantly better, and some do much worse. The simple change suggested by ygale (put the xor before the multiply) does a quite reasonable job in your test suite. The number of collisions drop from 66966 to 1466. It also solves the original problem with (x, (x,y)), which should not be ignored because it could naturally occur if someone is using d.items() to compute a hash out of a dict built by d[x] = x,y. With this change it seems more difficult to find examples of regular families of tuples that all turn out to have the same hash value. Such examples do exist: (x, (1684537936,x)) yields 2000004 for any x. But the value 1684537936 has been carefully constructed for this purpose, it's the only number to misbehave like this :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-18 05:14 Message: Logged In: YES user_id=31435 Nothing wrong with being "better than random" -- it's not the purpose of Python's hash() to randomize, but to minimize hash collisions. That's why, e.g., hash(int) returns int unchanged. Then when indexing a dict by any contiguous range of ints (excepting -1), there are no collisions at all. That's a good thing. There's nothing in the structure of my example function geared toward the specific test instances used. It *should* do a good job of "spreading out" inputs that are "close together", and that's an important case, and the test inputs have a lot of that. It's a good thing that it avoids all collisions across them. About speed, the only answer is to time both, and across several major platforms. An extra mult may or may not cost more than blowing extra cache lines to suck up a table of ints. About the table of ints, it's always a dubious idea to multiply by an even number. Because the high bits of the product are thrown away, multiplying by an even number throws away information needlessly (it throws away as many useful bits as there are trailing zero bits in the multiplier). odd_number * 69069**i is always odd, so the multiplier in the table-free way is never even (or, worse, divisible by 4, 8, 16, etc). ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-17 16:39 Message: Logged In: YES user_id=1033539 Here is the Python code for the hash function that I am describing: def hx(x): if isinstance(x, tuple): result = 0x345678L for n, elt in enumerate(x): y = (tbl[n & 0x3f] + (n >> 6)) & 0xffffffffL result = (y * (result ^ hx(elt))) & 0xffffffffL result ^= len(x) else: result = hash(x) return result # 64 constants taken from MD5 # (see Modules/md5c.c in the Python sources) tbl = ( -0x28955b88, -0x173848aa, 0x242070db, -0x3e423112, -0xa83f051, 0x4787c62a, -0x57cfb9ed, -0x2b96aff, 0x698098d8, -0x74bb0851, -0xa44f, -0x76a32842, 0x6b901122, -0x2678e6d, -0x5986bc72, 0x49b40821, -0x9e1da9e, -0x3fbf4cc0, 0x265e5a51, -0x16493856, -0x29d0efa3, 0x2441453, -0x275e197f, -0x182c0438, 0x21e1cde6, -0x3cc8f82a, -0xb2af279, 0x455a14ed, -0x561c16fb, -0x3105c08, 0x676f02d9, -0x72d5b376, -0x5c6be, -0x788e097f, 0x6d9d6122, -0x21ac7f4, -0x5b4115bc, 0x4bdecfa9, -0x944b4a0, -0x41404390, 0x289b7ec6, -0x155ed806, -0x2b10cf7b, 0x4881d05, -0x262b2fc7, -0x1924661b, 0x1fa27cf8, -0x3b53a99b, -0xbd6ddbc, 0x432aff97, -0x546bdc59, -0x36c5fc7, 0x655b59c3, -0x70f3336e, -0x100b83, -0x7a7ba22f, 0x6fa87e4f, -0x1d31920, -0x5cfebcec, 0x4e0811a1, -0x8ac817e, -0x42c50dcb, 0x2ad7d2bb, -0x14792c6f ) ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-17 16:09 Message: Logged In: YES user_id=1033539 Tim, your test fixture is very nice. I verified your result that there are no collisions over the fixture set for the algorithm you specified. Based on that result, I would recommend that we NOT use your algorithm. The probability of no collisions is near zero for 250000 random samples taken from a set of 2^32 elements. So the result shows that your algorithm is far from a good hash function, though it is constructed to be great for that specific fixture set. I ran my algorithm on your fixture set using the table of 64 constants taken from MD5. I got 13 collisions, a reasonable result. It is not true that I repeat multipliers (very often) - note that I increment the elements of the table each time around, so the sequence repeats itself only after 2^38 tuple elements. Also, my algorithm runs at esentially the same speed as the current one: no additional multiplies, only a few adds and increments and such. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-17 00:54 Message: Logged In: YES user_id=31435 I can't make more time for this, so unassigned myself. This seems to make a good set of test inputs: N = whatever base = range(N) xp = [(i, j) for i in base for j in base] inps = base + [(i, j) for i in base for j in xp] + [(i, j) for i in xp for j in base] When N == 50, inps contains 250,050 distinct elements, and 66,966 of them generate a duplicated hash code. Most simple changes to the current algorithm don't do significantly better, and some do much worse. For example, just replacing the xor with an add zooms it to 119,666 duplicated hash codes across that set. Here's a hash function that yields no collisions across that set: def hx(x): if isinstance(x, tuple): result = 0x345678L mult = 1000003 for elt in x: y = hx(elt) result = (((result + y) & 0xffffffffL) * mult) & 0xffffffffL mult = (mult * 69069) & 0xffffffffL result ^= len(x) if result == -1: result = -2 else: result = hash(x) return result In C, none of the "& 0xffffffffL" clauses are needed; in Python code, they're needed to cut results back to 32 bits. 69069 is a well-known multiplier for a better-than-most pure multiplicative PRNG. This does add a multiply per loop trip over what we have today, but it can proceed in parallel with the existing multiply. Unlike a canned table, it won't repeat multipliers, (unless the tuple has more than a billion elements). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-17 00:09 Message: Logged In: YES user_id=31435 Noting that (X, (Y, Z)) and (Y, (X, Z)) hash to the same thing today too (module -1 endcases). For example, >>> hash((3, (4, 8))) == hash((4, (3, 8))) True >>> ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-16 15:25 Message: Logged In: YES user_id=1033539 Hmm, you are correct. This is appears to be an off-by-one problem: the original seed always gets multiplied by the constant (which is silly), and the last item in the tuple does not get multiplied (which causes the bug). The correct solution is to change: value = my_mul(const, value) ^ hash(item) in Steve's pseudo-code to: value = my_mul(const, value ^ hash(item)) Of course, you still get a lot more robustness for almost no cost if you vary "const" across the tuple via a table. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-03 16:40 Message: Logged In: YES user_id=764593 Note that in this case, the problem was because of nested tuples. X was the first element of both tuples, and both tuples had the same length -- so the X's would still have been multiplied by the same constant. (Not the same constant as Y, and hash(X, Y), but the same constant as each other.) A non-linear function might work, but would do bad things to other data. ---------------------------------------------------------------------- Comment By: Yitz Gale (ygale) Date: 2004-05-02 05:50 Message: Logged In: YES user_id=1033539 I suggest leaving the function as is, but replacing the constant with a value that varies as we step through the tuple. Take the values from a fixed table. When we reach the end of the table, start again from the beginning and mung the values slightly (e.g., increment them). True, there will always be the possibilities of collisions, but this will make it very unlikely except in very weird cases. Using a table of constants is a standard technique for hashes. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 20:45 Message: Logged In: YES user_id=764593 Wouldn't that just shift the pathological case from (x, (x, y)) to (x, (-x, y))? My point was that any hash function will act badly on *some* pattern of input, and if a pattern must be penalized, this might be a good pattern to choose. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-04-28 19:36 Message: Logged In: YES user_id=80475 The OP was not referring to "some collisions"; his app collapsed all entries to a single hash value. Changing XOR to + would partially eliminate the self cancelling property of this hash function. Also, I am concerned about the tuple hash using the same multiplier as the hash for other objects. In sets.py, a naive combination of the component hash values caused many distinct sets to collapse to a handful of possibilities -- while tuples do not have an identical issue, it does highlight the risks involved. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-04-28 17:50 Message: Logged In: YES user_id=764593 Any hash will have some collisions. If there is going to be predictably bad data, this is probably a good place to have it. The obvious alternatives are a more complicated hash (slows everything down), a different hash for embedded tuples (bad, since hash can't be cached then) or ignoring some elements when determining the hash (bad in the normal case of different data). I would also expect your workaround of data rearrangement to be sensible almost any time (X, (X, Y)) is really a common case. (The intuitive meaning for me is "X - then map X to Y", which could be done as (X, Y) or at least (X, (None, Y)), or perhaps d[X]=(X,Y).) ---------------------------------------------------------------------- Comment By: Steve Tregidgo (smst) Date: 2004-04-27 14:45 Message: Logged In: YES user_id=42335 I'll repeat the tuple hashing algorithm in a fixed-width font (with the first line following "for..." being the loop body): # assume 'my_mul' would multiply two numbers and return the low 32 bits value = seed for item in tpl: value = my_mul(const, value) ^ hash(item) value = value ^ len(tpl) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=942952&group_id=5470 From noreply at sourceforge.net Thu May 20 03:19:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 03:20:06 2004 Subject: [ python-Feature Requests-491033 ] asyncore - api doesn't provide doneevent Message-ID: Feature Requests item #491033, was opened at 2001-12-09 22:00 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=491033&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Joseph Wayne Norton (natsukashi) Assigned to: Nobody/Anonymous (nobody) Summary: asyncore - api doesn't provide doneevent Initial Comment: The asyncore api doesn't provide a way to invoke the "loop" method but to only perform one iteration. Since the loop method does some selection of which poll method to call before entering the while loop, I have to duplicate the same behavior in my own "dooneiteration" method. I would like some functionality similar to the Tk interp model where one can enter the mainloop or simply do one event. regards, - joe n. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-20 00:19 Message: Logged In: YES user_id=341410 Sorry about the double post, I was distracted by pretty colors or something. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-19 23:45 Message: Logged In: YES user_id=341410 I have posted the patch to implement the desired functionality. SF patch #957240 ; python.org/sf/957240 ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-19 23:28 Message: Logged In: YES user_id=341410 I've implemented and submitted a patch to implement the desired functionality in asyncore.loop via a 4th keyword argument called 'count'. The patch to current CVS can be found here: python.org/sf/957240 ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2001-12-10 09:41 Message: Logged In: YES user_id=31435 The Feature Requests tracker didn't have any Categories defined. I defined the same ones as the Bug tracker has, and purt this report in category "Python Library". ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2001-12-10 09:15 Message: Logged In: YES user_id=31435 Moved to the Feature Request tracker. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2001-12-10 06:47 Message: Logged In: YES user_id=6380 If you can supply a decent patch, we'd happily apply it to Python 2.3. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=491033&group_id=5470 From noreply at sourceforge.net Thu May 20 08:16:26 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 08:24:56 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-20 08:16 Message: Logged In: YES user_id=119306 I have an approach to have readline work well with threads while still acknowledging KeyboardInterrrupt. Using the alternate readline interface of rl_callback_handler_install() and rl_callback_read_char() along with a select(), we can recognize the interrupt signal when select returns EINTR and not need the signal handler at all. I just need to try my patch on a few more systems and try to put Anthony at ease. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 13:25 Message: Logged In: YES user_id=56897 There didn't seem to be an outcome from the python-dev discussion regarding system() and pthread_atfork(). The thread indicates that system() indeed is supposed to call atfork handlers, so therefore RedHat 9 is violating the pthread standard in that sense. (Whether or not they'll fix it is another issue.) There's also mention that os.system() may be changed to not call system() because of the atfork() problem. If the changes to avoid system() are implemented, would the pthread_atfork() approach still be problematic? As Martin Loewis points out, we could always implement the signal fixup in the child directly after the fork() if Python routines are being used to do the fork() in the first place. However if we're concerned about native modules that directly call fork() then it seems our choices are a pthread_atfork() approach or an approach where SIGINT isn't blocked. Without an async-signal-safe way to route a signal from one thread to another, I don't see how we can leave SIGINT unblocked in all threads. Re: Py_AddPendingCall. That approach might work in many cases, but I assume it doesn't work well when all threads are currently busy in native modules that are not well-behaved. For example, I have two threads: one in readline() and the other blocked in a native call that, like readline(), doesn't return control on EINTR. If the SIGINT is sent to the readline thread, the signal handler could check the thread ID and do the longjmp() since we're the proper thread to do so. If the SIGINT is sent to the other thread, the callback added by Py_AddPendingCall() won't necessarily be processed any time soon because no threads are going to return control (in a timely manner) to Python. To make matters worse, apparently even something as simple as pthread_self(), which we'd use to get the thread ID, isn't async-signal-safe on all platforms. From what I've read, none of the pthread functions are guaranteed to be async-signal-safe. :-( ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-13 12:48 Message: Logged In: YES user_id=119306 pthread_kill(). That is annoying, I have something nearly done that used it. I didn't double check the safety of pthread_kill. I saw that posix says that kill is safe to call from interrupt handlers and went from there. Can we note that we need a pthread_kill in a call to Py_AddPendingCall, and then handle it later? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-13 12:25 Message: Logged In: YES user_id=6656 Just to make life more entertaining, pthread_atfork isn't what you want, either. http://mail.python.org/pipermail/python-dev/2003-December/041309.html ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 12:20 Message: Logged In: YES user_id=56897 Argh! I thought we had a relatively clean solution, but it appears there's a stumbling block with the pthread_kill() approach. pthread_kill() is not guaranteed to be async-signal-safe, and nuggets found on the net indicate there's no portable way to redirect a process signal from one thread to another: http://groups.google.com/groups?q=pthread_kill+async+safe&start=30&hl=en&lr=&selm=3662B6A8.861984D5%40zko.dec.com&rnum=32 http://www.redhat.com/archives/phil-list/2003-December/msg00049.html Given that we can't safely call pthread_kill() from the SIGINT handler directly, there might be another way to solve our problems with pthread_atfork(). Here's my thinking: - Block SIGINT in all threads except the main (readline) thread. - Register via child process handler via pthread_atfork() that sets the SIGINT action for the child process back to the default. Unfortunately this fix isn't localized to the readline module as desired, but it may solve the problems. SIGINT routing will be forced to the readline thread, and child processes won't have SIGINT blocked, solving bug 756940. The IRIX thread signal delivery model (i.e.: broadcast) may cause problems since SIGINT may be pending when we attempt to set the action to default. Having SIGINT pending when the handler is changed to default would kill the child process. Maybe having the child process set the disposition to ignore and then to default would safely clear any pending SIGINT signal? I'll try to run some experiments with the pthread_atfork() approach soon, but work and home life for me is particularly busy lately. Apologies in advance if it takes me a while to respond or submit patches. If we're interested in a timely fix, would it be useful to break up the fix in two stages? I think we can all agree that the current approach of blocking ALL signals in created threads is a Bad Thing. What if we implement a quick, partial fix by simply change the existing code to only block SIGINT? This should be a two-line change to thread_pthread.h where "sigemptyset(&newmask); sigaddset(&newmask, SIGINT);" is used instead of "sigfillset(&newmask);". I see this partial fix having a number of benefits: - Easy change to make. No extra stuff to check for in configure or calls to things that may not exist or work properly. - Much less risky than trying to fix all the problems at once. The change only opens up signals to threads that Python-2.1 is already allowing through. - Should solve the SIGSEGV zombie problem and Guido's SIGTERM annoyance, although it would still have the problem reported in bug 756940. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-12 10:39 Message: Logged In: YES user_id=29957 This seems like a pragmatic and sensible approach to take, to me. It should probably be tested on the HP/UX boxes (google for 'HP/UX testdrive') I particularly like the idea of just putting a test in to block readline in the non-main thread. It seems the pythonic approach - since we can't guarantee behaviour that's anything but sane, it seems like a plan. Or at least make it issue a warning saying "don't do this" when readline is invoked from a non-main thread. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 10:11 Message: Logged In: YES user_id=6380 Sounds good. This solves the problem in the readline module, where it originates. BTW, if we can simplify things by only allowing readline() to be called from the main thread, that's fine with me. Doing console I/O from threads is insane anyway. We can start by assuming the signal broadcast problem is restricted to IRIX, and configure appropriately: define a test symbol for this and in configure, set this when IRIX is detected. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 09:54 Message: Logged In: YES user_id=56897 SIGINT is 'special' because that's the signal behind the problems reported in bug 465673. Given the readline module's setjmp/longjmp mechanism to process SIGINT, we simply cannot allow one thread to do the setjmp() and another thread to do the longjmp() when it receives SIGINT. Without the setjmp/longjmp stuff, SIGINT is no more special than any other asynchronous signal like SIGTERM, SIGUSR1, etc. It'd be great if we could get the desired behavior for SIGINT out of the readline module without setjmp/longjmp, but without help from the readline library I don't see an easy way to do this. The readline library insists on continuing the readline() call after a SIGINT is handled, and there doesn't appear to be any way to get it to abort the current readline() call short of modifying the readline library. If we're stuck with the setjmp/longjmp mechanism, I think we can solve the issues regarding readline() being called from another thread and exec'd processes from threads by using the pthread_kill() technique mentioned earlier. The steps would look something like this: - Do not block any signals (including SIGINT) in any threads. - When we initialize the readline module's jmp_buf via setjmp(), save off the current thread ID. Probably want to check for existing ownership of jmp_buf and flag an error if detected. - When the readline module SIGINT handler is invoked, check if the current thread owns jmp_buf. If we are the owning thread then execute the longjmp (or siglongjmp). If we're not the owning thread, then have the current thread execute pthread_kill(jmp_buf_owner_thread, SIGINT) and little else. This will defer the SIGINT to the only thread that can really process it correctly. - Since SIGINT isn't blocked in any thread, processes exec'd from threads should get the default behavior for SIGINT rather than having it blocked. The above algorithm has a race condition on thread implementations where all threads receive SIGINT. The race can cause SIGINT to be processed more than once. The jmp_buf owning thread might finish the processing of SIGINT before another thread starts its processing and re-sends SIGINT to the jmp_buf owning thread. If there's a way to know via configure that we're on a thread implementation that broadcasts SIGINT, we could #ifdef the code to use something like the getpid() hack in signalmodule.c to do the right thing. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 19:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 19:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 18:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 18:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 15:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 15:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 09:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 04:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 18:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 16:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 15:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Thu May 20 09:05:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 09:05:35 2004 Subject: [ python-Bugs-957381 ] bdist_rpm fails on redhat9, fc1, fc2 Message-ID: Bugs item #957381, was opened at 2004-05-20 08:05 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957381&group_id=5470 Category: Distutils Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jeff Epler (jepler) Assigned to: Nobody/Anonymous (nobody) Summary: bdist_rpm fails on redhat9, fc1, fc2 Initial Comment: distutils bdist_rpm has long been broken for recent versions of RPM (RedHat 9, Fedora Core 1 and 2) https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=88616 When an RPM contains an executable or shared library, a "-debuginfo" rpm is generated. For instance, the following two packages would be generated: foo-1.0.0-1.i386.rpm foo-debuginfo-1.0.0.1-i386.rpm When distutils is faced with this problem, it prints an error like AssertionError: unexpected number of RPM files found: ['build/bdist.linux-i686/rpm/RPMS/i386/foo-1.0.0-1.i386.rpm', build/bdist.linux-i686/rpm/RPMS/i386/foo-debuginfo-1.0.0-1.i386.rpm'] The bugzilla bug contains a proposed patch, but redhat/fedora developers chose not to accept it for their own build of Python. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957381&group_id=5470 From noreply at sourceforge.net Thu May 20 12:11:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 12:11:50 2004 Subject: [ python-Bugs-957485 ] os.lseek() doesn't work Message-ID: Bugs item #957485, was opened at 2004-05-20 18:11 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957485&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: michal cernoevic (cernoevic) Assigned to: Nobody/Anonymous (nobody) Summary: os.lseek() doesn't work Initial Comment: >>> f=file('.htaccess', 'r+') >>> s=f.readline() >>> f=os.lseek(f,0,0) Traceback (most recent call last): File "", line 1, in ? TypeError: an integer is required SunOS wwwinfoa 5.8 Generic_108528-24 sun4u sparc SUNW,Ultra-4 64-bit The nulls are really nuls. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957485&group_id=5470 From noreply at sourceforge.net Thu May 20 12:42:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 12:42:20 2004 Subject: [ python-Bugs-957505 ] SocketServer module documentation misleading Message-ID: Bugs item #957505, was opened at 2004-05-20 17:42 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957505&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Jon Giddy (jongiddy) Assigned to: Nobody/Anonymous (nobody) Summary: SocketServer module documentation misleading Initial Comment: The Python Library Reference page for the SocketServer module refers to RequestHandlerClass as though it was the name of an actual class. Actually, it should be either "request handler class" or "BaseRequestHandler". Secondly, in the second para of the BaseRequestHandler handle() method description, StreamRequestHandler and DatagramRequestHandler are referred to as mix-in classes when they are in fact subclasses of BaseRequestHandler. This error is also present in the comments at the start of the module. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957505&group_id=5470 From noreply at sourceforge.net Thu May 20 13:02:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 13:07:53 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-19 00:28 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-20 18:02 Message: Logged In: YES user_id=6656 This sounds cool! The only thing to be aware of is readline versioning... are these alternate interfaces a recent thing? ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-20 13:16 Message: Logged In: YES user_id=119306 I have an approach to have readline work well with threads while still acknowledging KeyboardInterrrupt. Using the alternate readline interface of rl_callback_handler_install() and rl_callback_read_char() along with a select(), we can recognize the interrupt signal when select returns EINTR and not need the signal handler at all. I just need to try my patch on a few more systems and try to put Anthony at ease. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 18:25 Message: Logged In: YES user_id=56897 There didn't seem to be an outcome from the python-dev discussion regarding system() and pthread_atfork(). The thread indicates that system() indeed is supposed to call atfork handlers, so therefore RedHat 9 is violating the pthread standard in that sense. (Whether or not they'll fix it is another issue.) There's also mention that os.system() may be changed to not call system() because of the atfork() problem. If the changes to avoid system() are implemented, would the pthread_atfork() approach still be problematic? As Martin Loewis points out, we could always implement the signal fixup in the child directly after the fork() if Python routines are being used to do the fork() in the first place. However if we're concerned about native modules that directly call fork() then it seems our choices are a pthread_atfork() approach or an approach where SIGINT isn't blocked. Without an async-signal-safe way to route a signal from one thread to another, I don't see how we can leave SIGINT unblocked in all threads. Re: Py_AddPendingCall. That approach might work in many cases, but I assume it doesn't work well when all threads are currently busy in native modules that are not well-behaved. For example, I have two threads: one in readline() and the other blocked in a native call that, like readline(), doesn't return control on EINTR. If the SIGINT is sent to the readline thread, the signal handler could check the thread ID and do the longjmp() since we're the proper thread to do so. If the SIGINT is sent to the other thread, the callback added by Py_AddPendingCall() won't necessarily be processed any time soon because no threads are going to return control (in a timely manner) to Python. To make matters worse, apparently even something as simple as pthread_self(), which we'd use to get the thread ID, isn't async-signal-safe on all platforms. From what I've read, none of the pthread functions are guaranteed to be async-signal-safe. :-( ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-13 17:48 Message: Logged In: YES user_id=119306 pthread_kill(). That is annoying, I have something nearly done that used it. I didn't double check the safety of pthread_kill. I saw that posix says that kill is safe to call from interrupt handlers and went from there. Can we note that we need a pthread_kill in a call to Py_AddPendingCall, and then handle it later? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-13 17:25 Message: Logged In: YES user_id=6656 Just to make life more entertaining, pthread_atfork isn't what you want, either. http://mail.python.org/pipermail/python-dev/2003-December/041309.html ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 17:20 Message: Logged In: YES user_id=56897 Argh! I thought we had a relatively clean solution, but it appears there's a stumbling block with the pthread_kill() approach. pthread_kill() is not guaranteed to be async-signal-safe, and nuggets found on the net indicate there's no portable way to redirect a process signal from one thread to another: http://groups.google.com/groups?q=pthread_kill+async+safe&start=30&hl=en&lr=&selm=3662B6A8.861984D5%40zko.dec.com&rnum=32 http://www.redhat.com/archives/phil-list/2003-December/msg00049.html Given that we can't safely call pthread_kill() from the SIGINT handler directly, there might be another way to solve our problems with pthread_atfork(). Here's my thinking: - Block SIGINT in all threads except the main (readline) thread. - Register via child process handler via pthread_atfork() that sets the SIGINT action for the child process back to the default. Unfortunately this fix isn't localized to the readline module as desired, but it may solve the problems. SIGINT routing will be forced to the readline thread, and child processes won't have SIGINT blocked, solving bug 756940. The IRIX thread signal delivery model (i.e.: broadcast) may cause problems since SIGINT may be pending when we attempt to set the action to default. Having SIGINT pending when the handler is changed to default would kill the child process. Maybe having the child process set the disposition to ignore and then to default would safely clear any pending SIGINT signal? I'll try to run some experiments with the pthread_atfork() approach soon, but work and home life for me is particularly busy lately. Apologies in advance if it takes me a while to respond or submit patches. If we're interested in a timely fix, would it be useful to break up the fix in two stages? I think we can all agree that the current approach of blocking ALL signals in created threads is a Bad Thing. What if we implement a quick, partial fix by simply change the existing code to only block SIGINT? This should be a two-line change to thread_pthread.h where "sigemptyset(&newmask); sigaddset(&newmask, SIGINT);" is used instead of "sigfillset(&newmask);". I see this partial fix having a number of benefits: - Easy change to make. No extra stuff to check for in configure or calls to things that may not exist or work properly. - Much less risky than trying to fix all the problems at once. The change only opens up signals to threads that Python-2.1 is already allowing through. - Should solve the SIGSEGV zombie problem and Guido's SIGTERM annoyance, although it would still have the problem reported in bug 756940. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-12 15:39 Message: Logged In: YES user_id=29957 This seems like a pragmatic and sensible approach to take, to me. It should probably be tested on the HP/UX boxes (google for 'HP/UX testdrive') I particularly like the idea of just putting a test in to block readline in the non-main thread. It seems the pythonic approach - since we can't guarantee behaviour that's anything but sane, it seems like a plan. Or at least make it issue a warning saying "don't do this" when readline is invoked from a non-main thread. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 15:11 Message: Logged In: YES user_id=6380 Sounds good. This solves the problem in the readline module, where it originates. BTW, if we can simplify things by only allowing readline() to be called from the main thread, that's fine with me. Doing console I/O from threads is insane anyway. We can start by assuming the signal broadcast problem is restricted to IRIX, and configure appropriately: define a test symbol for this and in configure, set this when IRIX is detected. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 14:54 Message: Logged In: YES user_id=56897 SIGINT is 'special' because that's the signal behind the problems reported in bug 465673. Given the readline module's setjmp/longjmp mechanism to process SIGINT, we simply cannot allow one thread to do the setjmp() and another thread to do the longjmp() when it receives SIGINT. Without the setjmp/longjmp stuff, SIGINT is no more special than any other asynchronous signal like SIGTERM, SIGUSR1, etc. It'd be great if we could get the desired behavior for SIGINT out of the readline module without setjmp/longjmp, but without help from the readline library I don't see an easy way to do this. The readline library insists on continuing the readline() call after a SIGINT is handled, and there doesn't appear to be any way to get it to abort the current readline() call short of modifying the readline library. If we're stuck with the setjmp/longjmp mechanism, I think we can solve the issues regarding readline() being called from another thread and exec'd processes from threads by using the pthread_kill() technique mentioned earlier. The steps would look something like this: - Do not block any signals (including SIGINT) in any threads. - When we initialize the readline module's jmp_buf via setjmp(), save off the current thread ID. Probably want to check for existing ownership of jmp_buf and flag an error if detected. - When the readline module SIGINT handler is invoked, check if the current thread owns jmp_buf. If we are the owning thread then execute the longjmp (or siglongjmp). If we're not the owning thread, then have the current thread execute pthread_kill(jmp_buf_owner_thread, SIGINT) and little else. This will defer the SIGINT to the only thread that can really process it correctly. - Since SIGINT isn't blocked in any thread, processes exec'd from threads should get the default behavior for SIGINT rather than having it blocked. The above algorithm has a race condition on thread implementations where all threads receive SIGINT. The race can cause SIGINT to be processed more than once. The jmp_buf owning thread might finish the processing of SIGINT before another thread starts its processing and re-sends SIGINT to the jmp_buf owning thread. If there's a way to know via configure that we're on a thread implementation that broadcasts SIGINT, we could #ifdef the code to use something like the getpid() hack in signalmodule.c to do the right thing. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-12 00:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 00:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 23:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 23:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 20:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 20:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 14:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 09:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 23:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 21:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 20:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 19:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 17:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 02:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 22:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 17:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 14:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 14:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 14:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 13:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 13:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 21:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 09:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 15:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 15:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-19 00:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Thu May 20 14:38:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 14:40:08 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-20 14:38 Message: Logged In: YES user_id=119306 The callback interface seems to have been added in readline 2.1, from 1997. There seem to be configure tests in the current Modules/readline.c code to search for features new to readline 2.1 so my current approach would be upping the minimum readline version from 2.0 to 2.1. If needed I could test for the callback interface and use it if available, but fall back to the readline() interface otherwise (and leave the thread and signal handling issues in place when used with older readline.) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-20 13:02 Message: Logged In: YES user_id=6656 This sounds cool! The only thing to be aware of is readline versioning... are these alternate interfaces a recent thing? ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-20 08:16 Message: Logged In: YES user_id=119306 I have an approach to have readline work well with threads while still acknowledging KeyboardInterrrupt. Using the alternate readline interface of rl_callback_handler_install() and rl_callback_read_char() along with a select(), we can recognize the interrupt signal when select returns EINTR and not need the signal handler at all. I just need to try my patch on a few more systems and try to put Anthony at ease. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 13:25 Message: Logged In: YES user_id=56897 There didn't seem to be an outcome from the python-dev discussion regarding system() and pthread_atfork(). The thread indicates that system() indeed is supposed to call atfork handlers, so therefore RedHat 9 is violating the pthread standard in that sense. (Whether or not they'll fix it is another issue.) There's also mention that os.system() may be changed to not call system() because of the atfork() problem. If the changes to avoid system() are implemented, would the pthread_atfork() approach still be problematic? As Martin Loewis points out, we could always implement the signal fixup in the child directly after the fork() if Python routines are being used to do the fork() in the first place. However if we're concerned about native modules that directly call fork() then it seems our choices are a pthread_atfork() approach or an approach where SIGINT isn't blocked. Without an async-signal-safe way to route a signal from one thread to another, I don't see how we can leave SIGINT unblocked in all threads. Re: Py_AddPendingCall. That approach might work in many cases, but I assume it doesn't work well when all threads are currently busy in native modules that are not well-behaved. For example, I have two threads: one in readline() and the other blocked in a native call that, like readline(), doesn't return control on EINTR. If the SIGINT is sent to the readline thread, the signal handler could check the thread ID and do the longjmp() since we're the proper thread to do so. If the SIGINT is sent to the other thread, the callback added by Py_AddPendingCall() won't necessarily be processed any time soon because no threads are going to return control (in a timely manner) to Python. To make matters worse, apparently even something as simple as pthread_self(), which we'd use to get the thread ID, isn't async-signal-safe on all platforms. From what I've read, none of the pthread functions are guaranteed to be async-signal-safe. :-( ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-13 12:48 Message: Logged In: YES user_id=119306 pthread_kill(). That is annoying, I have something nearly done that used it. I didn't double check the safety of pthread_kill. I saw that posix says that kill is safe to call from interrupt handlers and went from there. Can we note that we need a pthread_kill in a call to Py_AddPendingCall, and then handle it later? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-13 12:25 Message: Logged In: YES user_id=6656 Just to make life more entertaining, pthread_atfork isn't what you want, either. http://mail.python.org/pipermail/python-dev/2003-December/041309.html ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 12:20 Message: Logged In: YES user_id=56897 Argh! I thought we had a relatively clean solution, but it appears there's a stumbling block with the pthread_kill() approach. pthread_kill() is not guaranteed to be async-signal-safe, and nuggets found on the net indicate there's no portable way to redirect a process signal from one thread to another: http://groups.google.com/groups?q=pthread_kill+async+safe&start=30&hl=en&lr=&selm=3662B6A8.861984D5%40zko.dec.com&rnum=32 http://www.redhat.com/archives/phil-list/2003-December/msg00049.html Given that we can't safely call pthread_kill() from the SIGINT handler directly, there might be another way to solve our problems with pthread_atfork(). Here's my thinking: - Block SIGINT in all threads except the main (readline) thread. - Register via child process handler via pthread_atfork() that sets the SIGINT action for the child process back to the default. Unfortunately this fix isn't localized to the readline module as desired, but it may solve the problems. SIGINT routing will be forced to the readline thread, and child processes won't have SIGINT blocked, solving bug 756940. The IRIX thread signal delivery model (i.e.: broadcast) may cause problems since SIGINT may be pending when we attempt to set the action to default. Having SIGINT pending when the handler is changed to default would kill the child process. Maybe having the child process set the disposition to ignore and then to default would safely clear any pending SIGINT signal? I'll try to run some experiments with the pthread_atfork() approach soon, but work and home life for me is particularly busy lately. Apologies in advance if it takes me a while to respond or submit patches. If we're interested in a timely fix, would it be useful to break up the fix in two stages? I think we can all agree that the current approach of blocking ALL signals in created threads is a Bad Thing. What if we implement a quick, partial fix by simply change the existing code to only block SIGINT? This should be a two-line change to thread_pthread.h where "sigemptyset(&newmask); sigaddset(&newmask, SIGINT);" is used instead of "sigfillset(&newmask);". I see this partial fix having a number of benefits: - Easy change to make. No extra stuff to check for in configure or calls to things that may not exist or work properly. - Much less risky than trying to fix all the problems at once. The change only opens up signals to threads that Python-2.1 is already allowing through. - Should solve the SIGSEGV zombie problem and Guido's SIGTERM annoyance, although it would still have the problem reported in bug 756940. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-12 10:39 Message: Logged In: YES user_id=29957 This seems like a pragmatic and sensible approach to take, to me. It should probably be tested on the HP/UX boxes (google for 'HP/UX testdrive') I particularly like the idea of just putting a test in to block readline in the non-main thread. It seems the pythonic approach - since we can't guarantee behaviour that's anything but sane, it seems like a plan. Or at least make it issue a warning saying "don't do this" when readline is invoked from a non-main thread. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 10:11 Message: Logged In: YES user_id=6380 Sounds good. This solves the problem in the readline module, where it originates. BTW, if we can simplify things by only allowing readline() to be called from the main thread, that's fine with me. Doing console I/O from threads is insane anyway. We can start by assuming the signal broadcast problem is restricted to IRIX, and configure appropriately: define a test symbol for this and in configure, set this when IRIX is detected. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 09:54 Message: Logged In: YES user_id=56897 SIGINT is 'special' because that's the signal behind the problems reported in bug 465673. Given the readline module's setjmp/longjmp mechanism to process SIGINT, we simply cannot allow one thread to do the setjmp() and another thread to do the longjmp() when it receives SIGINT. Without the setjmp/longjmp stuff, SIGINT is no more special than any other asynchronous signal like SIGTERM, SIGUSR1, etc. It'd be great if we could get the desired behavior for SIGINT out of the readline module without setjmp/longjmp, but without help from the readline library I don't see an easy way to do this. The readline library insists on continuing the readline() call after a SIGINT is handled, and there doesn't appear to be any way to get it to abort the current readline() call short of modifying the readline library. If we're stuck with the setjmp/longjmp mechanism, I think we can solve the issues regarding readline() being called from another thread and exec'd processes from threads by using the pthread_kill() technique mentioned earlier. The steps would look something like this: - Do not block any signals (including SIGINT) in any threads. - When we initialize the readline module's jmp_buf via setjmp(), save off the current thread ID. Probably want to check for existing ownership of jmp_buf and flag an error if detected. - When the readline module SIGINT handler is invoked, check if the current thread owns jmp_buf. If we are the owning thread then execute the longjmp (or siglongjmp). If we're not the owning thread, then have the current thread execute pthread_kill(jmp_buf_owner_thread, SIGINT) and little else. This will defer the SIGINT to the only thread that can really process it correctly. - Since SIGINT isn't blocked in any thread, processes exec'd from threads should get the default behavior for SIGINT rather than having it blocked. The above algorithm has a race condition on thread implementations where all threads receive SIGINT. The race can cause SIGINT to be processed more than once. The jmp_buf owning thread might finish the processing of SIGINT before another thread starts its processing and re-sends SIGINT to the jmp_buf owning thread. If there's a way to know via configure that we're on a thread implementation that broadcasts SIGINT, we could #ifdef the code to use something like the getpid() hack in signalmodule.c to do the right thing. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 19:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 19:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 18:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 18:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 15:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 15:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 09:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 04:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 18:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 16:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 15:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Thu May 20 16:40:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 16:40:50 2004 Subject: [ python-Feature Requests-957652 ] Implement BundleBuilder GUI as plugin component Message-ID: Feature Requests item #957652, was opened at 2004-05-20 20:40 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=957652&group_id=5470 Category: Macintosh Group: None Status: Open Resolution: None Priority: 5 Submitted By: has (hhas) Assigned to: Nobody/Anonymous (nobody) Summary: Implement BundleBuilder GUI as plugin component Initial Comment: On Wed, 19 May 2004 10:53:42 +0200, Jack Jansen writes on PythonMac SIG: "In a future release I'd like to add a GUI interface to bundlebuilder, so you can gradually migrate from save as applet to bundlebuilder without the current steep learning curve." Here's a few suggestions on how this might ideally be implemented: - Would it be possible/practical to construct this as, say, a plug- in component rather than a standalone executable? That way IDEs could incorporate it directly, bringing up the BB dialog when the user selects 'Save as Applet...'. - Arranging the window so that 'Advanced' options are initially hidden behind one of those disclosure triangle thingies would keep it easy to use when folk just want to spit out a generic applet. - Also, it'd be good to provide 'BBGUI' as a separate module to the original BB module so that scripts and applications wanting to use BB's programmatic interface aren't made to import unnecessary GUI toolkit modules. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=957652&group_id=5470 From noreply at sourceforge.net Thu May 20 16:41:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 16:41:49 2004 Subject: [ python-Feature Requests-957652 ] Implement BundleBuilder GUI as plugin component Message-ID: Feature Requests item #957652, was opened at 2004-05-20 20:40 Message generated for change (Settings changed) made by hhas You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=957652&group_id=5470 Category: Macintosh Group: None Status: Open Resolution: None Priority: 5 Submitted By: has (hhas) >Assigned to: Jack Jansen (jackjansen) Summary: Implement BundleBuilder GUI as plugin component Initial Comment: On Wed, 19 May 2004 10:53:42 +0200, Jack Jansen writes on PythonMac SIG: "In a future release I'd like to add a GUI interface to bundlebuilder, so you can gradually migrate from save as applet to bundlebuilder without the current steep learning curve." Here's a few suggestions on how this might ideally be implemented: - Would it be possible/practical to construct this as, say, a plug- in component rather than a standalone executable? That way IDEs could incorporate it directly, bringing up the BB dialog when the user selects 'Save as Applet...'. - Arranging the window so that 'Advanced' options are initially hidden behind one of those disclosure triangle thingies would keep it easy to use when folk just want to spit out a generic applet. - Also, it'd be good to provide 'BBGUI' as a separate module to the original BB module so that scripts and applications wanting to use BB's programmatic interface aren't made to import unnecessary GUI toolkit modules. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=957652&group_id=5470 From noreply at sourceforge.net Thu May 20 16:45:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 16:45:27 2004 Subject: [ python-Bugs-796219 ] ntpath.expanduser() is still wrong Message-ID: Bugs item #796219, was opened at 2003-08-27 13:11 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=796219&group_id=5470 Category: Windows Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Guido van Rossum (gvanrossum) Assigned to: Guido van Rossum (gvanrossum) Summary: ntpath.expanduser() is still wrong Initial Comment: I found a system with the following setup: - os.getenv("HOME") returns "%USERPROFILE%" - os.getenv("USERPROFILE") returns the home directory Currently, ntpath.py doesn't expand ~ correctly in this case. The fix is pretty simple, I'll try to submit it if I have time. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-20 13:45 Message: Logged In: YES user_id=341410 I have submitted sf patch #957650 to fix the bug listed. There is still an issue when there actually exists folders with names that mirror environment variables, but regardless of whether we stat the filesystem, the meaning of such a thing is ambiguous. The patch also includes ~user\subpath functionality, which has been missing in ntpath. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-03-26 16:53 Message: Logged In: YES user_id=341410 After doing some more playing around, I discovered a few other examples that would kill either method described below: HOME=%USERDRIVE%%USERPATH% HOME=C:\%userprofiles\%USERNAME% The real trick is that % can exist in a path on windows, so even c:\%HOME% is a vaild path: Directory of D:\test 03/26/2004 03:58p . 03/26/2004 03:58p .. 03/26/2004 03:59p %HOME% 0 File(s) 0 bytes 3 Dir(s) 5,355,511,808 bytes free D:\test> I suppose the question remains as to what cases do we want to cover. If we assume that there aren't any % symbols in a path, then the code is straightforward, and I can have a patch for you in a few minutes. If % can be in a path, then the problem is a pain, and a miniature parser needs to be written to deal with it. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-03-20 14:49 Message: Logged In: YES user_id=6380 Unclear what I'm asked to do here. Josiah, could you produce an actual patch against CVS rather than random example code? If you have forward slashes, you should use os.path.normpath(). Why doesn't that work? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2003-09-30 07:00 Message: Logged In: YES user_id=31435 Assigned back to Guido. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2003-09-29 23:33 Message: Logged In: YES user_id=341410 I just noticed that I've got some forward slashes in various paths on my windows machines...here's some updated code: def expandfull(var, rem=3): if not rem: return os.path.expandvars(var) a = os.path.expandvars(var) b = [] d = [b.extend(i.split('\')) for i in a.split('/')] c = [] for i in b: if '%' in i: c.append(expandfull(i), rem-1) else: c.append(i) return '\'.join(c) ---------------------------------------------------------------------- Comment By: Christos Georgiou (tzot) Date: 2003-08-30 02:55 Message: Logged In: YES user_id=539787 I stand corrected; multiple backslashes inside a path are not merged into one on Windows. Thank you. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2003-08-29 11:48 Message: Logged In: YES user_id=341410 Sourceforge ate my double-backslashes. All '\' should be '\\'. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2003-08-29 11:20 Message: Logged In: YES user_id=341410 The code you offered won't work correctly for all environment variable returns. An example that would kill your code: %SYSTEMROOT%\System32 def expandfull(var, rem=3): if not rem: return expandvars(var) a = expandvars(var) b = a.split('\') c = [] for i in b: if '%' in i: c.append(expandfull(i), rem-1) else: c.append(i) return '\'.join(c) The above would work properly for all environment variables. ---------------------------------------------------------------------- Comment By: Christos Georgiou (tzot) Date: 2003-08-29 01:38 Message: Logged In: YES user_id=539787 If expandvars worked for nt variable syntax too, then just before the expanduser final return, the following code would suffice, right? max_recursion = 3 while '%' in userhome and max_recursion > 0: userhome = expandvars(userhome) max_recursion -= 1 ignoring the fact that path[1:] could contain variables too. Shouldn't expandvars be made to work with %var% format too? If yes, I'll offer code. ---------------------------------------------------------------------- Comment By: Jarek Zgoda (zgoda) Date: 2003-08-28 00:47 Message: Logged In: YES user_id=92222 This is very common setting on Windows2000 Professional. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=796219&group_id=5470 From noreply at sourceforge.net Thu May 20 20:06:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 20 20:06:52 2004 Subject: [ python-Bugs-947380 ] sys.path is wrong in some cases Message-ID: Bugs item #947380, was opened at 2004-05-03 17:46 Message generated for change (Comment added) made by josiahcarlson You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947380&group_id=5470 Category: Python Interpreter Core Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Ori Berger (orib) Assigned to: Nobody/Anonymous (nobody) Summary: sys.path is wrong in some cases Initial Comment: Python version tested: 2.3.3 (But if I read the cvs annotate correctly, this goes all the way back to 2.0) OS Version tested: Win2K (but any win32 version should behave the same). On Windows, sys,path sometimes contains the 'current working directory', in which the Python process was started, while the interpreter is still initializing; it shouldn't be there until after the interpreter had completed initializing and is ready for batch or interactive execution. How to reproduce: Use plain-vanilla 2.3.3 Python, without _any_ additional module installed. The HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.3\PythonPath registry key must NOT have any subkey (this is true for a fresh install, but might not be true if packages were installed earlier). Go to a subdirectory of your choice, e.g., C:\HoverCraft\Eels, and run (assuming Python is in c:\python23) > c:\python23\python -c "import sys; print sys.path" The first entry should be the current directory or ''. That's ok - it was added at the end of initialization. The second entry will be a .zip file where the .dll is; That's also ok. The third entry should be the current directory (or '') again. THIS IS NOT OK. How this was discovered: To give credit where credit is due, my friend Oren Gampel created a file called 'stat.py' in a directory, and from that moment on, Python started complaining about failing to import site. The reason is that 'site' imports 'ntpath', which imports 'stat', which unfortunately imported the new 'stat' rather than the library 'stat'. In some convoluted settings, this might have security implications - e.g., if an administrator starts a Python script in a directory to which a user has write permissions, this could result in a privelege escalation (even if the script has no import statements at all - the implicit "import site" is sufficient). I'm submitting a 2-line patch to the patch tracker (and one of them is a comment!) that seems to solve this problem. ---------------------------------------------------------------------- Comment By: Josiah Carlson (josiahcarlson) Date: 2004-05-20 17:06 Message: Logged In: YES user_id=341410 This is the old "module in current path shadows standard library module" issue which will be fixed in Python 2.4 via absolute and relative imports as stated in PEP 328: http://python.org/peps/pep-0328.html ---------------------------------------------------------------------- Comment By: Ori Berger (orib) Date: 2004-05-03 17:59 Message: Logged In: YES user_id=67862 Patch uploaded to http://python.org/sf/947386 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=947380&group_id=5470 From noreply at sourceforge.net Fri May 21 10:46:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 21 10:46:32 2004 Subject: [ python-Feature Requests-935915 ] os.nullfilename Message-ID: Feature Requests item #935915, was opened at 2004-04-16 05:44 Message generated for change (Comment added) made by jbelmonte You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=935915&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: John Belmonte (jbelmonte) Assigned to: Nobody/Anonymous (nobody) Summary: os.nullfilename Initial Comment: Just as the os library provides os.sep, etc., for the current OS, it should provide the name of the null file (e.g., "/dev/null" or "nul:"), so that there is a portable way to open a null file. Use of an object such as class nullFile: def write(self, data): pass is not sufficient because it does not provide a full file object interface (no access to file descriptor, etc.). See discussion at . ---------------------------------------------------------------------- >Comment By: John Belmonte (jbelmonte) Date: 2004-05-21 23:46 Message: Logged In: YES user_id=282299 I do intend to make a patch, but it may be some time before I get to it. Please give me a few weeks. If someone else would like to step in, that is fine, just let me know before you start the work. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-21 15:00 Message: Logged In: YES user_id=80475 Consider mentioning this on comp.lang.python. Perhaps someone will volunteer to write a patch. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-10 01:08 Message: Logged In: YES user_id=21627 Would you like to work on a patch? ---------------------------------------------------------------------- Comment By: David Albert Torpey (dtorp) Date: 2004-05-09 10:54 Message: Logged In: YES user_id=681258 I like this idea. It is more portable. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-04-16 05:52 Message: Logged In: YES user_id=21627 Move to feature requests tracker ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=935915&group_id=5470 From noreply at sourceforge.net Fri May 21 12:01:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 21 12:02:11 2004 Subject: [ python-Feature Requests-887237 ] Machine integers Message-ID: Feature Requests item #887237, was opened at 2004-01-29 20:34 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=887237&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Fredrik Johansson (fredrik_j) Assigned to: Nobody/Anonymous (nobody) Summary: Machine integers Initial Comment: I think it'd be useful with a built-in module (I propose the name 'mint' :), providing classes for emulating machine ints. There should be support for ints of different sizes (at least 8, 16, 32, 64), signedness and endianness. The machine ints would react to overflow (negative values) and bit-level operations in the same way their C equivalents would. One idea for a feature would be a intobj.pack() (or str() or whatever) method that could be used instead of the struct module's pack(). int16(12345).pack() -> "\x30\x39". int32(-1).pack() -> "\0xff\0xff\0xff\0xff", etc. A related idea is that these ints could provide slicing to extract individual sets of bits or bytes. For example, uint32()[8:16] could return an int (uint8?) made from the bits in the int's second byte (whether it's the second from the left or the right might be fuzzy, I do realize). Applications for this would be algorithmic code involving bit fiddling (useful for cryptographic applications, perhaps) and, obviously, programs that have to operate on any binary data. It might also resolve some issues related to the unification of longs and ints. ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-21 16:01 Message: Logged In: YES user_id=4771 A number of projects end up redefining Python classes that try to emulate the raw integers of C, which is a pain to do properly in Python. I think that a general-purpose 'mint' class would be generally useful. Writing them in Python is a bit of a pain (http://codespeak.net/svn/pypy/trunk/src/pypy/objspace/std/restricted_int.py). Pyrex seems to be a good way to generate it as a extension module (http://codespeak.net/svn/user/arigo/misc/machineint.pyx). ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-02-26 02:43 Message: Logged In: YES user_id=139309 If I were providing slicing on bits I would say that foo[n] is bool(foo & (1< 7 bcdreg.multiply(1000) --> 97000 None of this is for the mainstream user and it would be unlikely to ever be included in the standard library. The best approach would be to experiment with an API in pure python, perhaps publish it as a recipe, and if it is well received, put a C version on the vaults of parnassus. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=887237&group_id=5470 From noreply at sourceforge.net Fri May 21 12:14:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 21 12:14:23 2004 Subject: [ python-Feature Requests-906746 ] int object need more informative error message Message-ID: Feature Requests item #906746, was opened at 2004-02-29 03:14 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=906746&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Nobody/Anonymous (nobody) Summary: int object need more informative error message Initial Comment: When int objects encounter a NotImplementedError, they don't say much about what the problem is and are too unfriendly. Error messages can be more verbose. >>> reduce(int.__add__, (1, 2L)) NotImplemented >>> reduce(int.__add__, (1L, 2)) Traceback (most recent call last): File "", line 1, in ? TypeError: descriptor '__add__' requires a 'int' object but received a 'long' ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-21 16:14 Message: Logged In: YES user_id=4771 What you report are not error messages generated by integers, but part of a general internal mecanism. The golden rule with all __xxx__ special methods is that they should not be called directly. This is especially true for binary methods like __add__ which have pretty complex internal semantics. The expression x+y is not equivalent to x.__add__(y): roughly, althought it first tries x.__add__(y), it checks if this returned NotImplemented and calls y.__radd__(x) if it did. But there are a number of subtle rules and special cases. All special methods are meant to be called by language operators (e.g. x+y, `x`,...); for special usages like reduce(), all operators are also available as built-in functions either directly (e.g. repr) or in the module named 'operator' (e.g. operator.add). Finally, unbound method objects like int.__add__ are not to be used unless one is really aware of what is going on. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-21 06:09 Message: Logged In: YES user_id=80475 Unfortunately, there may not be any flexibility here. Instead of being just an error message, NotImplemented is a constant that has meaning to other layers of code. I don't see a way to change this or even attach a more informative error message. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=906746&group_id=5470 From noreply at sourceforge.net Fri May 21 12:25:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 21 12:25:53 2004 Subject: [ python-Feature Requests-778763 ] Optional type enforcement Message-ID: Feature Requests item #778763, was opened at 2003-07-28 06:18 Message generated for change (Comment added) made by arigo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=778763&group_id=5470 Category: Python Interpreter Core Group: None Status: Open Resolution: None Priority: 5 Submitted By: Matt Perry (occupant4) Assigned to: Nobody/Anonymous (nobody) Summary: Optional type enforcement Initial Comment: One of the things that makes Python easy to code in is the fact that you don't have to explicitly declare the types of variables, or associate a specific type with a variable. However, this can potentially lead to somewhat confusing and hard-to-maintain code, not to mention make it easier to introduce bugs. An idea struck me that it might be possible to combine the power and safety of a strictly type-checked language with the speed and ease of coding of Python. Normal variables as they stand now would be unaffected by this feature, and retain their polymorphic type capabilities. Allow the programmer to optionally declare the type of a variable somehow, be it with the C-like syntax "int x", or Pascal's (I think?) "x: int", or a new syntactic representation. Python could then issue a TypeError if the program attempts to assign a non-int to x. Obviously this feature could apply to any data type, such as string, list, or perhaps user-defined classes. Class member variables can be declared as now in the class definition body, with the same syntax to specify a type, ie: class Test: int x = 5 string name = "test" untyped = None In addition, adding a type specifier to a function's parameters could serve another means of distinguishing it from other functions, which could allow function overloading, ie: def double(int x): return 2*x def double(string x): return x + ' ' + x Advantages: - clarifies the author's intention in writing code, and could help prevent confusion as to what type a variable/parameter is. - helps prevent bugs due to type mistakes - doesn't interfere with existing code or coding style, as dynamically typed variables are still allowed - could allow function overloading if it was wanted Disadvantages: - implementing a type checker could be difficult? - potentially pollutes namespace with extra keywords like 'string' or 'list'. may need to choose different keywords. ---------------------------------------------------------------------- >Comment By: Armin Rigo (arigo) Date: 2004-05-21 16:25 Message: Logged In: YES user_id=4771 This idea has always been considered but there are very deep issues with it. You should give a look to Pyrex, whose language a more static and type-checked version of Python. It produces C extension modules for the regular Python interpreter, so it can be mixed with regular Python code easily. google:python+pyrex ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2003-08-05 11:52 Message: Logged In: YES user_id=80475 To make the most of polymorphism, interface checking would be preferable to enforcing specific types. For example, it is better to check that f is a filelike type rather than requiring that is actually be a file -- that way, StringIO objects can be substituted without breaking code. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=778763&group_id=5470 From noreply at sourceforge.net Fri May 21 14:48:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 21 14:48:15 2004 Subject: [ python-Feature Requests-906746 ] int object need more informative error message Message-ID: Feature Requests item #906746, was opened at 2004-02-29 03:14 Message generated for change (Comment added) made by nascheme You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=906746&group_id=5470 Category: None Group: None >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Nobody/Anonymous (nobody) Summary: int object need more informative error message Initial Comment: When int objects encounter a NotImplementedError, they don't say much about what the problem is and are too unfriendly. Error messages can be more verbose. >>> reduce(int.__add__, (1, 2L)) NotImplemented >>> reduce(int.__add__, (1L, 2)) Traceback (most recent call last): File "", line 1, in ? TypeError: descriptor '__add__' requires a 'int' object but received a 'long' ---------------------------------------------------------------------- >Comment By: Neil Schemenauer (nascheme) Date: 2004-05-21 18:48 Message: Logged In: YES user_id=35752 Armin is completely correct. One more point is that NotImplemented is not the same as NotImplementedError. The name similarity is unfortunate because they really have no relation. I'm marking the the bug as invalid as I don't think there is anything to fix. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-21 16:14 Message: Logged In: YES user_id=4771 What you report are not error messages generated by integers, but part of a general internal mecanism. The golden rule with all __xxx__ special methods is that they should not be called directly. This is especially true for binary methods like __add__ which have pretty complex internal semantics. The expression x+y is not equivalent to x.__add__(y): roughly, althought it first tries x.__add__(y), it checks if this returned NotImplemented and calls y.__radd__(x) if it did. But there are a number of subtle rules and special cases. All special methods are meant to be called by language operators (e.g. x+y, `x`,...); for special usages like reduce(), all operators are also available as built-in functions either directly (e.g. repr) or in the module named 'operator' (e.g. operator.add). Finally, unbound method objects like int.__add__ are not to be used unless one is really aware of what is going on. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-21 06:09 Message: Logged In: YES user_id=80475 Unfortunately, there may not be any flexibility here. Instead of being just an error message, NotImplemented is a constant that has meaning to other layers of code. I don't see a way to change this or even attach a more informative error message. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=906746&group_id=5470 From noreply at sourceforge.net Fri May 21 14:56:34 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 21 14:56:39 2004 Subject: [ python-Bugs-958180 ] logging functionality non-intuitive, levels confusing Message-ID: Bugs item #958180, was opened at 2004-05-21 12:56 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=958180&group_id=5470 Category: Python Library Group: Python 2.2.3 Status: Open Resolution: None Priority: 5 Submitted By: Jason R. Coombs (jaraco) Assigned to: Nobody/Anonymous (nobody) Summary: logging functionality non-intuitive, levels confusing Initial Comment: If one tries to define additional logging levels without overriding the logger class, one runs into trouble with the log level naming conventions. In particular, I observe the following strange behavior: >>> import logging >>> logging.basicConfig() >>> logging.root.level 30 >>> logging.root.log( logging.INFO, 'test1' ) >>> logging.root.log( 20, 'test1' ) >>> logging.root.log( 'info', 'test1' ) Level info:root:test1 >>> logging.root.log( 'INFO', 'test1' ) 20:root:test1 >>> logging.root.log( logging.WARNING, 'warning' ) WARNING:root:warning >>> logging.root.log( 30, 'warning' ) WARNING:root:warning >>> logging.root.log( 'warning', 'warning' ) Level warning:root:warning >>> logging.root.log( 'WARNING', 'warning' ) 30:root:warning Note that if a string parameter is passed as the lvl parameter of the log() method, the message is logged regardless of the current logging level, and the output is furthermore inconsistent. Additionally, when the user attempts to add a logging level, the user cannot reference that log level by name using any exposed (documented) interfaces. For example: >>> import logging >>> logging.basicConfig() >>> logging.root.level = logging.DEBUG # 10 >>> logging.addLevelName( 15, 'STATUS' ) >>> logging.root.log( 15, 'a status message' ) STATUS:root:a status message >>> logging.root.log( logging._levelNames['STATUS'], 'a status message' ) STATUS:root:a status message The only interface, as far as I can tell, to access the log level is to explicitly query the internal data structures. Changes I propose: (1) The log() methodshould either throw a TypeError when a string is passed for the log level or (better) it should accept string values and translate them transparently to the integer log levels that it currently expects. (2) The logging module should include functionality to allow the user to query a logging level from a name. def getLevelNumber( levelName ): return _levelNames[ levelName ] # error checking would be appropriate ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=958180&group_id=5470 From noreply at sourceforge.net Fri May 21 01:57:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 21 15:02:31 2004 Subject: [ python-Feature Requests-950644 ] Allow any lvalue for function definitions Message-ID: Feature Requests item #950644, was opened at 2004-05-08 20:52 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=950644&group_id=5470 Category: Parser/Compiler Group: None Status: Open Resolution: None Priority: 5 Submitted By: David Albert Torpey (dtorp) Assigned to: Nobody/Anonymous (nobody) Summary: Allow any lvalue for function definitions Initial Comment: A definition like: def M(x): return 2*x is the same as: M = lambda x: 2*x With the latter form, I can use any lvalue: A[0] = lambda x: 2*x B.f = lambda x: 2*x But with the first form, you're locked into just using a plain variable name. If this were fixed, it wouldn't break anything else but would be useful for making method definitons outside of a class definition: This came up when I was experimenting with David MacQuigg's ideas for prototype OO. I want to write something like: Account = Object.clone() Account.balance = 0 def Account.deposit(self, v): self.balance += v Unfortunately, the latter has to be written: def Account.deposit(self, v): self.balance += v Account.deposit = deposit ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-21 00:57 Message: Logged In: YES user_id=80475 I think this should be made possible. It allows for alternate coding styles wihout harming anything else. The Lua programming language has something similar. It is a lightweight, non-OO language that revolves around making the best possible use of namespaces. Direct assignments into a namespace come up in several contexts throughout the language and are key to Lua's flexibility (using one concept to meet all needs). My only concern is that "def" and "class" statements also have the side effect of binding the __name__ attribute. We would have to come up with a lambda style placeholder for the attribute. ---------------------------------------------------------------------- Comment By: Michael Chermside (mcherm) Date: 2004-05-19 19:56 Message: Logged In: YES user_id=99874 I'm highly dubious of this. I see little advantage to doing the definition and storing the value in a single line, mostly because I rarely want to do such a thing. Your example may be convincing in Prothon or some relative, but in Python the sensible way to do it is a normal method. I'd suggest that if you want this idea to go anywhere that you try posting this to c.l.py and seeing if you can drum up interest and support there. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=950644&group_id=5470 From noreply at sourceforge.net Fri May 21 02:02:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 22 03:01:13 2004 Subject: [ python-Feature Requests-927543 ] "with self:" statement Message-ID: Feature Requests item #927543, was opened at 2004-04-01 09:36 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=927543&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Thomas Dunne (tomdunne) Assigned to: Nobody/Anonymous (nobody) Summary: "with self:" statement Initial Comment: please forgive if this is documented somewhere; I looked and searched and could not find any up to date PEPs or comments about the status of this idea. I managed to find a thread from 2002 that talked about this in great detail: http://mail.python.org/pipermail/python-list/2002-July/112888.html but there was no conclusion... I know it's not a new idea and it shows up often... but here goes: How about a "with self:" statement, but more in the style of (forgive) VB then Pascal, ie referenced items in the self instance are preceded with a dot. present syntax: with self: ??returnvalue = self.x + self.y * self.width proposed syntax: with self: ??returnvalue = .x + .y * .width If no preceding dot is used, you run into an ambiguity when assigning values. How do you know if the variable is to be assigned in self or to an object in a higher scope... the preceding dot avoids that problem since it explicitly tags self items. (using nested "with" statements will lead to the same problem when assigning values. Assignment will have to be to the innermost "with object". Reading values should work ok though. ) I understand that I could just simply use "s." instead of "self." or assign all needed values in one preceding line: ??x,y,width = self.x, self.y, self.width This is a question that shows up every so often, the points made in the thread are good: http://mail.python.org/pipermail/python-list/2002-July/113831.html has this been recently discussed? is there a patch or PEP? (I might be able to provide a patch...) ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-21 01:02 Message: Logged In: YES user_id=80475 Guido occasionally mentions this as a possibility, so writing a PEP would be a useful next step. Though simple, this would change the feel of the language. Accordingly, it would need much public discussion before advancing. ---------------------------------------------------------------------- Comment By: Thomas Dunne (tomdunne) Date: 2004-05-05 16:01 Message: Logged In: YES user_id=1005553 Hey Greg, completely OT, but thought you should know: your sourceforge address redirect isn't correct! I sent an email to gward at users.sourceforge.net and the host (don't want to post for everyone to see here) responded that the redirected address didn't exist... ---------------------------------------------------------------------- Comment By: Thomas Dunne (tomdunne) Date: 2004-05-05 15:48 Message: Logged In: YES user_id=1005553 Hey Greg, thanks for the suggestion! I posted a PEP to peps@python.org, flame on! ;) (want a copy?) ---------------------------------------------------------------------- Comment By: Greg Ward (gward) Date: 2004-05-04 20:45 Message: Logged In: YES user_id=14422 A change of this magnitude demands a PEP. I don't recall having seen one, but I haven't been following python-dev for a while. If you want to do the right thing, write up a PEP and submit it to python-dev for the obligatory shooting-down-in-flames. Document the reasons why the feature was rejected in the PEP, and then the next time someone asks for it, your PEP can be trotted out. And don't take it personally when your idea gets shot down in flames. (Heck, who knows, maybe the python-dev crew will latch on and someone will implement it. But I doubt it...) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=927543&group_id=5470 From noreply at sourceforge.net Sat May 22 04:23:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 22 04:23:42 2004 Subject: [ python-Bugs-728330 ] Don't define _SGAPI on IRIX Message-ID: Bugs item #728330, was opened at 2003-04-27 02:21 Message generated for change (Comment added) made by rwgk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 Category: Installation Group: Python 2.3 Status: Open >Resolution: Remind >Priority: 9 Submitted By: Ralf W. Grosse-Kunstleve (rwgk) Assigned to: Neal Norwitz (nnorwitz) Summary: Don't define _SGAPI on IRIX Initial Comment: Python release: 2.3b1 Platform: IRIX 6.5 MIPSpro Compilers: Version 7.3.1.2m Commands used: ./configure --prefix=/usr/local_cci/Python-2.3b1t make The socket module fails to compile. The output from the compiler is attached. ---------------------------------------------------------------------- >Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-22 01:23 Message: Logged In: YES user_id=71407 Based on Maik's posting from 2004-03-08 08:53 I've worked out a small bug-fix patch relative to Python-2.3.4c1 that leads to successful creation of _socket.so under three different versions of IRIX: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.23 with MIPRpro 7.3.1.3 I've consistently used #if defined(__sgi) && _COMPILER_VERSION>700 to restrict the changes to IRIX. However, to be sure I've also verified that the Python installation is still OK under Redhat 8 and Tru64 Unix 5.1. I will upload the patch. It is also available here: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patch For completeness I've also posted these files: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patched_c http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_original_c I've made an effort to keep the patch minimally intrusive, but it is certainly not "clean." However, the status quo is so bad that it can only get better. We have not been able to switch to Python 2.3 because of the problems with the socket module. It would be fantastic if the bug-fix patch could be included in the 2.3.4 final distribution. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2004-03-08 08:53 Message: Logged In: YES user_id=768223 I got Python 2.3.3 compiled with MipsPro Compilers: Version 7.3.1.3m. The following changes were made to satisfy all needs. Line 206 in Modules/socketmodule.c #define _SGIAPI 1 NEW: #undef _XOPEN_SOURCE NEW: #define HAVE_GETADDRINFO 1 NEW: #define HAVE_GETNAMEINFO 1 #define HAVE_INET_PTON NEW: #include NEW: #include NEW: #include #include Line 269: /* #include "addrinfo.h" */ In addition to this I extend the CFLAGS with -D_BSD_TYPES to satisfy the u_short and friends definition. --maik./ ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-03-03 08:20 Message: Logged In: YES user_id=892601 Got it down to one error: % > ./python Lib/test/test_socket.py Traceback (most recent call last): File "Lib/test/test_socket.py", line 6, in ? import socket File "/usr/local/spider/lib/python2.3/socket.py", line 44, in ? import _socket ImportError: 210454:./python: rld: Fatal Error: unresolvable symbol in /usr/local/spider/lib/python2.3/lib-dynload/_socketmodule.so: _xpg5_accept xpg5_accept does appear in any library on our SGI's that are only 1-2 years old. It does appear in libc in very recent revisions of Irix (> 6.5.19?) Does anyone know how to get Python 2.3 to avoid the use of xpg5_accept? Thanks ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-01-12 10:59 Message: Logged In: YES user_id=892601 Has anyone encountered problems with different versions of IRIX 6.5? I finally got sockets to work in Python2.3.3, but only on the most up-to-date version of IRIX (uname -v = 01100304). On any other machine with a lower version (archetecture doesn't seem to matter), I get messages which all seem to be related to sockets: 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_vsnprintf 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recv 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_connect 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_bind 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_sendto 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_setsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_accept 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recvfrom 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_socket 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockname Upgrading all of our SGIs is not something I have control over. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-22 10:52 Message: Logged In: YES user_id=768223 Sorry if there is something unclear. Redefine means: Set a value which is required to satisfy ifdef-statements in the header files. As you see in the definitions, given in the comment, the value for _NO_XOPEN4 is set as a result of some dependencies automatically defined. A redefinition of _SGIAPI in the socketmodule.c is not sufficient alone. Therefore to satisfy all requirements in the header files, you should set the values for _NO_XOPEN4 and _NO_XOPEN5 also manually. Howto do in socketmodule.c? ... #define _SGIAPI 1 #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 ... If you are familiar with regular expressions, then you will know that _NO_XOPEN[45] is a shorthand for _NO_XOPEN4, _NO_XOPEN5. Sorry if there was something wrong in my comment. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-12-22 07:49 Message: Logged In: YES user_id=892601 Can someone please clarify mherta's comment from 2003-12-16? How should we "redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition"? Should those lines of code be added to socketmodule.c, or are they the source of the problem? Is _NO_XOPEN[45] shorthand for _NO_XOPEN[4] and _NO_XOPEN[5] or are they all different? Thanks. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-16 09:27 Message: Logged In: YES user_id=768223 I've tested the latest Python 2.3.3c1 to compile. The problem reported is still present. To avoid the error messages, about the incompatible types of several netdb-functions (like inet_pton), you should redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition. If there are error messages that types like u_short and u_char are not defined, please add -D_BSD_TYPES in the Makefile (BASECFLAGS is okay). If anybody don't need a executable compiled with MipsPro, gcc from the freeware.sgi.com could be a choice. The problem for this seems in /usr/include/standards.h: #define _NO_XOPEN4 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 >= 500))) #define _NO_XOPEN5 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 < 500))) As the first option is true (_XOPEN_SOURCE is set to 600 in pyconfig.h) the second option will be false. The constant #define NI_NUMERICHOST 0x00000002 /* return numeric form of hostname */ is enveloped in a clause #ifdef _NO_XOPEN4 && _NO_XOPEN5 (/usr/include/netdb.h) which is always false. In the manual pages there is a example which uses the constant NI_NUMERICHOST and compiles without error message. I'm not sure to cut off the file pyconfig.h from compiling the socketmodule.c? Any comments? ---------------------------------------------------------------------- Comment By: Hans Nowak (zephyrfalcon) Date: 2003-10-30 19:04 Message: Logged In: YES user_id=173607 My $0.02... Yesterday I successfully compiled Python 2.3.2, with socket module. I'm using an O2 running IRIX 6.5.11f. I didn't encounter any of the problems described before, and had no problems with _SGIAPI or anything. The reason it didn't compile was that u_long and u_char were not defined in some places; /usr/include/netinet/tcp.h to be precise, and a Python file (getaddrinfo.c). I'm using gcc 3.3 (the one downloaded from freeware.sgi.com). If anyone is interested, drop me a mail at hans at zephyrfalcon.org and I'll put the binary online somewhere. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-24 12:59 Message: Logged In: YES user_id=892601 Here's the output from 'gmake', using MIPSpro Compiler 7.30 on IRIX 6.5 Sorry, I don't know how to attach something using the "Add a Comment" box. Is there another way to post? The patch at www.python.org/2.3.2/patches/sgi-socketmodule.patch was edited into socketmodule.c : #include #endif /* GCC 3.2 on Irix 6.5 fails to define this variable at all. Based on the above code, I'd say that the SGI headers are just horked. */ #if defined(__sgi) && !defined(INET_ADDRSTRLEN) #define INET_ADDRSTRLEN 16 #endif /* Generic includes */ #include #include --------------------------------------------------------------------------- cc -OPT:Olimit=0 -DNDEBUG -n32 -O -I. -I./Include -DPy_BUILD_CORE -c ./Modules/socketmodule.c -o Modules/socketmodule.o cc-1047 cc: WARNING File = ./Modules/socketmodule.c, Line = 195 Macro "_SGIAPI" (declared at line 210 of "/usr/include/standards.h") has an incompatible redefinition. #define _SGIAPI 1 ^ cc-1020 cc: ERROR File = /usr/include/netdb.h, Line = 198 The identifier "socklen_t" is undefined. extern int iruserok_sa(const void *, socklen_t, int, const char *, ^ cc-1047 cc: WARNING File = ./Modules/addrinfo.h, Line = 145 Macro "_SS_ALIGNSIZE" (declared at line 246 of "/usr/include/sys/socket.h") has an incompatible redefinition. #define _SS_ALIGNSIZE (sizeof(PY_LONG_LONG)) ^ cc-1020 cc: ERROR File = ./Modules/socketmodule.c, Line = 822 The identifier "NI_NUMERICHOST" is undefined. NI_NUMERICHOST); ^ cc-1515 cc: ERROR File = ./Modules/socketmodule.c, Line = 3054 A value of type "int" cannot be assigned to an entity of type "const char *". retval = inet_ntop(af, packed, ip, sizeof(ip)); ^ 3 errors detected in the compilation of "./Modules/socketmodule.c". gmake: *** [Modules/socketmodule.o] Error 2 ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2003-10-23 03:25 Message: Logged In: YES user_id=6656 Just to be clear, please "attach" the build rather than pasting it inline. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2003-10-22 19:19 Message: Logged In: YES user_id=29957 Did you try 2.3.2 with the patch from the 2.3.2 build bugs page? If it failed, please post the build log of the failing bit to this bug. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-22 11:26 Message: Logged In: YES user_id=892601 Has there been any progress since August on getting Python-2.3 to compile on IRIX 6.5? I really need to have sockets working to use python. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-08-14 04:37 Message: Logged In: YES user_id=768223 I have the same problem on IRIX 6.5.19m and MIPSpro 7.3.1.3m on a FUEL 600Mhz. The situation seems a little bit strange on this system. But I have luck and get it compiled. As the current implementation is not clean, my resolution also. Which should be changed to get it work. The manpage for gethostbyaddr_r says: #include #include #include #include So I included this in the __sgi-clause in socketmodule.c. Looking in the header-files shows, that some variables are defined if #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 are set, before the includes. This defines some more variables, so that pyconfig.h should be changed. #define HAVE_ADDRINFO 1 #define HAVE_SOCKADDR_STORAGE 1 Sure there are warnings about the redefinition of the _(SGIAPI|NO_XOPEN4|NO_XOPEN5) but it compiles and the tests don't fail. Also my applications run without exception or coredump on FUEL, Octane, Octane 2. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-08-01 05:10 Message: Logged In: YES user_id=33168 Ralf, I will try to work on this, but it won't be for a week or two probably. Can you tell me which machines (by name) work and doesn't work. The only IRIX machine I seem to know about it rattler. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-08-01 00:34 Message: Logged In: YES user_id=71407 For the records: The Python2.3 (final) socket module still does not compile under IRIX 6.5.20. It only compiles under 6.5.10. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 11:07 Message: Logged In: YES user_id=31392 I applied a slight variant of Neal's patch and that seemed to fix things. It still seems wrong that we define _SGIAPI, but that will have to wait for 2.3.1. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 06:54 Message: Logged In: YES user_id=31392 I learned a little about inet_ntop() in the last few minutes. It looks like the size_t argument is an older spelling, and the current POSIX says it should be socklen_t. Regardless, it looks like we're using the new definition when IRIX is actually providing the older definition. Neal attached a patch that looks like it should clean up the problem. Have you tried it? I think a better solution would be to teach configure how to find inet_ntop() via netdb.h, but that's probably too hard to get done in the new couple of hours. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 06:42 Message: Logged In: YES user_id=31392 There seem to be two causes of worry. The actual compiler error in your build log and the use of _SGIAPI that Casavanti warns again. It looks like the compiler problems is because our inet_ntop() prototype is wrong. That is Python says the last argument is a socklen_t, but IRIX and Linux man pages both say size_t. Perhaps size_t and socklen_t are different sizes on IRIX, but not on Linux. What happens if you change it to size_t? The problematic code would appear to be these lines from socketmodule.c: #if defined(__sgi)&&_COMPILER_VERSION>700 && !_SGIAPI /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #define _SGIAPI 1 #include "netdb.h" #endif It's the only use the _SGIAPI symbol that Casavant says we shouldn't use. What happens if you remove the use _SGIAPI, i.e. just make it #if defined(__sgi) && _COMPILER_VERSION > 700 /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #include "netdb.h" #endif ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-16 17:27 Message: Logged In: YES user_id=71407 We have confirmed that the socket module in Python 2.3b1 and 2.3b2 fails to compile under these versions of IRIX: 6.5.10, MIPSpro 7.3.1.2m 6.5.19, MIPSpro 7.3.1.3m 6.5.20, MIPSpro 7.3.1.3m (latest available, to my knowledge) If Python 2.3 goes out without a working socket module it will be useless for many people in science where IRIX is still found quite frequently. Maybe even worse, we will also no longer be able to advertise Python as a language that "runs everywhere." ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-15 15:37 Message: Logged In: YES user_id=71407 Date: Tue, 15 Jul 2003 15:06:25 -0500 From: "Brent Casavant" To: "Ralf W. Grosse-Kunstleve" Subject: Re: IRIX & Python Ralf, I read through the sourceforge report. I don't have a sourceforge account so I didn't add there, but feel free to distribute my ramblings. First, they should be very careful about what version of the operating system they are using. Real IPv6 support was only added a few releases ago (6.5.20 I believe, though my memory is hazy). Some of the functions/macros corresponding to IPv6 were introduced prior to that point, but implementation may not have been complete. If they have to play tricks with #define values to get a symbol recongnized it most likely means that SGI didn't intend for that symbol to be used yet. I suspect the problems they are seeing are because they are using some version of IRIX 6.5.x that includes inet_ntoa() definitions under _SGIAPI, but for which real support was not yet complete. The namespace of symbols that begin with a single underscore and a capital letter is reserved for the OS vendors, and the fact that someone is trying to manually insert a definition of _SGIAPI into the source should be a big red warning flag. That said, there is a correct method to deal with the new capabilities of IRIX 6.5.x over time. This comes in two parts. A better explanation is in /usr/include/optional_sym.h First, when a new symbol is added to IRIX, the corresponding header file will always have the following line in it: #pragma optional some_new_symbol The presence of this #pragma will cause the linker to not complain if the symbol is missing when the executable is linked. If SGI neglects to do so it is a bug that should be reported to us. The second part is that calls to these "optional" symbols should be protected by a run-time check to see if the symbol is actually present. Just like this: #include #include . . . if (_MIPS_SYMBOL_PRESENT(some_new_symbol)) { /* New function is present, use it */ qux = some_new_symbol(foo, bar, baz); } else { /* New function isn't present, do something else */ qux = builtin_some_new_symbol(foo, bar, baz); } This ensures that a single executable will be able to function properly (or at least degrade gracefully) on all version of IRIX 6.5.x. In other words you can compile on 6.5.20 and run on 6.5.8, or vice versa. In particular, this means that compile-time detection of features such as inet_ntoa() is incorrect for IRIX 6.5.x -- these decisions should be made at run-time instead. If the decision is made at compile time the executable may well not execute on an older version of IRIX 6.5.x. Unfortunately GNU configure is not equipped to utilize such advanced binary-compatability mechanisms, so there's no "easy" way to get GNU configure'd software to run across all IRIX 6.5.x releases, short of compiling only on the original non-upgraded version of IRIX 6.5. Hope that helps, Brent Casavant On Tue, 15 Jul 2003, Ralf W. Grosse-Kunstleve wrote: > We are developing a Python-based (www.python.org) application for > IRIX. Until now the Python IRIX port was fully supported, but the last > two beta releases of Python (2.3b1 and 2.3b2) make me increasingly > worried because the crucial socket module (TCP/IP sockets) fails to > build. A while ago I've filed a bug report, but the Python developers > appear to be stuck: > > https://sourceforge.net/tracker/? func=detail&aid=728330&group_id=5470&atid=105470 > > Is there a way to bring this to the attention of developers at SGI to > hopefully get some help? > > Thank you in advance! > Ralf > -- Brent Casavant Silicon Graphics, Inc. Where am I? And what am I IRIX O.S. Engineer http://www.sgi.com/ doing in this handbasket? bcasavan@sgi.com 44.8562N 93.1355W 860F -- Unknown ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-06-30 07:21 Message: Logged In: YES user_id=71407 The socket module still fails to compile with Python 2.3b2, in addition to two other extensions. The full make log is attached. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-21 23:47 Message: Logged In: YES user_id=21627 I think it would be best if detect presence of inet_aton/inet_pton. Is IPv6 support possible on this system? Then it would be best if that was detected, as well. If it is not possible to detect aton/pton/ipv6, what is the problem with not using them? I'd like to have a look at the relevant headers (in particular to understand the duplicate definition of _SGIAPI), indeed. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-05-21 16:35 Message: Logged In: YES user_id=33168 Argh!!! This is a nightmare. The attached hack ^h^h^h patch fixes the problem, but ... _SGIAPI needs to be defined (~192 in socketmodule.c) in order to get the prototypes for inet_aton, inet_pton. But _SGIAPI can't be defined in the standard header files because XOPEN_SOURCE is defined. Martin, do you have any other suggestions? I don't see how to do this right within configure. If you want I can send you the header file(s). ---------------------------------------------------------------------- Comment By: alexandre gillet (gillet) Date: 2003-05-12 10:41 Message: Logged In: YES user_id=150999 I had the same problem compiling socket on Irix. We did find that on our system ENABLE_IPV6 is not define and all the error are cause by that. System that do not enable IPV6 are not well supported with the socketmodule.c code. We were able to compile socket after making few change in the code: I am not sure if I did it right but it compile. I will post my patch but I do not see any place to attach my file example: See line 2794,2800 + #ifdef ENABLE_IPV6 char packed[MAX(sizeof(struct in_addr), sizeof(struct in6_addr))]; + #else + char packed[sizeof(struct in_addr)]; + #endif I am not sure how to post the patch or file. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-05-05 11:44 Message: Logged In: YES user_id=71407 This is not in my area of expertise, but I've tried a couple of (probably stupid) things: #include in socketmodule.c instead of #define _SGIAPI. This resulted in even more errors. ./configure --disable-ipv6 Same errors as before. I am lost here. Sorry. But you still have ssh access to rattler.lbl.gov if that helps. Accounts for other Python developers are a possibility. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-04 05:39 Message: Logged In: YES user_id=21627 Can you analyse these compiler errors in more detail, please? What, in your opinion, is causing these problems, and how would you correct them? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 From noreply at sourceforge.net Fri May 21 04:46:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 22 05:46:49 2004 Subject: [ python-Bugs-957381 ] bdist_rpm fails on redhat9, fc1, fc2 Message-ID: Bugs item #957381, was opened at 2004-05-20 13:05 Message generated for change (Comment added) made by jeremysanders You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957381&group_id=5470 Category: Distutils Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jeff Epler (jepler) Assigned to: Nobody/Anonymous (nobody) Summary: bdist_rpm fails on redhat9, fc1, fc2 Initial Comment: distutils bdist_rpm has long been broken for recent versions of RPM (RedHat 9, Fedora Core 1 and 2) https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=88616 When an RPM contains an executable or shared library, a "-debuginfo" rpm is generated. For instance, the following two packages would be generated: foo-1.0.0-1.i386.rpm foo-debuginfo-1.0.0.1-i386.rpm When distutils is faced with this problem, it prints an error like AssertionError: unexpected number of RPM files found: ['build/bdist.linux-i686/rpm/RPMS/i386/foo-1.0.0-1.i386.rpm', build/bdist.linux-i686/rpm/RPMS/i386/foo-debuginfo-1.0.0-1.i386.rpm'] The bugzilla bug contains a proposed patch, but redhat/fedora developers chose not to accept it for their own build of Python. ---------------------------------------------------------------------- Comment By: Jeremy Sanders (jeremysanders) Date: 2004-05-21 08:46 Message: Logged In: YES user_id=8953 I've opened a bugzilla report for fedora 2 See https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=123598 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957381&group_id=5470 From noreply at sourceforge.net Fri May 21 02:00:02 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 22 07:34:28 2004 Subject: [ python-Feature Requests-935915 ] os.nullfilename Message-ID: Feature Requests item #935915, was opened at 2004-04-15 15:44 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=935915&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: John Belmonte (jbelmonte) Assigned to: Nobody/Anonymous (nobody) Summary: os.nullfilename Initial Comment: Just as the os library provides os.sep, etc., for the current OS, it should provide the name of the null file (e.g., "/dev/null" or "nul:"), so that there is a portable way to open a null file. Use of an object such as class nullFile: def write(self, data): pass is not sufficient because it does not provide a full file object interface (no access to file descriptor, etc.). See discussion at . ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-21 01:00 Message: Logged In: YES user_id=80475 Consider mentioning this on comp.lang.python. Perhaps someone will volunteer to write a patch. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-09 11:08 Message: Logged In: YES user_id=21627 Would you like to work on a patch? ---------------------------------------------------------------------- Comment By: David Albert Torpey (dtorp) Date: 2004-05-08 20:54 Message: Logged In: YES user_id=681258 I like this idea. It is more portable. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-04-15 15:52 Message: Logged In: YES user_id=21627 Move to feature requests tracker ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=935915&group_id=5470 From noreply at sourceforge.net Fri May 21 02:05:40 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 22 07:35:47 2004 Subject: [ python-Feature Requests-925537 ] dir(mod) OK or use vars(mod).keys()? Message-ID: Feature Requests item #925537, was opened at 2004-03-29 14:28 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=925537&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 4 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: dir(mod) OK or use vars(mod).keys()? Initial Comment: The documentation on dir() notes that its behavior may change across releases. Several library modules use dir(x) in ways that might break if it changed too much. Should these be changed to use vars(obj).keys() (and possibly sort()), or should the note on dir() be removed? My own preference would just be to provide some guidance, such as "The output of dir() will always include all public variables which do not have a magic meaning." I realize that the standard library itself could be updated if dir() changes in an uncomfortable way. My real concern is which spelling to use in my own code. The library examples suggest a simpler (and clearer) dir(), but the documentation still says otherwise. A quick search for modules using dir() showed possible trouble in at least cgitb, cmd, FCNTL, inspect, optparse, os, pickle, rlcompleter, SimpleXMLRPCServer, TERMIOS, tokenize, unittest, and urllib2. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-21 01:05 Message: Logged In: YES user_id=80475 One possible change is in the works. Someone, suggested that dir() starting including the __name__ attribute. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-03-30 19:51 Message: Logged In: YES user_id=31435 Ya, dir() was *intended* to be a convenience for use at the interactive shell. I expect the only promise Guido wants to make about it is that it will show him stuff he finds interesting as Python changes <0.7 wink>. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-03-30 16:09 Message: Logged In: YES user_id=21627 Ok, marking it as a feature request then: Explicitly pointing out that behaviour may change in the future has usually been done because change in behaviour is anticipated, or has happened in the past. Giving stronger guarantees about future versions a feature that is currently not provided. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-03-30 10:28 Message: Logged In: YES user_id=764593 The ideal fix would be in documentation (and therefore policy). I want to do something like import_list = [x for x in dir(other_mod) if wanted(x)] It happens to work today on my machine, but the documentation says that dir(x) may return different results in different releases. This means that I can't safely use it in released code. If there were some indication that dir(module) would continue to return the same names as vars(module).keys(), then I could use it. If the differences were only in variables not intended for export, that would also be OK -- but dir does not current promise this. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-03-29 17:11 Message: Logged In: YES user_id=21627 I fail to see a bug. Which specific usage of dir() does not work as intended? ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-03-29 14:58 Message: Logged In: YES user_id=764593 Correction: "The output of dir (module) will always include ..." The most common use is figuring out what to do with (import from) a module. Symmetry suggests the same for a no-argument call or a class or type object, but object instances do not use their __dict__ for their attributes; the results are already different there. -jJ ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-03-29 14:39 Message: Logged In: YES user_id=764593 If there concern is over attributes that are "public" but don't exist until called, then most uses of dir and vars are already broken. An alternative reasonable promise would be The output of dir(obj) will always include all (public?, non-magical?) names in vars(obj).keys(). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=925537&group_id=5470 From noreply at sourceforge.net Fri May 21 02:09:00 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 22 07:36:21 2004 Subject: [ python-Feature Requests-906746 ] int object need more informative error message Message-ID: Feature Requests item #906746, was opened at 2004-02-28 22:14 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=906746&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: George Yoshida (quiver) Assigned to: Nobody/Anonymous (nobody) Summary: int object need more informative error message Initial Comment: When int objects encounter a NotImplementedError, they don't say much about what the problem is and are too unfriendly. Error messages can be more verbose. >>> reduce(int.__add__, (1, 2L)) NotImplemented >>> reduce(int.__add__, (1L, 2)) Traceback (most recent call last): File "", line 1, in ? TypeError: descriptor '__add__' requires a 'int' object but received a 'long' ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-21 01:09 Message: Logged In: YES user_id=80475 Unfortunately, there may not be any flexibility here. Instead of being just an error message, NotImplemented is a constant that has meaning to other layers of code. I don't see a way to change this or even attach a more informative error message. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=906746&group_id=5470 From noreply at sourceforge.net Fri May 21 02:17:07 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 22 07:43:01 2004 Subject: [ python-Feature Requests-894936 ] Have a split corresponding with os.path.join Message-ID: Feature Requests item #894936, was opened at 2004-02-11 11:36 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=894936&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Nicolas Fleury (nidoizo) Assigned to: Nobody/Anonymous (nobody) Summary: Have a split corresponding with os.path.join Initial Comment: I would be nice to have a function to do the exact opposite of os.path.join, something like an os.path.splitall or fullsplit. Rationale: Currently, on Windows, using os.path.split produce the following result: os.path.split('Z:/xyz') => ('Z:/', 'xyz') It exceptionaly keep the \ after the drive. This exception makes str.split incompatible with os.path.join that is expecting the \. Spliting fully a path is useful when transforming absolute paths to relative paths. I could also be nice to have a function to do that in os.path. ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-21 01:17 Message: Logged In: YES user_id=80475 Do you have a patch in mind? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=894936&group_id=5470 From noreply at sourceforge.net Fri May 21 07:46:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 22 07:59:39 2004 Subject: [ python-Bugs-957381 ] bdist_rpm fails on redhat9, fc1, fc2 Message-ID: Bugs item #957381, was opened at 2004-05-20 08:05 Message generated for change (Comment added) made by jepler You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957381&group_id=5470 Category: Distutils Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jeff Epler (jepler) Assigned to: Nobody/Anonymous (nobody) Summary: bdist_rpm fails on redhat9, fc1, fc2 Initial Comment: distutils bdist_rpm has long been broken for recent versions of RPM (RedHat 9, Fedora Core 1 and 2) https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=88616 When an RPM contains an executable or shared library, a "-debuginfo" rpm is generated. For instance, the following two packages would be generated: foo-1.0.0-1.i386.rpm foo-debuginfo-1.0.0.1-i386.rpm When distutils is faced with this problem, it prints an error like AssertionError: unexpected number of RPM files found: ['build/bdist.linux-i686/rpm/RPMS/i386/foo-1.0.0-1.i386.rpm', build/bdist.linux-i686/rpm/RPMS/i386/foo-debuginfo-1.0.0-1.i386.rpm'] The bugzilla bug contains a proposed patch, but redhat/fedora developers chose not to accept it for their own build of Python. ---------------------------------------------------------------------- >Comment By: Jeff Epler (jepler) Date: 2004-05-21 06:46 Message: Logged In: YES user_id=2772 the "WONTFIX" closure of bugzilla 88616 was already from Fedora days (2004-04-05), so opening a new bug report is unlikely to do much on its own. (in fact, I don't know if it's likely to do more than get closed as a duplicate) Of course, I don't speak for Fedora. If a fix for this new RPM feature is included in Python (for 2.3.5 and 2.4) then I'd guess it's more likely to be added as a patch for a subsequent 2.3.3 or 2.3.4-based Python package, but again I don't speak for the Fedora developers. ---------------------------------------------------------------------- Comment By: Jeremy Sanders (jeremysanders) Date: 2004-05-21 03:46 Message: Logged In: YES user_id=8953 I've opened a bugzilla report for fedora 2 See https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=123598 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957381&group_id=5470 From noreply at sourceforge.net Sat May 22 12:53:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 22 12:53:19 2004 Subject: [ python-Bugs-215126 ] Over restricted type checking on eval() function Message-ID: Bugs item #215126, was opened at 2000-09-22 14:36 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=215126&group_id=5470 Category: Python Interpreter Core Group: Feature Request Status: Open >Resolution: None Priority: 5 Submitted By: Nobody/Anonymous (nobody) Assigned to: Raymond Hettinger (rhettinger) Summary: Over restricted type checking on eval() function Initial Comment: The built-in function eval() takes a string argument and a dictionary. The second argument should allow any instance which defines __getitem__ as opposed to just dictionaries. The following example creates a type error: eval, argument 2: expected dictionary, instance found class SpreadSheet: _cells = {} def __setitem__( self, key, formula ): self._cells[key] = formula def __getitem__( self, key ): return eval( self._cells[key], self ) ss = SpreadSheet() ss['a1'] = '5' ss['a2'] = 'a1*5' ss['a2'] ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-22 11:53 Message: Logged In: YES user_id=80475 +1 Amrin's idea provides most of the needed functionality with zero performance impact. Also, using a local dictionary for the application variables is likely to be just exactly what a programmer would want to do. ---------------------------------------------------------------------- Comment By: Armin Rigo (arigo) Date: 2004-05-19 13:21 Message: Logged In: YES user_id=4771 With minimal work and performance impact, we could allow frame->f_locals to be something else than a real dictionary. It looks like it would be easily possible as long as we don't touch the time-critical frame->f_globals. Code compiled by eval() always uses LOAD_NAME and STORE_NAME, which is anyway allowed to be a bit slower than LOAD_FAST or LOAD_GLOBAL. Note that PyDict_GetItem() & co., as called by LOAD/STORE_NAME, do a PyDict_Check() anyway. We could just replace it with PyDict_CheckExact() and if it fails fall back to calling the appropriate ob_type->tp_as_mapping method. This would turn some of the PyDict_Xxx() API into a general mapping API, half-way between its current dict-only usage and the fully abstract PyObject_Xxx() API. This is maybe a bit strange from the user's point of view, because eval("expr", mydict) still wouldn't work: only eval("expr", {}, mydict) would. which does which does Now, there is no way I can think of that code compiled by eval() could contain LOAD_GLOBAL or STORE_GLOBAL. The only way to tell the difference between eval("expr", mydict) and eval("expr", {}, mydict) appears to be by calling globals() or somehow inspecting the frame directly. Therefore it might be acceptable to redefine the two-argument eval(expr, dict) to be equivalent not to eval(expr, dict, dict) but eval(expr, {}, dict). This hack might be useful enough even if it won't work with the exec statement. ---------------------------------------------------------------------- Comment By: Philippe Fremy (pfremy) Date: 2004-03-03 04:40 Message: Logged In: YES user_id=233844 I have exactly the same need as the original poster. The only difference in my case is that I inhreted a dictionnary. I want to use the eval() function as a simple expression evaluator. I have the follwing dictionnary: d['a']='1' d['b']='2' d['c']='a+b' I want the following results: d[a] -> 1 d[b] -> 2 d[c] -> 3 To do that, I was planning to use the eval() function and overloading the __getitem__ of the global or local dictionnary: class MyDict( dict ) : def __getitem__( self, key ): print "__getitem__", key val = dict.__getitem__( self, key ) print "val = '%s'" % val return eval( val , self ) But it does not work: d[a]: __getitem__ a val = '1' -> 1 d[b]: __getitem__ b val = '2' -> 2 d[c]: __getitem__ c val = 'e+1' ERROR Traceback (most recent call last): File "test_parse_jaycos_config.py", line 83, in testMyDict self.assertEquals( d['c'], 2 ) File "parse_config_file.py", line 10, in __getitem__ return eval( val , self ) File "", line 0, in ? TypeError: cannot concatenate 'str' and 'int' objects d['c'] did fetch the 'a+1' value, which was passed to eval. However, eval() tried to evaluate the expression using the content of the dictionnary instead of using __getitem__. So it fetched the string '1' for 'a' instead of the value 1, so the result is not suitable for the addition. ---------------------------------------------------------------------- Comment By: Michael Chermside (mcherm) Date: 2004-01-12 17:01 Message: Logged In: YES user_id=99874 Hmm... I like this! Of course, I am wary of adding *yet another* special double- underscore function to satisfy a single special purpose, but this would satisfy all of *my* needs, and without any performance impact for lookups that are FOUND. Lookups that are NOT found would have a slight performance degrade (I know better than to speculate about the size of the effect without measuring it). I'm not really sure what percentage of dict lookups succeed. At any rate, what are these "other contexts" you mention in which a __keyerror__ would "also be useful"? Because if we can find other places where it is useful, that significantly improves the usefulness. OTOH, if the tests can be done ONLY for eval (I don't really think that's possible), then I'm certain no one cares about the performance of eval. ---------------------------------------------------------------------- Comment By: Gregory Smith (gregsmith) Date: 2004-01-12 14:38 Message: Logged In: YES user_id=292741 This may be a compromise solution, which could also be useful in other contexts: What if the object passed is derived from dict - presumably that doesn't help because the point is to do low-level calls to the actual dict's lookup functions. Now, suppose we modify the basic dict type, so that before throwing a KeyError, it checks to see if it is really a derived object with a method __keyerror__, and if so, calls that and returns its result (or lets it throw)? Now you can make objects that look like dicts, and act like them at the low level, but can automatically populate themselves when non-existent keys are requested. Of course, __keyerror__('x') does not have to make an 'x' entry in the dict; it could make no change, or add several entries, depending on the desired semantics regarding future lookups. It could be set up so that every lookup fails and is forwarded by __keyerror__ to the __getitem__ of another object, for instance. The cost of this to the 'normal' dict lookup is that the need to do PyDict_CheckExact() each time a lookup fails. ---------------------------------------------------------------------- Comment By: Michael Chermside (mcherm) Date: 2003-12-16 10:37 Message: Logged In: YES user_id=99874 I'll just add my voice as somebody who would find this to be "darn handy" if it could ever done without noticably impacting the speed of python code that DOESN'T use eval(). ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2000-09-22 23:18 Message: Added to PEP 42. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2000-09-22 14:42 Message: Changed Group to Feature Request. Should be added to PEP 42 (I'll do that if nobody beats me to it). CPython requires a genuine dict now for speed. I believe JPython allows any mapping object. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=215126&group_id=5470 From noreply at sourceforge.net Sat May 22 12:53:42 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 22 12:53:50 2004 Subject: [ python-Bugs-958180 ] logging functionality non-intuitive, levels confusing Message-ID: Bugs item #958180, was opened at 2004-05-21 13:56 Message generated for change (Settings changed) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=958180&group_id=5470 Category: Python Library Group: Python 2.2.3 Status: Open Resolution: None >Priority: 6 Submitted By: Jason R. Coombs (jaraco) >Assigned to: Vinay Sajip (vsajip) Summary: logging functionality non-intuitive, levels confusing Initial Comment: If one tries to define additional logging levels without overriding the logger class, one runs into trouble with the log level naming conventions. In particular, I observe the following strange behavior: >>> import logging >>> logging.basicConfig() >>> logging.root.level 30 >>> logging.root.log( logging.INFO, 'test1' ) >>> logging.root.log( 20, 'test1' ) >>> logging.root.log( 'info', 'test1' ) Level info:root:test1 >>> logging.root.log( 'INFO', 'test1' ) 20:root:test1 >>> logging.root.log( logging.WARNING, 'warning' ) WARNING:root:warning >>> logging.root.log( 30, 'warning' ) WARNING:root:warning >>> logging.root.log( 'warning', 'warning' ) Level warning:root:warning >>> logging.root.log( 'WARNING', 'warning' ) 30:root:warning Note that if a string parameter is passed as the lvl parameter of the log() method, the message is logged regardless of the current logging level, and the output is furthermore inconsistent. Additionally, when the user attempts to add a logging level, the user cannot reference that log level by name using any exposed (documented) interfaces. For example: >>> import logging >>> logging.basicConfig() >>> logging.root.level = logging.DEBUG # 10 >>> logging.addLevelName( 15, 'STATUS' ) >>> logging.root.log( 15, 'a status message' ) STATUS:root:a status message >>> logging.root.log( logging._levelNames['STATUS'], 'a status message' ) STATUS:root:a status message The only interface, as far as I can tell, to access the log level is to explicitly query the internal data structures. Changes I propose: (1) The log() methodshould either throw a TypeError when a string is passed for the log level or (better) it should accept string values and translate them transparently to the integer log levels that it currently expects. (2) The logging module should include functionality to allow the user to query a logging level from a name. def getLevelNumber( levelName ): return _levelNames[ levelName ] # error checking would be appropriate ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=958180&group_id=5470 From noreply at sourceforge.net Sat May 22 13:23:09 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 22 13:23:13 2004 Subject: [ python-Feature Requests-878458 ] optparse help text is not easily extensible Message-ID: Feature Requests item #878458, was opened at 2004-01-16 13:59 Message generated for change (Settings changed) made by gward You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=878458&group_id=5470 >Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Ian Bicking (ianbicking) >Assigned to: Nobody/Anonymous (nobody) Summary: optparse help text is not easily extensible Initial Comment: optparse doesn't allow for paragraph/narrative help text. This text would typically go below the option listing. There is currently no place to describe positional arguments, and this would be a simple way to allow for that. This could be provided as an extra keyword argument to OptionParser.__init__, included in the help in the function OptionParser.format_help. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=878458&group_id=5470 From noreply at sourceforge.net Sun May 23 03:58:19 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 23 03:58:24 2004 Subject: [ python-Bugs-944119 ] Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty Message-ID: Bugs item #944119, was opened at 2004-04-29 11:05 Message generated for change (Comment added) made by aimacintyre You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944119&group_id=5470 Category: None Group: Platform-specific Status: Open Resolution: None Priority: 5 Submitted By: Steve Levis (levis501) Assigned to: Nobody/Anonymous (nobody) Summary: Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty Initial Comment: The attached python file reproduces a bug whereby Python does not receive alarm signals from FreeBSD 4.9. This also appears to be the case in FreeBSD 4.8, and may be related to the addition of HyperThreading in FreeBSD 4.8. If Python (either 2.2.3 or 2.3.3) is configured with -?without-threads before compiling, the bug does not exist. ---------------------------------------------------------------------- >Comment By: Andrew I MacIntyre (aimacintyre) Date: 2004-05-23 17:58 Message: Logged In: YES user_id=250749 There are a few things I haven't been able to figure out with this problem, but I think this is what's going on: - on FreeBSD the read(2) call is not interruptable, as ptys appear not to be considered "slow devices" by the kernel (whether this behavior is *BSD specific I don't really know; ordinary files and pipes aresimilarly affected so forkpty() is just the messenger here); - Python's signal handling is documented to behave such that signal handlers only get invoked between Python VM opcodes. As the os.read() call is a single VM instruction, the nett effect is that the signal handler will not execute until the os.read() call/instruction completes... which it never does. This also affects other signals, such as SIGINT :-( The signal handler being invoked between opcodes explains why the C version works and the Python version doesn't. Using select.select() (or select.poll()) is one way of avoiding this trap, as both select(2) and poll(2) are interruptable (in addition to supporting timeouts directly). I expect this approach would also be considered to be generally portable. I haven't dug into why a single-threaded build avoids the Python VM opcode scheduling - I can only speculate that because the VM doesn't have any thread locking, it can actually execute the signal handler while the os.read() call is still pending. I don't expect there to be a practical solution to this, and so I'm suggesting that the bug be closed as "Won't fix" (read "Can't fix"). ---------------------------------------------------------------------- Comment By: Steve Levis (levis501) Date: 2004-05-19 05:55 Message: Logged In: YES user_id=309698 I've uploaded failure_demo.c, which can be compiled with gcc -o failure_demo failure_demo.c -lutil. Written in C, the signal is properly delivered, and the test does not fail on either FreeBSD 4.7 or 4.9. ---------------------------------------------------------------------- Comment By: Andrew I MacIntyre (aimacintyre) Date: 2004-05-18 19:05 Message: Logged In: YES user_id=250749 Hmm... are you in a position to write a C test case for this? The reason I ask is that I have seen other signal related problems involving a Python interpreter built --with-threads on FreeBSD 4.x which were traced back to issues with libc_r, and I suspect that this problem is similar. ---------------------------------------------------------------------- Comment By: Andrew I MacIntyre (aimacintyre) Date: 2004-05-18 19:03 Message: Logged In: YES user_id=250749 Hmm... are you in a position to write a C test case for this? The reason I ask is that I have seen other signal related problems involving a Python interpreter built --with-threads on FreeBSD 4.x which were traced back to issues with libc_r, and I suspect that this problem is similar. ---------------------------------------------------------------------- Comment By: Steve Levis (levis501) Date: 2004-05-06 03:08 Message: Logged In: YES user_id=309698 Ooops, again! I ran on the standard kernel but had python compiled --without-threads, so of course it worked. So, to make a long story short, the bug exists -- ignore my last post. ---------------------------------------------------------------------- Comment By: Steve Levis (levis501) Date: 2004-05-06 02:59 Message: Logged In: YES user_id=309698 I was apparently not running on a standard FreeBSD kernel. It's working now. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944119&group_id=5470 From noreply at sourceforge.net Sun May 23 04:11:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 23 04:11:26 2004 Subject: [ python-Bugs-957198 ] C/C++ extensions w/ Python + Mingw (Windows) Message-ID: Bugs item #957198, was opened at 2004-05-20 13:36 Message generated for change (Comment added) made by aimacintyre You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957198&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Connelly (connelly) Assigned to: Nobody/Anonymous (nobody) Summary: C/C++ extensions w/ Python + Mingw (Windows) Initial Comment: I am asking you to distribute libpython23.a with Python. This is a library built by the Mingw compiler (www.mingw.org). When this file is present, it greatly simplifies the process of installing a C/C++ extension. The below document explains how make this library, and install a C/C++ extension with the Mingw compiler. Currently, a library is provided for the proprietary MSVC++ compiler (python23.lib), but not the open source Mingw compiler. Normally, one uses the following procedure to build and install a C/C++ extension: python setup.py build --compiler=your_compiler python setup.py install For Python 2.3.3 on Windows, with the Mingw (Minimalist GNU) compiler, the following steps must be taken: 1. Find your Mingw bin directory. Copy gcc.exe to cc.exe. 2. Get PExports from either of: http://sebsauvage.net/python/pexports-0.42h.zip http://starship.python.net/crew/kernr/mingw32/pexports-0.42h.zip Extract pexports.exe to your Mingw bin directory. 3. Find pythonxx.dll. It should be in your main Python directory. Do the following: pexports python23.dll > python23.def dlltool --dllname python23.dll --def python23.def --output-lib libpython23.a 4. Copy libpythonxx.a to \python\libs. 5. Patch distutils. Locate \python\lib\distutils\msvccompiler.py, open it, and find the following lines (around line 211): if len (self.__paths) == 0: raise DistutilsPlatformError, ("Python was built with version %s of Visual Studio, " "and extensions need to be built with the same " "version of the compiler, but it isn't installed." % self.__version) Delete these. 6. Move back to the directory of your extension. Do the following: python setup.py build --compiler=mingw32 python setup.py install Ideally, only step 6 should be required to install an extension. I submitted the patch for step 5 to python.sourceforge.net. Steps 2-4 can be avoided if the libpythonxx.a file is distributed with Python. Step 1 can probably be avoided with another patch. This document is based on http://sebsauvage.net/python/mingw.html, which was written for Mingw + Python 2.2. Thanks, Connelly ---------------------------------------------------------------------- >Comment By: Andrew I MacIntyre (aimacintyre) Date: 2004-05-23 18:11 Message: Logged In: YES user_id=250749 While older releases of MinGW certainly required this library fiddling, such as the MingW 1.1 pkg with gcc 2.95, I had been under the impression that recent versions could quite happily use the MSVC python23.lib. If you have tried this and had it fail, which version of MinGW? Using the --compiler=mingw32 build option should not require patching Distutils, though you probably have to use it for both build and install invocations. You can just issue to the install command (with the --compiler switch) which will both build and install the extention. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957198&group_id=5470 From noreply at sourceforge.net Mon May 24 04:01:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 24 04:01:38 2004 Subject: [ python-Bugs-959291 ] PythonIDE crashes on very large scripts folder Message-ID: Bugs item #959291, was opened at 2004-05-24 10:01 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=959291&group_id=5470 Category: Macintosh Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jack Jansen (jackjansen) Assigned to: Jack Jansen (jackjansen) Summary: PythonIDE crashes on very large scripts folder Initial Comment: The MacPython IDE crashes if your scripts folder is very large (or, alternatively, if you select a very large folder as the scripts folder), and here "large" counts files in subdirs, etc. The problem is that we have a fixed number of menu entry IDs, and we run out of them. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=959291&group_id=5470 From noreply at sourceforge.net Mon May 24 07:32:13 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 24 07:32:19 2004 Subject: [ python-Bugs-959379 ] Implicit close() should check for errors Message-ID: Bugs item #959379, was opened at 2004-05-24 13:32 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=959379&group_id=5470 Category: Python Interpreter Core Group: Python 2.2.3 Status: Open Resolution: None Priority: 5 Submitted By: Peter ?strand (astrand) Assigned to: Nobody/Anonymous (nobody) Summary: Implicit close() should check for errors Initial Comment: As we all know, the fileobjects destructor invokes the close() method automatically. But, most people are not aware of that errors from close() are silently ignored. This can lead to silent data loss. Consider this example: $ python -c 'open("foo", "w").write("aaa")' No traceback or warning message is printed, but the file is zero bytes large, because the close() system call returned EDQUOT. Another similiar example is: $ python -c 'f=open("foo", "w"); f.write("aaa")' When using an explicit close(), you get a traceback: $ python -c 'f=open("foo", "w"); f.write("aaa"); f.close()' Traceback (most recent call last): File "", line 1, in ? IOError: [Errno 122] Disk quota exceeded I'm aware of that exceptions cannot be raised in destructors, but wouldn't it be possible to at least print a warning message? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=959379&group_id=5470 From noreply at sourceforge.net Mon May 24 10:21:28 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 24 10:21:35 2004 Subject: [ python-Bugs-958180 ] logging functionality non-intuitive, levels confusing Message-ID: Bugs item #958180, was opened at 2004-05-21 18:56 Message generated for change (Comment added) made by vsajip You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=958180&group_id=5470 Category: Python Library Group: Python 2.2.3 Status: Open Resolution: None Priority: 6 Submitted By: Jason R. Coombs (jaraco) Assigned to: Vinay Sajip (vsajip) Summary: logging functionality non-intuitive, levels confusing Initial Comment: If one tries to define additional logging levels without overriding the logger class, one runs into trouble with the log level naming conventions. In particular, I observe the following strange behavior: >>> import logging >>> logging.basicConfig() >>> logging.root.level 30 >>> logging.root.log( logging.INFO, 'test1' ) >>> logging.root.log( 20, 'test1' ) >>> logging.root.log( 'info', 'test1' ) Level info:root:test1 >>> logging.root.log( 'INFO', 'test1' ) 20:root:test1 >>> logging.root.log( logging.WARNING, 'warning' ) WARNING:root:warning >>> logging.root.log( 30, 'warning' ) WARNING:root:warning >>> logging.root.log( 'warning', 'warning' ) Level warning:root:warning >>> logging.root.log( 'WARNING', 'warning' ) 30:root:warning Note that if a string parameter is passed as the lvl parameter of the log() method, the message is logged regardless of the current logging level, and the output is furthermore inconsistent. Additionally, when the user attempts to add a logging level, the user cannot reference that log level by name using any exposed (documented) interfaces. For example: >>> import logging >>> logging.basicConfig() >>> logging.root.level = logging.DEBUG # 10 >>> logging.addLevelName( 15, 'STATUS' ) >>> logging.root.log( 15, 'a status message' ) STATUS:root:a status message >>> logging.root.log( logging._levelNames['STATUS'], 'a status message' ) STATUS:root:a status message The only interface, as far as I can tell, to access the log level is to explicitly query the internal data structures. Changes I propose: (1) The log() methodshould either throw a TypeError when a string is passed for the log level or (better) it should accept string values and translate them transparently to the integer log levels that it currently expects. (2) The logging module should include functionality to allow the user to query a logging level from a name. def getLevelNumber( levelName ): return _levelNames[ levelName ] # error checking would be appropriate ---------------------------------------------------------------------- >Comment By: Vinay Sajip (vsajip) Date: 2004-05-24 14:21 Message: Logged In: YES user_id=308438 Point 1 - agreed. Point 2 - getLevelName() already handles this, as it adds both name->value and value->name mappings to _levelNames. The point about type checks is valid. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=958180&group_id=5470 From noreply at sourceforge.net Mon May 24 13:16:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 24 13:17:02 2004 Subject: [ python-Bugs-959576 ] Can't build Python on POSIX w/o $HOME Message-ID: Bugs item #959576, was opened at 2004-05-24 13:16 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=959576&group_id=5470 Category: Build Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Barry A. Warsaw (bwarsaw) Assigned to: Nobody/Anonymous (nobody) Summary: Can't build Python on POSIX w/o $HOME Initial Comment: If you're building Python on os.name == 'posix' but when there is no $HOME defined in your environment, you cannot build Python. This is because in the bowels of distutils, you end up in check_environ(), which has these lines: if os.name == 'posix' and not os.environ.has_key('HOME'): import pwd os.environ['HOME'] = pwd.getpwuid(os.getuid())[5] However, in a from-scratch build, the pwd module isn't built by the time you get here. I found this when using SCons to build Python, since by default the enclosing environment isn't passed to subprocesses. I can work around this in my builds but Python's setup.py should probably synthesize a $HOME if one doesn't exist. (Or maybe someone has a better idea for a workaround). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=959576&group_id=5470 From noreply at sourceforge.net Tue May 25 09:12:43 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 25 09:12:50 2004 Subject: [ python-Feature Requests-894936 ] Have a split corresponding with os.path.join Message-ID: Feature Requests item #894936, was opened at 2004-02-11 11:36 Message generated for change (Comment added) made by nidoizo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=894936&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Nicolas Fleury (nidoizo) Assigned to: Nobody/Anonymous (nobody) Summary: Have a split corresponding with os.path.join Initial Comment: I would be nice to have a function to do the exact opposite of os.path.join, something like an os.path.splitall or fullsplit. Rationale: Currently, on Windows, using os.path.split produce the following result: os.path.split('Z:/xyz') => ('Z:/', 'xyz') It exceptionaly keep the \ after the drive. This exception makes str.split incompatible with os.path.join that is expecting the \. Spliting fully a path is useful when transforming absolute paths to relative paths. I could also be nice to have a function to do that in os.path. ---------------------------------------------------------------------- >Comment By: Nicolas Fleury (nidoizo) Date: 2004-05-25 09:12 Message: Logged In: YES user_id=151545 I have attached a possible implementation (not tested on Unix). Feel free to ask if I can do more. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-21 02:17 Message: Logged In: YES user_id=80475 Do you have a patch in mind? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=894936&group_id=5470 From noreply at sourceforge.net Tue May 25 09:13:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 25 09:13:52 2004 Subject: [ python-Feature Requests-894936 ] Have a split corresponding with os.path.join Message-ID: Feature Requests item #894936, was opened at 2004-02-11 11:36 Message generated for change (Comment added) made by nidoizo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=894936&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Nicolas Fleury (nidoizo) Assigned to: Nobody/Anonymous (nobody) Summary: Have a split corresponding with os.path.join Initial Comment: I would be nice to have a function to do the exact opposite of os.path.join, something like an os.path.splitall or fullsplit. Rationale: Currently, on Windows, using os.path.split produce the following result: os.path.split('Z:/xyz') => ('Z:/', 'xyz') It exceptionaly keep the \ after the drive. This exception makes str.split incompatible with os.path.join that is expecting the \. Spliting fully a path is useful when transforming absolute paths to relative paths. I could also be nice to have a function to do that in os.path. ---------------------------------------------------------------------- >Comment By: Nicolas Fleury (nidoizo) Date: 2004-05-25 09:13 Message: Logged In: YES user_id=151545 I have attached a possible implementation (not tested on Unix). Feel free to ask if I can do more. ---------------------------------------------------------------------- Comment By: Nicolas Fleury (nidoizo) Date: 2004-05-25 09:12 Message: Logged In: YES user_id=151545 I have attached a possible implementation (not tested on Unix). Feel free to ask if I can do more. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-21 02:17 Message: Logged In: YES user_id=80475 Do you have a patch in mind? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=894936&group_id=5470 From noreply at sourceforge.net Tue May 25 14:23:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 25 14:23:43 2004 Subject: [ python-Bugs-957198 ] C/C++ extensions w/ Python + Mingw (Windows) Message-ID: Bugs item #957198, was opened at 2004-05-20 03:36 Message generated for change (Comment added) made by connelly You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957198&group_id=5470 Category: None Group: None Status: Open Resolution: None Priority: 5 Submitted By: Connelly (connelly) Assigned to: Nobody/Anonymous (nobody) Summary: C/C++ extensions w/ Python + Mingw (Windows) Initial Comment: I am asking you to distribute libpython23.a with Python. This is a library built by the Mingw compiler (www.mingw.org). When this file is present, it greatly simplifies the process of installing a C/C++ extension. The below document explains how make this library, and install a C/C++ extension with the Mingw compiler. Currently, a library is provided for the proprietary MSVC++ compiler (python23.lib), but not the open source Mingw compiler. Normally, one uses the following procedure to build and install a C/C++ extension: python setup.py build --compiler=your_compiler python setup.py install For Python 2.3.3 on Windows, with the Mingw (Minimalist GNU) compiler, the following steps must be taken: 1. Find your Mingw bin directory. Copy gcc.exe to cc.exe. 2. Get PExports from either of: http://sebsauvage.net/python/pexports-0.42h.zip http://starship.python.net/crew/kernr/mingw32/pexports-0.42h.zip Extract pexports.exe to your Mingw bin directory. 3. Find pythonxx.dll. It should be in your main Python directory. Do the following: pexports python23.dll > python23.def dlltool --dllname python23.dll --def python23.def --output-lib libpython23.a 4. Copy libpythonxx.a to \python\libs. 5. Patch distutils. Locate \python\lib\distutils\msvccompiler.py, open it, and find the following lines (around line 211): if len (self.__paths) == 0: raise DistutilsPlatformError, ("Python was built with version %s of Visual Studio, " "and extensions need to be built with the same " "version of the compiler, but it isn't installed." % self.__version) Delete these. 6. Move back to the directory of your extension. Do the following: python setup.py build --compiler=mingw32 python setup.py install Ideally, only step 6 should be required to install an extension. I submitted the patch for step 5 to python.sourceforge.net. Steps 2-4 can be avoided if the libpythonxx.a file is distributed with Python. Step 1 can probably be avoided with another patch. This document is based on http://sebsauvage.net/python/mingw.html, which was written for Mingw + Python 2.2. Thanks, Connelly ---------------------------------------------------------------------- >Comment By: Connelly (connelly) Date: 2004-05-25 18:23 Message: Logged In: YES user_id=1039782 I am using Mingw 3.1.0-1, released on Sep 15, 2003. It is the 'current' release of Mingw. I'm using Python 2.3.3. Issuing python setup.py install --compiler=mingw32 causes an error. I'm not sure which error -- I'll post it here tomorrow when I'm at the right machine. So I left off the --compiler option for the install step. This produced the error "Python was built with version %s of Visual Studio, and extensions need to be built with the same version of the compiler, but it isn't installed." This error is not produced if VC++ is installed. Thus you need to find a machine WITHOUT VC++ to test out the build process with Mingw32. I don't know how to tell MinGW to use python23.lib. Perhaps this is all the result of not passing the right flag to 'python setup.py install'? After all, it's using msvccompiler.py, which seems suspicious. I don't think I'm the only one having this problem. See: http://randomthoughts.vandorp.ca/WK/blog/758?t=item ---------------------------------------------------------------------- Comment By: Andrew I MacIntyre (aimacintyre) Date: 2004-05-23 08:11 Message: Logged In: YES user_id=250749 While older releases of MinGW certainly required this library fiddling, such as the MingW 1.1 pkg with gcc 2.95, I had been under the impression that recent versions could quite happily use the MSVC python23.lib. If you have tried this and had it fail, which version of MinGW? Using the --compiler=mingw32 build option should not require patching Distutils, though you probably have to use it for both build and install invocations. You can just issue to the install command (with the --compiler switch) which will both build and install the extention. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957198&group_id=5470 From noreply at sourceforge.net Tue May 25 15:07:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 25 15:07:17 2004 Subject: [ python-Bugs-960325 ] "require " configure option Message-ID: Bugs item #960325, was opened at 2004-05-25 21:07 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960325&group_id=5470 Category: Build Group: Feature Request Status: Open Resolution: None Priority: 5 Submitted By: Hallvard B Furuseth (hfuru) Assigned to: Nobody/Anonymous (nobody) Summary: "require " configure option Initial Comment: I'd like to be able to configure Python so that Configure or Make will fail if a particular feature is unavailable. Currently I'm concerned with SSL, which just gets a warning from Make: building '_ssl' extension *** WARNING: renaming "_ssl" since importing it failed: ld.so.1: ./python: fatal: libssl.so.0.9.8: open failed: No such file or directory Since that's buried in a lot of Make output, it's easy to miss. Besides, for semi-automatic builds it's in any case good to get a non-success exit status from the build process. Looking at the Make output, I see the bz2 extension is another example where this might be useful. Maybe the option would simply be '--enable-ssl', unless you want that to merely try to build with ssl. Or '--require=ssl,bz2,...'. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960325&group_id=5470 From noreply at sourceforge.net Tue May 25 15:30:17 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 25 15:30:21 2004 Subject: [ python-Bugs-960340 ] Poor documentation of new-style classes Message-ID: Bugs item #960340, was opened at 2004-05-25 21:30 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960340&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Hallvard B Furuseth (hfuru) Assigned to: Nobody/Anonymous (nobody) Summary: Poor documentation of new-style classes Initial Comment: The Python Reference Manual (info file python-ref) talks a lot about new-style classes, but does not say what they are, except in a brief note buried in node 'Coercion rules'. The library reference does say that object() creates such classes, that too lacks a way to look up 'new-style classes' and find object(). Also, since 'object' is a type, it seems strange that the Library Reference has it in the 'Built-in Functions' node instead of a node about (callable) types. The same applies to several other types. If you want to keep them there, at least add index entries for them in the Class-Exception-Object Index. This refers to the doc in info-2.3.3.tar.bz2 from . ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960340&group_id=5470 From noreply at sourceforge.net Tue May 25 18:13:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 25 18:14:26 2004 Subject: [ python-Bugs-960340 ] Poor documentation of new-style classes Message-ID: Bugs item #960340, was opened at 2004-05-25 15:30 Message generated for change (Comment added) made by jimjjewett You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960340&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Hallvard B Furuseth (hfuru) Assigned to: Nobody/Anonymous (nobody) Summary: Poor documentation of new-style classes Initial Comment: The Python Reference Manual (info file python-ref) talks a lot about new-style classes, but does not say what they are, except in a brief note buried in node 'Coercion rules'. The library reference does say that object() creates such classes, that too lacks a way to look up 'new-style classes' and find object(). Also, since 'object' is a type, it seems strange that the Library Reference has it in the 'Built-in Functions' node instead of a node about (callable) types. The same applies to several other types. If you want to keep them there, at least add index entries for them in the Class-Exception-Object Index. This refers to the doc in info-2.3.3.tar.bz2 from . ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-25 18:13 Message: Logged In: YES user_id=764593 object() doesn't create a new-style class; it creates an instance of class object. Note that the definition of a new-style class is just a class inheriting from object, so object itself is a new-style class. That said, the distributed documentation should probably have something more about "new-style" vs "old-style" classes, and should have a reference in the index. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960340&group_id=5470 From noreply at sourceforge.net Tue May 25 18:20:53 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 25 18:20:57 2004 Subject: [ python-Bugs-960448 ] grammar for "class" inheritance production slightly wrong Message-ID: Bugs item #960448, was opened at 2004-05-25 18:20 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960448&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: grammar for "class" inheritance production slightly wrong Initial Comment: http://www.python.org/dev/doc/devel/ref/class.html lists the grammar for a class statement, including an optional inheritance production of "(" [expression_list] ")". expression_list is *not* optional. If the inheritance production (the parentheses) is present, then the expression list must be present (non-empty) as well. """ >>> class C(): SyntaxError: invalid syntax """ ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960448&group_id=5470 From noreply at sourceforge.net Tue May 25 18:33:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Tue May 25 18:33:22 2004 Subject: [ python-Feature Requests-960454 ] old/new class documentation Message-ID: Feature Requests item #960454, was opened at 2004-05-25 18:33 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=960454&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: old/new class documentation Initial Comment: The distributed documentation refers to old and new-style classes, but they do not appear in the index, and the difference is not explained. (Users who ask are referred to documentation on www.python.org, which is not currently shipped with the default distrubution.) As best I understand it, the only differences are: (1) New-style classes inherit from object. Because of this inheritance, new-style classes have a few extra capabilities, such as descriptors and super. (2) Method Resolution Order can be different in cases of multiple inheritance. (3) New style classes take precedence over old-style classes when doing rich comparison. (4) If rich comparison fails, numeric coercion will be attempted if -- and only if -- at least one of the objects is an old-style class. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=960454&group_id=5470 From noreply at sourceforge.net Wed May 26 00:45:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 00:46:06 2004 Subject: [ python-Bugs-960340 ] Poor documentation of new-style classes Message-ID: Bugs item #960340, was opened at 2004-05-25 14:30 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960340&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Hallvard B Furuseth (hfuru) Assigned to: Nobody/Anonymous (nobody) Summary: Poor documentation of new-style classes Initial Comment: The Python Reference Manual (info file python-ref) talks a lot about new-style classes, but does not say what they are, except in a brief note buried in node 'Coercion rules'. The library reference does say that object() creates such classes, that too lacks a way to look up 'new-style classes' and find object(). Also, since 'object' is a type, it seems strange that the Library Reference has it in the 'Built-in Functions' node instead of a node about (callable) types. The same applies to several other types. If you want to keep them there, at least add index entries for them in the Class-Exception-Object Index. This refers to the doc in info-2.3.3.tar.bz2 from . ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-25 23:45 Message: Logged In: YES user_id=80475 The glossary in the tutorial was added to meet this need. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-25 17:13 Message: Logged In: YES user_id=764593 object() doesn't create a new-style class; it creates an instance of class object. Note that the definition of a new-style class is just a class inheriting from object, so object itself is a new-style class. That said, the distributed documentation should probably have something more about "new-style" vs "old-style" classes, and should have a reference in the index. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960340&group_id=5470 From noreply at sourceforge.net Wed May 26 07:28:57 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 07:34:13 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 23:28 Message generated for change (Comment added) made by dmaurer You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Dieter Maurer (dmaurer) Date: 2004-05-26 11:28 Message: Logged In: YES user_id=265829 The Python documentation currently asserts that signals are delivered only to the main thread. I think we can deviate from this assertion for signals that are not normally used by applications but are used by the OS to indicate abnormal execution conditions (like SIGSEGV, SIGBUS and friends). We should at least make sure that such abnormal conditions lead to a proper process shutdown -- as early as possible. I doubt that we should change the assertion for signals usually used by applications. Patch 949332 seems to be an appropriate short term solution - until we come up with something better. I would really like it to land already in Python 2.3.4. I will apply it for our productive Python environments because I am convinced that it will improve behaviour compared to the current state. I can report back should we see unexpected behaviour. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-20 18:38 Message: Logged In: YES user_id=119306 The callback interface seems to have been added in readline 2.1, from 1997. There seem to be configure tests in the current Modules/readline.c code to search for features new to readline 2.1 so my current approach would be upping the minimum readline version from 2.0 to 2.1. If needed I could test for the callback interface and use it if available, but fall back to the readline() interface otherwise (and leave the thread and signal handling issues in place when used with older readline.) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-20 17:02 Message: Logged In: YES user_id=6656 This sounds cool! The only thing to be aware of is readline versioning... are these alternate interfaces a recent thing? ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-20 12:16 Message: Logged In: YES user_id=119306 I have an approach to have readline work well with threads while still acknowledging KeyboardInterrrupt. Using the alternate readline interface of rl_callback_handler_install() and rl_callback_read_char() along with a select(), we can recognize the interrupt signal when select returns EINTR and not need the signal handler at all. I just need to try my patch on a few more systems and try to put Anthony at ease. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 17:25 Message: Logged In: YES user_id=56897 There didn't seem to be an outcome from the python-dev discussion regarding system() and pthread_atfork(). The thread indicates that system() indeed is supposed to call atfork handlers, so therefore RedHat 9 is violating the pthread standard in that sense. (Whether or not they'll fix it is another issue.) There's also mention that os.system() may be changed to not call system() because of the atfork() problem. If the changes to avoid system() are implemented, would the pthread_atfork() approach still be problematic? As Martin Loewis points out, we could always implement the signal fixup in the child directly after the fork() if Python routines are being used to do the fork() in the first place. However if we're concerned about native modules that directly call fork() then it seems our choices are a pthread_atfork() approach or an approach where SIGINT isn't blocked. Without an async-signal-safe way to route a signal from one thread to another, I don't see how we can leave SIGINT unblocked in all threads. Re: Py_AddPendingCall. That approach might work in many cases, but I assume it doesn't work well when all threads are currently busy in native modules that are not well-behaved. For example, I have two threads: one in readline() and the other blocked in a native call that, like readline(), doesn't return control on EINTR. If the SIGINT is sent to the readline thread, the signal handler could check the thread ID and do the longjmp() since we're the proper thread to do so. If the SIGINT is sent to the other thread, the callback added by Py_AddPendingCall() won't necessarily be processed any time soon because no threads are going to return control (in a timely manner) to Python. To make matters worse, apparently even something as simple as pthread_self(), which we'd use to get the thread ID, isn't async-signal-safe on all platforms. From what I've read, none of the pthread functions are guaranteed to be async-signal-safe. :-( ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-13 16:48 Message: Logged In: YES user_id=119306 pthread_kill(). That is annoying, I have something nearly done that used it. I didn't double check the safety of pthread_kill. I saw that posix says that kill is safe to call from interrupt handlers and went from there. Can we note that we need a pthread_kill in a call to Py_AddPendingCall, and then handle it later? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-13 16:25 Message: Logged In: YES user_id=6656 Just to make life more entertaining, pthread_atfork isn't what you want, either. http://mail.python.org/pipermail/python-dev/2003-December/041309.html ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 16:20 Message: Logged In: YES user_id=56897 Argh! I thought we had a relatively clean solution, but it appears there's a stumbling block with the pthread_kill() approach. pthread_kill() is not guaranteed to be async-signal-safe, and nuggets found on the net indicate there's no portable way to redirect a process signal from one thread to another: http://groups.google.com/groups?q=pthread_kill+async+safe&start=30&hl=en&lr=&selm=3662B6A8.861984D5%40zko.dec.com&rnum=32 http://www.redhat.com/archives/phil-list/2003-December/msg00049.html Given that we can't safely call pthread_kill() from the SIGINT handler directly, there might be another way to solve our problems with pthread_atfork(). Here's my thinking: - Block SIGINT in all threads except the main (readline) thread. - Register via child process handler via pthread_atfork() that sets the SIGINT action for the child process back to the default. Unfortunately this fix isn't localized to the readline module as desired, but it may solve the problems. SIGINT routing will be forced to the readline thread, and child processes won't have SIGINT blocked, solving bug 756940. The IRIX thread signal delivery model (i.e.: broadcast) may cause problems since SIGINT may be pending when we attempt to set the action to default. Having SIGINT pending when the handler is changed to default would kill the child process. Maybe having the child process set the disposition to ignore and then to default would safely clear any pending SIGINT signal? I'll try to run some experiments with the pthread_atfork() approach soon, but work and home life for me is particularly busy lately. Apologies in advance if it takes me a while to respond or submit patches. If we're interested in a timely fix, would it be useful to break up the fix in two stages? I think we can all agree that the current approach of blocking ALL signals in created threads is a Bad Thing. What if we implement a quick, partial fix by simply change the existing code to only block SIGINT? This should be a two-line change to thread_pthread.h where "sigemptyset(&newmask); sigaddset(&newmask, SIGINT);" is used instead of "sigfillset(&newmask);". I see this partial fix having a number of benefits: - Easy change to make. No extra stuff to check for in configure or calls to things that may not exist or work properly. - Much less risky than trying to fix all the problems at once. The change only opens up signals to threads that Python-2.1 is already allowing through. - Should solve the SIGSEGV zombie problem and Guido's SIGTERM annoyance, although it would still have the problem reported in bug 756940. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-12 14:39 Message: Logged In: YES user_id=29957 This seems like a pragmatic and sensible approach to take, to me. It should probably be tested on the HP/UX boxes (google for 'HP/UX testdrive') I particularly like the idea of just putting a test in to block readline in the non-main thread. It seems the pythonic approach - since we can't guarantee behaviour that's anything but sane, it seems like a plan. Or at least make it issue a warning saying "don't do this" when readline is invoked from a non-main thread. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 14:11 Message: Logged In: YES user_id=6380 Sounds good. This solves the problem in the readline module, where it originates. BTW, if we can simplify things by only allowing readline() to be called from the main thread, that's fine with me. Doing console I/O from threads is insane anyway. We can start by assuming the signal broadcast problem is restricted to IRIX, and configure appropriately: define a test symbol for this and in configure, set this when IRIX is detected. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 13:54 Message: Logged In: YES user_id=56897 SIGINT is 'special' because that's the signal behind the problems reported in bug 465673. Given the readline module's setjmp/longjmp mechanism to process SIGINT, we simply cannot allow one thread to do the setjmp() and another thread to do the longjmp() when it receives SIGINT. Without the setjmp/longjmp stuff, SIGINT is no more special than any other asynchronous signal like SIGTERM, SIGUSR1, etc. It'd be great if we could get the desired behavior for SIGINT out of the readline module without setjmp/longjmp, but without help from the readline library I don't see an easy way to do this. The readline library insists on continuing the readline() call after a SIGINT is handled, and there doesn't appear to be any way to get it to abort the current readline() call short of modifying the readline library. If we're stuck with the setjmp/longjmp mechanism, I think we can solve the issues regarding readline() being called from another thread and exec'd processes from threads by using the pthread_kill() technique mentioned earlier. The steps would look something like this: - Do not block any signals (including SIGINT) in any threads. - When we initialize the readline module's jmp_buf via setjmp(), save off the current thread ID. Probably want to check for existing ownership of jmp_buf and flag an error if detected. - When the readline module SIGINT handler is invoked, check if the current thread owns jmp_buf. If we are the owning thread then execute the longjmp (or siglongjmp). If we're not the owning thread, then have the current thread execute pthread_kill(jmp_buf_owner_thread, SIGINT) and little else. This will defer the SIGINT to the only thread that can really process it correctly. - Since SIGINT isn't blocked in any thread, processes exec'd from threads should get the default behavior for SIGINT rather than having it blocked. The above algorithm has a race condition on thread implementations where all threads receive SIGINT. The race can cause SIGINT to be processed more than once. The jmp_buf owning thread might finish the processing of SIGINT before another thread starts its processing and re-sends SIGINT to the jmp_buf owning thread. If there's a way to know via configure that we're on a thread implementation that broadcasts SIGINT, we could #ifdef the code to use something like the getpid() hack in signalmodule.c to do the right thing. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 23:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 23:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 22:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 22:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 19:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 19:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 13:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 08:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 22:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 20:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 19:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 18:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 16:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 01:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 21:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 16:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 13:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 13:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 13:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 12:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 12:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 20:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 08:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 14:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 14:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 23:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Wed May 26 09:34:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 09:34:55 2004 Subject: [ python-Feature Requests-960821 ] Add an updating load function in pickle Message-ID: Feature Requests item #960821, was opened at 2004-05-26 09:34 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=960821&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Nicolas Fleury (nidoizo) Assigned to: Nobody/Anonymous (nobody) Summary: Add an updating load function in pickle Initial Comment: It would be very nice to have a function in pickle to load an object without creating a new instance, but by using an existing one, meaning that __init__, etc. of an existing instance is called. Something like: updateload(object, file) (I have no patch in mind.) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=960821&group_id=5470 From noreply at sourceforge.net Wed May 26 09:48:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 09:48:35 2004 Subject: [ python-Bugs-960340 ] Poor documentation of new-style classes Message-ID: Bugs item #960340, was opened at 2004-05-25 15:30 Message generated for change (Comment added) made by jimjjewett You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960340&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Hallvard B Furuseth (hfuru) Assigned to: Nobody/Anonymous (nobody) Summary: Poor documentation of new-style classes Initial Comment: The Python Reference Manual (info file python-ref) talks a lot about new-style classes, but does not say what they are, except in a brief note buried in node 'Coercion rules'. The library reference does say that object() creates such classes, that too lacks a way to look up 'new-style classes' and find object(). Also, since 'object' is a type, it seems strange that the Library Reference has it in the 'Built-in Functions' node instead of a node about (callable) types. The same applies to several other types. If you want to keep them there, at least add index entries for them in the Class-Exception-Object Index. This refers to the doc in info-2.3.3.tar.bz2 from . ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-26 09:48 Message: Logged In: YES user_id=764593 That's a start, but I do think "classic class", "old class", and "new-style class" should show up in the index to the language reference as well. One obvious (but perhaps not sufficient?) place is the programmers note at the bottom of 7.6, class definitions. Just change: "For new-style classes, descriptors ..." to: "For new-style classes (those inheriting from object), descriptors ..." The language lawyer reference also seems like the right place to list all the differences between classic and new classes, but I am less certain how to do that properly. (And it starts to be an Enhancement request.) ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-26 00:45 Message: Logged In: YES user_id=80475 The glossary in the tutorial was added to meet this need. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-25 18:13 Message: Logged In: YES user_id=764593 object() doesn't create a new-style class; it creates an instance of class object. Note that the definition of a new-style class is just a class inheriting from object, so object itself is a new-style class. That said, the distributed documentation should probably have something more about "new-style" vs "old-style" classes, and should have a reference in the index. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960340&group_id=5470 From noreply at sourceforge.net Wed May 26 10:03:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 10:03:23 2004 Subject: [ python-Feature Requests-960845 ] overall index to distributed documentation Message-ID: Feature Requests item #960845, was opened at 2004-05-26 10:03 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=960845&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: overall index to distributed documentation Initial Comment: The default distribution comes with 11 pieces of documentation, and it isn't always clear which one is appropriate. (Language Reference? Library? Module? Tutorial?) It would help if there were a composite index. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=960845&group_id=5470 From noreply at sourceforge.net Wed May 26 10:05:57 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 10:06:03 2004 Subject: [ python-Feature Requests-960454 ] old/new class documentation Message-ID: Feature Requests item #960454, was opened at 2004-05-25 18:33 Message generated for change (Comment added) made by jimjjewett You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=960454&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: old/new class documentation Initial Comment: The distributed documentation refers to old and new-style classes, but they do not appear in the index, and the difference is not explained. (Users who ask are referred to documentation on www.python.org, which is not currently shipped with the default distrubution.) As best I understand it, the only differences are: (1) New-style classes inherit from object. Because of this inheritance, new-style classes have a few extra capabilities, such as descriptors and super. (2) Method Resolution Order can be different in cases of multiple inheritance. (3) New style classes take precedence over old-style classes when doing rich comparison. (4) If rich comparison fails, numeric coercion will be attempted if -- and only if -- at least one of the objects is an old-style class. ---------------------------------------------------------------------- >Comment By: Jim Jewett (jimjjewett) Date: 2004-05-26 10:05 Message: Logged In: YES user_id=764593 Given that the differences are almost -- but not entirely -- extra options for new-style classes, the language lawyer section on classes (ref/7.6) might be a good location. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=960454&group_id=5470 From noreply at sourceforge.net Wed May 26 10:17:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 10:18:03 2004 Subject: [ python-Bugs-960860 ] botched html for index subheadings Message-ID: Bugs item #960860, was opened at 2004-05-26 10:17 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960860&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: botched html for index subheadings Initial Comment: In the index, if a topic has subtopics, it should be listed as """
topic
subtopic1
subtopic2
""" In some cases (such as the reference manual entries for module, or object(continued)), it is correct. In other cases (such as the entries for name, or the first column of object), the "
...
" wrapper is left out, so that the subentries appear to be normal entries that just happen to be out of alphabetical order. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960860&group_id=5470 From noreply at sourceforge.net Wed May 26 10:37:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 10:37:45 2004 Subject: [ python-Bugs-960874 ] codecs.lookup can raise exceptions other than LookupError Message-ID: Bugs item #960874, was opened at 2004-05-26 14:37 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 5 Submitted By: John Ehresman (jpe) Assigned to: M.-A. Lemburg (lemburg) Summary: codecs.lookup can raise exceptions other than LookupError Initial Comment: codecs.lookup raises ValueError when given an empty string and UnicodeEncodeError when given a unicode object that can't be converted to a str in the default encoding. I'd expect it to raise LookupError when passed any basestring instance. For example: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import codecs >>> codecs.lookup('') Traceback (most recent call last): File "", line 1, in ? File "c:\python23\lib\encodings\__init__.py", line 84, in search_function globals(), locals(), _import_tail) ValueError: Empty module name >>> codecs.lookup(u'\uabcd') Traceback (most recent call last): File "", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\uabcd' in position 0: ordinal not in range (128) >>> ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 From noreply at sourceforge.net Wed May 26 11:21:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 11:21:43 2004 Subject: [ python-Feature Requests-960454 ] old/new class documentation Message-ID: Feature Requests item #960454, was opened at 2004-05-25 18:33 Message generated for change (Comment added) made by jimjjewett You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=960454&group_id=5470 Category: Documentation Group: None Status: Open Resolution: None Priority: 5 Submitted By: Jim Jewett (jimjjewett) Assigned to: Nobody/Anonymous (nobody) Summary: old/new class documentation Initial Comment: The distributed documentation refers to old and new-style classes, but they do not appear in the index, and the difference is not explained. (Users who ask are referred to documentation on www.python.org, which is not currently shipped with the default distrubution.) As best I understand it, the only differences are: (1) New-style classes inherit from object. Because of this inheritance, new-style classes have a few extra capabilities, such as descriptors and super. (2) Method Resolution Order can be different in cases of multiple inheritance. (3) New style classes take precedence over old-style classes when doing rich comparison. (4) If rich comparison fails, numeric coercion will be attempted if -- and only if -- at least one of the objects is an old-style class. ---------------------------------------------------------------------- >Comment By: Jim Jewett (jimjjewett) Date: 2004-05-26 11:21 Message: Logged In: YES user_id=764593 Found a new difference -- you can't assign to the __bases__ or __name__ of a new-style class, and you can't usually assign to it's __class__. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-26 10:05 Message: Logged In: YES user_id=764593 Given that the differences are almost -- but not entirely -- extra options for new-style classes, the language lawyer section on classes (ref/7.6) might be a good location. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=960454&group_id=5470 From noreply at sourceforge.net Wed May 26 11:43:29 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 11:43:45 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-19 09:28 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-27 01:43 Message: Logged In: YES user_id=29957 This will not be in 2.3.4, as I've already stated. a) We've already cut the release candidate, and the final release is less than 12 hours away b) this is a high-risk patch -- anything in the area of threads and signals is very risky. Here's how it should go forward: First, the trunk should be patched. This fix can then be in 2.4a1. Once we're through to 2.4 final, we'll know whether the fix is good enough for the 2.3 series. Finally, after 2.4 final comes out, there will be a 2.3.5. This fix can be in that, assuming it's fine in 2.4. ---------------------------------------------------------------------- Comment By: Dieter Maurer (dmaurer) Date: 2004-05-26 21:28 Message: Logged In: YES user_id=265829 The Python documentation currently asserts that signals are delivered only to the main thread. I think we can deviate from this assertion for signals that are not normally used by applications but are used by the OS to indicate abnormal execution conditions (like SIGSEGV, SIGBUS and friends). We should at least make sure that such abnormal conditions lead to a proper process shutdown -- as early as possible. I doubt that we should change the assertion for signals usually used by applications. Patch 949332 seems to be an appropriate short term solution - until we come up with something better. I would really like it to land already in Python 2.3.4. I will apply it for our productive Python environments because I am convinced that it will improve behaviour compared to the current state. I can report back should we see unexpected behaviour. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-21 04:38 Message: Logged In: YES user_id=119306 The callback interface seems to have been added in readline 2.1, from 1997. There seem to be configure tests in the current Modules/readline.c code to search for features new to readline 2.1 so my current approach would be upping the minimum readline version from 2.0 to 2.1. If needed I could test for the callback interface and use it if available, but fall back to the readline() interface otherwise (and leave the thread and signal handling issues in place when used with older readline.) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-21 03:02 Message: Logged In: YES user_id=6656 This sounds cool! The only thing to be aware of is readline versioning... are these alternate interfaces a recent thing? ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-20 22:16 Message: Logged In: YES user_id=119306 I have an approach to have readline work well with threads while still acknowledging KeyboardInterrrupt. Using the alternate readline interface of rl_callback_handler_install() and rl_callback_read_char() along with a select(), we can recognize the interrupt signal when select returns EINTR and not need the signal handler at all. I just need to try my patch on a few more systems and try to put Anthony at ease. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-14 03:25 Message: Logged In: YES user_id=56897 There didn't seem to be an outcome from the python-dev discussion regarding system() and pthread_atfork(). The thread indicates that system() indeed is supposed to call atfork handlers, so therefore RedHat 9 is violating the pthread standard in that sense. (Whether or not they'll fix it is another issue.) There's also mention that os.system() may be changed to not call system() because of the atfork() problem. If the changes to avoid system() are implemented, would the pthread_atfork() approach still be problematic? As Martin Loewis points out, we could always implement the signal fixup in the child directly after the fork() if Python routines are being used to do the fork() in the first place. However if we're concerned about native modules that directly call fork() then it seems our choices are a pthread_atfork() approach or an approach where SIGINT isn't blocked. Without an async-signal-safe way to route a signal from one thread to another, I don't see how we can leave SIGINT unblocked in all threads. Re: Py_AddPendingCall. That approach might work in many cases, but I assume it doesn't work well when all threads are currently busy in native modules that are not well-behaved. For example, I have two threads: one in readline() and the other blocked in a native call that, like readline(), doesn't return control on EINTR. If the SIGINT is sent to the readline thread, the signal handler could check the thread ID and do the longjmp() since we're the proper thread to do so. If the SIGINT is sent to the other thread, the callback added by Py_AddPendingCall() won't necessarily be processed any time soon because no threads are going to return control (in a timely manner) to Python. To make matters worse, apparently even something as simple as pthread_self(), which we'd use to get the thread ID, isn't async-signal-safe on all platforms. From what I've read, none of the pthread functions are guaranteed to be async-signal-safe. :-( ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-14 02:48 Message: Logged In: YES user_id=119306 pthread_kill(). That is annoying, I have something nearly done that used it. I didn't double check the safety of pthread_kill. I saw that posix says that kill is safe to call from interrupt handlers and went from there. Can we note that we need a pthread_kill in a call to Py_AddPendingCall, and then handle it later? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-14 02:25 Message: Logged In: YES user_id=6656 Just to make life more entertaining, pthread_atfork isn't what you want, either. http://mail.python.org/pipermail/python-dev/2003-December/041309.html ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-14 02:20 Message: Logged In: YES user_id=56897 Argh! I thought we had a relatively clean solution, but it appears there's a stumbling block with the pthread_kill() approach. pthread_kill() is not guaranteed to be async-signal-safe, and nuggets found on the net indicate there's no portable way to redirect a process signal from one thread to another: http://groups.google.com/groups?q=pthread_kill+async+safe&start=30&hl=en&lr=&selm=3662B6A8.861984D5%40zko.dec.com&rnum=32 http://www.redhat.com/archives/phil-list/2003-December/msg00049.html Given that we can't safely call pthread_kill() from the SIGINT handler directly, there might be another way to solve our problems with pthread_atfork(). Here's my thinking: - Block SIGINT in all threads except the main (readline) thread. - Register via child process handler via pthread_atfork() that sets the SIGINT action for the child process back to the default. Unfortunately this fix isn't localized to the readline module as desired, but it may solve the problems. SIGINT routing will be forced to the readline thread, and child processes won't have SIGINT blocked, solving bug 756940. The IRIX thread signal delivery model (i.e.: broadcast) may cause problems since SIGINT may be pending when we attempt to set the action to default. Having SIGINT pending when the handler is changed to default would kill the child process. Maybe having the child process set the disposition to ignore and then to default would safely clear any pending SIGINT signal? I'll try to run some experiments with the pthread_atfork() approach soon, but work and home life for me is particularly busy lately. Apologies in advance if it takes me a while to respond or submit patches. If we're interested in a timely fix, would it be useful to break up the fix in two stages? I think we can all agree that the current approach of blocking ALL signals in created threads is a Bad Thing. What if we implement a quick, partial fix by simply change the existing code to only block SIGINT? This should be a two-line change to thread_pthread.h where "sigemptyset(&newmask); sigaddset(&newmask, SIGINT);" is used instead of "sigfillset(&newmask);". I see this partial fix having a number of benefits: - Easy change to make. No extra stuff to check for in configure or calls to things that may not exist or work properly. - Much less risky than trying to fix all the problems at once. The change only opens up signals to threads that Python-2.1 is already allowing through. - Should solve the SIGSEGV zombie problem and Guido's SIGTERM annoyance, although it would still have the problem reported in bug 756940. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-13 00:39 Message: Logged In: YES user_id=29957 This seems like a pragmatic and sensible approach to take, to me. It should probably be tested on the HP/UX boxes (google for 'HP/UX testdrive') I particularly like the idea of just putting a test in to block readline in the non-main thread. It seems the pythonic approach - since we can't guarantee behaviour that's anything but sane, it seems like a plan. Or at least make it issue a warning saying "don't do this" when readline is invoked from a non-main thread. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-13 00:11 Message: Logged In: YES user_id=6380 Sounds good. This solves the problem in the readline module, where it originates. BTW, if we can simplify things by only allowing readline() to be called from the main thread, that's fine with me. Doing console I/O from threads is insane anyway. We can start by assuming the signal broadcast problem is restricted to IRIX, and configure appropriately: define a test symbol for this and in configure, set this when IRIX is detected. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 23:54 Message: Logged In: YES user_id=56897 SIGINT is 'special' because that's the signal behind the problems reported in bug 465673. Given the readline module's setjmp/longjmp mechanism to process SIGINT, we simply cannot allow one thread to do the setjmp() and another thread to do the longjmp() when it receives SIGINT. Without the setjmp/longjmp stuff, SIGINT is no more special than any other asynchronous signal like SIGTERM, SIGUSR1, etc. It'd be great if we could get the desired behavior for SIGINT out of the readline module without setjmp/longjmp, but without help from the readline library I don't see an easy way to do this. The readline library insists on continuing the readline() call after a SIGINT is handled, and there doesn't appear to be any way to get it to abort the current readline() call short of modifying the readline library. If we're stuck with the setjmp/longjmp mechanism, I think we can solve the issues regarding readline() being called from another thread and exec'd processes from threads by using the pthread_kill() technique mentioned earlier. The steps would look something like this: - Do not block any signals (including SIGINT) in any threads. - When we initialize the readline module's jmp_buf via setjmp(), save off the current thread ID. Probably want to check for existing ownership of jmp_buf and flag an error if detected. - When the readline module SIGINT handler is invoked, check if the current thread owns jmp_buf. If we are the owning thread then execute the longjmp (or siglongjmp). If we're not the owning thread, then have the current thread execute pthread_kill(jmp_buf_owner_thread, SIGINT) and little else. This will defer the SIGINT to the only thread that can really process it correctly. - Since SIGINT isn't blocked in any thread, processes exec'd from threads should get the default behavior for SIGINT rather than having it blocked. The above algorithm has a race condition on thread implementations where all threads receive SIGINT. The race can cause SIGINT to be processed more than once. The jmp_buf owning thread might finish the processing of SIGINT before another thread starts its processing and re-sends SIGINT to the jmp_buf owning thread. If there's a way to know via configure that we're on a thread implementation that broadcasts SIGINT, we could #ifdef the code to use something like the getpid() hack in signalmodule.c to do the right thing. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-12 09:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 09:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 08:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-12 08:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 05:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-12 05:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 23:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 18:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 08:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-11 06:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 05:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 04:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-11 02:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 11:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-08 07:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-08 02:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 23:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 23:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 23:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 22:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 22:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 06:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 18:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-05 00:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-05 00:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-19 09:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Wed May 26 12:27:47 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 12:28:20 2004 Subject: [ python-Bugs-728330 ] Don't define _SGAPI on IRIX Message-ID: Bugs item #728330, was opened at 2003-04-27 10:21 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 Category: Installation Group: Python 2.3 Status: Open Resolution: Remind Priority: 9 Submitted By: Ralf W. Grosse-Kunstleve (rwgk) Assigned to: Neal Norwitz (nnorwitz) Summary: Don't define _SGAPI on IRIX Initial Comment: Python release: 2.3b1 Platform: IRIX 6.5 MIPSpro Compilers: Version 7.3.1.2m Commands used: ./configure --prefix=/usr/local_cci/Python-2.3b1t make The socket module fails to compile. The output from the compiler is attached. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-26 17:27 Message: Logged In: YES user_id=6656 Is this really The Right Thing? I still see /* XXX Using _SGIAPI is the wrong thing, but I don't know what the right thing is. */ Is there any chance of the Right Thing actually happening? If not, I'll check this into HEAD, and if that seems to work, move it to release23-maint in time for 2.3.5. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-22 09:23 Message: Logged In: YES user_id=71407 Based on Maik's posting from 2004-03-08 08:53 I've worked out a small bug-fix patch relative to Python-2.3.4c1 that leads to successful creation of _socket.so under three different versions of IRIX: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.23 with MIPRpro 7.3.1.3 I've consistently used #if defined(__sgi) && _COMPILER_VERSION>700 to restrict the changes to IRIX. However, to be sure I've also verified that the Python installation is still OK under Redhat 8 and Tru64 Unix 5.1. I will upload the patch. It is also available here: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patch For completeness I've also posted these files: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patched_c http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_original_c I've made an effort to keep the patch minimally intrusive, but it is certainly not "clean." However, the status quo is so bad that it can only get better. We have not been able to switch to Python 2.3 because of the problems with the socket module. It would be fantastic if the bug-fix patch could be included in the 2.3.4 final distribution. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2004-03-08 16:53 Message: Logged In: YES user_id=768223 I got Python 2.3.3 compiled with MipsPro Compilers: Version 7.3.1.3m. The following changes were made to satisfy all needs. Line 206 in Modules/socketmodule.c #define _SGIAPI 1 NEW: #undef _XOPEN_SOURCE NEW: #define HAVE_GETADDRINFO 1 NEW: #define HAVE_GETNAMEINFO 1 #define HAVE_INET_PTON NEW: #include NEW: #include NEW: #include #include Line 269: /* #include "addrinfo.h" */ In addition to this I extend the CFLAGS with -D_BSD_TYPES to satisfy the u_short and friends definition. --maik./ ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-03-03 16:20 Message: Logged In: YES user_id=892601 Got it down to one error: % > ./python Lib/test/test_socket.py Traceback (most recent call last): File "Lib/test/test_socket.py", line 6, in ? import socket File "/usr/local/spider/lib/python2.3/socket.py", line 44, in ? import _socket ImportError: 210454:./python: rld: Fatal Error: unresolvable symbol in /usr/local/spider/lib/python2.3/lib-dynload/_socketmodule.so: _xpg5_accept xpg5_accept does appear in any library on our SGI's that are only 1-2 years old. It does appear in libc in very recent revisions of Irix (> 6.5.19?) Does anyone know how to get Python 2.3 to avoid the use of xpg5_accept? Thanks ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-01-12 18:59 Message: Logged In: YES user_id=892601 Has anyone encountered problems with different versions of IRIX 6.5? I finally got sockets to work in Python2.3.3, but only on the most up-to-date version of IRIX (uname -v = 01100304). On any other machine with a lower version (archetecture doesn't seem to matter), I get messages which all seem to be related to sockets: 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_vsnprintf 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recv 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_connect 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_bind 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_sendto 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_setsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_accept 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recvfrom 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_socket 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockname Upgrading all of our SGIs is not something I have control over. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-22 18:52 Message: Logged In: YES user_id=768223 Sorry if there is something unclear. Redefine means: Set a value which is required to satisfy ifdef-statements in the header files. As you see in the definitions, given in the comment, the value for _NO_XOPEN4 is set as a result of some dependencies automatically defined. A redefinition of _SGIAPI in the socketmodule.c is not sufficient alone. Therefore to satisfy all requirements in the header files, you should set the values for _NO_XOPEN4 and _NO_XOPEN5 also manually. Howto do in socketmodule.c? ... #define _SGIAPI 1 #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 ... If you are familiar with regular expressions, then you will know that _NO_XOPEN[45] is a shorthand for _NO_XOPEN4, _NO_XOPEN5. Sorry if there was something wrong in my comment. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-12-22 15:49 Message: Logged In: YES user_id=892601 Can someone please clarify mherta's comment from 2003-12-16? How should we "redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition"? Should those lines of code be added to socketmodule.c, or are they the source of the problem? Is _NO_XOPEN[45] shorthand for _NO_XOPEN[4] and _NO_XOPEN[5] or are they all different? Thanks. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-16 17:27 Message: Logged In: YES user_id=768223 I've tested the latest Python 2.3.3c1 to compile. The problem reported is still present. To avoid the error messages, about the incompatible types of several netdb-functions (like inet_pton), you should redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition. If there are error messages that types like u_short and u_char are not defined, please add -D_BSD_TYPES in the Makefile (BASECFLAGS is okay). If anybody don't need a executable compiled with MipsPro, gcc from the freeware.sgi.com could be a choice. The problem for this seems in /usr/include/standards.h: #define _NO_XOPEN4 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 >= 500))) #define _NO_XOPEN5 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 < 500))) As the first option is true (_XOPEN_SOURCE is set to 600 in pyconfig.h) the second option will be false. The constant #define NI_NUMERICHOST 0x00000002 /* return numeric form of hostname */ is enveloped in a clause #ifdef _NO_XOPEN4 && _NO_XOPEN5 (/usr/include/netdb.h) which is always false. In the manual pages there is a example which uses the constant NI_NUMERICHOST and compiles without error message. I'm not sure to cut off the file pyconfig.h from compiling the socketmodule.c? Any comments? ---------------------------------------------------------------------- Comment By: Hans Nowak (zephyrfalcon) Date: 2003-10-31 03:04 Message: Logged In: YES user_id=173607 My $0.02... Yesterday I successfully compiled Python 2.3.2, with socket module. I'm using an O2 running IRIX 6.5.11f. I didn't encounter any of the problems described before, and had no problems with _SGIAPI or anything. The reason it didn't compile was that u_long and u_char were not defined in some places; /usr/include/netinet/tcp.h to be precise, and a Python file (getaddrinfo.c). I'm using gcc 3.3 (the one downloaded from freeware.sgi.com). If anyone is interested, drop me a mail at hans at zephyrfalcon.org and I'll put the binary online somewhere. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-24 20:59 Message: Logged In: YES user_id=892601 Here's the output from 'gmake', using MIPSpro Compiler 7.30 on IRIX 6.5 Sorry, I don't know how to attach something using the "Add a Comment" box. Is there another way to post? The patch at www.python.org/2.3.2/patches/sgi-socketmodule.patch was edited into socketmodule.c : #include #endif /* GCC 3.2 on Irix 6.5 fails to define this variable at all. Based on the above code, I'd say that the SGI headers are just horked. */ #if defined(__sgi) && !defined(INET_ADDRSTRLEN) #define INET_ADDRSTRLEN 16 #endif /* Generic includes */ #include #include --------------------------------------------------------------------------- cc -OPT:Olimit=0 -DNDEBUG -n32 -O -I. -I./Include -DPy_BUILD_CORE -c ./Modules/socketmodule.c -o Modules/socketmodule.o cc-1047 cc: WARNING File = ./Modules/socketmodule.c, Line = 195 Macro "_SGIAPI" (declared at line 210 of "/usr/include/standards.h") has an incompatible redefinition. #define _SGIAPI 1 ^ cc-1020 cc: ERROR File = /usr/include/netdb.h, Line = 198 The identifier "socklen_t" is undefined. extern int iruserok_sa(const void *, socklen_t, int, const char *, ^ cc-1047 cc: WARNING File = ./Modules/addrinfo.h, Line = 145 Macro "_SS_ALIGNSIZE" (declared at line 246 of "/usr/include/sys/socket.h") has an incompatible redefinition. #define _SS_ALIGNSIZE (sizeof(PY_LONG_LONG)) ^ cc-1020 cc: ERROR File = ./Modules/socketmodule.c, Line = 822 The identifier "NI_NUMERICHOST" is undefined. NI_NUMERICHOST); ^ cc-1515 cc: ERROR File = ./Modules/socketmodule.c, Line = 3054 A value of type "int" cannot be assigned to an entity of type "const char *". retval = inet_ntop(af, packed, ip, sizeof(ip)); ^ 3 errors detected in the compilation of "./Modules/socketmodule.c". gmake: *** [Modules/socketmodule.o] Error 2 ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2003-10-23 11:25 Message: Logged In: YES user_id=6656 Just to be clear, please "attach" the build rather than pasting it inline. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2003-10-23 03:19 Message: Logged In: YES user_id=29957 Did you try 2.3.2 with the patch from the 2.3.2 build bugs page? If it failed, please post the build log of the failing bit to this bug. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-22 19:26 Message: Logged In: YES user_id=892601 Has there been any progress since August on getting Python-2.3 to compile on IRIX 6.5? I really need to have sockets working to use python. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-08-14 12:37 Message: Logged In: YES user_id=768223 I have the same problem on IRIX 6.5.19m and MIPSpro 7.3.1.3m on a FUEL 600Mhz. The situation seems a little bit strange on this system. But I have luck and get it compiled. As the current implementation is not clean, my resolution also. Which should be changed to get it work. The manpage for gethostbyaddr_r says: #include #include #include #include So I included this in the __sgi-clause in socketmodule.c. Looking in the header-files shows, that some variables are defined if #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 are set, before the includes. This defines some more variables, so that pyconfig.h should be changed. #define HAVE_ADDRINFO 1 #define HAVE_SOCKADDR_STORAGE 1 Sure there are warnings about the redefinition of the _(SGIAPI|NO_XOPEN4|NO_XOPEN5) but it compiles and the tests don't fail. Also my applications run without exception or coredump on FUEL, Octane, Octane 2. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-08-01 13:10 Message: Logged In: YES user_id=33168 Ralf, I will try to work on this, but it won't be for a week or two probably. Can you tell me which machines (by name) work and doesn't work. The only IRIX machine I seem to know about it rattler. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-08-01 08:34 Message: Logged In: YES user_id=71407 For the records: The Python2.3 (final) socket module still does not compile under IRIX 6.5.20. It only compiles under 6.5.10. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 19:07 Message: Logged In: YES user_id=31392 I applied a slight variant of Neal's patch and that seemed to fix things. It still seems wrong that we define _SGIAPI, but that will have to wait for 2.3.1. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 14:54 Message: Logged In: YES user_id=31392 I learned a little about inet_ntop() in the last few minutes. It looks like the size_t argument is an older spelling, and the current POSIX says it should be socklen_t. Regardless, it looks like we're using the new definition when IRIX is actually providing the older definition. Neal attached a patch that looks like it should clean up the problem. Have you tried it? I think a better solution would be to teach configure how to find inet_ntop() via netdb.h, but that's probably too hard to get done in the new couple of hours. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 14:42 Message: Logged In: YES user_id=31392 There seem to be two causes of worry. The actual compiler error in your build log and the use of _SGIAPI that Casavanti warns again. It looks like the compiler problems is because our inet_ntop() prototype is wrong. That is Python says the last argument is a socklen_t, but IRIX and Linux man pages both say size_t. Perhaps size_t and socklen_t are different sizes on IRIX, but not on Linux. What happens if you change it to size_t? The problematic code would appear to be these lines from socketmodule.c: #if defined(__sgi)&&_COMPILER_VERSION>700 && !_SGIAPI /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #define _SGIAPI 1 #include "netdb.h" #endif It's the only use the _SGIAPI symbol that Casavant says we shouldn't use. What happens if you remove the use _SGIAPI, i.e. just make it #if defined(__sgi) && _COMPILER_VERSION > 700 /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #include "netdb.h" #endif ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-17 01:27 Message: Logged In: YES user_id=71407 We have confirmed that the socket module in Python 2.3b1 and 2.3b2 fails to compile under these versions of IRIX: 6.5.10, MIPSpro 7.3.1.2m 6.5.19, MIPSpro 7.3.1.3m 6.5.20, MIPSpro 7.3.1.3m (latest available, to my knowledge) If Python 2.3 goes out without a working socket module it will be useless for many people in science where IRIX is still found quite frequently. Maybe even worse, we will also no longer be able to advertise Python as a language that "runs everywhere." ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-15 23:37 Message: Logged In: YES user_id=71407 Date: Tue, 15 Jul 2003 15:06:25 -0500 From: "Brent Casavant" To: "Ralf W. Grosse-Kunstleve" Subject: Re: IRIX & Python Ralf, I read through the sourceforge report. I don't have a sourceforge account so I didn't add there, but feel free to distribute my ramblings. First, they should be very careful about what version of the operating system they are using. Real IPv6 support was only added a few releases ago (6.5.20 I believe, though my memory is hazy). Some of the functions/macros corresponding to IPv6 were introduced prior to that point, but implementation may not have been complete. If they have to play tricks with #define values to get a symbol recongnized it most likely means that SGI didn't intend for that symbol to be used yet. I suspect the problems they are seeing are because they are using some version of IRIX 6.5.x that includes inet_ntoa() definitions under _SGIAPI, but for which real support was not yet complete. The namespace of symbols that begin with a single underscore and a capital letter is reserved for the OS vendors, and the fact that someone is trying to manually insert a definition of _SGIAPI into the source should be a big red warning flag. That said, there is a correct method to deal with the new capabilities of IRIX 6.5.x over time. This comes in two parts. A better explanation is in /usr/include/optional_sym.h First, when a new symbol is added to IRIX, the corresponding header file will always have the following line in it: #pragma optional some_new_symbol The presence of this #pragma will cause the linker to not complain if the symbol is missing when the executable is linked. If SGI neglects to do so it is a bug that should be reported to us. The second part is that calls to these "optional" symbols should be protected by a run-time check to see if the symbol is actually present. Just like this: #include #include . . . if (_MIPS_SYMBOL_PRESENT(some_new_symbol)) { /* New function is present, use it */ qux = some_new_symbol(foo, bar, baz); } else { /* New function isn't present, do something else */ qux = builtin_some_new_symbol(foo, bar, baz); } This ensures that a single executable will be able to function properly (or at least degrade gracefully) on all version of IRIX 6.5.x. In other words you can compile on 6.5.20 and run on 6.5.8, or vice versa. In particular, this means that compile-time detection of features such as inet_ntoa() is incorrect for IRIX 6.5.x -- these decisions should be made at run-time instead. If the decision is made at compile time the executable may well not execute on an older version of IRIX 6.5.x. Unfortunately GNU configure is not equipped to utilize such advanced binary-compatability mechanisms, so there's no "easy" way to get GNU configure'd software to run across all IRIX 6.5.x releases, short of compiling only on the original non-upgraded version of IRIX 6.5. Hope that helps, Brent Casavant On Tue, 15 Jul 2003, Ralf W. Grosse-Kunstleve wrote: > We are developing a Python-based (www.python.org) application for > IRIX. Until now the Python IRIX port was fully supported, but the last > two beta releases of Python (2.3b1 and 2.3b2) make me increasingly > worried because the crucial socket module (TCP/IP sockets) fails to > build. A while ago I've filed a bug report, but the Python developers > appear to be stuck: > > https://sourceforge.net/tracker/? func=detail&aid=728330&group_id=5470&atid=105470 > > Is there a way to bring this to the attention of developers at SGI to > hopefully get some help? > > Thank you in advance! > Ralf > -- Brent Casavant Silicon Graphics, Inc. Where am I? And what am I IRIX O.S. Engineer http://www.sgi.com/ doing in this handbasket? bcasavan@sgi.com 44.8562N 93.1355W 860F -- Unknown ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-06-30 15:21 Message: Logged In: YES user_id=71407 The socket module still fails to compile with Python 2.3b2, in addition to two other extensions. The full make log is attached. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-22 07:47 Message: Logged In: YES user_id=21627 I think it would be best if detect presence of inet_aton/inet_pton. Is IPv6 support possible on this system? Then it would be best if that was detected, as well. If it is not possible to detect aton/pton/ipv6, what is the problem with not using them? I'd like to have a look at the relevant headers (in particular to understand the duplicate definition of _SGIAPI), indeed. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-05-22 00:35 Message: Logged In: YES user_id=33168 Argh!!! This is a nightmare. The attached hack ^h^h^h patch fixes the problem, but ... _SGIAPI needs to be defined (~192 in socketmodule.c) in order to get the prototypes for inet_aton, inet_pton. But _SGIAPI can't be defined in the standard header files because XOPEN_SOURCE is defined. Martin, do you have any other suggestions? I don't see how to do this right within configure. If you want I can send you the header file(s). ---------------------------------------------------------------------- Comment By: alexandre gillet (gillet) Date: 2003-05-12 18:41 Message: Logged In: YES user_id=150999 I had the same problem compiling socket on Irix. We did find that on our system ENABLE_IPV6 is not define and all the error are cause by that. System that do not enable IPV6 are not well supported with the socketmodule.c code. We were able to compile socket after making few change in the code: I am not sure if I did it right but it compile. I will post my patch but I do not see any place to attach my file example: See line 2794,2800 + #ifdef ENABLE_IPV6 char packed[MAX(sizeof(struct in_addr), sizeof(struct in6_addr))]; + #else + char packed[sizeof(struct in_addr)]; + #endif I am not sure how to post the patch or file. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-05-05 19:44 Message: Logged In: YES user_id=71407 This is not in my area of expertise, but I've tried a couple of (probably stupid) things: #include in socketmodule.c instead of #define _SGIAPI. This resulted in even more errors. ./configure --disable-ipv6 Same errors as before. I am lost here. Sorry. But you still have ssh access to rattler.lbl.gov if that helps. Accounts for other Python developers are a possibility. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-04 13:39 Message: Logged In: YES user_id=21627 Can you analyse these compiler errors in more detail, please? What, in your opinion, is causing these problems, and how would you correct them? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 From noreply at sourceforge.net Wed May 26 12:32:48 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 12:33:11 2004 Subject: [ python-Bugs-960874 ] codecs.lookup can raise exceptions other than LookupError Message-ID: Bugs item #960874, was opened at 2004-05-26 15:37 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 5 Submitted By: John Ehresman (jpe) Assigned to: M.-A. Lemburg (lemburg) Summary: codecs.lookup can raise exceptions other than LookupError Initial Comment: codecs.lookup raises ValueError when given an empty string and UnicodeEncodeError when given a unicode object that can't be converted to a str in the default encoding. I'd expect it to raise LookupError when passed any basestring instance. For example: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import codecs >>> codecs.lookup('') Traceback (most recent call last): File "", line 1, in ? File "c:\python23\lib\encodings\__init__.py", line 84, in search_function globals(), locals(), _import_tail) ValueError: Empty module name >>> codecs.lookup(u'\uabcd') Traceback (most recent call last): File "", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\uabcd' in position 0: ordinal not in range (128) >>> ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-26 17:32 Message: Logged In: YES user_id=6656 What exactly are you complaining about? I'd expect codecs.lookup to raise TypeError if called with no arguments or an integer. I believe it's documented somewhere that encoding names must be ascii only, but I must admit I don't recall where. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 From noreply at sourceforge.net Wed May 26 12:36:09 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 12:36:44 2004 Subject: [ python-Bugs-924301 ] A leak case with cmd.py & readline & exception Message-ID: Bugs item #924301, was opened at 2004-03-27 00:28 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924301&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Sverker Nilsson (svenil) Assigned to: Michael Hudson (mwh) Summary: A leak case with cmd.py & readline & exception Initial Comment: A leak to do with readline & cmd, in Python 2.3. I found out what hold on to my interactive objects too long ('for ever') in certain circumstances. The circumstance had to do with an exception being raised in Cmd.cmdloop and handled (or not handled) outside of Cmd.cmdloop. In cmd.py, class Cmd, in cmdloop(), if an exception is raised and propagated out from the interior of cmdloop, the function postloop() is not called. The default function of this, (in 2.3) when the readline library is present, is to restore the completer, via: readline.set_completer(self.old_completer) If this is not done, the newly (by preloop) inserted completer will remain. Even if the loop is called again and run without exception, the new completer will remain, because then in postloop the old completer will be set to our new completer. When we exit, the completer will remain the one we set. This will hold on to our object, aka 'leak'. - In cmd.py in 2.2 no attempt was made to restore the completer, so this was also a kind of leak, but it was replaced the next time a Cmd instance was initialized. Now, however, the next time we will not replace the old completer, but both of them will remain in memory. The old one will be stored as self.old_completer. If we terminate with an exception, bad luck... the stored completer retains both of the instances. If we terminate normally, the old one will be retained. In no case do we restore the space of the first instance. The only way that would happen, would be if the second instance first exited the loop with an exception, and then entered the loop again an exited normally. But then, the second instance is retained instead! If each instance happens to terminate with an exception, it seems well possible that an ever increasing chain of leaking instances will be accumulated. My fix is to always call the postloop, given the preloop succeeded. This is done via a try:finally clause. def cmdloop(self, intro=None): ... self.preloop() try: ... finally: # Make sure postloop called self.postloop() I am attaching my patched version of cmd.py. It was originally from the tarball of Python 2.3.3 downloaded from Python.org some month or so ago in which cmd.py had this size & date: 14504 Feb 19 2003 cmd.py Best regards, Sverker Nilsson ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-26 17:36 Message: Logged In: YES user_id=6656 What do you think of the attached? This makes the documentation of pre & post loop more accurate again, which I think is nice. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-19 10:38 Message: Logged In: YES user_id=6656 This is where I go "I wish I'd reviewed that patch more carefully". In particular, the documentation of {pre,post}loop is now out of date. I wonder setting/getting the completer in these functions was a good idea. Hmm. This bug report confuses me :-) but I can certainly see the intent of the patch... ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-19 02:52 Message: Logged In: YES user_id=80475 Michael, this touches some of your code. Do you want to handle this one? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924301&group_id=5470 From noreply at sourceforge.net Wed May 26 12:52:41 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 12:52:50 2004 Subject: [ python-Bugs-696535 ] Python 2.4: Warn about omitted mutable_flag. Message-ID: Bugs item #696535, was opened at 2003-03-03 13:25 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=696535&group_id=5470 Category: Extension Modules Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Michael Hudson (mwh) >Assigned to: Martin v. L?wis (loewis) Summary: Python 2.4: Warn about omitted mutable_flag. Initial Comment: We need to add a warning to fcntl.ioctl() for 2.4. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-26 17:52 Message: Logged In: YES user_id=6656 Is the attached all that is needed? Do we want a test that the warning is indeed emitted? Reading http://python.org/sf/555817 first might be good for the memory. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=696535&group_id=5470 From noreply at sourceforge.net Wed May 26 12:59:33 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 13:00:31 2004 Subject: [ python-Bugs-728330 ] Don't define _SGAPI on IRIX Message-ID: Bugs item #728330, was opened at 2003-04-27 02:21 Message generated for change (Comment added) made by rwgk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 Category: Installation Group: Python 2.3 Status: Open Resolution: Remind Priority: 9 Submitted By: Ralf W. Grosse-Kunstleve (rwgk) Assigned to: Neal Norwitz (nnorwitz) Summary: Don't define _SGAPI on IRIX Initial Comment: Python release: 2.3b1 Platform: IRIX 6.5 MIPSpro Compilers: Version 7.3.1.2m Commands used: ./configure --prefix=/usr/local_cci/Python-2.3b1t make The socket module fails to compile. The output from the compiler is attached. ---------------------------------------------------------------------- >Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 09:59 Message: Logged In: YES user_id=71407 > Is there any chance of the Right Thing actually happening? The experience of the last 13 months suggests "No." SGI's freeware team has not responded to my report for more than a year. I am absolutely not a SGI or socket expert. I only worked on the patch because I was desperate enough and because Maik had already done the groundwork. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 09:27 Message: Logged In: YES user_id=6656 Is this really The Right Thing? I still see /* XXX Using _SGIAPI is the wrong thing, but I don't know what the right thing is. */ Is there any chance of the Right Thing actually happening? If not, I'll check this into HEAD, and if that seems to work, move it to release23-maint in time for 2.3.5. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-22 01:23 Message: Logged In: YES user_id=71407 Based on Maik's posting from 2004-03-08 08:53 I've worked out a small bug-fix patch relative to Python-2.3.4c1 that leads to successful creation of _socket.so under three different versions of IRIX: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.23 with MIPRpro 7.3.1.3 I've consistently used #if defined(__sgi) && _COMPILER_VERSION>700 to restrict the changes to IRIX. However, to be sure I've also verified that the Python installation is still OK under Redhat 8 and Tru64 Unix 5.1. I will upload the patch. It is also available here: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patch For completeness I've also posted these files: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patched_c http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_original_c I've made an effort to keep the patch minimally intrusive, but it is certainly not "clean." However, the status quo is so bad that it can only get better. We have not been able to switch to Python 2.3 because of the problems with the socket module. It would be fantastic if the bug-fix patch could be included in the 2.3.4 final distribution. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2004-03-08 08:53 Message: Logged In: YES user_id=768223 I got Python 2.3.3 compiled with MipsPro Compilers: Version 7.3.1.3m. The following changes were made to satisfy all needs. Line 206 in Modules/socketmodule.c #define _SGIAPI 1 NEW: #undef _XOPEN_SOURCE NEW: #define HAVE_GETADDRINFO 1 NEW: #define HAVE_GETNAMEINFO 1 #define HAVE_INET_PTON NEW: #include NEW: #include NEW: #include #include Line 269: /* #include "addrinfo.h" */ In addition to this I extend the CFLAGS with -D_BSD_TYPES to satisfy the u_short and friends definition. --maik./ ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-03-03 08:20 Message: Logged In: YES user_id=892601 Got it down to one error: % > ./python Lib/test/test_socket.py Traceback (most recent call last): File "Lib/test/test_socket.py", line 6, in ? import socket File "/usr/local/spider/lib/python2.3/socket.py", line 44, in ? import _socket ImportError: 210454:./python: rld: Fatal Error: unresolvable symbol in /usr/local/spider/lib/python2.3/lib-dynload/_socketmodule.so: _xpg5_accept xpg5_accept does appear in any library on our SGI's that are only 1-2 years old. It does appear in libc in very recent revisions of Irix (> 6.5.19?) Does anyone know how to get Python 2.3 to avoid the use of xpg5_accept? Thanks ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-01-12 10:59 Message: Logged In: YES user_id=892601 Has anyone encountered problems with different versions of IRIX 6.5? I finally got sockets to work in Python2.3.3, but only on the most up-to-date version of IRIX (uname -v = 01100304). On any other machine with a lower version (archetecture doesn't seem to matter), I get messages which all seem to be related to sockets: 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_vsnprintf 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recv 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_connect 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_bind 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_sendto 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_setsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_accept 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recvfrom 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_socket 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockname Upgrading all of our SGIs is not something I have control over. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-22 10:52 Message: Logged In: YES user_id=768223 Sorry if there is something unclear. Redefine means: Set a value which is required to satisfy ifdef-statements in the header files. As you see in the definitions, given in the comment, the value for _NO_XOPEN4 is set as a result of some dependencies automatically defined. A redefinition of _SGIAPI in the socketmodule.c is not sufficient alone. Therefore to satisfy all requirements in the header files, you should set the values for _NO_XOPEN4 and _NO_XOPEN5 also manually. Howto do in socketmodule.c? ... #define _SGIAPI 1 #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 ... If you are familiar with regular expressions, then you will know that _NO_XOPEN[45] is a shorthand for _NO_XOPEN4, _NO_XOPEN5. Sorry if there was something wrong in my comment. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-12-22 07:49 Message: Logged In: YES user_id=892601 Can someone please clarify mherta's comment from 2003-12-16? How should we "redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition"? Should those lines of code be added to socketmodule.c, or are they the source of the problem? Is _NO_XOPEN[45] shorthand for _NO_XOPEN[4] and _NO_XOPEN[5] or are they all different? Thanks. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-16 09:27 Message: Logged In: YES user_id=768223 I've tested the latest Python 2.3.3c1 to compile. The problem reported is still present. To avoid the error messages, about the incompatible types of several netdb-functions (like inet_pton), you should redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition. If there are error messages that types like u_short and u_char are not defined, please add -D_BSD_TYPES in the Makefile (BASECFLAGS is okay). If anybody don't need a executable compiled with MipsPro, gcc from the freeware.sgi.com could be a choice. The problem for this seems in /usr/include/standards.h: #define _NO_XOPEN4 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 >= 500))) #define _NO_XOPEN5 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 < 500))) As the first option is true (_XOPEN_SOURCE is set to 600 in pyconfig.h) the second option will be false. The constant #define NI_NUMERICHOST 0x00000002 /* return numeric form of hostname */ is enveloped in a clause #ifdef _NO_XOPEN4 && _NO_XOPEN5 (/usr/include/netdb.h) which is always false. In the manual pages there is a example which uses the constant NI_NUMERICHOST and compiles without error message. I'm not sure to cut off the file pyconfig.h from compiling the socketmodule.c? Any comments? ---------------------------------------------------------------------- Comment By: Hans Nowak (zephyrfalcon) Date: 2003-10-30 19:04 Message: Logged In: YES user_id=173607 My $0.02... Yesterday I successfully compiled Python 2.3.2, with socket module. I'm using an O2 running IRIX 6.5.11f. I didn't encounter any of the problems described before, and had no problems with _SGIAPI or anything. The reason it didn't compile was that u_long and u_char were not defined in some places; /usr/include/netinet/tcp.h to be precise, and a Python file (getaddrinfo.c). I'm using gcc 3.3 (the one downloaded from freeware.sgi.com). If anyone is interested, drop me a mail at hans at zephyrfalcon.org and I'll put the binary online somewhere. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-24 12:59 Message: Logged In: YES user_id=892601 Here's the output from 'gmake', using MIPSpro Compiler 7.30 on IRIX 6.5 Sorry, I don't know how to attach something using the "Add a Comment" box. Is there another way to post? The patch at www.python.org/2.3.2/patches/sgi-socketmodule.patch was edited into socketmodule.c : #include #endif /* GCC 3.2 on Irix 6.5 fails to define this variable at all. Based on the above code, I'd say that the SGI headers are just horked. */ #if defined(__sgi) && !defined(INET_ADDRSTRLEN) #define INET_ADDRSTRLEN 16 #endif /* Generic includes */ #include #include --------------------------------------------------------------------------- cc -OPT:Olimit=0 -DNDEBUG -n32 -O -I. -I./Include -DPy_BUILD_CORE -c ./Modules/socketmodule.c -o Modules/socketmodule.o cc-1047 cc: WARNING File = ./Modules/socketmodule.c, Line = 195 Macro "_SGIAPI" (declared at line 210 of "/usr/include/standards.h") has an incompatible redefinition. #define _SGIAPI 1 ^ cc-1020 cc: ERROR File = /usr/include/netdb.h, Line = 198 The identifier "socklen_t" is undefined. extern int iruserok_sa(const void *, socklen_t, int, const char *, ^ cc-1047 cc: WARNING File = ./Modules/addrinfo.h, Line = 145 Macro "_SS_ALIGNSIZE" (declared at line 246 of "/usr/include/sys/socket.h") has an incompatible redefinition. #define _SS_ALIGNSIZE (sizeof(PY_LONG_LONG)) ^ cc-1020 cc: ERROR File = ./Modules/socketmodule.c, Line = 822 The identifier "NI_NUMERICHOST" is undefined. NI_NUMERICHOST); ^ cc-1515 cc: ERROR File = ./Modules/socketmodule.c, Line = 3054 A value of type "int" cannot be assigned to an entity of type "const char *". retval = inet_ntop(af, packed, ip, sizeof(ip)); ^ 3 errors detected in the compilation of "./Modules/socketmodule.c". gmake: *** [Modules/socketmodule.o] Error 2 ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2003-10-23 03:25 Message: Logged In: YES user_id=6656 Just to be clear, please "attach" the build rather than pasting it inline. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2003-10-22 19:19 Message: Logged In: YES user_id=29957 Did you try 2.3.2 with the patch from the 2.3.2 build bugs page? If it failed, please post the build log of the failing bit to this bug. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-22 11:26 Message: Logged In: YES user_id=892601 Has there been any progress since August on getting Python-2.3 to compile on IRIX 6.5? I really need to have sockets working to use python. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-08-14 04:37 Message: Logged In: YES user_id=768223 I have the same problem on IRIX 6.5.19m and MIPSpro 7.3.1.3m on a FUEL 600Mhz. The situation seems a little bit strange on this system. But I have luck and get it compiled. As the current implementation is not clean, my resolution also. Which should be changed to get it work. The manpage for gethostbyaddr_r says: #include #include #include #include So I included this in the __sgi-clause in socketmodule.c. Looking in the header-files shows, that some variables are defined if #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 are set, before the includes. This defines some more variables, so that pyconfig.h should be changed. #define HAVE_ADDRINFO 1 #define HAVE_SOCKADDR_STORAGE 1 Sure there are warnings about the redefinition of the _(SGIAPI|NO_XOPEN4|NO_XOPEN5) but it compiles and the tests don't fail. Also my applications run without exception or coredump on FUEL, Octane, Octane 2. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-08-01 05:10 Message: Logged In: YES user_id=33168 Ralf, I will try to work on this, but it won't be for a week or two probably. Can you tell me which machines (by name) work and doesn't work. The only IRIX machine I seem to know about it rattler. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-08-01 00:34 Message: Logged In: YES user_id=71407 For the records: The Python2.3 (final) socket module still does not compile under IRIX 6.5.20. It only compiles under 6.5.10. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 11:07 Message: Logged In: YES user_id=31392 I applied a slight variant of Neal's patch and that seemed to fix things. It still seems wrong that we define _SGIAPI, but that will have to wait for 2.3.1. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 06:54 Message: Logged In: YES user_id=31392 I learned a little about inet_ntop() in the last few minutes. It looks like the size_t argument is an older spelling, and the current POSIX says it should be socklen_t. Regardless, it looks like we're using the new definition when IRIX is actually providing the older definition. Neal attached a patch that looks like it should clean up the problem. Have you tried it? I think a better solution would be to teach configure how to find inet_ntop() via netdb.h, but that's probably too hard to get done in the new couple of hours. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 06:42 Message: Logged In: YES user_id=31392 There seem to be two causes of worry. The actual compiler error in your build log and the use of _SGIAPI that Casavanti warns again. It looks like the compiler problems is because our inet_ntop() prototype is wrong. That is Python says the last argument is a socklen_t, but IRIX and Linux man pages both say size_t. Perhaps size_t and socklen_t are different sizes on IRIX, but not on Linux. What happens if you change it to size_t? The problematic code would appear to be these lines from socketmodule.c: #if defined(__sgi)&&_COMPILER_VERSION>700 && !_SGIAPI /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #define _SGIAPI 1 #include "netdb.h" #endif It's the only use the _SGIAPI symbol that Casavant says we shouldn't use. What happens if you remove the use _SGIAPI, i.e. just make it #if defined(__sgi) && _COMPILER_VERSION > 700 /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #include "netdb.h" #endif ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-16 17:27 Message: Logged In: YES user_id=71407 We have confirmed that the socket module in Python 2.3b1 and 2.3b2 fails to compile under these versions of IRIX: 6.5.10, MIPSpro 7.3.1.2m 6.5.19, MIPSpro 7.3.1.3m 6.5.20, MIPSpro 7.3.1.3m (latest available, to my knowledge) If Python 2.3 goes out without a working socket module it will be useless for many people in science where IRIX is still found quite frequently. Maybe even worse, we will also no longer be able to advertise Python as a language that "runs everywhere." ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-15 15:37 Message: Logged In: YES user_id=71407 Date: Tue, 15 Jul 2003 15:06:25 -0500 From: "Brent Casavant" To: "Ralf W. Grosse-Kunstleve" Subject: Re: IRIX & Python Ralf, I read through the sourceforge report. I don't have a sourceforge account so I didn't add there, but feel free to distribute my ramblings. First, they should be very careful about what version of the operating system they are using. Real IPv6 support was only added a few releases ago (6.5.20 I believe, though my memory is hazy). Some of the functions/macros corresponding to IPv6 were introduced prior to that point, but implementation may not have been complete. If they have to play tricks with #define values to get a symbol recongnized it most likely means that SGI didn't intend for that symbol to be used yet. I suspect the problems they are seeing are because they are using some version of IRIX 6.5.x that includes inet_ntoa() definitions under _SGIAPI, but for which real support was not yet complete. The namespace of symbols that begin with a single underscore and a capital letter is reserved for the OS vendors, and the fact that someone is trying to manually insert a definition of _SGIAPI into the source should be a big red warning flag. That said, there is a correct method to deal with the new capabilities of IRIX 6.5.x over time. This comes in two parts. A better explanation is in /usr/include/optional_sym.h First, when a new symbol is added to IRIX, the corresponding header file will always have the following line in it: #pragma optional some_new_symbol The presence of this #pragma will cause the linker to not complain if the symbol is missing when the executable is linked. If SGI neglects to do so it is a bug that should be reported to us. The second part is that calls to these "optional" symbols should be protected by a run-time check to see if the symbol is actually present. Just like this: #include #include . . . if (_MIPS_SYMBOL_PRESENT(some_new_symbol)) { /* New function is present, use it */ qux = some_new_symbol(foo, bar, baz); } else { /* New function isn't present, do something else */ qux = builtin_some_new_symbol(foo, bar, baz); } This ensures that a single executable will be able to function properly (or at least degrade gracefully) on all version of IRIX 6.5.x. In other words you can compile on 6.5.20 and run on 6.5.8, or vice versa. In particular, this means that compile-time detection of features such as inet_ntoa() is incorrect for IRIX 6.5.x -- these decisions should be made at run-time instead. If the decision is made at compile time the executable may well not execute on an older version of IRIX 6.5.x. Unfortunately GNU configure is not equipped to utilize such advanced binary-compatability mechanisms, so there's no "easy" way to get GNU configure'd software to run across all IRIX 6.5.x releases, short of compiling only on the original non-upgraded version of IRIX 6.5. Hope that helps, Brent Casavant On Tue, 15 Jul 2003, Ralf W. Grosse-Kunstleve wrote: > We are developing a Python-based (www.python.org) application for > IRIX. Until now the Python IRIX port was fully supported, but the last > two beta releases of Python (2.3b1 and 2.3b2) make me increasingly > worried because the crucial socket module (TCP/IP sockets) fails to > build. A while ago I've filed a bug report, but the Python developers > appear to be stuck: > > https://sourceforge.net/tracker/? func=detail&aid=728330&group_id=5470&atid=105470 > > Is there a way to bring this to the attention of developers at SGI to > hopefully get some help? > > Thank you in advance! > Ralf > -- Brent Casavant Silicon Graphics, Inc. Where am I? And what am I IRIX O.S. Engineer http://www.sgi.com/ doing in this handbasket? bcasavan@sgi.com 44.8562N 93.1355W 860F -- Unknown ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-06-30 07:21 Message: Logged In: YES user_id=71407 The socket module still fails to compile with Python 2.3b2, in addition to two other extensions. The full make log is attached. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-21 23:47 Message: Logged In: YES user_id=21627 I think it would be best if detect presence of inet_aton/inet_pton. Is IPv6 support possible on this system? Then it would be best if that was detected, as well. If it is not possible to detect aton/pton/ipv6, what is the problem with not using them? I'd like to have a look at the relevant headers (in particular to understand the duplicate definition of _SGIAPI), indeed. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-05-21 16:35 Message: Logged In: YES user_id=33168 Argh!!! This is a nightmare. The attached hack ^h^h^h patch fixes the problem, but ... _SGIAPI needs to be defined (~192 in socketmodule.c) in order to get the prototypes for inet_aton, inet_pton. But _SGIAPI can't be defined in the standard header files because XOPEN_SOURCE is defined. Martin, do you have any other suggestions? I don't see how to do this right within configure. If you want I can send you the header file(s). ---------------------------------------------------------------------- Comment By: alexandre gillet (gillet) Date: 2003-05-12 10:41 Message: Logged In: YES user_id=150999 I had the same problem compiling socket on Irix. We did find that on our system ENABLE_IPV6 is not define and all the error are cause by that. System that do not enable IPV6 are not well supported with the socketmodule.c code. We were able to compile socket after making few change in the code: I am not sure if I did it right but it compile. I will post my patch but I do not see any place to attach my file example: See line 2794,2800 + #ifdef ENABLE_IPV6 char packed[MAX(sizeof(struct in_addr), sizeof(struct in6_addr))]; + #else + char packed[sizeof(struct in_addr)]; + #endif I am not sure how to post the patch or file. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-05-05 11:44 Message: Logged In: YES user_id=71407 This is not in my area of expertise, but I've tried a couple of (probably stupid) things: #include in socketmodule.c instead of #define _SGIAPI. This resulted in even more errors. ./configure --disable-ipv6 Same errors as before. I am lost here. Sorry. But you still have ssh access to rattler.lbl.gov if that helps. Accounts for other Python developers are a possibility. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-04 05:39 Message: Logged In: YES user_id=21627 Can you analyse these compiler errors in more detail, please? What, in your opinion, is causing these problems, and how would you correct them? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 From noreply at sourceforge.net Wed May 26 13:07:24 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 13:07:52 2004 Subject: [ python-Bugs-728330 ] Don't define _SGAPI on IRIX Message-ID: Bugs item #728330, was opened at 2003-04-27 10:21 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 Category: Installation Group: Python 2.3 Status: Open Resolution: Remind Priority: 9 Submitted By: Ralf W. Grosse-Kunstleve (rwgk) Assigned to: Neal Norwitz (nnorwitz) Summary: Don't define _SGAPI on IRIX Initial Comment: Python release: 2.3b1 Platform: IRIX 6.5 MIPSpro Compilers: Version 7.3.1.2m Commands used: ./configure --prefix=/usr/local_cci/Python-2.3b1t make The socket module fails to compile. The output from the compiler is attached. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-26 18:07 Message: Logged In: YES user_id=6656 OK, so I've commited revision 1.289 of Modules/socketmodule.c. Can you check if CVS HEAD works for you now? ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 17:59 Message: Logged In: YES user_id=71407 > Is there any chance of the Right Thing actually happening? The experience of the last 13 months suggests "No." SGI's freeware team has not responded to my report for more than a year. I am absolutely not a SGI or socket expert. I only worked on the patch because I was desperate enough and because Maik had already done the groundwork. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 17:27 Message: Logged In: YES user_id=6656 Is this really The Right Thing? I still see /* XXX Using _SGIAPI is the wrong thing, but I don't know what the right thing is. */ Is there any chance of the Right Thing actually happening? If not, I'll check this into HEAD, and if that seems to work, move it to release23-maint in time for 2.3.5. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-22 09:23 Message: Logged In: YES user_id=71407 Based on Maik's posting from 2004-03-08 08:53 I've worked out a small bug-fix patch relative to Python-2.3.4c1 that leads to successful creation of _socket.so under three different versions of IRIX: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.23 with MIPRpro 7.3.1.3 I've consistently used #if defined(__sgi) && _COMPILER_VERSION>700 to restrict the changes to IRIX. However, to be sure I've also verified that the Python installation is still OK under Redhat 8 and Tru64 Unix 5.1. I will upload the patch. It is also available here: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patch For completeness I've also posted these files: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patched_c http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_original_c I've made an effort to keep the patch minimally intrusive, but it is certainly not "clean." However, the status quo is so bad that it can only get better. We have not been able to switch to Python 2.3 because of the problems with the socket module. It would be fantastic if the bug-fix patch could be included in the 2.3.4 final distribution. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2004-03-08 16:53 Message: Logged In: YES user_id=768223 I got Python 2.3.3 compiled with MipsPro Compilers: Version 7.3.1.3m. The following changes were made to satisfy all needs. Line 206 in Modules/socketmodule.c #define _SGIAPI 1 NEW: #undef _XOPEN_SOURCE NEW: #define HAVE_GETADDRINFO 1 NEW: #define HAVE_GETNAMEINFO 1 #define HAVE_INET_PTON NEW: #include NEW: #include NEW: #include #include Line 269: /* #include "addrinfo.h" */ In addition to this I extend the CFLAGS with -D_BSD_TYPES to satisfy the u_short and friends definition. --maik./ ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-03-03 16:20 Message: Logged In: YES user_id=892601 Got it down to one error: % > ./python Lib/test/test_socket.py Traceback (most recent call last): File "Lib/test/test_socket.py", line 6, in ? import socket File "/usr/local/spider/lib/python2.3/socket.py", line 44, in ? import _socket ImportError: 210454:./python: rld: Fatal Error: unresolvable symbol in /usr/local/spider/lib/python2.3/lib-dynload/_socketmodule.so: _xpg5_accept xpg5_accept does appear in any library on our SGI's that are only 1-2 years old. It does appear in libc in very recent revisions of Irix (> 6.5.19?) Does anyone know how to get Python 2.3 to avoid the use of xpg5_accept? Thanks ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-01-12 18:59 Message: Logged In: YES user_id=892601 Has anyone encountered problems with different versions of IRIX 6.5? I finally got sockets to work in Python2.3.3, but only on the most up-to-date version of IRIX (uname -v = 01100304). On any other machine with a lower version (archetecture doesn't seem to matter), I get messages which all seem to be related to sockets: 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_vsnprintf 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recv 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_connect 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_bind 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_sendto 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_setsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_accept 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recvfrom 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_socket 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockname Upgrading all of our SGIs is not something I have control over. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-22 18:52 Message: Logged In: YES user_id=768223 Sorry if there is something unclear. Redefine means: Set a value which is required to satisfy ifdef-statements in the header files. As you see in the definitions, given in the comment, the value for _NO_XOPEN4 is set as a result of some dependencies automatically defined. A redefinition of _SGIAPI in the socketmodule.c is not sufficient alone. Therefore to satisfy all requirements in the header files, you should set the values for _NO_XOPEN4 and _NO_XOPEN5 also manually. Howto do in socketmodule.c? ... #define _SGIAPI 1 #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 ... If you are familiar with regular expressions, then you will know that _NO_XOPEN[45] is a shorthand for _NO_XOPEN4, _NO_XOPEN5. Sorry if there was something wrong in my comment. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-12-22 15:49 Message: Logged In: YES user_id=892601 Can someone please clarify mherta's comment from 2003-12-16? How should we "redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition"? Should those lines of code be added to socketmodule.c, or are they the source of the problem? Is _NO_XOPEN[45] shorthand for _NO_XOPEN[4] and _NO_XOPEN[5] or are they all different? Thanks. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-16 17:27 Message: Logged In: YES user_id=768223 I've tested the latest Python 2.3.3c1 to compile. The problem reported is still present. To avoid the error messages, about the incompatible types of several netdb-functions (like inet_pton), you should redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition. If there are error messages that types like u_short and u_char are not defined, please add -D_BSD_TYPES in the Makefile (BASECFLAGS is okay). If anybody don't need a executable compiled with MipsPro, gcc from the freeware.sgi.com could be a choice. The problem for this seems in /usr/include/standards.h: #define _NO_XOPEN4 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 >= 500))) #define _NO_XOPEN5 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 < 500))) As the first option is true (_XOPEN_SOURCE is set to 600 in pyconfig.h) the second option will be false. The constant #define NI_NUMERICHOST 0x00000002 /* return numeric form of hostname */ is enveloped in a clause #ifdef _NO_XOPEN4 && _NO_XOPEN5 (/usr/include/netdb.h) which is always false. In the manual pages there is a example which uses the constant NI_NUMERICHOST and compiles without error message. I'm not sure to cut off the file pyconfig.h from compiling the socketmodule.c? Any comments? ---------------------------------------------------------------------- Comment By: Hans Nowak (zephyrfalcon) Date: 2003-10-31 03:04 Message: Logged In: YES user_id=173607 My $0.02... Yesterday I successfully compiled Python 2.3.2, with socket module. I'm using an O2 running IRIX 6.5.11f. I didn't encounter any of the problems described before, and had no problems with _SGIAPI or anything. The reason it didn't compile was that u_long and u_char were not defined in some places; /usr/include/netinet/tcp.h to be precise, and a Python file (getaddrinfo.c). I'm using gcc 3.3 (the one downloaded from freeware.sgi.com). If anyone is interested, drop me a mail at hans at zephyrfalcon.org and I'll put the binary online somewhere. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-24 20:59 Message: Logged In: YES user_id=892601 Here's the output from 'gmake', using MIPSpro Compiler 7.30 on IRIX 6.5 Sorry, I don't know how to attach something using the "Add a Comment" box. Is there another way to post? The patch at www.python.org/2.3.2/patches/sgi-socketmodule.patch was edited into socketmodule.c : #include #endif /* GCC 3.2 on Irix 6.5 fails to define this variable at all. Based on the above code, I'd say that the SGI headers are just horked. */ #if defined(__sgi) && !defined(INET_ADDRSTRLEN) #define INET_ADDRSTRLEN 16 #endif /* Generic includes */ #include #include --------------------------------------------------------------------------- cc -OPT:Olimit=0 -DNDEBUG -n32 -O -I. -I./Include -DPy_BUILD_CORE -c ./Modules/socketmodule.c -o Modules/socketmodule.o cc-1047 cc: WARNING File = ./Modules/socketmodule.c, Line = 195 Macro "_SGIAPI" (declared at line 210 of "/usr/include/standards.h") has an incompatible redefinition. #define _SGIAPI 1 ^ cc-1020 cc: ERROR File = /usr/include/netdb.h, Line = 198 The identifier "socklen_t" is undefined. extern int iruserok_sa(const void *, socklen_t, int, const char *, ^ cc-1047 cc: WARNING File = ./Modules/addrinfo.h, Line = 145 Macro "_SS_ALIGNSIZE" (declared at line 246 of "/usr/include/sys/socket.h") has an incompatible redefinition. #define _SS_ALIGNSIZE (sizeof(PY_LONG_LONG)) ^ cc-1020 cc: ERROR File = ./Modules/socketmodule.c, Line = 822 The identifier "NI_NUMERICHOST" is undefined. NI_NUMERICHOST); ^ cc-1515 cc: ERROR File = ./Modules/socketmodule.c, Line = 3054 A value of type "int" cannot be assigned to an entity of type "const char *". retval = inet_ntop(af, packed, ip, sizeof(ip)); ^ 3 errors detected in the compilation of "./Modules/socketmodule.c". gmake: *** [Modules/socketmodule.o] Error 2 ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2003-10-23 11:25 Message: Logged In: YES user_id=6656 Just to be clear, please "attach" the build rather than pasting it inline. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2003-10-23 03:19 Message: Logged In: YES user_id=29957 Did you try 2.3.2 with the patch from the 2.3.2 build bugs page? If it failed, please post the build log of the failing bit to this bug. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-22 19:26 Message: Logged In: YES user_id=892601 Has there been any progress since August on getting Python-2.3 to compile on IRIX 6.5? I really need to have sockets working to use python. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-08-14 12:37 Message: Logged In: YES user_id=768223 I have the same problem on IRIX 6.5.19m and MIPSpro 7.3.1.3m on a FUEL 600Mhz. The situation seems a little bit strange on this system. But I have luck and get it compiled. As the current implementation is not clean, my resolution also. Which should be changed to get it work. The manpage for gethostbyaddr_r says: #include #include #include #include So I included this in the __sgi-clause in socketmodule.c. Looking in the header-files shows, that some variables are defined if #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 are set, before the includes. This defines some more variables, so that pyconfig.h should be changed. #define HAVE_ADDRINFO 1 #define HAVE_SOCKADDR_STORAGE 1 Sure there are warnings about the redefinition of the _(SGIAPI|NO_XOPEN4|NO_XOPEN5) but it compiles and the tests don't fail. Also my applications run without exception or coredump on FUEL, Octane, Octane 2. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-08-01 13:10 Message: Logged In: YES user_id=33168 Ralf, I will try to work on this, but it won't be for a week or two probably. Can you tell me which machines (by name) work and doesn't work. The only IRIX machine I seem to know about it rattler. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-08-01 08:34 Message: Logged In: YES user_id=71407 For the records: The Python2.3 (final) socket module still does not compile under IRIX 6.5.20. It only compiles under 6.5.10. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 19:07 Message: Logged In: YES user_id=31392 I applied a slight variant of Neal's patch and that seemed to fix things. It still seems wrong that we define _SGIAPI, but that will have to wait for 2.3.1. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 14:54 Message: Logged In: YES user_id=31392 I learned a little about inet_ntop() in the last few minutes. It looks like the size_t argument is an older spelling, and the current POSIX says it should be socklen_t. Regardless, it looks like we're using the new definition when IRIX is actually providing the older definition. Neal attached a patch that looks like it should clean up the problem. Have you tried it? I think a better solution would be to teach configure how to find inet_ntop() via netdb.h, but that's probably too hard to get done in the new couple of hours. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 14:42 Message: Logged In: YES user_id=31392 There seem to be two causes of worry. The actual compiler error in your build log and the use of _SGIAPI that Casavanti warns again. It looks like the compiler problems is because our inet_ntop() prototype is wrong. That is Python says the last argument is a socklen_t, but IRIX and Linux man pages both say size_t. Perhaps size_t and socklen_t are different sizes on IRIX, but not on Linux. What happens if you change it to size_t? The problematic code would appear to be these lines from socketmodule.c: #if defined(__sgi)&&_COMPILER_VERSION>700 && !_SGIAPI /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #define _SGIAPI 1 #include "netdb.h" #endif It's the only use the _SGIAPI symbol that Casavant says we shouldn't use. What happens if you remove the use _SGIAPI, i.e. just make it #if defined(__sgi) && _COMPILER_VERSION > 700 /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #include "netdb.h" #endif ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-17 01:27 Message: Logged In: YES user_id=71407 We have confirmed that the socket module in Python 2.3b1 and 2.3b2 fails to compile under these versions of IRIX: 6.5.10, MIPSpro 7.3.1.2m 6.5.19, MIPSpro 7.3.1.3m 6.5.20, MIPSpro 7.3.1.3m (latest available, to my knowledge) If Python 2.3 goes out without a working socket module it will be useless for many people in science where IRIX is still found quite frequently. Maybe even worse, we will also no longer be able to advertise Python as a language that "runs everywhere." ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-15 23:37 Message: Logged In: YES user_id=71407 Date: Tue, 15 Jul 2003 15:06:25 -0500 From: "Brent Casavant" To: "Ralf W. Grosse-Kunstleve" Subject: Re: IRIX & Python Ralf, I read through the sourceforge report. I don't have a sourceforge account so I didn't add there, but feel free to distribute my ramblings. First, they should be very careful about what version of the operating system they are using. Real IPv6 support was only added a few releases ago (6.5.20 I believe, though my memory is hazy). Some of the functions/macros corresponding to IPv6 were introduced prior to that point, but implementation may not have been complete. If they have to play tricks with #define values to get a symbol recongnized it most likely means that SGI didn't intend for that symbol to be used yet. I suspect the problems they are seeing are because they are using some version of IRIX 6.5.x that includes inet_ntoa() definitions under _SGIAPI, but for which real support was not yet complete. The namespace of symbols that begin with a single underscore and a capital letter is reserved for the OS vendors, and the fact that someone is trying to manually insert a definition of _SGIAPI into the source should be a big red warning flag. That said, there is a correct method to deal with the new capabilities of IRIX 6.5.x over time. This comes in two parts. A better explanation is in /usr/include/optional_sym.h First, when a new symbol is added to IRIX, the corresponding header file will always have the following line in it: #pragma optional some_new_symbol The presence of this #pragma will cause the linker to not complain if the symbol is missing when the executable is linked. If SGI neglects to do so it is a bug that should be reported to us. The second part is that calls to these "optional" symbols should be protected by a run-time check to see if the symbol is actually present. Just like this: #include #include . . . if (_MIPS_SYMBOL_PRESENT(some_new_symbol)) { /* New function is present, use it */ qux = some_new_symbol(foo, bar, baz); } else { /* New function isn't present, do something else */ qux = builtin_some_new_symbol(foo, bar, baz); } This ensures that a single executable will be able to function properly (or at least degrade gracefully) on all version of IRIX 6.5.x. In other words you can compile on 6.5.20 and run on 6.5.8, or vice versa. In particular, this means that compile-time detection of features such as inet_ntoa() is incorrect for IRIX 6.5.x -- these decisions should be made at run-time instead. If the decision is made at compile time the executable may well not execute on an older version of IRIX 6.5.x. Unfortunately GNU configure is not equipped to utilize such advanced binary-compatability mechanisms, so there's no "easy" way to get GNU configure'd software to run across all IRIX 6.5.x releases, short of compiling only on the original non-upgraded version of IRIX 6.5. Hope that helps, Brent Casavant On Tue, 15 Jul 2003, Ralf W. Grosse-Kunstleve wrote: > We are developing a Python-based (www.python.org) application for > IRIX. Until now the Python IRIX port was fully supported, but the last > two beta releases of Python (2.3b1 and 2.3b2) make me increasingly > worried because the crucial socket module (TCP/IP sockets) fails to > build. A while ago I've filed a bug report, but the Python developers > appear to be stuck: > > https://sourceforge.net/tracker/? func=detail&aid=728330&group_id=5470&atid=105470 > > Is there a way to bring this to the attention of developers at SGI to > hopefully get some help? > > Thank you in advance! > Ralf > -- Brent Casavant Silicon Graphics, Inc. Where am I? And what am I IRIX O.S. Engineer http://www.sgi.com/ doing in this handbasket? bcasavan@sgi.com 44.8562N 93.1355W 860F -- Unknown ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-06-30 15:21 Message: Logged In: YES user_id=71407 The socket module still fails to compile with Python 2.3b2, in addition to two other extensions. The full make log is attached. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-22 07:47 Message: Logged In: YES user_id=21627 I think it would be best if detect presence of inet_aton/inet_pton. Is IPv6 support possible on this system? Then it would be best if that was detected, as well. If it is not possible to detect aton/pton/ipv6, what is the problem with not using them? I'd like to have a look at the relevant headers (in particular to understand the duplicate definition of _SGIAPI), indeed. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-05-22 00:35 Message: Logged In: YES user_id=33168 Argh!!! This is a nightmare. The attached hack ^h^h^h patch fixes the problem, but ... _SGIAPI needs to be defined (~192 in socketmodule.c) in order to get the prototypes for inet_aton, inet_pton. But _SGIAPI can't be defined in the standard header files because XOPEN_SOURCE is defined. Martin, do you have any other suggestions? I don't see how to do this right within configure. If you want I can send you the header file(s). ---------------------------------------------------------------------- Comment By: alexandre gillet (gillet) Date: 2003-05-12 18:41 Message: Logged In: YES user_id=150999 I had the same problem compiling socket on Irix. We did find that on our system ENABLE_IPV6 is not define and all the error are cause by that. System that do not enable IPV6 are not well supported with the socketmodule.c code. We were able to compile socket after making few change in the code: I am not sure if I did it right but it compile. I will post my patch but I do not see any place to attach my file example: See line 2794,2800 + #ifdef ENABLE_IPV6 char packed[MAX(sizeof(struct in_addr), sizeof(struct in6_addr))]; + #else + char packed[sizeof(struct in_addr)]; + #endif I am not sure how to post the patch or file. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-05-05 19:44 Message: Logged In: YES user_id=71407 This is not in my area of expertise, but I've tried a couple of (probably stupid) things: #include in socketmodule.c instead of #define _SGIAPI. This resulted in even more errors. ./configure --disable-ipv6 Same errors as before. I am lost here. Sorry. But you still have ssh access to rattler.lbl.gov if that helps. Accounts for other Python developers are a possibility. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-04 13:39 Message: Logged In: YES user_id=21627 Can you analyse these compiler errors in more detail, please? What, in your opinion, is causing these problems, and how would you correct them? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 From noreply at sourceforge.net Wed May 26 13:09:18 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 13:09:27 2004 Subject: [ python-Bugs-960874 ] codecs.lookup can raise exceptions other than LookupError Message-ID: Bugs item #960874, was opened at 2004-05-26 14:37 Message generated for change (Comment added) made by jpe You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 5 Submitted By: John Ehresman (jpe) Assigned to: M.-A. Lemburg (lemburg) Summary: codecs.lookup can raise exceptions other than LookupError Initial Comment: codecs.lookup raises ValueError when given an empty string and UnicodeEncodeError when given a unicode object that can't be converted to a str in the default encoding. I'd expect it to raise LookupError when passed any basestring instance. For example: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import codecs >>> codecs.lookup('') Traceback (most recent call last): File "", line 1, in ? File "c:\python23\lib\encodings\__init__.py", line 84, in search_function globals(), locals(), _import_tail) ValueError: Empty module name >>> codecs.lookup(u'\uabcd') Traceback (most recent call last): File "", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\uabcd' in position 0: ordinal not in range (128) >>> ---------------------------------------------------------------------- >Comment By: John Ehresman (jpe) Date: 2004-05-26 17:09 Message: Logged In: YES user_id=22785 The other exceptions occur when strings or unicode objects are passed in as an argument. The string that it fails on is the empty string (''). I can see disallowing non-ascii names, but '' should raise a LookupError. My use case is to see if an user supplied unicode string is a valid encoding, so any check that the lookup function does not do, I will need to do before calling it. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 16:32 Message: Logged In: YES user_id=6656 What exactly are you complaining about? I'd expect codecs.lookup to raise TypeError if called with no arguments or an integer. I believe it's documented somewhere that encoding names must be ascii only, but I must admit I don't recall where. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 From noreply at sourceforge.net Wed May 26 13:13:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 13:13:35 2004 Subject: [ python-Bugs-960874 ] codecs.lookup can raise exceptions other than LookupError Message-ID: Bugs item #960874, was opened at 2004-05-26 15:37 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 5 Submitted By: John Ehresman (jpe) Assigned to: M.-A. Lemburg (lemburg) Summary: codecs.lookup can raise exceptions other than LookupError Initial Comment: codecs.lookup raises ValueError when given an empty string and UnicodeEncodeError when given a unicode object that can't be converted to a str in the default encoding. I'd expect it to raise LookupError when passed any basestring instance. For example: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import codecs >>> codecs.lookup('') Traceback (most recent call last): File "", line 1, in ? File "c:\python23\lib\encodings\__init__.py", line 84, in search_function globals(), locals(), _import_tail) ValueError: Empty module name >>> codecs.lookup(u'\uabcd') Traceback (most recent call last): File "", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\uabcd' in position 0: ordinal not in range (128) >>> ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-26 18:13 Message: Logged In: YES user_id=6656 This much seems to be fixed in CVS, actually :-) ---------------------------------------------------------------------- Comment By: John Ehresman (jpe) Date: 2004-05-26 18:09 Message: Logged In: YES user_id=22785 The other exceptions occur when strings or unicode objects are passed in as an argument. The string that it fails on is the empty string (''). I can see disallowing non-ascii names, but '' should raise a LookupError. My use case is to see if an user supplied unicode string is a valid encoding, so any check that the lookup function does not do, I will need to do before calling it. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 17:32 Message: Logged In: YES user_id=6656 What exactly are you complaining about? I'd expect codecs.lookup to raise TypeError if called with no arguments or an integer. I believe it's documented somewhere that encoding names must be ascii only, but I must admit I don't recall where. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 From noreply at sourceforge.net Wed May 26 13:19:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 13:19:21 2004 Subject: [ python-Bugs-756924 ] SIGSEGV causes hung threads (Linux) Message-ID: Bugs item #756924, was opened at 2003-06-18 19:28 Message generated for change (Comment added) made by langmead You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 Category: Threads Group: Python 2.4 Status: Open Resolution: None Priority: 5 Submitted By: Greg Jones (morngnstar) Assigned to: Nobody/Anonymous (nobody) Summary: SIGSEGV causes hung threads (Linux) Initial Comment: When a segmentation fault happens on Linux in any thread but the main thread, the program exits, but zombie threads remain behind. Steps to reproduce: 1. Download attached tar and extract files zombie.py and zombieCmodule.c. 2. Compile and link zombieCmodule.c as a shared library (or whatever other method you prefer for making a Python extension module). 3. Put the output from step 2 (zombieC.so) in your lib/python directory. 4. Run python2.2 zombie.py. 5. After the program exits, run ps. zombie.py launches several threads that just loop forever, and one that calls a C function in zombieC. The latter prints "NULL!" then segfaults intentionally, printing "Segmentation fault". Then the program exits, returning control back to the shell. Expected, and Python 2.1 behavior: No Python threads appear in the output of ps. Actual Python 2.2 behavior: 5 Python threads appear in the output of ps. To kill them, you have to apply kill -9 to each one individually. Not only does this bug leave around messy zombie threads, but the threads left behind hold on to program resources. For example, if the program binds a socket, that port cannot be bound again until someone kills the threads. Of course programs should not generate segfaults, but if they do they should fail gracefully. I have identified the cause of this bug. The old Python 2.1 behavior can be restored by removing these lines of Python/thread_pthread.h: sigfillset(&newmask); SET_THREAD_SIGMASK(SIG_BLOCK, &newmask, &oldmask); ... and ... SET_THREAD_SIGMASK(SIG_SETMASK, &oldmask, NULL); I guess even SIGSEGV gets blocked by this code, and somehow that prevents the default behavior of segfaults from working correctly. I'm not suggesting that removing this code is a good way to fix this bug. This is just an example to show that it seems to be the blocking of signals that causes this bug. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-26 13:19 Message: Logged In: YES user_id=119306 The patch in is probably a better change to consider rather than the LinuxThreads specific workarounds of 948614 or the false synchronous/ asynchronous dichotomy of 949332. (948614 was just looking at things in terms of LinuxThreads specific quirks, and when comments started popping up about similar problems in HP/UX and others it became apparent that a more general approach was needed. 949332 was a hope that a minimally intrusive variation might get be more palatable for 2.3.4, but I can perfectly understand your reluctance.) The new patch, 960406, seems to solve the problems Jason Lowe reported in (at least with my testing under Linux, Solaris, Irix, HP/UX, and True64 Unix) but does not exhibit the problems described here. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-26 11:43 Message: Logged In: YES user_id=29957 This will not be in 2.3.4, as I've already stated. a) We've already cut the release candidate, and the final release is less than 12 hours away b) this is a high-risk patch -- anything in the area of threads and signals is very risky. Here's how it should go forward: First, the trunk should be patched. This fix can then be in 2.4a1. Once we're through to 2.4 final, we'll know whether the fix is good enough for the 2.3 series. Finally, after 2.4 final comes out, there will be a 2.3.5. This fix can be in that, assuming it's fine in 2.4. ---------------------------------------------------------------------- Comment By: Dieter Maurer (dmaurer) Date: 2004-05-26 07:28 Message: Logged In: YES user_id=265829 The Python documentation currently asserts that signals are delivered only to the main thread. I think we can deviate from this assertion for signals that are not normally used by applications but are used by the OS to indicate abnormal execution conditions (like SIGSEGV, SIGBUS and friends). We should at least make sure that such abnormal conditions lead to a proper process shutdown -- as early as possible. I doubt that we should change the assertion for signals usually used by applications. Patch 949332 seems to be an appropriate short term solution - until we come up with something better. I would really like it to land already in Python 2.3.4. I will apply it for our productive Python environments because I am convinced that it will improve behaviour compared to the current state. I can report back should we see unexpected behaviour. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-20 14:38 Message: Logged In: YES user_id=119306 The callback interface seems to have been added in readline 2.1, from 1997. There seem to be configure tests in the current Modules/readline.c code to search for features new to readline 2.1 so my current approach would be upping the minimum readline version from 2.0 to 2.1. If needed I could test for the callback interface and use it if available, but fall back to the readline() interface otherwise (and leave the thread and signal handling issues in place when used with older readline.) ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-20 13:02 Message: Logged In: YES user_id=6656 This sounds cool! The only thing to be aware of is readline versioning... are these alternate interfaces a recent thing? ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-20 08:16 Message: Logged In: YES user_id=119306 I have an approach to have readline work well with threads while still acknowledging KeyboardInterrrupt. Using the alternate readline interface of rl_callback_handler_install() and rl_callback_read_char() along with a select(), we can recognize the interrupt signal when select returns EINTR and not need the signal handler at all. I just need to try my patch on a few more systems and try to put Anthony at ease. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 13:25 Message: Logged In: YES user_id=56897 There didn't seem to be an outcome from the python-dev discussion regarding system() and pthread_atfork(). The thread indicates that system() indeed is supposed to call atfork handlers, so therefore RedHat 9 is violating the pthread standard in that sense. (Whether or not they'll fix it is another issue.) There's also mention that os.system() may be changed to not call system() because of the atfork() problem. If the changes to avoid system() are implemented, would the pthread_atfork() approach still be problematic? As Martin Loewis points out, we could always implement the signal fixup in the child directly after the fork() if Python routines are being used to do the fork() in the first place. However if we're concerned about native modules that directly call fork() then it seems our choices are a pthread_atfork() approach or an approach where SIGINT isn't blocked. Without an async-signal-safe way to route a signal from one thread to another, I don't see how we can leave SIGINT unblocked in all threads. Re: Py_AddPendingCall. That approach might work in many cases, but I assume it doesn't work well when all threads are currently busy in native modules that are not well-behaved. For example, I have two threads: one in readline() and the other blocked in a native call that, like readline(), doesn't return control on EINTR. If the SIGINT is sent to the readline thread, the signal handler could check the thread ID and do the longjmp() since we're the proper thread to do so. If the SIGINT is sent to the other thread, the callback added by Py_AddPendingCall() won't necessarily be processed any time soon because no threads are going to return control (in a timely manner) to Python. To make matters worse, apparently even something as simple as pthread_self(), which we'd use to get the thread ID, isn't async-signal-safe on all platforms. From what I've read, none of the pthread functions are guaranteed to be async-signal-safe. :-( ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-13 12:48 Message: Logged In: YES user_id=119306 pthread_kill(). That is annoying, I have something nearly done that used it. I didn't double check the safety of pthread_kill. I saw that posix says that kill is safe to call from interrupt handlers and went from there. Can we note that we need a pthread_kill in a call to Py_AddPendingCall, and then handle it later? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-13 12:25 Message: Logged In: YES user_id=6656 Just to make life more entertaining, pthread_atfork isn't what you want, either. http://mail.python.org/pipermail/python-dev/2003-December/041309.html ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-13 12:20 Message: Logged In: YES user_id=56897 Argh! I thought we had a relatively clean solution, but it appears there's a stumbling block with the pthread_kill() approach. pthread_kill() is not guaranteed to be async-signal-safe, and nuggets found on the net indicate there's no portable way to redirect a process signal from one thread to another: http://groups.google.com/groups?q=pthread_kill+async+safe&start=30&hl=en&lr=&selm=3662B6A8.861984D5%40zko.dec.com&rnum=32 http://www.redhat.com/archives/phil-list/2003-December/msg00049.html Given that we can't safely call pthread_kill() from the SIGINT handler directly, there might be another way to solve our problems with pthread_atfork(). Here's my thinking: - Block SIGINT in all threads except the main (readline) thread. - Register via child process handler via pthread_atfork() that sets the SIGINT action for the child process back to the default. Unfortunately this fix isn't localized to the readline module as desired, but it may solve the problems. SIGINT routing will be forced to the readline thread, and child processes won't have SIGINT blocked, solving bug 756940. The IRIX thread signal delivery model (i.e.: broadcast) may cause problems since SIGINT may be pending when we attempt to set the action to default. Having SIGINT pending when the handler is changed to default would kill the child process. Maybe having the child process set the disposition to ignore and then to default would safely clear any pending SIGINT signal? I'll try to run some experiments with the pthread_atfork() approach soon, but work and home life for me is particularly busy lately. Apologies in advance if it takes me a while to respond or submit patches. If we're interested in a timely fix, would it be useful to break up the fix in two stages? I think we can all agree that the current approach of blocking ALL signals in created threads is a Bad Thing. What if we implement a quick, partial fix by simply change the existing code to only block SIGINT? This should be a two-line change to thread_pthread.h where "sigemptyset(&newmask); sigaddset(&newmask, SIGINT);" is used instead of "sigfillset(&newmask);". I see this partial fix having a number of benefits: - Easy change to make. No extra stuff to check for in configure or calls to things that may not exist or work properly. - Much less risky than trying to fix all the problems at once. The change only opens up signals to threads that Python-2.1 is already allowing through. - Should solve the SIGSEGV zombie problem and Guido's SIGTERM annoyance, although it would still have the problem reported in bug 756940. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-12 10:39 Message: Logged In: YES user_id=29957 This seems like a pragmatic and sensible approach to take, to me. It should probably be tested on the HP/UX boxes (google for 'HP/UX testdrive') I particularly like the idea of just putting a test in to block readline in the non-main thread. It seems the pythonic approach - since we can't guarantee behaviour that's anything but sane, it seems like a plan. Or at least make it issue a warning saying "don't do this" when readline is invoked from a non-main thread. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-12 10:11 Message: Logged In: YES user_id=6380 Sounds good. This solves the problem in the readline module, where it originates. BTW, if we can simplify things by only allowing readline() to be called from the main thread, that's fine with me. Doing console I/O from threads is insane anyway. We can start by assuming the signal broadcast problem is restricted to IRIX, and configure appropriately: define a test symbol for this and in configure, set this when IRIX is detected. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-12 09:54 Message: Logged In: YES user_id=56897 SIGINT is 'special' because that's the signal behind the problems reported in bug 465673. Given the readline module's setjmp/longjmp mechanism to process SIGINT, we simply cannot allow one thread to do the setjmp() and another thread to do the longjmp() when it receives SIGINT. Without the setjmp/longjmp stuff, SIGINT is no more special than any other asynchronous signal like SIGTERM, SIGUSR1, etc. It'd be great if we could get the desired behavior for SIGINT out of the readline module without setjmp/longjmp, but without help from the readline library I don't see an easy way to do this. The readline library insists on continuing the readline() call after a SIGINT is handled, and there doesn't appear to be any way to get it to abort the current readline() call short of modifying the readline library. If we're stuck with the setjmp/longjmp mechanism, I think we can solve the issues regarding readline() being called from another thread and exec'd processes from threads by using the pthread_kill() technique mentioned earlier. The steps would look something like this: - Do not block any signals (including SIGINT) in any threads. - When we initialize the readline module's jmp_buf via setjmp(), save off the current thread ID. Probably want to check for existing ownership of jmp_buf and flag an error if detected. - When the readline module SIGINT handler is invoked, check if the current thread owns jmp_buf. If we are the owning thread then execute the longjmp (or siglongjmp). If we're not the owning thread, then have the current thread execute pthread_kill(jmp_buf_owner_thread, SIGINT) and little else. This will defer the SIGINT to the only thread that can really process it correctly. - Since SIGINT isn't blocked in any thread, processes exec'd from threads should get the default behavior for SIGINT rather than having it blocked. The above algorithm has a race condition on thread implementations where all threads receive SIGINT. The race can cause SIGINT to be processed more than once. The jmp_buf owning thread might finish the processing of SIGINT before another thread starts its processing and re-sends SIGINT to the jmp_buf owning thread. If there's a way to know via configure that we're on a thread implementation that broadcasts SIGINT, we could #ifdef the code to use something like the getpid() hack in signalmodule.c to do the right thing. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 19:41 Message: Logged In: YES user_id=119306 The only thing special about SIGINT is that the readline module uses PyOS_setsig to set it, and when readline's special SIGINT handler is set, it throws all of the careful thread handling in Modules/sigmodule.c: signal_handler out the window. Now that I say it out loud, PyOS_setsig some consideration on its own. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 19:04 Message: Logged In: YES user_id=6380 And I think it is possible to call readline() from any thread. (Though it would be a problem if multiple threads were doing this simultaneously :-) ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 18:54 Message: Logged In: YES user_id=6380 But if you still block SIGINT (why is SIGINT special?) in all threads, processes forked from threads will be started with SIGINT blocked, and that's still wrong. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-11 18:46 Message: Logged In: YES user_id=119306 I was handwaving a bit over the "arrangements" to make with the siglongjump. It is probable that blocking SIGINT from all spawned threads will be the easiest. It will also work in both the pthreads and LWP case (signal sent to one unblocked thread in the process) and the LinuxThreads and SGI threads case (signal broadcast to the process group, which includes each thread individually.) The only thing I wanted to double check was whether readline could be executed by any thread other than the main thread. If so, the SIGINT handler needs to check not whether it is the main thread, but rather if it is the (or *a*?) thread that currently is in the middle of a readline call. ---------------------------------------------------------------------- Comment By: Jason Lowe (jasonlowe) Date: 2004-05-11 15:57 Message: Logged In: YES user_id=56897 Ack. I thought I was logged in for that previous comment which was from me (jasonlowe). ---------------------------------------------------------------------- Comment By: Nobody/Anonymous (nobody) Date: 2004-05-11 15:52 Message: Logged In: NO I agree, the original patch I submitted is horribly ham-fisted because it blocks all signals. I'm kicking myself for not forseeing the problems with SIGSEGV, SIGTERM, etc. as reported in 756924. The original problem I was trying to fix was that the wrong thread (i.e.: any thread but the main thread) would receive the SIGINT and end up doing the longjmp() to the context saved by the main thread. Then we have two threads executing on the main thread's stack which is a Bad Thing. With the way the readline support currently handles SIGINT via setjmp()/longjmp(), you really want the main thread and only the main thread to get the SIGINT and perform that longjmp(). Would it be reasonable to block only SIGINT (and not other signals) for all threads but the main thread? That would force SIGINT to be handled by the main thread and eliminate the worry that the wrong thread will do the longjmp() into the main thread's context in the readline code. I agree with large parts of langmead's proposed approach to fixing this, but I do have concerns about the combination of these two parts: * Remove the section of thread creation that blocks signals. * Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) According to the book "Programming with Threads" by Kleinman, Shah, and Smaalders: "Asynchronously generated signals are sent to the process as a whole where they may be serviced by any thread that has the signal unmasked. If more than one thread is able to receive a signal sent to the process, only one is chosen." If we leave SIGINT unmasked on all threads, then the signal handler will need to check the thread ID, and if not the main thread, use pthread_kill(main_thread, SIGINT) to defer the work to the main thread. In that sense, it'd be simpler to block SIGINT in all threads and force the system to route the SIGINT to the main thread directly. Of course if a particular threads implementation doesn't have the desired asynchronous signal routing behavior, maybe leaving SIGINT unmasked and using the pthread_kill(main_thread, SIGINT) technique could work around that. So to sum up, I'm in complete agreement with unblocking most if not all signals in other threads and with langmead's proposals to leverage the benefits provided by sigaction() and siglongjmp() when possible. I have one question though. Would it be reasonable to force SIGINT to be handled only by the main thread, or is there a need for Python threads other than the main thread to handle/receive SIGINT? If the latter then the setjmp()/longjmp() mechanism currently used in the readline module is going to be problematic. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-11 09:59 Message: Logged In: YES user_id=6380 I'm beginning to think that langmead may be on to something: that blocking all signals in all threads is Just Plain Wrong (tm). The Zope SIGSEGV problem is just an example; I have my own beef with SIGTERM, which ends up blocked (together with all other signals) in child processes started from a thread. I would love to see langmead's patch! (For Python 2.4.) Make sure to test the behavior from 465673 (and possible 219772?) after removing the signal blocking but before adding the new fixes, and again after applying those, to make sure 465673 is really gone. Also, I'd like to hear from jasonlowe, who submitted bug 465673 and the patch that caused all the problems, 468347. Maybe his signal-fu has increased since 2001. It would be a miracle to get this into 2.3.4 though... ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-11 04:52 Message: Logged In: YES user_id=6656 That does indeed sound reasonable, but not for 2.3.4 (professional cowardice, I'm afraid). Good luck! ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 18:46 Message: Logged In: YES user_id=119306 The original bug that added the signal blocking, #465673, seems to be exposing itself via a combination of threads and readline. Is it possible that it is the problem is there and not within the signal handling code itself? (especially since it installs and removes a SIGINT handler, possibly causing a race condition with the code within the signal handler when it re-installs itself. On systems that have sigaction, should python need to re-install handlers at all? ) I'm tempted to try to the following, and if it works submit a patch. Does this seem like it would be the right direction? * Remove the section of thread creation that blocks signals. * Look for sections of code may have reentrancy issues, like: ** On machines with reliable signals, keep the signal handler installed, rather than reinstalling it within the handler. ** Change Py_AddPendingCall to use a real semaphore, if available, rather than a busy flag. ** Change readline.c to use more thread safe constructs where available (arrange so that the longjmp out of the signal handler is only executed for the thread that is using readline, and use siglongjmp if available) and then see if issues like this one are solved without reintroducing issues from 465673. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-10 16:06 Message: Logged In: YES user_id=31435 Unassigned (was assigned to Guido, but doesn't sound like he's going to do more with it). ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-10 15:47 Message: Logged In: YES user_id=6380 I agree with Anthony, too much risk for 2.3.4. I don't claim to understand this code any more; in particular the signal blocking code that's currently there wasn't written by me and if I checked it in, I did it hoping for the best... Langmead is right about signal asynchrony. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-10 14:45 Message: Logged In: YES user_id=119306 Unfortunately, in pthreads the "synchronous" doesn't apply to a signal number, but its method of delivery. You can deliver a "SIGSEGV" asynchronously with the "kill" command, and you send normally asynchronous signals with pthread_kill. What does is unblock signals like SIGSEGV which are likely to be sent synchronously from the OS and are unlikely to be handled by normal processes as asynchronous handlers. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-10 12:08 Message: Logged In: YES user_id=29957 I'd strongly prefer that this go into the trunk, and sooner, rather than later. I'd even more strongly prefer that this not go anywhere near the release23-maint branch, at least until _after_ 2.3.4 is done. If there ends up being a nice easy way to do this, great! We can cut a 2.3.5 around the same time as 2.4 final. Putting this into 2.3.4 seems, to me, to be a hell of a risk. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-09 21:13 Message: Logged In: YES user_id=31435 Whether you're off the hook depends on whether you're determined to be . I don't run on Unixish systems, so I'm not a lot of use here. The problem you've had with unkillable subprocesses also affects Zope, and you'll recall that the zdaemon business tries to keep Zope sites running via signal cruft. As people have tried to move from Python 2.1 to Python 2.3, they're discovering that Zope sites fail hard because of the signal- blocking added after 2.1: "when an external python module segfaults during a zope request ... the remaining worker threads are deadlocked", from http://tinyurl.com/2qslw and zdaemon doesn't do its job then. Andrew has in mind a scheme for not blocking "synchronous" signals, which makes sense to me, but I'm out of touch with this stuff. If you can't review it, who can? It would sure be nice to get a resolution into 2.3.4, although I understand that may be perceived as too risky. The alternative from my immediate POV is that people running Zope-based apps on some Unixish systems stay away from Python 2.3, which is a real shame. For that matter, unless this is resolved, I suppose they'll stay away from Python 2.4 too. ---------------------------------------------------------------------- Comment By: Guido van Rossum (gvanrossum) Date: 2004-05-07 17:26 Message: Logged In: YES user_id=6380 (You had me confused for a bit -- I thought you meant Python 2.3, but you meant file revision 2.3, which was in 1994...) It can't be as simple as that; the 1994 code (rev 2.3) initializes both main_pid and main_thread, and checks one or the other in different places. The NOTES in that version don't shed much light on the issue except claiming that checking getpid() is a hack that works on three platforms named: SGI, Solaris, and POSIX threads. The code that is re-initializing main_pid in PyOS_AfterFork()was added much later (rev 2.30, in 1997). Here's my theory. On SGI IRIX, like on pre-2.6-kernel-Linux, getpid() differs per thread, and SIGINT is sent to each thread. The getpid() test here does the right thing: it only sets the flag once (in the handler in the main thread). On Solaris, the getpid() test is a no-op, which is fine sice only one thread gets the signal handler. Those were the only two cases that the getpid() test really cared for; the NOTES section speculated that it would also work with POSIX threads if the signal was only delivered to the main thread. Conclusion: the getpid() test was *not* a mistake, and replacing it with a get_thread_ident() test is not the right answer. But the getpid() test is probably not correct for all pthreads implementations, and some fix may be necessary. I also agree that blocking all signals is too aggressive, but am not sure what to do about this either. (It has caused some problems in my own code where I was spawning a subprocess in a thread, and the subprocess inherited the blocked signals, causing it to be unkillable except through SIGKILL.) Am I off the hook now? ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-07 12:50 Message: Logged In: YES user_id=31435 Assigned to Guido to get an answer to one of the questions here: Guido, signal_handler() checks getpid() against main_pid, and has ever since revision 2.3 (when you first taught signalmodule.c about threads). But on every pthreads box except for Linux, get_pid() should always equal main_pid (even after a fork). What was the intent? I read the comments the same as Andrew does here, that the intent was to check thread identity, not process identity. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:59 Message: Logged In: YES user_id=119306 mwh wrote: "when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all." The bug report that caused the patch to be created was originally reported on Solaris, which has a more correct pthreads implementation. I'm now wondering if that problem was not caused by signals being handled by the spawned threads, but rather that the signal handler does a check for "if (getpid() == main_pid)" rather than "(PyThread_get_thread_ident() == main_thread)". One a standard's compliant pthreads implementation, and even on Solaris, getpid() will always "==" "main_pid". For the Linux case, we may have a more modern working threads implementation now, but when the old LinuxThreads style behavior was out and deployed for 8 years or so, it will probably be around for a while. ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-07 09:48 Message: Logged In: YES user_id=119306 There are two different thread related patches that I submitted, I agree that is pretty radical. (Its the one that tests at configure time for LinuxThreads peculiarities and alters the thread spawning and signal related activities accordingly.) A different related signal patch might be more appealing to you. It only unblocks signals like segmentation faults that creates synchronously sends to itself and that a pthreads implementation will always send to the faulting thread. (whether it blocks it or not.) ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 09:06 Message: Logged In: YES user_id=29957 Any patches in this area, I'd prefer to see on the trunk, along with tests to exercise it (and confirm that it's not breaking something else). We can then give it a solid testing during the 2.4 release cycle. I don't want to have to stretch the bugfix release cycle out to have alphas, betas and the like. This seems like huge piles of no-fun. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-07 08:56 Message: Logged In: YES user_id=6656 Note that there is an attempt at a configure test in 948614, but it seems very LinuxThreads specific. I agree with Anthony that this area is very scary. The last thing we want to do a fortnight before release is break things somewhere they currently work. On the gripping hand, when there's a modern, actually working implementation of pthreads, I don't think we actually need to block signals at all. I certainly don't have the threads-fu to come up with appropriate configure/pyport.h magic though. I'm not sure I have the energy to test a patch on all the testdrive, snake farm and SF compile farm machines either. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-07 08:39 Message: Logged In: YES user_id=29957 We're a week out from release-candidate, and this seems (to me) to be an area that's fraught with risk. The terms "HP/UX" and "threads" have also cropped up, which, for me, is a marker of "here be sodding great big dragons". I don't mind delaying the release if it's necessary, and there's a definite path to getting a nice clean fix in that won't break things for some other class of platform. This stuff looks like being a beast to test for, though. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-06 16:05 Message: Logged In: YES user_id=31435 Boosting priority, hoping to attract interest before 2.3.4. Patch 949332 looks relevant. ---------------------------------------------------------------------- Comment By: Kjetil Jacobsen (kjetilja) Date: 2004-05-05 04:28 Message: Logged In: YES user_id=5685 I've experienced similar behaviour with hung threads on other platforms such as HP/UX, so we should consider letting through some signals to all threads on all platforms. For instance, very few apps use signal handlers for SIGILL, SIGFPE, SIGSEGV, SIGBUS and SIGABRT, so unblocking those signals should not cause much breakage compared to the breakage caused by blocking all signals. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-05-04 10:44 Message: Logged In: YES user_id=31435 Noting that this has become a semi-frequent topic on the zope-dev mailing list, most recently in the "Segfault and Deadlock" thread starting here: ---------------------------------------------------------------------- Comment By: Andrew Langmead (langmead) Date: 2004-05-04 10:00 Message: Logged In: YES user_id=119306 The issue is that the threading implementation in Linux kernels previous to 2.6 diverged from the pthreads standard for signal handling. Normally signals are sent to the process and can be handled by any thread. In the LinuxThreads implementation of pthreads, signals are sent to a specific thread. If that thread blocks signals (which is what happens to all threads spawned in Python 2.2) then those signals do not get routed to a thread with them unblocked (what Python calls the "main thread") The new threading facility in Linux 2.6, the NPTL, does not have this signal handling bug. A simple python script that shows the problem is included below. This will hang in Linux kernels before 2.6 or RedHat customized kernels before RH9. #!/usr/bin/python import signal import thread import os def handle_signals(sig, frame): pass def send_signals(): os.kill(os.getpid(), signal.SIGSEGV) signal.signal(signal.SIGSEGV, handle_signals) thread.start_new_thread(send_signals, ()) signal.pause() ---------------------------------------------------------------------- Comment By: Greg Jones (morngnstar) Date: 2003-06-18 19:54 Message: Logged In: YES user_id=554883 Related to Bug #756940. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=756924&group_id=5470 From noreply at sourceforge.net Wed May 26 13:35:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 13:35:54 2004 Subject: [ python-Bugs-960995 ] test_zlib is too slow Message-ID: Bugs item #960995, was opened at 2004-05-26 18:35 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960995&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: test_zlib is too slow Initial Comment: I don't know what it's doing, but I've never seen it fail and waiting for it has certainly wasted quite a lot of my life :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960995&group_id=5470 From noreply at sourceforge.net Wed May 26 13:36:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 13:41:03 2004 Subject: [ python-Bugs-899109 ] 1==float('nan') Message-ID: Bugs item #899109, was opened at 2004-02-17 21:47 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=899109&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 >Status: Closed Resolution: Fixed Priority: 3 Submitted By: Arman Bostani (arman0) Assigned to: Michael Hudson (mwh) Summary: 1==float('nan') Initial Comment: In python 2.3.1 on Linux/x86, X==float('nan') evaluates to True given any number value for X. I admit to not knowing much about IEEE arithmetic. But, is this Kosher? -arman ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-26 18:36 Message: Logged In: YES user_id=6656 I removed float_compare as discussed below. Closing. Objects/floatobject.c revision 2.130 ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-03-01 13:48 Message: Logged In: YES user_id=6656 If I'm reading the pybench tea leaves correctly, you don't have anything to worry about here. CompareFloatsIntegers is significantly faster (~30%), everything else is in the noise. Removing float_compare speeds up CompareFloatsIntegers even more (~40%, now) and has little impact on anything else. I don't believe PyObject_Compare() is a performance hotspot when the types involved implement rich comparisons. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-02-26 13:09 Message: Logged In: YES user_id=38388 Michael, please be more careful about these changes: we certainly do not want to lose a significant number of cycles just to remedy this rather uncommon error situation. The cleanest solution seems to be to just raise an exception in the compare logic (which is perfectly OK). Removing the tp_compare slot will result in PyObject_Compare() to take a different path and is likely going to cause a performance hit. Whatever you chose to do about this, please make sure that we don't lose performance in float compares. pybench has a special test for float comparisons which you could use as basis for this. Thanks. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-02-26 12:51 Message: Logged In: YES user_id=6656 Hmm. I agree the situation wrt. float_compare as in CVS is a problem. Not sure I have the courage to have float_compare starting to set exceptions, though. Just removing float_compare seems to be a conservative change (yes, it can give silly answers, but this is nothing new). I certainly don't have the courage to change PyObject_Compare unless someone like Tim or Guido says I should :-) ---------------------------------------------------------------------- Comment By: Paul Eggert (eggert) Date: 2004-02-20 21:10 Message: Logged In: YES user_id=17848 Sorry, I didn't intend a rant, and I apologize if any offense was given. I'll try to use a milder tone henceforth. To answer Michael's question: yes, now that your patch is installed, float_compare is incompatible with rich comparison; for example cmp(NaN,1.0) returns 0 even though NaN==1.0 returns False. Another problem: cmp() initially uses pointer comparison to compare objects, so if x is a NaN then cmp(x,x) returns 0. I don't think that removing float_compare will fix the problem, as it would simply cause cmp to return an incorrect value when given a NaN. If you prefer IEEE semantics to sorting NaNs before numbers, another possibility is for cmp to raise an exception when given a NaN, since in that case cmp can't return any value that is compatible with IEEE floating point. Here's a patch to implement this proposal. It does not rely on isnan or a substitute, so I hope this answers Tim's objection. Note that PyObject_Compare needs a minor change, too; since user-defined types may have the same problem that Float does, it's simplest just to omit the optimization there. =================================================================== RCS file: Objects/floatobject.c,v retrieving revision 2.3.3.2 retrieving revision 2.3.3.3 diff -pu -r2.3.3.2 -r2.3.3.3 --- Objects/floatobject.c 2004/02/20 19:45:01 2.3.3.2 +++ Objects/floatobject.c 2004/02/20 21:04:20 2.3.3.3 @@ -367,7 +367,15 @@ float_compare(PyFloatObject *v, PyFloatO { double i = v->ob_fval; double j = w->ob_fval; - return (i < j) ? -1 : (i > j) ? 1 : 0; + if (i < j) + return -1; + if (i > j) + return 1; + if (i == j) + return 0; + PyErr_SetString(PyExc_FloatingPointError, + "cannot compare NaNs using cmp"); + return -2; } static PyObject* =================================================================== RCS file: Objects/object.c,v retrieving revision 2.3.3.0 retrieving revision 2.3.3.1 diff -pu -r2.3.3.0 -r2.3.3.1 --- Objects/object.c 2003/04/18 00:45:58 2.3.3.0 +++ Objects/object.c 2004/02/20 21:04:20 2.3.3.1 @@ -877,8 +877,6 @@ PyObject_Compare(PyObject *v, PyObject * PyErr_BadInternalCall(); return -1; } - if (v == w) - return 0; vtp = v->ob_type; compare_nesting++; if (compare_nesting > NESTING_LIMIT && ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-02-20 11:09 Message: Logged In: YES user_id=6656 Paul, I appreciate your attention to detail, but your last comment is essentially a rant that puts words into Tim's and my mouth. My patch (which is what got checked in) *is* attached to this report, or you can see it on the python-checkins list. There was some ancillary discussion on python-dev (under the admittedly inscrutable title of "Weekly Python Bug/Patch Summary") about this, where it was uncovered that MSVC 7.1 (which is what Python 2.4 will be compiled with) adheres to the C99 rules for comparisons involving NaNs. So with my patch, on all major platforms, float comparisons are likely to follow C99 rules in Python 2.4. We're still not going to promise anything, though. I admit to not doing much research on whether I should/could remove float_compare. It seems that perhaps I should. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-02-19 20:22 Message: Logged In: YES user_id=31435 I specifically said C89. Python doesn't require C99 support, and can't yet, because the majority of compilers used to compile Python aren't yet C99 compilers. Even in C99, that x! =x is 1 for NaN is optional behavior, depending on whether the implementation chooses to define __STDC_IEC_559__. The point of my example was solely the result of x==x. It doesn't matter how a NaN got put into x, MSVC 6 generates code that returns true regardless. It wasn't constructed to confuse anything -- to the contrary, it shows about the most straightforward way to *get* a NaN under MSVC 6, short of playing tricks with bit representation. Michael's patch is a more efficient approach regardless. If Python is to grow an isnan macro (which his patch doesn't need), then it shoiuld grow an isnan() macro that works on all platforms with 754 HW (else it appears to promise thing it can't deliver). Going on to call NaN "less than" everything else goes beyond anything C99 says, and is a debate of its own. No, he didn't remove float_compare(). He could, but only cmp () will invoke it now. ---------------------------------------------------------------------- Comment By: Paul Eggert (eggert) Date: 2004-02-19 19:55 Message: Logged In: YES user_id=17848 C99 does say what happens when NaNs are compared. See, for example, section 7.12.14, which says that NaNs are unordered with respect to other floating point values, and section 7.12.14.1, which says that x>y is equivalent to isgreater(x,y) and therefore returns false if either x or y is a NaN (unless the "invalid" floating-point exception occurs, which isn't happening here and even if it did happen the original code would be broken anyway). Your argument is that any code that uses NaNs returns garbage and so we should give up. But then you have no logical reason to either favor or oppose this patch, as it doesn't alter Python's behavior on non-NaN arguments. It's clear the the patch does fix the comparison problem on most existing platforms. It actually works on Solaris sparc, and I'm sure that it will work on a wide variety of other important platforms. The fact that you've found an incompatibility between MSVC 6 and C99 on one example does not mean that the proposed patch won't work: it merely means that MSVC 6 won't work on your example (which is deliberately constructed to confuse pre-C99 compilers). The proposed patch doesn't contain confusing code like that, so it should work even on pre-C99 compilers. I'm not speaking just from theory here. Similar code has been in GNU sort since 1999, and it works fine in practice. Since the patch fixes the bug on many (perhaps all) real platforms, and it can't introduce a bug on any platform, the patch is a win. Michael Hudson said "Consider the attached." but nothing was attached. I gather from his comments that he added rich comparisons for floats. This will work if the code never invokes the 2.3.3 float_compare -- so can I take it that he removed float_compare? ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-02-19 19:39 Message: Logged In: YES user_id=6656 Tim said yes. Objects/floatobject.c revision 2.127 Misc/NEWS revision 1.936 ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-02-19 19:01 Message: Logged In: YES user_id=6656 Consider the attached. It implements rich comparisons for floats, which makes the behaviour on linux at least very much less silly. All tests except for the considered-hopeless-on-this box audio tests and the new test_tcl pass. Even test_bsddb passed! Twice! (I'm not sure I can claim credit for this :-) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2004-02-19 17:34 Message: Logged In: YES user_id=31435 Sorry, the patch can't work: C89 says nothing about what happens when NaNs are compared, and it's in fact not the case across platforms that "x != x" returns 1 when x is a NaN. The most important counter-example for Python is that the following prints 1.#INF -1.#IND 1 under MSVC 6. #include void main() { double x = 1e300 * 1e300; printf("%g\n", x); x -= x; printf("%g\n", x); printf("%d\n", x == x); } ---------------------------------------------------------------------- Comment By: Paul Eggert (eggert) Date: 2004-02-19 06:39 Message: Logged In: YES user_id=17848 There's an easy fix. Python has long mishandled IEEE special values (see bugs 445484 and 737648, and see PEP 42 "Non-accidental IEEE-754 support" as well as PEP 754), and solving the problem in general will be some work. However, solving the NaN-comparison problem is easy: simply sort NaNs consistently before all numbers. This is the same algorithm that GNU "sort -g" has used for quite some time. =================================================================== RCS file: Include/pyport.h,v retrieving revision 2.3.3.0 diff -pu -r2.3.3.0 Include/pyport.h --- Include/pyport.h 2003/09/30 14:56:50 2.3.3.0 +++ Include/pyport.h 2004/02/19 06:06:10 @@ -221,6 +221,13 @@ extern "C" { #define Py_SAFE_DOWNCAST(VALUE, WIDE, NARROW) (NARROW)(VALUE) #endif +/* Py_IS_NAN(X) + * Return 1 if float or double arg is not a number (NaN), else 0. + * Caution: + * X is evaluated more than once. + */ +#define Py_IS_NAN(X) ((X) != (X)) + /* Py_IS_INFINITY(X) * Return 1 if float or double arg is an infinity, else 0. * Caution: =================================================================== RCS file: Objects/floatobject.c,v retrieving revision 2.3.3.0 diff -pu -r2.3.3.0 Objects/floatobject.c --- Objects/floatobject.c 2003/06/28 20:04:24 2.3.3.0 +++ Objects/floatobject.c 2004/02/19 06:03:37 @@ -367,7 +367,21 @@ float_compare(PyFloatObject *v, PyFloatO { double i = v->ob_fval; double j = w->ob_fval; - return (i < j) ? -1 : (i > j) ? 1 : 0; + int c; + /* Because of NaNs IEEE arithmetic is not a total order. + * Python works better with total orders, so use the same + * total order that GNU sort does: NaNs sort before numbers, + * and NaNs are sorted by internal bit-pattern. + */ + return ((i < j) ? -1 + : (i > j) ? 1 + : (i == j) ? 0 + : !Py_IS_NAN(j) ? -1 /* i is NaN, j is not */ + : !Py_IS_NAN(i) ? 1 /* j is NAN, i is not */ + : /* i and j are both NaNs; compare them bitwise */ + ((c = memcmp(&v->ob_fval, &w->ob_fval, sizeof(v->ob_fval))) + < 0) ? -1 + : (c > 0)); } static long ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-02-18 17:46 Message: Logged In: YES user_id=6656 Man, that's strange (I see the same behaviour on my redhat 9 box with 2.3.3). I don't have time to dig right now, but if you want to, playing with equivalent C programs would be a very good start. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=899109&group_id=5470 From noreply at sourceforge.net Wed May 26 14:14:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 14:14:29 2004 Subject: [ python-Bugs-728330 ] Don't define _SGAPI on IRIX Message-ID: Bugs item #728330, was opened at 2003-04-27 02:21 Message generated for change (Comment added) made by rwgk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 Category: Installation Group: Python 2.3 Status: Open Resolution: Remind Priority: 9 Submitted By: Ralf W. Grosse-Kunstleve (rwgk) Assigned to: Neal Norwitz (nnorwitz) Summary: Don't define _SGAPI on IRIX Initial Comment: Python release: 2.3b1 Platform: IRIX 6.5 MIPSpro Compilers: Version 7.3.1.2m Commands used: ./configure --prefix=/usr/local_cci/Python-2.3b1t make The socket module fails to compile. The output from the compiler is attached. ---------------------------------------------------------------------- >Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 11:14 Message: Logged In: YES user_id=71407 I checked out the python cvs head from scratch and ran configure in the usual way, with the usual result. However, make doesn't work: % make don't know how to make Parser/printgrammar.o (bu42). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 10:07 Message: Logged In: YES user_id=6656 OK, so I've commited revision 1.289 of Modules/socketmodule.c. Can you check if CVS HEAD works for you now? ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 09:59 Message: Logged In: YES user_id=71407 > Is there any chance of the Right Thing actually happening? The experience of the last 13 months suggests "No." SGI's freeware team has not responded to my report for more than a year. I am absolutely not a SGI or socket expert. I only worked on the patch because I was desperate enough and because Maik had already done the groundwork. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 09:27 Message: Logged In: YES user_id=6656 Is this really The Right Thing? I still see /* XXX Using _SGIAPI is the wrong thing, but I don't know what the right thing is. */ Is there any chance of the Right Thing actually happening? If not, I'll check this into HEAD, and if that seems to work, move it to release23-maint in time for 2.3.5. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-22 01:23 Message: Logged In: YES user_id=71407 Based on Maik's posting from 2004-03-08 08:53 I've worked out a small bug-fix patch relative to Python-2.3.4c1 that leads to successful creation of _socket.so under three different versions of IRIX: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.23 with MIPRpro 7.3.1.3 I've consistently used #if defined(__sgi) && _COMPILER_VERSION>700 to restrict the changes to IRIX. However, to be sure I've also verified that the Python installation is still OK under Redhat 8 and Tru64 Unix 5.1. I will upload the patch. It is also available here: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patch For completeness I've also posted these files: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patched_c http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_original_c I've made an effort to keep the patch minimally intrusive, but it is certainly not "clean." However, the status quo is so bad that it can only get better. We have not been able to switch to Python 2.3 because of the problems with the socket module. It would be fantastic if the bug-fix patch could be included in the 2.3.4 final distribution. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2004-03-08 08:53 Message: Logged In: YES user_id=768223 I got Python 2.3.3 compiled with MipsPro Compilers: Version 7.3.1.3m. The following changes were made to satisfy all needs. Line 206 in Modules/socketmodule.c #define _SGIAPI 1 NEW: #undef _XOPEN_SOURCE NEW: #define HAVE_GETADDRINFO 1 NEW: #define HAVE_GETNAMEINFO 1 #define HAVE_INET_PTON NEW: #include NEW: #include NEW: #include #include Line 269: /* #include "addrinfo.h" */ In addition to this I extend the CFLAGS with -D_BSD_TYPES to satisfy the u_short and friends definition. --maik./ ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-03-03 08:20 Message: Logged In: YES user_id=892601 Got it down to one error: % > ./python Lib/test/test_socket.py Traceback (most recent call last): File "Lib/test/test_socket.py", line 6, in ? import socket File "/usr/local/spider/lib/python2.3/socket.py", line 44, in ? import _socket ImportError: 210454:./python: rld: Fatal Error: unresolvable symbol in /usr/local/spider/lib/python2.3/lib-dynload/_socketmodule.so: _xpg5_accept xpg5_accept does appear in any library on our SGI's that are only 1-2 years old. It does appear in libc in very recent revisions of Irix (> 6.5.19?) Does anyone know how to get Python 2.3 to avoid the use of xpg5_accept? Thanks ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-01-12 10:59 Message: Logged In: YES user_id=892601 Has anyone encountered problems with different versions of IRIX 6.5? I finally got sockets to work in Python2.3.3, but only on the most up-to-date version of IRIX (uname -v = 01100304). On any other machine with a lower version (archetecture doesn't seem to matter), I get messages which all seem to be related to sockets: 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_vsnprintf 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recv 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_connect 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_bind 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_sendto 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_setsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_accept 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recvfrom 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_socket 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockname Upgrading all of our SGIs is not something I have control over. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-22 10:52 Message: Logged In: YES user_id=768223 Sorry if there is something unclear. Redefine means: Set a value which is required to satisfy ifdef-statements in the header files. As you see in the definitions, given in the comment, the value for _NO_XOPEN4 is set as a result of some dependencies automatically defined. A redefinition of _SGIAPI in the socketmodule.c is not sufficient alone. Therefore to satisfy all requirements in the header files, you should set the values for _NO_XOPEN4 and _NO_XOPEN5 also manually. Howto do in socketmodule.c? ... #define _SGIAPI 1 #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 ... If you are familiar with regular expressions, then you will know that _NO_XOPEN[45] is a shorthand for _NO_XOPEN4, _NO_XOPEN5. Sorry if there was something wrong in my comment. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-12-22 07:49 Message: Logged In: YES user_id=892601 Can someone please clarify mherta's comment from 2003-12-16? How should we "redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition"? Should those lines of code be added to socketmodule.c, or are they the source of the problem? Is _NO_XOPEN[45] shorthand for _NO_XOPEN[4] and _NO_XOPEN[5] or are they all different? Thanks. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-16 09:27 Message: Logged In: YES user_id=768223 I've tested the latest Python 2.3.3c1 to compile. The problem reported is still present. To avoid the error messages, about the incompatible types of several netdb-functions (like inet_pton), you should redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition. If there are error messages that types like u_short and u_char are not defined, please add -D_BSD_TYPES in the Makefile (BASECFLAGS is okay). If anybody don't need a executable compiled with MipsPro, gcc from the freeware.sgi.com could be a choice. The problem for this seems in /usr/include/standards.h: #define _NO_XOPEN4 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 >= 500))) #define _NO_XOPEN5 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 < 500))) As the first option is true (_XOPEN_SOURCE is set to 600 in pyconfig.h) the second option will be false. The constant #define NI_NUMERICHOST 0x00000002 /* return numeric form of hostname */ is enveloped in a clause #ifdef _NO_XOPEN4 && _NO_XOPEN5 (/usr/include/netdb.h) which is always false. In the manual pages there is a example which uses the constant NI_NUMERICHOST and compiles without error message. I'm not sure to cut off the file pyconfig.h from compiling the socketmodule.c? Any comments? ---------------------------------------------------------------------- Comment By: Hans Nowak (zephyrfalcon) Date: 2003-10-30 19:04 Message: Logged In: YES user_id=173607 My $0.02... Yesterday I successfully compiled Python 2.3.2, with socket module. I'm using an O2 running IRIX 6.5.11f. I didn't encounter any of the problems described before, and had no problems with _SGIAPI or anything. The reason it didn't compile was that u_long and u_char were not defined in some places; /usr/include/netinet/tcp.h to be precise, and a Python file (getaddrinfo.c). I'm using gcc 3.3 (the one downloaded from freeware.sgi.com). If anyone is interested, drop me a mail at hans at zephyrfalcon.org and I'll put the binary online somewhere. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-24 12:59 Message: Logged In: YES user_id=892601 Here's the output from 'gmake', using MIPSpro Compiler 7.30 on IRIX 6.5 Sorry, I don't know how to attach something using the "Add a Comment" box. Is there another way to post? The patch at www.python.org/2.3.2/patches/sgi-socketmodule.patch was edited into socketmodule.c : #include #endif /* GCC 3.2 on Irix 6.5 fails to define this variable at all. Based on the above code, I'd say that the SGI headers are just horked. */ #if defined(__sgi) && !defined(INET_ADDRSTRLEN) #define INET_ADDRSTRLEN 16 #endif /* Generic includes */ #include #include --------------------------------------------------------------------------- cc -OPT:Olimit=0 -DNDEBUG -n32 -O -I. -I./Include -DPy_BUILD_CORE -c ./Modules/socketmodule.c -o Modules/socketmodule.o cc-1047 cc: WARNING File = ./Modules/socketmodule.c, Line = 195 Macro "_SGIAPI" (declared at line 210 of "/usr/include/standards.h") has an incompatible redefinition. #define _SGIAPI 1 ^ cc-1020 cc: ERROR File = /usr/include/netdb.h, Line = 198 The identifier "socklen_t" is undefined. extern int iruserok_sa(const void *, socklen_t, int, const char *, ^ cc-1047 cc: WARNING File = ./Modules/addrinfo.h, Line = 145 Macro "_SS_ALIGNSIZE" (declared at line 246 of "/usr/include/sys/socket.h") has an incompatible redefinition. #define _SS_ALIGNSIZE (sizeof(PY_LONG_LONG)) ^ cc-1020 cc: ERROR File = ./Modules/socketmodule.c, Line = 822 The identifier "NI_NUMERICHOST" is undefined. NI_NUMERICHOST); ^ cc-1515 cc: ERROR File = ./Modules/socketmodule.c, Line = 3054 A value of type "int" cannot be assigned to an entity of type "const char *". retval = inet_ntop(af, packed, ip, sizeof(ip)); ^ 3 errors detected in the compilation of "./Modules/socketmodule.c". gmake: *** [Modules/socketmodule.o] Error 2 ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2003-10-23 03:25 Message: Logged In: YES user_id=6656 Just to be clear, please "attach" the build rather than pasting it inline. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2003-10-22 19:19 Message: Logged In: YES user_id=29957 Did you try 2.3.2 with the patch from the 2.3.2 build bugs page? If it failed, please post the build log of the failing bit to this bug. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-22 11:26 Message: Logged In: YES user_id=892601 Has there been any progress since August on getting Python-2.3 to compile on IRIX 6.5? I really need to have sockets working to use python. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-08-14 04:37 Message: Logged In: YES user_id=768223 I have the same problem on IRIX 6.5.19m and MIPSpro 7.3.1.3m on a FUEL 600Mhz. The situation seems a little bit strange on this system. But I have luck and get it compiled. As the current implementation is not clean, my resolution also. Which should be changed to get it work. The manpage for gethostbyaddr_r says: #include #include #include #include So I included this in the __sgi-clause in socketmodule.c. Looking in the header-files shows, that some variables are defined if #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 are set, before the includes. This defines some more variables, so that pyconfig.h should be changed. #define HAVE_ADDRINFO 1 #define HAVE_SOCKADDR_STORAGE 1 Sure there are warnings about the redefinition of the _(SGIAPI|NO_XOPEN4|NO_XOPEN5) but it compiles and the tests don't fail. Also my applications run without exception or coredump on FUEL, Octane, Octane 2. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-08-01 05:10 Message: Logged In: YES user_id=33168 Ralf, I will try to work on this, but it won't be for a week or two probably. Can you tell me which machines (by name) work and doesn't work. The only IRIX machine I seem to know about it rattler. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-08-01 00:34 Message: Logged In: YES user_id=71407 For the records: The Python2.3 (final) socket module still does not compile under IRIX 6.5.20. It only compiles under 6.5.10. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 11:07 Message: Logged In: YES user_id=31392 I applied a slight variant of Neal's patch and that seemed to fix things. It still seems wrong that we define _SGIAPI, but that will have to wait for 2.3.1. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 06:54 Message: Logged In: YES user_id=31392 I learned a little about inet_ntop() in the last few minutes. It looks like the size_t argument is an older spelling, and the current POSIX says it should be socklen_t. Regardless, it looks like we're using the new definition when IRIX is actually providing the older definition. Neal attached a patch that looks like it should clean up the problem. Have you tried it? I think a better solution would be to teach configure how to find inet_ntop() via netdb.h, but that's probably too hard to get done in the new couple of hours. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 06:42 Message: Logged In: YES user_id=31392 There seem to be two causes of worry. The actual compiler error in your build log and the use of _SGIAPI that Casavanti warns again. It looks like the compiler problems is because our inet_ntop() prototype is wrong. That is Python says the last argument is a socklen_t, but IRIX and Linux man pages both say size_t. Perhaps size_t and socklen_t are different sizes on IRIX, but not on Linux. What happens if you change it to size_t? The problematic code would appear to be these lines from socketmodule.c: #if defined(__sgi)&&_COMPILER_VERSION>700 && !_SGIAPI /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #define _SGIAPI 1 #include "netdb.h" #endif It's the only use the _SGIAPI symbol that Casavant says we shouldn't use. What happens if you remove the use _SGIAPI, i.e. just make it #if defined(__sgi) && _COMPILER_VERSION > 700 /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #include "netdb.h" #endif ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-16 17:27 Message: Logged In: YES user_id=71407 We have confirmed that the socket module in Python 2.3b1 and 2.3b2 fails to compile under these versions of IRIX: 6.5.10, MIPSpro 7.3.1.2m 6.5.19, MIPSpro 7.3.1.3m 6.5.20, MIPSpro 7.3.1.3m (latest available, to my knowledge) If Python 2.3 goes out without a working socket module it will be useless for many people in science where IRIX is still found quite frequently. Maybe even worse, we will also no longer be able to advertise Python as a language that "runs everywhere." ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-15 15:37 Message: Logged In: YES user_id=71407 Date: Tue, 15 Jul 2003 15:06:25 -0500 From: "Brent Casavant" To: "Ralf W. Grosse-Kunstleve" Subject: Re: IRIX & Python Ralf, I read through the sourceforge report. I don't have a sourceforge account so I didn't add there, but feel free to distribute my ramblings. First, they should be very careful about what version of the operating system they are using. Real IPv6 support was only added a few releases ago (6.5.20 I believe, though my memory is hazy). Some of the functions/macros corresponding to IPv6 were introduced prior to that point, but implementation may not have been complete. If they have to play tricks with #define values to get a symbol recongnized it most likely means that SGI didn't intend for that symbol to be used yet. I suspect the problems they are seeing are because they are using some version of IRIX 6.5.x that includes inet_ntoa() definitions under _SGIAPI, but for which real support was not yet complete. The namespace of symbols that begin with a single underscore and a capital letter is reserved for the OS vendors, and the fact that someone is trying to manually insert a definition of _SGIAPI into the source should be a big red warning flag. That said, there is a correct method to deal with the new capabilities of IRIX 6.5.x over time. This comes in two parts. A better explanation is in /usr/include/optional_sym.h First, when a new symbol is added to IRIX, the corresponding header file will always have the following line in it: #pragma optional some_new_symbol The presence of this #pragma will cause the linker to not complain if the symbol is missing when the executable is linked. If SGI neglects to do so it is a bug that should be reported to us. The second part is that calls to these "optional" symbols should be protected by a run-time check to see if the symbol is actually present. Just like this: #include #include . . . if (_MIPS_SYMBOL_PRESENT(some_new_symbol)) { /* New function is present, use it */ qux = some_new_symbol(foo, bar, baz); } else { /* New function isn't present, do something else */ qux = builtin_some_new_symbol(foo, bar, baz); } This ensures that a single executable will be able to function properly (or at least degrade gracefully) on all version of IRIX 6.5.x. In other words you can compile on 6.5.20 and run on 6.5.8, or vice versa. In particular, this means that compile-time detection of features such as inet_ntoa() is incorrect for IRIX 6.5.x -- these decisions should be made at run-time instead. If the decision is made at compile time the executable may well not execute on an older version of IRIX 6.5.x. Unfortunately GNU configure is not equipped to utilize such advanced binary-compatability mechanisms, so there's no "easy" way to get GNU configure'd software to run across all IRIX 6.5.x releases, short of compiling only on the original non-upgraded version of IRIX 6.5. Hope that helps, Brent Casavant On Tue, 15 Jul 2003, Ralf W. Grosse-Kunstleve wrote: > We are developing a Python-based (www.python.org) application for > IRIX. Until now the Python IRIX port was fully supported, but the last > two beta releases of Python (2.3b1 and 2.3b2) make me increasingly > worried because the crucial socket module (TCP/IP sockets) fails to > build. A while ago I've filed a bug report, but the Python developers > appear to be stuck: > > https://sourceforge.net/tracker/? func=detail&aid=728330&group_id=5470&atid=105470 > > Is there a way to bring this to the attention of developers at SGI to > hopefully get some help? > > Thank you in advance! > Ralf > -- Brent Casavant Silicon Graphics, Inc. Where am I? And what am I IRIX O.S. Engineer http://www.sgi.com/ doing in this handbasket? bcasavan@sgi.com 44.8562N 93.1355W 860F -- Unknown ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-06-30 07:21 Message: Logged In: YES user_id=71407 The socket module still fails to compile with Python 2.3b2, in addition to two other extensions. The full make log is attached. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-21 23:47 Message: Logged In: YES user_id=21627 I think it would be best if detect presence of inet_aton/inet_pton. Is IPv6 support possible on this system? Then it would be best if that was detected, as well. If it is not possible to detect aton/pton/ipv6, what is the problem with not using them? I'd like to have a look at the relevant headers (in particular to understand the duplicate definition of _SGIAPI), indeed. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-05-21 16:35 Message: Logged In: YES user_id=33168 Argh!!! This is a nightmare. The attached hack ^h^h^h patch fixes the problem, but ... _SGIAPI needs to be defined (~192 in socketmodule.c) in order to get the prototypes for inet_aton, inet_pton. But _SGIAPI can't be defined in the standard header files because XOPEN_SOURCE is defined. Martin, do you have any other suggestions? I don't see how to do this right within configure. If you want I can send you the header file(s). ---------------------------------------------------------------------- Comment By: alexandre gillet (gillet) Date: 2003-05-12 10:41 Message: Logged In: YES user_id=150999 I had the same problem compiling socket on Irix. We did find that on our system ENABLE_IPV6 is not define and all the error are cause by that. System that do not enable IPV6 are not well supported with the socketmodule.c code. We were able to compile socket after making few change in the code: I am not sure if I did it right but it compile. I will post my patch but I do not see any place to attach my file example: See line 2794,2800 + #ifdef ENABLE_IPV6 char packed[MAX(sizeof(struct in_addr), sizeof(struct in6_addr))]; + #else + char packed[sizeof(struct in_addr)]; + #endif I am not sure how to post the patch or file. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-05-05 11:44 Message: Logged In: YES user_id=71407 This is not in my area of expertise, but I've tried a couple of (probably stupid) things: #include in socketmodule.c instead of #define _SGIAPI. This resulted in even more errors. ./configure --disable-ipv6 Same errors as before. I am lost here. Sorry. But you still have ssh access to rattler.lbl.gov if that helps. Accounts for other Python developers are a possibility. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-04 05:39 Message: Logged In: YES user_id=21627 Can you analyse these compiler errors in more detail, please? What, in your opinion, is causing these problems, and how would you correct them? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 From noreply at sourceforge.net Wed May 26 14:17:59 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 14:18:05 2004 Subject: [ python-Bugs-960995 ] test_zlib is too slow Message-ID: Bugs item #960995, was opened at 2004-05-26 12:35 Message generated for change (Comment added) made by rhettinger You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960995&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: test_zlib is too slow Initial Comment: I don't know what it's doing, but I've never seen it fail and waiting for it has certainly wasted quite a lot of my life :-) ---------------------------------------------------------------------- >Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-26 13:17 Message: Logged In: YES user_id=80475 I hate this slow test. If you want to label this as explictly called resource, regrtest -u zlib , then be my guest. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960995&group_id=5470 From noreply at sourceforge.net Wed May 26 14:19:56 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 14:21:00 2004 Subject: [ python-Bugs-728330 ] Don't define _SGAPI on IRIX Message-ID: Bugs item #728330, was opened at 2003-04-27 10:21 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 Category: Installation Group: Python 2.3 Status: Open Resolution: Remind Priority: 9 Submitted By: Ralf W. Grosse-Kunstleve (rwgk) Assigned to: Neal Norwitz (nnorwitz) Summary: Don't define _SGAPI on IRIX Initial Comment: Python release: 2.3b1 Platform: IRIX 6.5 MIPSpro Compilers: Version 7.3.1.2m Commands used: ./configure --prefix=/usr/local_cci/Python-2.3b1t make The socket module fails to compile. The output from the compiler is attached. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-26 19:19 Message: Logged In: YES user_id=6656 Uh, that's odd. Is IRIX make particularly braindead? Can you try gmake? Maybe we've let something non-portable slip in. At any rate, I think this should be done in a separate report... ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 19:14 Message: Logged In: YES user_id=71407 I checked out the python cvs head from scratch and ran configure in the usual way, with the usual result. However, make doesn't work: % make don't know how to make Parser/printgrammar.o (bu42). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 18:07 Message: Logged In: YES user_id=6656 OK, so I've commited revision 1.289 of Modules/socketmodule.c. Can you check if CVS HEAD works for you now? ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 17:59 Message: Logged In: YES user_id=71407 > Is there any chance of the Right Thing actually happening? The experience of the last 13 months suggests "No." SGI's freeware team has not responded to my report for more than a year. I am absolutely not a SGI or socket expert. I only worked on the patch because I was desperate enough and because Maik had already done the groundwork. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 17:27 Message: Logged In: YES user_id=6656 Is this really The Right Thing? I still see /* XXX Using _SGIAPI is the wrong thing, but I don't know what the right thing is. */ Is there any chance of the Right Thing actually happening? If not, I'll check this into HEAD, and if that seems to work, move it to release23-maint in time for 2.3.5. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-22 09:23 Message: Logged In: YES user_id=71407 Based on Maik's posting from 2004-03-08 08:53 I've worked out a small bug-fix patch relative to Python-2.3.4c1 that leads to successful creation of _socket.so under three different versions of IRIX: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.23 with MIPRpro 7.3.1.3 I've consistently used #if defined(__sgi) && _COMPILER_VERSION>700 to restrict the changes to IRIX. However, to be sure I've also verified that the Python installation is still OK under Redhat 8 and Tru64 Unix 5.1. I will upload the patch. It is also available here: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patch For completeness I've also posted these files: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patched_c http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_original_c I've made an effort to keep the patch minimally intrusive, but it is certainly not "clean." However, the status quo is so bad that it can only get better. We have not been able to switch to Python 2.3 because of the problems with the socket module. It would be fantastic if the bug-fix patch could be included in the 2.3.4 final distribution. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2004-03-08 16:53 Message: Logged In: YES user_id=768223 I got Python 2.3.3 compiled with MipsPro Compilers: Version 7.3.1.3m. The following changes were made to satisfy all needs. Line 206 in Modules/socketmodule.c #define _SGIAPI 1 NEW: #undef _XOPEN_SOURCE NEW: #define HAVE_GETADDRINFO 1 NEW: #define HAVE_GETNAMEINFO 1 #define HAVE_INET_PTON NEW: #include NEW: #include NEW: #include #include Line 269: /* #include "addrinfo.h" */ In addition to this I extend the CFLAGS with -D_BSD_TYPES to satisfy the u_short and friends definition. --maik./ ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-03-03 16:20 Message: Logged In: YES user_id=892601 Got it down to one error: % > ./python Lib/test/test_socket.py Traceback (most recent call last): File "Lib/test/test_socket.py", line 6, in ? import socket File "/usr/local/spider/lib/python2.3/socket.py", line 44, in ? import _socket ImportError: 210454:./python: rld: Fatal Error: unresolvable symbol in /usr/local/spider/lib/python2.3/lib-dynload/_socketmodule.so: _xpg5_accept xpg5_accept does appear in any library on our SGI's that are only 1-2 years old. It does appear in libc in very recent revisions of Irix (> 6.5.19?) Does anyone know how to get Python 2.3 to avoid the use of xpg5_accept? Thanks ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-01-12 18:59 Message: Logged In: YES user_id=892601 Has anyone encountered problems with different versions of IRIX 6.5? I finally got sockets to work in Python2.3.3, but only on the most up-to-date version of IRIX (uname -v = 01100304). On any other machine with a lower version (archetecture doesn't seem to matter), I get messages which all seem to be related to sockets: 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_vsnprintf 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recv 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_connect 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_bind 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_sendto 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_setsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_accept 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recvfrom 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_socket 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockname Upgrading all of our SGIs is not something I have control over. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-22 18:52 Message: Logged In: YES user_id=768223 Sorry if there is something unclear. Redefine means: Set a value which is required to satisfy ifdef-statements in the header files. As you see in the definitions, given in the comment, the value for _NO_XOPEN4 is set as a result of some dependencies automatically defined. A redefinition of _SGIAPI in the socketmodule.c is not sufficient alone. Therefore to satisfy all requirements in the header files, you should set the values for _NO_XOPEN4 and _NO_XOPEN5 also manually. Howto do in socketmodule.c? ... #define _SGIAPI 1 #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 ... If you are familiar with regular expressions, then you will know that _NO_XOPEN[45] is a shorthand for _NO_XOPEN4, _NO_XOPEN5. Sorry if there was something wrong in my comment. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-12-22 15:49 Message: Logged In: YES user_id=892601 Can someone please clarify mherta's comment from 2003-12-16? How should we "redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition"? Should those lines of code be added to socketmodule.c, or are they the source of the problem? Is _NO_XOPEN[45] shorthand for _NO_XOPEN[4] and _NO_XOPEN[5] or are they all different? Thanks. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-16 17:27 Message: Logged In: YES user_id=768223 I've tested the latest Python 2.3.3c1 to compile. The problem reported is still present. To avoid the error messages, about the incompatible types of several netdb-functions (like inet_pton), you should redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition. If there are error messages that types like u_short and u_char are not defined, please add -D_BSD_TYPES in the Makefile (BASECFLAGS is okay). If anybody don't need a executable compiled with MipsPro, gcc from the freeware.sgi.com could be a choice. The problem for this seems in /usr/include/standards.h: #define _NO_XOPEN4 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 >= 500))) #define _NO_XOPEN5 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 < 500))) As the first option is true (_XOPEN_SOURCE is set to 600 in pyconfig.h) the second option will be false. The constant #define NI_NUMERICHOST 0x00000002 /* return numeric form of hostname */ is enveloped in a clause #ifdef _NO_XOPEN4 && _NO_XOPEN5 (/usr/include/netdb.h) which is always false. In the manual pages there is a example which uses the constant NI_NUMERICHOST and compiles without error message. I'm not sure to cut off the file pyconfig.h from compiling the socketmodule.c? Any comments? ---------------------------------------------------------------------- Comment By: Hans Nowak (zephyrfalcon) Date: 2003-10-31 03:04 Message: Logged In: YES user_id=173607 My $0.02... Yesterday I successfully compiled Python 2.3.2, with socket module. I'm using an O2 running IRIX 6.5.11f. I didn't encounter any of the problems described before, and had no problems with _SGIAPI or anything. The reason it didn't compile was that u_long and u_char were not defined in some places; /usr/include/netinet/tcp.h to be precise, and a Python file (getaddrinfo.c). I'm using gcc 3.3 (the one downloaded from freeware.sgi.com). If anyone is interested, drop me a mail at hans at zephyrfalcon.org and I'll put the binary online somewhere. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-24 20:59 Message: Logged In: YES user_id=892601 Here's the output from 'gmake', using MIPSpro Compiler 7.30 on IRIX 6.5 Sorry, I don't know how to attach something using the "Add a Comment" box. Is there another way to post? The patch at www.python.org/2.3.2/patches/sgi-socketmodule.patch was edited into socketmodule.c : #include #endif /* GCC 3.2 on Irix 6.5 fails to define this variable at all. Based on the above code, I'd say that the SGI headers are just horked. */ #if defined(__sgi) && !defined(INET_ADDRSTRLEN) #define INET_ADDRSTRLEN 16 #endif /* Generic includes */ #include #include --------------------------------------------------------------------------- cc -OPT:Olimit=0 -DNDEBUG -n32 -O -I. -I./Include -DPy_BUILD_CORE -c ./Modules/socketmodule.c -o Modules/socketmodule.o cc-1047 cc: WARNING File = ./Modules/socketmodule.c, Line = 195 Macro "_SGIAPI" (declared at line 210 of "/usr/include/standards.h") has an incompatible redefinition. #define _SGIAPI 1 ^ cc-1020 cc: ERROR File = /usr/include/netdb.h, Line = 198 The identifier "socklen_t" is undefined. extern int iruserok_sa(const void *, socklen_t, int, const char *, ^ cc-1047 cc: WARNING File = ./Modules/addrinfo.h, Line = 145 Macro "_SS_ALIGNSIZE" (declared at line 246 of "/usr/include/sys/socket.h") has an incompatible redefinition. #define _SS_ALIGNSIZE (sizeof(PY_LONG_LONG)) ^ cc-1020 cc: ERROR File = ./Modules/socketmodule.c, Line = 822 The identifier "NI_NUMERICHOST" is undefined. NI_NUMERICHOST); ^ cc-1515 cc: ERROR File = ./Modules/socketmodule.c, Line = 3054 A value of type "int" cannot be assigned to an entity of type "const char *". retval = inet_ntop(af, packed, ip, sizeof(ip)); ^ 3 errors detected in the compilation of "./Modules/socketmodule.c". gmake: *** [Modules/socketmodule.o] Error 2 ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2003-10-23 11:25 Message: Logged In: YES user_id=6656 Just to be clear, please "attach" the build rather than pasting it inline. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2003-10-23 03:19 Message: Logged In: YES user_id=29957 Did you try 2.3.2 with the patch from the 2.3.2 build bugs page? If it failed, please post the build log of the failing bit to this bug. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-22 19:26 Message: Logged In: YES user_id=892601 Has there been any progress since August on getting Python-2.3 to compile on IRIX 6.5? I really need to have sockets working to use python. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-08-14 12:37 Message: Logged In: YES user_id=768223 I have the same problem on IRIX 6.5.19m and MIPSpro 7.3.1.3m on a FUEL 600Mhz. The situation seems a little bit strange on this system. But I have luck and get it compiled. As the current implementation is not clean, my resolution also. Which should be changed to get it work. The manpage for gethostbyaddr_r says: #include #include #include #include So I included this in the __sgi-clause in socketmodule.c. Looking in the header-files shows, that some variables are defined if #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 are set, before the includes. This defines some more variables, so that pyconfig.h should be changed. #define HAVE_ADDRINFO 1 #define HAVE_SOCKADDR_STORAGE 1 Sure there are warnings about the redefinition of the _(SGIAPI|NO_XOPEN4|NO_XOPEN5) but it compiles and the tests don't fail. Also my applications run without exception or coredump on FUEL, Octane, Octane 2. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-08-01 13:10 Message: Logged In: YES user_id=33168 Ralf, I will try to work on this, but it won't be for a week or two probably. Can you tell me which machines (by name) work and doesn't work. The only IRIX machine I seem to know about it rattler. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-08-01 08:34 Message: Logged In: YES user_id=71407 For the records: The Python2.3 (final) socket module still does not compile under IRIX 6.5.20. It only compiles under 6.5.10. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 19:07 Message: Logged In: YES user_id=31392 I applied a slight variant of Neal's patch and that seemed to fix things. It still seems wrong that we define _SGIAPI, but that will have to wait for 2.3.1. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 14:54 Message: Logged In: YES user_id=31392 I learned a little about inet_ntop() in the last few minutes. It looks like the size_t argument is an older spelling, and the current POSIX says it should be socklen_t. Regardless, it looks like we're using the new definition when IRIX is actually providing the older definition. Neal attached a patch that looks like it should clean up the problem. Have you tried it? I think a better solution would be to teach configure how to find inet_ntop() via netdb.h, but that's probably too hard to get done in the new couple of hours. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 14:42 Message: Logged In: YES user_id=31392 There seem to be two causes of worry. The actual compiler error in your build log and the use of _SGIAPI that Casavanti warns again. It looks like the compiler problems is because our inet_ntop() prototype is wrong. That is Python says the last argument is a socklen_t, but IRIX and Linux man pages both say size_t. Perhaps size_t and socklen_t are different sizes on IRIX, but not on Linux. What happens if you change it to size_t? The problematic code would appear to be these lines from socketmodule.c: #if defined(__sgi)&&_COMPILER_VERSION>700 && !_SGIAPI /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #define _SGIAPI 1 #include "netdb.h" #endif It's the only use the _SGIAPI symbol that Casavant says we shouldn't use. What happens if you remove the use _SGIAPI, i.e. just make it #if defined(__sgi) && _COMPILER_VERSION > 700 /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #include "netdb.h" #endif ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-17 01:27 Message: Logged In: YES user_id=71407 We have confirmed that the socket module in Python 2.3b1 and 2.3b2 fails to compile under these versions of IRIX: 6.5.10, MIPSpro 7.3.1.2m 6.5.19, MIPSpro 7.3.1.3m 6.5.20, MIPSpro 7.3.1.3m (latest available, to my knowledge) If Python 2.3 goes out without a working socket module it will be useless for many people in science where IRIX is still found quite frequently. Maybe even worse, we will also no longer be able to advertise Python as a language that "runs everywhere." ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-15 23:37 Message: Logged In: YES user_id=71407 Date: Tue, 15 Jul 2003 15:06:25 -0500 From: "Brent Casavant" To: "Ralf W. Grosse-Kunstleve" Subject: Re: IRIX & Python Ralf, I read through the sourceforge report. I don't have a sourceforge account so I didn't add there, but feel free to distribute my ramblings. First, they should be very careful about what version of the operating system they are using. Real IPv6 support was only added a few releases ago (6.5.20 I believe, though my memory is hazy). Some of the functions/macros corresponding to IPv6 were introduced prior to that point, but implementation may not have been complete. If they have to play tricks with #define values to get a symbol recongnized it most likely means that SGI didn't intend for that symbol to be used yet. I suspect the problems they are seeing are because they are using some version of IRIX 6.5.x that includes inet_ntoa() definitions under _SGIAPI, but for which real support was not yet complete. The namespace of symbols that begin with a single underscore and a capital letter is reserved for the OS vendors, and the fact that someone is trying to manually insert a definition of _SGIAPI into the source should be a big red warning flag. That said, there is a correct method to deal with the new capabilities of IRIX 6.5.x over time. This comes in two parts. A better explanation is in /usr/include/optional_sym.h First, when a new symbol is added to IRIX, the corresponding header file will always have the following line in it: #pragma optional some_new_symbol The presence of this #pragma will cause the linker to not complain if the symbol is missing when the executable is linked. If SGI neglects to do so it is a bug that should be reported to us. The second part is that calls to these "optional" symbols should be protected by a run-time check to see if the symbol is actually present. Just like this: #include #include . . . if (_MIPS_SYMBOL_PRESENT(some_new_symbol)) { /* New function is present, use it */ qux = some_new_symbol(foo, bar, baz); } else { /* New function isn't present, do something else */ qux = builtin_some_new_symbol(foo, bar, baz); } This ensures that a single executable will be able to function properly (or at least degrade gracefully) on all version of IRIX 6.5.x. In other words you can compile on 6.5.20 and run on 6.5.8, or vice versa. In particular, this means that compile-time detection of features such as inet_ntoa() is incorrect for IRIX 6.5.x -- these decisions should be made at run-time instead. If the decision is made at compile time the executable may well not execute on an older version of IRIX 6.5.x. Unfortunately GNU configure is not equipped to utilize such advanced binary-compatability mechanisms, so there's no "easy" way to get GNU configure'd software to run across all IRIX 6.5.x releases, short of compiling only on the original non-upgraded version of IRIX 6.5. Hope that helps, Brent Casavant On Tue, 15 Jul 2003, Ralf W. Grosse-Kunstleve wrote: > We are developing a Python-based (www.python.org) application for > IRIX. Until now the Python IRIX port was fully supported, but the last > two beta releases of Python (2.3b1 and 2.3b2) make me increasingly > worried because the crucial socket module (TCP/IP sockets) fails to > build. A while ago I've filed a bug report, but the Python developers > appear to be stuck: > > https://sourceforge.net/tracker/? func=detail&aid=728330&group_id=5470&atid=105470 > > Is there a way to bring this to the attention of developers at SGI to > hopefully get some help? > > Thank you in advance! > Ralf > -- Brent Casavant Silicon Graphics, Inc. Where am I? And what am I IRIX O.S. Engineer http://www.sgi.com/ doing in this handbasket? bcasavan@sgi.com 44.8562N 93.1355W 860F -- Unknown ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-06-30 15:21 Message: Logged In: YES user_id=71407 The socket module still fails to compile with Python 2.3b2, in addition to two other extensions. The full make log is attached. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-22 07:47 Message: Logged In: YES user_id=21627 I think it would be best if detect presence of inet_aton/inet_pton. Is IPv6 support possible on this system? Then it would be best if that was detected, as well. If it is not possible to detect aton/pton/ipv6, what is the problem with not using them? I'd like to have a look at the relevant headers (in particular to understand the duplicate definition of _SGIAPI), indeed. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-05-22 00:35 Message: Logged In: YES user_id=33168 Argh!!! This is a nightmare. The attached hack ^h^h^h patch fixes the problem, but ... _SGIAPI needs to be defined (~192 in socketmodule.c) in order to get the prototypes for inet_aton, inet_pton. But _SGIAPI can't be defined in the standard header files because XOPEN_SOURCE is defined. Martin, do you have any other suggestions? I don't see how to do this right within configure. If you want I can send you the header file(s). ---------------------------------------------------------------------- Comment By: alexandre gillet (gillet) Date: 2003-05-12 18:41 Message: Logged In: YES user_id=150999 I had the same problem compiling socket on Irix. We did find that on our system ENABLE_IPV6 is not define and all the error are cause by that. System that do not enable IPV6 are not well supported with the socketmodule.c code. We were able to compile socket after making few change in the code: I am not sure if I did it right but it compile. I will post my patch but I do not see any place to attach my file example: See line 2794,2800 + #ifdef ENABLE_IPV6 char packed[MAX(sizeof(struct in_addr), sizeof(struct in6_addr))]; + #else + char packed[sizeof(struct in_addr)]; + #endif I am not sure how to post the patch or file. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-05-05 19:44 Message: Logged In: YES user_id=71407 This is not in my area of expertise, but I've tried a couple of (probably stupid) things: #include in socketmodule.c instead of #define _SGIAPI. This resulted in even more errors. ./configure --disable-ipv6 Same errors as before. I am lost here. Sorry. But you still have ssh access to rattler.lbl.gov if that helps. Accounts for other Python developers are a possibility. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-04 13:39 Message: Logged In: YES user_id=21627 Can you analyse these compiler errors in more detail, please? What, in your opinion, is causing these problems, and how would you correct them? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 From noreply at sourceforge.net Wed May 26 14:45:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 14:45:50 2004 Subject: [ python-Bugs-960995 ] test_zlib is too slow Message-ID: Bugs item #960995, was opened at 2004-05-26 13:35 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960995&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: test_zlib is too slow Initial Comment: I don't know what it's doing, but I've never seen it fail and waiting for it has certainly wasted quite a lot of my life :-) ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-26 14:45 Message: Logged In: YES user_id=31435 I'm sure most of the cases in test_zlib are quite zippy (yes, pun intended). Do the right thing: determine which cases are the time hogs, and pare them down. By eyeball, only these subtests consume enough time to notice: test_manydecompimax test_manydecompimaxflush test_manydecompinc s/_many/_some/ isn't enough on its own . ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-26 14:17 Message: Logged In: YES user_id=80475 I hate this slow test. If you want to label this as explictly called resource, regrtest -u zlib , then be my guest. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960995&group_id=5470 From noreply at sourceforge.net Wed May 26 14:47:16 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 14:49:27 2004 Subject: [ python-Bugs-960874 ] codecs.lookup can raise exceptions other than LookupError Message-ID: Bugs item #960874, was opened at 2004-05-26 14:37 Message generated for change (Comment added) made by jpe You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 5 Submitted By: John Ehresman (jpe) Assigned to: M.-A. Lemburg (lemburg) Summary: codecs.lookup can raise exceptions other than LookupError Initial Comment: codecs.lookup raises ValueError when given an empty string and UnicodeEncodeError when given a unicode object that can't be converted to a str in the default encoding. I'd expect it to raise LookupError when passed any basestring instance. For example: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import codecs >>> codecs.lookup('') Traceback (most recent call last): File "", line 1, in ? File "c:\python23\lib\encodings\__init__.py", line 84, in search_function globals(), locals(), _import_tail) ValueError: Empty module name >>> codecs.lookup(u'\uabcd') Traceback (most recent call last): File "", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\uabcd' in position 0: ordinal not in range (128) >>> ---------------------------------------------------------------------- >Comment By: John Ehresman (jpe) Date: 2004-05-26 18:47 Message: Logged In: YES user_id=22785 Yes, it does look like lookup('') is fixed in CVS. So the question is whether lookup() of something that isn't convertable in the current encoding to a char* should raise a LookupError. I can live with it not, though if it did, it would make it a bit easier to determine if an arbitrary unicode string is a name of a supported encoding. I'm willing to put together a patch to raise LookupError if that's what the behavior should be ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 17:13 Message: Logged In: YES user_id=6656 This much seems to be fixed in CVS, actually :-) ---------------------------------------------------------------------- Comment By: John Ehresman (jpe) Date: 2004-05-26 17:09 Message: Logged In: YES user_id=22785 The other exceptions occur when strings or unicode objects are passed in as an argument. The string that it fails on is the empty string (''). I can see disallowing non-ascii names, but '' should raise a LookupError. My use case is to see if an user supplied unicode string is a valid encoding, so any check that the lookup function does not do, I will need to do before calling it. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 16:32 Message: Logged In: YES user_id=6656 What exactly are you complaining about? I'd expect codecs.lookup to raise TypeError if called with no arguments or an integer. I believe it's documented somewhere that encoding names must be ascii only, but I must admit I don't recall where. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 From noreply at sourceforge.net Wed May 26 14:53:45 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 14:53:56 2004 Subject: [ python-Bugs-960874 ] codecs.lookup can raise exceptions other than LookupError Message-ID: Bugs item #960874, was opened at 2004-05-26 15:37 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 5 Submitted By: John Ehresman (jpe) Assigned to: M.-A. Lemburg (lemburg) Summary: codecs.lookup can raise exceptions other than LookupError Initial Comment: codecs.lookup raises ValueError when given an empty string and UnicodeEncodeError when given a unicode object that can't be converted to a str in the default encoding. I'd expect it to raise LookupError when passed any basestring instance. For example: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import codecs >>> codecs.lookup('') Traceback (most recent call last): File "", line 1, in ? File "c:\python23\lib\encodings\__init__.py", line 84, in search_function globals(), locals(), _import_tail) ValueError: Empty module name >>> codecs.lookup(u'\uabcd') Traceback (most recent call last): File "", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\uabcd' in position 0: ordinal not in range (128) >>> ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-26 19:53 Message: Logged In: YES user_id=6656 Well, *I* don't think that's a particularly good idea. I don't know if Marc-Andr? feels differently. ---------------------------------------------------------------------- Comment By: John Ehresman (jpe) Date: 2004-05-26 19:47 Message: Logged In: YES user_id=22785 Yes, it does look like lookup('') is fixed in CVS. So the question is whether lookup() of something that isn't convertable in the current encoding to a char* should raise a LookupError. I can live with it not, though if it did, it would make it a bit easier to determine if an arbitrary unicode string is a name of a supported encoding. I'm willing to put together a patch to raise LookupError if that's what the behavior should be ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 18:13 Message: Logged In: YES user_id=6656 This much seems to be fixed in CVS, actually :-) ---------------------------------------------------------------------- Comment By: John Ehresman (jpe) Date: 2004-05-26 18:09 Message: Logged In: YES user_id=22785 The other exceptions occur when strings or unicode objects are passed in as an argument. The string that it fails on is the empty string (''). I can see disallowing non-ascii names, but '' should raise a LookupError. My use case is to see if an user supplied unicode string is a valid encoding, so any check that the lookup function does not do, I will need to do before calling it. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 17:32 Message: Logged In: YES user_id=6656 What exactly are you complaining about? I'd expect codecs.lookup to raise TypeError if called with no arguments or an integer. I believe it's documented somewhere that encoding names must be ascii only, but I must admit I don't recall where. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 From noreply at sourceforge.net Wed May 26 15:17:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 15:18:06 2004 Subject: [ python-Bugs-960874 ] codecs.lookup can raise exceptions other than LookupError Message-ID: Bugs item #960874, was opened at 2004-05-26 16:37 Message generated for change (Comment added) made by lemburg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 Category: Unicode Group: None >Status: Closed >Resolution: Wont Fix Priority: 5 Submitted By: John Ehresman (jpe) Assigned to: M.-A. Lemburg (lemburg) Summary: codecs.lookup can raise exceptions other than LookupError Initial Comment: codecs.lookup raises ValueError when given an empty string and UnicodeEncodeError when given a unicode object that can't be converted to a str in the default encoding. I'd expect it to raise LookupError when passed any basestring instance. For example: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import codecs >>> codecs.lookup('') Traceback (most recent call last): File "", line 1, in ? File "c:\python23\lib\encodings\__init__.py", line 84, in search_function globals(), locals(), _import_tail) ValueError: Empty module name >>> codecs.lookup(u'\uabcd') Traceback (most recent call last): File "", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\uabcd' in position 0: ordinal not in range (128) >>> ---------------------------------------------------------------------- >Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 21:17 Message: Logged In: YES user_id=38388 I don't think we should change anything. First of all, the lookup function interfaces to a codec search function and these can raise all kinds of errors, so it is not guaranteed that you will only see LookupErrors (the same is true for most other Python APIs, e.g. most can generate MemoryErrors). Possible other errors are ValueErrors, NameErrors, ImportErrors, etc. etc. depending on the search function that happens to process your request. Second, the name you enter as argument usually maps to a Python module and/or package name, so it *has* to be ASCII. The fact that you can enter Unicode names for the codec name if only by virtue of the automagical conversion of Unicode to strings. Again, this happens in a lot of places in Python and is not specific to lookup(). Closing this request. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 20:53 Message: Logged In: YES user_id=6656 Well, *I* don't think that's a particularly good idea. I don't know if Marc-Andr? feels differently. ---------------------------------------------------------------------- Comment By: John Ehresman (jpe) Date: 2004-05-26 20:47 Message: Logged In: YES user_id=22785 Yes, it does look like lookup('') is fixed in CVS. So the question is whether lookup() of something that isn't convertable in the current encoding to a char* should raise a LookupError. I can live with it not, though if it did, it would make it a bit easier to determine if an arbitrary unicode string is a name of a supported encoding. I'm willing to put together a patch to raise LookupError if that's what the behavior should be ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 19:13 Message: Logged In: YES user_id=6656 This much seems to be fixed in CVS, actually :-) ---------------------------------------------------------------------- Comment By: John Ehresman (jpe) Date: 2004-05-26 19:09 Message: Logged In: YES user_id=22785 The other exceptions occur when strings or unicode objects are passed in as an argument. The string that it fails on is the empty string (''). I can see disallowing non-ascii names, but '' should raise a LookupError. My use case is to see if an user supplied unicode string is a valid encoding, so any check that the lookup function does not do, I will need to do before calling it. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 18:32 Message: Logged In: YES user_id=6656 What exactly are you complaining about? I'd expect codecs.lookup to raise TypeError if called with no arguments or an integer. I believe it's documented somewhere that encoding names must be ascii only, but I must admit I don't recall where. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 From noreply at sourceforge.net Wed May 26 15:33:50 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 15:34:14 2004 Subject: [ python-Bugs-960874 ] codecs.lookup can raise exceptions other than LookupError Message-ID: Bugs item #960874, was opened at 2004-05-26 14:37 Message generated for change (Comment added) made by jpe You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 Category: Unicode Group: None Status: Closed Resolution: Wont Fix Priority: 5 Submitted By: John Ehresman (jpe) Assigned to: M.-A. Lemburg (lemburg) Summary: codecs.lookup can raise exceptions other than LookupError Initial Comment: codecs.lookup raises ValueError when given an empty string and UnicodeEncodeError when given a unicode object that can't be converted to a str in the default encoding. I'd expect it to raise LookupError when passed any basestring instance. For example: Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import codecs >>> codecs.lookup('') Traceback (most recent call last): File "", line 1, in ? File "c:\python23\lib\encodings\__init__.py", line 84, in search_function globals(), locals(), _import_tail) ValueError: Empty module name >>> codecs.lookup(u'\uabcd') Traceback (most recent call last): File "", line 1, in ? UnicodeEncodeError: 'ascii' codec can't encode character u'\uabcd' in position 0: ordinal not in range (128) >>> ---------------------------------------------------------------------- >Comment By: John Ehresman (jpe) Date: 2004-05-26 19:33 Message: Logged In: YES user_id=22785 Okay, that works for me. We might want to update the documentation, which seems to imply that LookupError will be raised if the name is invalid -- my mental model was that it acted more like a dictionary. I was just trying to avoid a catch all handler to catch expected failures (an encoding being unavailable is exepect because I know I may be feeding junk to it; but out of memory wouldn't be, though I know it can happen anywhere). Thanks for the quick response :). ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 19:17 Message: Logged In: YES user_id=38388 I don't think we should change anything. First of all, the lookup function interfaces to a codec search function and these can raise all kinds of errors, so it is not guaranteed that you will only see LookupErrors (the same is true for most other Python APIs, e.g. most can generate MemoryErrors). Possible other errors are ValueErrors, NameErrors, ImportErrors, etc. etc. depending on the search function that happens to process your request. Second, the name you enter as argument usually maps to a Python module and/or package name, so it *has* to be ASCII. The fact that you can enter Unicode names for the codec name if only by virtue of the automagical conversion of Unicode to strings. Again, this happens in a lot of places in Python and is not specific to lookup(). Closing this request. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 18:53 Message: Logged In: YES user_id=6656 Well, *I* don't think that's a particularly good idea. I don't know if Marc-Andr? feels differently. ---------------------------------------------------------------------- Comment By: John Ehresman (jpe) Date: 2004-05-26 18:47 Message: Logged In: YES user_id=22785 Yes, it does look like lookup('') is fixed in CVS. So the question is whether lookup() of something that isn't convertable in the current encoding to a char* should raise a LookupError. I can live with it not, though if it did, it would make it a bit easier to determine if an arbitrary unicode string is a name of a supported encoding. I'm willing to put together a patch to raise LookupError if that's what the behavior should be ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 17:13 Message: Logged In: YES user_id=6656 This much seems to be fixed in CVS, actually :-) ---------------------------------------------------------------------- Comment By: John Ehresman (jpe) Date: 2004-05-26 17:09 Message: Logged In: YES user_id=22785 The other exceptions occur when strings or unicode objects are passed in as an argument. The string that it fails on is the empty string (''). I can see disallowing non-ascii names, but '' should raise a LookupError. My use case is to see if an user supplied unicode string is a valid encoding, so any check that the lookup function does not do, I will need to do before calling it. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 16:32 Message: Logged In: YES user_id=6656 What exactly are you complaining about? I'd expect codecs.lookup to raise TypeError if called with no arguments or an integer. I believe it's documented somewhere that encoding names must be ascii only, but I must admit I don't recall where. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960874&group_id=5470 From noreply at sourceforge.net Wed May 26 15:41:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 15:41:47 2004 Subject: [ python-Feature Requests-920680 ] readline not implemented for UTF-16 Message-ID: Feature Requests item #920680, was opened at 2004-03-21 23:37 Message generated for change (Comment added) made by lemburg You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 3 Submitted By: Bob Ippolito (etrepum) Assigned to: M.-A. Lemburg (lemburg) Summary: readline not implemented for UTF-16 Initial Comment: The StreamReader for UTF-16 (all three of them) doesn't implement readline. ---------------------------------------------------------------------- >Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 21:41 Message: Logged In: YES user_id=38388 I don't have time to review this now, but will get back to it after EuroPython if you ping me. Thanks. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-20 01:10 Message: Logged In: YES user_id=764593 It might be just an upload/download quirk, but when I tried, this concatenated short lines. u"\n".join(...) worked better, but I'm not sure how that plays with other line breaks. It might work better to stick a class around the realine functions, so that self.buff can always be a (state-preserved) list; just return the first row, until the list length gets to one, then concatenate to that and resplit. ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-05-19 20:38 Message: Logged In: YES user_id=139309 Attaching a revised monkeypatch: * splitlines is used (I wasn't aware of the other unicode EOL markers) * 256 bytes is the new default buffer size Why do you want sized and unsized to be in the same function? They're both dispatched from readline as appropriate, and they are very different code paths. It would be much uglier as one function, so I'm not going to do it in my own code. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-19 10:19 Message: Logged In: YES user_id=38388 Thanks for the patch. Some comments: * Unicode has a lot more line-end markers than just LF; you should use .splitlines() to break lines at all of them * please collapse both methods (sized + unsized) into one method and default to 256 bytes for the buffer size ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-05-19 01:22 Message: Logged In: YES user_id=139309 I've attached a monkeypatch to get readline support for utf-16 codecs.. import utf16reader utf16reader.install() It can be trivially inserted into the utf16 encodings implementation.. it would be really cool if someone would audit the implementation and sneak it in before Python 2.4 :) ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-03-21 23:54 Message: Logged In: YES user_id=139309 I don't need it enough to write a patch, but this is what I used instead.. and it seems like it might work: try: for line in inFile: tline = translator(line) outFile.write(tline) except NotImplementedError: BUFFER = 16384 bytes = inFile.read(BUFFER) while bytes: lines = bytes.split(u'\n') bytes = lines.pop() for line in lines: tline = translator(line) outFile.write(tline) newbytes = inFile.read(BUFFER) bytes += newbytes if not newbytes and bytes: bytes += u'\n' ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-21 23:44 Message: Logged In: YES user_id=38388 Patches are welcome :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 From noreply at sourceforge.net Wed May 26 15:46:52 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 15:47:09 2004 Subject: [ python-Feature Requests-920680 ] readline not implemented for UTF-16 Message-ID: Feature Requests item #920680, was opened at 2004-03-21 17:37 Message generated for change (Comment added) made by etrepum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 3 Submitted By: Bob Ippolito (etrepum) Assigned to: M.-A. Lemburg (lemburg) Summary: readline not implemented for UTF-16 Initial Comment: The StreamReader for UTF-16 (all three of them) doesn't implement readline. ---------------------------------------------------------------------- >Comment By: Bob Ippolito (etrepum) Date: 2004-05-26 15:46 Message: Logged In: YES user_id=139309 Can you please give an example of a case where short lines get concatenated? I can't fix it if I don't know what's wrong. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 15:41 Message: Logged In: YES user_id=38388 I don't have time to review this now, but will get back to it after EuroPython if you ping me. Thanks. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-19 19:10 Message: Logged In: YES user_id=764593 It might be just an upload/download quirk, but when I tried, this concatenated short lines. u"\n".join(...) worked better, but I'm not sure how that plays with other line breaks. It might work better to stick a class around the realine functions, so that self.buff can always be a (state-preserved) list; just return the first row, until the list length gets to one, then concatenate to that and resplit. ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-05-19 14:38 Message: Logged In: YES user_id=139309 Attaching a revised monkeypatch: * splitlines is used (I wasn't aware of the other unicode EOL markers) * 256 bytes is the new default buffer size Why do you want sized and unsized to be in the same function? They're both dispatched from readline as appropriate, and they are very different code paths. It would be much uglier as one function, so I'm not going to do it in my own code. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-19 04:19 Message: Logged In: YES user_id=38388 Thanks for the patch. Some comments: * Unicode has a lot more line-end markers than just LF; you should use .splitlines() to break lines at all of them * please collapse both methods (sized + unsized) into one method and default to 256 bytes for the buffer size ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-05-18 19:22 Message: Logged In: YES user_id=139309 I've attached a monkeypatch to get readline support for utf-16 codecs.. import utf16reader utf16reader.install() It can be trivially inserted into the utf16 encodings implementation.. it would be really cool if someone would audit the implementation and sneak it in before Python 2.4 :) ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-03-21 17:54 Message: Logged In: YES user_id=139309 I don't need it enough to write a patch, but this is what I used instead.. and it seems like it might work: try: for line in inFile: tline = translator(line) outFile.write(tline) except NotImplementedError: BUFFER = 16384 bytes = inFile.read(BUFFER) while bytes: lines = bytes.split(u'\n') bytes = lines.pop() for line in lines: tline = translator(line) outFile.write(tline) newbytes = inFile.read(BUFFER) bytes += newbytes if not newbytes and bytes: bytes += u'\n' ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-21 17:44 Message: Logged In: YES user_id=38388 Patches are welcome :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 From noreply at sourceforge.net Wed May 26 15:52:26 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 15:52:35 2004 Subject: [ python-Bugs-960995 ] test_zlib is too slow Message-ID: Bugs item #960995, was opened at 2004-05-26 10:35 Message generated for change (Comment added) made by bcannon You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960995&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: test_zlib is too slow Initial Comment: I don't know what it's doing, but I've never seen it fail and waiting for it has certainly wasted quite a lot of my life :-) ---------------------------------------------------------------------- >Comment By: Brett Cannon (bcannon) Date: 2004-05-26 12:52 Message: Logged In: YES user_id=357491 A quick look at the tests Tim lists shows that each of those run the basic incremental decompression test 8 times, from the normal size to 2**8 time the base size; creates a list from [1<. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-26 11:17 Message: Logged In: YES user_id=80475 I hate this slow test. If you want to label this as explictly called resource, regrtest -u zlib , then be my guest. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960995&group_id=5470 From noreply at sourceforge.net Wed May 26 15:52:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Wed May 26 15:52:44 2004 Subject: [ python-Feature Requests-920680 ] readline not implemented for UTF-16 Message-ID: Feature Requests item #920680, was opened at 2004-03-21 17:37 Message generated for change (Comment added) made by etrepum You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 Category: Unicode Group: None Status: Open Resolution: None Priority: 3 Submitted By: Bob Ippolito (etrepum) Assigned to: M.-A. Lemburg (lemburg) Summary: readline not implemented for UTF-16 Initial Comment: The StreamReader for UTF-16 (all three of them) doesn't implement readline. ---------------------------------------------------------------------- >Comment By: Bob Ippolito (etrepum) Date: 2004-05-26 15:52 Message: Logged In: YES user_id=139309 Also, I've moved the latest copy of the code to my public repository at: http://svn.red-bean.com/bob/unicode/trunk/utf16reader.py this should be free of any quirks, but I still can't reproduce whatever problem jim is having. ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-05-26 15:46 Message: Logged In: YES user_id=139309 Can you please give an example of a case where short lines get concatenated? I can't fix it if I don't know what's wrong. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-26 15:41 Message: Logged In: YES user_id=38388 I don't have time to review this now, but will get back to it after EuroPython if you ping me. Thanks. ---------------------------------------------------------------------- Comment By: Jim Jewett (jimjjewett) Date: 2004-05-19 19:10 Message: Logged In: YES user_id=764593 It might be just an upload/download quirk, but when I tried, this concatenated short lines. u"\n".join(...) worked better, but I'm not sure how that plays with other line breaks. It might work better to stick a class around the realine functions, so that self.buff can always be a (state-preserved) list; just return the first row, until the list length gets to one, then concatenate to that and resplit. ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-05-19 14:38 Message: Logged In: YES user_id=139309 Attaching a revised monkeypatch: * splitlines is used (I wasn't aware of the other unicode EOL markers) * 256 bytes is the new default buffer size Why do you want sized and unsized to be in the same function? They're both dispatched from readline as appropriate, and they are very different code paths. It would be much uglier as one function, so I'm not going to do it in my own code. ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-05-19 04:19 Message: Logged In: YES user_id=38388 Thanks for the patch. Some comments: * Unicode has a lot more line-end markers than just LF; you should use .splitlines() to break lines at all of them * please collapse both methods (sized + unsized) into one method and default to 256 bytes for the buffer size ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-05-18 19:22 Message: Logged In: YES user_id=139309 I've attached a monkeypatch to get readline support for utf-16 codecs.. import utf16reader utf16reader.install() It can be trivially inserted into the utf16 encodings implementation.. it would be really cool if someone would audit the implementation and sneak it in before Python 2.4 :) ---------------------------------------------------------------------- Comment By: Bob Ippolito (etrepum) Date: 2004-03-21 17:54 Message: Logged In: YES user_id=139309 I don't need it enough to write a patch, but this is what I used instead.. and it seems like it might work: try: for line in inFile: tline = translator(line) outFile.write(tline) except NotImplementedError: BUFFER = 16384 bytes = inFile.read(BUFFER) while bytes: lines = bytes.split(u'\n') bytes = lines.pop() for line in lines: tline = translator(line) outFile.write(tline) newbytes = inFile.read(BUFFER) bytes += newbytes if not newbytes and bytes: bytes += u'\n' ---------------------------------------------------------------------- Comment By: M.-A. Lemburg (lemburg) Date: 2004-03-21 17:44 Message: Logged In: YES user_id=38388 Patches are welcome :-) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=920680&group_id=5470 From noreply at sourceforge.net Thu May 27 00:14:36 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 27 00:14:57 2004 Subject: [ python-Bugs-728330 ] Don't define _SGAPI on IRIX Message-ID: Bugs item #728330, was opened at 2003-04-27 02:21 Message generated for change (Comment added) made by rwgk You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 Category: Installation Group: Python 2.3 Status: Open Resolution: Remind Priority: 9 Submitted By: Ralf W. Grosse-Kunstleve (rwgk) Assigned to: Neal Norwitz (nnorwitz) Summary: Don't define _SGAPI on IRIX Initial Comment: Python release: 2.3b1 Platform: IRIX 6.5 MIPSpro Compilers: Version 7.3.1.2m Commands used: ./configure --prefix=/usr/local_cci/Python-2.3b1t make The socket module fails to compile. The output from the compiler is attached. ---------------------------------------------------------------------- >Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 21:14 Message: Logged In: YES user_id=71407 > Can you try gmake? Yes, it works with gmake. I've verified that the socket module on the CVS head builds and runs correctly on the three platforms mentioned in my message from 2004-05- 22 01:23. There are not even warnings. > Is IRIX make particularly braindead? I don't think so. It works with most auto-configured packages incl. all earlier versions of Python. But I'll save this for another day when the first 2.4 beta release is out. Thank you for your help! I think this tracker issue should be closed. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 11:19 Message: Logged In: YES user_id=6656 Uh, that's odd. Is IRIX make particularly braindead? Can you try gmake? Maybe we've let something non-portable slip in. At any rate, I think this should be done in a separate report... ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 11:14 Message: Logged In: YES user_id=71407 I checked out the python cvs head from scratch and ran configure in the usual way, with the usual result. However, make doesn't work: % make don't know how to make Parser/printgrammar.o (bu42). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 10:07 Message: Logged In: YES user_id=6656 OK, so I've commited revision 1.289 of Modules/socketmodule.c. Can you check if CVS HEAD works for you now? ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 09:59 Message: Logged In: YES user_id=71407 > Is there any chance of the Right Thing actually happening? The experience of the last 13 months suggests "No." SGI's freeware team has not responded to my report for more than a year. I am absolutely not a SGI or socket expert. I only worked on the patch because I was desperate enough and because Maik had already done the groundwork. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 09:27 Message: Logged In: YES user_id=6656 Is this really The Right Thing? I still see /* XXX Using _SGIAPI is the wrong thing, but I don't know what the right thing is. */ Is there any chance of the Right Thing actually happening? If not, I'll check this into HEAD, and if that seems to work, move it to release23-maint in time for 2.3.5. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-22 01:23 Message: Logged In: YES user_id=71407 Based on Maik's posting from 2004-03-08 08:53 I've worked out a small bug-fix patch relative to Python-2.3.4c1 that leads to successful creation of _socket.so under three different versions of IRIX: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.23 with MIPRpro 7.3.1.3 I've consistently used #if defined(__sgi) && _COMPILER_VERSION>700 to restrict the changes to IRIX. However, to be sure I've also verified that the Python installation is still OK under Redhat 8 and Tru64 Unix 5.1. I will upload the patch. It is also available here: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patch For completeness I've also posted these files: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patched_c http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_original_c I've made an effort to keep the patch minimally intrusive, but it is certainly not "clean." However, the status quo is so bad that it can only get better. We have not been able to switch to Python 2.3 because of the problems with the socket module. It would be fantastic if the bug-fix patch could be included in the 2.3.4 final distribution. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2004-03-08 08:53 Message: Logged In: YES user_id=768223 I got Python 2.3.3 compiled with MipsPro Compilers: Version 7.3.1.3m. The following changes were made to satisfy all needs. Line 206 in Modules/socketmodule.c #define _SGIAPI 1 NEW: #undef _XOPEN_SOURCE NEW: #define HAVE_GETADDRINFO 1 NEW: #define HAVE_GETNAMEINFO 1 #define HAVE_INET_PTON NEW: #include NEW: #include NEW: #include #include Line 269: /* #include "addrinfo.h" */ In addition to this I extend the CFLAGS with -D_BSD_TYPES to satisfy the u_short and friends definition. --maik./ ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-03-03 08:20 Message: Logged In: YES user_id=892601 Got it down to one error: % > ./python Lib/test/test_socket.py Traceback (most recent call last): File "Lib/test/test_socket.py", line 6, in ? import socket File "/usr/local/spider/lib/python2.3/socket.py", line 44, in ? import _socket ImportError: 210454:./python: rld: Fatal Error: unresolvable symbol in /usr/local/spider/lib/python2.3/lib-dynload/_socketmodule.so: _xpg5_accept xpg5_accept does appear in any library on our SGI's that are only 1-2 years old. It does appear in libc in very recent revisions of Irix (> 6.5.19?) Does anyone know how to get Python 2.3 to avoid the use of xpg5_accept? Thanks ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-01-12 10:59 Message: Logged In: YES user_id=892601 Has anyone encountered problems with different versions of IRIX 6.5? I finally got sockets to work in Python2.3.3, but only on the most up-to-date version of IRIX (uname -v = 01100304). On any other machine with a lower version (archetecture doesn't seem to matter), I get messages which all seem to be related to sockets: 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_vsnprintf 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recv 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_connect 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_bind 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_sendto 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_setsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_accept 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recvfrom 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_socket 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockname Upgrading all of our SGIs is not something I have control over. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-22 10:52 Message: Logged In: YES user_id=768223 Sorry if there is something unclear. Redefine means: Set a value which is required to satisfy ifdef-statements in the header files. As you see in the definitions, given in the comment, the value for _NO_XOPEN4 is set as a result of some dependencies automatically defined. A redefinition of _SGIAPI in the socketmodule.c is not sufficient alone. Therefore to satisfy all requirements in the header files, you should set the values for _NO_XOPEN4 and _NO_XOPEN5 also manually. Howto do in socketmodule.c? ... #define _SGIAPI 1 #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 ... If you are familiar with regular expressions, then you will know that _NO_XOPEN[45] is a shorthand for _NO_XOPEN4, _NO_XOPEN5. Sorry if there was something wrong in my comment. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-12-22 07:49 Message: Logged In: YES user_id=892601 Can someone please clarify mherta's comment from 2003-12-16? How should we "redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition"? Should those lines of code be added to socketmodule.c, or are they the source of the problem? Is _NO_XOPEN[45] shorthand for _NO_XOPEN[4] and _NO_XOPEN[5] or are they all different? Thanks. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-16 09:27 Message: Logged In: YES user_id=768223 I've tested the latest Python 2.3.3c1 to compile. The problem reported is still present. To avoid the error messages, about the incompatible types of several netdb-functions (like inet_pton), you should redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition. If there are error messages that types like u_short and u_char are not defined, please add -D_BSD_TYPES in the Makefile (BASECFLAGS is okay). If anybody don't need a executable compiled with MipsPro, gcc from the freeware.sgi.com could be a choice. The problem for this seems in /usr/include/standards.h: #define _NO_XOPEN4 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 >= 500))) #define _NO_XOPEN5 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 < 500))) As the first option is true (_XOPEN_SOURCE is set to 600 in pyconfig.h) the second option will be false. The constant #define NI_NUMERICHOST 0x00000002 /* return numeric form of hostname */ is enveloped in a clause #ifdef _NO_XOPEN4 && _NO_XOPEN5 (/usr/include/netdb.h) which is always false. In the manual pages there is a example which uses the constant NI_NUMERICHOST and compiles without error message. I'm not sure to cut off the file pyconfig.h from compiling the socketmodule.c? Any comments? ---------------------------------------------------------------------- Comment By: Hans Nowak (zephyrfalcon) Date: 2003-10-30 19:04 Message: Logged In: YES user_id=173607 My $0.02... Yesterday I successfully compiled Python 2.3.2, with socket module. I'm using an O2 running IRIX 6.5.11f. I didn't encounter any of the problems described before, and had no problems with _SGIAPI or anything. The reason it didn't compile was that u_long and u_char were not defined in some places; /usr/include/netinet/tcp.h to be precise, and a Python file (getaddrinfo.c). I'm using gcc 3.3 (the one downloaded from freeware.sgi.com). If anyone is interested, drop me a mail at hans at zephyrfalcon.org and I'll put the binary online somewhere. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-24 12:59 Message: Logged In: YES user_id=892601 Here's the output from 'gmake', using MIPSpro Compiler 7.30 on IRIX 6.5 Sorry, I don't know how to attach something using the "Add a Comment" box. Is there another way to post? The patch at www.python.org/2.3.2/patches/sgi-socketmodule.patch was edited into socketmodule.c : #include #endif /* GCC 3.2 on Irix 6.5 fails to define this variable at all. Based on the above code, I'd say that the SGI headers are just horked. */ #if defined(__sgi) && !defined(INET_ADDRSTRLEN) #define INET_ADDRSTRLEN 16 #endif /* Generic includes */ #include #include --------------------------------------------------------------------------- cc -OPT:Olimit=0 -DNDEBUG -n32 -O -I. -I./Include -DPy_BUILD_CORE -c ./Modules/socketmodule.c -o Modules/socketmodule.o cc-1047 cc: WARNING File = ./Modules/socketmodule.c, Line = 195 Macro "_SGIAPI" (declared at line 210 of "/usr/include/standards.h") has an incompatible redefinition. #define _SGIAPI 1 ^ cc-1020 cc: ERROR File = /usr/include/netdb.h, Line = 198 The identifier "socklen_t" is undefined. extern int iruserok_sa(const void *, socklen_t, int, const char *, ^ cc-1047 cc: WARNING File = ./Modules/addrinfo.h, Line = 145 Macro "_SS_ALIGNSIZE" (declared at line 246 of "/usr/include/sys/socket.h") has an incompatible redefinition. #define _SS_ALIGNSIZE (sizeof(PY_LONG_LONG)) ^ cc-1020 cc: ERROR File = ./Modules/socketmodule.c, Line = 822 The identifier "NI_NUMERICHOST" is undefined. NI_NUMERICHOST); ^ cc-1515 cc: ERROR File = ./Modules/socketmodule.c, Line = 3054 A value of type "int" cannot be assigned to an entity of type "const char *". retval = inet_ntop(af, packed, ip, sizeof(ip)); ^ 3 errors detected in the compilation of "./Modules/socketmodule.c". gmake: *** [Modules/socketmodule.o] Error 2 ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2003-10-23 03:25 Message: Logged In: YES user_id=6656 Just to be clear, please "attach" the build rather than pasting it inline. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2003-10-22 19:19 Message: Logged In: YES user_id=29957 Did you try 2.3.2 with the patch from the 2.3.2 build bugs page? If it failed, please post the build log of the failing bit to this bug. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-22 11:26 Message: Logged In: YES user_id=892601 Has there been any progress since August on getting Python-2.3 to compile on IRIX 6.5? I really need to have sockets working to use python. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-08-14 04:37 Message: Logged In: YES user_id=768223 I have the same problem on IRIX 6.5.19m and MIPSpro 7.3.1.3m on a FUEL 600Mhz. The situation seems a little bit strange on this system. But I have luck and get it compiled. As the current implementation is not clean, my resolution also. Which should be changed to get it work. The manpage for gethostbyaddr_r says: #include #include #include #include So I included this in the __sgi-clause in socketmodule.c. Looking in the header-files shows, that some variables are defined if #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 are set, before the includes. This defines some more variables, so that pyconfig.h should be changed. #define HAVE_ADDRINFO 1 #define HAVE_SOCKADDR_STORAGE 1 Sure there are warnings about the redefinition of the _(SGIAPI|NO_XOPEN4|NO_XOPEN5) but it compiles and the tests don't fail. Also my applications run without exception or coredump on FUEL, Octane, Octane 2. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-08-01 05:10 Message: Logged In: YES user_id=33168 Ralf, I will try to work on this, but it won't be for a week or two probably. Can you tell me which machines (by name) work and doesn't work. The only IRIX machine I seem to know about it rattler. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-08-01 00:34 Message: Logged In: YES user_id=71407 For the records: The Python2.3 (final) socket module still does not compile under IRIX 6.5.20. It only compiles under 6.5.10. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 11:07 Message: Logged In: YES user_id=31392 I applied a slight variant of Neal's patch and that seemed to fix things. It still seems wrong that we define _SGIAPI, but that will have to wait for 2.3.1. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 06:54 Message: Logged In: YES user_id=31392 I learned a little about inet_ntop() in the last few minutes. It looks like the size_t argument is an older spelling, and the current POSIX says it should be socklen_t. Regardless, it looks like we're using the new definition when IRIX is actually providing the older definition. Neal attached a patch that looks like it should clean up the problem. Have you tried it? I think a better solution would be to teach configure how to find inet_ntop() via netdb.h, but that's probably too hard to get done in the new couple of hours. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 06:42 Message: Logged In: YES user_id=31392 There seem to be two causes of worry. The actual compiler error in your build log and the use of _SGIAPI that Casavanti warns again. It looks like the compiler problems is because our inet_ntop() prototype is wrong. That is Python says the last argument is a socklen_t, but IRIX and Linux man pages both say size_t. Perhaps size_t and socklen_t are different sizes on IRIX, but not on Linux. What happens if you change it to size_t? The problematic code would appear to be these lines from socketmodule.c: #if defined(__sgi)&&_COMPILER_VERSION>700 && !_SGIAPI /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #define _SGIAPI 1 #include "netdb.h" #endif It's the only use the _SGIAPI symbol that Casavant says we shouldn't use. What happens if you remove the use _SGIAPI, i.e. just make it #if defined(__sgi) && _COMPILER_VERSION > 700 /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #include "netdb.h" #endif ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-16 17:27 Message: Logged In: YES user_id=71407 We have confirmed that the socket module in Python 2.3b1 and 2.3b2 fails to compile under these versions of IRIX: 6.5.10, MIPSpro 7.3.1.2m 6.5.19, MIPSpro 7.3.1.3m 6.5.20, MIPSpro 7.3.1.3m (latest available, to my knowledge) If Python 2.3 goes out without a working socket module it will be useless for many people in science where IRIX is still found quite frequently. Maybe even worse, we will also no longer be able to advertise Python as a language that "runs everywhere." ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-15 15:37 Message: Logged In: YES user_id=71407 Date: Tue, 15 Jul 2003 15:06:25 -0500 From: "Brent Casavant" To: "Ralf W. Grosse-Kunstleve" Subject: Re: IRIX & Python Ralf, I read through the sourceforge report. I don't have a sourceforge account so I didn't add there, but feel free to distribute my ramblings. First, they should be very careful about what version of the operating system they are using. Real IPv6 support was only added a few releases ago (6.5.20 I believe, though my memory is hazy). Some of the functions/macros corresponding to IPv6 were introduced prior to that point, but implementation may not have been complete. If they have to play tricks with #define values to get a symbol recongnized it most likely means that SGI didn't intend for that symbol to be used yet. I suspect the problems they are seeing are because they are using some version of IRIX 6.5.x that includes inet_ntoa() definitions under _SGIAPI, but for which real support was not yet complete. The namespace of symbols that begin with a single underscore and a capital letter is reserved for the OS vendors, and the fact that someone is trying to manually insert a definition of _SGIAPI into the source should be a big red warning flag. That said, there is a correct method to deal with the new capabilities of IRIX 6.5.x over time. This comes in two parts. A better explanation is in /usr/include/optional_sym.h First, when a new symbol is added to IRIX, the corresponding header file will always have the following line in it: #pragma optional some_new_symbol The presence of this #pragma will cause the linker to not complain if the symbol is missing when the executable is linked. If SGI neglects to do so it is a bug that should be reported to us. The second part is that calls to these "optional" symbols should be protected by a run-time check to see if the symbol is actually present. Just like this: #include #include . . . if (_MIPS_SYMBOL_PRESENT(some_new_symbol)) { /* New function is present, use it */ qux = some_new_symbol(foo, bar, baz); } else { /* New function isn't present, do something else */ qux = builtin_some_new_symbol(foo, bar, baz); } This ensures that a single executable will be able to function properly (or at least degrade gracefully) on all version of IRIX 6.5.x. In other words you can compile on 6.5.20 and run on 6.5.8, or vice versa. In particular, this means that compile-time detection of features such as inet_ntoa() is incorrect for IRIX 6.5.x -- these decisions should be made at run-time instead. If the decision is made at compile time the executable may well not execute on an older version of IRIX 6.5.x. Unfortunately GNU configure is not equipped to utilize such advanced binary-compatability mechanisms, so there's no "easy" way to get GNU configure'd software to run across all IRIX 6.5.x releases, short of compiling only on the original non-upgraded version of IRIX 6.5. Hope that helps, Brent Casavant On Tue, 15 Jul 2003, Ralf W. Grosse-Kunstleve wrote: > We are developing a Python-based (www.python.org) application for > IRIX. Until now the Python IRIX port was fully supported, but the last > two beta releases of Python (2.3b1 and 2.3b2) make me increasingly > worried because the crucial socket module (TCP/IP sockets) fails to > build. A while ago I've filed a bug report, but the Python developers > appear to be stuck: > > https://sourceforge.net/tracker/? func=detail&aid=728330&group_id=5470&atid=105470 > > Is there a way to bring this to the attention of developers at SGI to > hopefully get some help? > > Thank you in advance! > Ralf > -- Brent Casavant Silicon Graphics, Inc. Where am I? And what am I IRIX O.S. Engineer http://www.sgi.com/ doing in this handbasket? bcasavan@sgi.com 44.8562N 93.1355W 860F -- Unknown ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-06-30 07:21 Message: Logged In: YES user_id=71407 The socket module still fails to compile with Python 2.3b2, in addition to two other extensions. The full make log is attached. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-21 23:47 Message: Logged In: YES user_id=21627 I think it would be best if detect presence of inet_aton/inet_pton. Is IPv6 support possible on this system? Then it would be best if that was detected, as well. If it is not possible to detect aton/pton/ipv6, what is the problem with not using them? I'd like to have a look at the relevant headers (in particular to understand the duplicate definition of _SGIAPI), indeed. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-05-21 16:35 Message: Logged In: YES user_id=33168 Argh!!! This is a nightmare. The attached hack ^h^h^h patch fixes the problem, but ... _SGIAPI needs to be defined (~192 in socketmodule.c) in order to get the prototypes for inet_aton, inet_pton. But _SGIAPI can't be defined in the standard header files because XOPEN_SOURCE is defined. Martin, do you have any other suggestions? I don't see how to do this right within configure. If you want I can send you the header file(s). ---------------------------------------------------------------------- Comment By: alexandre gillet (gillet) Date: 2003-05-12 10:41 Message: Logged In: YES user_id=150999 I had the same problem compiling socket on Irix. We did find that on our system ENABLE_IPV6 is not define and all the error are cause by that. System that do not enable IPV6 are not well supported with the socketmodule.c code. We were able to compile socket after making few change in the code: I am not sure if I did it right but it compile. I will post my patch but I do not see any place to attach my file example: See line 2794,2800 + #ifdef ENABLE_IPV6 char packed[MAX(sizeof(struct in_addr), sizeof(struct in6_addr))]; + #else + char packed[sizeof(struct in_addr)]; + #endif I am not sure how to post the patch or file. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-05-05 11:44 Message: Logged In: YES user_id=71407 This is not in my area of expertise, but I've tried a couple of (probably stupid) things: #include in socketmodule.c instead of #define _SGIAPI. This resulted in even more errors. ./configure --disable-ipv6 Same errors as before. I am lost here. Sorry. But you still have ssh access to rattler.lbl.gov if that helps. Accounts for other Python developers are a possibility. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-04 05:39 Message: Logged In: YES user_id=21627 Can you analyse these compiler errors in more detail, please? What, in your opinion, is causing these problems, and how would you correct them? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 From noreply at sourceforge.net Thu May 27 01:41:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 27 01:43:02 2004 Subject: [ python-Bugs-728330 ] Don't define _SGAPI on IRIX Message-ID: Bugs item #728330, was opened at 2003-04-27 19:21 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 Category: Installation Group: Python 2.3 Status: Open Resolution: Remind Priority: 9 Submitted By: Ralf W. Grosse-Kunstleve (rwgk) Assigned to: Neal Norwitz (nnorwitz) Summary: Don't define _SGAPI on IRIX Initial Comment: Python release: 2.3b1 Platform: IRIX 6.5 MIPSpro Compilers: Version 7.3.1.2m Commands used: ./configure --prefix=/usr/local_cci/Python-2.3b1t make The socket module fails to compile. The output from the compiler is attached. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-27 15:41 Message: Logged In: YES user_id=29957 If the patch is good, I'll make sure it's included in the 2.3.4 'bugs' page. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-27 14:14 Message: Logged In: YES user_id=71407 > Can you try gmake? Yes, it works with gmake. I've verified that the socket module on the CVS head builds and runs correctly on the three platforms mentioned in my message from 2004-05- 22 01:23. There are not even warnings. > Is IRIX make particularly braindead? I don't think so. It works with most auto-configured packages incl. all earlier versions of Python. But I'll save this for another day when the first 2.4 beta release is out. Thank you for your help! I think this tracker issue should be closed. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-27 04:19 Message: Logged In: YES user_id=6656 Uh, that's odd. Is IRIX make particularly braindead? Can you try gmake? Maybe we've let something non-portable slip in. At any rate, I think this should be done in a separate report... ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-27 04:14 Message: Logged In: YES user_id=71407 I checked out the python cvs head from scratch and ran configure in the usual way, with the usual result. However, make doesn't work: % make don't know how to make Parser/printgrammar.o (bu42). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-27 03:07 Message: Logged In: YES user_id=6656 OK, so I've commited revision 1.289 of Modules/socketmodule.c. Can you check if CVS HEAD works for you now? ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-27 02:59 Message: Logged In: YES user_id=71407 > Is there any chance of the Right Thing actually happening? The experience of the last 13 months suggests "No." SGI's freeware team has not responded to my report for more than a year. I am absolutely not a SGI or socket expert. I only worked on the patch because I was desperate enough and because Maik had already done the groundwork. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-27 02:27 Message: Logged In: YES user_id=6656 Is this really The Right Thing? I still see /* XXX Using _SGIAPI is the wrong thing, but I don't know what the right thing is. */ Is there any chance of the Right Thing actually happening? If not, I'll check this into HEAD, and if that seems to work, move it to release23-maint in time for 2.3.5. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-22 18:23 Message: Logged In: YES user_id=71407 Based on Maik's posting from 2004-03-08 08:53 I've worked out a small bug-fix patch relative to Python-2.3.4c1 that leads to successful creation of _socket.so under three different versions of IRIX: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.23 with MIPRpro 7.3.1.3 I've consistently used #if defined(__sgi) && _COMPILER_VERSION>700 to restrict the changes to IRIX. However, to be sure I've also verified that the Python installation is still OK under Redhat 8 and Tru64 Unix 5.1. I will upload the patch. It is also available here: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patch For completeness I've also posted these files: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patched_c http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_original_c I've made an effort to keep the patch minimally intrusive, but it is certainly not "clean." However, the status quo is so bad that it can only get better. We have not been able to switch to Python 2.3 because of the problems with the socket module. It would be fantastic if the bug-fix patch could be included in the 2.3.4 final distribution. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2004-03-09 03:53 Message: Logged In: YES user_id=768223 I got Python 2.3.3 compiled with MipsPro Compilers: Version 7.3.1.3m. The following changes were made to satisfy all needs. Line 206 in Modules/socketmodule.c #define _SGIAPI 1 NEW: #undef _XOPEN_SOURCE NEW: #define HAVE_GETADDRINFO 1 NEW: #define HAVE_GETNAMEINFO 1 #define HAVE_INET_PTON NEW: #include NEW: #include NEW: #include #include Line 269: /* #include "addrinfo.h" */ In addition to this I extend the CFLAGS with -D_BSD_TYPES to satisfy the u_short and friends definition. --maik./ ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-03-04 03:20 Message: Logged In: YES user_id=892601 Got it down to one error: % > ./python Lib/test/test_socket.py Traceback (most recent call last): File "Lib/test/test_socket.py", line 6, in ? import socket File "/usr/local/spider/lib/python2.3/socket.py", line 44, in ? import _socket ImportError: 210454:./python: rld: Fatal Error: unresolvable symbol in /usr/local/spider/lib/python2.3/lib-dynload/_socketmodule.so: _xpg5_accept xpg5_accept does appear in any library on our SGI's that are only 1-2 years old. It does appear in libc in very recent revisions of Irix (> 6.5.19?) Does anyone know how to get Python 2.3 to avoid the use of xpg5_accept? Thanks ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-01-13 05:59 Message: Logged In: YES user_id=892601 Has anyone encountered problems with different versions of IRIX 6.5? I finally got sockets to work in Python2.3.3, but only on the most up-to-date version of IRIX (uname -v = 01100304). On any other machine with a lower version (archetecture doesn't seem to matter), I get messages which all seem to be related to sockets: 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_vsnprintf 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recv 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_connect 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_bind 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_sendto 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_setsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_accept 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recvfrom 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_socket 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockname Upgrading all of our SGIs is not something I have control over. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-23 05:52 Message: Logged In: YES user_id=768223 Sorry if there is something unclear. Redefine means: Set a value which is required to satisfy ifdef-statements in the header files. As you see in the definitions, given in the comment, the value for _NO_XOPEN4 is set as a result of some dependencies automatically defined. A redefinition of _SGIAPI in the socketmodule.c is not sufficient alone. Therefore to satisfy all requirements in the header files, you should set the values for _NO_XOPEN4 and _NO_XOPEN5 also manually. Howto do in socketmodule.c? ... #define _SGIAPI 1 #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 ... If you are familiar with regular expressions, then you will know that _NO_XOPEN[45] is a shorthand for _NO_XOPEN4, _NO_XOPEN5. Sorry if there was something wrong in my comment. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-12-23 02:49 Message: Logged In: YES user_id=892601 Can someone please clarify mherta's comment from 2003-12-16? How should we "redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition"? Should those lines of code be added to socketmodule.c, or are they the source of the problem? Is _NO_XOPEN[45] shorthand for _NO_XOPEN[4] and _NO_XOPEN[5] or are they all different? Thanks. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-17 04:27 Message: Logged In: YES user_id=768223 I've tested the latest Python 2.3.3c1 to compile. The problem reported is still present. To avoid the error messages, about the incompatible types of several netdb-functions (like inet_pton), you should redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition. If there are error messages that types like u_short and u_char are not defined, please add -D_BSD_TYPES in the Makefile (BASECFLAGS is okay). If anybody don't need a executable compiled with MipsPro, gcc from the freeware.sgi.com could be a choice. The problem for this seems in /usr/include/standards.h: #define _NO_XOPEN4 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 >= 500))) #define _NO_XOPEN5 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 < 500))) As the first option is true (_XOPEN_SOURCE is set to 600 in pyconfig.h) the second option will be false. The constant #define NI_NUMERICHOST 0x00000002 /* return numeric form of hostname */ is enveloped in a clause #ifdef _NO_XOPEN4 && _NO_XOPEN5 (/usr/include/netdb.h) which is always false. In the manual pages there is a example which uses the constant NI_NUMERICHOST and compiles without error message. I'm not sure to cut off the file pyconfig.h from compiling the socketmodule.c? Any comments? ---------------------------------------------------------------------- Comment By: Hans Nowak (zephyrfalcon) Date: 2003-10-31 14:04 Message: Logged In: YES user_id=173607 My $0.02... Yesterday I successfully compiled Python 2.3.2, with socket module. I'm using an O2 running IRIX 6.5.11f. I didn't encounter any of the problems described before, and had no problems with _SGIAPI or anything. The reason it didn't compile was that u_long and u_char were not defined in some places; /usr/include/netinet/tcp.h to be precise, and a Python file (getaddrinfo.c). I'm using gcc 3.3 (the one downloaded from freeware.sgi.com). If anyone is interested, drop me a mail at hans at zephyrfalcon.org and I'll put the binary online somewhere. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-25 05:59 Message: Logged In: YES user_id=892601 Here's the output from 'gmake', using MIPSpro Compiler 7.30 on IRIX 6.5 Sorry, I don't know how to attach something using the "Add a Comment" box. Is there another way to post? The patch at www.python.org/2.3.2/patches/sgi-socketmodule.patch was edited into socketmodule.c : #include #endif /* GCC 3.2 on Irix 6.5 fails to define this variable at all. Based on the above code, I'd say that the SGI headers are just horked. */ #if defined(__sgi) && !defined(INET_ADDRSTRLEN) #define INET_ADDRSTRLEN 16 #endif /* Generic includes */ #include #include --------------------------------------------------------------------------- cc -OPT:Olimit=0 -DNDEBUG -n32 -O -I. -I./Include -DPy_BUILD_CORE -c ./Modules/socketmodule.c -o Modules/socketmodule.o cc-1047 cc: WARNING File = ./Modules/socketmodule.c, Line = 195 Macro "_SGIAPI" (declared at line 210 of "/usr/include/standards.h") has an incompatible redefinition. #define _SGIAPI 1 ^ cc-1020 cc: ERROR File = /usr/include/netdb.h, Line = 198 The identifier "socklen_t" is undefined. extern int iruserok_sa(const void *, socklen_t, int, const char *, ^ cc-1047 cc: WARNING File = ./Modules/addrinfo.h, Line = 145 Macro "_SS_ALIGNSIZE" (declared at line 246 of "/usr/include/sys/socket.h") has an incompatible redefinition. #define _SS_ALIGNSIZE (sizeof(PY_LONG_LONG)) ^ cc-1020 cc: ERROR File = ./Modules/socketmodule.c, Line = 822 The identifier "NI_NUMERICHOST" is undefined. NI_NUMERICHOST); ^ cc-1515 cc: ERROR File = ./Modules/socketmodule.c, Line = 3054 A value of type "int" cannot be assigned to an entity of type "const char *". retval = inet_ntop(af, packed, ip, sizeof(ip)); ^ 3 errors detected in the compilation of "./Modules/socketmodule.c". gmake: *** [Modules/socketmodule.o] Error 2 ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2003-10-23 20:25 Message: Logged In: YES user_id=6656 Just to be clear, please "attach" the build rather than pasting it inline. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2003-10-23 12:19 Message: Logged In: YES user_id=29957 Did you try 2.3.2 with the patch from the 2.3.2 build bugs page? If it failed, please post the build log of the failing bit to this bug. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-23 04:26 Message: Logged In: YES user_id=892601 Has there been any progress since August on getting Python-2.3 to compile on IRIX 6.5? I really need to have sockets working to use python. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-08-14 21:37 Message: Logged In: YES user_id=768223 I have the same problem on IRIX 6.5.19m and MIPSpro 7.3.1.3m on a FUEL 600Mhz. The situation seems a little bit strange on this system. But I have luck and get it compiled. As the current implementation is not clean, my resolution also. Which should be changed to get it work. The manpage for gethostbyaddr_r says: #include #include #include #include So I included this in the __sgi-clause in socketmodule.c. Looking in the header-files shows, that some variables are defined if #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 are set, before the includes. This defines some more variables, so that pyconfig.h should be changed. #define HAVE_ADDRINFO 1 #define HAVE_SOCKADDR_STORAGE 1 Sure there are warnings about the redefinition of the _(SGIAPI|NO_XOPEN4|NO_XOPEN5) but it compiles and the tests don't fail. Also my applications run without exception or coredump on FUEL, Octane, Octane 2. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-08-01 22:10 Message: Logged In: YES user_id=33168 Ralf, I will try to work on this, but it won't be for a week or two probably. Can you tell me which machines (by name) work and doesn't work. The only IRIX machine I seem to know about it rattler. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-08-01 17:34 Message: Logged In: YES user_id=71407 For the records: The Python2.3 (final) socket module still does not compile under IRIX 6.5.20. It only compiles under 6.5.10. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-18 04:07 Message: Logged In: YES user_id=31392 I applied a slight variant of Neal's patch and that seemed to fix things. It still seems wrong that we define _SGIAPI, but that will have to wait for 2.3.1. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 23:54 Message: Logged In: YES user_id=31392 I learned a little about inet_ntop() in the last few minutes. It looks like the size_t argument is an older spelling, and the current POSIX says it should be socklen_t. Regardless, it looks like we're using the new definition when IRIX is actually providing the older definition. Neal attached a patch that looks like it should clean up the problem. Have you tried it? I think a better solution would be to teach configure how to find inet_ntop() via netdb.h, but that's probably too hard to get done in the new couple of hours. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 23:42 Message: Logged In: YES user_id=31392 There seem to be two causes of worry. The actual compiler error in your build log and the use of _SGIAPI that Casavanti warns again. It looks like the compiler problems is because our inet_ntop() prototype is wrong. That is Python says the last argument is a socklen_t, but IRIX and Linux man pages both say size_t. Perhaps size_t and socklen_t are different sizes on IRIX, but not on Linux. What happens if you change it to size_t? The problematic code would appear to be these lines from socketmodule.c: #if defined(__sgi)&&_COMPILER_VERSION>700 && !_SGIAPI /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #define _SGIAPI 1 #include "netdb.h" #endif It's the only use the _SGIAPI symbol that Casavant says we shouldn't use. What happens if you remove the use _SGIAPI, i.e. just make it #if defined(__sgi) && _COMPILER_VERSION > 700 /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #include "netdb.h" #endif ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-17 10:27 Message: Logged In: YES user_id=71407 We have confirmed that the socket module in Python 2.3b1 and 2.3b2 fails to compile under these versions of IRIX: 6.5.10, MIPSpro 7.3.1.2m 6.5.19, MIPSpro 7.3.1.3m 6.5.20, MIPSpro 7.3.1.3m (latest available, to my knowledge) If Python 2.3 goes out without a working socket module it will be useless for many people in science where IRIX is still found quite frequently. Maybe even worse, we will also no longer be able to advertise Python as a language that "runs everywhere." ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-16 08:37 Message: Logged In: YES user_id=71407 Date: Tue, 15 Jul 2003 15:06:25 -0500 From: "Brent Casavant" To: "Ralf W. Grosse-Kunstleve" Subject: Re: IRIX & Python Ralf, I read through the sourceforge report. I don't have a sourceforge account so I didn't add there, but feel free to distribute my ramblings. First, they should be very careful about what version of the operating system they are using. Real IPv6 support was only added a few releases ago (6.5.20 I believe, though my memory is hazy). Some of the functions/macros corresponding to IPv6 were introduced prior to that point, but implementation may not have been complete. If they have to play tricks with #define values to get a symbol recongnized it most likely means that SGI didn't intend for that symbol to be used yet. I suspect the problems they are seeing are because they are using some version of IRIX 6.5.x that includes inet_ntoa() definitions under _SGIAPI, but for which real support was not yet complete. The namespace of symbols that begin with a single underscore and a capital letter is reserved for the OS vendors, and the fact that someone is trying to manually insert a definition of _SGIAPI into the source should be a big red warning flag. That said, there is a correct method to deal with the new capabilities of IRIX 6.5.x over time. This comes in two parts. A better explanation is in /usr/include/optional_sym.h First, when a new symbol is added to IRIX, the corresponding header file will always have the following line in it: #pragma optional some_new_symbol The presence of this #pragma will cause the linker to not complain if the symbol is missing when the executable is linked. If SGI neglects to do so it is a bug that should be reported to us. The second part is that calls to these "optional" symbols should be protected by a run-time check to see if the symbol is actually present. Just like this: #include #include . . . if (_MIPS_SYMBOL_PRESENT(some_new_symbol)) { /* New function is present, use it */ qux = some_new_symbol(foo, bar, baz); } else { /* New function isn't present, do something else */ qux = builtin_some_new_symbol(foo, bar, baz); } This ensures that a single executable will be able to function properly (or at least degrade gracefully) on all version of IRIX 6.5.x. In other words you can compile on 6.5.20 and run on 6.5.8, or vice versa. In particular, this means that compile-time detection of features such as inet_ntoa() is incorrect for IRIX 6.5.x -- these decisions should be made at run-time instead. If the decision is made at compile time the executable may well not execute on an older version of IRIX 6.5.x. Unfortunately GNU configure is not equipped to utilize such advanced binary-compatability mechanisms, so there's no "easy" way to get GNU configure'd software to run across all IRIX 6.5.x releases, short of compiling only on the original non-upgraded version of IRIX 6.5. Hope that helps, Brent Casavant On Tue, 15 Jul 2003, Ralf W. Grosse-Kunstleve wrote: > We are developing a Python-based (www.python.org) application for > IRIX. Until now the Python IRIX port was fully supported, but the last > two beta releases of Python (2.3b1 and 2.3b2) make me increasingly > worried because the crucial socket module (TCP/IP sockets) fails to > build. A while ago I've filed a bug report, but the Python developers > appear to be stuck: > > https://sourceforge.net/tracker/? func=detail&aid=728330&group_id=5470&atid=105470 > > Is there a way to bring this to the attention of developers at SGI to > hopefully get some help? > > Thank you in advance! > Ralf > -- Brent Casavant Silicon Graphics, Inc. Where am I? And what am I IRIX O.S. Engineer http://www.sgi.com/ doing in this handbasket? bcasavan@sgi.com 44.8562N 93.1355W 860F -- Unknown ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-01 00:21 Message: Logged In: YES user_id=71407 The socket module still fails to compile with Python 2.3b2, in addition to two other extensions. The full make log is attached. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-22 16:47 Message: Logged In: YES user_id=21627 I think it would be best if detect presence of inet_aton/inet_pton. Is IPv6 support possible on this system? Then it would be best if that was detected, as well. If it is not possible to detect aton/pton/ipv6, what is the problem with not using them? I'd like to have a look at the relevant headers (in particular to understand the duplicate definition of _SGIAPI), indeed. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-05-22 09:35 Message: Logged In: YES user_id=33168 Argh!!! This is a nightmare. The attached hack ^h^h^h patch fixes the problem, but ... _SGIAPI needs to be defined (~192 in socketmodule.c) in order to get the prototypes for inet_aton, inet_pton. But _SGIAPI can't be defined in the standard header files because XOPEN_SOURCE is defined. Martin, do you have any other suggestions? I don't see how to do this right within configure. If you want I can send you the header file(s). ---------------------------------------------------------------------- Comment By: alexandre gillet (gillet) Date: 2003-05-13 03:41 Message: Logged In: YES user_id=150999 I had the same problem compiling socket on Irix. We did find that on our system ENABLE_IPV6 is not define and all the error are cause by that. System that do not enable IPV6 are not well supported with the socketmodule.c code. We were able to compile socket after making few change in the code: I am not sure if I did it right but it compile. I will post my patch but I do not see any place to attach my file example: See line 2794,2800 + #ifdef ENABLE_IPV6 char packed[MAX(sizeof(struct in_addr), sizeof(struct in6_addr))]; + #else + char packed[sizeof(struct in_addr)]; + #endif I am not sure how to post the patch or file. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-05-06 04:44 Message: Logged In: YES user_id=71407 This is not in my area of expertise, but I've tried a couple of (probably stupid) things: #include in socketmodule.c instead of #define _SGIAPI. This resulted in even more errors. ./configure --disable-ipv6 Same errors as before. I am lost here. Sorry. But you still have ssh access to rattler.lbl.gov if that helps. Accounts for other Python developers are a possibility. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-04 22:39 Message: Logged In: YES user_id=21627 Can you analyse these compiler errors in more detail, please? What, in your opinion, is causing these problems, and how would you correct them? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 From noreply at sourceforge.net Thu May 27 02:24:25 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 27 02:24:38 2004 Subject: [ python-Bugs-960995 ] test_zlib is too slow Message-ID: Bugs item #960995, was opened at 2004-05-26 13:35 Message generated for change (Comment added) made by tim_one You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960995&group_id=5470 Category: Python Library Group: Python 2.4 Status: Open Resolution: None Priority: 3 Submitted By: Michael Hudson (mwh) Assigned to: Nobody/Anonymous (nobody) Summary: test_zlib is too slow Initial Comment: I don't know what it's doing, but I've never seen it fail and waiting for it has certainly wasted quite a lot of my life :-) ---------------------------------------------------------------------- >Comment By: Tim Peters (tim_one) Date: 2004-05-27 02:24 Message: Logged In: YES user_id=31435 Persevere: taking tests you don't understand and just moving them to artificially bloat the time taken by an unrelated test is so lazy on so many counts I won't make you feel bad by belaboring the point . Moving them to yet another -u option doomed to be unused is possibly worse. IOW, fix the problem, don't shuffle it around. Or, IOOW, pare the expensive ones down. Since they never fail for anyone, it's not like they're testing something delicate. Does it *need* to try so many distinct cases? That will take some thought, but it's a real help that you already know the answer . ---------------------------------------------------------------------- Comment By: Brett Cannon (bcannon) Date: 2004-05-26 15:52 Message: Logged In: YES user_id=357491 A quick look at the tests Tim lists shows that each of those run the basic incremental decompression test 8 times, from the normal size to 2**8 time the base size; creates a list from [1<. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-26 14:17 Message: Logged In: YES user_id=80475 I hate this slow test. If you want to label this as explictly called resource, regrtest -u zlib , then be my guest. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=960995&group_id=5470 From noreply at sourceforge.net Thu May 27 03:17:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 27 03:17:46 2004 Subject: [ python-Bugs-728330 ] Don't define _SGAPI on IRIX Message-ID: Bugs item #728330, was opened at 2003-04-27 19:21 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 Category: Installation Group: Python 2.3 Status: Open Resolution: Remind Priority: 9 Submitted By: Ralf W. Grosse-Kunstleve (rwgk) Assigned to: Neal Norwitz (nnorwitz) Summary: Don't define _SGAPI on IRIX Initial Comment: Python release: 2.3b1 Platform: IRIX 6.5 MIPSpro Compilers: Version 7.3.1.2m Commands used: ./configure --prefix=/usr/local_cci/Python-2.3b1t make The socket module fails to compile. The output from the compiler is attached. ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-27 17:17 Message: Logged In: YES user_id=29957 There's a note on the 2.3.4 bugs page pointing at this bug. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-27 15:41 Message: Logged In: YES user_id=29957 If the patch is good, I'll make sure it's included in the 2.3.4 'bugs' page. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-27 14:14 Message: Logged In: YES user_id=71407 > Can you try gmake? Yes, it works with gmake. I've verified that the socket module on the CVS head builds and runs correctly on the three platforms mentioned in my message from 2004-05- 22 01:23. There are not even warnings. > Is IRIX make particularly braindead? I don't think so. It works with most auto-configured packages incl. all earlier versions of Python. But I'll save this for another day when the first 2.4 beta release is out. Thank you for your help! I think this tracker issue should be closed. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-27 04:19 Message: Logged In: YES user_id=6656 Uh, that's odd. Is IRIX make particularly braindead? Can you try gmake? Maybe we've let something non-portable slip in. At any rate, I think this should be done in a separate report... ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-27 04:14 Message: Logged In: YES user_id=71407 I checked out the python cvs head from scratch and ran configure in the usual way, with the usual result. However, make doesn't work: % make don't know how to make Parser/printgrammar.o (bu42). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-27 03:07 Message: Logged In: YES user_id=6656 OK, so I've commited revision 1.289 of Modules/socketmodule.c. Can you check if CVS HEAD works for you now? ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-27 02:59 Message: Logged In: YES user_id=71407 > Is there any chance of the Right Thing actually happening? The experience of the last 13 months suggests "No." SGI's freeware team has not responded to my report for more than a year. I am absolutely not a SGI or socket expert. I only worked on the patch because I was desperate enough and because Maik had already done the groundwork. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-27 02:27 Message: Logged In: YES user_id=6656 Is this really The Right Thing? I still see /* XXX Using _SGIAPI is the wrong thing, but I don't know what the right thing is. */ Is there any chance of the Right Thing actually happening? If not, I'll check this into HEAD, and if that seems to work, move it to release23-maint in time for 2.3.5. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-22 18:23 Message: Logged In: YES user_id=71407 Based on Maik's posting from 2004-03-08 08:53 I've worked out a small bug-fix patch relative to Python-2.3.4c1 that leads to successful creation of _socket.so under three different versions of IRIX: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.23 with MIPRpro 7.3.1.3 I've consistently used #if defined(__sgi) && _COMPILER_VERSION>700 to restrict the changes to IRIX. However, to be sure I've also verified that the Python installation is still OK under Redhat 8 and Tru64 Unix 5.1. I will upload the patch. It is also available here: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patch For completeness I've also posted these files: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patched_c http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_original_c I've made an effort to keep the patch minimally intrusive, but it is certainly not "clean." However, the status quo is so bad that it can only get better. We have not been able to switch to Python 2.3 because of the problems with the socket module. It would be fantastic if the bug-fix patch could be included in the 2.3.4 final distribution. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2004-03-09 03:53 Message: Logged In: YES user_id=768223 I got Python 2.3.3 compiled with MipsPro Compilers: Version 7.3.1.3m. The following changes were made to satisfy all needs. Line 206 in Modules/socketmodule.c #define _SGIAPI 1 NEW: #undef _XOPEN_SOURCE NEW: #define HAVE_GETADDRINFO 1 NEW: #define HAVE_GETNAMEINFO 1 #define HAVE_INET_PTON NEW: #include NEW: #include NEW: #include #include Line 269: /* #include "addrinfo.h" */ In addition to this I extend the CFLAGS with -D_BSD_TYPES to satisfy the u_short and friends definition. --maik./ ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-03-04 03:20 Message: Logged In: YES user_id=892601 Got it down to one error: % > ./python Lib/test/test_socket.py Traceback (most recent call last): File "Lib/test/test_socket.py", line 6, in ? import socket File "/usr/local/spider/lib/python2.3/socket.py", line 44, in ? import _socket ImportError: 210454:./python: rld: Fatal Error: unresolvable symbol in /usr/local/spider/lib/python2.3/lib-dynload/_socketmodule.so: _xpg5_accept xpg5_accept does appear in any library on our SGI's that are only 1-2 years old. It does appear in libc in very recent revisions of Irix (> 6.5.19?) Does anyone know how to get Python 2.3 to avoid the use of xpg5_accept? Thanks ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-01-13 05:59 Message: Logged In: YES user_id=892601 Has anyone encountered problems with different versions of IRIX 6.5? I finally got sockets to work in Python2.3.3, but only on the most up-to-date version of IRIX (uname -v = 01100304). On any other machine with a lower version (archetecture doesn't seem to matter), I get messages which all seem to be related to sockets: 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_vsnprintf 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recv 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_connect 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_bind 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_sendto 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_setsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_accept 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recvfrom 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_socket 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockname Upgrading all of our SGIs is not something I have control over. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-23 05:52 Message: Logged In: YES user_id=768223 Sorry if there is something unclear. Redefine means: Set a value which is required to satisfy ifdef-statements in the header files. As you see in the definitions, given in the comment, the value for _NO_XOPEN4 is set as a result of some dependencies automatically defined. A redefinition of _SGIAPI in the socketmodule.c is not sufficient alone. Therefore to satisfy all requirements in the header files, you should set the values for _NO_XOPEN4 and _NO_XOPEN5 also manually. Howto do in socketmodule.c? ... #define _SGIAPI 1 #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 ... If you are familiar with regular expressions, then you will know that _NO_XOPEN[45] is a shorthand for _NO_XOPEN4, _NO_XOPEN5. Sorry if there was something wrong in my comment. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-12-23 02:49 Message: Logged In: YES user_id=892601 Can someone please clarify mherta's comment from 2003-12-16? How should we "redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition"? Should those lines of code be added to socketmodule.c, or are they the source of the problem? Is _NO_XOPEN[45] shorthand for _NO_XOPEN[4] and _NO_XOPEN[5] or are they all different? Thanks. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-17 04:27 Message: Logged In: YES user_id=768223 I've tested the latest Python 2.3.3c1 to compile. The problem reported is still present. To avoid the error messages, about the incompatible types of several netdb-functions (like inet_pton), you should redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition. If there are error messages that types like u_short and u_char are not defined, please add -D_BSD_TYPES in the Makefile (BASECFLAGS is okay). If anybody don't need a executable compiled with MipsPro, gcc from the freeware.sgi.com could be a choice. The problem for this seems in /usr/include/standards.h: #define _NO_XOPEN4 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 >= 500))) #define _NO_XOPEN5 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 < 500))) As the first option is true (_XOPEN_SOURCE is set to 600 in pyconfig.h) the second option will be false. The constant #define NI_NUMERICHOST 0x00000002 /* return numeric form of hostname */ is enveloped in a clause #ifdef _NO_XOPEN4 && _NO_XOPEN5 (/usr/include/netdb.h) which is always false. In the manual pages there is a example which uses the constant NI_NUMERICHOST and compiles without error message. I'm not sure to cut off the file pyconfig.h from compiling the socketmodule.c? Any comments? ---------------------------------------------------------------------- Comment By: Hans Nowak (zephyrfalcon) Date: 2003-10-31 14:04 Message: Logged In: YES user_id=173607 My $0.02... Yesterday I successfully compiled Python 2.3.2, with socket module. I'm using an O2 running IRIX 6.5.11f. I didn't encounter any of the problems described before, and had no problems with _SGIAPI or anything. The reason it didn't compile was that u_long and u_char were not defined in some places; /usr/include/netinet/tcp.h to be precise, and a Python file (getaddrinfo.c). I'm using gcc 3.3 (the one downloaded from freeware.sgi.com). If anyone is interested, drop me a mail at hans at zephyrfalcon.org and I'll put the binary online somewhere. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-25 05:59 Message: Logged In: YES user_id=892601 Here's the output from 'gmake', using MIPSpro Compiler 7.30 on IRIX 6.5 Sorry, I don't know how to attach something using the "Add a Comment" box. Is there another way to post? The patch at www.python.org/2.3.2/patches/sgi-socketmodule.patch was edited into socketmodule.c : #include #endif /* GCC 3.2 on Irix 6.5 fails to define this variable at all. Based on the above code, I'd say that the SGI headers are just horked. */ #if defined(__sgi) && !defined(INET_ADDRSTRLEN) #define INET_ADDRSTRLEN 16 #endif /* Generic includes */ #include #include --------------------------------------------------------------------------- cc -OPT:Olimit=0 -DNDEBUG -n32 -O -I. -I./Include -DPy_BUILD_CORE -c ./Modules/socketmodule.c -o Modules/socketmodule.o cc-1047 cc: WARNING File = ./Modules/socketmodule.c, Line = 195 Macro "_SGIAPI" (declared at line 210 of "/usr/include/standards.h") has an incompatible redefinition. #define _SGIAPI 1 ^ cc-1020 cc: ERROR File = /usr/include/netdb.h, Line = 198 The identifier "socklen_t" is undefined. extern int iruserok_sa(const void *, socklen_t, int, const char *, ^ cc-1047 cc: WARNING File = ./Modules/addrinfo.h, Line = 145 Macro "_SS_ALIGNSIZE" (declared at line 246 of "/usr/include/sys/socket.h") has an incompatible redefinition. #define _SS_ALIGNSIZE (sizeof(PY_LONG_LONG)) ^ cc-1020 cc: ERROR File = ./Modules/socketmodule.c, Line = 822 The identifier "NI_NUMERICHOST" is undefined. NI_NUMERICHOST); ^ cc-1515 cc: ERROR File = ./Modules/socketmodule.c, Line = 3054 A value of type "int" cannot be assigned to an entity of type "const char *". retval = inet_ntop(af, packed, ip, sizeof(ip)); ^ 3 errors detected in the compilation of "./Modules/socketmodule.c". gmake: *** [Modules/socketmodule.o] Error 2 ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2003-10-23 20:25 Message: Logged In: YES user_id=6656 Just to be clear, please "attach" the build rather than pasting it inline. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2003-10-23 12:19 Message: Logged In: YES user_id=29957 Did you try 2.3.2 with the patch from the 2.3.2 build bugs page? If it failed, please post the build log of the failing bit to this bug. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-23 04:26 Message: Logged In: YES user_id=892601 Has there been any progress since August on getting Python-2.3 to compile on IRIX 6.5? I really need to have sockets working to use python. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-08-14 21:37 Message: Logged In: YES user_id=768223 I have the same problem on IRIX 6.5.19m and MIPSpro 7.3.1.3m on a FUEL 600Mhz. The situation seems a little bit strange on this system. But I have luck and get it compiled. As the current implementation is not clean, my resolution also. Which should be changed to get it work. The manpage for gethostbyaddr_r says: #include #include #include #include So I included this in the __sgi-clause in socketmodule.c. Looking in the header-files shows, that some variables are defined if #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 are set, before the includes. This defines some more variables, so that pyconfig.h should be changed. #define HAVE_ADDRINFO 1 #define HAVE_SOCKADDR_STORAGE 1 Sure there are warnings about the redefinition of the _(SGIAPI|NO_XOPEN4|NO_XOPEN5) but it compiles and the tests don't fail. Also my applications run without exception or coredump on FUEL, Octane, Octane 2. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-08-01 22:10 Message: Logged In: YES user_id=33168 Ralf, I will try to work on this, but it won't be for a week or two probably. Can you tell me which machines (by name) work and doesn't work. The only IRIX machine I seem to know about it rattler. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-08-01 17:34 Message: Logged In: YES user_id=71407 For the records: The Python2.3 (final) socket module still does not compile under IRIX 6.5.20. It only compiles under 6.5.10. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-18 04:07 Message: Logged In: YES user_id=31392 I applied a slight variant of Neal's patch and that seemed to fix things. It still seems wrong that we define _SGIAPI, but that will have to wait for 2.3.1. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 23:54 Message: Logged In: YES user_id=31392 I learned a little about inet_ntop() in the last few minutes. It looks like the size_t argument is an older spelling, and the current POSIX says it should be socklen_t. Regardless, it looks like we're using the new definition when IRIX is actually providing the older definition. Neal attached a patch that looks like it should clean up the problem. Have you tried it? I think a better solution would be to teach configure how to find inet_ntop() via netdb.h, but that's probably too hard to get done in the new couple of hours. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 23:42 Message: Logged In: YES user_id=31392 There seem to be two causes of worry. The actual compiler error in your build log and the use of _SGIAPI that Casavanti warns again. It looks like the compiler problems is because our inet_ntop() prototype is wrong. That is Python says the last argument is a socklen_t, but IRIX and Linux man pages both say size_t. Perhaps size_t and socklen_t are different sizes on IRIX, but not on Linux. What happens if you change it to size_t? The problematic code would appear to be these lines from socketmodule.c: #if defined(__sgi)&&_COMPILER_VERSION>700 && !_SGIAPI /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #define _SGIAPI 1 #include "netdb.h" #endif It's the only use the _SGIAPI symbol that Casavant says we shouldn't use. What happens if you remove the use _SGIAPI, i.e. just make it #if defined(__sgi) && _COMPILER_VERSION > 700 /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #include "netdb.h" #endif ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-17 10:27 Message: Logged In: YES user_id=71407 We have confirmed that the socket module in Python 2.3b1 and 2.3b2 fails to compile under these versions of IRIX: 6.5.10, MIPSpro 7.3.1.2m 6.5.19, MIPSpro 7.3.1.3m 6.5.20, MIPSpro 7.3.1.3m (latest available, to my knowledge) If Python 2.3 goes out without a working socket module it will be useless for many people in science where IRIX is still found quite frequently. Maybe even worse, we will also no longer be able to advertise Python as a language that "runs everywhere." ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-16 08:37 Message: Logged In: YES user_id=71407 Date: Tue, 15 Jul 2003 15:06:25 -0500 From: "Brent Casavant" To: "Ralf W. Grosse-Kunstleve" Subject: Re: IRIX & Python Ralf, I read through the sourceforge report. I don't have a sourceforge account so I didn't add there, but feel free to distribute my ramblings. First, they should be very careful about what version of the operating system they are using. Real IPv6 support was only added a few releases ago (6.5.20 I believe, though my memory is hazy). Some of the functions/macros corresponding to IPv6 were introduced prior to that point, but implementation may not have been complete. If they have to play tricks with #define values to get a symbol recongnized it most likely means that SGI didn't intend for that symbol to be used yet. I suspect the problems they are seeing are because they are using some version of IRIX 6.5.x that includes inet_ntoa() definitions under _SGIAPI, but for which real support was not yet complete. The namespace of symbols that begin with a single underscore and a capital letter is reserved for the OS vendors, and the fact that someone is trying to manually insert a definition of _SGIAPI into the source should be a big red warning flag. That said, there is a correct method to deal with the new capabilities of IRIX 6.5.x over time. This comes in two parts. A better explanation is in /usr/include/optional_sym.h First, when a new symbol is added to IRIX, the corresponding header file will always have the following line in it: #pragma optional some_new_symbol The presence of this #pragma will cause the linker to not complain if the symbol is missing when the executable is linked. If SGI neglects to do so it is a bug that should be reported to us. The second part is that calls to these "optional" symbols should be protected by a run-time check to see if the symbol is actually present. Just like this: #include #include . . . if (_MIPS_SYMBOL_PRESENT(some_new_symbol)) { /* New function is present, use it */ qux = some_new_symbol(foo, bar, baz); } else { /* New function isn't present, do something else */ qux = builtin_some_new_symbol(foo, bar, baz); } This ensures that a single executable will be able to function properly (or at least degrade gracefully) on all version of IRIX 6.5.x. In other words you can compile on 6.5.20 and run on 6.5.8, or vice versa. In particular, this means that compile-time detection of features such as inet_ntoa() is incorrect for IRIX 6.5.x -- these decisions should be made at run-time instead. If the decision is made at compile time the executable may well not execute on an older version of IRIX 6.5.x. Unfortunately GNU configure is not equipped to utilize such advanced binary-compatability mechanisms, so there's no "easy" way to get GNU configure'd software to run across all IRIX 6.5.x releases, short of compiling only on the original non-upgraded version of IRIX 6.5. Hope that helps, Brent Casavant On Tue, 15 Jul 2003, Ralf W. Grosse-Kunstleve wrote: > We are developing a Python-based (www.python.org) application for > IRIX. Until now the Python IRIX port was fully supported, but the last > two beta releases of Python (2.3b1 and 2.3b2) make me increasingly > worried because the crucial socket module (TCP/IP sockets) fails to > build. A while ago I've filed a bug report, but the Python developers > appear to be stuck: > > https://sourceforge.net/tracker/? func=detail&aid=728330&group_id=5470&atid=105470 > > Is there a way to bring this to the attention of developers at SGI to > hopefully get some help? > > Thank you in advance! > Ralf > -- Brent Casavant Silicon Graphics, Inc. Where am I? And what am I IRIX O.S. Engineer http://www.sgi.com/ doing in this handbasket? bcasavan@sgi.com 44.8562N 93.1355W 860F -- Unknown ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-01 00:21 Message: Logged In: YES user_id=71407 The socket module still fails to compile with Python 2.3b2, in addition to two other extensions. The full make log is attached. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-22 16:47 Message: Logged In: YES user_id=21627 I think it would be best if detect presence of inet_aton/inet_pton. Is IPv6 support possible on this system? Then it would be best if that was detected, as well. If it is not possible to detect aton/pton/ipv6, what is the problem with not using them? I'd like to have a look at the relevant headers (in particular to understand the duplicate definition of _SGIAPI), indeed. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-05-22 09:35 Message: Logged In: YES user_id=33168 Argh!!! This is a nightmare. The attached hack ^h^h^h patch fixes the problem, but ... _SGIAPI needs to be defined (~192 in socketmodule.c) in order to get the prototypes for inet_aton, inet_pton. But _SGIAPI can't be defined in the standard header files because XOPEN_SOURCE is defined. Martin, do you have any other suggestions? I don't see how to do this right within configure. If you want I can send you the header file(s). ---------------------------------------------------------------------- Comment By: alexandre gillet (gillet) Date: 2003-05-13 03:41 Message: Logged In: YES user_id=150999 I had the same problem compiling socket on Irix. We did find that on our system ENABLE_IPV6 is not define and all the error are cause by that. System that do not enable IPV6 are not well supported with the socketmodule.c code. We were able to compile socket after making few change in the code: I am not sure if I did it right but it compile. I will post my patch but I do not see any place to attach my file example: See line 2794,2800 + #ifdef ENABLE_IPV6 char packed[MAX(sizeof(struct in_addr), sizeof(struct in6_addr))]; + #else + char packed[sizeof(struct in_addr)]; + #endif I am not sure how to post the patch or file. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-05-06 04:44 Message: Logged In: YES user_id=71407 This is not in my area of expertise, but I've tried a couple of (probably stupid) things: #include in socketmodule.c instead of #define _SGIAPI. This resulted in even more errors. ./configure --disable-ipv6 Same errors as before. I am lost here. Sorry. But you still have ssh access to rattler.lbl.gov if that helps. Accounts for other Python developers are a possibility. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-04 22:39 Message: Logged In: YES user_id=21627 Can you analyse these compiler errors in more detail, please? What, in your opinion, is causing these problems, and how would you correct them? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 From noreply at sourceforge.net Thu May 27 06:28:05 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 27 06:29:44 2004 Subject: [ python-Bugs-728330 ] Don't define _SGAPI on IRIX Message-ID: Bugs item #728330, was opened at 2003-04-27 10:21 Message generated for change (Comment added) made by mwh You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 Category: Installation Group: Python 2.3 >Status: Closed Resolution: Remind Priority: 9 Submitted By: Ralf W. Grosse-Kunstleve (rwgk) Assigned to: Neal Norwitz (nnorwitz) Summary: Don't define _SGAPI on IRIX Initial Comment: Python release: 2.3b1 Platform: IRIX 6.5 MIPSpro Compilers: Version 7.3.1.2m Commands used: ./configure --prefix=/usr/local_cci/Python-2.3b1t make The socket module fails to compile. The output from the compiler is attached. ---------------------------------------------------------------------- >Comment By: Michael Hudson (mwh) Date: 2004-05-27 11:28 Message: Logged In: YES user_id=6656 Closing! ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-27 08:17 Message: Logged In: YES user_id=29957 There's a note on the 2.3.4 bugs page pointing at this bug. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-27 06:41 Message: Logged In: YES user_id=29957 If the patch is good, I'll make sure it's included in the 2.3.4 'bugs' page. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-27 05:14 Message: Logged In: YES user_id=71407 > Can you try gmake? Yes, it works with gmake. I've verified that the socket module on the CVS head builds and runs correctly on the three platforms mentioned in my message from 2004-05- 22 01:23. There are not even warnings. > Is IRIX make particularly braindead? I don't think so. It works with most auto-configured packages incl. all earlier versions of Python. But I'll save this for another day when the first 2.4 beta release is out. Thank you for your help! I think this tracker issue should be closed. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 19:19 Message: Logged In: YES user_id=6656 Uh, that's odd. Is IRIX make particularly braindead? Can you try gmake? Maybe we've let something non-portable slip in. At any rate, I think this should be done in a separate report... ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 19:14 Message: Logged In: YES user_id=71407 I checked out the python cvs head from scratch and ran configure in the usual way, with the usual result. However, make doesn't work: % make don't know how to make Parser/printgrammar.o (bu42). ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 18:07 Message: Logged In: YES user_id=6656 OK, so I've commited revision 1.289 of Modules/socketmodule.c. Can you check if CVS HEAD works for you now? ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-26 17:59 Message: Logged In: YES user_id=71407 > Is there any chance of the Right Thing actually happening? The experience of the last 13 months suggests "No." SGI's freeware team has not responded to my report for more than a year. I am absolutely not a SGI or socket expert. I only worked on the patch because I was desperate enough and because Maik had already done the groundwork. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 17:27 Message: Logged In: YES user_id=6656 Is this really The Right Thing? I still see /* XXX Using _SGIAPI is the wrong thing, but I don't know what the right thing is. */ Is there any chance of the Right Thing actually happening? If not, I'll check this into HEAD, and if that seems to work, move it to release23-maint in time for 2.3.5. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2004-05-22 09:23 Message: Logged In: YES user_id=71407 Based on Maik's posting from 2004-03-08 08:53 I've worked out a small bug-fix patch relative to Python-2.3.4c1 that leads to successful creation of _socket.so under three different versions of IRIX: 6.5.10 with MIPSpro 7.3.1.2 6.5.21 with MIPSpro 7.41 6.5.23 with MIPRpro 7.3.1.3 I've consistently used #if defined(__sgi) && _COMPILER_VERSION>700 to restrict the changes to IRIX. However, to be sure I've also verified that the Python installation is still OK under Redhat 8 and Tru64 Unix 5.1. I will upload the patch. It is also available here: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patch For completeness I've also posted these files: http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_patched_c http://cci.lbl.gov/~rwgk/bugs/python/py234c1_socketmod ule_original_c I've made an effort to keep the patch minimally intrusive, but it is certainly not "clean." However, the status quo is so bad that it can only get better. We have not been able to switch to Python 2.3 because of the problems with the socket module. It would be fantastic if the bug-fix patch could be included in the 2.3.4 final distribution. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2004-03-08 16:53 Message: Logged In: YES user_id=768223 I got Python 2.3.3 compiled with MipsPro Compilers: Version 7.3.1.3m. The following changes were made to satisfy all needs. Line 206 in Modules/socketmodule.c #define _SGIAPI 1 NEW: #undef _XOPEN_SOURCE NEW: #define HAVE_GETADDRINFO 1 NEW: #define HAVE_GETNAMEINFO 1 #define HAVE_INET_PTON NEW: #include NEW: #include NEW: #include #include Line 269: /* #include "addrinfo.h" */ In addition to this I extend the CFLAGS with -D_BSD_TYPES to satisfy the u_short and friends definition. --maik./ ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-03-03 16:20 Message: Logged In: YES user_id=892601 Got it down to one error: % > ./python Lib/test/test_socket.py Traceback (most recent call last): File "Lib/test/test_socket.py", line 6, in ? import socket File "/usr/local/spider/lib/python2.3/socket.py", line 44, in ? import _socket ImportError: 210454:./python: rld: Fatal Error: unresolvable symbol in /usr/local/spider/lib/python2.3/lib-dynload/_socketmodule.so: _xpg5_accept xpg5_accept does appear in any library on our SGI's that are only 1-2 years old. It does appear in libc in very recent revisions of Irix (> 6.5.19?) Does anyone know how to get Python 2.3 to avoid the use of xpg5_accept? Thanks ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2004-01-12 18:59 Message: Logged In: YES user_id=892601 Has anyone encountered problems with different versions of IRIX 6.5? I finally got sockets to work in Python2.3.3, but only on the most up-to-date version of IRIX (uname -v = 01100304). On any other machine with a lower version (archetecture doesn't seem to matter), I get messages which all seem to be related to sockets: 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_vsnprintf 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recv 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_connect 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_bind 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_sendto 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_setsockopt 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_accept 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_recvfrom 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_socket 152272:python2.3: rld: Error: unresolvable symbol in python2.3: _xpg5_getsockname Upgrading all of our SGIs is not something I have control over. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-22 18:52 Message: Logged In: YES user_id=768223 Sorry if there is something unclear. Redefine means: Set a value which is required to satisfy ifdef-statements in the header files. As you see in the definitions, given in the comment, the value for _NO_XOPEN4 is set as a result of some dependencies automatically defined. A redefinition of _SGIAPI in the socketmodule.c is not sufficient alone. Therefore to satisfy all requirements in the header files, you should set the values for _NO_XOPEN4 and _NO_XOPEN5 also manually. Howto do in socketmodule.c? ... #define _SGIAPI 1 #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 ... If you are familiar with regular expressions, then you will know that _NO_XOPEN[45] is a shorthand for _NO_XOPEN4, _NO_XOPEN5. Sorry if there was something wrong in my comment. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-12-22 15:49 Message: Logged In: YES user_id=892601 Can someone please clarify mherta's comment from 2003-12-16? How should we "redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition"? Should those lines of code be added to socketmodule.c, or are they the source of the problem? Is _NO_XOPEN[45] shorthand for _NO_XOPEN[4] and _NO_XOPEN[5] or are they all different? Thanks. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-12-16 17:27 Message: Logged In: YES user_id=768223 I've tested the latest Python 2.3.3c1 to compile. The problem reported is still present. To avoid the error messages, about the incompatible types of several netdb-functions (like inet_pton), you should redefine _NO_XOPEN[45] in the socketmodule.c at the same position like the _SGIAPI redefinition. If there are error messages that types like u_short and u_char are not defined, please add -D_BSD_TYPES in the Makefile (BASECFLAGS is okay). If anybody don't need a executable compiled with MipsPro, gcc from the freeware.sgi.com could be a choice. The problem for this seems in /usr/include/standards.h: #define _NO_XOPEN4 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 >= 500))) #define _NO_XOPEN5 (!defined(_XOPEN_SOURCE) || (defined(_XOPEN_SOURCE) && (_XOPEN_SOURCE+0 < 500))) As the first option is true (_XOPEN_SOURCE is set to 600 in pyconfig.h) the second option will be false. The constant #define NI_NUMERICHOST 0x00000002 /* return numeric form of hostname */ is enveloped in a clause #ifdef _NO_XOPEN4 && _NO_XOPEN5 (/usr/include/netdb.h) which is always false. In the manual pages there is a example which uses the constant NI_NUMERICHOST and compiles without error message. I'm not sure to cut off the file pyconfig.h from compiling the socketmodule.c? Any comments? ---------------------------------------------------------------------- Comment By: Hans Nowak (zephyrfalcon) Date: 2003-10-31 03:04 Message: Logged In: YES user_id=173607 My $0.02... Yesterday I successfully compiled Python 2.3.2, with socket module. I'm using an O2 running IRIX 6.5.11f. I didn't encounter any of the problems described before, and had no problems with _SGIAPI or anything. The reason it didn't compile was that u_long and u_char were not defined in some places; /usr/include/netinet/tcp.h to be precise, and a Python file (getaddrinfo.c). I'm using gcc 3.3 (the one downloaded from freeware.sgi.com). If anyone is interested, drop me a mail at hans at zephyrfalcon.org and I'll put the binary online somewhere. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-24 20:59 Message: Logged In: YES user_id=892601 Here's the output from 'gmake', using MIPSpro Compiler 7.30 on IRIX 6.5 Sorry, I don't know how to attach something using the "Add a Comment" box. Is there another way to post? The patch at www.python.org/2.3.2/patches/sgi-socketmodule.patch was edited into socketmodule.c : #include #endif /* GCC 3.2 on Irix 6.5 fails to define this variable at all. Based on the above code, I'd say that the SGI headers are just horked. */ #if defined(__sgi) && !defined(INET_ADDRSTRLEN) #define INET_ADDRSTRLEN 16 #endif /* Generic includes */ #include #include --------------------------------------------------------------------------- cc -OPT:Olimit=0 -DNDEBUG -n32 -O -I. -I./Include -DPy_BUILD_CORE -c ./Modules/socketmodule.c -o Modules/socketmodule.o cc-1047 cc: WARNING File = ./Modules/socketmodule.c, Line = 195 Macro "_SGIAPI" (declared at line 210 of "/usr/include/standards.h") has an incompatible redefinition. #define _SGIAPI 1 ^ cc-1020 cc: ERROR File = /usr/include/netdb.h, Line = 198 The identifier "socklen_t" is undefined. extern int iruserok_sa(const void *, socklen_t, int, const char *, ^ cc-1047 cc: WARNING File = ./Modules/addrinfo.h, Line = 145 Macro "_SS_ALIGNSIZE" (declared at line 246 of "/usr/include/sys/socket.h") has an incompatible redefinition. #define _SS_ALIGNSIZE (sizeof(PY_LONG_LONG)) ^ cc-1020 cc: ERROR File = ./Modules/socketmodule.c, Line = 822 The identifier "NI_NUMERICHOST" is undefined. NI_NUMERICHOST); ^ cc-1515 cc: ERROR File = ./Modules/socketmodule.c, Line = 3054 A value of type "int" cannot be assigned to an entity of type "const char *". retval = inet_ntop(af, packed, ip, sizeof(ip)); ^ 3 errors detected in the compilation of "./Modules/socketmodule.c". gmake: *** [Modules/socketmodule.o] Error 2 ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2003-10-23 11:25 Message: Logged In: YES user_id=6656 Just to be clear, please "attach" the build rather than pasting it inline. ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2003-10-23 03:19 Message: Logged In: YES user_id=29957 Did you try 2.3.2 with the patch from the 2.3.2 build bugs page? If it failed, please post the build log of the failing bit to this bug. ---------------------------------------------------------------------- Comment By: Bill Baxter (billbaxter26) Date: 2003-10-22 19:26 Message: Logged In: YES user_id=892601 Has there been any progress since August on getting Python-2.3 to compile on IRIX 6.5? I really need to have sockets working to use python. ---------------------------------------------------------------------- Comment By: Maik Hertha (mhertha) Date: 2003-08-14 12:37 Message: Logged In: YES user_id=768223 I have the same problem on IRIX 6.5.19m and MIPSpro 7.3.1.3m on a FUEL 600Mhz. The situation seems a little bit strange on this system. But I have luck and get it compiled. As the current implementation is not clean, my resolution also. Which should be changed to get it work. The manpage for gethostbyaddr_r says: #include #include #include #include So I included this in the __sgi-clause in socketmodule.c. Looking in the header-files shows, that some variables are defined if #define _NO_XOPEN4 1 #define _NO_XOPEN5 1 are set, before the includes. This defines some more variables, so that pyconfig.h should be changed. #define HAVE_ADDRINFO 1 #define HAVE_SOCKADDR_STORAGE 1 Sure there are warnings about the redefinition of the _(SGIAPI|NO_XOPEN4|NO_XOPEN5) but it compiles and the tests don't fail. Also my applications run without exception or coredump on FUEL, Octane, Octane 2. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-08-01 13:10 Message: Logged In: YES user_id=33168 Ralf, I will try to work on this, but it won't be for a week or two probably. Can you tell me which machines (by name) work and doesn't work. The only IRIX machine I seem to know about it rattler. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-08-01 08:34 Message: Logged In: YES user_id=71407 For the records: The Python2.3 (final) socket module still does not compile under IRIX 6.5.20. It only compiles under 6.5.10. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 19:07 Message: Logged In: YES user_id=31392 I applied a slight variant of Neal's patch and that seemed to fix things. It still seems wrong that we define _SGIAPI, but that will have to wait for 2.3.1. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 14:54 Message: Logged In: YES user_id=31392 I learned a little about inet_ntop() in the last few minutes. It looks like the size_t argument is an older spelling, and the current POSIX says it should be socklen_t. Regardless, it looks like we're using the new definition when IRIX is actually providing the older definition. Neal attached a patch that looks like it should clean up the problem. Have you tried it? I think a better solution would be to teach configure how to find inet_ntop() via netdb.h, but that's probably too hard to get done in the new couple of hours. ---------------------------------------------------------------------- Comment By: Jeremy Hylton (jhylton) Date: 2003-07-17 14:42 Message: Logged In: YES user_id=31392 There seem to be two causes of worry. The actual compiler error in your build log and the use of _SGIAPI that Casavanti warns again. It looks like the compiler problems is because our inet_ntop() prototype is wrong. That is Python says the last argument is a socklen_t, but IRIX and Linux man pages both say size_t. Perhaps size_t and socklen_t are different sizes on IRIX, but not on Linux. What happens if you change it to size_t? The problematic code would appear to be these lines from socketmodule.c: #if defined(__sgi)&&_COMPILER_VERSION>700 && !_SGIAPI /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #define _SGIAPI 1 #include "netdb.h" #endif It's the only use the _SGIAPI symbol that Casavant says we shouldn't use. What happens if you remove the use _SGIAPI, i.e. just make it #if defined(__sgi) && _COMPILER_VERSION > 700 /* make sure that the reentrant (gethostbyaddr_r etc) functions are declared correctly if compiling with MIPSPro 7.x in ANSI C mode (default) */ #include "netdb.h" #endif ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-17 01:27 Message: Logged In: YES user_id=71407 We have confirmed that the socket module in Python 2.3b1 and 2.3b2 fails to compile under these versions of IRIX: 6.5.10, MIPSpro 7.3.1.2m 6.5.19, MIPSpro 7.3.1.3m 6.5.20, MIPSpro 7.3.1.3m (latest available, to my knowledge) If Python 2.3 goes out without a working socket module it will be useless for many people in science where IRIX is still found quite frequently. Maybe even worse, we will also no longer be able to advertise Python as a language that "runs everywhere." ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-07-15 23:37 Message: Logged In: YES user_id=71407 Date: Tue, 15 Jul 2003 15:06:25 -0500 From: "Brent Casavant" To: "Ralf W. Grosse-Kunstleve" Subject: Re: IRIX & Python Ralf, I read through the sourceforge report. I don't have a sourceforge account so I didn't add there, but feel free to distribute my ramblings. First, they should be very careful about what version of the operating system they are using. Real IPv6 support was only added a few releases ago (6.5.20 I believe, though my memory is hazy). Some of the functions/macros corresponding to IPv6 were introduced prior to that point, but implementation may not have been complete. If they have to play tricks with #define values to get a symbol recongnized it most likely means that SGI didn't intend for that symbol to be used yet. I suspect the problems they are seeing are because they are using some version of IRIX 6.5.x that includes inet_ntoa() definitions under _SGIAPI, but for which real support was not yet complete. The namespace of symbols that begin with a single underscore and a capital letter is reserved for the OS vendors, and the fact that someone is trying to manually insert a definition of _SGIAPI into the source should be a big red warning flag. That said, there is a correct method to deal with the new capabilities of IRIX 6.5.x over time. This comes in two parts. A better explanation is in /usr/include/optional_sym.h First, when a new symbol is added to IRIX, the corresponding header file will always have the following line in it: #pragma optional some_new_symbol The presence of this #pragma will cause the linker to not complain if the symbol is missing when the executable is linked. If SGI neglects to do so it is a bug that should be reported to us. The second part is that calls to these "optional" symbols should be protected by a run-time check to see if the symbol is actually present. Just like this: #include #include . . . if (_MIPS_SYMBOL_PRESENT(some_new_symbol)) { /* New function is present, use it */ qux = some_new_symbol(foo, bar, baz); } else { /* New function isn't present, do something else */ qux = builtin_some_new_symbol(foo, bar, baz); } This ensures that a single executable will be able to function properly (or at least degrade gracefully) on all version of IRIX 6.5.x. In other words you can compile on 6.5.20 and run on 6.5.8, or vice versa. In particular, this means that compile-time detection of features such as inet_ntoa() is incorrect for IRIX 6.5.x -- these decisions should be made at run-time instead. If the decision is made at compile time the executable may well not execute on an older version of IRIX 6.5.x. Unfortunately GNU configure is not equipped to utilize such advanced binary-compatability mechanisms, so there's no "easy" way to get GNU configure'd software to run across all IRIX 6.5.x releases, short of compiling only on the original non-upgraded version of IRIX 6.5. Hope that helps, Brent Casavant On Tue, 15 Jul 2003, Ralf W. Grosse-Kunstleve wrote: > We are developing a Python-based (www.python.org) application for > IRIX. Until now the Python IRIX port was fully supported, but the last > two beta releases of Python (2.3b1 and 2.3b2) make me increasingly > worried because the crucial socket module (TCP/IP sockets) fails to > build. A while ago I've filed a bug report, but the Python developers > appear to be stuck: > > https://sourceforge.net/tracker/? func=detail&aid=728330&group_id=5470&atid=105470 > > Is there a way to bring this to the attention of developers at SGI to > hopefully get some help? > > Thank you in advance! > Ralf > -- Brent Casavant Silicon Graphics, Inc. Where am I? And what am I IRIX O.S. Engineer http://www.sgi.com/ doing in this handbasket? bcasavan@sgi.com 44.8562N 93.1355W 860F -- Unknown ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-06-30 15:21 Message: Logged In: YES user_id=71407 The socket module still fails to compile with Python 2.3b2, in addition to two other extensions. The full make log is attached. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-22 07:47 Message: Logged In: YES user_id=21627 I think it would be best if detect presence of inet_aton/inet_pton. Is IPv6 support possible on this system? Then it would be best if that was detected, as well. If it is not possible to detect aton/pton/ipv6, what is the problem with not using them? I'd like to have a look at the relevant headers (in particular to understand the duplicate definition of _SGIAPI), indeed. ---------------------------------------------------------------------- Comment By: Neal Norwitz (nnorwitz) Date: 2003-05-22 00:35 Message: Logged In: YES user_id=33168 Argh!!! This is a nightmare. The attached hack ^h^h^h patch fixes the problem, but ... _SGIAPI needs to be defined (~192 in socketmodule.c) in order to get the prototypes for inet_aton, inet_pton. But _SGIAPI can't be defined in the standard header files because XOPEN_SOURCE is defined. Martin, do you have any other suggestions? I don't see how to do this right within configure. If you want I can send you the header file(s). ---------------------------------------------------------------------- Comment By: alexandre gillet (gillet) Date: 2003-05-12 18:41 Message: Logged In: YES user_id=150999 I had the same problem compiling socket on Irix. We did find that on our system ENABLE_IPV6 is not define and all the error are cause by that. System that do not enable IPV6 are not well supported with the socketmodule.c code. We were able to compile socket after making few change in the code: I am not sure if I did it right but it compile. I will post my patch but I do not see any place to attach my file example: See line 2794,2800 + #ifdef ENABLE_IPV6 char packed[MAX(sizeof(struct in_addr), sizeof(struct in6_addr))]; + #else + char packed[sizeof(struct in_addr)]; + #endif I am not sure how to post the patch or file. ---------------------------------------------------------------------- Comment By: Ralf W. Grosse-Kunstleve (rwgk) Date: 2003-05-05 19:44 Message: Logged In: YES user_id=71407 This is not in my area of expertise, but I've tried a couple of (probably stupid) things: #include in socketmodule.c instead of #define _SGIAPI. This resulted in even more errors. ./configure --disable-ipv6 Same errors as before. I am lost here. Sorry. But you still have ssh access to rattler.lbl.gov if that helps. Accounts for other Python developers are a possibility. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2003-05-04 13:39 Message: Logged In: YES user_id=21627 Can you analyse these compiler errors in more detail, please? What, in your opinion, is causing these problems, and how would you correct them? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=728330&group_id=5470 From noreply at sourceforge.net Thu May 27 09:04:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 27 09:04:17 2004 Subject: [ python-Feature Requests-960821 ] Add an updating load function in pickle Message-ID: Feature Requests item #960821, was opened at 2004-05-26 09:34 Message generated for change (Comment added) made by nidoizo You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=960821&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Nicolas Fleury (nidoizo) Assigned to: Nobody/Anonymous (nobody) Summary: Add an updating load function in pickle Initial Comment: It would be very nice to have a function in pickle to load an object without creating a new instance, but by using an existing one, meaning that __init__, etc. of an existing instance is called. Something like: updateload(object, file) (I have no patch in mind.) ---------------------------------------------------------------------- >Comment By: Nicolas Fleury (nidoizo) Date: 2004-05-27 09:04 Message: Logged In: YES user_id=151545 I realize it might not be clear. What is needed is simply a function that does the same as pickle.load but that uses an existing instance instead of creating another one. All the rest after that point is the same. It means this function would raise an exception if the instance is not of the good type, etc. (Forget the part about __init__). ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=960821&group_id=5470 From noreply at sourceforge.net Thu May 27 15:59:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 27 15:59:33 2004 Subject: [ python-Bugs-961805 ] Text.edit_modified() fails Message-ID: Bugs item #961805, was opened at 2004-05-27 12:59 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=961805&group_id=5470 Category: Tkinter Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Eric P. (ericpett) Assigned to: Martin v. L?wis (loewis) Summary: Text.edit_modified() fails Initial Comment: Text.edit_modified() fails because it calls _getints() on the result of the underlying tk.call(). The tk.call() is returning a boolean, not a string, and the _getints() fails. Here's the last part of an example traceback: File "/usr/local/src/staff/pett/chimera/devel/Ilabel/ __init__.py", line 53, in textCB if not text.edit_modified(): File "/usr/local/src/staff/chimera-build/IRIX-X11/foreign/ Python-2.3.2/lib/python2.3/lib-tk/Tkinter.py", line 2831, in edit_modified return self.edit("modified", arg) File "/usr/local/src/staff/chimera-build/IRIX-X11/foreign/ Python-2.3.2/lib/python2.3/lib-tk/Tkinter.py", line 2819, in edit return self._getints( File "/usr/local/src/staff/chimera-build/IRIX-X11/foreign/ Python-2.3.2/lib/python2.3/lib-tk/Tkinter.py", line 972, in _getints return tuple(map(getint, self.tk.splitlist(string))) TypeError: coercing to Unicode: need string or buffer, bool found ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=961805&group_id=5470 From noreply at sourceforge.net Thu May 27 16:52:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Thu May 27 16:52:45 2004 Subject: [ python-Bugs-961805 ] Text.edit_modified() fails Message-ID: Bugs item #961805, was opened at 2004-05-27 12:59 Message generated for change (Comment added) made by ericpett You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=961805&group_id=5470 Category: Tkinter Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Eric P. (ericpett) Assigned to: Martin v. L?wis (loewis) Summary: Text.edit_modified() fails Initial Comment: Text.edit_modified() fails because it calls _getints() on the result of the underlying tk.call(). The tk.call() is returning a boolean, not a string, and the _getints() fails. Here's the last part of an example traceback: File "/usr/local/src/staff/pett/chimera/devel/Ilabel/ __init__.py", line 53, in textCB if not text.edit_modified(): File "/usr/local/src/staff/chimera-build/IRIX-X11/foreign/ Python-2.3.2/lib/python2.3/lib-tk/Tkinter.py", line 2831, in edit_modified return self.edit("modified", arg) File "/usr/local/src/staff/chimera-build/IRIX-X11/foreign/ Python-2.3.2/lib/python2.3/lib-tk/Tkinter.py", line 2819, in edit return self._getints( File "/usr/local/src/staff/chimera-build/IRIX-X11/foreign/ Python-2.3.2/lib/python2.3/lib-tk/Tkinter.py", line 972, in _getints return tuple(map(getint, self.tk.splitlist(string))) TypeError: coercing to Unicode: need string or buffer, bool found ---------------------------------------------------------------------- >Comment By: Eric P. (ericpett) Date: 2004-05-27 13:52 Message: Logged In: YES user_id=1051353 I should mention that it's the argless form of edit_modified() that fails (querying the flag state) -- setting the flag works. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=961805&group_id=5470 From noreply at sourceforge.net Fri May 28 10:04:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 10:04:14 2004 Subject: [ python-Bugs-962226 ] Python 2.3.4 on Linux: test test_grp failed Message-ID: Bugs item #962226, was opened at 2004-05-28 16:04 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962226&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Michael Str?der (stroeder) Assigned to: Nobody/Anonymous (nobody) Summary: Python 2.3.4 on Linux: test test_grp failed Initial Comment: Build on SuSE Linux 9.0: test test_grp failed -- Traceback (most recent call last): File "/home/michael/src/Python-2.3.4/Lib/test/test_grp.py", line 71, in test_errors self.assertRaises(KeyError, grp.getgrnam, fakename) File "/home/michael/src/Python-2.3.4/Lib/unittest.py", line 295, in failUnlessRaises raise self.failureException, excName AssertionError: KeyError ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962226&group_id=5470 From noreply at sourceforge.net Fri May 28 10:26:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 10:26:43 2004 Subject: [ python-Bugs-962226 ] Python 2.3.4 on Linux: test test_grp failed Message-ID: Bugs item #962226, was opened at 2004-05-29 00:04 Message generated for change (Comment added) made by anthonybaxter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962226&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Michael Str?der (stroeder) Assigned to: Nobody/Anonymous (nobody) Summary: Python 2.3.4 on Linux: test test_grp failed Initial Comment: Build on SuSE Linux 9.0: test test_grp failed -- Traceback (most recent call last): File "/home/michael/src/Python-2.3.4/Lib/test/test_grp.py", line 71, in test_errors self.assertRaises(KeyError, grp.getgrnam, fakename) File "/home/michael/src/Python-2.3.4/Lib/unittest.py", line 295, in failUnlessRaises raise self.failureException, excName AssertionError: KeyError ---------------------------------------------------------------------- >Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-29 00:26 Message: Logged In: YES user_id=29957 Do you use YP/NIS? If so, this is a known problem, and is documented on the bugs page for 2.3.4. I really should get around to checking those fixes in... ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962226&group_id=5470 From noreply at sourceforge.net Fri May 28 10:43:10 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 10:43:28 2004 Subject: [ python-Bugs-962226 ] Python 2.3.4 on Linux: test test_grp failed Message-ID: Bugs item #962226, was opened at 2004-05-28 16:04 Message generated for change (Comment added) made by stroeder You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962226&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Michael Str?der (stroeder) Assigned to: Nobody/Anonymous (nobody) Summary: Python 2.3.4 on Linux: test test_grp failed Initial Comment: Build on SuSE Linux 9.0: test test_grp failed -- Traceback (most recent call last): File "/home/michael/src/Python-2.3.4/Lib/test/test_grp.py", line 71, in test_errors self.assertRaises(KeyError, grp.getgrnam, fakename) File "/home/michael/src/Python-2.3.4/Lib/unittest.py", line 295, in failUnlessRaises raise self.failureException, excName AssertionError: KeyError ---------------------------------------------------------------------- >Comment By: Michael Str?der (stroeder) Date: 2004-05-28 16:43 Message: Logged In: YES user_id=64920 I do not use NIS. But I don't know what SuSE really puts into the "compat" NSS module. >From my /etc/nsswitch.conf: passwd: compat group: compat ---------------------------------------------------------------------- Comment By: Anthony Baxter (anthonybaxter) Date: 2004-05-28 16:26 Message: Logged In: YES user_id=29957 Do you use YP/NIS? If so, this is a known problem, and is documented on the bugs page for 2.3.4. I really should get around to checking those fixes in... ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962226&group_id=5470 From noreply at sourceforge.net Fri May 28 16:39:58 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 16:40:02 2004 Subject: [ python-Bugs-962442 ] Konqueror can't render docs because of malformed HTML Message-ID: Bugs item #962442, was opened at 2004-05-28 22:39 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962442&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Klaus-J Wolf (yanestra) Assigned to: Nobody/Anonymous (nobody) Summary: Konqueror can't render docs because of malformed HTML Initial Comment: At least on page Python-Docs-2.3.3/lib/re-syntax.html Konqueror doesn't render the
definition part correctly (the
part is not being displayed). The reason appears to be malformed HTML, as to cite Stephan Kulow of KDE Project: "No, this is not correct HTML and you might want to report it to the python maintainers. I wonder why they use
(which is correct HTML) when they put a

in there (which is not correct). Send your snippet through tidy to see how it looks as correct HTML (and as such it's rendered correctly by konqueror)" You might want to visit KDE's bugzilla: http://bugs.kde.org/show_bug.cgi?id=82335 ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962442&group_id=5470 From noreply at sourceforge.net Fri May 28 17:46:55 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Fri May 28 17:47:02 2004 Subject: [ python-Bugs-962471 ] PyModule_AddIntConstant documented to take an int, not a lon Message-ID: Bugs item #962471, was opened at 2004-05-28 21:46 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962471&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Matthias Klose (doko) Assigned to: Nobody/Anonymous (nobody) Summary: PyModule_AddIntConstant documented to take an int, not a lon Initial Comment: [forwarded from http://bugs.debian.org/250826] In 7.5.4 (Module Objects) PyModule_AddIntConstant is documented to take an int as third parameter, however it takes an long. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962471&group_id=5470 From noreply at sourceforge.net Sat May 29 01:35:51 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 29 01:35:55 2004 Subject: [ python-Bugs-962602 ] misspelling -- warngin -> warning Message-ID: Bugs item #962602, was opened at 2004-05-29 05:35 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962602&group_id=5470 Category: Parser/Compiler Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Evil Mr Henry (evilmrhenry) Assigned to: Nobody/Anonymous (nobody) Summary: misspelling -- warngin -> warning Initial Comment: distutils/filelist.py, line 169, "warning" is spelled "warngin". ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962602&group_id=5470 From noreply at sourceforge.net Sat May 29 03:32:04 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 29 03:32:08 2004 Subject: [ python-Bugs-962631 ] Typo in Lib/test/test_sax.py can confuse Message-ID: Bugs item #962631, was opened at 2004-05-29 01:32 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962631&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Bryan Blackburn (blb) Assigned to: Nobody/Anonymous (nobody) Summary: Typo in Lib/test/test_sax.py can confuse Initial Comment: Not a major bug, but can be confusing...when test_sax.py is run verbosely, it'll say 'Failed' for passing tests. Patch to fix attached. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962631&group_id=5470 From noreply at sourceforge.net Sat May 29 03:41:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 29 03:41:19 2004 Subject: [ python-Bugs-962633 ] macfs and macostools tests fail on UFS Message-ID: Bugs item #962633, was opened at 2004-05-29 01:41 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962633&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Bryan Blackburn (blb) Assigned to: Nobody/Anonymous (nobody) Summary: macfs and macostools tests fail on UFS Initial Comment: Two Mac-specific tests (macfs and macostools) will fail if 'make test' is run on a UFS volume. macfs fails since SetDates() doesn't affect all timestamps for UFS like it does on HFS+. This causes GetDates() to return unexpected values, hence the failure. macostools fails for similar reasons, but related to forks, since UFS doesn't have any. Not sure if this is something to be fixed (few use UFS) or should simply be pointed out in documentation. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962633&group_id=5470 From noreply at sourceforge.net Sat May 29 03:43:06 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 29 03:43:17 2004 Subject: [ python-Bugs-924301 ] A leak case with cmd.py & readline & exception Message-ID: Bugs item #924301, was opened at 2004-03-27 01:28 Message generated for change (Comment added) made by svenil You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924301&group_id=5470 Category: Python Library Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Sverker Nilsson (svenil) Assigned to: Michael Hudson (mwh) Summary: A leak case with cmd.py & readline & exception Initial Comment: A leak to do with readline & cmd, in Python 2.3. I found out what hold on to my interactive objects too long ('for ever') in certain circumstances. The circumstance had to do with an exception being raised in Cmd.cmdloop and handled (or not handled) outside of Cmd.cmdloop. In cmd.py, class Cmd, in cmdloop(), if an exception is raised and propagated out from the interior of cmdloop, the function postloop() is not called. The default function of this, (in 2.3) when the readline library is present, is to restore the completer, via: readline.set_completer(self.old_completer) If this is not done, the newly (by preloop) inserted completer will remain. Even if the loop is called again and run without exception, the new completer will remain, because then in postloop the old completer will be set to our new completer. When we exit, the completer will remain the one we set. This will hold on to our object, aka 'leak'. - In cmd.py in 2.2 no attempt was made to restore the completer, so this was also a kind of leak, but it was replaced the next time a Cmd instance was initialized. Now, however, the next time we will not replace the old completer, but both of them will remain in memory. The old one will be stored as self.old_completer. If we terminate with an exception, bad luck... the stored completer retains both of the instances. If we terminate normally, the old one will be retained. In no case do we restore the space of the first instance. The only way that would happen, would be if the second instance first exited the loop with an exception, and then entered the loop again an exited normally. But then, the second instance is retained instead! If each instance happens to terminate with an exception, it seems well possible that an ever increasing chain of leaking instances will be accumulated. My fix is to always call the postloop, given the preloop succeeded. This is done via a try:finally clause. def cmdloop(self, intro=None): ... self.preloop() try: ... finally: # Make sure postloop called self.postloop() I am attaching my patched version of cmd.py. It was originally from the tarball of Python 2.3.3 downloaded from Python.org some month or so ago in which cmd.py had this size & date: 14504 Feb 19 2003 cmd.py Best regards, Sverker Nilsson ---------------------------------------------------------------------- >Comment By: Sverker Nilsson (svenil) Date: 2004-05-29 09:43 Message: Logged In: YES user_id=356603 I couldn't find a new attached file. I acknowledge some problems with my original patch, but have no other suggestion at the moment. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-26 18:36 Message: Logged In: YES user_id=6656 What do you think of the attached? This makes the documentation of pre & post loop more accurate again, which I think is nice. ---------------------------------------------------------------------- Comment By: Michael Hudson (mwh) Date: 2004-05-19 11:38 Message: Logged In: YES user_id=6656 This is where I go "I wish I'd reviewed that patch more carefully". In particular, the documentation of {pre,post}loop is now out of date. I wonder setting/getting the completer in these functions was a good idea. Hmm. This bug report confuses me :-) but I can certainly see the intent of the patch... ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-19 03:52 Message: Logged In: YES user_id=80475 Michael, this touches some of your code. Do you want to handle this one? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=924301&group_id=5470 From noreply at sourceforge.net Sat May 29 10:58:27 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 29 10:58:30 2004 Subject: [ python-Bugs-962772 ] when both maintainer and author provided, author discarded Message-ID: Bugs item #962772, was opened at 2004-05-29 23:58 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962772&group_id=5470 Category: Distutils Group: None Status: Open Resolution: None Priority: 5 Submitted By: John Belmonte (jbelmonte) Assigned to: Nobody/Anonymous (nobody) Summary: when both maintainer and author provided, author discarded Initial Comment: When both maintainer and author contacts are passed to distutil's setup, the resulting PKG-INFO file has only an author field (set to the maintainer contact!). The original author contact information is discarded. Both maintainer and author contacts should be preserved. If PKG-INFO doesn't support a maintainer contact, it should be added to the spec. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962772&group_id=5470 From noreply at sourceforge.net Sat May 29 17:54:01 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 29 17:54:05 2004 Subject: [ python-Bugs-962918 ] 2.3.4 can not be installed over 2.3.3 Message-ID: Bugs item #962918, was opened at 2004-05-29 23:54 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962918&group_id=5470 Category: Build Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Jarek Zgoda (zgoda) Assigned to: Nobody/Anonymous (nobody) Summary: 2.3.4 can not be installed over 2.3.3 Initial Comment: When trying to install newly-compiled 2.3.4 over existing 2.3.3 I ran into "libinstall error" on zipfile.py. After removing whole /usr/lib/python2.3 and installing from scratch the installation was succesful. I think it's some omission from installer script, since I didn't have any problem when I upgraded 2.3.0 to 2.3.2 ant thet to 2.3.3. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962918&group_id=5470 From noreply at sourceforge.net Sat May 29 19:36:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sat May 29 19:36:50 2004 Subject: [ python-Feature Requests-935915 ] os.nullfilename Message-ID: Feature Requests item #935915, was opened at 2004-04-16 05:44 Message generated for change (Comment added) made by jbelmonte You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=935915&group_id=5470 >Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: John Belmonte (jbelmonte) Assigned to: Nobody/Anonymous (nobody) Summary: os.nullfilename Initial Comment: Just as the os library provides os.sep, etc., for the current OS, it should provide the name of the null file (e.g., "/dev/null" or "nul:"), so that there is a portable way to open a null file. Use of an object such as class nullFile: def write(self, data): pass is not sufficient because it does not provide a full file object interface (no access to file descriptor, etc.). See discussion at . ---------------------------------------------------------------------- >Comment By: John Belmonte (jbelmonte) Date: 2004-05-30 08:36 Message: Logged In: YES user_id=282299 Attaching patch against Python HEAD, including docs and test. ---------------------------------------------------------------------- Comment By: John Belmonte (jbelmonte) Date: 2004-05-21 23:46 Message: Logged In: YES user_id=282299 I do intend to make a patch, but it may be some time before I get to it. Please give me a few weeks. If someone else would like to step in, that is fine, just let me know before you start the work. ---------------------------------------------------------------------- Comment By: Raymond Hettinger (rhettinger) Date: 2004-05-21 15:00 Message: Logged In: YES user_id=80475 Consider mentioning this on comp.lang.python. Perhaps someone will volunteer to write a patch. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-05-10 01:08 Message: Logged In: YES user_id=21627 Would you like to work on a patch? ---------------------------------------------------------------------- Comment By: David Albert Torpey (dtorp) Date: 2004-05-09 10:54 Message: Logged In: YES user_id=681258 I like this idea. It is more portable. ---------------------------------------------------------------------- Comment By: Martin v. L?wis (loewis) Date: 2004-04-16 05:52 Message: Logged In: YES user_id=21627 Move to feature requests tracker ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=935915&group_id=5470 From noreply at sourceforge.net Sun May 30 15:52:23 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 30 15:52:28 2004 Subject: [ python-Bugs-963246 ] Multiple Metaclass inheritance limitation Message-ID: Bugs item #963246, was opened at 2004-05-30 11:52 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=963246&group_id=5470 Category: Python Interpreter Core Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Greg Chapman (glchapman) Assigned to: Nobody/Anonymous (nobody) Summary: Multiple Metaclass inheritance limitation Initial Comment: I'm not sure if this constitutes a bug or a limitation (which should be documented?), but the following doesn't work with Python 2.3.4. Assume CMeta is a type defined in C, with tp_base = PyType_Type tp_new = some C function which uses the mro to call the inherited tp_new tp_flags includes Py_TPFLAGS_BASETYPE class PyMeta(type): def __new__(meta, name, bases, attrs): return super(PyMeta, meta).__new__(meta, name, bases, attrs) class MetaTest(CMeta, PyMeta): pass class Test: __metaclass__ = MetaTest The attempt to define Test generates a TypeError: "type.__new__(MetaTest) is not safe, use CMeta.__new__()". The above error is generated (in tp_new_wrapper) by the super call in PyMeta.__new__, but this is only reached as a result of an initial call to CMeta.tp_new (which, using the mro to find the next "__new__" method, finds and calls PyMeta.__new__). It may be there is no good way to allow the above scenario, but I just thought I'd point it out in case someone can think of a workaround. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=963246&group_id=5470 From noreply at sourceforge.net Sun May 30 19:21:08 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 30 19:21:19 2004 Subject: [ python-Feature Requests-481171 ] Hang using files named prn.txt, etc Message-ID: Feature Requests item #481171, was opened at 2001-11-13 05:43 Message generated for change (Comment added) made by davidhopwood You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=481171&group_id=5470 Category: None Group: None Status: Open Resolution: Wont Fix Priority: 5 Submitted By: Andrew Bennetts (spiv) Assigned to: Nobody/Anonymous (nobody) Summary: Hang using files named prn.txt, etc Initial Comment: Windows reserves many possible filenames as reserved device names. Examples are prn.xxx, con.xxx, com1.xxx aux.xxx, etc, where xxx can be any extension. Attempts to create/read/write to these files can hang the interpreter, but ideally would just throw an OSError. A partial list of device names is mentioned at: http://cert.uni- stuttgart.de/archive/bugtraq/2001/07/msg00086.html ---------------------------------------------------------------------- Comment By: David Hopwood (davidhopwood) Date: 2004-05-31 00:21 Message: Logged In: YES user_id=1053197 Device filenames can be detected by creating an empty directory, and testing whether the filename exists in that directory: > md c:\empty > if exist c:\empty\lpt1.txt echo foo foo > if exist c:\empty\lpt1.txt\lpt1.txt echo foo foo > if exist c:\empty\harmless echo foo > But note: > if exist c:\empty\c:\lpt1.txt echo foo > if exist c:\empty\:\lpt1.txt echo foo > I don't know why filenames containing ':' are treated differently, but it might be wise to strip any ':' characters from the path first. ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2001-11-14 05:51 Message: Logged In: YES user_id=31435 It's cool to reopen it, but I reclassified it as a feature request and will add it to PEP 42. DDNs can be constructed dynamically, so no fixed list of magic names is adequate. I don't have the knowledge needed to do what is adequate, so the only hope for this is that somebody in the Python community who does will notice and volunteer their time to fix it. In the meantime, you might want to install a real operating system . ---------------------------------------------------------------------- Comment By: Andrew Bennetts (spiv) Date: 2001-11-14 01:20 Message: Logged In: YES user_id=50945 Yes, I read the message (and most of the thread) :) Apologies for re-opening this bug, but I believe that something can still be done about it. I'm using Windows 2K (SP 2), so those patches don't apply to me. Those patches merely prevent windows crashing on certain types of accesses to devices, e.g. the now well-known C:\NUL\NUL bug. They don't prevent applications from hanging when trying to access prn.txt. Further in the thread starting at that URL, there is some discussion of possible work-arounds for safely opening/creating files. I'm not a Win32 C programmer, so I've no idea about the technical feasibility of these solutions in Python. While I agree that the real problem is in the OS, it seems to be standard behaviour now for apps to workaround it. For example, Textpad (www.textpad.com) will just pop up a dialog saying "You cannot create this file: this name is reserved device name" or something similar. Whether they've just hard-coded a list of known names or if they're doing something more advanced I don't know. This caused a fair bit of confusion for me recently: Python hung inexplicably in splitting a CSV file of stock data into seperate files, and one of the Australian Stock Exchange's stock symbols is "PRN". The script was extremely simple, so I eventually realised what was going on, but I dread an unwary person having to debug this problem in a larger application. If you think this sort of ugliness doesn't belong in Python then feel free to close this bug again, and I won't feel offended :) ---------------------------------------------------------------------- Comment By: Tim Peters (tim_one) Date: 2001-11-13 17:41 Message: Logged In: YES user_id=31435 Did you read the message you referenced with that URL? I agree with it: """ Conclusion : applications should not filter out DDNs, because they don't fix the problem (basically they make it even worse), the OS patch is better because it fixes *ALL* problems, ... CONCLUSION : patch your OS, and stop whining about so called 'bugs' in applications, you will never be able to completely patch the problem that way. """ They're right: applications cannot fix this; if it's a problem for you, then you should download and apply the MS OS patches linked to from that article. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=355470&aid=481171&group_id=5470 From noreply at sourceforge.net Sun May 30 19:50:15 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Sun May 30 19:50:18 2004 Subject: [ python-Bugs-963321 ] Acroread aborts printing PDF documentation Message-ID: Bugs item #963321, was opened at 2004-05-30 23:50 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=963321&group_id=5470 Category: Documentation Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Howard B. Golden (hgolden) Assigned to: Nobody/Anonymous (nobody) Summary: Acroread aborts printing PDF documentation Initial Comment: Using acroread 5.08 under Linux 2.6.5, the May 20, 2004 version of the documentation (release 2.3.4) is displayed successfully. However, when I attempt to print the documentation, acroread aborts (prints "Aborted" on standard error). (It is possible that this is an acroread bug, but I'm reporting it here in case it indicates a problem with PDF files generated by the documentation building process.) ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=963321&group_id=5470 From noreply at sourceforge.net Mon May 31 05:39:38 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 05:39:49 2004 Subject: [ python-Bugs-963494 ] packman upgrade issue Message-ID: Bugs item #963494, was opened at 2004-05-31 11:39 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=963494&group_id=5470 Category: Macintosh Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Ronald Oussoren (ronaldoussoren) Assigned to: Jack Jansen (jackjansen) Summary: packman upgrade issue Initial Comment: When you upgrade a package using packman the installer doesn't remove the old version before installing the new version. The end result is that old files might interfere with the correct operation of the upgraded package. I ran into this with an upgrade from PyObjC 1.0 to PyObjC 1.1. Some extension modules have moved between those two releases. When upgrading using PackMan the old extension modules still exists, and cause problems when you try to use the package. Three possible solutions: 1) Remove the existing package directory before installing the upgrade 2) Add pre/post install/upgrade scripts 3) Use a real package database ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=963494&group_id=5470 From noreply at sourceforge.net Mon May 31 11:19:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 11:19:10 2004 Subject: [ python-Bugs-962602 ] misspelling -- warngin -> warning Message-ID: Bugs item #962602, was opened at 2004-05-29 07:35 Message generated for change (Comment added) made by doerwalter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962602&group_id=5470 Category: Parser/Compiler Group: Python 2.3 >Status: Closed >Resolution: Fixed Priority: 5 Submitted By: Evil Mr Henry (evilmrhenry) Assigned to: Nobody/Anonymous (nobody) Summary: misspelling -- warngin -> warning Initial Comment: distutils/filelist.py, line 169, "warning" is spelled "warngin". ---------------------------------------------------------------------- >Comment By: Walter D?rwald (doerwalter) Date: 2004-05-31 17:19 Message: Logged In: YES user_id=89016 Fix checked in as Lib/distutils/filelist.py 1.16. Thanks for the report. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962602&group_id=5470 From noreply at sourceforge.net Mon May 31 17:09:20 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 17:10:27 2004 Subject: [ python-Bugs-962918 ] 2.3.4 can not be installed over 2.3.3 Message-ID: Bugs item #962918, was opened at 2004-05-29 23:54 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962918&group_id=5470 Category: Build Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Jarek Zgoda (zgoda) Assigned to: Nobody/Anonymous (nobody) Summary: 2.3.4 can not be installed over 2.3.3 Initial Comment: When trying to install newly-compiled 2.3.4 over existing 2.3.3 I ran into "libinstall error" on zipfile.py. After removing whole /usr/lib/python2.3 and installing from scratch the installation was succesful. I think it's some omission from installer script, since I didn't have any problem when I upgraded 2.3.0 to 2.3.2 ant thet to 2.3.3. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 23:09 Message: Logged In: YES user_id=21627 What operating system are you using? Can you report the precise wording of the error that you get, and the command that you have invoked to trigger this error? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=962918&group_id=5470 From noreply at sourceforge.net Mon May 31 17:12:37 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 17:12:42 2004 Subject: [ python-Bugs-957485 ] os.lseek() doesn't work Message-ID: Bugs item #957485, was opened at 2004-05-20 18:11 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957485&group_id=5470 Category: Python Library Group: Python 2.3 >Status: Closed >Resolution: Invalid Priority: 5 Submitted By: michal cernoevic (cernoevic) Assigned to: Nobody/Anonymous (nobody) Summary: os.lseek() doesn't work Initial Comment: >>> f=file('.htaccess', 'r+') >>> s=f.readline() >>> f=os.lseek(f,0,0) Traceback (most recent call last): File "", line 1, in ? TypeError: an integer is required SunOS wwwinfoa 5.8 Generic_108528-24 sun4u sparc SUNW,Ultra-4 64-bit The nulls are really nuls. ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 23:12 Message: Logged In: YES user_id=21627 This is not a bug in Python; you are using lstat incorrectly. The first argument for lstat must be an integer, not a file object. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=957485&group_id=5470 From noreply at sourceforge.net Mon May 31 17:15:14 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 17:15:18 2004 Subject: [ python-Bugs-944394 ] No examples or usage docs for urllib2 Message-ID: Bugs item #944394, was opened at 2004-04-29 13:02 Message generated for change (Comment added) made by loewis You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944394&group_id=5470 Category: Python Library Group: None Status: Open Resolution: None Priority: 5 Submitted By: Chris Withers (fresh) Assigned to: Nobody/Anonymous (nobody) Summary: No examples or usage docs for urllib2 Initial Comment: Hi there, I'm sure I reported this before, but it's a couple of major releases later, and there's still no usage docs for urllib2. The examples given are too trivial to be helpful, but I'm guessing people are using the module so there must be some examples out there somewhere ;-) With a bit o fhelp from Moshez, I found the docstring in the module source. At the very least, it'd be handy if that appeared somewhere at: http://docs.python.org/lib/module-urllib2.html But really, mroe extensive and helpful documentation on this cool new module would be very handy. Chris ---------------------------------------------------------------------- >Comment By: Martin v. L?wis (loewis) Date: 2004-05-31 23:15 Message: Logged In: YES user_id=21627 Are you willing to provide examples? ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=944394&group_id=5470 From noreply at sourceforge.net Mon May 31 17:16:49 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 17:16:52 2004 Subject: [ python-Bugs-963825 ] Distutils should be able to produce Debian packages (.deb) Message-ID: Bugs item #963825, was opened at 2004-05-31 23:16 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=963825&group_id=5470 Category: Distutils Group: Feature Request Status: Open Resolution: None Priority: 5 Submitted By: Kaleissin (kaleissin) Assigned to: Nobody/Anonymous (nobody) Summary: Distutils should be able to produce Debian packages (.deb) Initial Comment: Distutils should be able to produce Debian packages (.deb), it's the Other[TM] large linux package-format after all. It might perhaps be done by using "alien" on an .rpm, but for the simple cases of pure Python modules and packages it ought to be handled internally. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=963825&group_id=5470 From noreply at sourceforge.net Mon May 31 17:49:12 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 17:49:16 2004 Subject: [ python-Bugs-963845 ] There ought to be a way to uninstall Message-ID: Bugs item #963845, was opened at 2004-05-31 23:49 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=963845&group_id=5470 Category: Distutils Group: Feature Request Status: Open Resolution: None Priority: 5 Submitted By: Kaleissin (kaleissin) Assigned to: Nobody/Anonymous (nobody) Summary: There ought to be a way to uninstall Initial Comment: There ought to be a way to uninstall, if for no other reason that it is polite behavior in a civilized society :) The usual way to uninstall something from a "package" seems to be to keep a list of all installed files somewhere. Doing the equivalent of a "find . -print" at the right point during "bdist/build" should do the trick, or maybe a command of its own, the filelist to be stored in the sdist-result? Such a list of files would also be useful for packagers, be it maintainers of .rpm, .deb or *BSD ports. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=963845&group_id=5470 From noreply at sourceforge.net Mon May 31 22:18:03 2004 From: noreply at sourceforge.net (SourceForge.net) Date: Mon May 31 22:18:07 2004 Subject: [ python-Bugs-963956 ] Bad error mesage when subclassing a module Message-ID: Bugs item #963956, was opened at 2004-06-01 02:18 Message generated for change (Tracker Item Submitted) made by Item Submitter You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=963956&group_id=5470 Category: Type/class unification Group: Python 2.3 Status: Open Resolution: None Priority: 5 Submitted By: Edward C. Jones (edcjones) Assigned to: Nobody/Anonymous (nobody) Summary: Bad error mesage when subclassing a module Initial Comment: I made a common mistake and ended up trying to subclass a module. Here are two small python files: ---- a.py: class a(object): pass ---- b.py: import a class B(a): # Should be "a.a". pass ---- If I run "python b.py" I get the uninformative error message: Traceback (most recent call last): File "./b.py", line 3, in ? class B(a): TypeError: function takes at most 2 arguments (3 given) I think the message should say that "a" is not a class. ---------------------------------------------------------------------- You can respond by visiting: https://sourceforge.net/tracker/?func=detail&atid=105470&aid=963956&group_id=5470